A longitudinal assessment of therapist multicultural competence process and outcome

Therapist multicultural competence (MCC) has been utilized as a tool to reduce prevalent health disparities. Numerous MCC theories have been developed and the American Psychological Association has adopted MCC guidelines aimed to infuse multicultural initiatives into psychology training programs. Over the last decade, we have begun to develop a vital empirical base supporting the utility of therapist MCC. There is now strong evidence of the relationship between increased therapist MCC and improved clinical outcomes. However, the meaning of the MCC and outcome correlation remains ambiguous due to issues in interpreting a process-outcome relationship in a dyadic interaction. The present study sought to clarify the therapist MCC and outcome correlation by utilizing multilevel models to (1) measure the amount of variability attributed to the therapist versus the client in client-rated therapist MCC and (2) assess the different sources of MCC variability (client/therapist) in relationship to client outcomes. Using a large repeated measures dataset (n = 1,458 clients; k = 35 therapists; w = 8,497 observations) from a university counseling center, the present study found a small amount of variability in MCC ratings was attributed to the therapist. This small level of variability was consistent across client demographics (race, sexual orientation, and religious affiliation). In examining MCC and outcome, there was a near zero relationship between client-rated therapist MCC and client outcomes. Again, these results were consistent across client demographics. In contrast, ratings of the therapeutic alliance and client treatment satisfaction significantly predicted client outcomes. The implications of these findings, including a discussion of MCC measurement and future research directions, are discussed

A LONGITUDINAL ASSESSMENT OF THERAPIST MULTICULTURAL COMPETENCE PROCESS AND OUTCOME by Brian T. Pace A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Educational Psychology The University of Utah December 2018 Copyright © Brian T. Pace 2018 All Rights Reserved The University of Utah Graduate School STATEMENT OF DISSERTATION APPROVAL The dissertation of Brian T. Pace has been approved by the following supervisory committee members: Zac E. Imel , Chair 04/30/2018 Date Approved David Robert Davies , Member 04/30/2018 Date Approved Lois Huebner , Member 04/30/2018 Date Approved John C. Kircher , Member 04/30/2018 Date Approved Karen W. Tao , Member 04/30/2018 Date Approved and by the Department/College/School of Anne Cook , Chair/Dean of Educational Psychology and by David B. Kieda, Dean of The Graduate School. ABSTRACT Therapist multicultural competence (MCC) has been utilized as a tool to reduce prevalent health disparities. Numerous MCC theories have been developed and the American Psychological Association has adopted MCC guidelines aimed to infuse multicultural initiatives into psychology training programs. Over the last decade, we have begun to develop a vital empirical base supporting the utility of therapist MCC. There is now strong evidence of the relationship between increased therapist MCC and improved clinical outcomes. However, the meaning of the MCC and outcome correlation remains ambiguous due to issues in interpreting a process-outcome relationship in a dyadic interaction. The present study sought to clarify the therapist MCC and outcome correlation by utilizing multilevel models to (1) measure the amount of variability attributed to the therapist versus the client in client-rated therapist MCC and (2) assess the different sources of MCC variability (client/therapist) in relationship to client outcomes. Using a large repeated measures dataset (n = 1,458 clients; k = 35 therapists; w = 8,497 observations) from a university counseling center, the present study found a small amount of variability in MCC ratings was attributed to the therapist. This small level of variability was consistent across client demographics (race, sexual orientation, and religious affiliation). In examining MCC and outcome, there was a near zero relationship between client-rated therapist MCC and client outcomes. Again, these results were consistent across client demographics. In contrast, ratings of the therapeutic alliance and client treatment satisfaction significantly predicted client outcomes. The implications of these findings, including a discussion of MCC measurement and future research directions, are discussed. iv This dissertation is dedicated to those engaged in the nonviolent fight for equality and social change. “Not everything that is faced can be changed, but nothing can be changed until it is faced.” -James A. Baldwin TABLE OF CONTENTS ABSTRACT ....................................................................................................................... iii LIST OF TABLES .............................................................................................................. x LIST OF FIGURES ........................................................................................................... xi ACKNOWLEDGEMENTS .............................................................................................. xii Chapters 1 INTRODUCTION ........................................................................................................ 1 Mental Health Disparities .......................................................................................... 4 Efforts to Address Disparities .................................................................................... 8 Definitions of Key Terms ................................................................................. 9 Brief History of the Multicultural Movement .......................................................... 10 The 2003 APA Multicultural Guidelines ................................................................. 12 The 2008 APA Multicultural Guidelines Implementation Report ........................... 15 Efforts to Establish a Multicultural Competencies Framework ............................... 18 Empirical Basis for Therapist Multicultural Competence ....................................... 21 MCC Measurement Tools ............................................................................... 22 Empirical Support for Therapist MCC and Psychotherapy Process-Outcome 25 Issues in Measuring the Dyad in Psychotherapy Process-Outcome Research......... 28 Variance Partitioning in Dyadic Interactions .................................................. 29 Variance Partitioning in Therapist MCC Research ........................................ 34 Untangling Process and Outcome Correlations ....................................................... 37 MCC Correlated Variability With Outcomes ................................................. 40 Summary and Next Steps ......................................................................................... 41 Study Rationale and Hypotheses ............................................................................. 43 Hypothesis 1: Sources of Variability in Therapist MCC Ratings ................... 45 Hypothesis 2: Therapist MCC Variability and Clinical Outcomes ................ 45 2 METHOD ................................................................................................................... 48 Participants and Procedure ....................................................................................... 48 Measures .................................................................................................................. 50 Client Psychological Distress ......................................................................... 50 Therapist Multicultural Competence (MCC) .................................................. 51 The Working Alliance..................................................................................... 51 Treatment Satisfaction .................................................................................... 52 Statistical Analyses .................................................................................................. 52 Analysis 1: Sources of Variability in Therapist MCC Ratings ....................... 55 Analysis 2: Therapist MCC Variability and Clinical Outcomes .................... 57 3 RESULTS ................................................................................................................... 60 Client Level Descriptive Statistics ........................................................................... 60 Therapist Level Descriptive Statistics ..................................................................... 61 Variance Partitioning ............................................................................................... 61 Exploratory Variance Partitioning Across Sexual Orientation and Religious Identities ................................................................................................................... 63 Client Sexual Orientation ................................................................................ 63 Client Religious Affiliation............................................................................. 64 Relationship Between MCC, Alliance, and Treatment Satisfaction With Outcome 65 Test of Within and Between Relationships Using Multilevel Models .................... 66 MCC Ratings .................................................................................................. 67 Alliance Ratings .............................................................................................. 67 Treatment Satisfaction Ratings ....................................................................... 68 Relationship Between Process and Outcome Among Client Sexual Orientation .... 69 MCC Ratings .................................................................................................. 70 Alliance Ratings .............................................................................................. 70 Treatment Satisfaction Ratings ....................................................................... 71 Relationship Between Process and Outcome Among Client Religious Status ........ 71 MCC Ratings .................................................................................................. 71 Alliance Ratings .............................................................................................. 72 Treatment Satisfaction Ratings ....................................................................... 72 4 DISCUSSION ............................................................................................................. 84 Variability in MCC Ratings Attributed to the Therapist Was Small ....................... 85 A Bulk of the Variability in MCC Ratings Was Attributed to the Client ....... 86 Few Differences in MCC ICCs Across Client Cultural Identities .................. 88 Little Evidence for a Relationship Between MCC Ratings and Clinical Outcomes 89 The Steadiness of the Empirical MCC and Outcome Research Base ............. 91 Alliance and Treatment Satisfaction Ratings Predicted Clinical Outcome ............. 92 Summary .................................................................................................................. 95 Three Key Findings......................................................................................... 95 Limitations ............................................................................................................... 96 The MCC Items............................................................................................... 96 Method of MCC Measurement ....................................................................... 97 Sample Diversity ............................................................................................. 97 Heterogeneity Within Client Identity Subsamples ......................................... 98 Naturalistic Data ............................................................................................. 99 Future Directions ................................................................................................... 100 viii Continue to Test MCC Measures Across Diverse Samples.......................... 100 Observer Ratings of Therapist MCC ............................................................ 101 Examining Raw Therapy Data ...................................................................... 101 REFERENCES ............................................................................................................... 103 ix LIST OF TABLES Tables 1 The APA’s Six Multicultural Guidelines ................................................................. 47 2 Means and Standard Deviations for Each Process Measure .................................... 73 3 Intraclass Correlation Coefficients (ICCs) and Density Intervals ........................... 76 4 Sexual Orientation Intraclass Correlation Coefficients and Density Intervals ........ 78 5 Religious Intraclass Correlation Coefficients and Density Intervals ....................... 78 6 Zero-Order Correlations Between Process Measures and Client Outcomes ........... 79 7 Within and Between Outcome Results Stratified by Client R-EM Status ............... 81 8 Within and Between Outcome Results Stratified by Client’s Sexual Orientation ... 82 9 Within and Between Outcome Results Stratified by Client’s Religious Affiliation 83 LIST OF FIGURES Figures 1 Distribution of client responses to each process variable .........................................73 2 Distribution of mean therapist rating on each process variable ................................74 3 Posterior distributions for each level of analysis by each process variable ..............75 4 Model-derived therapist random effects estimates ...................................................77 5 Simple scatterplots for each process variable ...........................................................80 ACKNOWLEDGEMENTS Although this document is the culmination of my doctoral work, many individuals have supported me along the way and shaped my identity as a psychotherapist and researcher. First and foremost, I want to thank my family. I was fortunate to have been raised by parents that strongly encouraged me to pursue my passions. My siblings have always been a source of support for me and I would not trade our relationships for anything. I also want to thank a few of my early mentors, including Odessa Cole who encouraged me during my undergraduate studies to volunteer at the UW-Madison psychotherapy training clinic. It was at this clinic that I learned the career path I wanted to pursue. I also want to thank Simon Goldberg for his patience in teaching me how to conduct a meta-analysis over long Saturdays on the 13th floor of the Educational Sciences building in Madison, Wisconsin. Finally, I want to thank my advisors and mentors over the years: Takuya Minami, Bruce Wampold, and Zac Imel, whose impact on my development as a psychotherapist and researcher is not taken for granted. CHAPTER 1 INTRODUCTION There is consistent evidence of disparities in both the quality and access to mental health treatment among racial and ethnic minority (R-EM) groups (Institute of Medicine, 2003; Substance Abuse and Mental Health Administration, 2015). A national survey found that compared to White individuals, R-EM individuals accessed treatment less often and had lower quality of mental health care (Alegria et al., 2008). With the recent passage of the Affordable Care Act, there has been an increase in overall mental health treatment utilization, however, disparities have remained (Creedon & Lê Cook, 2016). A primary strategy for reducing disparities has involved infusing multicultural competence (MCC) paradigms into psychology training programs. The governing body of psychology, the American Psychological Association (APA), has called for the implementation of multicultural competencies into psychology training programs, including education, practice, research, and policy domains (APA, 2003, 2008). Many mental health counselors now train within MCC frameworks that foster attitudes and beliefs, knowledge, and skills in working with individuals from diverse backgrounds (Pieterse, Evans, Risner-Butner, Collins, & Mason, 2009). The training process is ongoing and aspirational in nature and seeks to “create conditions that maximize the optimal development of client and client systems” (D. W. Sue & Sue, 2012, p. 46). 2 Given this broad emphasis on MCC training in mental health, it is important to conduct detailed research on therapist MCC, its impact on treatment outcome, and if ratings of MCC correspond to improved measures of mental health treatment and outcome. There is now solid initial evidence that MCC is correlated with other indicators of quality treatment and treatment outcome. In a recent meta-analysis of 18 studies (N = 2,191) there was a significant correlation between increased MCC ratings and improved treatment outcome (r = .29, MCC ratings accounting for approximately 8% of the variability in outcome) as well as significant relationships between MCC ratings and other measures of treatment process (r = .75; Tao, Owen, Pace, & Imel, 2015). Despite these promising findings, the meaning of the correlation between MCC and measures of treatment process and outcome is ambiguous. There are two primary issues when interpreting the relationship of a treatment process measure (i.e., a measure of what happens during a therapy interaction) to a treatment outcome in psychotherapy (see DeRubeis, Brotman, & Gibbons, 2005). First, during a dyadic interaction, variability in a measure targeting a specific person can actually be the result of features of the interaction itself - and not the person who is purportedly being targeted by the measure. Research has not yet clearly established the primary source of variability in MCC measures. Essentially, MCC measures may provide information about whether therapists differ from each other in their cultural competence, or scores may be driven by patient-topatient variability. If the source of MCC variability is between therapists, then this suggests that MCC measures provide useful information about which therapists are effectively navigating cultural factors during session relative to other therapists. In contrast, if the variability is within therapists’ caseloads (i.e., patient-to-patient), then the 3 measure is simply picking up client differences in ratings and little can be gleaned about therapists’ differences in addressing cultural factors during session. Second, it is not clear how these different sources of variability relate to treatment outcome (i.e., does Therapist A’s increased MCC rating correlate with better client outcomes or are therapist differences in MCC unrelated to outcome). If patient-to-patient variability is most associated with treatment outcome, this would suggest that efforts to improve therapist MCC might not impact the quality of treatment received by R-EM clients. Using a large dataset (n = 1,458 clients; k = 35 therapists; w = 8,497 observations) collected at the University of Utah Counseling Center, the purpose of the proposed project is to clarify the meaning of the correlation between MCC and outcome in psychotherapy. These data consist of repeated observations of client-rated therapist MCC, treatment satisfaction, the working alliance, and clinical distress. These data were collected prior to every clinical contact. This design provides the power to utilize advanced statistical models that can partition the variability in ratings between therapist and client, as well as correlate this disaggregated variability with treatment outcome. These analyses will directly address the shortcomings in MCC research that have commonly relied on correlating one client rating of therapist MCC with the client’s posttreatment distress score. This cross-sectional design does not provide the ability to partition variability at both the therapist and client level and correlate these separate sources of variability with outcome. The present study’s results may have important clinical implications. For example, if the MCC measure detects differences in therapist MCC, and this variability correlates with treatment outcome, then we can potentially work to identify specific therapist MCC behaviors, and shape clinical training and 4 supervision going forward. The present dissertation project is presented in four chapters. In the first and current chapter, I will discuss mental health disparities, outline efforts that have been undertaken to address these disparities, and examine the empirical support for therapist MCC. Additionally, I will describe measurement concerns in process-outcome research and how these will be addressed in the present study to clarify the MCC and outcome relationship. In the second chapter, I will introduce the study method, including the sample demographics, process measures, and statistical analyses. The third chapter will describe the results across all study analyses. In the fourth and final chapter, I will discuss the results and implications as they relate to MCC theory, measurement, and clinical practice. Mental Health Disparities There is consistent evidence that indicates racial-ethnic minority (R-EM) disparities in both access to mental health care and quality of care received, while provider discrimination constitutes a major barrier to reducing disparities (Alegría et al., 2008; Ault-Brutus, 2012; Broman, 2012; Chen & Rizzo, 2010; Creedon & Le Cook, 2016; Institute of Medicine, 2003; Manseau & Case, 2014; McGuire & Miranda, 2008; Roll, Kennedy, Tran, & Howell, 2013; SAMHSA, 2015). Research and efforts to reduce disparities has increased in the past twenty years (Safran et al., 2009). With the passage of the Affordable Care Act’s Medicaid expansion, there has been an overall increase in the utilization of mental health services and reduced uninsured disparities, yet no reduction in treatment access disparities (Creedon & Lê Cook, 2016; Mechanic & Olfson, 5 2016; McMorrow, Long, Kenney, & Anderson, 2015). Given that the United States is becoming increasingly diverse (e.g., it is projected by 2044 that over half of the population will be persons of color; U.S. Bureau of the Census, 2015), the continued existence of racial-ethnic disparities presents ethical and moral dilemmas in a healthcare field designed “as a resource that is tied to social justice, opportunity, and the quality of life for individuals and groups” (Institute of Medicine, 2003, p. 36). National survey data has provided evidence of differential access to mental health treatment across racial-ethnic groups (SAMHSA, 2015). Access is typically defined as any clinical contact with a mental health professional over the prior year. A single appointment with a provider would qualify as having accessed treatment (Alegría et al., 2008; SAMHSA, 2015). Using this metric, in a national sample of individuals diagnosed with depressive disorders (n = 8,762), White individuals accessed treatment significantly more often than Latino, Asian, and African American individuals (Alegría et al., 2008). More recently, a larger sample (n = 45,900) was collected by the Substance Abuse and Mental Health Administration (SAMHSA) and found similar mental health access disparities. Specifically, White, American Indian or Alaska Native, or individuals identifying with two or more races had higher rates of mental health treatment than Black, Asian, or Hispanic individuals (2015). In a separate study of comorbid mental health and substance use concerns, White individuals were more likely to have received treatment than Black or Latino individuals (Nam, Matejkowski, & Lee, 2016). There is increasing evidence that individuals that end up accessing mental health treatment have differential quality in their care. The definition of quality of care in mental healthcare has proved challenging to define and efforts have lagged in relation to 6 physical healthcare (Kilbourne, Keyser, & Pincus, 2010; Lora, 2013; McGlynn, Norquist, Wells, Sullivan, & Liberman, 1988) There have been recent efforts to standardize the definition (Kilbourne et al., 2010), but currently, quality of care is typically defined by the diagnostic treatment guidelines from which the patient is seeking services (Powell, 2011; Young, Klap, Sherbourne, & Wells, 2001). For example, the guidelines for treatment of depression in adults include either a minimum of four visits and a continued prescription antidepressant medication for at least 30 days, or at least eight visits to a mental health specialist (e.g., psychiatrist, psychologist) with no medication (Wang, Berglund, & Kessler, 2010). Using these depression treatment guidelines, Algeria et al. (2008) found that among those who accessed treatment for depression, non-Latino White individuals had received higher quality of care than African-American individuals. Specifically, non-Latino White individuals received higher quality of care 33% of the time relative to 25% for Latino, 18.9% for Asian, and 10.4% for African American individuals. In a separate study examining depressive and anxiety disorders (n = 1,636), African-American individuals received less quality of care relative to White individuals, and individuals with less than a high school education and low socioeconomic status also had lower quality of care (Young et al., 2001). Using a treatment completion metric (e.g., planned discharge from a specific treatment setting), African American and Hispanic individuals were less likely than White individuals to complete treatment for alcohol or drugs and socio-economic status largely accounted for the disparity (Saloner & Lê Cook, 2013). Provider discrimination, including racial bias and stereotyping, constitutes a major barrier to reducing disparities (McGuire & Miranda, 2008). Two recent audit 7 studies were conducted to examine potential provider bias and discrimination. An audit study is an approach that does not rely on self-report, but rather is a form of field experiment where actors (e.g., clients seeking services) approach subjects (e.g., psychotherapists) and the actor variable is manipulated to test for differences in response by the subject (Shin, Smith, Welch, & Ezeofar, 2016). Among voicemails to psychotherapists (n = 371) that emulated help-seeking messages, middle-class sounding messages were three times more likely to receive a response. Among middle-class messages, White individuals were more likely to be offered a response than African American individuals (Kugelmass, 2016). In a separate study using a similar callback method, individuals with a stereotypical African-American name versus non-Latino White name were 12% less likely to receive a return telephone call and a potential offer for therapy services (Shin et al., 2016). Both studies demonstrated highly concerning evidence of provider discrimination towards clients requesting therapy services. Despite assumed therapists’ efforts to work towards egalitarianism, there seems to be evidence for implicit racial biases, stereotypes towards working class individuals and people of color, and in-group favoritism. These findings provide important evidence of the need in addressing system-level discrimination towards individuals seeking treatment (Snowden & Yamada, 2005; Thornicroft, 2008). The research surrounding mental health disparities signals a clear cause of concern for the psychology field. There is overwhelming evidence that an individual’s experience of discrimination in public life is related to increased mental health concerns (Bostwick, Boyd, Hughes, West, & McCabe, 2014; Krieger, 2014; Thornicroft et al., 2016) and increased suicidal and self-harm behaviors (Farrelly et al., 2015). Yet, many 8 times individuals from marginalized racial-ethnic groups fail to access quality treatment and may face provider racial discrimination upon attempts to enter treatment. Recent changes in psychology have sought to ameliorate mental health disparities and the advantages some individuals have compared to others in accessing quality treatment. Efforts to Address Disparities The American Psychological Association (APA), aware of mental health disparities and an increasingly diverse society, established guidelines on multicultural education, training, research, practice, and organizational change. APA directly acknowledged psychology’s history and the need for structural changes in mental health services by stating: Psychology has traditionally been defined by and based upon Anglo Western middle class, Eurocentric perspectives and assumptions. The traditional approaches to psychological research, education, and practice have not always considered the influence and impact of culture, race, and ethnicity, and their roles in psychological theory, research, and therapy have largely gone unexplored. There has been a growing need to develop a deeper knowledge and awareness of race and ethnicity in psychology and to integrate race and ethnicity into the practice, research, education, and ethics of psychology. (2008, p. 2) In the following section, I will first define key terms. Second, I will briefly discuss the history of the multicultural movement, including the development of multicultural competencies by mental health organizations and critical reviews. Third, I will describe APA’s guidelines on infusing multicultural initiatives in education, training, research, practice, and organizational change. Fourth, I will describe the 2008 APA implementation effort, including reactions from the field of psychology. 9 Definitions of Key Terms In defining race, the APA report describes the controversy surrounding the term, particularly the debate between race being biologically versus socially determined. APA uses a socially determined definition and outlines race as the categorization of individuals, by other individuals, based on physical characteristics (e.g., skin color) and the generalizations and stereotypes that come with these characteristics. In contrast, ethnicity is defined as “the acceptance of the group mores and practices of one’s culture of origin and concomitant sense of belonging” (p. 380). An individual may have multiple ethnic identities and thus the saliency of one may increase or decrease based on the individual’s context. The term culture has been defined as “the belief systems and value orientations that influence customs, norms, practices, and social institutions, including psychological processes (e.g., language, caretaking practices, media, educational systems) and organizations (e.g., media, educational systems)” (APA, 2003, p. 380). An individual’s culture, one that varies from person to person, is one that is shaped through transmitted beliefs, values, and practices. The term multicultural counseling competence (MCC) has been described as “professionals who possess the necessary skills to work with clients from diverse backgrounds” (Holcomb-McCoy & Myers, 1999, p. 294), and includes three components: (a) the counselor’s self-awareness of their worldview, including beliefs, attitudes, assumptions, biases, and values; (b) the counselor’s knowledge of their own particular cultural background as well as working to understand the client’s unique cultural background, and (c) the counselor’s skills in developing appropriate interventions in 10 working with culturally diverse clients (D. W. Sue, Arredondo, & McDavis, 1992). It is important to distinguish MCC from multicultural orientation (MCO) in that the latter has been described as a therapist’s “way of being” with the client that is guided by potentially salient cultural factors within the therapeutic relationship (Owen, Tao, Leach, Rodolfa, 2011, p. 1). A major component to the MCO conceptualization is cultural humility, where the therapist is demonstrating an “interpersonal stance that is other-oriented…in relation to aspects of cultural identity that are most important to the client” (Hook, Davis, Owen, Worthington, & Utsey, 2013). In the present dissertation, I will refer to therapist’s cultural work as MCC versus MCO due to the aim in assessing how well, or competently, the therapist addressed cultural discussions during the therapy hour. Brief History of the Multicultural Movement The history of the inclusion of diversity and multicultural initiatives into psychology is relatively brief, yet diversity initiatives have expanded since their inception. Historical events, such as the 1954 Brown vs. Board of Education Supreme Court Decision and the Civil Rights Act of 1964 are cited as critical events that paved the way for the multicultural guidelines (APA, 2003, 2008; Arredondo & Perez, 2006). In 1973, the National Institute of Mental Health sponsored a conference in Vail, Colorado. The focus of the conference was establishing training guidelines that addressed societal concerns. Among over 150 resolutions was the first explicit recommendation surrounding diversity initiatives. The committee stated: It is important that training programs maximize the degree of cultural diversity characterizing their students. Facilitating the admission, retention, and graduation of students from underrepresented groups will not only correct an obvious injustice, but will simultaneously add multicultural dimensions to the training 11 context, a clearly more efficient way of preparing professions to function in a pluralistic society. (Korman, 1974, p. 444) As training programs sought to address diversity concerns, the next two decades saw multiple changes across the field as a whole. Ethnic minority organizations began assembling within APA, including the Association of Black Psychologists, the Asian American Psychological Association, the Society of Indian Psychologists, and others. There was an explicit push against the ethnocentric history of psychology’s past and texts were published that addressed racial bias in scientific research that occurred over the years. In 1981, the first multicultural competencies by APA were established (Arredondo & Perez, 2006; for a list of the competencies see D. W. Sue et al., 1982). In the following 20 years, the APA, the American Counseling Association (ACA), and the Association of Multicultural Counseling and Development (AMCD) developed and evolved. In 1991, the AMCD outlined 31 competencies and expanded in the following year to include 119 explanatory statements and three additional competencies to address organizational change. In 2002, the ACA adopted the original 31 guidelines (Arredondo & Perez, 2006). The efforts to establish multicultural competencies faced some criticism. A critical review by Weinrach and Thomas (2002) questioned if clinicians should be required to adhere to the competencies outlined by the AMCD. This led to a special section in the Journal of Mental Health Counseling on criticisms of the competencies (Patterson, 2004; Weinrach & Thomas, 2004) and responses by scholars in the multicultural counseling community (Arredondo & Toporek, 2004; Coleman, 2004; Vontress & Jackson, 2004). In their initial article in 2002 and follow-up article in 2004, Weinrach and Thomas described concerns over the competencies, including an overfocus on race and racial differences, a lack of empirical support, and confusion 12 surrounding the use of the terms multiculturalism and diversity. Across the responses, one article agreed and cited that there was no evidence for the implementation of the competencies and recommended a universal approach to developing “methods and approaches that are effective with all kinds of clients” (Patterson, 2004, p. 72). The remaining responses refuted the criticisms. Arredondo and Toporek (2004) outlined the historical context on why race matters and discussed several recommendations for future research surrounding the benefits of the competencies. Coleman (2004) also refuted a number of Weinrach and Thomas’s criticisms, and emphasized the fluidity over time when developing best practices when working with clients. Finally, Vontress and Jackson (2004) applauded the efforts of the AMCD and highlighted the importance of promoting racial and ethnic equality in counseling. The 2003 APA Multicultural Guidelines As the field entered the 21 st century, APA outlined guidelines across multicultural education, training, research, practice, and organizational change. The 2003 report included six guidelines and the scope of the guidelines were explicitly limited to race/ethnicity. The authors stated these guidelines represented “the latest step in an ongoing effort to provide psychologists in the United States with a framework for services to an increasingly diverse population and to assist psychologists in the provision of those services” (p. 381). The first two guidelines focused on the individual, including recognition of individual differences among race/ethnicity and cultural identities as well as knowledge of cultural differences. The remaining four guidelines addressed multiculturalism and cultural sensitivity across the psychology domains of education, 13 research, practice, and organizational change (see Table 1). Guideline 1 focused on the psychologist, including a psychologist’s ability to increase self-awareness of their cultural background, cultural differences with others, including biases and assumptions, and how these impact interactions with others from diverse racial/ethnic groups. The guideline was largely based on research that an individual’s worldview, shaped by their cultural background and development, impacts interactions with others (D. W. Sue et al., 1982, 1992). Human beings automatically categorize others, which includes separating people into an in-group and out-group and thus develop assumptions, biases, and stereotypes towards out-group members (Hornsey, 2008; Tskhay & Rule, 2013). This can have negative effects on inter-group relations (Hornsey & Hogg, 2000). Increasing one’s self-awareness and out-group contact are cited as methods for reducing the impact of negative out-group assumptions and biases. (APA, 2003). Guideline 2 encompasses the psychologist working to increase their multicultural sensitivity, knowledge, and understanding of racially and ethnically diverse individuals. A primary focus on Guideline 2 is the pursuit of knowledge and increased awareness. This includes reading and learning different identity models (e.g., Minority Identity Model; Atkinson, Morten, & Sue, 1998; Patterson 1996), increasing one’s knowledge of different culture’s psychological theories (Betancourt & López, 1993), understanding concerns unique to specific populations (e.g., immigrants or refugees; Vostanis, 2014), reading federal legislation that impacts specific groups (e.g., the United States Court of Appeals for the Ninth Court’s rule on Executive Order 13769; Washington v. Trump, 2017), and overall working to understand “how the individual’s ethnic and racial identity 14 status and development affect beliefs, emotions, behavior, and interaction styles” (APA, 2003, p. 386; Hughes, Kielcot, Keith, & Demo, 2015; LaFromboise, Coleman, & Gerton, 1993; Sellers, Copeland-Linder, Martin, & Lewis, 2006). The remaining guidelines address broader initiatives across multicultural education, training, research, practice, and organizational change. Guideline 3 addresses psychological education and calls on training programs to infuse cultural diversity into their curriculum and training initiatives. Guideline 4 calls for psychology researchers to take into account cultural variables when collecting, analyzing, and interpreting their results. Guideline 6 addresses multiculturalism at an organizational level and works to develop culturally informed policies and practices (APA, 2003). For the purposes of the present dissertation, Guidelines 3, 4, and 6 will not be reviewed in detail due to their proxy relation to the examination of therapist MCC. Guideline 5 is the specific application of Guidelines 1 and 2 into clinical practice. It outlines that a Eurocentric therapy model (i.e., one developed by White middle-class men to work with White middle-class men) significantly reduces psychologists’ ability to effectively treat diverse individuals (APA, 2003; Lee & Ramirez III, 2000; Pilgrim 2014). The Multicultural Guidelines outline three domains from which Guideline 5 operates. First, the guideline highlights the importance of assessing the contextual background of the client, including cultural and sociopolitical factors that may impact the client’s behaviors, thoughts, or feelings (similar to Guideline 1 above). Second, the guideline emphasizes the limitations of assessment and diagnostic instruments when working with individuals from diverse backgrounds and important research avenues to create culturally competent assessments. For example, test bias can be abated by 15 examining the theoretical underpinnings of assessments as well as the content validity of items and normative samples used when developing norms (Balkin, Heard, Lee, & Wines, 2014). Third, the guideline notes the importance of working from culturally sensitive interventions and looking beyond the traditional therapeutic approaches (e.g., cognitive behavioral therapy or psychodynamic interventions) to effectively treat diverse individuals (APA, 2003). For example, researchers have found empirical support for the inclusion of religiosity and spiritualism as it aligns with the client’s value system and as it benefits treatment (de Mamani, Tuchman, & Duarte, 2010) After years of psychologists’ calls for implementation of formal guidelines (see Arredondo & Perez, 2006; D. W. Sue et al., 1982, 1992), APA responded in 2003 and provided a systemic view on the infusion of multiculturalism into psychology. The process of developing multicultural competencies and the 2003 guidelines has been described as a difficult process, but overall the release of the guidelines was a “cause for celebration” (Arredondo & Perez, 2006, p. 1). The guidelines were an important first step in psychology’s efforts to address an increasingly diverse society and disparities in mental health treatment. In 2008, the APA released an implementation report for the 2003 guidelines. The 2008 APA Multicultural Guidelines Implementation Report A critical component missing from the 2003 multicultural guidelines is a clear implementation plan. The 2008 report by the APA Task Force on the Implementation of the Multicultural Guidelines was established as a funded entity to ensure guideline implementation (hereafter referred to as infusion due to the Task Force’s 16 recommendation). Thus, direct recommendations are provided to boards, committees, and offices accountable to APA. Although the original guidelines are limited to race/ethnicity as the primary cultural variable, the 2008 implementation report noted that “when recommendations apply to other core identities, they should be applied,” unless these core identities already have specific guidelines and recommendations (p. 4). Finally, the Task Force noted that the infusion of recommendations pays particular attention to psychologists in training. The 2008 Task Force explicitly noted that the main focus of their infusion efforts are focused on Guidelines 3-6, specifically due to the level of focus (i.e., not individual) of these entities (education, training, research, and organizational change). Guidelines 1 and 2 are both focused on the individual level in an effort to encourage increased selfawareness, intergroup contact, and intergroup knowledge. In reference to Guidelines 1 and 2, the Task Force recommended that psychologists read the Multicultural Guidelines and material identified in their description, participate in cultural competence related continuing education sessions, and review materials from the Council of National Psychological Associations for the Advancement of Ethnic Minority Issues (CNPAAEMI). Guidelines 3-6 are not exclusive from Guidelines 1-2, specifically Guideline 5, which addresses recommendations related to psychologists’ multicultural practice, a core area related to MCC. The Guideline states that, “Psychologists strive to apply culturally appropriate skills in clinical and other applied psychological practices” (APA, 2008, p. 15). Within this guideline, the Task Force made two recommendations. First, the report recommended that clinicians work from and develop multicultural competencies 17 surrounding cultural awareness during assessment and intervention. This notes a skillsbased focus surrounding practices incorporating cultural perspectives. Second, the Task Force recommended that psychologists consider both Western and non-Western psychological interventions (APA, 2008). Psychologists have outlined a few limitations regarding the current APA guidelines and implementation efforts. One primary limitation was that the development of the guidelines was derived from a largely etic, rather than emic perspective (Cheung, 2012; Cheung, van de Vijver, & Leong, 2011; Whaley & Davis, 2007). An etic approach works from a Western template in understanding culture and cultural differences and then testing or applying these with different cultural groups (Cheung et al., 2011). Essentially, theories are developed and researched within predominately Western populations are then generalized or tested on different cultural groups. In contrast, an emic approach looks to build psychological theory from within a specific cultural group and seeks to better understand the “culture-specific meaning of psychological phenomena from the perspective of the insiders” (Cheung, 2012, p. 725). Psychologists have proposed to work from a combined etic-emic approach and utilize the broad information we have across cultures (etic) while still working within specific cultural groups (emic) to support development of theory, assessment, and interventions (Cheung et al., 2011; Helfrich, 1999). Another limitation of the APA guidelines is the lack empirical evidence (Whaley & Davis, 2007). Over the past decade there has been an increase in empirical studies broadly examining culture in psychology, yet little research has cited and directly examined the guidelines themselves (Foaud, Santana, & Ghosh, 2017). 18 Efforts to Establish a Multicultural Competencies Framework As political efforts were underway to establish multicultural guidelines and infusion strategies, psychologists were working to establish multicultural frameworks within the field. In 1982, D. W. Sue and colleagues wrote a position paper in The Counseling Psychologist that aimed to dispel myths surrounding the field’s need for cross-cultural counseling competencies (e.g., at that time, the belief that mental health research and interventions were appropriate across minority groups). They defined crosscultural counseling as, “Any counseling relationship in which two or more of the participants differ with respect to cultural background, values, and lifestyle” (p. 47). Different definitions and theoretical frameworks developed over the following three decades. The frameworks have included, “cultural sensitivity,” “cultural responsiveness,” “multicultural competence,” and “multicultural orientation” (Owen, Tao, Leach, & Rodolfa, 2011; S. Sue, Zane, Hall, & Berger, 2009). Across the varying discussions surrounding the preferred framework and definition, there seems to be a consensus that cultural considerations are important in working with individuals from diverse backgrounds. These discussions have helped shape a number of current cultural competence models. Multiple models of cultural competence have been theorized and they tend to pay attention to self-awareness, knowledge, and skills on the part of the clinician (see Whaley & Davis, 2007). Huey and colleagues (2014) have examined the different models of cultural competence and highlight how each conceptualizes cultural competence slightly differently. Skills-based models (Pedersen, 1978; D. W. Sue et al., 1992; S. Sue, 1998) view 19 cultural competence from the lens of provider awareness of their own cultural biases and assumptions as well as knowledge of other groups. The client is considered in the context of their cultural background and appropriate interventions are developed as these factors interact in the therapeutic relationship. A skills-based model assesses therapist characteristics. For example, the tripartite model developed by D. W. Sue and colleagues (1992), discuss three areas of cultural competence: (a) therapist awareness of one’s own assumptions, values, and biases; (b) understanding the worldview of culturally diverse clients; and (c) developing appropriate intervention strategies and techniques (D. W. Sue et al., 1992; S. Sue, 1998). This model, described in greater detail below, is the flagship model in the field and assisted in forming the APA 2003 guidelines (Chu, Leino, Pflum, & Sue, 2016). Process-oriented models (López, 1997; Owen, Tao, et al., 2011; S. Sue, 1998) focus on the direct process between therapist and client in-session versus treatment tailoring or training therapists with skills and knowledge of cultural contexts. This broader approach emphasizes the integration of the client’s perspective into treatment planning and aims to address cultural variability within broader racial-ethnic groups (Whaley & Davis, 2007). A primary emphasis of the process model is a lack of reliance on therapist knowledge about a specific cultural group, but rather how the therapist ascribes meaning about a client’s concern based on the client’s culture-specific perspective (López, 2002). A process-oriented approach has demonstrated better treatment engagement in a primary care setting among a sample of Chinese Americans with diagnosed major depressive disorder (Yeung et al., 2010). Recently, Owen and colleagues defined multicultural orientation as an other-oriented interpersonal stance 20 (cultural humility) that therapists demonstrate when engaging in cultural discussions (Hook et al., 2013; Owen, Tao, et al., 2011). Therapist MCO has been associated with other therapy processes such as therapist MCC, the working alliance, and treatment outcomes (Hook et al., 2013; Owen et al., 2015). Adaptation models (Bernal, Bonilla, & Bellido, 1995; Bernal, Jiménez-Chafey, & Domenech Rodríguez, 2009; Whaley & Davis, 2007) are one of the more common approaches, and they view cultural competence through a culturally adaptive treatment lens. Here, rather than focus on the therapist’s skills and proficiencies, the systemic nature of the treatment is modified. Factors such as language, metaphors, or overall content of the sessions are adjusted based on the client population. For example, the ecological validity model was developed in work with Latino populations and has been adapted to both CBT and interpersonal therapy (Bernal et al., 1995; Rosselló & Bernal, 1999; Rosselló, Bernal, & Rivera-Medina, 2012). In a separate study, culturally adapted CBT was found to be efficacious when working with Cambodian refugees diagnosed with PTSD and comorbid panic attacks (Hinton et al., 2005). The skills-based tripartite MCC model developed by D. W. Sue and colleagues in 1982 has been the standard in the field. This model was expanded in 1992 and again in 1998 to include a focus on multiculturalism at the organizational level, including policy and employee multicultural interventions (D. W. Sue et al., 1992; S. Sue, 1998; Worthington, Soth-McNett, & Moreno, 2007). The model is conceptualized as a 3x3 matrix with three characteristics mapping on to three dimensions. The three characteristics include: (a) therapist awareness of their own assumptions, values, and biases, (b) understanding the worldview of culturally diverse clients, and (c) developing 21 appropriate intervention strategies and techniques (D. W. Sue et al., 1992; S. Sue, 1998). The three dimensions include: (a) beliefs and attitudes; (b) knowledge; and (c) skills. In total, this creates nine competency areas. For example, there are recommendations under the therapist awareness (characteristic) and knowledge (dimension) competency, including: having an awareness of one’s cultural heritage; understanding how racism, oppression, discrimination, or stereotyping affects the counselor; acknowledging a counselor’s own racist attitudes or beliefs; and an awareness of one’s social impact on individuals from different R-EM groups (e.g., communication style; D. W. Sue et al., 1992; S. Sue, 1998). The tripartite model was operationalized in 1996 (Arredondo, 1996). The operationalizing of this model included explanatory statements for each of the nine competency areas. Under the same therapist awareness (characteristic) and knowledge (dimension) competency, the explanatory statement recommended that an individual be able to display a number of described competencies, including the ability to identify five features of one’s culture of origin and how this impacts their work with culturally different clients (see Arredondo, 1996, for a complete list). However, there have been some debate as to how therapist multicultural competencies have been operationalized, including the semantic difference between multicultural competencies and competence, with the former being a subset of demonstrable behaviors within competence (Ridley & Shaw-Ridley, 2011). Empirical Basis for Therapist Multicultural Competence As frameworks were infused into counseling training programs and clinicians began to practice the multiculturally competencies (Pieterse et al., 2009), researchers 22 worked to provide empirical support for therapist multicultural competence (MCC). In this section, I will outline the empirical basis for therapist MCC by first examining the measurement tools commonly used to assess therapist MCC. Second, I will present the research findings that have explored therapist MCC with other psychotherapy processes and treatment outcome, including a comprehensive meta-analysis conducted by Tao and colleagues (2015). Third, I will discuss limitations in our attempts to examine MCC in relation to psychotherapy process and outcome. MCC Measurement Tools In an effort to better understand therapist MCC, researchers have worked to design measurement tools to collect data from therapists, supervisors or observers, and clients. Therapist self-report cultural competence measures have proven problematic given that we know therapists may not be the best judge of their own abilities (Walfish, McAlister, O’Donnell, & Lambert, 2012). Supervisor or observer rated measures are rarely used due to the labor-intensive process of observational coding (Pace et al., 2016) and the lack of a standardized measurement tools (Drinane, Owen, Adelson, & Rodolfa, 2014). There is also evidence that observer-rated assessments are only detecting the frequency of cultural discussions versus rating therapist skillfulness in these discussions (Dunn, Smith, & Montoya, 2006; Worthington et al., 2000). Client self-report MCC measures are commonly used and attempt to measure insession processes surrounding cultural discussions as perceived by the client. These MCC measures have been classified as both direct and indirect (Tao et al., 2015). Indirect measures attempt to measure a specific construct that is hypothesized to be proxy 23 indicators of therapist MCC. For example, the Racial Microaggressions in Counseling Scale (RMCS) is a 10-item measure that assesses instances of racial microagressions, or “subtle and commonplace exchanges that somehow convey insulting or demeaning messages to people of color” (Constantine, 2007, p. 2). This 10-item measure uses a 3point Likert scale to assess the degree to which racial microaggressions occurred during the session. This measure was recently revised to categorically rate if a racial microaggression occurred or did not occur during the session (Constantine, 2007; Owen, Tao, Imel, Wampold, & Rodolfa, 2014). Study results have shown that increased racial microagressions have been related to decreased ratings of the therapeutic alliance, client ratings of therapist MCC, and client counseling satisfaction (Constantine, 2007). Another indirect measure is the Cultural Humility Scale (CHS). The CHS is a 12item client-rated measure of their counselor's ability to be interpersonally other-oriented and work to understand the cultural background of the client during session (Hook et al., 2013). First, the client rates core aspects of their cultural background that are important to them. Second, the client rates the therapist across 12-items on a scale of 1 (Strongly Disagree) to 5 (Strongly Agree). For example, the prompt reads “Regarding the core aspect(s) of my cultural background, my counselor…” and example items include “Is respectful,” “Is genuinely interested in learning more” and reverse scored items such as “Acts superior” and “Makes assumptions about me.” Studies have shown that increased ratings of therapist cultural humility are related with higher alliance scores even when controlling for ratings of therapist MCC (assessed using the Cross-Cultural Counseling Inventory-Revised, see below), and clients that identified religion/spirituality as their salient cultural identity had better therapy outcomes (Hook et al., 2013; Owen et al., 24 2014). Cultural humility via the CHS taps into the “underlying mechanism through which MCC is actualized, encompassing complex therapeutic processes necessary in cultural discussions and factors that reflect knowledge, skills, and awareness (e.g., tripartite model)” (Owen et al., 2015, p. 5). In contrast, direct measures attempt to assess therapist MCC by asking the client to rate how well, or poorly, the therapist addressed cultural discussions during the therapy session. The only client self-report direct measure of therapist MCC is the Cross-Cultural Counseling Inventory-Revised (CCCI-R; Hook et al., 2013; LaFromboise, Coleman, & Hernandez, 1991). Developed to measure client ratings of therapist beliefs/attitudes, knowledge, and skills as outlined by D.W. Sue et al. (1982), the 20-item measure is commonly used in assessing therapist MCC. An example CCCI-R item asks the client to rate, “My counselor is aware of his or her own cultural heritage” on a 6-point scale ranging from 1 (strongly disagree) to 6 (strongly agree). However, there are particular limitations with the CCCI-R. First, the CCCI-R was originally developed as an observerrated measure and then the items were reworded as it was adapted into a direct client selfreport measure. No study of the psychometric properties of the revised self-report measure exists (Ridley & Shaw-Ridley, 2011). Second, a recent validity study found that only seven of the 20 items were appropriate for client-rated therapist MCC behaviors (Drinane et al., 2014). Overall, there are limitations in the way we attempt to measure therapist MCC. First, there is a limited number of empirically supported measures available. The only measure that was developed to tap into the skills, knowledge, awareness, domains of MCC is the CCCI-R and there are concerns regarding the validity of that measure. 25 Related, there are also construct validity concerns across MCC measures in general as there has been little agreement across client-self report, observer rated, and therapist selfreport measures (Hoyt, Warbasse, & Chu, 2006) across a number of studies (Constantine, 2001; Dillon et al., 2016; Fuertes et al., 2006; Worthingon et al., 2000). These validity concerns call into question our ability to accurately measure therapist MCC and if clients are able to adequately rate their therapists on a domain such as MCC (Ridley & ShawRidley, 2011). Empirical Support for Therapist MCC and Psychotherapy Process-Outcome A 20-year content analysis assessed the empirical studies on therapist MCC (Worthington et al., 2007). Previous content analyses have examined R-EM groups more broadly (e.g., Arredondo, Rosen, Rice, Perez, & Tovar-Gamero, 2005; Pope-Davis, Ligiero, Liang, & Codrington, 2001), but the Worthington et al. study focused on empirical research of therapist MCC. Among 75 identified studies, 48% of the studies were process-outcome and categorized by two outcomes: (a) client perceptions of counselors (e.g., counselor credibility, counselor MCCs, counselor effectiveness, and general counseling competencies) and (b) client outcomes (e.g., client self-disclosure, client attrition, and client satisfaction). A closer look at the actual studies reveals major empirical shortcomings. First, zero studies examined therapist MCC in relation to client improvement or worsening in counseling. Second, a number of the studies provided support for the D. W. Sue et al. tripartite model (1992), yet only a few studies utilized real therapy clients versus pseudo-clients (i.e., students rating audiotapes of culturally 26 unresponsive versus culturally responsive therapy scripts; Atkinson, Casas, & Abreu, 1992). In the studies that utilized real therapy clients, Constantine (2001, 2002) found that R-EM counselors-in-training demonstrated higher MCCs than their White colleagues, and that R-EM client-rated therapist MCC accounted for a significant amount of variance in client satisfaction above general counseling competence ratings. Third, only four studies examined MCC process to assess counselor in-session behaviors in relation to MCC (Worthington et al., 2007). For example, therapists demonstrating culturally congruent interventions with Asian-Americans (e.g., immediate resolution of problems) received higher working alliance scores than insight-oriented therapists (Kim, Li, & Lang, 2002). Worthington et al. noted limitations with the empirical MCC literature and called for the field to continue MCC research and pay attention to the R-EM composition of samples, begin testing real counseling dyads, and continue to replicate past findings. Since the 2007 Worthington et al. review, researchers have analyzed therapist MCC in real counseling dyads, including both client-rated therapist MCC and dyadic studies that survey both the client and therapist. In the first dyadic study ever conducted, Fuertes et al. (2006) found that among a sample of 51 therapy dyads, client-rated therapist MCC (as measured by the CCCI-R) was significantly correlated with increased client-rated therapist empathy (r = .81), client-rated working alliance (r = .73), and client-rated treatment satisfaction (r = .83). Client-rated therapist MCC was not associated with therapist self-rated MCC, working alliance, or satisfaction. Clients that generally viewed their therapist as more competent (general competence, not MCC) was correlated with therapists rating themselves more highly on MCC (r = .40). Despite the 27 promising findings that increased client-rated therapist MCC was related to other psychotherapy processes, the study was limited by both a small sample size and no assessment of client outcome. Since 2000, there have been additional studies that assessed client-rated therapist MCC from real therapy dyads and correlated these with other psychotherapy processes (e.g., the working alliance) and client outcomes (Davis et al., 2016; Owen et al., 2014; Owen, Leach, et al., 2011; Owen, Tao, et al., 2011). A recent meta-analysis aggregated 18 studies of 20 independent samples that tested MCC process and outcome (Tao et al., 2015). The studies utilized samples with varying percentages of R-EM clients (range 30100%) and were composed largely of students receiving services at university counseling centers. There were significant relationships between MCC and client outcome (r = .29, MCC ratings accounting for approximately 8% of the variability in client outcome) and MCC in relation to commonly used psychotherapy process measures, including the therapeutic alliance (r = .61), client satisfaction (r = .72), general counseling competencies (r = .62), and session depth (r = .58). The medium-to-large correlations between MCC and other process measures (range .58-.72) provided evidence that MCC and other processes occur in tandem during session, i.e., a therapist who is rated highly on a MCC measure is likely to also have a strong therapeutic alliance. The correlation between MCC and outcome was nearly identical in size to one of the most robust predictors of psychotherapy outcome, the therapeutic alliance (r = 0.275; Horvath, Del Re, Flückiger, & Symonds, 2011). Despite the promising empirical research supporting MCC process and outcome, the Tao et al. (2015) meta-analysis highlights limitations that remain. First a number of 28 studies in the meta-analysis had small samples sizes (e.g., n = 40, Constantine, 2007; n = 19; Morton, 2011; n = 15, Ward, 2002), although there were, importantly, a number of studies with larger samples sizes (n = 472, Hook et al., 2013; n = 232, Owen et al., 2010; n = 176, Owen, Tao, et al., 2011). Second, all but one of the studies assessed client outcomes using a broad psychological well-being measure. Only one study, an unpublished dissertation, assessed symptom reduction (e.g., anxiety or depression) and found an overall null MCC and outcome effect (Sariemento, 2012). Third, only seven studies assessed MCC in relation to treatment outcome. Fourth, all MCC studies have relied on a cross-sectional study design that either measures one-time point during a client’s treatment episode or retrospectively assesses clients who have recently ended counseling. The lack of any longitudinal study limits our ability to better understand the relationship between therapist MCC and client outcome. The limitations in therapist MCC research are related to larger measurement concerns in psychotherapy process and outcome research as a whole. Issues in Measuring the Dyad in Psychotherapy Process-Outcome Research A broader issue in process and outcome research is the importance of accounting for the dyadic interaction inherent in psychotherapy. At a fundamental level, psychotherapy is a conversation between two individuals, a therapist and a client, that exists in an emotionally charged therapeutic relationship (Frank & Frank, 1991; Wampold & Imel, 2015). Yet most research has not formally considered this relational context and have treated individuals as separate entities. For example, early psychotherapy research focused on client factors (e.g., personality traits or motivation) 29 that were related to improvement or worsening in counseling (Luborsky, Chandler, Auerbach, Cohen, & Bachrach, 1971; Smith & Glass, 1977). More recently, research has examined therapist factors (e.g., a therapist’s ability to build strong therapeutic alliances) that are related client outcomes (Baldwin & Imel, 2007). This focus on the therapist is warranted given they are an important factor in the dyad and there is evidence that some therapists achieve better outcomes than other therapists (Baldwin & Imel, 2013; CritsCristoph et al., 1991). However, there are methodological complexities when assessing the psychotherapy dyad. For example, a therapist’s ability to develop strong alliances with their clients may be the result of specific therapist trait-like qualities in building a positive alliance, but this ability may also be influenced by the client as some clients may be easier to develop a positive alliance with than others. This potential mutual influence of the client and the therapist is important to assess when measuring therapist behaviors in a dyadic interaction (Imel et al., 2011). This influence impacts both the interpretation of therapist ratings on a specific domain as well as the interpretation of a process and outcome correlation. In the following section, I will outline the methods that have addressed these complexities, including the different sources of variability when measuring a dyadic interaction and the potential differential relationship between these sources of variability and client outcome. Variance Partitioning in Dyadic Interactions During a dyadic interaction like psychotherapy, there are several potential sources of influence on measures we obtain from the interaction. The Social Relations Model (SRM; Kenny, 1994) provides a platform from which we are able to understand how 30 interpersonal behavior can be influenced by three sources of variability: (a) the perceiver, (b) the target, and (c) the relationship. This model parses variability in an individual’s ratings of others (perceiver effect), ratings of other individuals on one specific person (target effect), and how one individual uniquely views another individual (relationship effect). When applied to psychotherapy, the possible sources of variability include: (a) the client, (b) the therapist, and (c) the relationship between the client and therapist. Thus, when we measure a psychotherapy process, the measure targeting a specific person can actually be the result of features of the interaction itself – and not the person who is purportedly being targeted by the measure. For example, when asking the client to rate their therapist’s MCC, the client may be more likely give high ratings based on the client’s perception of positive interactions that are unique to the dyad versus the therapist actually exhibiting MCC behaviors during the therapy session. There are features of the data that emerge from psychotherapy dyads that allow researchers to model the potential influence of the client, therapist, or relationship. One of these features is the nested structure of psychotherapy where therapists treat several clients, but each client is treated by one specific therapist. Multilevel models take advantage of this nested structure and are able to untangle the different sources of variability when measuring the therapist on a particular domain (Baldwin & Imel, 2013). For example, in a model where clients (level-1) are nested within therapists (level-2), the model decomposes the overall variance in the dependent variable (e.g., client-rated therapist MCC) into a therapist variance estimate and remaining model variance estimate. These estimates are used to calculate the intra-class correlation coefficient (ICC), which is a ratio of the therapist variance divided by the total variance in the model. The ICC 31 describes the proportion of variance in the dependent variable that is attributed to the therapist (between-therapist variability) as well as the remaining variance (withintherapist variability). A measure with large between-therapist variability (i.e., a large ICC) would indicate that clients of the same therapist are rating their therapist more similarly than clients who saw different therapists. More practically, this would mean that differences exist between therapists on their average ratings across clients. In contrast, a measure with a larger within-therapist variability estimate would be indicative of a wide dispersion of client ratings within a therapist’s caseload and less differences between therapists (Baldwin & Imel, 2013; Radenbush & Byrk, 2002). The final result of these multilevel models and ICC estimates is a measure of how much variance in ratings (e.g., MCC or alliance) is attributed to the therapist, client, or possibly the relationship. For example, if the therapist accounts for a large amount of variance in client-rated therapist MCC, this suggests that therapists are consistently exhibiting higher or lower MCC behaviors across their caseloads as compared to other therapists – thus the measure appears to be discriminating between therapists. However, if the client accounts for majority of the variance, this would suggest that therapists are inconsistently exhibiting MCC across clients in their caseloads and that the majority of the variability in MCC ratings is between clients within a therapist’s caseload. Thus, the measure is largely picking up client characteristics, with less information gleaned about the particular therapist. Finally, if majority of the variance is at the relationship level, there are likely important factors inherent in the interaction that are influencing the scores on the MCC measure. However, the relationship variance component is difficult to assess as it requires a round-robin design where multiple clients rate multiple therapists (Marcus 32 et al., 2009). Psychotherapy researchers have begun to apply this method to better understand observer ratings of adherence (i.e., the degree of therapist treatment compliance) and therapist competence (i.e., the level of therapist skillfulness; Webb, DeRubeis, & Barber, 2010). When measuring the therapist on adherence or competence, one would expect to observe increased between-therapist variability (i.e., differences between therapists in average ratings and stability of ratings across their caseloads). This would indicate that therapists are exhibiting trait-like qualities of adherence and competence across clients in their caseloads. However, results have found more within-therapist variability than expected, suggesting therapists do not consistently adhere and competently deliver a specific treatment across their caseload and thus, ratings are influenced by the client. For example, in a study that assessed therapist adherence and competence in Motivational Enhancement Therapy (MET) there was increased within-therapist variability relative to between-therapist variability on MET competence ratings (e.g., frequency and skillfulness in MET interventions) as well as the therapist’s engagement in METinconsistent behaviors (e.g., confrontational statements that run counter to MET theory). The client’s level of pretreatment motivation and the client’s days of use during treatment significantly predicted adherence and competence ratings, suggesting two client aspects that may impact the therapists’ ability to deliver MET (Imel et al., 2011). Similar results were found in a study that assessed therapist variability in a trial of cognitive behavioral therapy (CBT) for panic disorder. There was increased within-therapist variability relative to between-therapist variability in therapists’ competence of delivering CBT. The patient’s level of interpersonal aggression significantly predicted therapist adherence and 33 competence, however, even when controlling for this client characteristic, withintherapist variability remained significant (Boswell et al., 2013). Across both MET and CBT adherence and competence studies (Boswell et al., 2013; Imel et al., 2011) the use of multilevel models provided a platform to better understand that the client’s influence on therapist adherence and competence ratings was larger than expected, suggesting contextual factors inherent in the psychotherapy dyad (e.g., client level of pretreatment motivation or interpersonal aggression) that impact the therapist’s ability to consistently deliver these treatments. These findings suggest that rather than adherence and competence measuring therapist traits, there is a more complex interpersonal interaction at play that impacts therapist performance. This social relations modeling approach to understanding sources of variability in a measure has also been applied to the therapeutic alliance. The aim of this work is to examine if some therapists are consistently better at building a positive therapeutic alliance across clients in their caseload relative to other therapists, or if the therapist’s ability to develop a positive alliance depends on the client that they are working with. In a meta-analysis of 15 studies that examined how much variance in alliance ratings were accounted for by the therapist (n = 10,287 clients, k = 838 therapists) the mean ICC was 9%. However, there was a wide range of ICC estimates across studies (range of ICCs = .0001-.33) that may be due to varying treatment settings (e.g., training clinics versus clinical trials), different measures of the alliance, and varying sample sizes (Baldwin & Imel, 2013). Yet, even the largest ICC estimate of .33 (Dinger, Strack, Leichsenring, Wilmers, & Schauenburg, 2008) suggests a large amount of variability in alliance ratings is attributed to the client, relationship, or error. These results suggest there are important 34 influences (e.g., the client or relationship) on a therapist’s ability to develop strong alliances (Baldwin & Imel, 2013). Variance Partitioning in Therapist MCC Research A small but growing number of studies have partitioned variance in MCC ratings to assess the variance attributed to the therapist. The first study assessed the relationship between racial microaggressions, therapist MCC, the working alliance, and treatment satisfaction ratings across 40 African-American clients working with 19 White counselors (Constantine, 2007). Using the 10-item Racial Microagressions in Counseling Scale (RMCS) the ICC was 0.76, indicating that a substantial amount of the variability in the RMCS scale (76%) was attributed to the therapist, much larger than estimates of adherence or the alliance discussed previously. This large between-therapist variability estimate suggests therapist differences in average RMCS ratings and that clients’ perceptions of their therapists’ use (or lack of use) of microagressions was generally consistent across one’s caseload. For the 20-item Cross Cultural Counseling Inventory (CCCI-R), the ICC was 0.26. Again, clients’ perceptions of their therapists’ MCC was somewhat consistent (26% of variability was attributed to the therapist), however the remaining within-therapist variability (74%) is indicative of a client influence in MCC ratings, in that some clients perceived their therapist as more MCC than other clients within a therapist’s caseload. Overall, these results are interpreted with caution as a major limitation was the small sample size (40 clients working with 19 therapists). Large samples are required to make more precise ICC estimates (Baldwin, Imel, & Atkins, 2012; Schiefele et al., 2017). 35 A more recent study found substantially smaller between-therapist variability in MCC ratings. In a sample of 143 clients (46% R-EM) and 31 therapists, Owen and colleagues found that the therapist accounted for less than 1% (ICC = .00001) of the variability in client-rated MCC via the 20-item CCCI-R (Owen, Leach, et al., 2011). This suggests few differences between therapist in average MCC ratings and high inconsistency across client perceptions of therapist MCC within a therapist’s caseload. This substantial within-therapist variability suggests that a therapist MCC rating was highly dependent on the client (i.e., varied from client to client). Owen and colleagues (2015) followed-up with another study that used different MCC measures, the 12-item Cultural Humility Scale (CHS) and ratings of missed cultural opportunities using the 4item Cultural (Missed) Opportunities (CMO) scale. In a sample of 247 clients (49% REM) and 50 therapists, therapist ICCs were 21.9% for the CHS scale and 5.3% for the CMO scale. Thus, there was a greater consistency across client perceptions of their therapist’s cultural humility, yet still large within-therapist caseload variability. In contrast, the CMO scale had less between-therapist variability. Taken together, within an individual therapist, ratings were more variable across clients in their caseload on perceived cultural missed opportunities than cultural humility. In a separate MCC study, Dillon and colleagues (2016) assessed 133 clients (100% R-EM) and 24 therapists, where both clients and therapists rated MCC using the 20-item CCCI-R. Among client-rated therapist MCC, 3.46% of the variance was attributed to the therapist (between-therapist variability), indicating some differentiation among therapists with majority of the variance attributed to the client, the relationship, or error. Among therapist-rated MCC, 98% of the variance was attributed to the therapist, 36 indicating that therapists consistently rated themselves the same across clients in their caseload. This was the only study to partition therapist self-rated MCC. Overall, these findings provided slightly larger between-therapist variability estimates using the CCCI-R than the Owen et al. (2015) study (3.46% vs. <1%). These results suggest increased consistency on client perceptions of their therapist’s MCC along with increased therapist differences in average MCC ratings. Yet again, there is a large amount of within-therapist variability suggesting that the therapist MCC is largely dependent on the client. Across all of the MCC studies that have partitioned variance, there was a wide dispersion of between-therapist and within-therapist variability. These inconsistencies point to several limitations. First, sample sizes across studies were small, which impacts the ability to reliably assess therapist variability estimates (Baldwin & Imel, 2013). Second, varying MCC measures were used and it is possible that MCC measures may not be stable in detecting therapist behaviors. For example, one of the most commonly used measures, the CCCI-R, detected essentially no between-therapist variability in one study (Owen, Leach, et al., 2011) yet increased variability in another study (Dillon et al., 2016). Third, the cross-sectional design commonly used in MCC studies limits the partitioning of variability to between therapists and within therapists. This provides no assessment of the separate sources of variability embedded in the within-therapist variability estimate, that is, client variance versus error variance. A three-level repeated measures design would be required to separate therapist and client variance from error variance (Raudenbush & Bryk, 2002). Despite these limitations, by measuring the differential sources of variability in MCC measures, we are able to better understand the degree to 37 which the MCC measure is consistently assessing therapist MCC. These results are important as they inform our efforts to understand the potential differential relationship between variability in MCC ratings attributed to therapist (or to the client) in relation to client outcome. Untangling Process and Outcome Correlations It is common practice in process and outcome research to collect ratings on a specific domain and correlate these ratings with client outcomes. For example, 190 studies have assessed the correlation between the therapeutic alliance and client outcome. The aggregated correlation across all studies (r = .275) has been described as perhaps the most robust predictor of treatment outcomes (Horvath et al., 2011; Wampold & Imel, 2015). This finding has often been interpreted that therapists that develop strong alliances are more likely to have better outcomes. A similar interpretation has been suggested in the therapist MCC literature. In a meta-analysis of 18 studies (described above), the aggregated correlation (r = .29), supported the hypothesis that therapists perceived as more multiculturally competent have, on average, better outcomes (Tao et al., 2015). However, the meaning of the correlation in both the alliance and MCC literature remains ambiguous. To clarify this ambiguity, one must examine the different sources of variability inherent in a process and outcome correlation. A correlation between a process measure and outcome is considered a “total correlation” in that it includes all possible sources of variance (client, therapist, and relationship). Thus, interpreting the alliance correlation as “therapists with higher alliance scores are more likely to have better outcomes” can lead to a contextual fallacy, or an 38 error in inference. This occurs when interpreting the total correlation under the assumption that both within-therapist variability (level-1) and between-therapist variability (level-2) are equally related to outcome. When in reality, the therapist and client variability estimates may have a different relationship with client outcome (Baldwin & Imel, 2013). A classic example is found in the education literature on the relationship between a student’s socioeconomic status and math performance. Researchers have found a larger correlation between a school’s average SES and math scores versus a smaller correlation between student’s SES and math test scores within a school. This finding indicates that schools with higher average student SES tend to have better math scores (between-school). Alternatively, variability in student’s SES within a school is less related to math test scores (within-school). In other words, the SES of the individual within a school matters less when compared to the school’s overall average SES in relation to math outcomes (Raudenbush & Bryk, 2002). Overall, this finding indicates that the correlation within and between schools may be different in magnitude and thus is important to account for the potential differences when interpreting a ‘total correlation.’ In an initial application of this concept to the psychotherapy literature, Baldwin and colleagues (2007) sought to “untangle the alliance-outcome correlation” (p. 1). This was a first test of the potential differential relationship of within-therapist and betweentherapist variability in alliance ratings in relation to outcome. There is an important implication in the alliance and outcome interpretation if the sources of variability are not equally related to client outcome. For example, if there is a between-therapist variability effect, then we can work to better understand therapist differences in alliances behaviors. 39 In contrast, if there is a within-therapist variability effect then we could explore why therapists are better at building strong alliances with some clients versus others. Among a sample of 331 clients seen by 80 therapists, Baldwin et al. (2007) found that between-therapist variability (i.e., therapist differences) in alliance ratings significantly correlated with client outcomes. In contrast, within-therapist variability (i.e., client differences in ratings within a therapist’s caseload) was not significantly related to outcome. In other words, therapists who had higher average alliance scores relative to other therapists generally had better outcomes and clients’ higher alliance ratings relative to other clients seen by the same therapist did not predict outcome. There was also evidence of a “contextual effect,” which tested the significance between the withintherapist and between-therapist effects. This indicated that if two clients with the same alliance rating saw two therapists that differed in their mean alliance rating, the client working with the higher-rated therapist would have a better outcome – even though both clients had the same alliance rating. These results have been replicated by a number of studies (Crits-Christoph et al., 2009; Dinger et al., 2008; Zuroff, Kelly, Leybman, Blatt, & Wampold, 2010). However, one study found the reverse was true and that withintherapist variability versus between-therapist variability significantly correlated with client outcome. This study concluded these results may have differed due to the smaller sample size, particularly few clients per therapist (Falkenström, Granström, & Holmqvist, 2014). 40 MCC Correlated Variability With Outcomes Only two studies have examined the potential differential relationship between within-therapist and between-therapist variability in MCC ratings and client outcome. In a sample of 143 clients (46% R-EM) and 31 therapists, therapists’ average MCC scores were unrelated to outcome (between-therapist) whereas higher client scores within a therapist caseload were related to better outcome (within-therapist; Owen, Leach, et al., 2011). These results were replicated in a separate study that assessed 133 clients (100% R-EM) and 24 therapists. Therapist differences in average MCC ratings were unrelated to client outcomes (between-therapist). Yet, clients with higher MCC ratings relative to other clients in their therapist’s caseload generally reported better clinical outcomes (within-therapist variability; Dillon et al., 2016). These results support an alternative interpretation from the alliance research. In the alliance, therapists that consistently demonstrated positive working alliances had better outcomes and variability within the therapist caseload was unrelated to outcome. However, the MCC research has found the opposite. Specifically, it is not betweentherapist variability but rather within-therapist caseload variability that is related to client outcome. This suggests that the differences between Therapist A or Therapist B on their average MCC rating is less important than differences between clients’ MCC ratings within a therapist caseload. Clients with higher ratings relative to other clients seeing the same therapist are more likely to have better outcomes. However, there are two particular limitations across both studies, including small sample sizes and the use of an MCC measure (CCCI-R) that may not adequately discriminate therapist differences on MCC. 41 Summary and Next Steps In summary, one measurement concern in process and outcome research is the difficulty in assessing a specific individual in a dyadic interaction. When we ask the client to rate their therapist, we tend to infer the rating is reflective of therapist attributes on that domain. However, there are multiple sources of variability in the dyadic interaction (client, therapist, relationship) that may influence this rating. Variance partitioning provides an approach to clarify the sources of variability in these ratings. This approach decomposes the variance in the dependent variable (e.g., ratings of therapist MCC) into between-therapist and within-therapist variance estimates. Thus, we can better understand variability that is attributed to the therapist versus the client. For example, adherence and competence ratings are generally assumed to be consistent across therapists. A competent therapist should be able to exhibit this competency across clients in their caseload (high between-therapist variability). However, studies have found the opposite. Therapists vary in adherence and competence ratings more than we would expect across clients in their caseload. These results parallel the alliance literature where there was some between-therapist variability in alliance ratings, but majority of the variability was within therapist caseloads. The research on therapist MCC is mixed with a wide dispersion of estimates. One study displayed substantial between-therapist variability (e.g., ICC = .76; Constantine, 2007) and another showed a near zero betweentherapist variability estimate (e.g., ICC = 0.0001; Owen, Leach, et al., 2011). The inconsistencies are likely accounted for by the variety of measures used to assess therapist MCC and small sample sizes that reduce the precision in ICC estimates. A second measurement concern is the potential differential relationship between 42 the sources of variability in a process and outcome correlation. This is vital in clarifying the interpretation of a total correlation. The statistical approach builds on variance partitioning and aims to assess if the variability at one level of analysis (e.g., the therapist or between-therapist variability) is differentially related to outcome when compared against another level of analysis (e.g., the client or within-therapist variability). In the alliance literature, preliminary findings suggest that between-therapist variability (i.e., therapist differences on average alliance scores) is related to client outcome versus within-therapist variability (i.e., client scores within a therapist caseload). This finding lends evidence that therapists with consistently higher alliance ratings achieve, on average, better client outcomes. In return, client ratings within a therapist are unrelated to outcome. Among MCC ratings, results from two studies have found the opposite. It was within-therapist variability that correlated with client outcome, not between-therapist variability. Thus, clients with higher MCC ratings within a therapist’s caseload generally had better outcomes. However, this finding is not particularly surprising given that in these two studies the between-therapist variability was small to nonexistent (ICCs = 3.46% and .0001%; Dillon et al., 2016; Owen, Leach, et al., 2011). One interpretation is that the MCC measures are not adequately discriminating differences between therapists, but instead the measures are largely picking up client differences in their perception of therapist MCC. With less variability between-therapists, and essentially zero in one study, there is little variance to explain in relation to outcome (Singer & Willett, 2003). Important limitations remain in the interpretation of the relationship between therapist MCC and outcome. First, studies remain limited with small sample sizes. This impacts our ability to detect precise ICC estimates and the power to detect effects when 43 examining the sources of variability with outcome. Second, of the few studies that have partitioned variance, there are inconsistencies across the different measures in regards to detecting therapist differences. Future work is required to verify that client-rated measures are detecting meaningful information about the therapist. It is possible that clients may not be good raters of therapist MCC (Shaw & Ridley-Shaw, 2011), or that therapists are performing equally well in MCC ratings. Third, the commonly used crosssectional design limits our ability to partition out client variability from the error term (Raudenbush & Byrk, 2002). By utilizing a longitudinal repeated measures design, we would be able to better understand variability in MCC ratings attributed to the therapist versus the client. Fourth, no therapist MCC study has assessed the contextual effect, which tests the significance between the within-therapist and between-therapist effects and would further clarify the interpretation of the MCC and outcome correlation. Study Rationale and Hypotheses The purpose of the present dissertation project is to clarify the MCC and outcome correlation. I propose to address several limitations in MCC research. First, to address small sample sizes commonly used in MCC research, I plan to use a large dataset provided by the University of Utah Counseling Center. The dataset includes repeated observations (n = 8,497) from 1,458 clients working with 35 therapists. This sample size is above the recommended 1,200 clients and therapists suitable to derive reliable ICC estimates (Schiefele et al., 2017). Second, few studies have examined variability in MCC ratings that are attributable to the therapist and no study has partitioned client variance from error variance. This is an important step in understanding how well our MCC 44 measures are actually detecting information about the therapist. To address this limitation, I will run Bayesian three-level mixed-effects models that will partition variance at both the client (within-therapist) and therapist (between-therapist) level on items selected from a recently developed MCC measure, the Cultural (Missed) Opportunities Scale. An initial assessment of this measure has found a promising between-therapist variability estimate relative to those detected using the CCCI-R (Dillon et al., 2016; Owen, Leach, et al., 2011; Owen et al., 2015). Third, only two studies have examined the correlations between within-therapist MCC variability and betweentherapist MCC variability and client outcomes. The preliminary finding that withintherapist and not between-therapist variability was related to outcome are to be interpreted with caution due to small samples sizes and the use of a measure that detected small therapist differences. Also, both of the studies failed to test if the within and between coefficients were significantly different, a test that would measure how client outcomes would differ for two clients with the same MCC rating but working with different therapists. To address this limitation, I will run Bayesian two-level mixedeffects models that will test within-therapist MCC versus between-therapist MCC in relation to outcome. I will also test the contextual effect to assess a potential significant difference in the magnitude that a client would improve when working with a therapist with a high mean MCC rating versus a therapist with low mean MCC rating. Overall, these data will comprise the first ever longitudinal assessment of MCC process and outcome. Based on the review of the literature, I have two core hypotheses. 45 Hypothesis 1: Sources of Variability in Therapist MCC Ratings There are two important sources of variability to measure in MCC ratings, between-therapist variability (i.e., how much therapists vary from one another in their mean MCC ratings) and within-therapist variability (i.e., variability in client ratings within a therapist’s caseload). I predict that a small amount between-therapist variability will be detected (around 5-7%) and that majority of the variability will be withintherapist and error. I will also assess variability in relation to client’s R-EM status and examine if the different sources of variability (between and within) increase for therapists working with White compared to R-EM clients. Across analyses, therapist MCC will be compared with ratings of the therapeutic alliance and treatment satisfaction. Additionally, exploratory analyses will assess variability in ratings based on client’s sexual orientation and religious affiliation. Hypothesis 2: Therapist MCC Variability and Clinical Outcomes I predict that between-therapist variability in MCC ratings will correlate with client outcomes. Thus, clients treated by therapists with higher average MCC ratings will generally achieve better outcomes. I also predict that the contextual effect, the difference between the within-therapist and between-therapist variability effects, will be significant. This has two important implications. First, a significant contextual effect would confirm that there are meaningful differences between the within-therapist and between-therapist components in the total correlation. Second, it would clarify how client outcomes would differ between two clients with the same MCC rating but worked with therapists that differed in average their average MCC ratings. This hypothesis runs counter to two 46 previous studies that have examined within-therapist and between-therapist MCC effects in relation to outcome. However, I believe that with a larger sample, new and promising items, and surveying clients currently in therapy will assist in detecting betweentherapists differences in relation to outcome. Additionally, and similar to hypothesis one, I will examine differences across client demographics (race-ethnicity, sexual orientation and religious affiliation) to explore potential differential MCC and outcome relationships between client identities. Similar to the variance partitioning models, I will also compare these the MCC results with ratings of both the therapist alliance and client treatment satisfaction in relation to client outcome. 47 Table 1 The APA’s Six Multicultural Guidelines Guideline Level of Focus Guideline 1: Psychologists are encouraged to recognize that, as cultural beings, they may hold attitudes and beliefs that can detrimentally influence their perceptions of and interactions with individuals who are ethnically and racially different from themselves. Individual Guideline 2: Psychologists are encouraged to recognize the importance of multicultural sensitivity/responsiveness to, knowledge of, and understanding about ethnically and racially different individuals. Individual Guideline 3: As educators, psychologists are encouraged to employ the constructs of multiculturalism and diversity in psychological education. Education Guideline 4: Culturally sensitive psychological researchers are encouraged to recognize the importance of conducting culture-centered and ethical psychological Research Guideline 5: Psychologists are encouraged to apply culturally appropriate skills in clinical and other applied psychological practices. Practice Guideline 6: Psychologists are encouraged to use organizational change processes to support culturally informed organizational (policy) development and practices. Organization The 2003 APA Multicultural Guidelines (see American Psychological Association, 2008). CHAPTER 2 METHOD Participants and Procedure The data analyzed for this study were collected from clients that attended individual counseling appointments at the University of Utah Counseling Center (UCC) between January 2015 and June 2017. As part of the regular intake procedure at the UCC, clients fill out background demographic information and initial levels of symptom distress. After the intake interview, clients are assigned with an individual therapist. Prior to every individual therapy session, clients fill out a symptom severity measure along with process measures that assess client-rated therapist MCC, alliance, and overall treatment satisfaction from their previous session. Two datasets were created to assess (1) variance partitioning and (2) the relationship between process measures and clinical outcomes. For the variance partitioning dataset inclusion criteria, clients had to fill out symptom severity and process measures and met with their therapist for at least two sessions (to provide at least one rating on the process measures given they are unable to rate their therapist at the first session). Therapists had to see a minimum of 10 clients (5 White and 5 R-EM) and this number was selected in an effort to have more precise therapist-level estimates (i.e., mean therapist rating across each process measure). The second dataset had additional inclusion 49 criteria, including selecting the first client treatment episode where clients responded to a process measure. Additionally, the modal therapist was taken for each treatment episode. It is not uncommon at the UCC for clients to meet with a different therapist either sporadically at one point during treatment (e.g., crisis intervention) or to change therapists at times of trainee turnover. Due to this, the modal therapist was assigned as the client’s therapist for the complete treatment episode. For exploratory hypotheses (LGBT and religion), therapists had to see at least five clients in each identity category (e.g., religious/nonreligious). Using these inclusion criteria, the first dataset (variance partitioning) consisted of 8,497 process measure observations from 1,458 clients that saw 35 therapists. Clients attended a mean number of 9.2 sessions (SD = 12.2). Therapists averaged 45 clients per caseload (SD = 25.9, range = 14-116). There were 407 clients that identified as R-EM. Therapists averaged 29% R-EM clients per caseload. The client demographics include: 57% women, 41% men, <1% transgender, and <1% individuals self-identified their gender. Among race-ethnicity, 71% percent of individuals identified as White, 10% as Asian-American, 9% as Latino/a, 5% as multiracial, 2% African-American, <1% as American Indian or Alaskan, <1% as Pacific Islander, and 2% self-identified their raceethnicity. 80% percent of individuals identified as heterosexual, 9% as bisexual, 4% as gay, 1% as lesbian, 2% as questioning, and 3% self-identified their sexual orientation. Among sexual orientation, 36% of clients identified with a religious affiliation (18% LDS), 51% of clients identified no religious affiliation, and 15% identified an “other” religious preference or preferred not to answer. The second dataset (outcome) consisted of 1,421 clients that saw 36 therapists. 50 Clients attended a mean number of 8.5 sessions (SD = 8.2). Therapists averaged 39 clients per caseload (SD = 25.3, range = 14-116). There were 403 R-EM clients and therapists averaged 29% R-EM clients per caseload. The client demographics were nearly identical to the first dataset. Measures Client Psychological Distress Client distress was measured with the Counseling Center Assessment of Psychological Symptoms (CCAPS; Center for Collegiate Mental Health, 2012). The CCAPS is a multidimensional assessment developed for college counseling centers and the student population. Clients are asked to rate how well a statement describes them over the past two weeks on a 4-point scale from “not at all like me” (0) to “extremely like me” (4). An example item from the depression subscale is, “I don't enjoy being around people as much as I used to.” The 64-item measure is used only at intake to measure client distress across eight subscales: depression, generalized anxiety, social anxiety, academic distress, eating concerns, family distress, hostility, and substance use. The 64-item measure demonstrated high internal consistency across all subscales (range  = .82-.92). The 34-item measure is used at every session post-intake and measures client distress across seven subscales, excluding family distress, and demonstrated high internal consistency across the seven subscales (range  = .82-.92). 51 Therapist Multicultural Competence (MCC) Therapist MCC was assessed using two items from the Cultural (Missed) Opportunities Scale (Owen et al., 2015). Prior to answering the items, clients are provided with a brief description that states: “There are several different aspects of one’s cultural background that may be important to a person, including (but not limited to) race, ethnicity, nationality, gender, age, sexual orientation, religion, disability, socioeconomic status, and size. Some things may be more central or important to one’s identity as a person, whereas other things may be less central or important.” After this description, the first item states, “My therapist discussed my cultural background in a way that worked for me.” The second item states, “I wish my therapist would have encouraged me to discuss my cultural background more.” These are rated on a 5-point scale from strongly disagree (1) to strongly agree (5) with the option to provide a neutral response (3). These two items were selected from a 5-item measure that had adequate reliability ( = .79), yet when the item “My therapist discussed my cultural background in a way that worked for me,” was removed, the 4-item measure had increased reliability ( = .86; Owen et al., 2015). The second item was reverse scored and combined with the first item to create an MCC total score. These items were selected from a measure intending to measure MCO, however, the items themselves reflect the behavioral piece of the counselor’s ability to discuss the client’s background and the frequency of this discussion. The Working Alliance The therapeutic alliance was assessed using four items from the Working Alliance Inventory (WAI; Horvath & Greenberg, 1989). The WAI assesses three facets of the 52 therapeutic relationship, (a) agreement on the goals of therapy, (b) the tasks that therapist and client agree to address during therapy, and (c) the perceived bond between therapist and client. The four items are rated on a 7-point scale ranging from “never” (1) to “always” (7). For the present study, we selected one item from each goals, tasks, and two items from the bond subscale. The therapeutic alliance has proven to be a robust predictor of treatment outcomes (Horvath et al., 2011) and overall has demonstrated high internal consistency ( = .93; Horvath & Greenberg, 1989). In a separate using three of the four WAI items, researchers found high internal consistency ( = .90; Imel, Hubbard, Rutter, & Simon, 2013). Treatment Satisfaction A single client satisfaction item was used from the Group Health Patient Experience Survey (GHPES) that assesses treatment satisfaction on a 10-point scale ranging from “worst possible counseling” (0) to the “best possible counseling” (10). Client satisfaction ratings are common in health maintenance organizations and research has shown that therapists that have higher alliance ratings generally have more satisfied clients (Imel et al., 2013). Statistical Analyses Data were analyzed using a Bayesian statistical approach that relies on a probability distribution framework to estimate uncertainty in model parameters. Rather than estimating a population parameter via a sampling distribution (i.e., running a model over an infinite number of hypothetical possible samples, as found in classical 53 “frequentist” statistical inference), Bayesian statistics estimate the probability of a population parameter given the actual data (Hox, Moerbeek, & van de Schoot, 2017; Krushke, 2015). In Bayesian statistics, each population parameter of interest is assigned a “prior” probability distribution before running the model. This probability distribution is called the prior distribution and these distributions can be informative or uninformative. An informative prior indicates a strong belief about the shape of the unknown parameter’s distribution. In contrast, an uninformative prior suggests we know very little about the population parameter of interest and an uninformative prior will have very little impact on the model estimate (Hox et al., 2017). The prior distribution is combined with a likelihood, or a distribution that best fits the collected data (e.g., Gaussian for continuous data). The model (prior + likelihood) generates a posterior distribution, or a distribution of the possible values of the parameter of interest. The posterior distribution is the outcome of the model and is estimated via a Markov chain Monte Carlo (MCMC) simulation procedure. The posterior distribution is used to obtain point estimates of the parameter as well as a 95% highest density interval (HDI). The 95% HDI is a measure of uncertainty of the true parameter value (Krushke, 2015). For the present analyses, I ran multiple Bayesian mixed effects models for both variance estimates (Hypothesis 1) and correlated variability with clinical outcomes (Hypothesis 2). Across all models, I used noninformative priors that weakly informed parameter estimates while also providing posterior distributions for each estimate. Relatively flat normally distributed priors were used for the fixed effects and inverse Wishart priors for the variance components. Each model had four separate MCMC chains and ran across 100,000 iterations, selecting every 10th iterative draw, also known as the 54 thinning interval, and included a burn-in phase (also described as a “warm-up”) where the first 50,000 estimates were thrown out. The thinning interval was used to reduce potential dependence from one simulation draw to the next. The burn-in phase reduced the influence of the starting point for each MCMC chain, as early iterations may be less likely to converge given the exploration of the parameter space. A conservative recommendation is to discard the first half of each chain (Gelman et al., 2014). The chains for each parameter were combined to form a single posterior distribution, comprised of 20,000 model estimates. Model checks are critical for Bayesian models. Convergence (i.e., that the iterative draws comprise a posterior distribution for the parameter of interest) was done by (a) manually examining the trace plots, (b) assessing autocorrelation across chains, and (c) calculating the Gelman-Rubin statistic. A trace-plot shows the iterative draw (xaxis) and the parameter value (y-axis) and we examine the plots to ensure they draws are relatively stationary across iterations (i.e., no major shifts or getting stuck at a specific parameter value) and adequately mix across draws (i.e., exploring the parameter space). Autocorrelation values near zero indicate good mixing across draws. Higher autocorrelation values would require changes to the thinning and burn-in commands. The Gelman-Rubin statistic is a measure of how well the MCMC chains converged and the expected value is around 1 (Gelman & Rubin, 1992). Problems in model convergence, as evidenced by trace-plots, high autocorrelation, and a small or large Gelman-Rubin statistic are generally due to error in specifying the prior, the likelihood distribution, or fitting overly complex models, for example, many random effects (Baldwin & Larson, 2017). 55 Bayesian methods offer a clear advantage to the frequentist statistical approach when partitioning variance. Bayesian estimation procedures provide more stable estimates, particularly when the grouping variable sample sizes are small (Hox et al., 2017). These methods are also capable of running more complex models with multiple fixed and random effects or stratifying the random effects by client or therapist level variables (e.g., race-ethnicity status; Gelman et al., 2016). All Bayesian mixed effects models were fit using the MCMCglmm package (Hadfield, 2010) and model checks were conducted with the coda package (Plummer et al., 2016) within the R statistical program (R Core Team, 2017). Analysis 1: Sources of Variability in Therapist MCC Ratings I ran multiple models to extract variability estimates across MCC, alliance, and treatment satisfaction ratings. These models provided variance estimates in MCC ratings that could be compared to alliance and treatment satisfaction ratings. To examine the source of variability in the complete sample, I used the following equation: 𝑌𝑖𝑘𝑗 = 𝛾000 + 𝜇𝑘 [𝑅𝐸𝑀] + 𝜇𝑗 [𝑅𝐸𝑀] + 𝑒𝑖𝑘𝑗 [𝑅𝐸𝑀] (1) where 𝑌𝑖𝑘𝑗 is process variable (MCC, alliance, treatment satisfaction) with the ith observation nested in the kth client and nested in the jth therapist; 𝛾000 is the grand mean of the process variable, 𝜇𝑘 is the random effect for client (level-2), 𝜇𝑗 is the random effect for therapist (level-3) and 𝑒𝑖𝑘𝑗 is the residual error term. Both random effects and the residual error term were also stratified by client’s R-EM status, thus providing a variance 56 estimate for both White and R-EM clients at each level. For example, the random effect for client (𝜇𝑘 ) will comprise two variance estimates, White clients and R-EM clients. The random effects at each level provide variance estimates that will be used to calculate the intra-class correlation coefficient (ICC): 2 𝜎𝑡ℎ𝑒𝑟𝑎𝑝𝑖𝑠𝑡 [𝑅𝐸𝑀] 2 2 𝜎𝑐𝑙𝑖𝑒𝑛𝑡[𝑅𝐸𝑀] + 𝜎𝑡ℎ𝑒𝑟𝑎𝑝𝑖𝑠𝑡 [𝑅𝐸𝑀] + 𝑒𝑖𝑘𝑗 [𝑅𝐸𝑀] (2) where 𝜎2therapist[𝑅𝐸𝑀] is the amount of variance at the therapist level (level-3) across REM clients and the denominator is the remaining variance across R-EM clients in the model. This ratio will provide the amount of variability in the process measure ratings attributable to the therapist working with R-EM clients. A particular strength in the repeated measures design is the ability to partition the variance at the client level (level-2) from model error. This analysis is not available in cross-sectional designs where the client provides only one rating, and thus client level variance is combined with the error term. Additionally, ICCs will be calculated for each level of the model: therapist, client, and error. For example, to calculate variability in the process measure ratings attributable to the client, I will run a separate ICC as follows: 2 𝜎𝑐𝑙𝑖𝑒𝑛𝑡 [𝑅𝐸𝑀] 2 2 [𝑅𝐸𝑀]+ 𝜎𝑡ℎ𝑒𝑟𝑎𝑝𝑖𝑠𝑡 [𝑅𝐸𝑀]+ 𝑒𝑖𝑘𝑗 [𝑅𝐸𝑀} 𝜎𝑐𝑙𝑖𝑒𝑛𝑡 (3) where I will replace the therapist level variance at the numerator with the client level variance estimate. Subsequently, to calculate variance attributable to error, the numerator 57 will be replaced with the residual error term. Variance estimates were calculated from each parameter estimate’s posterior distribution. Given the therapist-level posterior distributions were likely to be skewed towards zero, the median estimate was used and the 95% HDI was calculated using the 2.75% and 97.5% quantile estimates. A particular strength of the Bayesian posterior distribution is the ability to measure the difference in magnitude between ICC estimates (e.g., the therapist White ICC versus the therapist R-EM ICC). Differences between ICCs were calculated by subtracting posterior distribution A (e.g., therapist White ICC) from posterior distribution B (e.g., therapist R-EM ICC). The median point estimate and 95% HDI of this new “difference distribution” was calculated. A 95% HDI that does not overlap zero lends support that the probability of these ICC estimates being truly different is beyond 95% certainty. To explore variability in process ratings across different client cultural identities, I ran additional models that examined client’s sexual orientation (LGBT or straight) as well as client’s religious affiliation (religious or not religious). These variables replaced client’s R-EM in the above models. Analysis 2: Therapist MCC Variability and Clinical Outcomes To calculate the relationship between MCC within-therapist and betweentherapist variability and clinical outcomes across client demographics, I ran a series of models: 𝑌𝑖𝑗 = 𝛾00 + 𝛾01 + 𝛾20 (𝑧𝑖𝑗 − 𝑧̅𝑗 ) ∗ 𝑅𝐸𝑀 + 𝛾30 (𝑧̅𝑗 − 𝑧̅) ∗ 𝑅𝐸𝑀 + [𝑈0𝑗 + 𝑅𝑖𝑗 ] (4) 58 where 𝑌𝑖𝑗 is the posttest CCAPS score for the ith client seen by the jth therapist; 𝛾00 is the grand mean and intercept for the White client outcome, 𝛾01 is the pretest estimate (controlling for patient pretreatment severity), 𝛾20 is the within-therapist MCC slope for White clients that is calculated by taking the ith patient’s mean processes measure rating (𝑧𝑖𝑗 ) minus their jth therapist’s mean process measure rating (𝑧̅𝑗 ), 𝛾30 is the betweentherapist MCC slope for White clients that is calculated by taking the therapist’s mean process measure rating (𝑧̅𝑗 ) minus the overall therapist grand process measure rating mean (𝑧̅), 𝑈0𝑗 is the therapist variance estimate and 𝑅𝑖𝑗 is the residual error variance estimate. Both within-therapist (𝛾20 ) and between-therapist (𝛾30 ) parameter estimates are tested with an interaction term to calculate the R-EM slopes for both parameters. For exploratory hypothesis, this interaction term was replaced based on the identity status of interest (e.g., LGBT and religion). Similar to the variance partitioning Bayesian models, all parameter estimates described above were drawn from separate posterior distributions. Again, the posterior median was used as a measure of central tendency due to potential skewed distributions. 95% HDI intervals were calculated for each parameter estimate. Differences between within-therapist and between-therapist estimates across White and R-EM clients were tested by subtracting each posterior distribution (e.g., White within-therapist and R-EM within-therapist) and calculating a median and 95% HDI of the new “difference distribution.” A 95% HDI on the new posterior distribution, or the difference distribution, that does not overlap zero lends support that the probability of the parameter estimates truly being different is beyond 95% certainty. Additionally, I tested if the between-therapist and within-therapist estimates were 59 significantly different from one another, also known as the contextual effect (Raudenbush & Bryk, 2002). In frequentist statistics, this test is conducted by re-centering the 𝛾20 fixed effect 𝑧𝑖𝑗 around the grand therapist mean (𝑧̅) instead of the jth therapist mean (𝑧̅𝑗 ). This changes the between-therapist fixed effect to the difference between the within-therapist and between-therapist fixed effects (Baldwin et al., 2007). However, with Bayesian models, this test is conducted by simply subtracting the between-therapist posterior distribution from the within-therapist posterior distribution and calculating a 95% HDI on the difference posterior distribution. This is identical to the test of the differences between White and R-EM estimates described above. The contextual effect is interpreted as the expected posttest CCAPS difference between two clients with the same process measure rating but whose therapists differ by one standard deviation unit in their mean process measure score. CHAPTER 3 RESULTS Client Level Descriptive Statistics The means and standard deviations for all observations are reported in Table 2 across each of the three process measures (MCC, alliance, and treatment satisfaction) for the complete, White, and R-EM sample. The MCC total score was calculated by taking the total of the two items (one item was reverse scored) and observations fall on a range of 2 to 10. The alliance score is a mean rating of each of the four alliance items and falls on a range of 1 to 7. Finally, the treatment satisfaction score is a single item on a range of 0 to 10. Overall, the mean ratings between the complete sample, White sample, and R-EM sample were similar and standard deviations were nearly identical. Clients generally experienced therapists that displayed above average multicultural competence, strong alliance building behaviors, and these clients were generally satisfied with their treatment (Table 2). The distributions for the complete sample are plotted in Figure 1. Among MCC total score, 40% of client observations rated both MCC items “3” or “neutral” (61% for Item 1 and 46% for Item 2). The modal response for alliance scores was 7 out of 7, indicating a strong experience of alliance between client and therapist. The modal response for treatment rating was 10 out of 10, indicating many clients experienced their 61 previous session as the “best counseling ever.” Therapist Level Descriptive Statistics Therapist means and standard deviations were calculated at the therapist level (k = 35). Overall, therapists had above average mean MCC ratings (M = 7.3, SD = 0.4), high alliance ratings (M = 6.1, SD = 0.2), and high treatment satisfaction ratings (M = 8.6, SD = 0.3). Range estimates indicated that the difference between the highest and lowest preforming therapist on mean MCC was 1.5 points (range = 6.6-8.1), alliance was 1.0 points (range = 5.6-6.6), and treatment satisfaction was 1.4 points (range = 7.9-9.3). The distributions of therapist mean ratings are displayed in Figure 2. Variance Partitioning The ICC posterior distributions across all three process measures are shown in Figure 3. The median was used as the measure of central tendency due to the skewed distributions, particularly at the therapist level, and 2.5%, 97.5% quantiles were used to measure 95% highest density intervals (HDI). ICC results are reported in Table 3. Results are presented for the complete sample, White sample, and R-EM sample. Across all process ratings, a small amount of variability was attributed to the therapist (range = 0.02-0.05), a substantial amount of variability was attributed to the client (range = 0.690.72), and a medium amount of variability was attributed to residual error (range = 0.230.28). Among the complete sample, there were no significant differences in the therapist or client MCC ICC estimates when compared with alliance or treatment satisfaction 62 ratings. There was greater variability attributed to residual error in MCC ratings relative to alliance ratings (median = 0.05, 95% HDI [0.02, 0.07]). Among White clients, there was a significant amount of residual MCC variability when compared to alliance ratings (median = 0.08, 95% HDI [0.05, 0.11]) and treatment satisfaction ratings (median = 0.04, 95% HDI [0.01, 0.07]). Among R-EM clients, larger variability was attributed to clients for MCC ratings relative to treatment satisfaction ratings (median = 0.08, 95% HDI [0.002, 0.15]). ICC estimates between White and R-EM samples were compared across each of the three process measures (e.g., therapist White MCC and therapist R-EM MCC). Therapist ICC estimates between White and R-EM clients were similar across all measures (i.e., credible interval overlapped zero). Among client-level ICC estimates, White clients accounted for a larger amount of variability than R-EM clients in alliance ratings (median = 0.07, 95% HDI [0.01, 0.14]) and treatment satisfaction ratings (median = 0.06, 95% HDI [0.0004, 0.13]). Among residual ICC estimates, there was increased residual variability attributed to R-EM treatment rating scores relative to White clients (median = -0.07, 95% HDI [-0.12, -0.02]). Between-therapist variability is a measure of the spread between therapists in mean ratings on a specific process measure. To visualize this therapist-level variability, I plotted caterpillar plots for each process measure for both the White and R-EM sample. Displayed in Figure 4, each therapist’s random effect estimate (dot) and 95% HDI (vertical line) were extracted from the mixed effects models. The blue dots represent therapist random effects across their R-EM clients and the yellow dots represent therapist random effects across their White clients. The effects are plotted using White clients as a 63 reference group to visualize the comparison in therapist effects with White clients relative to R-EM clients. The random effects estimates descend from right to left, with the highest mean scored therapists on the right side of the plot and the lowest mean scored therapist on the left side of the plot. This provides a visualization of how well, or poorly, therapists were rated by both their White and R-EM clients. For example, therapist 81 had above average MCC ratings by both White and R-EM clients, the highest mean alliance rating among White clients, a below average mean alliance rating by R-EM clients, and was about average in their mean treatment ratings across both groups. In another example, therapist 340 had significantly higher mean alliance ratings with their White clients than their R-EM clients. Exploratory Variance Partitioning Across Sexual Orientation and Religious Identities I ran additional exploratory models that replaced client race-ethnicity with either client sexual orientation or religious affiliation. Similar to the race-ethnicity analyses, an exclusion criterion was set where therapists had to see at least 10 clients and five from each group (e.g., five LGBT clients and five heterosexual clients). Client Sexual Orientation Among the client sexual orientation, there were 1,144 clients (6,543 observations) that worked with 20 therapists, 883 straight clients met with 20 therapists (M = 47 clients per therapist, SD = 19.6, range = 22-93), and 169 LGBT clients met with 20 therapists (M = 9 clients per therapist, SD = 3.4, range = 5-16). 64 Overall, straight and LGBT ICC estimates across each level (therapist, client, residual) trended in a similar magnitude to White and R-EM client estimates (Table 4). A bulk of the variability was accounted for at the client level, a medium amount at the residual level, and a small amount at the therapist level. Among process measure comparisons (e.g., MCC ICCs versus alliance MCCs), the therapist MCC ICC was significantly smaller than the therapist alliance ICC (median = 0.05, 95% HDI [0.02, 0.07]) and the treatment rating ICC (median = 0.05, 95% HDI [0.02, 0.07]) for the straight client sample. Subsequently, there was larger residual error ICCs for MCC ratings when compared to both alliance (median = 0.09, 95% HDI [0.06, 0.13]) and treatment satisfaction (median = 0.07, 95% HDI [0.03, 0.11]) among straight clients. There were no differences between ICCs for LGBT clients. When straight and LGBT ICC estimates were compared (e.g., ICCs between straight clients and LGBT clients) there were no significant differences across all ICCs between straight clients and LGBT clients (i.e., the credible intervals all overlapped zero). Client Religious Affiliation There were 1,400 clients that identified as religious or not religious that worked with 31 therapists. For clients that identified as religious, 484 clients worked with 31 therapists (M = 17 clients per therapist, SD = 8.9, range = 6-36). For those that identified as not religious, 702 clients worked with 31 therapists (M = 24.7 clients per therapist, SD = 13.8, range = 6-62). Overall, religious and nonreligious ICC estimates across each level (therapist, 65 client, residual) trended in a similar magnitude to White/R-EM and straight/LGBT client estimates (Table 5). A bulk of the variability was accounted for at the client level, a medium amount at the residual level, and a small amount at the therapist level. Among process measure comparisons (e.g., MCC ICCs versus alliance MCCs), the therapist MCC ICC was significantly smaller than the therapist alliance ICC (median = -0.06, 95% HDI [-0.12, -0.02]) and the treatment rating ICC (median = -0.08, 95% HDI [-0.16, -0.04]) for the nonreligious client sample. There was increased variability at the client level for MCC ratings relative to treatment ratings for nonreligious clients (median = 0.07, 95% HDI [0.1, 0.13]) and larger residual variability for MCC relative to alliance for nonreligious clients (median = 0.06, 95% HDI [0.02, 0.09]). There were no differences between ICCs among religious clients. When nonreligious and religious ICC estimates were compared (e.g., ICCs between religious clients and nonreligious clients) variability in treatment rating scores attributed to the therapist was larger for nonreligious clients than religious clients (median = 0.07, 95% HDI [0.01, 0.14]). The remaining ICC comparisons were similar (i.e., credible intervals overlapped zero). Relationship Between MCC, Alliance, and Treatment Satisfaction With Outcome Due to significant differences between White and R-EM client’s posttest score [median = -0.13, 95% HDI [-0.24, -0.02]), all outcome models were controlled for using the client’s pretest score. By including pretest scores, there were no significant differences between White and R-EM clients on their residualized change score [median 66 = -0.02, 95% HDI [-0.1, 0.07]). Zero-order correlations were computed between all three process measures and the client’s residual change score (i.e., the client’s posttest score controlling for their pretest score, or where they began treatment). The results are listed in Table 6 and scatterplots are visualized in Figure 5. Overall, the correlation between MCC rating and client’s residual change score was nonsignificant and near zero (r = -0.03). Thus, there was little relationship between increased or decreased MCC scores and client outcomes. Additionally, correlations between MCC ratings and both the alliance and treatment ratings were in the small to medium range (range = 0.22-0.28; Cohen, 1988). Finally, the correlations between both alliance (r = -0.21) and treatment rating (r = -0.20) with client’s residual change score were significant and in the small to medium range. To calculate the client’s percent improvement for each process measure, I ran three simple multilevel models with client distress level as the dependent variable and client mean rating for the process measure as the independent variable (while controlling for client pretest score). For every one unit increase in a client’s mean MCC score, their outcomes improved by 0.4%. For every one unit increase in a client’s mean alliance score, their outcomes improved by 7%. For every one unit increase in a client’s treatment rating score, their outcomes improved by 4.4%. Test of Within and Between Relationships Using Multilevel Models For each process measure, multilevel models were run to examine the withintherapist and between-therapist estimates for the complete sample. Additionally, these within and between estimates then stratified by client’s R-EM status. 67 MCC Ratings Among MCC ratings, the complete sample within-therapist estimate was near zero (𝛾𝑤𝑖𝑡ℎ𝑖𝑛 = -0.01, 95% HDI [-0.04, 0.02]) as well as the between-therapist estimate (𝛾𝑏𝑒𝑡𝑤𝑒𝑒𝑛 = -0.02, 95% HDI [-0.17, 0.12]). The contextual effect was not significant. These estimates were stratified by R-EM status (see Table 7) and both within-therapist and between-therapist estimates were near zero across White and R-EM clients, indicating little relationship between MCC ratings and client improvement for both groups. All credible intervals for within-therapist and between-therapist estimates overlapped zero. Within-therapist estimates are interpreted such that for every one unit increase in a client’s mean MCC score, the client’s outcome improved by 0.1% for White clients and 0.8% for R-EM clients relative to other clients within their therapist’s caseload. There was no significant difference between the White and R-EM withintherapist parameter estimates (median = 0.02, 95% HDI [-0.04, 0.09]). Between-therapist estimates are interpreted such that for every one unit increase in a therapist’s mean MCC score (relative to other therapists), their client outcomes improved by 1.2% for White clients and 0.1% for R-EM clients. There was no significant difference between the White and R-EM between-therapist parameter estimates (median = -0.03, 95% HDI [0.31, 0.24]). Both tests of the contextual effect were not significant. Alliance Ratings Among alliance ratings, the complete sample within-therapist estimate (𝛾𝑤𝑖𝑡ℎ𝑖𝑛 = 0.17, 95% HDI [-0.22, -0.12]) and the between-therapist estimate (𝛾𝑏𝑒𝑡𝑤𝑒𝑒𝑛 = = -0.28, 95% HDI [-0.48, -0.08]) were both significant (i.e., the credible interval did not overlap 68 zero). The difference between the within-therapist and between-therapist estimates was not significant (median = -0.11, 95% HDI [-0.32, 0.09]). These estimates were stratified by R-EM status (see Table 7). For within-therapist alliance ratings, both White and R-EM clients with higher alliance ratings tended to have more improvement relative to other clients within their therapist’s caseload. For every one unit increase in a client’s within-therapist mean alliance score, outcomes improved by 6.4% for White clients and 7.9% for R-EM clients. The density intervals indicated that the parameter estimate for both White and R-EM within-therapist estimates was below zero beyond a 95% probability. In comparing the two within-therapist estimates, there was no significant difference between the White and R-EM within-therapist parameter estimates (median = 0.05, 95% HDI [-0.06, 0.15]). For between-therapist ratings, the effect for therapists working with White clients was significant. For every one unit increase in a therapist’s mean alliance rating, their clients’ outcome improved by 13.7% for White clients. The between-therapist R-EM effect was not significant. There was no significant difference between the White and R-EM between-therapist parameter estimates (median = -0.25, 95% HDI [-0.65, 0.16]). Neither test of the contextual effect was significant. Treatment Satisfaction Ratings Among treatment satisfaction ratings, the complete sample within-therapist estimate (𝛾𝑤𝑖𝑡ℎ𝑖𝑛 = -0.11, 95% HDI [-0.14, -0.08]) and the between-therapist estimate (𝛾𝑏𝑒𝑡𝑤𝑒𝑒𝑛 = = -0.16, 95% HDI [-0.29, -0.02]) were both significant (i.e., the credible interval did not include zero). The difference between the within-therapist and between- 69 therapist estimates was not significant (median = -0.05, 95% HDI [-0.18, 0.09]). These estimates were stratified by R-EM status (see Table 7). For within-therapist treatment rating scores, both White and R-EM clients with higher treatment satisfaction ratings tended to have more improvement relative to other clients within their therapist’s caseload. For every one unit increase in a client’s mean treatment rating score, outcomes improved by 4% for White clients and 5.6% for R-EM. The highest density intervals indicated that the parameter estimate for both White and REM within-therapist estimates was below zero beyond a 95% probability. There was no significant difference between the White and R-EM within-therapist parameter estimates (median = 0.05, 95% HDI [-0.02, 0.11]). For between-therapist treatment rating scores, the effect for therapists working with White clients was significant. For every one unit increase in a therapist’s mean alliance rating, their clients’ outcome improved by 8.7% for White clients. The betweentherapist R-EM effect was not significant. There was no significant difference between the White and R-EM between-therapist parameter estimates (median = -0.22, 95% HDI [-0.49, 0.04]). Neither test of the contextual effect was significant. Relationship Between Process and Outcome Among Client Sexual Orientation To explore the potential differential relationship across the three process measures and unique diverse client identities, I reran all models substituting client R-EM status with client sexual orientation. These stratified effects based on the client’s sexual orientation, straight or LGBT, are listed in Table 8. 70 MCC Ratings The within-therapist and between-therapist estimates for straight and LGBT clients were nonsignificant. Although nonsignificant, the within-therapist and betweentherapist parameter estimates for LGBT clients trended in a positive direction, indicating that as client and therapist average MCC scores increased, their outcomes generally worsened. There was no significant difference between LGBT and straight clients across effects (within-therapist and between-therapist) and both tests of the contextual effect were not significant. Alliance Ratings Among within-therapist estimates, straight clients with higher alliance scores tended to have better outcomes (6.4% improvement per unit increase) than other straight clients in their therapist’s caseload. The effect for LGBT clients trended in a similar direction but was not significant. The difference between the straight and LGBT withintherapist estimate was not significant (median = -0.03, 95% HDI [-0.19, 0.12]). There was a similar pattern of results for between-therapist effects, where therapists with higher average alliance generally had better outcomes (13.3% improvement per unit increase) among their straight clients. In contrast, the LGBT between-therapist effect was not significant. The difference between these two effects was also not significant (median = 0.12, 95% HDI [-0.46, 0.70]). The contextual effects were not significant. 71 Treatment Satisfaction Ratings Among treatment satisfaction ratings, both within-therapist estimates were significant for LGBT and straight clients. For every one unit increase in a therapist’s mean treatment satisfaction rating, their clients’ outcome improved by 3.6% for straight clients and 4.4% for LGBT clients. The difference between the LGBT and straight within-therapist estimates was not significant (median = 0.02, 95% HDI [-0.07, 0.12]). Neither between-therapist effect was significant. Both contextual effects were not significant. Relationship Between Process and Outcome Among Client Religious Status Similar to the exploratory analyses that examined client’s sexual orientation, I reran all models substituting client’s sexual orientation with their religious affiliation (religious/nonreligious). The results are listed in Table 9. MCC Ratings Among within-therapist estimates, religious clients with higher MCC ratings tended to have better outcomes (2.4% improvement) relative to other religious clients within their therapist’s caseload. The effect for nonreligious clients was not significant. The difference between religious and nonreligious within-therapist estimates was significant (median = -0.06, 95% HDI [-0.11, -0.01]), indicating that religious clients with higher MCC ratings had significantly better outcomes than nonreligious clients within their therapist’s caseload. Neither between-therapist estimate was significant. Both 72 tests of the contextual effect (test of the difference between within-therapist and betweentherapist estimates) were not significant. Alliance Ratings Both within-therapist estimates for religious and nonreligious clients were significant. Thus, for every one unit increase in a therapist’s mean alliance rating, their clients’ outcome improved by 9% for religious clients and 8% for nonreligious clients. The difference between these estimates was not significant (median = 0.03, 95% HDI [-0.08, 0.13]). Among between-therapist estimates, there was a significant effect for nonreligious clients, indicating that therapists with higher average alliance ratings tended to have better outcomes (14.4% improvement) among nonreligious clients in their caseloads. The difference between the religious and nonreligious between-therapist estimates was not significant (median = -0.12, 95% HDI [-0.51, 0.26]). Both tests of contextual effect were not significant. Treatment Satisfaction Ratings Similar to alliance ratings, both within-therapist estimates were significant among treatment satisfaction ratings. Thus, for every one unit increase in a therapist’s mean treatment satisfaction rating, their clients’ outcomes improved by 6.4% for religious clients and 4.4% for nonreligious clients. The difference between these within-therapist effects (religious/nonreligious) was not significant (median = 0.05, 95% HDI [-0.02, 0.11]). Both of the between therapist estimates were not significant and both tests of the contextual effect were not significant. 73 Table 2 Means and Standard Deviations for Each Process Measure White Process Measure All Clients Clients MCC Total 7.2 (1.5) 7.3 (1.6) Alliance Mean 6.1 (0.8) 6.1 (0.8) Treatment Rating 8.7 (1.4) 8.7 (1.4) Note: Mean estimate and (sd). Figure 1. Distribution of client responses to each process variable. R-EM Clients 7.0 (1.4) 6.0 (0.8) 8.6 (1.4) 74 Figure 2. Distribution of mean therapist rating on each process variable. Figure 3. Posterior distributions for each level of analysis by each process variable. The vertical line is the median estimate of each distribution. The shaded area is the 95% HDI. 75 76 Table 3 Intraclass Correlation Coefficients (ICCs) and Density Intervals Complete Sample White Clients RE-M Clients Process Measure (n = 1458, k = 35) (n = 1026, k = 35) (n = 407, k = 35) MCC Therapist ICC 0.02 [0.01, 0.05] 0.02 [0.01, 0.06] 0.02 [0.001, 0.09] Client ICC 0.70 [0.68, 0.72] 0.69 [0.66, 0.72] 0.71 [0.65, 0.75] Residual ICC 0.28 [0.26, 0.30] 0.28 [0.26, 0.30] 0.27 [0.23, 0.30] Alliance Therapist ICC 0.05 [0.03, 0.09] 0.06 [0.03, 0.11] 0.09 [0.04, 0.17] Client ICC 0.72 [0.68, 0.74] 0.74 [0.69, 0.77] 0.66 [0.59, 0.72] Residual ICC 0.23 [0.21, 0.25] 0.20 [0.18, 0.22] 0.24 [0.20, 0.29] Treatment Rating Therapist ICC 0.04 [0.02, 0.08] 0.06 [0.03, 0.12] 0.06 [0.02, 0.12] Client ICC 0.69 [0.66, 0.72] 0.69 [0.65, 0.72] 0.63 [0.57, 0.68] Residual ICC 0.26 [0.24, 0.28] 0.24 [0.22, 0.26] 0.31 [0.27, 0.36] Note: ICC estimates as calculated taking the median of each posterior distribution. MCC = multicultural competence. HDI = 95% highest density interval. n = number of clients. k = number of therapists. 77 Figure 4. Model-derived therapist random effects estimates. Each blue dot is the therapist’s mean process rating for R-EM clients. Each yellow dot corresponds to the process rating for White clients. The vertical lines are the corresponding 95% HDIs. The dashed horizontal line is the mean process measure rating. 78 Table 4 Sexual Orientation Intraclass Correlation Coefficients and Density Intervals Straight Clients LGBT Clients Process Measure (n = 883, k = 20) (n = 169, k = 20) MCC Therapist ICC 0.02 [0.004, 0.06] 0.03 [0.002, 0.13] Client ICC 0.68 [0.65, 0.71] 0.70 [0.61, 0.76] Residual ICC 0.30 [0.27, 0.32] 0.26 [0.21, 0.31] Alliance Therapist ICC 0.07 [0.04, 0.16] 0.09 [0.02, 0.24] Client ICC 0.72 [0.65, 0.76] 0.66 [0.54, 0.75] Residual ICC 0.20 [0.18, 0.23] 0.24 [0.19, 0.29] Treatment Rating Therapist ICC 0.09 [0.05, 0.19] 0.14 [0.05, 0.31] Client ICC 0.68 [0.61, 0.72] 0.59 [0.46, 0.68] Residual ICC 0.23 [0.20, 0.25] 0.27 [0.20, 0.33] Note: ICC estimates as calculated taking the posterior mode of each posterior distribution. MCC = multicultural competence. HDI = 95% highest density interval. n = number of clients. k = number of therapists. Table 5 Religious Intraclass Correlation Coefficients and Density Intervals Nonreligious Clients Religious Clients Process Measure (n = 702, k = 31) (n = 484, k = 31) MCC Therapist ICC 0.01 [0.002, 0.03] 0.05 [0.01, 0.11] Client ICC 0.71 [0.69, 0.74] 0.68 [0.62, 0.72] Residual ICC 0.27 [0.25, 0.30] 0.27 [0.24, 0.31] Alliance Therapist ICC 0.07 [0.04, 0.13] 0.02 [0.004, 0.07] Client ICC 0.71 [0.66, 0.75] 0.74 [0.69, 0.77] Residual ICC 0.21 [0.19, 0.24] 0.24 [0.21, 0.27] Treatment Rating Therapist ICC 0.09 [0.05, 0.17] 0.02 [0.003, 0.07] Client ICC 0.65 [0.59, 0.69] 0.71 [0.66, 0.75] Residual ICC 0.26 [0.23, 0.28] 0.26 [0.23, 0.30] Note: ICC estimates as calculated taking the posterior mode of each posterior distribution. MCC = multicultural competence. HDI = 95% highest density interval. 79 Table 6 Zero-Order Correlations Between Process Measures and Client Outcomes Complete Sample MCC Alliance Tx Rating MCC Alliance 0.28* Tx Rating 0.22* 0.78* Outcome -0.03 -0.21* -0.20* White Sample MCC Alliance Tx Rating MCC Alliance 0.26* Tx Rating 0.20* 0.80* Outcome -0.01 -0.20* -0.20* R-EM Sample MCC Alliance Tx Rating MCC Alliance 0.32* Tx Rating 0.27* 0.74* Outcome -0.06 -0.23* -0.23* Note. *p < 0.05. 80 Figure 5. Simple scatterplots for each process variable. The y-axis is the patient’s residualized gain score. The x-axis is the patient’s mean rating for each process variable. The blue dots represent R-EM clients and the yellow dots represent White clients. Table 7 Within and Between Outcome Results Stratified by Client R-EM Status MCC Posterior Median -0.52 0.62 Alliance Posterior Median 95% HDI -0.53 -0.58, -0.48 0.61 0.57, 0.65 Treatment Satisfaction Posterior Median 95% HDI -0.53 -0.59, -0.48 0.61 0.57, 0.65 Parameter Estimates 95 %HDI Intercept 𝛾00 -0.57, -0.46 Pretest 𝛾01 0.58, 0.66 Within Therapist White 𝛾20 -0.002 -0.03, 0.03 -0.16 -0.21, -0.10 -0.10 -0.13, -0.06 R-EM 𝛾20 -0.02 -0.08, .03 -0.20 -0.29, -0.12 -0.14 -0.20, -0.09 Between Therapist White 𝛾30 -0.03 -0.20, 0.13 -0.35 -0.58, -0.13 -0.22 -0.37, -0.07 R-EM 𝛾30 0.002 -0.26, 0.26 -0.10 -0.46, 0.26 0.01 -0.23, 0.24 Contextual Effect White -0.03 -0.19, 0.13 -0.20 -0.42, 0.04 -0.12 -0.28, 0.03 R-EM 0.03 -0.23, 0.29 0.10 -0.27, 0.46 0.15 -0.09, 0.39 Therapist Variance 0.01 0.003 0.003 Residual Variance 0.51 0.49 0.49 Note: The posterior median is the point estimate taken from each parameter’s posterior distribution. HDI = 95% highest density interval. 81 Table 8 Within and Between Outcome Results Stratified by Client’s Sexual Orientation MCC Alliance Posterior Median 95% HDI -0.53 -0.59, -0.47 0.60 0.56, 0.65 Treatment Satisfaction Posterior Median 95% HDI -0.53 -0.59, -0.47 0.61 0.56, 0.65 Posterior Parameter Estimates Median 95 %HDI Intercept 𝛾00 -0.51 -0.58, -0.45 Pretest 𝛾01 0.61 0.57, 0.67 Within Therapist Straight 𝛾20 -0.01 -0.04, 0.03 -0.16 -0.21, -0.10 -0.09 -0.12, -0.05 LGBT 𝛾20 0.08 -0.01, 0.16 -0.12 -0.26, 0.03 -0.11 -0.21, -0.02 Between Therapist Straight 𝛾30 -0.08 -0.28, 0.11 -0.34 -0.63, -0.05 -0.17 -0.37, 0.03 LGBT 𝛾30 0.22 -0.15, 0.60 -0.47 -1.02, 0.08 -0.28 -0.64, 0.08 Contextual Effect Straight -0.07 -0.28, 0.12 -0.19 -0.48, 0.11 -0.09 -0.29, 0.12 LGBT 0.14 -0.22, 0.51 -0.35 -0.92, 0.22 -0.16 -0.54, 0.22 Therapist Variance 0.01 0.003 0.003 Residual Variance 0.51 0.48 0.48 Note: The posterior median is the point estimate taken from each parameter’s posterior distribution. HDI = 95% highest density interval. 82 Table 9 Within and Between Outcome Results Stratified by Client’s Religious Affiliation MCC Alliance Posterior Median 95% HDI -0.52 -0.58, -0.45 0.60 0.56, 0.65 Treatment Satisfaction Posterior Median 95% HDI -0.51 -0.58, -0.45 0.61 0.56, 0.65 Posterior Parameter Estimates Median 95 %HDI Intercept 𝛾00 -0.48 -0.55, -0.42 Pretest 𝛾01 0.61 0.57, 0.66 Within Therapist Religious 𝛾20 -0.06 -0.11, -0.01 -0.23 -0.31, -0.15 -0.16 -0.20, -0.11 Nonreligious 𝛾20 0.01 -0.03, 0.05 -0.20 -0.27, -0.14 -0.11 -0.15, -0.07 Between Therapist Religious 𝛾30 -0.17 -0.41, 0.06 -0.24 -0.55, 0.06 -0.09 -0.29, 0.12 Nonreligious 𝛾30 0.06 -0.16, 0.28 -0.37 -0.66, -0.08 -0.18 -0.37, 0.002 Contextual Effect Religious -0.11 -0.35, 0.12 -0.02 -0.33, 0.30 0.07 -0.14, 0.28 Nonreligious 0.05 -0.17, 0.27 -0.17 -0.46, 0.12 -0.07 -0.26, 0.11 Therapist Variance 0.01 0.004 0.01 Residual Variance 0.50 0.48 0.47 Note: The posterior median is the point estimate taken from each parameter’s posterior distribution. HDI = 95% highest density interval. 83 CHAPTER 4 DISCUSSION Researchers have targeted therapist multicultural competence (MCC) as a potential tool in reducing prevalent health disparities. Different MCC theories have been developed (Owen et al., 2011; D. W. Sue et al., 1992), and the American Psychological Association has outlined specific MCC guidelines and devoted a task force to monitor the implementation of the guidelines (APA 2003, 2008). However, only recently have we developed an empirical base that supports the relationship between therapist MCC and clinical outcomes. The Tao et al. (2015) meta-analysis synthesized a bulk of MCC research and has subsequently lent significant support to the relationship between MCC, other psychotherapy processes, and clinical outcomes. Yet, limitations remain. There are measurement and analytical concerns that, when addressed, can further our understanding of therapist MCC. The present study sought to begin that process by employing a large naturalist dataset to better understand therapist MCC in relation to the therapeutic alliance, client treatment satisfaction, and clinical outcomes. Analyses focused on two main areas: (1) assessing the extent to which our MCC measures are actually providing information about the therapist versus the client and measurement error, as well as (2) untangling the MCC and outcome relationship by looking at the different sources of variability (within-therapist and between-therapist) embedded in the relationship. A 85 number of important findings were discovered across analyses. Variability in MCC Ratings Attributed to the Therapist Was Small Among the complete sample, variability attributed to the therapist in MCC ratings was small. Observed ICCs for therapist MCC were about half of the magnitude of therapist alliance and treatment satisfaction, yet not significantly different. In previous studies, therapist MCC ICCs have varied from less than 1% to upwards of 76% (Constantine, 2007; Owen, Leach, et al., 2011), with limitations in the varying use of MCC measures and small sample sizes. The current finding aligns more closely with two recent MCC studies that used the Cultural (Missed) Opportunities scale and detected a therapist ICC of 5.3% (Owen et al., 2015) and another study that detected a therapist ICC of 3.5% with the Cross-Cultural Counseling Inventory Revised (CCCI-R; Dillon et al., 2016). These recent studies had larger samples than early MCC work and, including the present study, provide initial evidence of therapist MCC ICCs in the 2-5% range, about half the size of the 9% average therapist alliance ICC (Baldwin & Imel, 2013). There are two leading explanations for small therapist MCC ICCs in the present study. First, it is possible that therapists in the sample are all performing equally well in terms of MCC behaviors. A counseling center is likely to have an increased focus on cultural training versus, for example, a community outpatient clinic. This uniformity could lead to less therapist variability in MCC. However, the range of provider experience from trainees to licensed psychologists, and heterogeneity among mentalhealth service disciplines (e.g., social work, counseling psychology, clinical psychology), may actually signal the opposite and predict increased provider variability. Second, if 86 provider differences do in fact exist, it is possible that the items selected for the present study are not adequately detecting these differences. The CMO items in the present study were selected based on promising initial results (Owen et al., 2015) and perceived face validity of the items, but it is possible that the items fell short in their measurement of multiculturally competent therapist behaviors. However, simply pointing to the items as the deterrent in measuring therapist MCC may be short-sighted and the problem is potentially multifactorial, and this is addressed throughout the discussion below. A Bulk of the Variability in MCC Ratings Was Attributed to the Client This was the first study to quantify the amount of variability that we can attribute to the client versus measurement error. No study has examined therapist MCC with a repeated measures design that allows for separation of client and error variance. Prior cross-sectional studies have examined MCC at a single time point and interpreted the error variance as within-therapist variability, but this estimate is necessarily inflated. We have yet to understand how much of the variability is actually attributed to the client versus simply error. When examining variability in MCC ratings, I found a substantial amount of variability at the client level. This indicates that client characteristics likely account for variability in MCC ratings. It is not uncommon for clients to account for majority of the variability in a process measure, yet it is important to know that about a quarter of the variability across MCC, alliance, and treatment ratings, was attributed to error, or potential random responding. As described earlier, we intend to develop process measures that assess some level of a therapist’s behavior, for example, the therapist’s ability to build strong alliances 87 or ability to address cultural conversations, yet oftentimes these measures are range restricted towards positive ratings. In the present study, clients often rated their therapists high on alliance behaviors (rated 7 out of 7) and that the client has received the “best counseling ever” (10 out of 10). For MCC items, the pattern of responses was different and clients often responded neutral to the MCC items. For Item 1, the neutral response rate was 61%, for Item 2, the neutral response rate was 46%, and 40% of clients responded neutral-neutral to both items when surveyed prior to their session. This neutral response pattern is difficult, if not impossible to interpret. It is possible that the client felt unsure how to respond, preferred not to respond, or maybe cultural conversations did not occur in the previous session or were irrelevant. It is possible that some clients are greatly impacted by one or many of their cultural identities and these are salient in reference to their presenting concerns; alternatively one’s cultural identity may not be as relevant and neutral was an adequate response. Additionally, there may be a “local dependence” among the two items in which similar item wording may have influenced the client to respond likewise to both items (Emmons, Sijtsma, & Meijer, 2007). When we combine this neutral response pattern with the client explaining majority of the variability in MCC items, we are left with questions surrounding what these items are actually measuring. It is possible that these items are not actually targeting specific therapist MCC behaviors. There is certainly value in a client’s insight into the therapeutic process (Boswell, Kraus, Miller, & Lambert, 2015), yet how we interpret the client’s responses may require less attribution to specific therapist behaviors and more to the client’s general therapy experience. This experience may lack insight into more convoluted constructs such as therapist multicultural competence, which leads to 88 questions surrounding the utility of client-rated therapist MCC (Ridley & Ridley-Shaw, 2011). In reexamining the items, it may be beneficial to instead ask the client different session-related questions, such as “Were any of your cultural identities explored during the previous session?,” or “Do you find value in discussing your cultural identities in session?,” and related to concerns of safety “Do you feel safe discussing your cultural identities in session?.” It may also be beneficial to obtain information about client experiences in the time between treatment sessions, such as “Were you negatively/positively impacted by one or more of your cultural identities in the previous week?.” These sources of information may be beneficial for the therapist to follow-up on during session, similar to therapists trained to respond to specific scores on depression, suicide, or homicidal items in the CCAPS distress measure. Few Differences in MCC ICCs Across Client Cultural Identities I was also interested in how variability in MCC ratings may change, or be stable, across different client demographics. Across all three client identities (R-EM, sexual orientation, and religious affiliation) therapist, client, and residual ICCs followed a similar pattern: small therapist ICCs, large client ICCs, and medium residual ICCs. Yet, therapist MCCs were significantly smaller than therapist alliance and treatment satisfaction ICCs among White clients, straight clients, and nonreligious clients. This indicates that measures of the alliance and treatment rating better discriminated between therapists that were good/bad at building alliances and high/low on client treatment satisfaction versus therapist MCC behaviors among these client identities. Of note, each of these client identities (White, straight, nonreligious) had a larger sample size than their 89 comparison group (R-EM, LGBT, religious). It is possible that as sample sizes increase across the complete sample (and subsequent client identities), therapist MCC ICCs would be significantly smaller than therapist alliance and treatment rating ICCs. Our measures are developed with the intention of providing information about therapist behaviors during session, thus significantly small ICCs are problematic. Statistically, there is little variability between therapists to explain in relation to clinical outcomes, that is, does a therapist with high MCC ratings actually obtain better outcomes when compared to a therapist with low MCC ratings. Clinically, measures unable to detect therapist differences reduce our ability to identify which therapist(s) would benefit from increased supervision or additional trainings in an effort to improve client outcomes. Little Evidence for a Relationship Between MCC Ratings and Clinical Outcomes The correlation between ratings of therapist MCC and clinical outcomes was nonsignificant and near zero. This was an unanticipated finding and runs counter to past studies (Dillon et al., 2016; Owen et al., 2011; Tao et al., 2015). Despite the nonsignificant relationship between MCC and outcome, within-therapist and betweentherapist estimates were analyzed across client demographic variables. The only association between MCC ratings and better outcomes was the within-therapist estimate for religious clients. Subsequently, this religious client within-therapist effect was significantly different than nonreligious clients, indicating that religious clients with higher MCC ratings had significantly better outcomes than nonreligious clients. This 90 finding may be related to the unique religious culture in Salt Lake City, Utah, and sheds light on the importance of therapist behaviors in navigating conversations surrounding religion. The remaining within-therapist and between-therapist estimates were nonsignificant and often near zero. Given there was a near zero overall correlation between MCC and outcomes, it is not surprising that the within-therapist and between-therapist estimates were not significant. Two previous studies have examined within-therapist and between-therapist effects and both found that within-therapist variability predicted better client outcomes (Dillon et al., 2016; Owen, Leach, et al., 2011). That is, clients with higher MCC ratings relative to other clients in their therapist’s caseload had better outcomes. The salient difference between the two previous studies and the present study is the measurement of MCC. The Owen et al. and Dillon et. al. study both used the CCCI-R to assess MCC and both samples perceived high therapist MCC (range 5.03-5.29 out of a possible 6, with a standard deviation around 0.5). In contrast, the present study used MCC items from the CMO and the modal response was neutral-neutral (40% of the sample). This lack of higher-rated MCCs may attenuate the MCC to outcome relationship due to a range restriction near the middle of the item scale. To examine the impact of the neutral response pattern, I conducted a post-hoc analysis with these responses removed from the dataset. With zero neutral-neutral responses (Item 1 and Item 2), the therapist MCC ICC decreased from 2% to 0.5% and the correlation between MCC and outcome for the total sample was significant at r = -0.07 (p = 0.02). Whereas the original correlation was r = -0.03, the observed difference is only minimally larger. In comparing White and R-EM clients, both 91 correlations in the post-hoc analysis were small and nonsignificant (White = -0.03; R-EM = -0.10). Overall, removing the neutral-neutral responses had little effect on the MCC and outcome correlation. The Steadiness of the Empirical MCC and Outcome Research Base The nonsignificant finding in the present study may seem surprising, but as we look closer at the MCC and outcome literature, there are inconsistencies across studies in the Tao et al. (2015) meta-analysis that may inflate the MCC and outcome relationship. In the meta-analysis, there were seven MCC and outcome studies that varied greatly in their sample size, sample composition, and measurement of both MCC and outcome. For example, assessment of outcome was variable – four studies used the Schwartz Outcome Scale, two studies used the Patient Estimate of Improvement (PEI), and only one study used a symptom measure (anxiety via the GAD-7). Similarly, the use of MCC measures varied, with only three direct measures of MCC (CCCI-R) and the remaining four studies using an indirect measure of MCC (Cultural Humility Scale, Racial Microaggressions in Counseling Scale, and Microaggressions Against Women Scale). The only study to use a direct MCC measure (CCCI-R) and a symptom measure was an unpublished dissertation that found a small, null effect (r =0.12; Sarmiento, 2012). Additionally, in the Tao et al. meta-analysis, a number of studies used a retrospective cross-sectional design where clients were asked to recall a recent treatment episode and rate their distress/improvement along with their therapist’s MCC behaviors. Five of the seven studies used this retrospective design and only two studies asked clients that were currently in treatment. Among those two studies, one study used a Mechanical 92 Turk survey method and had the largest effect size in ratings of Cultural Humility correlated with Patient Estimate of Improvement (r = 0.59; Hook et al., 2013). The second study was the unpublished dissertation with a null effect (Sarmiento, 2012). Given there are questions as to how well the client can rate a construct such as therapist MCC, it seems this concern is only exacerbated when we ask clients to retrospectively rate their therapist on nuanced MCC behaviors after treatment has ended. Yet, parts of this concern are shared in the present study in which clients are asked to rate their previous therapy session (discussed further below). However, the present study intended to remedy many of these prior methodological concerns by using a repeated measures design with clients actively in treatment. There needs to be continued work in improving our ability to measure MCC. This work includes questioning whether indirect versus direct measures best tap into therapist MCC behaviors, utilizing designs in which clients are actively in treatment, and continuing this work across diverse samples. Alliance and Treatment Satisfaction Ratings Predicted Clinical Outcome I included ratings of the therapeutic alliance and client treatment satisfaction as comparisons to therapist MCC. The correlation between MCC ratings and therapeutic alliance in the present study was small, and about half the size of the aggregated correlation detected in the Tao et al. meta-analysis (r = 0.25 versus r = 0.61). Many of the studies in the meta-analysis used the short-form Working Alliance Inventory (WAI) and items from the WAI were used in the present study. Two factors may explain the attenuated correlation in the present study. First, it is possible that the reduced number of positive MCC ratings (i.e., the neutral responses described above) lead to an attenuation 93 in the correlation between MCC and both alliance and treatment ratings. Second, studies often correlate a single rating of the alliance, which may impact the size of the correlation. In the present study, it is likely that taking the client’s mean rating for each measure reduced within-person measurement error, thus providing a better estimate of the client’s true score. Yet, this may have also suppressed some variability and lead to a smaller correlation. Alternatively, it is possible that early observations of the alliance or therapist MCC may have more value or significance and lead to larger observed correlations. Correlations between both alliance and treatment satisfaction with clinical outcomes were small and significant. The alliance to outcome correlation was near the overall aggregate correlation found across 190 studies (r = 0.275; Horvath et al., 2011). Similarly, the treatment rating to outcome correlation was of similar magnitude and also significant. When examining these correlations with the White client sample versus REM client sample, the correlations were similar in magnitude, however, differences in White/R-EM within-therapist and between-therapist estimates revealed some interesting patterns. For alliance ratings, both the within and between estimates were significant, indicating that higher client alliance scores within a therapist’s caseload relative to other clients predicted better outcomes. The interpretation is similar for between-therapists therapists with higher mean average alliance ratings relative to other therapists at the counseling center had better outcomes. Although not significantly different as measured by the contextual effect, the between-therapist alliance estimate was observed to be larger in magnitude than the within-therapist alliance estimate. This pattern held for White 94 clients, yet not for R-EM clients. For R-EM clients, the pattern switched and the withintherapist estimate was observed larger than between-therapist estimate. Although this difference was not significant, it was within-therapist variability that was significantly related with better client outcomes versus between-therapist variability among R-EM clients. This finding illustrates that when R-EM clients rate their therapist’s ability to build alliances, it is not between-therapist differences that predict outcome, but rather within-therapist variability that predicts outcome. In other words, a therapist’s mean alliance score does not predict their outcome with R-EM clients above and beyond other therapists at the center. In contrast, a R-EM client’s individual alliance score relative to other R-EM clients in the therapist’s caseload is generally indicative of better outcomes. Essentially, therapists are better at building alliances with some R-EM clients versus others in their caseloads, and those clients more satisfied with the alliance have (on average) better outcomes. This pattern was similar for treatment ratings as well, with significant within-therapist and between-therapist estimates for White clients, yet only a significant within-therapist effect for R-EM clients. The treatment rating betweentherapist estimate for R-EM clients was near zero. Taken together, the alliance results for the complete sample mirror the seminal study that untangled the alliance and outcome correlation (Baldwin et al., 2007), yet the present study did not have a significant contextual effect (the 95% HDI was -0.32, 0.09). It is possible that a slight increase in sample size would lead to a significant contextual effect. This was the first study to examine within-therapist and between-therapist in alliance ratings relative to different client identity statuses. With increased data, some clearer patterns may begin to emerge on alliance behaviors among R-EM clients given 95 the therapist alliance R-EM ICC was 9%, yet the relationship with outcome was nonsignificant. Instead, it was within-therapist variability (i.e., client variability) that predicted outcome. The clinical implication is that therapists may have difficulties building strong alliance with some R-EM clients relative to other R-EM clients. This finding may assist in shifting the nature of supervision toward examining R-EM clients with low alliance ratings and addressing potential barriers in building stronger therapeutic alliances. Summary Three Key Findings The present dissertation aimed to address several limitations in MCC research and found three key findings. 1. There was small MCC between therapist variability using select items from a new MCC measure. The therapist MCC ICC in the complete sample was smaller than hypothesized and smaller in magnitude than the two previous studies that examined MCC variability. The therapist MCC ICC was not significantly smaller than therapist alliance and treatment rating ICCs among the complete sample. However, exploratory analyses across specific cultural identities revealed significantly smaller therapist MCC ICC relative to therapist alliance and treatment rating ICCs for White clients, straight clients, and religious clients. 2. There was a near zero, nonsignificant relationship between MCC and outcome that was consistent across diverse client identities. In contrast to previous MCC and outcome studies, the present study did not detect a relationship between client-rated MCC 96 and clinical outcomes in the sample as a whole. Only among religious clients was there a significant within-therapist effect, potentially indicative of a unique religious environment in Utah. 3. Higher alliance and treatment satisfaction ratings both predicted improved client outcomes. Both alliance ratings and treatment satisfaction ratings significantly predicted client outcomes. Among the complete sample, both within-therapist and between-therapist effects were significant. There was no significant contextual effect, although the 95% HDI was close to zero. Among White clients both within-therapist and between-therapist variability predicted improved outcomes. In contrast, only within-therapist variability predicted improved outcomes for R-EM clients. Limitations The MCC Items One limitation in the present study is the use of two MCC items from the fiveitem Cultural (Missed) Opportunities (CMO) scale. In a validation study of the CMO, client ratings of missed cultural opportunities correlated with worse clinical outcomes (as hypothesized) across both a patient improvement outcome (r = -0.50) and general wellbeing outcome (r = -0.27; Owen et al., 2015). However, this validation study also used a retrospective design with clients not actively in treatment. Classical test theory outlines how an increased number of test items reduces measurement error, yet many health professionals must use short questionnaires due to the individual’s time or item fatigue (Eisinga, te Grotenhuis, & Pelzer, 2013). The current items are administered after a series of 32 questions regarding the client’s distress levels. With this in mind, the counseling 97 center aimed to assess MCC (and other processes) with a few items that were deemed to have good content validity. The creators of the CMO scale noted three psychologists agreed on the content validity of the items, although removing one of the items, one used in the current study, did increase the Cronbach alpha coefficient (yet the results were the same with or without the item; Owen et al., 2015). Method of MCC Measurement The present study used a partial retrospective design where clients were asked to rate their previous session prior to their upcoming session. Concerns applicable to previous retrospective designed MCC studies are also applicable in the present study, yet with a smaller time gap between a session and client rating. Administering process measures after the session is ideal (Henry, Moffitt, Caspi, Langley, & Silva, 1994) yet was not feasible given the high number of clients at the counseling center and data collection procedures. It is possible that asking the client to rate their therapist prior to session adds additional pressures to provide positive or non-answer neutral responses. Additionally, when asking the client to rate their therapist behaviors from a previous session, we may instead be detecting a signal of the client’s general therapy experience and not actual therapist behaviors. Sample Diversity The sample of clients in the present study was large, however, there remains a limited sample of clients with minority identity statuses. For example, when assessing client sexual orientation, 84% of the clients identified as straight and 16% LGBT. 98 Bayesian methods provide an ideal analytic approach to small samples, however there remained too few clients to look at additional demographics such as international students. As more data is collected, it is possible to look further at different client identities. Related to sample diversity, it is possible that aspects of diversity unique to Utah may have influenced the results relative to other settings in the United States or abroad. For example, religion may be more salient for clients at the University of Utah versus a university in a more secular area. Additionally, issues related to race and ethnicity may be less salient for University of Utah college students than other more diverse campuses or in parts of the country where racial issues have reached the national news (e.g., Ferguson, Missouri). Thus, the results from the present study may not hold in different settings. Future studies should examine the MCC and outcome relationship with student and nonstudent populations and examine client demographic variables that may be unique to the research sample. Heterogeneity Within Client Identity Subsamples Related to sample diversity, there were a number of identities within each grouped client identity (race/ethnicity, sexual orientation, and religious affiliation) that were aggregated for statistical purposes. For example, the R-EM variable was comprised of Asian-American, Latino/a, multiracial, African-American, American Indian or Alaskan, and Pacific Islander individuals. There was a similar aggregation for the sexual orientation variable (bisexual, gay, and lesbian) and religion variable. The resulting method of grouping individuals into broad categories reduces one’s ability to detect 99 differences between the groups. Certain identities may experience increased discriminatory behaviors or microaggressions and subsequently may have increased treatment disparities or variable ratings of therapist MCC. Further detailed analyses may provide important evidence on patterns of MCC responses among specific client identities as well as possible disparities in treatment satisfaction, alliance ratings, client outcomes, or treatment length. Naturalistic Data Naturalistic data is inherently messy. Clients meet with one provider for an intake session, then are assigned a therapist for individual therapy, and that therapist might leave the center resulting in a client transfer to a new therapist. In cleaning naturalistic data, I made a number of decision that may or may not have impacted the present results. For example, by taking the modal therapist for each episode I may have attributed client ratings for one therapist to a separate therapist. Additionally, I decided to take the mean rating of the client for each process measure (MCC, alliance, and treatment satisfaction) with the intention of reducing within-person measurement. It is possible that early therapy ratings provide a different or more unique perspective of the client’s experience in therapy (e.g., at session 3 that is commonly done in research studies). Finally, I only examined individual therapy treatment episodes and did not exclude clients actively involved in other treatments at the center such as medication management, clinical workshops (e.g., meditation groups), or group therapy. 100 Future Directions Continue to Test MCC Measures Across Diverse Samples There is no shortage of MCC theory in our field, thus we must continue to test ratings of therapist MCC with clinical outcomes to best determine what our measures are telling us (by calculating and reporting ICCs) and exhausting strong/weak measures and items (Drinane et al., 2014). This measure testing includes direct MCC measures as well as indirect MCC measures such as racial microaggressions or cultural humility. It is possible that anchors in the CMO scale may drive individuals to select “neutral” and difficulties interpreting this “neutral” response along with range restriction can limit our understanding of MCC and client outcomes. Related, it may be that discussions around an individual’s cultural identity are more important for some clients than others. Some individuals may identify more strongly with discussing their identities in relationship to their presenting concerns and maybe these clients more likely to respond in a nonneutral way. It is often assumed in the literature that R-EM individuals are more likely to benefit from MCC-related conversations, yet empirically this need to be tested further. In the Tao et al. meta-analysis (2015), for example, a client’s racial-ethnic status did not moderate the MCC to outcome relationship. Additionally, and discussed above, new items may be helpful to assess MCC by examining if cultural conversations occurred in session, if they are important to the client, or if the client had any negative/positive experiences related to their cultural identities in the previous week. 101 Observer Ratings of Therapist MCC There is certainly value in the client’s perspective in self-report ratings (Chan, 2010), yet some have questioned if clients have the knowledge and skills to adequately rate a construct such as MCC (Ridley & Ridley-Shaw, 2011). To date, no study has examined observer-rated MCC in relation to client improvement or worsening in counseling. This dearth of observer-rated MCC research is surprising given the commonly used CCCI-R measure was originally developed for observer ratings prior to the items being translated to client self-report (Fuertes et al., 2006; Owen, Leach, et al., 2011). The lack of observer-rated studies is likely due to the costly and time-consuming nature of observational coding. However, there is potential that observer ratings may be able to detect some of the more nuanced therapist MCC behaviors that occur throughout the therapy hour. Examining Raw Therapy Data As technologies such as natural language processing and machine learning begin to advance, we can continue to apply these methods to psychotherapy datasets (Imel, Caperton, Tanana, & Atkins, 2017). New technologies provide the ability to learn more about psychotherapy processes such as therapist MCC, alliance, and treatment satisfaction. It is within the raw data of psychotherapy, the audio and words of a session, that we can begin to create classifiers that identify discussions around race, instances of racial microaggressions, and other MCC-related constructs. This approach may let us answer simple questions such as how often race or another cultural variable were discussed during session. Additionally, we can explore if the therapist or the client more 102 frequently initiated conversations regarding cultural differences. We can also address more complex questions such as whether there are changes in voice and vocal tone when the therapist and/or client discuss race, sexual orientation, or religion. With these technologies we can potentially explore a new layer of psychotherapy sessions and better understand many of the processes unfolding during the therapy hour. REFERENCES Alegría, M., Chatterji, P., Wells, K., Cao, Z., Chen, C.-n., Takeuchi, D., … Meng, X.-L. (2008). Disparity in depression treatment among racial and ethnic minority populations in the United States. Psychiatric Services, 59(11), 1264–1272. https://doi.org/10.1176/appi.ps.59.11.1264 American Psychological Association. (2003). Guidelines on multicultural education, training, research, practice, and organizational change for psychologists. American Psychologist, 58(5), 377-402. https://doi.org/10.1037/0003066X.58.5.377 American Psychological Association. (2008). Report of the Task Force on the Implementation of the Multicultural Guidelines. Washington, D.C.: Author. Retrieved from http://www.apa.org/pi/ Arredondo, P., & Perez, P. (2006). Historical perspectives on the multicultural guidelines and contemporary applications. Professional Psychology: Research and Practice, 37(1), 1–5. https://doi.org/10.1037/0735-7028.37.1.1 Arredondo, P., Rosen, D. C., Rice, T., Perez, P., & Tovar-Gamero, Z. G. (2005). Multicultural counseling: A 10-year content analysis of the Journal of Counseling & Development. Journal of Counseling & Development, 83(2), 155–161. Arredondo, P., & Toporek, R. (2004). Multicultural competencies = ethical practice. Journal of Mental Health Counseling, 26(1) , 44-55. Arredondo, P., Toporek, R., Brown, S., Jones, J., Locke, D. C., Sanchez, J., & Stadler, H. (1996). Operationalization of the multicultural counseling competencies. Journal of Multicultural Counseling and Development, 24, 42–78. Atkinson, D. R., Casas, A., & Abreu, J. (1992). Mexican-American acculturation, counselor ethnicity and cultural sensitivity, and perceived counselor competence. Journal of Counseling Psychology, 39(4), 515-520. Atkinson, D. R., Morten, G., & Sue, D. W. (1998). Counseling American minorities (5th ed.). New York: McGraw-Hill. Ault-Brutus, A. A. (2012). Changes in racial-ethnic disparities in use and adequacy of mental health care in the United States, 1990–2003. Psychiatric Services. Retrieved from http://ps.psychiatryonline.org/doi/abs/10.1176/appi.ps.201000397 104 Baldwin, S. A., & Imel, Z. E. (2013). Therapist effects: Findings and methods. In M. J. Lambert (Ed.), Bergin and Garfield’s handbook of psychotherapy and behavior change (5th ed., pp. 258–297). Hoboken, NJ: Wiley. Baldwin, S. A., Imel, Z. E., & Atkins, D. C. (2012). The influence of therapist variance on the dependability of therapists’ alliance scores: A brief comment on “The dependability of alliance assessments: The alliance–outcome correlation is larger than you think” (Crits-Christoph et al., 2011). Journal of Consulting and Clinical Psychology, 80(5), 947–951. https://doi.org/10.1037/a0027935 Baldwin, S. A., & Larson, M. J. (2017). An introduction to using Bayesian linear regression with clinical data. Behaviour Research and Therapy, 98, 58-75. https://doi.org/10.1016/j.brat.2016.12.016 Baldwin, S. A., Wampold, B. E., & Imel, Z. E. (2007). Untangling the alliance-outcome correlation: Exploring the relative importance of therapist and patient variability in the alliance. Journal of Consulting and Clinical Psychology, 75(6), 842–852. https://doi.org/10.1037/0022-006X.75.6.842 Balkin, R. S., Heard, C. C., Lee, S., & Wines, L. A. (2014). A primer for evaluating test bias and test fairness: Implications for multicultural assessment. Journal of Professional Counseling, Practice, Theory, & Research, 41(1), 42. Bernal, G., Bonilla, J., & Bellido, C. (1995). Ecological validity and cultural sensitivity for outcome research: Issues for the cultural adaptation and development of psychosocial treatments with Hispanics. Journal of Abnormal Child Psychology, 23(1), 67-82. Bernal, G., Jiménez-Chafey, M. I., & Domenech Rodríguez, M. M. (2009). Cultural adaptation of treatments: A resource for considering culture in evidence-based practice. Professional Psychology: Research and Practice, 40(4), 361–368. https://doi.org/10.1037/a0016401 Betancourt, H., & López, S. R. (1993). The study of culture, ethnicity, and race in American psychology. American Psychologist, 48(6), 629. Boswell, J. F., Gallagher, M. W., Sauer-Zavala, S. E., Bullis, J., Gorman, J. M., Shear, M. K., … Barlow, D. H. (2013). Patient characteristics and variability in adherence and competence in cognitive-behavioral therapy for panic disorder. Journal of Consulting and Clinical Psychology, 81(3), 443–454. https://doi.org/10.1037/a0031437 Boswell, J. F., Kraus, D. R., Miller, S. D., & Lambert, M. J. (2015). Implementing routine outcome monitoring in clinical practice; Benefits, challenges, and solutions. Psychotherapy Research, 25(1), 6-19. http://dx.doi.org/10.1080/10503307.2013.817696 105 Bostwick, W. B., Boyd, C. J., Hughes, T. L., West, B. T., & McCabe, S. E. (2014). Discrimination and mental health among lesbian, gay, and bisexual adults in the United States. American Journal of Orthopsychiatry, 84(1), 35–45. https://doi.org/10.1037/h0098851 Broman, C. L. (2012). Race differences in the receipt of mental health services among young adults. Psychological Services, 9(1), 38–48. https://doi.org/10.1037/a0027089 Center for Collegiate Mental Health (2012). CCAPS 2012 technical manual. University Park, PA: Author. Chan, D. (2009). So why ask me? Are self-report data really that bad? In C. E. Lance & R. J. Vandenberg (Eds.), Statistical methodological myths and urban legends: Doctrine verity and fable in the organizational and social science (pp. 309-332). New York, NY: Routledge. Chen, J., & Rizzo, J. (2010). Racial and ethnic disparities in use of psychotherapy: Evidence from U.S. national survey data. Psychiatric Services, 61(4), 364–372. Cheung, F. M. (2012). Mainstreaming culture in psychology. American Psychologist, 67(8), 721–730. https://doi.org/10.1037/a0029876 Cheung, F. M., van de Vijver, F. J. R., & Leong, F. T. L. (2011). Toward a new approach to the study of personality in culture. American Psychologist, 66(7), 593–603. https://doi.org/10.1037/a0022389 Chu, J., Leino, A., Pflum, S., & Sue, S. (2016). A model for the theoretical basis of cultural competency to guide psychotherapy. Professional Psychology: Research and Practice, 47(1), 18–29. https://doi.org/10.1037/pro0000055 Coleman, H. L. K. (2004). Multicultural counseling competencies in a pluralistic society. Journal of Mental Health Counseling, 26(1), 56-66. Constantine, M. G. (2001). Predictors of observer ratings of multicultural counseling competence in Black, Latino, and White American trainees. Journal of Counseling Psychology, 48(4), 456–462. https://doi.org/10.1037//00220167.48.4.456 Constantine, M. G. (2002). Predictors of satisfaction with counseling: Racial and ethnic minority clients’ attitudes toward counseling and ratings of their counselors’ general and multicultural counseling competence. Journal of Counseling Psychology, 49(2), 255–263. https://doi.org/10.1037//0022-0167.49.2.255 Constantine, M. G. (2007). Racial microaggressions against African American clients in cross-racial counseling relationships. Journal of Counseling Psychology, 54(1), 1–16. https://doi.org/10.1037/0022-0167.54.1.1 106 Creedon, T. B., & Cook, B. L. (2016). Access to mental health care increased but not for substance use, while disparities remain. Health Affairs, 35(6), 1017–1021. https://doi.org/10.1377/hlthaff.2016.0098 Crits-Christoph, P., Baranackie, K., Kurcias, J., Beck, A., Carroll, K., Perry, K., … Zitrin, C. (1991). Meta‐analysis of therapist effects in psychotherapy outcome studies. Psychotherapy Research, 1(2), 81–91. https://doi.org/10.1080/10503309112331335511 Crits-Christoph, P., Gallop, R., Temes, C. M., Woody, G., Ball S. A., Martino, S., & Carroll, K. M., The alliance in motivational enhancement therapy and counseling as usual for substance use problems. Journal of Consulting and Clinical Psychology, 77(6), 1125-1135. https://doi.org/10.1037/a0017045 Davis, D. E., DeBlaere, C., Brubaker, K., Owen, J., Jordan, T. A., Hook, J. N., & Van Tongeren, D. R. (2016). Microaggressions and perceptions of cultural humility in counseling. Journal of Counseling & Development, 94(4), 483–493. https://doi.org/10.1002/jcad.12107 DeRubeis, R. J., Brotman, M. A., & Gibbons, C. J. (2005). A conceptual and methodological analysis of the nonspecifics argument. Clinical Psychology: Science and Practice, 12(2), 174–183. https://doi.org/10.1093/clipsy.bpi022 Dillon, F. R., Odera, L., Fons-Scheyd, A., Sheu, H-B., Ebersole, R. C., & Spanierman, L. B. (2016). A dyadic study of multicultural counseling competence. Journal of Counseling Psychology, 63(1), 57–66. https://doi.org/10.1037/cou0000118 Dinger, U., Strack, M., Leichsenring, F., Wilmers, F., & Schauenburg, H. (2008). Therapist effects on outcome and alliance in inpatient psychotherapy. Journal of Clinical Psychology, 64, 344-354. doi:10.1002/jclp.20443 Drinane, J. M., Owen, J., Adelson, J. L., & Rodolfa, E. (2014). Multicultural competencies: What are we measuring? Psychotherapy Research, 26(3), 342–351. https://doi.org/10.1080/10503307.2014.983581 Dunn, T. W., Smith, T. B., & Montoya, J. A. (2006). Multicultural competency instrumentation: A review and analysis of reliability generalization. Journal of Counseling and Development, 84(4), 471. Eisenberg, D., Hunt, J., Speer, N., & Zivin, K. (2011). Mental health service utilization among college students in the United States: The Journal of Nervous and Mental Disease, 199(5), 301–308. https://doi.org/10.1097/NMD.0b013e3182175123 Eisinga, R., & te Grotenhuis, M. (2012). The reliability of a two-item scale: Pearson, Cronbach, or Spearman-Brown? International Journal of Public Health, 58, 637642. https://doi.org/10.1007/s00038-012-0416-3 107 Ellis, B. H., Murray, K., & Barrett, C. (2014). Understanding the mental health of refugees: Trauma, stress, and the cultural context. In The Massachusetts General Hospital textbook on diversity and cultural sensitivity in mental health (pp. 165187). New York: Springer. Emons, W. H. M., Sijtsma, K., & Meijer, R. R. (2007). On the consistency of individual classification using short scales. Psychological Methods, 12(1), 105-120. https://doi.org/ 10.1037/1082-989X.12.1.105 Falkenström, F., Grantström, F., & Holmqvist, R. (2014). Working alliance predicts psychotherapy outcome even while controlling for prior symptom improvement. Psychotherapy Research, 24(2), 146-159. http://dx.doi.org/10.1080/10503307.2013.847985 Farrelly, S., Jeffery, D., Rüsch, N., Williams, P., Thornicroft, G., & Clement, S. (2015). The link between mental health-related discrimination and suicidality: Service user perspectives. Psychological Medicine, 45(10), 2013–2022. https://doi.org/10.1017/S0033291714003158 Fouad, N. A., Santana, M., & Ghosh, A. (2017). Empirical influence of the multicultural guidelines: A brief report. Cultural Diversity and Ethnic Minority Psychology. https://doi.org/10.1037/cdp0000136 Frank, J. D., & Frank, J. B. (1991). Persuasion and healing: A comparative study of psychotherapy (3rd ed.). Baltimore: Johns Hopkins University Press. Fuertes, J. N., Stracuzzi, T. I., Bennett, J., Scheinholtz, J., Mislowack, A., Hersh, M., & Cheng, D. (2006). Therapist multicultural competency: A study of therapy dyads. Psychotherapy: Theory, Research, Practice, Training, 43(4), 480–490. https://doi.org/10.1037/0033-3204.43.4.480 Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Doing Bayesian data analysis (3rd ed.). Boca Raton, FL: CRC Press. Gelman, A., & Rubin, D. R. (1992). Inference from iterative simulation using multiple sequences. Statistical Sciences, 7(4), 457-511. Hadfield, J. (2017). MCMCglmm: MCMC generalised linear mixed models. R Package version 2.5. Helfrich, H. (1999). Beyond the dilemma of cross-cultural psychology: Resolving the tension between etic and emic approaches. Culture & Psychology, 5(2), 131–153. Henry, B., Moffitt, T. E., Caspi, A., Langley, J., & Silvia, P. A. (1994). On the "remembrance of things past": A longitudinal evaluation of the retrospective method. Psychological Assessment, 6(2), 92-101. 108 Hinton, D. E., Chhean, D., Pich, V., Safren, S. A., Hofmann, S. G., & Pollack, M. H. (2005). A randomized controlled trial of cognitive-behavior therapy for Cambodian refugees with treatment-resistant PTSD and panick attacks: A crossover design. Journal of Traumatic Stress, 18(6), 617-629. https://doi.org/10.1002/jts.20070 Holcomb-McCoy, C. C., & Myers, J. E. (1999). Multicultural competence and counselor training: A national survey. Journal of Counseling and Development: JCD, 77(3), 294. Hook, J. N., Davis, D. E., Owen, J., Worthington, E. L., & Utsey, S. O. (2013). Cultural humility: Measuring openness to culturally diverse clients. Journal of Counseling Psychology, 60(3), 353–366. https://doi.org/10.1037/a0032595 Hornsey, M. J. (2008). Social identity theory and self-categorization theory: A historical review. Social and Personality Psychology Compass, 2(1), 204–222. https://doi.org/10.1111/j.1751-9004.2007.00066.x Hornsey, M. J., & Hogg, M. A. (2000). Assimilation and diversity: An integrative model of subgroup relations. Personality and Social Psychology Review, 4(2), 143–156. Horvath, A. O., Del Re, A. C., Flückiger, C., & Symonds, D. (2011). Alliance in individual psychotherapy. Psychotherapy, 48(1), 9–16. https://doi.org/10.1037/a0022186 Horvath, A. O., & Greenberg, L. S. (1989). Development and validation of the Working Alliance Inventory. Journal of Counseling Psychology, 36(2), 223. Hox, J. J., Moerbeek, M., & van de Schoot, R. (2010). Multilevel analysis: Techniques and applications (3rd ed.). New York, NY: Routledge. Hoyt, W. T., Warbasse, R. E., & Chu, E. Y. (2006). Construct validation in counseling psychology research. The Counseling Psychologist, 34(6), 769-805. https://doi.org/10.1177/0011000006287389 Huey, S. J., Tilley, J. L., Jones, E. O., & Smith, C. A. (2014). The contribution of cultural competence to evidence-based care for ethnically diverse populations. Annual Review of Clinical Psychology, 10(1), 305–338. https://doi.org/10.1146/annurevclinpsy-032813-153729 Hughes, M., Kiecolt, K. J., Keith, V. M., & Demo, D. H. (2015). Racial identity and well-being among African Americans. Social Psychology Quarterly, 78(1), 25– 48. https://doi.org/10.1177/0190272514554043 Imel, Z. E., Baer, J. S., Martino, S., Ball, S. A., & Carroll, K. M. (2011). Mutual influence in therapist competence and adherence to motivational enhancement therapy. Drug and Alcohol Dependence, 115(3), 229–236. https://doi.org/10.1016/j.drugalcdep.2010.11.010 109 Imel, Z. E., Baldwin, S., Atkins, D. C., Owen, J., Baardseth, T., & Wampold, B. E. (2011). Racial/ethnic disparities in therapist effectiveness: A conceptualization and initial study of cultural competence. Journal of Counseling Psychology, 58(3), 290–298. https://doi.org/10.1037/a0023284 Imel, Z. E., Baldwin, S. A., Baer, J. S., Hartzler, B., Dunn, C., Rosengren, D. B., & Atkins, D. C. (2014). Evaluating therapist adherence in motivational interviewing by comparing performance with standardized and real patients. Journal of Consulting and Clinical Psychology, 82(3), 472–481. https://doi.org/10.1037/a0036158 Imel, Z. E., Caperton, D. D., Tanana, M., & Atkins, D. C. (2017). Technology-enhanced human interaction in psychotherapy. Journal of Counseling Psychology, 64(4), 385-393. https://doi.org/ 10.1037/cou0000213 Imel, Z. E., Hubbard, R. A., Rutter, C. M., & Simon, G. (2013). Patient-rated alliance as a measure of therapist performance in two clinical settings. Journal of Consulting and Clinical Psychology, 81(1), 154–165. https://doi.org/10.1037/a0030903 Imel, Z. E., Sheng, E., Baldwin, S. A., & Atkins, D. C. (2015). Removing very lowperforming therapists: A simulation of performance-based retention in psychotherapy. Psychotherapy, 52(3), 329–336. https://doi.org/10.1037/pst0000023 Institute of Medicine. (2003). Unequal treatment: Confronting racial and ethnic disparities in health care. Washington, D.C.: National Academies Press. Retrieved from http://www.nap.edu/catalog/10260 Kenny, D. A. (1994). Interpersonal perception: A social relations analysis. New York: Guilford Press. Kenny, D. A., Kashy, D. A., & Cook, W. L. (2006). Dyadic data analysis. New York: Guilford Press. Kilbourne, A. M., Keyser, D., & Pincus, H. A. (2010). Challenges and opportunities in measuring the quality of mental health care. The Canadian Journal of Psychiatry, 55(9), 549-557. Kim, B. S. K., Li, L. C., & Liang, T. H. (2002). Effects of Asian American client adherence to Asian cultural values, session goal, and counselor emphasis of client expression on career counseling process. Journal of Counseling Psychology, 49(3), 342–354. https://doi.org/10.1037//0022-0167.49.3.342 Kochhar, R., & Fry, R. (2014). Wealth inequality has widened along racial, ethnic lines since end of Great Recession. Pew Research Center, 12. Retrieved from http://www.pewresearch.org/fact-tank/2014/12/12/racial-wealth-gaps-greatrecession/ 110 Korman, M. (1974). National conference on levels and patterns of professional training in psychology: The major themes. American Psychologist, 29(6), 441. Krieger, N. (2014). Discrimination and health inequities. International Journal of Health Services, 44(4), 643–710. https://doi.org/10.2190/HS.44.4.b Kruschke, J. K. (2015). Doing Bayesian data analysis: A tutorial with R and BUGS (2nd ed.). San Diego, CA: Academic Press. Kugelmass, H. (2016). “Sorry, I’m not accepting new patients” an audit study of access to mental health care. Journal of Health and Social Behavior, 57(2), 168–183. LaFromboise, T., Coleman, H., & Gerton, J. (1993). Psychological impact of biculturalism. Psychological Bulletin, 114(3), 395-412. LaFromboise, T. D., Coleman, H. L., & Hernandez, A. (1991). Development and factor structure of the Cross-Cultural Counseling Inventory—Revised. Professional Psychology: Research and Practice, 22(5), 380. Lee, R. L., & Ramirez, M., III. (2000). The history, current status, and future of multicultural psychotherapy. In Cuéllar, I. & Paniagua, F. A. (Eds.), Handbook of multicultural mental health: Assessment and treatment of diverse populations (pp. 280-303). San Diego, California: Academic Press Lopez, S. R. (1997). Cultural competence in psychotherapy: A guide for clinicians and their supervisors. In C. E. Watkins (Ed.), Handbook of psychotherapy supervision (pp. 570-588). Hoboken, NJ: Wiley. Lopez, S. R., Kopelowicz, A., & Canive, J. M. (2002). Strategies in developing culturally congruent family interventions for schizophrenia: The case of Hispanics. In H. P. Lefley & D. L. Johnson (Eds.), Family interventions in mental illness: International perspective. (pp. 61-90). Westport, CT: Praeger. Lora, A. (2013). Call for information, call for quality in mental health care. Epidemiology and Psychiatric Sciences, 22(1), 9–13. https://doi.org/10.1017/S2045796012000716 Luborsky, L., Chandler, M., Auerbach, A. H., & Cohen, J. (1971). Factors influencing the outcome of psychotherapy: A review of quantiative research. Psychological Bulletin, 75(3), 145-185. Luborsky, L., Singer, B., & Luborsky, L. (1975). Comparative studies of psychotherapies: Is it true that everyone has won and all must have prizes? Archives of General Psychiatry, 32(8), 995–8. Manseau, M., & Case, B. G. (2014). Racial-ethnic disparities in outpatient mental health visits to US physicians, 1993–2008. Psychiatric Services, 65(1), 59–67. 111 Marcus, D. K., Kashy, D. A., & Baldwin, S. A. (2009). Studying psychotherapy using the one-with-many design: The therapeutic alliance as an exemplar. Journal of Counseling Psychology, 56(4), 537–548. https://doi.org/10.1037/a0017291 McGlynn, E. A., Norquist, G. S., Wells, K. B., Sullivan, G., & Liberman, R. P. (1988). Quality-of-care research in mental health: Responding to the challenge. Inquiry, 25(1), 157-170. McGuire, T. G., & Miranda, J. (2008). New evidence regarding racial and ethnic disparities in mental health: Policy implications. Health Affairs, 27(2), 393–403. https://doi.org/10.1377/hlthaff.27.2.393 McMorrow, S., Long, S. K., Kenney, G. M., & Anderson, N. (2015). Uninsurance disparities have narrowed for Black and Hispanic adults under The Affordable Care Act. Health Affairs, 34(10), 1774–1778. https://doi.org/10.1377/hlthaff.2015.0757 Mechanic, D., & Olfson, M. (2016). The relevance of the Affordable Care Act for improving mental health care. Annual Review of Clinical Psychology, 12(1), 515– 542. https://doi.org/10.1146/annurev-clinpsy-021815-092936 Morton, E. (2011). The incidence of racial microaggressions in the cross-racial counseling dyad (Unpublished doctoral dissertation). Saint Louis University, St. Louis, MO. Nam, E., Matejkowski, J., & Lee, S. (2016). Racial/ethnic differences in contemporaneous use of mental health and substance use treatment among individuals experiencing both mental illness and substance use disorders. Psychiatric Quarterly, 88(1), 185–198. https://doi.org/10.1007/s11126-016-94440 Neville, H. A., Lilly, R. L., Duran, G., Lee, R. M., & Browne, L. (2000). Construction and initial validation of the Color-Blind Racial Attitudes Scale (CoBRAS). Journal of Counseling Psychology, 47, 59–70. http://dx.doi.org/10.1037/00220167.47.1.59 Owen, J., Jordan, T. A., II, Turner, D., Davis, D. E., Hook, J. N., & Leach, M. M. (2014). Therapists' multicultural orientation: Client perceptions of cultural humility, spiritual/religious commitment, and therapy outcomes. Journal of Psychology & Theology, 42(1), 91-98. Owen, J., Leach, M. M., Wampold, B., & Rodolfa, E. (2011). Client and therapist variability in clients’ perceptions of their therapists’ multicultural competencies. Journal of Counseling Psychology, 58(1), 1–9. https://doi.org/10.1037/a0021496 112 Owen, J., Tao, K. W., Drinane, J. M., Hook, J., Davis, D. E., & Kune, N. F. (2015). Client perceptions of therapists’ multicultural orientation: Cultural (missed) opportunities and cultural humility. Professional Psychology: Research and Practice, 47(1), 30-37. https://doi.org/10.1037/pro0000046 Owen, J., Tao, K. W., Imel, Z. E., Wampold, B. E., & Rodolfa, E. (2014). Addressing racial and ethnic microaggressions in therapy. Professional Psychology: Research and Practice, 45(4), 283–290. https://doi.org/10.1037/a0037420 Owen, J. J., Tao, K., Leach, M. M., & Rodolfa, E. (2011). Clients’ perceptions of their psychotherapists’ multicultural orientation. Psychotherapy, 48(3), 274–282. https://doi.org/10.1037/a0022065 Pace, B. T., Tanana, M., Xiao, B., Dembe, A., Soma, C. S., Steyvers, M., … Imel, Z. E. (2016). What about the words?: Natural language processing in psychotherapy. Psychotherapy Bulletin, 50(1), 14-18. Patterson, C. H. (1996). Multicultural counseling: From diversity to universality. Journal of Counseling and Development, 74, 227-231. Patterson, C. H. (2004). Do we need multicultural counseling competencies? Journal of Mental Health Counseling, 26(1), 67-73. Pedersen, P. B. (1978). Four dimensions of cross-cultural skill in counselor training. Personnel and Guidance Journal, 56(2), 480-484. Pieterse, A. L., Evans, S. A., Risner-Butner, A., Collins, N. M., & Mason, L. B. (2009). Multicultural competence and social justice training in counseling psychology and counselor education: A review and analysis of a sample of multicultural course syllabi. The Counseling Psychologist, 37(1), 93–115. https://doi.org/10.1177/0011000008319986 Pilgrim, D. (2014). Historical resonances of the DSM-5 dispute: American exceptionalism or Eurocentrism? History of the Human Sciences, 27(2), 97–117. https://doi.org/10.1177/0952695114527998 Plummer, M. (2016). coda: Output Analysis and Diagnostics for MCMC. R Package version 0.19-1. Pope-Davis, D. B., Ligiero, D. P., Liang, C., & Codrington, J. (2001). Fifteen years of the Journal of Multicultural Counseling and Development: A content analysis. Journal of Multicultural Counseling and Development, 29(4), 226. Powell, B. J. (2011). Quality of care and implementation research in children's mental health. Psychiatric Services, 62(1), 103. R Core Team (2017). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. http://www.R-project.org/. 113 Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models (2nd ed.). Thousand Oaks, CA: Sage. Ridley, C. R., & Shaw-Ridley, M. (2011). Multicultural counseling competencies: An analysis of research on clients’ perceptions: Comment on Owen, Leach, Wampold, and Rodolfa (2011). Journal of Counseling Psychology, 58(1), 16–21. https://doi.org/10.1037/a0022221 Roll, J. M., Kennedy, J., Tran, M., & Howell, D. (2013). Disparities in unmet need for mental health services in the United States, 1997-2010. Psychiatric Services, 64(1), 80-82. Rosselló, J., & Bernal, G. (1999). The efficacy of cognitive-behavioral and interpersonal treatments for depression in Puerto Rican adolescents. Journal of Consulting and Clinical Psychology, 67(5), 734. Rosselló, J., Bernal, G., & Rivera-Medina, C. (2012). Individual and group CBT and IPT for Puerto Rican adolescents with depressive symptoms. Journal of Latina/O Psychology, 1(S), 36–51. https://doi.org/10.1037/2168-1678.1.S.36 Safran, M. A., Mays, R. A., Jr, Huang, L. N., McCuan, R., Pham, P. K., Fisher, S. K., … Trachtenberg, A. (2009). Mental health disparities. American Journal of Public Health, 99(11), 1962–1966. Saloner, B., & Cook, B. L. (2013). Blacks and Hispanics are less likely than Whites to complete addiction treatment, largely due to socioeconomic factors. Health Affairs, 32(1), 135–145. https://doi.org/10.1377/hlthaff.2011.0983 Schiefele A-K., Lutz, W., Barkham, M., Rubel, J., Böhnke, J., Delgadillo, J., … Lambert, M. J. (2016). Reliability of therapist effects in practice-based psychotherapy research: A guide for the planning of future studies. Administration and Policy in Mental Health, 44(5), 598-613. https://doi.org/19.1007/s10488-016-0736-3 Sellers, R. M., Copeland-Linder, N., Martin, P. P., & Lewis, R. (2006). Racial identity matters: The relationship between racial discrimination and psychological functioning in African American adolescents. Journal of Research on Adolescence, 16(2), 187–216. Shin, R. Q., Smith, L. C., Welch, J. C., & Ezeofor, I. (2016). Is Allison more likely than Lakisha to receive a callback from counseling professionals? A racism audit study. The Counseling Psychologist, 44(8), 1187-1211. https://doi.org/11000016668814. Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis: Modeling change and event occurrence. New York, NY: Oxford University Press. Smith, M. L., & Glass, G. V. (1977). Meta-analysis of psychotherapy outcome studies. American Psychologist, 32(9), 752–760. 114 Snowden, L. R., & Yamada, A.-M. (2005). Cultural differences in access to care. Annual Review of Clinical Psychology, 1(1), 143–166. https://doi.org/10.1146/annurev.clinpsy.1.102803.143846 Substance Abuse and Mental Health Services Administration. (2015). Racial/ethnic differences in mental health service use among adults. HHS Publication No. SMA-15-4906. Rockville, MD: Author. Sue, D. W., Arredondo, P., & McDavis, R. J. (1992). Multicultural counseling competencies and stands: A call to the profession. Journal of Counseling & Development, 70, 477–486. Sue, D. W., Bernier, J. E., Durran, A., Feinberg, L., Pedersen, P., Smith, E. J., & Vasquez-Nuttall, E. (1982). Position paper: Cross-cultural counseling competencies. The Counseling Psychologist, 10, 45–52. Sue, D. W., & Sue, D. (2012). Counseling the culturally diverse: Theory and practice (6th ed.). Hoboken, NJ: Wiley. Sue, S. (1998). In search of cultural competence in psychotherapy and counseling. American Psychologist, 53(4), 440. Sue, S., Zane, N., Nagayama Hall, G. C., & Berger, L. K. (2009). The case for cultural competency in psychotherapeutic interventions. Annual Review of Psychology, 60(1), 525–548. https://doi.org/10.1146/annurev.psych.60.110707.163651 Tao, K. W., Owen, J., Pace, B. T., & Imel, Z. E. (2015). A meta-analysis of multicultural competencies and psychotherapy process and outcome. Journal of Counseling Psychology, 62(3), 337–350. https://doi.org/10.1037/cou0000086 Thornicroft, G. (2008). Stigma and discrimination limit access to mental health care. Epidemiologia E Psichiatria Sociale, 17(1), 14–19. https://doi.org/10.1017/S1121189X00002621 Thornicroft, G., Mehta, N., Clement, S., Evans-Lacko, S., Doherty, M., Rose, D., … Henderson, C. (2016). Evidence for effective interventions to reduce mentalhealth-related stigma and discrimination. The Lancet, 387(10023), 1123–1132. Tskhay, K. O., & Rule, N. O. (2013). Accuracy in categorizing perceptually ambiguous groups: A review and meta-analysis. Personality and Social Psychology Review, 17(1), 72–86. U.S. Bureau of the Census. (2015). Projections of the size and composition of the US population: 2014 to 2060. Current Population Reports, 1-13. Retrieved from http://www.census.gov/content/dam/Census/library/publications/2015/demo/p251143.pdf 115 Vontress, C. E., & Jackson, M. L. (2004). Reactions to the multicultural counseling competencies debate. Journal of Mental Health Counseling, 26(1), 74-80. Vostanis, P. (2014). Meeting the mental health needs of refugees and asylum seekers. The British Journal of Psychiatry, 204(3), 176–177. https://doi.org/10.1192/bjp.bp.113.134742 Walfish, S., McAlister, B., O’Donnell, P., & Lambert, M. J. (2012). An investigation of self-assessment bias in mental health providers. Psychological Reports, 110(2), 639–644. Wampold, B. E., & Imel, Z. E. (2015). The great psychotherapy debate: The research evidence for what works in psychotherapy (2nd ed.). New York, NY: Routledge. Wang, P. S., Berglund, P., & Kessler, R. C. (2000). Recent care of common mental disorders in the United States. Journal of General Internal Medicine, 15(5), 284– 292. Ward, E. C. (2002). Cultural competence, cultural mistrust, working alliance and racial and ethnic minority clients’ experience of counseling: A mixed method study (Unpublished doctoral dissertation). University of Wisconsin-Madison, Madison, WI. Washington v. Trump, 2:17-cv-00141 (2017). Webb, C. A., DeRubeis, R. J., & Barber, J. P. (2010). Therapist adherence/competence and treatment outcome: A meta-analytic review. Journal of Consulting and Clinical Psychology, 78(2), 200–211. https://doi.org/10.1037/a0018912 Weinrach, S. G., & Thomas, K. R. (1996). The counseling profession's commitment to diversity-sensitive counseling: A critical reassessment Journal of Counseling & Development, 74, 472-477. Weinrach, S. G., & Thomas, K. R. (2002). A critical analysis of the multicultural counseling competencies: Implications for the practice of mental health counseling. Journal of Mental Health Counseling, 24(1), 20-35. Weisman de Mamani, A. G., Tuchman, N., & Duarte, E. A. (2010). Incorporating religion/spirituality into treatment for serious mental illness. Cognitive and Behavioral Practice, 17(4), 348–357. Whaley, A. L., & Davis, K. E. (2007). Cultural competence and evidence-based practice in mental health services: A complementary perspective. American Psychologist, 62(6), 563–574. https://doi.org/10.1037/0003-066X.62.6.563 Worthington, R. L., Mobley, M., Franks, R. P., & Tan, J. A. (2000). Multicultural counseling competencies: Verbal content, counselor attributions, and social desirability. Journal of Counseling Psychology, 47(4), 460. 116 Worthington, R. L., Soth-McNett, A. M., & Moreno, M. V. (2007). Multicultural counseling competencies research: A 20-year content analysis. Journal of Counseling Psychology, 54(4), 351–361. https://doi.org/10.1037/00220167.54.4.351 Yeung, A., Shyu, I., Fisher, L., Wu, S., Yang, H., & Fava, M. (2010). Culturally sensitive collaborative treatment for depressed Chinese Americans in primary care. American Journal of Public Health, 100(12), 2397–2402. Young, A. S., Klap, R., Sherbourne, C. D., & Wells, K. B. (2001). The quality of care for depressive and anxiety disorders in the United States. Archives of General Psychiatry, 58, 55-61. Zuroff, D. C., Kelly, A. C., Leybman, M. J., Blatt, S. J., & Wampold, B. E. (2010). Between-therapist and within-therapist differences in the quality of the therapeutic relationship: Effects of maladjustment and self-critical perfectionism. Journal of Clinical Psychology, 66(7), 681-697. https://doi.org/10.1002/jclp.20683