A pilot implementation study of school-based universal screening for language impairment

Researchers estimate that roughly 7%-8% of children will enter kindergarten with deficits in language that cannot be attributed to comorbid conditions such as intellectual deficits or other neurodevelopmental disorders. Despite its prevalence, less than a quarter of children are identified and receive treatment. Children with language impairments who go unidentified, and consequently, untreated at the entry of kindergarten, are at cumulative risk for academic and social difficulties during their formative and later school years. Public schools play a central role in the identification of this population. A vast majority of public-school districts identify children with language delay using a referral-based format that relies heavily on referral sources such as general education teachers and parents. Research has shown that vulnerabilities may exist in such systems. Currently, there is no policy supporting universal screening for language impairment identification in public schools. The primary purpose of this study was to conduct a practice-based hybrid-implementation pilot research study with a local school district to assess the feasibility of annual screenings for language impairment. This was achieved by addressing two specific aims. The first aim assessed the usefulness of the Redmond Sentence Recall (RSR) measure as a universal screener. The second aim targeted the facilitators and barriers to the systematic uptake of an annual-based screening format specific to this school district. Regarding the first aim, 165 kindergarten students from two school sites were administered the RSR screening measure. Of these, 19 students failed the screening and participated in confirmatory testing. The positive predictive rate associated with the RSR measure was .74, representing an improvement over estimates of preexisting district-level general education teacher referral positive predictive rates of .35. Positive case rates associated with the screening (7.88%) aligned with well-accepted prevalence rates. Regarding the second aim, two focus groups were conducted with district teaching assistants who administered the screener (focus group 1) and district Related Services administrators, kindergarten teachers, and speech-language pathologists (focus group 2). A list of facilitators and barriers to a universal screening format was identified. Survey statements were then developed from this list to assess more broadly levels of agreement within the school district. Accordingly, 66 speech-language pathologists and 88 general education teachers participated in the survey. Key findings included: 1) Wide support for an annual- over referral-based identification format. 2) The negative impact of burden on key stakeholders under both referral- and annual-based formats. 3) The identification of speech-language technicians or teaching assistants as preferred personnel to administer regular screenings. 4) The important role Response to Intervention programs stand to play to reduce burdens associated with universal screenings. 5) The need for increased awareness/education with regard to child language development and impairment among general education teachers and the public at large.

A PILOT IMPLEMENTATION STUDY OF SCHOOL-BASED UNIVERSAL SCREENING FOR LANGUAGE IMPAIRMENT by Tyler T. Christopulos A dissertation submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Speech-Language Pathology and Audiology Department of Communication Sciences and Disorders The University of Utah December 2019 Copyright © Tyler T. Christopulos 2019 All Rights Reserved The University of Utah Graduate School STATEMENT OF DISSERTATION APPROVAL The dissertation of Tyler T. Christopulos has been approved by the following supervisory committee members: Sean Redmond , Chair 9/3/2019 Date Approved Jacob Kean , Member 9/3/2019 Date Approved Christina Gringeri , Member 9/3/2019 Date Approved Stacy Manwaring , Member 9/3/2019 Date Approved Robert Kraemer , Member 9/3/2019 Date Approved and by the Department/College/School of Michael Blomgren , Chair/Dean of Communication Sciences and Disorders and by David B. Kieda, Dean of The Graduate School. ABSTRACT Researchers estimate that roughly 7%–8% of children will enter kindergarten with deficits in language that cannot be attributed to comorbid conditions such as intellectual deficits or other neurodevelopmental disorders. Despite its prevalence, less than a quarter of children are identified and receive treatment. Children with language impairments who go unidentified, and consequently, untreated at the entry of kindergarten, are at cumulative risk for academic and social difficulties during their formative and later school years. Public schools play a central role in the identification of this population. A vast majority of public-school districts identify children with language delay using a referral-based format that relies heavily on referral sources such as general education teachers and parents. Research has shown that vulnerabilities may exist in such systems. Currently, there is no policy supporting universal screening for language impairment identification in public schools. The primary purpose of this study was to conduct a practice-based hybrid-implementation pilot research study with a local school district to assess the feasibility of annual screenings for language impairment. This was achieved by addressing two specific aims. The first aim assessed the usefulness of the Redmond Sentence Recall (RSR) measure as a universal screener. The second aim targeted the facilitators and barriers to the systematic uptake of an annual-based screening format specific to this school district. Regarding the first aim, 165 kindergarten students from two school sites were administered the RSR screening measure. Of these, 19 students failed the screening and participated in confirmatory testing. The positive predictive rate associated with the RSR measure was .74, representing an improvement over estimates of preexisting district-level general education teacher referral positive predictive rates of .35. Positive case rates associated with the screening (7.88%) aligned with well-accepted prevalence rates. Regarding the second aim, two focus groups were conducted with district teaching assistants who administered the screener (focus group 1) and district Related Services administrators, kindergarten teachers, and speech-language pathologists (focus group 2). A list of facilitators and barriers to a universal screening format was identified. Survey statements were then developed from this list to assess more broadly levels of agreement within the school district. Accordingly, 66 speech-language pathologists and 88 general education teachers participated in the survey. Key findings included: 1) Wide support for an annual- over referral-based identification format. 2) The negative impact of burden on key stakeholders under both referral- and annual-based formats. 3) The identification of speech-language technicians or teaching assistants as preferred personnel to administer regular screenings. 4) The important role Response to Intervention programs stand to play to reduce burdens associated with universal screenings. 5) The need for increased awareness/education with regard to child language development and impairment among general education teachers and the public at large. iv TABLE OF CONTENTS ABSTRACT ....................................................................................................................iii LIST OF TABLES ..........................................................................................................viii . LIST OF FIGURES .........................................................................................................ix ACKNOWLEDGMENTS ...............................................................................................x Chapters 1. INTRODUCTION ………………………………………...…………………….…...1 Problem Statement ……………………………………………......….…………1 Purpose Statement ………………………………………………..…………… 2 Research Aims …………………………………………………………………3 2. LITERATURE BASED RATIONALE ……………………………………….....….4 Literature Review Overview …..……………………………..………………. 4 Specific Language Impairment Profile ……………………..………………….5 Primary Clinical Markers of SLI ……………………………………… 5 Short and Long-term Personal Impacts of SLI ………………………….8 Awareness of SLI ………………………………………………………… 11 Identification of SLI ……………………………………………………………12 Role of Screening for Language Impairment Identification ……………. 13 Types of Screening Measures ……………………………………..……14 Use of Diagnostic Evidence Data in Public Schools …………..………… 18 Redmond Sentence Recall Measure (RSR)……………..……………..… 21 Implementation Research …………………………………………………… 22 Structure of Implementation Research – a Review of Nilsen (2015) … 23 Implementation Research in Public Schools …………………………. 26 School-Based Guiding Conceptual Sources ………………………..… 31 Quality Enhancement Research Initiative (QUERI) ……………..…….32 Hybrid Implementation Design ………………………….………..……36 3. PRELIMINARY FILE REVIEW STUDY ………………...….……………….……45 Introduction ………………………………………………………………..……45 Research Questions…………………………………………………………….... 47 Methods …………………………………………………………………...….… 47 Results ……………………………………………………….………………….49 Discussion …………………………………………………………….……… 50 4. METHODS ………………………….…………………………….…...….……….. 54 Participants …………………………………………..…………………………54 Aim 1: Research Questions 1 and 2…………………………………….54 Aim 2: Research Question 3……………………………………………56 Aim 2: Research Question 4……………………………………………57 Procedures ………………………………………………………………………58 Aim 1: Research Questions 1 and 2…………………………………….59 Aim 2: Research Question 3…………………………………...……… 61 Aim 2: Research Question 4……………………………………………64 Analysis …………………………………………..………...……………..…...65 Aim 1: Research Questions 1 and 2…………………………………… 65 Aim 2: Research Question 3……………………………………………66 Aim 2: Research Question 4……………………………………………67 5. RESULTS ………………...…………………………………………..……………. 69 Aim 1: Research Questions 1 and 2: RSR Screener Results……………………69 Screenings and Confirmatory Testing………………………………….70 Aim 2: Research Questions 3: Focus Group Data Results …………………….72 Code Group Overview………………………………………………….73 Code Group Detail………………………………………………………74 Aim 2: Research Questions 4: Survey Data Results……………………….……85 Survey Statement Overview…………………………………………….85 6. DISCUSSION ………………………….…………………...…………..……….…..104 Review of Findings ………………………………………………………..……106 Aim 1: Research Questions 1 ………………………….………………106 Aim 1: Research Questions 2 ………………………………………….108 Aim 2: Research Question 3……………………………………………109 Aim 2: Research Question 4……………………………………………109 Qualitative Research Impact …………………………………..………………..116 Implementation Research Impact ……………………………..………………..118 Practical and Clinical Implications ………………………….…………………120 Prevalence Support for Increased Identification Efforts……………….121 Availability of Effective Measures ……………………………………... 122 vi The Impact of RTI on Burden Reeducation and SLTs or TAs as an Initial Good-Fit for Regular Screening Administration……………….. 122 The Need for Increased Efforts to Bolster Awareness and Education 123 of Language Impairment…………………………………………………… Limitations and Future Directions ……………………………………..……….125 125 Aim 1: Screening and Confirmatory Testing……………………………. 127 Aim 2: Focus Groups and Survey……………………………………….. Future QUERI Stages of this Study……………………………………..130 Summary and Conclusion …………………………………………..…………..131 Appendices A: REDMOND SENTENCE RECALL MEASURE ………….……………………….137 138 B: 32-POINT COREQ CHECKLIST…………………………....……………………… C: REDMOND SENTENCE RECALL (RSR) ADMINISTRATION AND SCORING PROTOCOL ……………………………………………………………… 142 D: FOCUS GROUP SEMISTRUCTURED CONVERSATION FACILITATORS……. 144 E: REFLEXIVITY STATEMENT ………………………………………….………….146 REFERENCES …………………………………………………………………….…… 149 vii LIST OF TABLES Tables 1. Psychometric Properties of the RSR Measure ………………………………………. 44 2. Referral Source Correctness ………………………………………………………….. 53 3. School SES Status and LI Occurrence ……………………………………….……… 53 4. Descriptive Statistics for Screening and Confirmatory Measures …………………… 98 5. Individual Failed Screening Results ………………………………………………….. 99 6. Partial Diagnostic Accuracy Values ………………………………………………….. 99 7. Framework Analysis Chart …………………………………………………………..100 8. Survey Results: Descriptive Data……………………………………………………101 9. Prevalence Estimates for Specific Language Impairment ………………………… 134 10. Positive Predictive Rates …………………………………………………………..134 11. Theme/Code Groups………………………………………………………………..135 LIST OF FIGURES Figures 1. Analytic Framework and Key Questions Regarding Screenings ……………………42 2. Past Tense Probe Sensitivity and Specificity Data ………………………………….42 43 3. General and Specific Implementation Theoretical Approaches ……………………… 43 4. Four-stage Research Pipeline ………………………………………………………… 5. Project Structure …………………………………………………………………… 68 6. Project Flow and Timeline …………………………………………………………..93 7. Overlapping Focus Code Groups ……………………………………………………93 8. Mapping of Code Groups to Survey Statements …………………………………… 94 9. SLP Survey Responses ……………………………………………….……………..95 10. General Education Kindergarten Teacher Survey Responses……………..……… 96 97 11. Combined Group Survey Responses ………………………………………………… ACKNOWLEDGMENTS We live in a day and age of movie superheroes. Their fictitious presence pervasively surrounds us in the media, advertising, and pop-culture always reminding us that greatness is possible. I once read that real superheroes live in the hearts of small children fighting big battles. I often think of these “real” superheroes and how we as adults have it in our power to help lift children in ways that will give them a chance to live happy, healthy, and productive lives. I firmly believe all children have a basic human right to experience these things regardless of their circumstances. While many reasons ultimately compelled me to pursue a Ph.D., the desire to help children who struggle to compete, and eventually succeed in life because of language difficulties, was paramount. I started this journey over 4 years ago and have had fantastic opportunities to meet and work with professionals who have indelibly impacted my life. First and foremost, I wish to thank the incredible professionals of the Davis School District for their willingness to work with me on this project. I think our society takes for granted and undervalues public-school educators. Of the many things I observed while conducting this project, I saw educators who genuinely care about the children they teach – often in the face of challenging circumstances. It was inspiring and reminded me that educators are often unrecognized superheroes. The administrators, related servers, teachers, and aids of this school district exemplified professionalism and caring of the highest level. It was sincerely my pleasure to be able to partner with them on this study. Second, I wish to express my appreciation to the faculty members who took time out of their busy professional schedules to serve on my committee. I was fortunate to have a remarkable group of scholars who agreed to mentor me through this experience. Specifically, I wish to thank my primary advisor, Dr. Sean Redmond, for his patience, insight, and expertise. He continually challenged me to push my boundaries and to think broader and deeper. I credit him for giving me my nascent confidence as a scholarly writer. I want to extend a special thank you to Dr. Christina Gringeri for her mentoring through the qualitative portion of this study. Exploration with her of ontological and epistemological elements of research has been most enlightening and has taught me always to view research through the lens of the participant. I also want to thank Dr. Jacob Kean for allowing me to learn through trial and error independently. Dr. Stacy Manwaring and Dr. Robert Kraemer taught me how research and academia can successfully be approached practically and pragmatically. These attributes have allowed me as an individual to achieve what I have thus far in my academic career. Third, I want to express my appreciation to the other important faculty, students, and staff in the Communication Sciences and Disorders and Educational Psychology departments at the University of Utah who played a supportive role in helping through this journey. Thank you to Dr. John Kircher, Dr. Andrea Ash, Dr. Kathy Chapman, Dr. Julie Wambaugh, Dr. Nelson Roy, Wei Wei, Sharon Benavides, and Theresa Pfaff. Finally, I want to thank my wife and two daughters for their unwavering support throughout the entire course of my graduate school. My wife Anna has been steadfast in her support. Her patience and love have served as the foundation for me to push through xi the tough times. My daughters Jessica and Olivia have been patient and supported me every single day. I could not have done this without the three of them. xii CHAPTER 1 INTRODUCTION Problem Statement Specific language impairment (SLI) is one of the most common early childhood disorders (Leonard, 2014). Researchers estimate the prevalence of SLI in 5-year-olds to be between 7-8%. This makes SLI more common than Down syndrome, fluency disorders, traumatic brain injury, and autism combined. Short and long-term pejorative impacts on children with SLI include elevated risks for poorer outcomes in academic achievement, difficulty establishing and maintaining social and personal relationships, peer victimization, sexual abuse, and reduced vocational opportunities, among others (Alonzo, Yeomans-Maldonado, Murphy, & Bevens, 2016; Beitchman et al., 2001; Beitchman et al., 1986; Catts, Fey, Tomblin, & Zhang, 2002; Clegg, Hollis, Mawhood, & Rutter, 2005; Conti-Ramsden & Botting, 2008; Redmond, 2011; Sullivan & Knutson, 2000). SLI continues into adulthood (see Poll, Betz, & Miller, 2010), places high costs on societies, and is an enduring source of frustration and indignity to those affected. In spite of these negative outcomes, SLI has a very low profile in public health and educational forums, specifically among educators working in the public school system. This is of particular concern as public schools represent the front-line for the identification, evaluation, and treatment of all children with disabilities, regardless of severity (i.e., 2 Child Find mandate; Wright & Wright, 2007). Currently, there is no policy for school districts in the state of Utah requiring universal screening of language impairment. Research over the last 30 years has produced findings that contribute considerably to the ability of professionals to accurately identify children with SLI. Unfortunately, research suggests that on average, only 15% of children who suffer from language impairment are identified and receive treatment (Morgan et al., 2016; Zhang & Tomblin, 2000). To aggravate the issue, current trends show an average of 15-20 years for research findings to be fully integrated into evidence-based policies and programs suitable for public use (Brownson, Colditz, & Proctor, 2018). This is a staggering amount of time lost that could be better used serving the needs of students in Utah public schools. Purpose Statement The purpose of this study was two-fold. First, it looked to determine the usefulness of the Redmond Sentence Recall screening measure (RSR) as a universal screener for use in the Davis School District (DSD), located in Davis County, Utah. Second, it looked to identify and examine potential barriers and/or facilitators to the systematic uptake of universal screening in DSD. Broadly speaking, this project seeks to improve the identification rates of children with SLI who attend public schools. More specifically, qualitative and quantitative data generated from the aims stand to inform not only DSD, but to also guide public-school administrators in other school districts regarding the feasibility of district-wide universal screening implementation for SLI. 3 Research Aims AIM 1: To determine the potential usefulness of the RSR task as a universal screener for DSD. This was addressed by using a hybrid implementation approach to answer the following research questions: 1. Does the RSR screener protocol identify language impairments at a higher rate than that associated with the current system in DSD utilizing general education teacher referrals when using a standard score cutoff of < 80 on the CELF-4? Specifically, does it provide an occurrence rate that is more consistent with prevalence expectations based on epidemiological reports? 2. What is the positive predictive rate of the RSR screener protocol when used by personnel in DSD? AIM 2: To identify and examine potential barriers and/or facilitators to the systematic uptake of universal screening in DSD. This was addressed by using a hybrid implementation approach to answer the following research questions: 3. What barriers and facilitators are reported by DSD project team members based on their experience in the facilitation and administration of the RSR screener protocol as articulated in focus group settings? 4. Do DSD speech and language pathologists and general education kindergarten teachers agree/disagree with the barriers and facilitators identified by DSD focus group members as indicated by their responses to survey statements? CHAPTER 2 LITERATURE BASED RATIONALE Literature Review Overview The aims of this project were guided by research in the fields of communication disorders and implementation research. The assessment of the feasibility of universal screening of language impairment in public schools requires an understanding of key components of language impairment. The first section of the literature review describes our current understanding of language impairment. In its subsections, important topics pertaining to specific language impairment such as clinical markers, outcomes, and current awareness are explored. The second section looks at issues central to the discussion regarding the use of an annual-based format as an alternative to referral-based systems such as types of screening measures and limitations of diagnostic accuracy studies in school settings. An important part of this section describes this project’s proposed screening measure of the Redmond Sentence Recall screening measure (Redmond, 2007). The final section of the literature review explores the implementation structures by which systematic uptake of universal screening in public schools can be best achieved. Specifically, it investigates the foundational and theoretical underpinnings of implementation research and past use of implementation research in the public-school setting. This section concludes by identifying a conceptual framework and model this 5 study can utilize for appraisal of facilitators and barriers of universal screening, namely the Quality Enhancement Research Initiative (QUERI) framework working within a hybrid implementation model. The terms universal screening, annual screening, and annual-screening format will be used interchangeably throughout this paper to describe the formal screening of kindergarten-age students in public schools once a year. Specific Language Impairment Profile Children with specific language impairment (SLI) demonstrate a substantial delay in language acquisition that cannot be attributed to hearing impairment, social difficulties, intellectual deficiency, or neurological damage (Leonard, 2014). It has been classified by the National Institute on Deafness and Other Communication Disorders (NIDCD) as one of the most prevalent neurodevelopmental disorders in children that impacts approximately 7-8% of kindergarten-aged children (Bishop, 2010; Boyle et al., 2011; Leonard, 2014; Norbury et al., 2016; Tomblin et al., 1997). Despite its prevalence, only a small portion of children with SLI receive intervention services by speech and language pathologists (SLP) in the public-school system (Morgan et al., 2016; Zhang & Tomblin, 2000). Key to understanding SLI is the ability to recognize core phenotypic characteristics. These are outlined in the following section. Primary Clinical Markers of SLI Language impairment in children is a highly heterogeneous condition. Much progress, however, has been made by researchers over the last 30 years in the identification of preschool and school-aged language markers of SLI in children. Of the 6 many investigated, two areas have been acknowledged as promising pathognomonic markers, namely morphosyntax and verbal memory. Morphosyntax, the use of morphological elements within the context of syntactic structure, is widely considered to be the leading primary clinical marker of SLI (Conti-Ramsden, Botting, & Faragher, 2001; Eadie, Fey, Douglas, & Parsons, 2002; Eisenberg & Guo, 2013; Rice & Wexler, 1996; Rice, Wexler, & Cleave, 1995; Rice, Wexler, & Hershberger, 1998; Schuele & Dykes, 2005). Within morphosyntax, tense and agreement has emerged as a particular area of deficit in English speaking children with SLI. For example, in the sentence: “He is walking the dog,” tense and agreement are marked on the finite verb “is.” Researchers agree that children diagnosed with SLI show clear and measurable impairment in the inclusion of grammatical morphemes that mark tense in obligatory contexts (Grela & Leonard, 2000; Hoover, Storkel, & Rice, 2012; Redmond, 2005; Rice & Blossom, 2013). Particular forms that have been shown to be difficult in children with SLI include past tense marking, both with irregular and regular past tense -ed (e.g., He thought and He kick-ed), and 3rd – person regular and irregular present tense (e.g., He think-s and He does), and auxiliary verb use (e.g., is, am, are, was, were; Ash & Redmond, 2014; Kamhi, 2014). There is much debate among researchers, however, regarding the mechanisms underlying children’s weaknesses in this area. Memory ability in the SLI population, specifically short-term and working memory, is at the heart of this debate. Two leading views have emerged as the most widely accepted. The first view attempts to frame SLI as a function of a deficit in core linguistic abilities which limits performance exclusively on verbal memory tasks (e.g., MacDonald & Christiansen, 2002). The second view differs 7 by arguing the deficit lies in core memory ability, thus limiting a child’s capacity to acquire language (e.g., Gathercole & Baddeley, 1990). It is not the intent of this study to argue the merits of these two positions, but only to acknowledge both highlight an inadequacy in verbal memory skills in children with SLI as a function of difficulties with short-term and working memory. Short-term memory refers to the ability of an individual to briefly retain information without the need for additional cognitive processing (Brown, 1958). Working memory, often considered a subset of short-term memory, refers to the ability of an individual to both briefly retain information and process/manipulate information (Baddeley & Hitch, 1974). The topic of memory deficit is of particular interest as children with SLI demonstrate normal ranges of intellect as measured by nonverbal intelligence tasks. While the relationship between memory and cognitive processing has much overlap, they are, in fact, distinct processes. Short-term memory can be better understood by comparing it to a visual-spatial sketchpad of sorts and a phonological loop, much like a recorded playback. These two systems vary in the amount of time they are able to maintain chunks of information. Working memory adds the additional element of manipulating those chunks, typically measured on cognitive tasks by reordering or grouping the chunks. These key memory skills are highlighted in measures such as sentence repetition and nonword repetition tasks (Conti-Ramsden, Botting, & Faragher, 2001; Dollaghan & Campbell, 1998; Redmond, 2005). These measures have proven valuable in the identification of children with SLI and will be reviewed in later sections of this report. Now that an understanding of how SLI may present in a child’s language pattern, it is crucial to understand how unidentified and untreated language impairment can 8 negatively influence the developmental trajectory of a child. This is addressed in the following section. Short- and Long-term Personal Impacts of SLI The realization of the broad range of potential negative outcomes that have been associated with SLI may be the most influential component in agitating towards universal screening in public schools. Research has uncovered interesting results regarding keyimpact outcomes of this population. For example, a meta-analysis by Yew and O’Kearney (2012) reviewed 19 studies that followed eight cohorts of children with SLI and typically developing (TD) controls from initial assessments performed at ages ranging from 3 to 8.8 years and follow-up duration ranging from 2 to 12 years. The specific dependent measures looked at the key psychiatric domains of emotional and behavioral outcomes later in childhood and adolescence. Their findings indicated SLI children were two times more likely to show disordered levels of internalizing problems, externalizing difficulties, and attention/hyperactivity problems than TD children. Higher rates of anxiety and depression in adolescents with SLI have also been linked to reduced language skills (Conti-Ramsden & Botting, 2008). Studies looking at peer-victimization in public schools show that children with SLI may be up to three times more likely to be bullied as compared to their TD peers (Conti-Ramsden & Botting, 2004). Redmond (2011) highlighted the role peer victimization plays in the social adjustment of students with language impairment. Results from his study showed that SLI status was linked with higher levels of peer victimization. A buffering effect related to the number of close friendships was found for 9 participants with ADHD and TD participants, but not for participants with SLI. Lyons and Roulstone (2018) conducted semistructured interviews with children meeting criteria for language impairment to inquire about their personal experiences regarding overall well-being. Results from their study showed potential risk factors to well-being included negative perceptions with their communication impairment as well as difficulties with peer relationships. They also found potential protective strategies such as hope, agency, and positive relationships played an important role in the mitigation of negative experiences. In a population-based epidemiological study done by Sullivan and Knutson (2000) looking at maltreatment in children with disabilities, children with speech and language impairments were at almost five times the risk for neglect and physical abuse, almost three times the risk for sexual abuse, and almost seven times the risk for emotional maltreatment. Studies looking at academic outcomes in children with SLI are particularly relevant for this study. Research shows a wide range of academic skills that are negatively impacted as a result of language impairment. For example, it is well documented that reading abilities in young children are linked to academic outcomes (e.g., Daneman, 1991; Naglieri, 2001; Stanovich, 2009; Wells, 1986). In an epidemiological study Catts, Fey, Tomblin, and Zang (2002) found that children with language impairment in kindergarten were nearly six times more likely to be classified as having a reading disability in second and fourth grades. Conversely, children with language impairment in kindergarten who had improved in spoken language abilities by second and fourth grades had better reading outcomes than those with persistent language 10 impairments. Listening comprehension abilities, a vital academic skill, have been shown to be negatively impacted by impaired language capabilities (Alonzo, YeomansMaldonado, Murphy, & Bevens, 2016). In a longitudinal study, Young et al. (2002) found that language-impaired young adults who were originally diagnosed at age 5 lagged considerably behind controls in all areas of academic achievement including reading, spelling, and math, even after controlling for intelligence. Research has begun to paint a compelling picture of the potential negative effects of unresolved language impairment that continue into adolescence and adulthood. A broad and nascent picture may exist indicating pervasive problems in areas such as difficulty with complex elements of syntax and figurative language (both in receptive and expressive domains), difficulty with social and personal relationships, and poor vocational outcomes (Clegg, Hollis, Mawhood, & Rutter, 2005; Conti-Ramsden & Botting, 2004; Howlin, Mawhood, & Rutter, 2000; Johnson et al., 1999; Stothard, Snowling, Bishop, Chipchase, & Kaplan, 1998; Tomblin, Freese, & Records, 1992; Whitehouse, Watt, Line, & Bishop, 2009). Studies looking at untreated speech and language delay in preschool children have shown variable persistence rates ranging anywhere from 40% to 60% (Law, Boyle, Harris, Harkness, & Nye, 1998). Now that a foundation has been established regarding key features of SLI and how it can pejoratively impact the course of an individual’s life, it is important to look at the current understanding of SLI by public educators as well as the general public. This is addressed in the following section. 11 Awareness of SLI An appropriate consideration of the experience of having SLI would include reference to the disorder as being highly unrecognizable. Currently, recognition and awareness of SLI are quite limited in both public health and educational forums. A leading reason for this points to difficulties in the areas of morphosyntax and memory as not overt as is the case with other conditions such as autism spectrum disorder (ASD), articulation disorders, stuttering, attention deficit hyperactive disorder (ADHD), or externalizing disorders. For example, public awareness of key ASD markers such as poor eye contact, repetitive behaviors, social disconnect, and difficulty with figurative language are well known to the general public (Prelock, 2006). Tipton and Blacher (2014) reported that 76% of survey respondents had more correct answers than neutral and incorrect ones regarding their views of the diagnostic characteristics and causes associated with ASD. This may be due in large part to the observability of these characteristics, thus making identification easier. As it relates to language impairment, teachers often mistake nonrelated issues such as speech or reading problems as valid and reliable indicators of language delay. Weiler, Schuele, Feldman, and Krimm (2018) looked at these specific areas as possible markers of language difficulties in kindergarteners. Results from their study found that children with language difficulties cannot be identified on the basis of articulation or emergent literacy screening. Consequently, as it relates to SLI, front-line advocates such as parents and general education teachers are put at a gross disadvantage. Accordingly, these children forfeit the opportunities of assessment, identification, and treatment available to all children as part of a free and appropriate public education. It is estimated that less than 25% of children 12 with SLI in public schools are identified and served (Morgan et al., 2016; Zhang & Tomblin, 2000). It may be argued that until policymakers, experts, and opinion leaders in public health, medicine, and education realize the increased health and societal costs starting in early childhood, and extending into adulthood, SLI will continue to have low levels of awareness. For example, Cronin, Mccabe, Viney, and Goodall (2017) investigated the relationship between language impairment in children ages 4-11, and health care costs over an 8-year period. Results from their study showed that language difficulties are associated with increased healthcare costs at key developmental milestones, notably early childhood, and as children approach adolescence. While long-term solutions to better identify children with SLI and appreciate the short and long-term impacts of the disorder must include increased awareness in public health and educational forums, universal screening measures stand to play an important role in the overall discussion. As a result, universal screening in the public-school setting presents an intriguing alternative to better identify children with language impairment. With a profile of SLI now presented, the next step lies in exploring the salient topic regarding best methods of identification. The following sections explore screening as it relates to language impairment identification as well as associated approaches. Identification of SLI Much progress has been made with regard to the development of measures sensitive to identifying SLI. The metrics by which a measure is deemed accurate are important to understand. Particularly, as it relates to SLI, a successful screening measure 13 should have the ability to identify difficulties in morphosyntax and/or short term and working memory, as these phenotypes have been identified as leading clinical markers. Role of Screening in Language Impairment Identification Language development is a critical indicator of future aptitude in overall development, cognitive ability, and academic success. Rates in referral accuracy from sources such as parents and general education teachers can be subpar and prone to a variety of biases (Morgan et al., 2016). Therefore, a logical alternative to reliance on a referral-based system where the primary front-line detectors are ill-equipped to recognize the characteristics of SLI, is that of universal screening. Twenty years ago, Law et al. (1998) began to investigate the need for universal screening in children with speech and language delay. They completed a systematic review that looked to establish whether sufficient evidence existed to warrant the introduction of universal screening for speech and language delays in children up to 7 years of age. Among others, a primary conclusion drawn from their study was that indeed sufficient evidence did exist to support universal screening and therefore it be appropriate that attempts should be made to identify children at risk of persistent speech and language difficulties as early as possible. More recently, a systematic review by Nelson, Nygren, Walker, and Panoscha (2006) performed on behalf of the US Preventative Service Task Force (USPSTF) sought to appraise the strengths and limits of evidence regarding the usefulness of screening for speech and language delay in the primary care setting and to determine the balance of benefits and adverse effects of routine screening. A portion of this review targeted evaluation measures that took less than 10 minutes to administer. They discovered a wide 14 variance of sensitivity and specificity in comparison to confirmatory measures. Most measures were not specifically designed for screening purposes and assessed different linguistic domains. No “gold standard” measure was identified for universal screening, and lack of consistency regarding confirmatory measures was found across studies. The USPSTF provides a useful analytic framework by which key questions regarding screening rational and methodology should follow (Figure 1). Types of Screening Measures Researchers have narrowed in on three approaches that have shown to be diagnostically accurate, namely nonword repetition tasks, tense-marking probes, and sentence recall tasks. These approaches have been shown to be useful in the identification of SLI, particularly because they act as a “work-around” to the SLI-deficit-source-debate by targeting both the language and memory systems. Rather than being limited to measures that target either language or memory systems, when the debate is far from settled, these tasks assess both areas at the same time. A meta-analysis by Pawlowska (2014) examined the extent to which these three markers of language impairment in English distinguished affected and unaffected participants. All three were found to aid in the process of initial identification. The conclusions also emphasized the need for additional studies looking at these three measures effectiveness both in small- and largescale studies. Nonword repetition tasks (NWR) involve the immediate repetition of verbally presented nonsense words (i.e., tayvok or voop) to a participant. The aim of this type of task targets the underlying mechanism of phonological short-term and working memory. 15 NWR tasks are typically designed such that neither the nonsense words themselves nor their primary syllables resemble known lexical words. In addition, the predictability of distinct phonemes within the nonsense words should be minimized. NWR tasks initially showed great promise when researchers began investigating them in the 1990s (e.g., Bishop, North, & Donlan, 1996; Gathercole & Baddeley, 1990; Montgomery, 1995). For example, Dollaghan and Campbell (1998) found no overlap in nonword repetition abilities between clinical and control groups. In fact, their findings were most impressive showing that a failed NWR task was associated with a positive likelihood ratio of 25 in addition to a 98% discrimination rate between typically developing and language impaired participant groups. These robust findings, however, have not been replicated in other studies conducted later and therefore warrant some caution when NWR tasks are considered as a preferred method of screening for SLI. Weismer et al. (2000) reported a much lower likelihood ratio of 6.5 for discriminating between groups on nonword repetition abilities. Gray (2003) reported sensitivity and specificity values for NWR slightly above 80%. Other studies have endorsed higher sensitivity, but paltry specificity values. For example, Conti-Ramsden and Hesketh (2003) and Conti-Ramsden (2003) reported robust sensitivity for NWR in kindergarten age children between 85%-100% but moderate specificity at 66%-78%. Taken as a whole, research potentially supports NRW as a useful primary marker of SLI but may require additional research to support its position as a measure that can reliably identify children with SLI. Tense-marking probes target children’s morphosyntactic abilities. Grammatical tense marking in English includes morphemes such as regular past tense (e.g., John walked), irregular past tense (e.g., John thought), 3rd-person regular singular (e.g., John walk- 16 s), and auxiliary verbs (is, am are, was, were). A robust body of research now supports a clear pattern of difficulty in English-speaking children with SLI in the use of these morphemes, specifically with regard to regular and irregular past tense forms (e.g., Leonard, 2014; Marchman, Wulfeck, & Weismer, 1999; Oetting & Horohov, 1997; Redmond, 2005; Rice, Tomblin, Hoffman, Richman, & Marquis, 2004; Rice & Wexler, 1996; Rice, Wexler, & Cleave, 1995; Rice, Wexler, & Hershberger, 1998). Past tense probes have been shown to have respectable diagnostic accuracy levels. In fact, of the three proposed measures, past tense probes may be the most investigated with regard to sensitivity and specificity values. Figure 2 reports on studies looking at these values across age ranges of 2-10 years of age as reported by Ash and Redmond (2014). The use of past tense probes as a screening measure for SLI may be of value, but comparatively speaking, the diagnostic accuracy is lower compared to other suggested options (i.e., sentence recall) specifically as it relates to sensitivity values. Sentence recall tasks involve the immediate repetition of verbally spoken sentences to a participant. This type of a task aims to target the underlying cognitive mechanism of short-term verbal memory as well as morphosyntactic abilities by requiring the child first to parse and interpret the stimuli sentence, process the sentence auditorily, and regenerate the sentence using their grammar abilities. Reduced sentence recall performance has been well documented in studies looking at SLI compared to typically developing groups (Archibald & Joanissee, 2009; Briscoe, Bishop, & Norbury, 2001; Conti-Ramsden, Botting, & Faragher, 2001; Eadie, Fey, Douglas, & Parsons, 2002; Laws & Bishop, 2003; Norbury, Bishop, & Briscoe, 2001; Redmond, 2005). In fact, sentence recall tasks have been routinely included as core subtests in leading standardized 17 measures of language abilities such as the Test of Language Development – fourth edition (TOLD-4) (Newcomer & Hammill, 2008) and the Clinical Evaluation of Language Fundamentals – fourth edition (CELF-4) (Semel, Wiig, & Secord, 2004). In a study looking at the accuracy of nonword repetition compared to sentence recall tasks to identify SLI in 400 school-aged children, Archibald and Joanisee (2009) found sentence recall to be more useful than nonword repetition. Results from their study reported sensitivity/specificity rates at 84.6 and 90.3, respectively (cutoff score at below 10th percentile) and a positive likelihood ratio of 8.7. Conti-Ramsden, Botting, and Faragher (2001) showed similarly high levels of sensitivity/specificity in sentence recall tasks over nonword repetition and past tense probes (90% and 85%, respectively) in 160 schoolaged children with SLI compared to typically developing controls. The particular strength of sentence recall tasks lies in their ability to target multiple aspects of language impairment (i.e., morphosyntax, verbal memory, and general memory). Sentence recall tasks appear to be more diagnostically accurate relative to other measures reviewed, such as nonword repetition and past tense probes. In addition to these data, administrative considerations must be weighed when looking at these measures for potential use as a universal screener. For example, NWR requires the administrator to be able to recognize unfamiliar phonemic combinations or productions used in nonsense words. Recording of responses also requires background and experience transcribing participant responses using the international phonetic alphabet (IPA). Past tense probes require a trained and experienced ear in being able to accurately identify and classify subtle omissions or substitutions that frequently accompany morphemic productions specifically with inflectional endings such as past tense -ed or 3rd-person singular regular 18 -s. Because universal screening measures require fast administration time, done with minimal training, and potentially conducted by nonspecialized personnel such as parent aids or classroom assistants, NWR and past tense probes would not be ideal selections. Sentence recall measures, on the other hand, are able to meet these criteria of universal screening administration. For example, sentence recall tasks require simple omission or addition notations by the administrator. Moreover, because stimuli sentences are basic in nature, administrators are able to understand responses by participants easily and make the necessary, simple notations. Taken as a whole, based on diagnostic accuracy reports, sentence recall tasks are a recommendable option for a universal screening measure for SLI identification. There are different indices of validity that reveal related, but different, characteristics of a screener. Some of these indices may not be readily available for routine monitoring in public schools. The following section examines these indices in detail. Use of Diagnostic Evidence Data in Public Schools Critical appraisal of an identification measure allows professionals to determine whether sufficient external evidence exists in order to use a particular diagnostic tool as a screening measure. A 2x2 table format is frequently used among researchers for examining evidence regarding the accuracy of a diagnostic measure. Rates of true/false positives and true/false negatives are used to calculate two sets of values used in diagnostic accuracy assessment, namely sensitivity/specificity and positive/negative ratios. Sensitivity refers to the proportion of participants who fail a particular screening measure and are shown to have the disorder as identified by confirmatory testing 19 (Dollaghan, 2007). This value is calculated using true positive and false negative values. Specificity, on the other hand, refers to the proportion of participants who pass a particular screening measure and are shown not to have the disorder (Dollaghan, 2007). This value is calculated using false positive and true negative values. The primary weakness of sensitivity and specificity values is in their susceptibility to variations in the base rate of the sample in which they are studied (Dollaghan, 2007). A point of importance is the realization that the diagnostic accuracy of any measure reflects the performance of that measure under certain conditions (Bossuyt, 2003). The vulnerability of sensitivity and specificity to base rate variations restricts their effectiveness as practical metrics, especially when derived from participant pools in which the base rate is low as is the case comparatively speaking with SLI. For example, if half of a participant pool has a particular disorder and half do not (i.e., a base rate of 50%), the probability of any one participant having the disorder prior to the administration of a screening measure is 50%. However, as is with the case with SLI, when there are fewer children with the disorder as compared to those with it, a threat is presented towards elevated rates of false positives. There is also a higher probability that a child will test negative (i.e., normal/pass) on a screening measure (Dollaghan, 2007). In this case, specificity will routinely be higher. Conversely, sensitivity will routinely be lower because there are fewer disordered children to detect. As a result, sensitivity and specificity values can be misleading with regard to a particular screening measure’s diagnostic accuracy. The approach often used by researchers to overcome the weakness in sensitivity/specificity values is calculation of positive/negative predictive ratios. Positive and negative likelihood ratios are highly beneficial because they are substantially less susceptible to 20 variations in base rates particularly because they are calculated from both sensitivity and specificity values concurrently (Dunn, Flax, Sliwinski, & Aram, 1996; Sackett, Haynes, Guyatt, & Tugwell, 1991). When base rates are low (or higher) than 50%, the use of likelihood ratios is suggested for reporting alongside sensitivity/specificity values (Plante & Vance, 1995). Use of these two metrics, however, can be problematic in public schools. A primary challenge with sensitivity/specificity and positive/negative likelihood ratios is with the issue of verification bias. This occurs if the diagnostic standing of some, but not all students, in a sample, is verified (Dollaghan, 2007). This is of particular concern when dealing with confirmatory testing via a reference standard specifically as it relates to participants who pass a screening measure. Resource and logistical constraints in public schools make it impossible to administer a reference standard to all students. Therefore, values that include, or are calculated from, both sensitivity and specificity data, will always be subject to verification bias because only a small portion of the total population will be administered the reference standard (i.e., true/false positives). A third set of values can be calculated using just true/false positives rates and stand to be the most practical when routinely assessing the diagnostic accuracy of measures in public schools, namely positive/negative predictive values. Positive predictive values (PPV) stand to be most useful for public schools. PPV is defined as the proportion of participants with a positive screening result (i.e., fail the screener) who have the target disorder (Sackett et al., 1991). Negative predictive values (NPV) are described as the proportion of participants with a negative screening result (i.e., pass the screener) who do not have the target disorder (Sackett et al., 1991). NPVs are computed 21 by use of true/false negatives, thus relying on data that contribute to the verification bias problem for public schools. As is the case with likelihood ratios, performance of positive and negative values is tied to the base rate or prevalence of a disorder, therefore making the setting in which they are calculated an important factor. For example, PPVs that are calculated using participants from a clinical setting would be substantially less informative as compared to a measure’s predictive abilities taken from a communitybased sample such as a public-school setting. In other words, the predictive value of a specific measure cannot be constant. Rather, it must vary with the proportion of participants who truly have a target disorder among those who undergo the screening evaluation (Sackett et al., 1991). Now that an understanding has been established as to what type of measure best identifies children with language impairment in public schools, and what rates are most informative and achievable for measurement values, the task at hand becomes to identify a specific sentence recall measure for recommendation as a universal screener. Redmond Sentence Recall Measure (RSR) Redmond (2007) developed an experimental sentence recall measure that has been shown to have respectable diagnostic accuracy of children with SLI between the ages of 5;0 – 6;11. The measure is comprised of 16 stimuli sentences that are each composed of 10 words (10-14 syllables) and contain an equal number of active and passive sentences (Appendix A). Table 1 details the original published psychometric properties of the RSR measure (Redmond, 2007) compared with replication data from Archibald and Joanissee (2009). As is evident, values are robust from both studies and 22 show the task can reliably differentiate SLI and typically developing groups. In addition to strong psychometric properties, three specific features of the RSR measure make it a compelling choice for a language screening tool. First, the sentences are challenging enough to ensure that typically developing controls do not reach ceiling. Second, the sentences draw on the clinical marker of past tense. Third, the sentences are scored on a graded scoring system (0-2) as opposed to a binary system (i.e., correct/incorrect). This contributes as well to the task’s ability to avoid ceiling effects among the typically developing group. Logistically speaking, administration of the RSR measure can be completed by nonspecialized serviced personnel such as general education teachers, classroom aides, and/or classroom volunteers. Interrater consistency for the RSR measure was high at .988 (r =.990; significant at p < .001) Administration and scoring training for the tasks is minimal. Administration time for the measure ranges from 7-10 minutes (Redmond, Thompson, & Goldstein, 2011). Taken as a whole, the RSR measure stands to be a valuable tool for use in public schools for initial screening of language impairment. Implementation Research Current trends show an average of 15-20 years for research findings to be fully implemented in clinic and health care settings (Brownson, Colditz, & Proctor, 2018). This applies to all aspects of health care and represents a staggering amount of time lost that could be better used serving the needs of patients and clients both in adult and child populations. In addition, certain areas of health care and behavioral and social sciences change so rapidly that by the time findings reach the clinical setting, they are outdated, 23 irrelevant, and ultimately not useful (Brownson, Colditz, & Proctor, 2018; Curran, Bauer, Mittman, Pyne, & Stetler, 2012; Stetler, Damschroder, Helfrich, & Hagedom, 2011). Implementation research is a comparatively new academic field that is growing considerably. It looks to shorten this gap by addressing challenges associated with the systematic uptake of research findings into routine practice to improve the quality and effectiveness of patient and client care (Eccles & Mittman, 2006). Implementation research is defined as the active process of integrating evidence-based findings within a specific setting with the end goal of systematic uptake and adoption (Rabin & Brownson, 2018). Before investigating specific approaches that stand to be useful in school-based implementation projects, it is important to have a working understanding of the structural foundation and theory of implementation research. The following section addresses these topics. Structure of Implementation Research – a Review of Nilsen (2015) Although early efforts in implementation research have always been empirically driven, the incorporation of theoretical foundations was limited. Critical reviews looking at this early period have compared the process to an expensive version of trial and error (Eccles & Mittman, 2006). As a result, implementation endeavors often produced limited outcomes highlighting the need for increased use of theoretical models in project development. Particularly, reviews have emphasized the limited use of theoretical underpinnings, making it difficult to explain and understand why and how implementation actions succeed or fail. Nilsen (2015) attempted to provide a narrative review of the terms theories, models, and frameworks and apply them to past 24 implementation research. Specifically, the study proposed a hierarchal system that distinguishes between different levels of approaches. As a result, it endeavors to bring greater clarity to the formation of better understood and more commonly used terminology in which implementation researchers can utilize. Before being able to understand the use of the above mentioned terms in an implementation-specific context, it is important to appreciate that these terms are frequently used to create the basis of most research fields. Generally speaking, theories can be defined as a set of parameters or analytical principles that attempt to explain a set of phenomena. Models work within a particular theory and attempt to simplify a phenomenon. Frameworks can represent a plan that attempts to identify variables that can account for a phenomenon. In an implementation context, these terms take on a more technical meaning. It is noted that as the field of implementation research has evolved over the years, the three terms have frequently been used interchangeably in the literature, but there are, in fact, distinct differences between them. As it relates to implementation research and science, theories carry predictive ability and attempt to explain causal mechanisms. For example, a theory would attempt to explain the extent to which the attitudes and beliefs of publicschool personnel concerning a newly implemented assessment measure can predict adherence to proposed guidelines. Models can be defined in implementation research as the steps needed to guide the actual process of implementation. In a way, models can be looked at as the “nuts and bolts” of implementation research as they outline to stakeholders how a proposed plan will be operationalized. For example, a model would lay out specific steps required to implement an assessment measure in a particular setting. 25 Frameworks in implementation research may attempt to explain or evaluate factors believed to influence implementation outcomes. It asks the salient question of what barriers may stand in the way of the implementation of an assessment measure in terms of personnel, policies, and logistics. Theoretical approaches Nilsen’s review of the implementation literature identified three general theoretical approaches which can be broken down into five categories of specific theoretical approaches (see Figure 3). The first general approach focuses on guiding or describing the actual process of translating research into practice. This is accomplished by the use of process models. The purpose of process models is to specify the steps, phases, or stages required to translate research into practice. Process models are facilitative in nature by offering practical guidance in the planning and execution of implementation. They should offer a proposed number of stages that should be followed in the process of moving research findings to the clinical setting. Stakeholders may often be most interested in process models as they are the means that guide or cause change. Process models emphasize the importance of deliberate and meticulous planning, specifically in the preliminary stages of implementation efforts. In other words, process models provide stakeholders with operationalized measures that should proceed in a step-wise, orderly, and linear fashion. The second general approach emphasizes the understanding of and/or explaining the influences that impact implementation outcomes. For the purpose of this proposal, this general aim is to be addressed by the use of determinate frameworks. The aim of 26 determinate frameworks is to identify the domains specific to determinants that act as enablers or barriers in implementation outcomes. In other words, they seek to identify the independent variable(s) of a particular endeavor and explain how they might negatively or positively impact project outcomes (i.e., dependent variable(s)). Determinate frameworks can also be abstract in nature by attempting to hypothesize the interaction between certain determinants. Unlike process models, determinate frameworks do not address how change takes place, but rather, they have the ability to assess determinants at different levels of an organization and beyond. They help stakeholders realize that implementation is a multidimensional phenomenon with many interacting influences. The third general approach is centered on the evaluation of implementation outcomes. This is accomplished by the use of evaluation frameworks. Specifically, evaluation frameworks look to assess a range of implementation components that directly determine the ultimate success or failure of an endeavor. It is important to point out that although evaluation frameworks are considered a stand-alone theoretical category and approach, the preceding two areas can also be scrutinized for success or failure thus highlighting the multilayered nature of implementation research. This is because each theoretical approach ideally must be operationalized and measured against its own unique concepts and constructs. Implementation Research in Public Schools Public school settings hold unique potential for the promotion of health. Particularly, this claim is evident in two ways. The following subsections highlight the important roles regarding the amount of time a child spends in the educational system and 27 the unique reach that system has. Time spent in school Children spend the majority of their formative years in schools, thus making this setting the ideal place for teaching, awareness, and healthy behavior learning. It is also an ideal place for the promotion of basic preventative services such as health screenings. By their nature, schools have the potential for tremendous reach and impact that cannot be found in other settings such as hospitals or other health care sites. For example, as Brownson, Colditz, and Proctor (2018) aptly pointed out, a typically developing child may spend an average of only 30 minutes with a clinical professional at an annual wellness check-up. This is vastly disproportionate to the time a child spends in school of 7 hours a day over 180 days during the year (~75,600 minutes). As a result, the potential reach of health program services such as awareness, assessment, and intervention stand to positively impact children with accumulating effects across the formative years of development. Research on public health implementation in public schools is still in the initial period of development (Brownson, Colditz, & Proctor, 2018) although several key studies have laid important groundwork for future projects that can be mirrored for studies based in communication disorders such as universal screening for language impairment. For example, Locke et al. (2016) studied the individual and organizational factors that impact the implementation of interventions for children with autism spectrum disorders (ASD) in public schools. They posited that although many ASD interventions have indicated efficacy in health-based settings such as university-based clinics, few have 28 been effectively implemented and sustained in schools. The primary dependent variable looked at was ASD intervention program fidelity (i.e., dose, adherence, and competence). Independent variables examined were ASD providers’ attitudes toward evidence-based practices, organizational culture and climate, implementation climate, and organizational leadership. Other studies have looked at similar variables that impact the systemic uptake of research-based evidence in public school. Massey, Armstrong, Boroughs, Henson, and McCash (2005) looked at challenges to implementation, operation, and sustainability of mental health services in public schools. They suggested that difficulties associated with operating programs in schools often prevent evidence-based practices from being fully implemented and sustained as intended. They were able to identify major challenges encountered by school personnel such as struggles in determining the organizational placement of the target program, identifying lines of authority and accountability, attaining organizational support for operations, forming program-relevant procedures, and recognizing and clarifying the scope and expectations of individual programs. Interestingly, no difficulties were found with regard to requirements for program documentation and accountability or with regard to the flexibility of programs in responding to the needs of schools. Poduska et al. (2009) looked at a classroom-based intervention directed at preventing early risk factors for drug abuse. They pointed out that although a number of preventative interventions carried out within public schools have demonstrated both short-term and long-term effectiveness in epidemiologically based randomized field trials, programs are not often maintained with fidelity over time. Particularly, they looked to examine the importance of the complex multilevel structure of a school district and its 29 impact on long-term maintenance of an intervention. Forman and Barakat (2011) looked at contributing factors to the low use of evidence-based cognitive-behavioral interventions that are appropriate for use with children and adolescents in the public-school setting. As is the emerging theme in this section with the studies looked at thus far, this study highlighted the fact that despite the potential for cognitive-behavioral therapies to improve or reduce the number of child and adolescent mental health problems, their use in schools remained low. Specifically, they looked at factors they identified as having the most influence on implementation success in public schools, namely, school organizational structure, program characteristics, training and technical assistance, administrator support, and fit with school goals, policies, and programs. Studies such as these are important as it relates to implementation projects targeting communication disorders in the public school for two reasons. First, it emphasizes that although important implementation work in schools has been done in fields such as ASD, mental health, drug addiction, and behavioral treatment, studies specifically detailing the relationship relating to the implementation of language impairment screening, assessment, and/or treatment are lacking. Second, these studies highlight the importance of general theoretical approaches that target factors that influence implementation outcomes. Reach of public schools Implementation research in public schools provides an opportunity to influence whole populations that otherwise would be difficult to reach regarding awareness, 30 assessment, and intervention. It has been well documented in the communication disorders literature that maternal and family variables, in particular socio-economic status (SES), have a strong link to language learning (e.g., Dollaghan et al., 1999; Hart & Risley, 1995; Hoff-Ginsberg, 1994; Huttenlocher, Haight, Bryk, Seltzer, & Lyons, 1991; Wells, 1985; Zubrick, Taylor, Rice, & Slegers, 2007). In addition, some studies have shown SES to be a predictor of language development (Entwisle & Astone, 1994; Tomblin, 1996). For example, Hoff (2003) found children from high-SES families had higher productive vocabularies than those from mid- and low-SES families although recent findings from Sperry, Sperry, and Miller (2018) have now challenged this longheld position. Rural populations have been shown to have a number of barriers that affect their ability to access speech and language services both in assessment and treatment (Caldwell, Ford, Wallace, Wang, & Takahashi, 2017; Law, Reilly, & Snow, 2013; Smith, Humphreys, & Wilson, 2008; Verdon, Smith-Tamaray, & Mcallister, 2011). For example, O’Callaghan, McAllister, and Wilson (2005) reported that parents of children with communication disorders living in rural areas experienced limited choices of clinicians, high cost of travel, long distances to travel in order to access services, limited access to public transportation, poor understanding of available speech and language services, and delays in assessment and treatment as a result of waiting lists. Research has documented the challenges minority populations face regarding access to health and primary care (Caldwell et al., 2017; Canedo, Miller, Schlundt, Fadden, & Sanderson, 2017; Flores, Abreu, Olivar, & Kastner, 1998; Szczepura, 2005; Wang & Luo, 2005). For example, Newacheck, Hughes, and Stoddard (1996) reported 31 that minority children fared consistently worse than those from nonmajority groups in areas such as type of site availability, travel time, wait time, access to after-hours care, and availability of regular clinical staff. Particularly, they reported these children were twice as likely to have limited access to usual sources of care, nearly twice as likely to wait for services an hour or more at clinics, and utilized only half as many clinical services after adjusting for health status. Thus, by weaving health promotion and delivery of health services into the public-school agenda, underserved populations such as low SES, rural, and minorities stand to be benefited. Now that an understanding regarding the foundational and theoretical underpinnings of implementation research has been established, identification of a specific approach by which universal screening of language impairment can best be achieved in the public-school setting becomes the next important step. School-Based Guiding Conceptual Sources There are wide-ranging sources of process models, determinant frameworks, and evaluation frameworks that have been developed by researchers in the field of implementation science. These sources span a broad range of fields of origin including nursing, health care/services, public health, mental health, and strategic management (Tabak, Khoong, Chambers, & Brownson, 2018). Unfortunately, this range currently does not include public education. In fact, Tabak et al. (2012) reported on the most commonly cited models and model developers in the literature as well as respective fields of origin. According to their review, not one approach has origins in, and developed specifically for, public education. Brownson et al. (2018) lend support to this finding by 32 asserting research with regard to public health implementation in public schools is in an initial stage of development, and as a result, the majority of the literature currently lacks categorical theoretical models and frameworks. The primary source of general theoretical input for school-based implementation studies has been rooted in classical theories of organizational and individual change. For example, Domitrovich et al. (2010) reported that studies done in schools are often grounded in prevention science where factors that increase and guard against risk are foundational (Flay et al., 2005). Prevention science is also considerably informed by developmental models and ecological theory, which underscore the dynamic influences of the environment on behavior (Brofenbrenner & Morris, 1998; Kelly, Ryan, Altman, & Stelzner, 2000). These environmental influences can be systematized into specific contexts, ranging from closer influences such as the family to more distal aspects, such as the community or school. The use of systems currently in place in public schools has been emphasized as a recommended model for implementation (i.e., integrated models) (Domitrovich et al., 2010). As a result of the primary use of classical theoretical approaches to implementation in public schools, researchers looking to conduct implementation projects in this setting must rely on, and integrate, specific models and frameworks that have been developed for use in other fields. The following section looks at a suitable option for this project. Quality Enhancement Research Initiative (QUERI) Stetler, Mittan, and Francis (2008) developed a six-step structure that outlines methods that facilitate the process of implementation. Originally designed for implementation projects in Veterans Affairs hospital settings, QUERI is a process model, 33 determinant framework, and evaluation framework that focuses on the identification of stages that implementation should move through. High-level comparisons can be drawn between the VA and public school organizational and system structures. For example, both are federally governed institutions that are densely multilayered and complex entities. Because of the organizational and system complexity, policy change is often slow. Reliance on funding from taxpayers contributes to this measured response to needed policy change. As a result, stakeholders in positions to enact change, as well as recipients that stand to benefit from policy change, are often bogged down by bureaucracy and red tape (U.S. Department of Veterans Affairs; U.S. Department of Education). The QUERI model could be particularly useful for implementation projects in the public-school setting because of these similarities. Studies have shown QUERI to be compatible with socio-ecological targets of organizational approaches as well as system-based interventions and policy change (Demakis, McQueen, Kizer, & Feussner, 2000; Hagedorn et al., 2006). Working within the six steps of The QUERI approach is a four-phase framework that could be particularly useful for implementation projects within the public-school setting. The following will provide a foundational overview of the QUERI six steps followed by a detailed look at the four-phase model. Step one of the QUERI process model looks to identify the primary area of need (e.g., intervention, assessment, policy needs, etc.). Step two looks to identify evidencebased guidelines, recommendations, and best practices. Step three focuses on the identification of potential barriers and facilitators to improvement (i.e., determinant framework). Step four calls for the implementation of the specific improvement program (i.e., process model). Stage five and six focus on the evaluation of outcomes (i.e., 34 evaluation framework). The following phases would be conducted largely under steps four, five, and six of the QUERI steps just described. QUERI phase 1 Phase 1 calls for a small pilot project that serves to develop and refine an implementation plan and to assess basic feasibility. It is typically performed at a single site over a short 12-18-month timeframe. The primary goals during this phase are to minimize costs to the research team and stakeholders as well as provide basic information such as “lessons learned.” A phase 1 stage for a language screening implementation project in the public-school setting could take place at a single elementary school site with a possible time frame of one to two terms (~ 6 months). Lessons learned might involve the identification of employee resistance or support, limitations regarding site logistics, or in the case of a hybrid implementation design, the screening measure is shown to be insufficient to accurately identify children with language impairment. It is at this phase that a separate evaluation framework can be brought into direct measurement and evaluation. This particular project looks to utilize a phase 1 approach. QUERI phase 2 Phase 2 looks to build on the lessons learned from the pilot program by implementing small clinical trials to further refine and evaluate a particular implementation program. This is usually performed across four to eight sites over a similar time frame to that of phase 1. The goals of this phase look to utilize an active research team who oversees the development of measurement tools and evaluation 35 methods. A phase 2 stage for a school-based language screening implementation project would take place at multiple elementary school sites (four to eight) with a similar time frame, as outlined in phase 1. QUERI phase 3 Phase 3 aims to expand the project to a much wider scale typically, across 10 to 20 sites. This phase should involve a reduced need for real-time refinements of the implementation approach. The emphasis of this phase turns to measurements of impacts on the research team, site employees, and patients. A phase 3 stage for a language screening implementation project would take place across at least half of the elementary schools in a district. It would measure impacts of the research team, teachers (both participating general education and special services personnel), and students who are being screened. QUERI phase 4 Phase 4 proposes a full organizational roll-out of the program. The project now utilizes comprehensive research and delivery teams to support as needed based on data collected from phase 3 measurements and evaluations. Real-time minor adjustments to the program are made as needed. A phase 4 stage for a public-school project would involve the inclusion of all elementary schools within a particular district. Research and delivery teams would be fully integrated with district personnel working towards a full turnover of operations. Models and frameworks that utilize a “ramping-up” approach when dealing with 36 large organizational systems have shown to be successful. For example, the National Implementation Research Network (NIRM) utilizes a similar step or phase approach to that of the QUERI model. NIRM has been involved in projects involving the Office of Special Education on a national level with success (U.S. Office of Special Education). With a model and a specific phase identified as a good fit for use in initial feasibility studies such as this project, attention is now turned to the best type of implementation design that allows for the combination of implementation studies (i.e., identification of barriers and facilitators) and effectiveness studies (i.e. use of RSR measure). This is explored in the following section. Hybrid Implementation Design Research studies move through stages which serve specific functions. Efficacy stage studies conduct research in highly controlled environments where all independent variables have been identified and controlled. Effectiveness studies, on the other hand, conduct research in less than ideal environments (i.e., real-world settings) where every independent variable may or may not be identified and cannot be controlled. Randomized or pragmatic control trials are conducted at this stage. Pragmatic control studies aim to test the effectiveness of a specific assessment or intervention in wide-ranging routine clinical practice whereby the setting cannot be controlled and well-defined (Patsopoulos, 2011). They are designed to test assessments and interventions in the full range of everyday clinical settings in order to increase applicability and generalizability. The implementation phase looks to adopt successful findings into routine use at the clinical or practitioner level. Once full systematic uptake has occurred, improved outcomes should 37 be realized benefiting individuals, providers, organizations, and systems. Figure 4 outlines these stages from beginning to end. Traditional implementation studies take place as a standalone stage within the research pipeline. It takes an average of 15-20 years for research findings to achieve full systemic integration into a setting (i.e., stages 1-3). Many factors contribute to this research-to-service gap such as educational/knowledge insufficiencies (Freemantle et al., 1999), time limitations of clinicians (Fruth, 2010), lack of decision support tools and feedback means (Cabana et al., 1999), inadequately aligned incentives (Reschovsky, Hadley, & Landon, 2006), and a host of other organizational climate and cultural considerations (Grole, Wensing, Eccles, & Davis, 2013; Racine, 2006). Suggestions have been made to blend certain stages in order to reduce this large research-to-service gap (Glasgow, Lichtenstein, & Marcus, 2003; Wells, 1999). For example, intermingling the efficacy and effectiveness stages of treatment or assessment development could advance the speed of knowledge findings. Curran et al. (2012) contend that such blending can deliver benefits over following stages of research independently such as quicker translational advances in clinical intervention acceptance, more efficient implementation tactics, and increased information for key stakeholders. As a result, hybrid implementation designs have evolved to play a pivotal role in the reduction of researchto-service gap time. Hybrid designs complement three types that can usefully drive a particular study depending on the status of efficacy and/or effectiveness stages and primary and secondary aims. 38 Hybrid type 1 The first hybrid type places the primary emphasis on testing a clinical assessment or intervention procedure while the secondary emphasis is geared towards gathering information regarding facilitators and barriers for future uptake in a specific setting. Modest modifications to effectiveness studies could take place without jeopardizing the capability to achieve the stated primary aims of a project. Use of determinant frameworks is particularly useful at this stage to collect valuable information. As a result, subsequent implementation research trials or phase (e.g., QUERI 1-4 phases) stand to be optimized for best possible outcomes. Curran et al. (2012) recommend the following particular conditions for the use of a type 1 hybrid approach. First, strong face validity for a particular clinical assessment or intervention should exist that would be applicable to a specific setting, population, or delivery method in question. Second, ideally independent replication of the target assessment of intervention. Third, negligible risk exists with the target assessment or intervention both in direct and indirect forms as a result of the replacement of a known adequate intervention. The National Institute of Mental Health used a type 1 hybrid implementation design to test the clinical effectiveness of an intervention for anxiety disorders (i.e., primary emphasis) while concurrently conducting a qualitative process evaluation for stakeholders directly impacted by the implementation of the intervention (i.e., secondary emphasis) (Joesch et al., 2012; Sullivan et al., 2007). Specifically, the secondary emphasis looked to identify facilitators/barriers for the uptake of the intervention, sustainment of the implementation postcompletion of the project, and potential modifications of the intervention in order to better facilitate adoption and sustainability. 39 Hybrid type 2 The second hybrid type places equal emphasis on effectiveness outcomes and implementation methods. It endeavors to enact a more direct blending of clinical effectiveness and implementation of research aims in support of faster uptake of findings. Curran et al. (2012) state that a particular assessment or intervention in question need not necessarily be subject to randomized, highly powered designs. They justify this by asserting that the reality of ever-decreasing availability of research funding sources forces research studies to move forward with less than the ideal sample size. This reality could also be rationalized by realizing that effectiveness studies often yield different results from those of efficacy studies due to uncontrolled conditions or no research team support of delivery. Recommended conditions for a type 2 hybrid design expand those of hybrid type 1. First, a compelling case should exist for strong face validity as it relates to the clinical and implementation interventions/strategies that would encourage uptake to a particular setting, population, or delivery/implementation methods in question. Second, a strong base of ancillary evidence for the clinical and implementation interventions/strategies that would support uptake to a particular setting, population, or delivery/implementation method. Third, direct and indirect risks should be reduced with regard to the clinical and implementation interventions/strategies as a result of the replacement of a known adequate intervention/strategy. Fourth, momentum regarding implementation should exist within the particular clinical organization or system for routine adoption of the target assessment or intervention. Solomon et al. (2005) utilized a type 2 hybrid design by studying different intervention approaches for bone fracture prevention. Simultaneously, they assessed outcomes regarding key actors’ educational 40 understanding of the new intervention approach. Hybrid type 3 The third hybrid type places the primary emphasis on testing an implementation strategy and a secondary emphasis on the collection of information regarding a particular assessment or treatment outcomes. In an ideal nonhybrid research scenario, a certain assessment or treatment procedure should undergo numerous independent replications. By doing so, this helps finalize the effectiveness phase of the research pipeline. A hybrid type 3 approach assumes much further observation of effectiveness results than that of hybrid types 1 and 2 but still seeks to continue to collect evidence of clinical effectiveness while pressing hard for full systematic uptake. Curran et al. (2012) recommend that all four conditions outlined as requirements for a hybrid type 2 approach be met, in addition to a fifth. Evidence should exist to support a particular implementation intervention/strategy being tested as entirely supportable and feasible in the clinical and organization structure under study. Kirchner et al. (2011) used a type 3 hybrid design to conduct an implementation facilitation strategy in the VA hospital setting across 16 sites. The strategy consisted of internal and facilitation (plus numerous implementation tools and aids) to support the implementation of integrated primary care and mental health. Certain components of the strategy had shown strong effectiveness outcomes from previous trials, but a small portion lacked similar results. As a result, while the main outcomes were implementation focused, the study also concurrently collected additional clinical data regarding outcomes of those lacking. The primary aim of this project is to identify and examine potential barriers 41 and/or facilitators to the systematic uptake of universal screening in DSD, assessment of the usefulness of the RSR measure secondary, therefore categorizing it as a hybrid type 3 model. 42 Figure 1. Analytic Framework and Key Questions Regarding Screenings; Nelson, Nygren, Walker, & Panoscha, 2006 Figure 2. Past Tense Probes Sensitivity and Specificity Data; Adapted from Ash & Redmond, 2014 43 Figure 3. General and Specific Implementation Theoretical Approaches; Adapted from Nilsen, 2015 Figure 4. Four-stage Research Pipeline; Adapted from Curran et al., 2012 44 Table 1. Psychometric Properties of the RSR Measure Redmond Archibald & Archibald & (2007)a Joanisse (2009)b Joanisse (2009)c Sensitivity .94 .96 .85 Specificity .84 .76 .90 Positive Likelihood Ratio 8.15 4.0 8.7 Negative Likelihood .06 nrd nrd Ratio Positive Predictive Value .68 .63e .79e Negative Predictive Value .98 .98e .93e a Cutoff score: <85 on CELF-4; ages 5;0 – 6;11. b Cutoff score: <85 on CELF-4; ages 6;0 – 9;0. c Cutoff score: <80 on CELF-4; ages 6;0 – 9;0. d Values not reported. e Values not reported in original study but were calculated for this report CHAPTER 3 PRELIMINARY FILE REVIEW STUDY In an attempt to set the stage for universal screening as a more useful method of identifying language impairment in school-age children over that of a referral-based system, analysis of the baseline levels of success of that system is important. Christopulos and Kean (2018) conducted a study looking at the success rates of Davis School District’s (DSD) current referral system across four elementary school sites within the district. Results from the study provide additional support for this proposal’s exploration of universal screening. Introduction General education teachers are tasked with the identification of children who may be disadvantaged for a variety of reasons, including difficulties with behavior, literacy, social skills, academics, and many other areas. Among the areas teachers are asked to identify for possible assessment and intervention by special education is in the area of language development. Previous studies have shown that even experienced teachers find it difficult to identify important language elements in the domains of phonology, morphology, and syntax (Moats, 1994). Results indicated that even experienced and motivated teachers often understood too little about language structure to be able to 46 identify salient elements and provide appropriate instruction in this area. Recent data show that although teachers are attuned to differences in students’ language development, social skills, vocabulary acquisition, and literacy ability, they struggle to identify children’s specific language needs in order to better inform screening and assessment measures by the school-based SLP (Girolamo, Rice, Selin, & Wang, 2017). These data show that general education teachers were indeed making referrals to school SLPs, but their accuracy may be negatively impacted due to the incredible demands of required knowledge in a wide range of subject topics. Studies have shown a disparity in true positive and true negative rates (i.e., sensitivity and specificity) when relying on general education teachers as the primary source of referrals for children with language impairment to special education services. For example, Jessup, Ward, Cahill, and Keating (2008) looked at the accuracy of teacher identification of speech and language impairment in 286 kindergarten children compared to the formal screening measure of the Kindergarten Development Check (KDC) (Office for Educational Review, 2003). They found specificity rates in teachers to be high, with 93% and 97% of students with typically developing speech and language skills, respectively. However, sensitivity was comparatively low at only 50% for speech impairment and 15% for language impairment, indicating that 50% of students presenting with speech impairment and 85% of students with language impairment were not identified by teachers as having such a difficulty. Taken as a whole, these studies highlight the potential vulnerabilities of the current referral system as they relate to the identification of children with language impairment in the public-school setting. For this preliminary analysis, we addressed the 47 following research questions: Research Questions 1. What is the positive predictive rate of general education teachers when identifying and referring students with language impairment to special education in Davis School District? 2. Do the variables of teacher age, sex, years of teaching experience, and years of education predict general education teachers’ ability in identifying children with language impairment in Davis School District? Methods Four elementary school sites in Davis School District (DSD) located in Davis County, Utah were included in this study. Sites were selected based on DSD’s own classification of school socioeconomic status (SES). Based on that classification, sites were labeled for this study as: School #1: Higher SES, school #2: Middle SES, school #3: Low SES; non-Title 1, and school #4 Low SES; Title 1. • Student Participants: The records of 177 students enrolled in grades prekindergarten to 6th grade, and who were referred to special education, were reviewed. Of the 177 student records reviewed, 44 were referred for possible language impairment. Ethnic distribution of the students referred for possible language impairment was primarily Caucasian (77.27%) followed by Hispanic/Latino (11.36%), and American and Native Hawaiian/Pacific Islander (2.27%). One student was mixed race (2.27%), and three were unknown (6.82%). 48 Sex distribution favored males (61%) over females (39%). All students in this study spoke English as their primary language. • Teacher Participants: Information about the referring general education teacher was collected regarding age, sex, years of teaching experience, and level of education. • Measures: The file review was restricted to students enrolled in the participating school sites during the 2015-2016 school year. Student and teacher lists were obtained by the principal investigator with the University of Utah IRB and DSD approval. Student information was gathered primarily through electronic record review. For information not available on a small percentage of students, physical files were reviewed at the respective school sites. All teacher information was provided by DSD as deidentified. The file review focused on the following information. Students: grade, age, sex, source of referral, the reason for referral, and placement status (i.e., eligible/not eligible for services). Teacher: age, sex, years of teaching experience, and level of education. • Scoring and Analysis: The dependent variable of language referral correctness by teachers was scored binarily as 0 = correct and 1 = incorrect. A referral was considered correct if an IEP goal was written that corresponded to the reason for the referral from a particular source. Descriptive statistics were generated to determine the correctness of language impairment referrals. Binomial regression was used to determine the contribution of the teacher predictive variables of age, sex, years of teaching experience, and level of education. 49 Results • Correctness (Research Question #1) (Table 2): Three sources were found to make referrals to special education for language impairment assessment: general education teachers, early intervention, and parents. General education teachers had the highest proportion of referrals (45.45%), followed by parents (31.83%), and early intervention (22.72%). Individual source positive predictive rates on students referred to special education services showed early intervention as the most correct of the three sources at 90%, followed by parents at 78.75%. General education teachers were the least correct only correctly identifying 35% (i.e., true positives). It is noted these data do not account for those students with language impairment who were not identified by these three sources (i.e., false/true negatives). • Predictor Variables (Research Question #2): Binomial regression showed the model of general education teachers’ sex, age, years of teaching experience, and level of education as predictor variables only accounted for approxiamltey12% of model variance (R2 = .119). Individual predictor variables did not reach statistical significance: sex (p = .630), age (p = .375), years of teaching experience (p = .788), and level of education (p = .820). A test assessing multicollinearity of the model revealed moderate levels of correlation between age and years of teaching experience. Therefore, age was removed from the model, but still no statistical significance was observed. • School SES Status and LI Occurrence (Table 3): The middle SES school reported the highest proportion of referrals at 43.18% followed by the low SES; non-Title 50 1 school at 29.55%, the low SES; Title 1 school at 22.73%, and higher SES school 4.55%. The occurrence of language impairment at the respective sites were the middle SES school at 3.24%, low SES; non-Title 1 school at 1.43 %, low SES; Title 1 school at 1.56%, and higher SES school .19%. The total pooled occurrence of language impairment across all four schools was 1.38. Discussion Findings from this preliminary study support the contention that reliance on a referral system whereby general education teachers are a primary source of referral of children with language impairment to special education for assessment is error prone. Referrals provided by early intervention agencies were the most correct. This is not surprising as specialists involved at the early intervention level have had considerably more training in language development and disorders as compared to general education teachers and parents. Parents were approximately two times more likely to correctly identify concerns regarding language development as compared to general education teachers. This finding could be interpreted in different ways. First, it may be surprising that general education teachers were less correct in the identification of children with language impairment as most are exposed to topics on language development as part of their formal education. However, several studies have highlighted that teachers often feel ill-equipped to identify key markers of language deviance or delay due to little or no training, with knowledge needing to be attained primarily through personal experience and books (Marshall, Ralph, & Palmer, 2002; Sadler, 2005). Second, findings support a broad range of research identifying parents as a reliable source for identification of 51 developmental delay (Fenson et al., 2007; Law & Roy, 2008). For example, parent report measures take advantage of the vast knowledge possessed by caregivers regarding their child’s behaviors across a variety of settings. As a result, caregivers can shed more light on a child’s skill set as compared to naturalistic assessments or forced-choice standardized measures. Regarding general education teacher variables, it was particularly interesting that years of teaching experience was not a statistically significant predictor in language referral ability. There is some research that has shown this particular factor to be influential in teachers’ referral abilities (Podell & Soodak, 1993; Schwartz, Wolfe, & Cassar, 1997). School SES classification showed a disproportionate distribution of language impairment referrals across the four schools with the middle SES site having the highest occurrence rate. The higher SES site reported the lowest. Studies looking at SES contribution to language impairment have reported more cognitively stimulating and lower stress environments benefit children’s language environments, thus reducing the need for special education services (Mcloyd, 1998). Studies have also pointed to less supportive parenting as a factor in children experiencing depressed language growth due to lower quality and less frequent verbal interactions (Hart & Risley, 1995) although recently this long-held position has been challenged (Sperry, Sperry, & Miller, 2018). Findings from this study are supportive of these past findings with the low occurrence rate from the higher SES school, but less supportive with the middle SES school site demonstrating the highest occurrence rates over both low SES school sites. Individual and pooled occurrence rates across all four school sites were lower 52 than estimated rates of nonspecific language impairment at 19% (Beitchman, Nair, Clegg, & Patel, 1986) and estimated rates of SLI at 7-8% (Tomblin et al., 1997). DSD reports frequent use of a cutoff score of 1.25 SD on standardized tests by school speech-language pathologists to identify language impairments. No students in this sample received standardized testing to assess their nonverbal IQ, thus preventing a differentiation between cases of nonspecific and specific language impairment. 53 Table 2. Referral Source Correctness Proportion of Referral Source Referralsa True Positives General Education 7 45.45% Teacher 9 Early Intervention 22.72% 11 Parents 31.83% Combined 100% 27 a Total number of referrals for language impairment: 44 False Positive 13 Positive Predictive Rate .35 1 3 17 .90 .79 .61 Table 3. School SES Status and LI Occurrence School Site Proportion of Referralsa 4.55% 43.18% Individual School LI Occurrence .19% 3.24% School #1 (Higher SES) School #2 (Middle SES) School #3 (Low SES; non-Title 1 29.55% 1.43% SES) School #4 (Low SES; Title 1 SES) 22.73% 1.56% a Total number of referrals for language impairment: 44. b Total student enrollment across the four school sites: 3177 Total School Occurrence 1.38% CHAPTER 4 METHODS This study used a hybrid implementation, type 3 design (Curran et al., 2012) (Figure 5). Working within the hybrid design, the guiding conceptual aspects of a process model and determinant framework from the QUERI approach (Stetler et al., 2008) shaped the effectiveness and implementation components. Portions of the implementation study generated qualitative data from focus groups and followed the standards of reporting qualitative research outlined in the Consolidated Criteria for Reporting Qualitative Research (COREQ) to guard against potential sources of bias (Tong, Sainsbury, & Craig, 2007). The COREQ is a 32-item checklist developed for explicit and comprehensive reporting of qualitative studies, particularly regarding the use of in-depth interviews and focus groups. Appendix B outlines the checklist and how applicable items have been addressed. The first research aim was guided by the effectiveness portion, and the second research aim was guided by the implementation portion. Participants Aim 1: Research Questions 1 and 2 Students who participated in the screenings were recruited from all kindergarten level classrooms at two elementary school sites in the Davis School District (DSD) 55 located in Davis County, Utah. DSD elementary and secondary schools serve more than 72,000 students from over 330,000 residences, with an average median income of nearly $77,000, unemployment rate of 3.6%, 24% of adults within the community hold bachelor’s degrees, and 10.6% hold graduate degrees (Davis School District, 2018). Screenings took place during the first semester of the 2018-2019 school year. The kindergarten grade is particularly relevant for this aim under investigation as this grade level includes the period when designations of children’s language impairment are conventionally made (Beitchman et al., 1996; Redmond, 2005; Tomblin et al., 1997). In order to obtain a desired screener failure count of 15-20 participants, it was estimated a target total sample size of 120-150 would be needed (Bujang & Adnan, 2016; Glascoe, 2005, Tomblin et al., 1997). DSD reports an average per school kindergarten enrollment of 75 students, thus making two elementary school sites optimal for this target. In the end, a total of 165 students participated in the screenings. Screening participants met the following inclusion/exclusion criteria: Inclusion criteria were as follows: 1) Must meet DSD eligibility for kindergarten enrollment (i.e., must be at least 5 years of age on or before September 1st, 2018, certified copy of birth certificate, immunization record is completed and up-to-date, and valid social security number). Exclusion criteria were as follows: 1) Student has a known intellectual disability (i.e., Down syndrome, nonverbal cognitive deficit, etc.). 2) Any language other than English is spoken as the first language. Although negative consent (i.e., reverse or “opt-out” consent) is used by DSD for all universal screenings conducted for all students in kindergarten, DSD required the use of positive or “opt-in” consent given the exploratory nature of this study. Consent forms 56 were given to parents of all students from the two participating school sites by kindergarten teachers as part of a back-to-school-night packet during the 1st week of school. Permission to administer the RSR screening protocol was obtained from 165 parents. All children who had returned a consent form indicating parental permission to participate and who had provided examiners with their assent were included in the sample. Aim 2: Research Question 3 Focus group 1 DSD identified district speech and language teaching assistants (TA) as the position of their choice to conduct the screenings. DSD’s full complement of eight TAs participated in the training for the RSR screening measure. Seven of the TAs were available to conduct language screenings, and six of those were invited to participate in this focus group. Accordingly, participants in this focus group included all individuals involved in the direct administration of the RSR measure and met the following inclusion/exclusion criteria: Inclusion criteria: 1) Recruited as a project screener by the principal investigator, 2) Participated in mandatory training of RSR measure administration and scoring procedures prior to the screening of students, 3) Participated in direct screening of students in project target schools. There were no exclusionary criteria for this group. 57 Focus group 2 A total of six members participated in the second focus group with position descriptions as follows: Two Related Services administrators who were directly involved in the DSD side of participation/planning of this study. These individuals would ultimately be responsible for any future implementation decisions regarding an annualbased screening format for language impairment identification in DSD. Two general education teachers from each participating school’s cohort of kindergarten teachers. Two SLPs from each participating school. Participants from this focus group met the following inclusion/exclusion criteria: Inclusion criteria: 1) Recruited as planning and oversite team member by the principal investigator, 2) Participated in key phases of project planning and oversite as determined by the principal investigator, 3) Is a direct stakeholder under an annual-based screening format. Exclusion criteria: 1) Participation in the RSR screening measure administration at project target schools. Positive consent forms for both focus groups were emailed to participants 1 week prior to the focus group meeting for review. Printed and signed consent forms were collected at the first of the focus group meetings. Aim 2: Research Question 4 Survey participants were recruited from the DSD Related Services department and general education teachers employed as part-time or full-time speech and language pathologists or kindergarten teachers. At the time of the study, DSD employed approximately 91 SLPs as practicing clinicians at the elementary and secondary school level and 132 kindergarten teachers. Future extensions of this project include district- 58 wide implementation of screening for language impairment. As part of this, preliminary input and support from district speech and language pathologists and general education teachers prior to execution would be critical for success as they stand to see increased caseloads as a result of universal screening. Survey participants met the following inclusion/exclusion criteria: Inclusion criteria: Full or part-time employment with DSD during the 2018-2019 school year in elementary schools (i.e., kindergarten teachers and SLPs) or secondary schools (SLPs). While extensions of this project would include language impairment screening for K-3rd grades (Redmond, 2005) input from secondary school speech and language pathologists stand to add to the overall discussion of potential/perceived barriers and facilitators. There were no exclusionary criteria for this group. Positive consent was obtained from each respondent via a consent cover letter that accompanied the survey. Procedures As part of the prospectus development of this study, the principal investigator presented the project rationale and premise to DSD directors from Special Education and Related Services departments in order to assess potential levels of district interest. Those in attendance expressed high interest to participate in the development and administration of the project. Project details were submitted to the University of Utah’s Institutional Review Board (IRB) and DSD internal review board (i.e., Department of Assessment) for approval. Permission from both institutions was obtained prior to any data collection. During the approval process, the principal investigator began working with selected DSD project team members in the identification of the target school sites. 59 Aim 1: Research Questions 1 and 2 Data collection for this aim consisted of RSR measure administration and scoring training for all DSD TAs, a screening phase, and a confirmatory testing phase. Part 1 RSR measure training DSD TAs took part in an approximately 1-hour training session held at the DSD Related Services building in August of 2018. Training consisted of a brief overview of the project, introduction of the RSR measure, administration procedures, and scoring procedures. Practice opportunities were given to each participant to administer and score the task while listening to previously recorded child audio sample files from Redmond (2005). Training participants were required to pass a training quiz with a score of 100%, practice administering the task to two adults, and pass scoring reliability with one hundred percent on the child audio sample file. See Appendix C for the project administration and scoring protocol. One hundred percent of the TAs passed all required reliability. Part 2 RSR screening phase Student intake information was collected by DSD as part of the standard kindergarten registration process. Positive consent procedures were followed as outlined by the University of Utah IRB and DSD Department of Assessment (see description in Aim 1 in participant section). Student participants were given the opportunity to provide a DSD team screening team member with their assent prior to RSR measure administration. Screening time needed for each student was approximately 7-10 minutes 60 (Archibald & Joanisse, 2009). Part 3: Confirmatory testing phase The principal investigator conducted all confirmatory testing for this study and was blinded to students’ performance on the screening measure. Because a primary interest for this study was on the identification of SLI, measures assessing language ability and nonverbal intellectual ability were used as confirmatory measures. The Clinical Evaluation of Fundamentals, 4th edition (CELF-4) was used for the reference language measure of language impairment status (Semel, Wiig, & Secord, 2004). The CELF-4 assess morphological and syntactical abilities in the areas of concepts and following directions, word structure, recalling sentences, formulated sentences, and sentence structure. Editions of the CELF have been widely used in the literature for standardized assessment and differential diagnosis for SLI (i.e., Conti-Ramsden, Botting, & Faragher, 2001; Lloyd, Paintin, & Botting, 2006; Loucas et al., 2008; Marton, Kelmenson, & Pinkhasova, 2007; Plante & Vance, 1995; Wetherell, Botting, & ContiRamsden, 2007). The Naglieri Nonverbal Ability Test (NNAT) was used for the nonverbal intelligence measure (Naglieri, 2003). The NNAT assesses individual differences in pattern completion, reasoning by analogy, and serial reasoning. The NNAT has been used to differentiate SLI from general language-impaired groups or confirm SLI status (e.g., Ash, Redmond, Timler, & Kean, 2017; Redmond, & Ash, 2014). These two confirmatory measures were used to determine the positive predictive value of the RSR task when administered by DSD TAs. SLI was considered confirmed if a student’s core language standard score on the CELF-4 was < 80 and a NNAT standard 61 score of ≥ 80. Using this particular cutoff score for the CELF has shown adequate levels of identification accuracy for language impairment (Spaulding, Plante, & Farinella, 2006; Tomblin, Records, & Zhang, 1996). Aim 2: Research Question 3 Qualitative research approaches can be used to capture a deeper understanding of individuals being observed during an event within a natural context. This type of method can be particularly useful as it allows participants to share complex and detailed narratives regarding beliefs, knowledge, experiences, and perspectives (Hesse-Biber, & Leavy, 2010; Merriam & Tisdell, 2016). Qualitative methods are usually considered most fitting for investigating phenomena that may be complex in nature. Qualitative research is not informed by researchers’ hypotheses. In fact, Creswell (2014) explains that inferential meaning can be drawn from realities or truths given a certain environment. Corbin, Strauss, and Strauss (2014) emphasize that qualitative methods do not control variables but rather describe them as they support particular perceptions and outcomes (Feuerstein, 2018). Qualitative methods include a wide range of approaches. This project used an evaluation study design approach to obtain DSD project team members’ perspectives on potential barriers and facilitators that stand to hinder/facilitate the systematic uptake of universal language screening in DSD. Focus group methods bring inherent flexibility and wide-ranging application to multiple contexts in research. Qualitative data are produced in focus group settings by moderated participant interaction. Particularly, high importance is placed on group members interacting with each other, as opposed to interacting directly with the 62 moderator, in order to provide deep and rich experiential information. Barbour (2007) points out that through interaction, “concepts can be interrogated, concerns aired, meanings conferred, and rationales for views and behavior developed” (p. 37). The underpinning methodological framework for this project is based on content analysis. The primary purpose of content analysis is to systematically describe types of communication such as written, spoken, or visual (Hesse-Biber, & Leavy, 2010). In this project’s case, the systematic analysis of spoken communication generated from focus groups aimed to identify key facilitators and barriers of language screening in DSD. Focus groups have been routinely used in both communications disorders and implementation research (e.g., Boster & McCarthy, 2018; Di Rezze, 2016; Guest, Namey, Taylor, Eley, & McKenna, 2017; Law, Roulstone, & Lindsay, 2015; Powell et al., 2013; Starr, Martini, & Kuo, 2016). Following the completion of student screenings, DSD team members involved in the project took part in one of two focus groups (see Aim 2 in participants section for group breakdowns). The number of participants per focus group followed the recommendations outlined by Barbour (2008) of a minimum of three and a maximum of eight. The use of two separate focus groups was appropriate due to the heterogeneous nature of team members involved with the RSR measure administration compared to those involved in planning and oversite of the project. In addition, the scope of inquiry is somewhat small and structured, thus reducing the number of needed participants and/or groups to reach saturation (See Guest, Bunce, & Johnson, 2006). Focus groups were moderated by the principal investigator using a series of guided, semistructured questions regarding team members’ experiences, perceptions, beliefs, and/or attitudes towards the 63 RSR measure screening event and potential barriers and facilitators that stand to hinder/ facilitate the systematic uptake of universal screening in DSD (Appendix D). In addition to the moderator, each session included an independent note-taker. Each focus group lasted approximately 60 minutes and was digitally recorded for video and audio. The moderator and note-taker completed field notes during and following the focus groups that documented the main themes of the sessions and any conversations pertinent to the study aims. Specifically, observations regarding descriptions of the groups as a whole, individual participants, and group dynamics were noted. Barbour (2008) notes the form and content of qualitative data generated from a focus group can be impacted by the moderator’s bias. Therefore, it is often recommended the moderator acknowledges any potential bias prior to the conduction of focus groups (i.e., reflexivity). A reflexivity exercise need not be overemphasized by magnifying a moderator’s uncertainty or discomfort, but rather to contribute to theoretically informed analysis (Barbour, 2008). Pascale (2010) stated: “Interpretation is always a matter of positionality – how one is situated within networks of power, geographies of privilege, and the histories of experience. It might seem that standards of good research would require scholars to be accountable for the many processes of interpretation involved in knowledge production.” (p. 72). Accordingly, a reflexivity statement by the principal investigator, as it relates to this project, is included in Appendix E. Moderator bias was assessed by listening to recorded playback of the first focus group session. Following completion, all focus group data were transcribed verbatim by Landmark Associates Inc., located in Phoenix, AZ (Landmark Associates, 2019). 64 Aim 2: Research Question 4 Data collected through survey methods have been widely used among researchers in behavioral and social sciences. Generally speaking, the goal of surveys is to generate statistics that are numerical descriptions (i.e., quantitative) about a certain feature of a study population (Fowler, 2013). The primary method of collecting information comes by way of asking participants questions. Respondent responses comprise the data to be analyzed. As is the case with most quantitative methods, information that is collected comprises a sample of the study population. Surveys have been widely utilized in communication disorders research (e.g., Chandler et al., 2007; Eaves, Wingert, Ho, & Mickelson, 2006; Lewis et al., 2016; Lue, 2001; Reetzke, Zou, Sheng, & Katsos, 2015). In fact, the use of data collection via parent checklists, a type of targeted or individual survey, provides valid and reliable information in the study of language impairment as is the case with the Children’s Communication Checklist (CCC-2), MacArthur-Bates Communicative Development Inventories (CDI), and the Language Development Survey (LDS). Use of surveys for data collection in implementation research is widely documented as well (e.g., Armstrong et al., 2017; Breiman, 2016; Neta et al., 2015; Palinkas et al., 2015; Ruben, 2000). A project-specific self-administered web-based survey was developed to assess DSD speech and language pathologists’ and kindergarten teachers’ views of potential facilitators and/or barriers to universal screening for language impairment. Respondents’ demographic information was collected as part of the survey including gender, age, years of teaching experience, and level of education, years working with children with language impairment, caseload size, number of children on caseload, and type of school 65 setting. Survey statements were informed by data generated from the focus groups. In order to ensure a relatively low burden to respondents, a target maximum total number of 12-15 statements was desired. In the end, 20 statements were developed. This number was needed to explore all main themes identified in the focus groups fully. All answers required closed-ended responses using a Likert-style scale set on the following dimension (Likert, 1932): 1) Agreement (e.g., strongly agree, agree, neutral, disagree, strongly disagree.) The basis that informed statement development included respondents’ responses to realized themes from focus group data. The principal investigator informed district SLPs and kindergarten teachers of the survey solicited participation via email. The survey was sent out via district email with a link to the online consent form and survey. The online survey platform used was mysurveylab.com. Prior to district-wide survey distribution, the online survey was pilot tested on this study’s committee members and four district SLPs to ensure face-validity and ease of use. Analysis This study employed a multimethod evaluation study design. Multiple methods of analysis were utilized over the course of this project. Aim 1: Research Questions 1 and 2 Pass/fail groups were constructed using a standard score of 80 as the cutoff point in association with the RSR measure. To examine the positive predictive rates of the RSR task, the CELF-4 was used as the reference standard to confirm students’ language status 66 (i.e., atypical language vs. typical language). Diagnostic values such as rates of true positives and false positives were generated and analyzed (Dollaghan, 2007). Positive predictive values were calculated for the RSR measure. Occurrence rates were calculated and compared against reported epidemiological rates of SLI. Aim 2: Research Question 3 Focus group transcriptions underwent systematic text condensation (STC). STC involves the process of selection, simplification, and transformation of data in order to allow for conclusions to be drawn, codes assigned, and categories (i.e., themes) to emerge (Malterud, 2012). A four-step process is involved in STC that was used for data extraction, analysis, and interpretation. 1) Overview of data. Impressions were formed by looking for preliminary themes. It is noted that although Malterud (2012) stated that preconceptions are to be bracketed while still realizing that an interpretative position determined by the project aim must be realized, this step was somewhat minimized with the aim’s primary intent of looking for themes of “barriers” and “facilitator” to universal screening for language impairment implementation. Other themes of interest that emerged were given full attention. 2) Identifying and sorting themes into codes. Transcript data were systematically reviewed line by line to identify meaning units. A meaning unit was defined as parts of the text that contain information directly related to the project aim. After meaning units were identified, coding began by sorting meaning units under identified themes from step 1. Coding is defined as the decontextualization of parts of text in order to assign a label that connects related meaning units into a code group. The naming of code groups is an elaboration from the themes from step 1. It is 67 important to note that codes were uniquely created to avoid preconceptions and to develop a deeper understanding of the data. 3) Reduction of codes into code groups. Code groups were created by organizing and reducing codes identified from step 2. This step was reductive in nature. 4) Synthesis and summarization of code groups. This step utilized framework analysis, as outlined by Ritchie and Spencer (2002). Framework analysis utilizes a grid or matrix system to help recognize or “frame” patterns in the code groups across both focus groups. The final grid summarized the patterning with regard to the identification of code groups under the broad themes of facilitators and barriers and others. Aim 2: Research Question 4 Means were calculated for each survey item across SLPs, kindergarten teachers, and their combined responses. Specifically, means were used to analyze individual and group trends in perspectives on facilitators and barriers regarding the implementation of universal screening of language impairment in public schools. Other potential descriptive statistics were not useful for the questions under consideration (i.e., standard deviations and ranges) as a function of the nature of the Likert scaled data collected. 68 Figure 5. Project Structure CHAPTER 5 RESULTS The results of this study are divided into three sections. The first section addresses Aim 1: Research Questions 1 and 2 (re: RSR screening data) and reports on findings from the administration of the RSR sentence recall task to students of kindergarten classes from two separate elementary schools in DSD. The second section addresses Aim 2: Research Question 3 (re: focus group data). This section reports on code groups derived from the overall themes of facilitators and barriers to the processes of implementing universal screening in the DSD context from two focus groups: Group 1: DSD Related Services teaching assistants (TA). Group 2: district stakeholders who were directly involved in the planning and oversite of the screening phase of this project, or who would be directly impacted by the implementation of a universal screening format. The final section addresses Aim 2: Research Question 4 (re: survey data). This section reports on findings from the survey administered to DSD general education kindergarten teachers and SLPs. Figure 6 shows the project flow and timeline. Aim 1: Research Question 1 and 2: RSR Screener Results Ten DSD teaching assistant (TAs) participated in a mandatory 1-hour training session for the RSR screener protocol conducted by the principal investigator. Areas of 70 background and description of the RSR screener, required materials, scoring sheet, administration, and scoring were reviewed in PowerPoint presentation format. TAs were given the opportunity to practice administering the screener to each other and on a child audio sample file. Participants were then required to demonstrate competency on three measures: 1) listening reliability, 2) scoring reliability, and 3) a nine-question written quiz. Passing cutoff on all three measures was 100%; 10/10 = 100% of participants passed reliability. Screenings and Confirmatory Testing Approvals by the University of Utah IRB board and DSD assessment department were received prior to consent forms being distributed to the kindergarten classes of the two elementary school sites. Participating students met the following inclusionary criteria: 1) Met DSD’s eligibility for kindergarten enrollment (i.e., must be at least 5 years of age on or before September 1st, 2018, certified copy of birth certificate, immunization record is completed and up-to-date, and valid social security number). Exclusionary criteria included: 1) Student has a known intellectual disability, 2) Any language other than English is spoken as the first language. A total of 182 kindergarten students met the eligibility criteria to participate in the screenings. Of the students eligible, 165 (90.66%) assented to participate. Descriptive statistics are shown in Table 4 for the measures of screening and confirmatory testing. Confirmatory measures included the Clinical Evaluation of Fundamentals, 4th edition (CELF-4) for language (Semel, Wiig, & Secord, 2004), and the Naglieri Nonverbal Ability Test (NNAT) for nonverbal intelligence (Naglieri, 2003). Standard scores on the sentence recall task were based on 71 the community sample from Redmond, Ash, Christopulos, and Pfaff (2019). One hundred and forty-six students passed with a mean standard score of 100.49 (range 80-129). Nineteen of the 165 students failed the screening protocol with a mean standard score of 70 (range 67-80). Confirmatory testing was completed on the 19 students who failed the screener using the CELF-4 Core Language subtests for language assessment and the NNAT for nonverbal intelligence assessment. Group means for the CELF-4 and NNAT were 74.89 and 98.32, respectively. Table 5 shows the individual results of this group, including demographics, findings diagnosis, and status of services at the time of testing. Three categories were used to classify results of confirmatory assessments: 1) Specific language impairment (SLI): students were classified as SLI if the CELF-4 result was at or below the cutoff score of 80 and NNAT result was above the cutoff score of 81 and above. Thirteen of the 19 students were classified as SLI. 2) Nonspecific language impairment (NSLI): students were classified as NSLI if the CELF-4 result was at or below the cutoff score of 80 and NNAT result was below the cutoff score of 81 and above. One student (#3) met these criteria. It is noted that their nonverbal test score was nearing criteria for SLI. 3) Typically developing (TD): students were classified as TD if both the CELF-4 and NNAT scores were 81 or above. Of the five students classified as such, two of them (#4, #13) had CELF-4 scores slightly above the cutoff score of 80. The identified occurrence rate of LI reported by Christopulos and Kean (2018) under the current referral-based format in DSD was 1.38%, well below prevalence expectations of SLI offered by Norbury et al. (2016), Tomblin et al. (1997), and Beitchman et al. (1986). The identification rate identified by this project utilizing a 72 screening format in DSD mirrors very closely expectations based on epidemiological reports at 7.88%. When factoring in potential comorbid conditions (SLI + NLI), the occurrence rate of LI increased to 9.09%. Table 6 provides true positive, false positive, and positive predictive rates (PPR) of the RSR screening protocol compared to referral sources identified in Christopulos and Kean (2018). Data used to calculate true and false negatives were not collected as part of the larger data set due to limitations placed on the study by DSD. Because students who passed the screening could not participate in confirmatory testing, other diagnostic values such as sensitivity/specificity could not be calculated. PPR for the protocol was .74. When compared to the other referral sources (i.e., general education teachers, parents, and early intervention), the RSR task was most comparable to the parent source PPR of .79, and well above the PPR associated with referrals from general education teachers of .35. Aim 2: Research Question 3: Focus Group Data Results Two focus groups were conducted to understand specific barriers and facilitators to the implementation of an annual screening format as experienced from their participation in training, facilitation, and administration of the RSR screener protocol. Focus group 1 included six DSD Related Services TAs. Focus 2 included two DSD Related Services administrators, two DSD SLPs, and two general education kindergarten teachers. Both focus group sessions were video recorded, and recordings were transcribed verbatim by Landmark Associates Inc., located in Phoenix, AZ (Landmark Associates, 2019). These transcriptions then underwent step process analysis for data extraction (i.e., 73 Systematic Text Condensation). See the methods section for a breakdown of these steps. Code Group Overview Twelve overarching code groups were identified from both focus groups: 1) Supportive attitudes and willingness to change, 2) Availability of resources/time constraints, 3) Benefits to educators/district, 4) Concern for the well-being of students, 5) Screening equipment, 6) Need for a validated screening tool, 7) Opinions of screener and screening administration, 8) Response to Intervention (RTI) need, 9) Training on screener protocol, 10) Understanding/education with regard to LI, 11) Use of existing policy/structure to facilitate annual screening, 12) Who best to administer screener. It is noted that several of the code groups could be classified as either a facilitator or barrier depending on the nature of the feedback provided. For example, responses regarding the code group of screening equipment highlighted issues such as the use of an mp3 player and headphones and the overall simplicity of the equipment as being helpful. These items were identified as facilitators. On the other hand, issues such as the need for clipboards and pencils wearing out during a screening session were presented as drawbacks and therefore identified as barriers. Table 7 shows framework analyses outlining the individual code groups identified from the focus group data analysis as respective facilitators or barriers. Framework analysis uses a grid or matrix system to help identify, or “frame” patterns in the code groups across both focus groups. 74 Code Group Detail Supportive attitudes and willingness to change (facilitator and barrier) Both groups identified attitudes of key stakeholders such as DSD SLPs, general education teachers, school site administrators, parents, and district administrators as important. Specifically, attitudes relating to the willingness to change from a current system of familiarity to a new system was fundamental. The following statement from a participant in focus group #2 in response to follow-up questioning on this topic captured the overall feelings of both groups: Interviewee 3: I mean administrators who think, “Well, this is going to be expensive.” I mean professionals, who may feel like this will usurp their role and responsibility, and maybe take their place. Anyone who’s resistant to change, they will have an attitude of, “Oh, that’s not going to work.” This is just another new thing. Kindergarten teachers expressed some resistance to change to an annual screening format if it meant increasing their workload burden as captured in this exchange between focus group #2 participants: Interviewee 3: Interviewee 2: Interviewee 5: Interviewee 3: Interviewee 5: Administrators tell me, they don’t want to put burdens on teachers. They would ask me, “Well, what burden does this have on the teacher?” We want the teacher teaching kids. We don’t wanna put any more on our teachers.” If the teachers had a good attitude about this. Well, finding the children takes the burden away from us because it’s helping with the child! This is me, just coming from me. If you’re really not asking me to do it, then I’m in, you know what I mean? Exactly. Exactly. I am back in Kindergarten for the first time and I am working, seriously, 10-to-12-hour days. My husband and I are almost divorced over teaching Kindergarten. It shouldn’t be that hard. Because he makes three or four or five times as much as I, and he works about one-tenth of the time. And that’s no joke. Anyway, but if I’m not gonna have to administer the test, really, what burden is it? 75 Others indicated they would have supportive attitudes and be willing to change as indicated by the following statement from focus group #1: Interviewee 3: Now, I think the SLPs have a great heart, and I think they do want to help as many students as they can, but they gotta keep their sanity at the same time. Availability of resources/time constraints (barrier) Both groups recognized that resources, in general, are limited in public education and the implementation of any new program would mean increased demands across multiple domains. Participants from focus group #1 had the following exchange underscoring this potential barrier: Interviewee 4: Interviewee 6: Interviewee 2: Interviewee 4: Interviewee 5: Interviewee 4: I think most SLPs are wanting to help everyone they can, but right now, a lot of them have 80, 90 kids on their caseload. We do this screener, and that's gonna find more. Even if you found five more, that's phenomenal when you're already overworked. That’s a lot of kids, yeah. Is the district going to hire more SLPs or SLTs to take over? To support? Yeah, not to take over, but to support the language, because a lot of them already feel like the water's too deep here, and they're just about drowned. Benefits to educators/district (facilitator) Focus group #2 identified that an annual screening format for language impairment would be beneficial for multiple stakeholders in the district, specifically for general education teachers as captured in this exchange: Interviewee 1: Interviewee 6: Interviewee 1: (As a general education teacher) I don’t pick up on language very easily. It’s hard for almost everybody. It is. Yeah. 76 Interviewee 6: Interviewee 3: Interviewee 2: Interviewee 1: Interviewee 2: The only referrals I usually get for language in any grade is if the psychologist picks up on patterns or strengths and weaknesses. That’s the only language referrals I get. What I hear you saying, as a Kindergarten teacher, you would feel much more confident if there were some kind of screener, something that you could run your kids through and then you’re insured. You don’t have to worry about it. Something tells you that they’re okay? I feel like I let kids slip through the cracks all the time because, I don't know, I’m not a speech pathologist. I don’t know what to listen for. You worry about it. Yeah, I would love just to, “Speech check?” “Okay.” “They’re good.” You know what I mean? That's awesome. Concern for well-being of students (facilitator) Even as discussion in both focus groups was quite robust with regard to barriers such as limited resources and burden placed on certain groups of professionals, both groups always returned to the salient point that student needs are paramount. Interviewee 2: Interviewee 3: What it comes down to though is, do you do what’s best for the child? This is what’s best for the child. For me, as a teacher. I don't care about costs. I don’t care about parents or anybody else, only what’s best for the child, and this would be helpful. Mm-hmm. If it’s gonna help your students. Mm-hmm. The following exchange between members of focus group #2 emphasized the experience often had by educators that teach children with language impairment. Interviewee 5: Interviewee 3: Interviewee 5: Sometimes, I think, I look at ‘em and I think they’re defiant students. I was just going to mention that. I really think, why are you not doing this or that? I get mad at ‘em. I’m like, “Why are you not doing what I’m telling you to do?” Then come to find out, they look at me and they really do not know what I am saying to them. They really cannot process. Then I find out—I don’t yell and scream and get mad at them—but then I feel bad that they really don’t understand cuz they are not processing the way that I process. Then we have to take a step back and work with them the way their poor little brains work, in a different way. 77 Screening equipment (facilitator and barrier) Focus group #1 provided useful feedback for both themes of facilitators and barriers as it related to the use of the screening equipment such as mp3 players, headphones, scoring sheets, writing instruments, etc. In terms of facilitators, most members reported the use of electronic equipment helpful to maintain consistency in sentence stimuli delivery. Use of headphones, as opposed to speakers, allowed young students, who may be prone to reduced attention because of surrounding auditory or visual stimuli in the immediate environment, to stay on task. Most reported use of the electronic equipment easy and straight forward. In terms of barriers, members expressed that clipboards would be useful in order to shield the recording of responses from the student. Members pointed out that pencils easily wore down due to the amount of writing required over the course of screening and that use of mechanical pencils may be beneficial. Some members reported that certain words on the recording were consistently difficult for students to understand. The age and use of the specific stand-alone mp3 players were commented on as being outdated. Finally, many members reported that the timing between stimuli sentences was difficult to gauge and control. Need for validated screening tool (barrier) Both groups emphasized the importance of having a screening measure that is valid and reliable to anchor an annual screening format. Members of focus group #2 paired this barrier with the barrier of availability of resources, specifically financial resources. The concern articulated was that if a screening tool was identified as valid and reliable but would incur cost to the district because of the need to purchase, this may 78 discourage the implementation of such a format. This barrier was identified as particularly relevant by members from both focus groups because DSD currently does not officially endorse a specific measure or tool for language impairment screening under the current referral-based system. Finally, the importance that a screening tool is valid and reliable for the bilingual or ESL student population was also emphasized by both groups. Opinions of screener and screening administration (facilitator and barrier) Focus group #1 offered beneficial feedback for both general themes of facilitators and barriers with regard to opinions of the screening measure itself and administration of it. In terms of facilitators, some members felt it was helpful to have instructions, sentence stimuli, and recording of responses condensed to one page. Many expressed initial apprehension that online scoring would be too difficult, but reported after one or two students, the recording of responses and online scoring was quite easy to pick up on. Some members even reported that administration of the screener was so straight-forward, they did not feel a stand-alone training was necessary. All highlighted the helpfulness and necessity for practice items. This helped set expectations and provided an opportunity for the students to understand how to navigate the task. Many reported the instructions were informative, well-worded, and helpful for the students. In terms of barriers, some members felt the condensed score sheet made the print difficult to read. Certain participants would have liked more space to record and annotate student responses. Others reported that at times, students had difficulty comprehending certain words used in the stimuli sentences. A few members questioned the number of stimuli sentences 79 because it was hard for some students to maintain attention for the full duration of the task. Others reported that often, students had a difficult time understanding the instructions. It was difficult relaying expectations to students because instructions were first given as part of the mp3 recorded file and a second time verbally read by the administrator. This seemed to confuse some students and negatively impact the pacing of the first few sentences of the screener. Some members reported the condensed one-page score sheet did not provide adequate room to record students’ responses with the needed corresponding notations. Finally, certain members reported they felt some of the stimuli sentences were too complex for kindergarteners. Response to intervention (RTI) need (facilitator) A key mediating facilitator underscored by both focus groups to the barriers of potential increased caseloads and burden on SLPs was the utilization of an RTI format such as multi-tier system of support (MTSS). Framework based tiered approaches such as MTSS implement systematic, evidence-based practices that maximize student achievement and aid in the early identification and support of students with learning needs. Critical components of the MTSS model include universal, targeted, and intensive levels of support. Universal or tier 1 support systems are provided to all students and include ongoing formative assessment such as universal screening. Targeted, or tier 2, support systems provide core instruction with supplemental, most often collective, targeted supports with increased time, intensity, explicitness, strategy, and response. Tier 2 support systems offer core instruction with intensive individually responsive supports in increased time, intensity, explicitness, strategy, and response. Tier 2 and 3 interventions 80 are typically accomplished in the classroom setting and are compulsory prior to any student being considered for qualification to special education services. Utilization of an RTI format was discussed as a key in “spreading out” of the burden rather than placing it exclusively on one group of professionals such as SLPs. Enthusiasm for RTI was somewhat tempered however by some members questioning whether general education teachers could actually follow through and implement tier 2 and tier 3 in-class interventions as highlighted by this exchange by members from focus group # 1: Interviewee 2: Interviewee 5: Interviewee 2: Interviewee 5: Interviewee 6: Interviewee 2: Interviewee 6: Well, that's just it. Once they go through this screening process, do they still have to do interventions with the classroom teachers? Yes, that would be the next step. Yeah, so the next step would be the intervention with the teachers. Then, so I can just see these SLPs. Some of the SLPs get so frustrated because the teachers will not—they won't follow through. I go and do interventions. Yeah, I do interventions. I do too, and so do the SLPs. They're saying the teachers aren't doin' them, the kid needs to qualify, so I go and do the interventions. What I see mainly coming from the SLPs is, like you said, more on my caseload, more to do. If the kids need to be caught, they need to be caught. That's what I was gonna say. They want the kids to succeed. This exchange continued by shifting focus to speech-language technicians (SLT). SLTs are not a professional position currently utilized in DSD, but this situation has been discussed among Related Services administrators as potentially changing in the near future. Focus group members identified this group of professionals as a plausible alternative to help reduce the burden placed on general education teachers and SLPs as a result of an RTI approach. 81 Interviewee 5: Interviewee 1: Interviewee 4: This could being something the SLT—do you know what I mean? The next step in making sure, if there are SLTs in the district, helping this load for the SLPs. That would definitely be a case for having SLTs. Yes, SLTs, wouldn't it? Training on screener protocol (facilitator and barrier) Focus group #1 offered valuable feedback with regard to their experiences in training for screening administration both in the themes of facilitators and barriers. With regard to facilitators, all members reported the training was thorough and informative. Specifically, the TAs valued the training opportunities where they were able to practice administration with supervision. Most members reported they felt prepared following the training to administer the screening protocol. Many of the members reported that training regarding the underlying mechanisms of language impairment (e.g., difficulties in the areas of morphosyntax and verbal working memory) helped them to understand better what they were targeting with an annual screening format. With regard to barriers, some members reported the time gap between training (August) and actual administration of the screening protocol (October and November) as being too long. Some reported they felt they had forgotten salient aspects of administration. This was negated somewhat by reviewing handout materials provided at the time of training immediately prior to screenings. Some members felt time during training would be better spent in practicing exclusively on children participants as opposed to both adults and children. 82 Understanding/education with regard to LI (barrier) Focus group #1 members seemed to struggle most with overall understanding and being educated with salient elements of language impairment such as prevalence, need for screening, and short and long-term outcomes of untreated language impairment. This carried over into certain issues raised pertaining to concerns with the screening protocol as expressed by these members: Interviewee 5: Interviewee 6: Interviewee 5: On a couple of them, I had to repeat it two times or three times before they understood what the instructions were. Oh, and you could tell by the practice one that they didn't quite understand it. Oh, they were just tripping over words, not getting it at all. I mean, these were the students that really scored low, and so they didn't— yeah, they didn't understand what was expected of them at all. I played it again and then I gave a verbal prompt as well with the recording. See, I did those three days, so I hit a lot of students, so some just really struggled with it. During postscreening review, some of the TAs struggled to make the link between a particular student struggling with the screener as a function of having a language disorder and that the protocol was doing precisely what it was intended for. Kindergarten teachers in focus group #2 expressed inadequacy with being comfortable identifying markers of language impairment as expressed in this exchange. Parents were also recognized as not being able to identify key markers: Interviewee 5: Interviewee 2: Interviewee 5: Interviewees: Interviewee 5: Well, I know that when I taught second-grade, they get to secondgrade and you’re thinking, “Why aren’t they learning?” Then come to find out, it’s a big, huge language thing. Yeah. And you’re thinking, well, why didn’t they catch this in Kindergarten? Mm-hmm. Mm-hmm. Why didn’t they catch this in First Grade? It’s because they can keep up, but by Second Grade, they really can’t keep up because it’s going—and I can see that now. You’re back in Kindergarten, they can look around. 83 Interviewee 3: Interviewee 5: Interviewee 2: Interviewee 5: That's what I was just gonna say. They can look around and figure out what the other kids are doing. Yeah. By Second Grade, students can look around and figure something is wrong with them. Interviewee 2: Yes. Interviewee 5: Now that I’ve been up, and now I’ve come down, you can pick it up a little bit. There’s a girl in my class, and you’re thinking, “What? Why can’t she understand?” I think there’s a language thing there. I talked to her parents about it and I think it’s because her parents think that that’s what normal. Interviewee 2: They think it’s normal. Interviewees: Yes. Mm-hmm. Interviewee 2: They think it’s normal because this is their first kid. Interviewees: Right. Yes. Interviewee 5: I’m like, “Well, it’s not really normal? What does she do when you give three directions?” They’re like, “Well, we don’t. We give her one.” Well, that’s not normal. By now, you should be able to say, “Go in your room. Put your pajamas—' You know what I mean?” Anyway, so it would be nice though because I don’t know though what to do to pick up on what is a language. Interviewee 2: I think it would be nice because now I know, a lot of times, it is a language problem. SLPs in focus group #2 reported some uncertainty with treatment approaches for children with LI. They stated that the breadth of training required at the graduate level prevented them from fully gaining a solid comfort level in language disorder assessment and treatment. Use of existing policy/structure to facilitate an annual screening format (facilitator) Members of focus group #2 emphasized that use of existing policies or procedures in DSD would support the transition to an annual screening format. For example, application of the current multitier system of support (MTSS), which is an RTI approach, is a new system recently implemented in DSD that stands to mediate several key barriers that may stand in the way of switching to an annual screening format. Utilization of SLTs 84 as a professional position in the district is an intriguing option as they stand ready to help reduce current caseload burden of SLPs, help in the regular administration of an annual screening protocol, and support tier 2 and 3 MTSS intervention. Who best to administer screener (barrier) A robust amount of consideration was given by both focus groups regarding the topic of who is best to administer the actual screening protocol. Five potential groups were identified: parents, general education teachers, TAs, SLTs, and SLPs. Generally speaking, most members felt parents were the least qualified due to lack of knowledge and experience. SLPs and general education teachers were next identified as not plausible options due to limited available time, although many members conceded they might be the most qualified, particularly the SLPs, for obvious reasons. TAs were identified as a suitable option, but currently, only a small amount of these professional positions are offered in the district making it untenable for such a small group to carry the heavy load of district-wide screening. The group identified as the best option to carry the load of regular screening administration was the SLTs. The addition of this position on a wide scale to the district has been considered for several years as a viable solution to help reduce the current burden on SLPs. They make for an intriguing choice due to the factors outlined in the use of existing policy/structure section. Figure 7 shows the code groups as identified by their respective focus group. Theme overlap is targeted. While the participatory roles and responsibilities of members of each focus group differed considerably, there was still a 58.33% overlap across the two groups of the 12 identified code groups. 85 Aim 2: Research Question 4: Survey Data Results Survey Statement Overview Following focus group data extraction, 20 survey statements were developed to gauge the extent DSD SLPs and general education kindergarten teachers agreed/disagreed with the barriers and facilitators identified by the focus groups. Each statement was directly associated with the facilitators and barriers identified by focus group participants. Figure 8 shows the mapping of the 12 identified code groups onto the 20 survey statements. In addition to these 20 statements, six demographic questions were asked to look at potential predictors (years of experience, level of college education, gender, and age). Surveys were sent via email on an individual basis to all participants via www.mysurveylab.com. Individual follow-up reminder emails were sent to those who had not taken the survey after 1 week. In total, 157 surveys were completed. Participation rates across the two groups were as follows: DSD SLPs: 69/91 = 76%. General education kindergarten teachers: 88/132 = 67%. Table 8 shows final descriptive data for the 20 corresponding survey statements. Figures 9, 10, and 11 show individual and combined group responses in chart format. Survey statement detail Of those who began the survey, completion rate was 100% among both general education teachers and SLPs. One survey statement (#14) was developed exclusively for SLPs. General education teachers were provided with a “does not apply to me” answer choice among the Likert responses. One teacher mistakenly selected the neutral answer 86 choice. Accordingly, survey results for this statement reflect the exclusion of the particular response. Statement #1: Yearly screenings (e.g,. hearing, vision, reading, etc.) are an effective way of identifying children with developmental difficulties. The vast majority (90%) of kindergarten teachers believed that screenings, in general, are an effective tool when identifying students with developmental difficulties with 47% of them strongly agreeing. Ninety-seven percent of SLPs thought screenings are effective with 36% of them strongly agreeing. Statement #2: Yearly screening, compared to a referral-based approach, is more effective for identifying school-aged children with language impairment. The next statement narrowed the focus of question #1 to ask specifically about the potential effectiveness of screening for language impairment. The support for this approach as being effective dropped compared to statement #1, but still remained high with 71% of kindergarten teachers and 72% of SLPs approving. Thirty percent of kindergarten teachers strongly agreed with this statement, while only 17% of SLPs did. Statement #3: A yearly screening format for language impairment identification would be well-received by district general education teachers. Eighty-seven percent of kindergarten teachers and 71% of SLPs positively responded that DSD general education teachers as a whole would benefit from an annual screening format for language impairment. Forty percent of kindergarten teachers strongly agreed with this, while only 17% of SLPs strongly agreed. Only 6% percent of SLPs did not believe that screening for language impairment would be well-received by teachers as compared to only 2% of kindergarten teachers. 87 Statement #4: A yearly screening format for language impairment identification would be well-received by district speech-language pathologists. Sixty percent of kindergarten teachers felt that SLPs would favorably react to yearly screening for language impairment. SLPs showed less enthusiasm with 50% indicating favor. Of these, 22% of kindergarten teachers strongly approved while only 9% of SLPs strongly approved. Of note, the neutral responses by both groups were robust at 36% for kindergarten teachers and 30% for SLPs. Disagreement rates followed the same trend as the previous question with 19% of SLPs disagreeing with this statement as compared to 5% of general education teachers. Statement #5: A yearly screening format would place an unmanageable burden on district speech-language pathologists. A prominent theme that emerged during the focus group exercise was sensitivity to the potential burden of increased caseload sizes that may result from an annual screening format for language impairment. Results from the kindergarten teachers were rather split with 34% supporting this statement and 26% not supporting. SLP responses neared majority with 44% in support. The minority view disputing the assertion was half of those in support (22%). As with the previous question, neutral responses were prominent at 40% for kindergarten teachers and 35% for SLPs indicating levels of ambivalence to a potential burden. Statement #6: A yearly screening format would reduce the burden on district general education teachers. This question reversed the polarity of the previous statement by addressing the stakeholders that currently bear a large portion of the burden of language impairment identification – general education teachers. Both kindergarten teachers and SLPs positively responded to this question at 67% and 70% respectively. 88 Thirty-one percent of kindergarten teachers strongly agreed with this statement. Seventeen percent of SLPs negatively responded. Statement #7: Resources are limited in the district (e.g., personnel, time constraints, etc.) and therefore would make a yearly screening format for language impairment difficult. This question highlighted a ubiquitous component of public-school education – limited resources. Fifty-seven percent of kindergarten teachers concurred with this. Among SLPs, endorsement of this statement was quite robust at 89%. Close to a quarter of kindergarten teachers did not support this statement (23%) highlighting that SLPs feel the limit on resources more than kindergarten teachers do. Statement #8: Current caseload sizes of district speech-language pathologists would prohibit the district moving to a yearly screening format for language impairment identification. A slight majority of both groups (52% of kindergarten teachers and 54% of SLPs) concurred with this statement. Twenty-two percent of SLPs did not feel it would negatively impact them. The next four statements targeted the theme of who best to administer the screener in an annual format. Statement #9: Parents are a good choice to administer a language screening measure to students. Low support for parents as a choice to administer a selected screening protocol was robust across both groups with 73% of kindergarten teachers and 82% of SLPs responding accordingly. Strong disagreement was at 25% for kindergarten teachers and 28% for SLPs. Statement #10: District teaching assistants are a good choice to administer a language screening measure to students. The group that was responsible for the 89 screening administration in this project received respectable support from kindergarten teachers at 53%. This number dropped with respect to SLP respondents to 35%. More importantly, 45% of SLPs opposed teaching assistants as a good option. Statement #11: District speech-language technicians are a good choice to administer a language screening measure to students. This group received overwhelming support as the ideal choice to administer a selected screening protocol. Nearly all of the kindergarten teachers (97%) supported this choice with 47% of them strongly agreeing. Support among SLPs was also high at 75% with nearly a quarter (23%) strongly agreeing. Statement #12: District general education teachers are a good choice to administer a language screening measure to students. Support from kindergarten teachers with regard to themselves (and other general education teachers) as a practical option to administer regular screening for language impairment was surprisingly high at 46% which outpaced low support for this option at 33%. SLPs felt this option was less viable at only 35% support it and 44% indicated opposition. The next two statements looked at the barrier identified by focus groups of SLPs being considerably impacted by increased caseload sizes as a result of an annual screening format. Statement #13: Yearly screenings for language impairments could result in an increase in referrals for assessments by district speech-language pathologists. Response to intervention (RTI) programs represents an effective method for reducing this increase. A potential mediating factor that emerged from the focus group data was that a multitier system of support, or RTI approach, something which is already in place in DSD, would 90 reduce the impact on SLPs’ caseloads. This view was widely supported by both kindergarten teachers and SLPs at 77% and 73% respectively. Statement #14: (For SLPs only. GE teachers select "does not apply to me"): I would be willing to see an increase to my caseload due to screening for language impairment in order to better identify students with language impairment. Interestingly, a robust majority of 71% of SLPs indicated they would be willing to see an increase in their caseload in order to serve the population identified by regular screening. This supported the facilitative theme identified by the focus groups of concern and well-being of students impacted by language impairment. The next four questions addressed the barrier related to awareness and education of language impairment. Statement #15: I am aware of research reporting the short and long-term impact of unidentified language impairment in school-aged children. Less than half of the kindergarten teachers indicated their support for this statement (43%) with a large number of them reporting a neutral position (30%). This number jumped considerably when answered by SLPs with 82% of them supporting this statement. Thirty percent of them strongly agreed. Statement #16: I am confident in my ability to identify students with a language disorder using evidence-based assessments. Over half of the kindergarten teacher respondents (60%) positively responded to this statement, but only 10% of them strongly agreed. Responses from SLPs, however, were quite robust with 95% of them positively responding to the statement. Nearly half of the SLPs strongly agreed (46%). Statement #17: I am confident in my ability to help/treat students with a language 91 disorder using evidence-based interventions. Participant responses to this statement varied considerably. Forty-one percent of kindergarten teachers felt informed by current research on this topic, while 26% did not. These numbers more than doubled when answered by SLPs. Ninety-four percent of SLPs felt informed with only 3% reporting they did not. These data may highlight the lack of targeted training general education teachers receive with regard to language development during their formal education. Statement #18: I feel I received adequate training regarding language impairment in my formal education (e.g., undergraduate, master’s, or PhD program). Kindergarten teachers’ responses to this statement support the position that training and course work with respect to language development/disorders in formal education programs is lacking. Only 17% of kindergarten teachers reported they felt they received adequate training. This number nearly quadrupled with those reporting the opposite (63%). Nearly a quarter strongly disagreed (22%). While the results for SLPs were more encouraging, only 67% reported they received adequate training. Seventeen percent of SLPs reported they did not. Proportionally speaking, neutral responses from both groups were modest at 17% for kindergarten teachers and 16% for SLPs. Statement #19: Children with language impairment in DSD have been an underserved population as compared to other groups (e.g., Autism, speech/articulation, ADHD, etc.). Rather high instances of neutral responses existed between kindergarten teachers and SLPs at 35% and 26%, respectively. Forty-two percent of Kindergarten teachers supported this statement while, 23% did not. SLP responses indicated higher support for this statement at 60% with 22% of them strongly agreeing. Statement #20: I would support DSD in adopting a yearly screening format over 92 the current referral-based format for the identification of children with language impairment. The final statement in the series targeted the overall support directly from both groups should DSD decide to adopt an annual screening format. High majorities from both groups existed, with 90% of kindergarten teachers responding positively. Thirty-eight percent of them strongly agreed. A lower number of SLPs indicated support at 74% with 22% of them strongly agreeing. Seven percent of SLPs would not support this type of a format and, 19% of them indicated a neutral response. 93 Phase 1 (Aug. – Nov.) Phase 2 (Dec. – Feb.) Phase 3 (Mar. – April) RSR protocol training for DSD TAs Focus group #1 (TAs) Survey development and approval Consent forms distributed and collected Focus group #2 (Stakeholders) Survey distribution School site #1 screenings and confirmatory testing Focus group data analysis Survey data analysis School site #2 screenings and confirmatory testing Figure 6. Project Flow and Timeline Figure 7. Overlapping Focus Code Groups 94 Survey Statements Code Groups Supportive attitudes and willingness to change Statement 1 Statement 2 Statement 3 Availability of resources/time constraints Statement 4 Benefits to educators/district Statement 5 Statement 6 Concern for well-being of students Response to Intervention (RTI) need Statement 7 Statement 8 Statement 9 Understanding/education with regard to LI Use of existing policy/structure to facilitate annual screening Who best to administer screener Training on screener protocol Statement 10 Statement 11 Statement 12 Statement 13 Statement 14 Statement 15 Need for validated screening tool Opinions of screener and screening administration Screening equipment Statement 16 Statement 17 Statement 18 Statement 19 Statement 20 Figure 8. Mapping of Code Groups to Survey Statements 95 SLP RESPONSES strongly agree agree neutral strongly disagree disagree Q19 Q17 Q15 Q13 Q11 Q9 Q7 Q5 Q3 Q1 -100% -80% -60% -40% Figure 9. SLP Survey Responses -20% 0% 20% 40% 60% 80% 100% 96 KINDERGARTEN TEACHER RESPONSES strongly agree agree neutral strongly disagree disagree Q19 Q17 Q15 Q13 Q11 Q9 Q7 Q5 Q3 Q1 -100% -80% -60% -40% -20% 0% 20% 40% 60% Figure 10. General Education Kindergarten Teacher Survey Responses 80% 100% 97 COMBINED RESPONSES strongly agree agree neutral strongly disagree disagree Q19 Q17 Q15 Q13 Q11 Q9 Q7 Q5 Q3 Q1 -100% -80% -60% -40% -20% 0% Figure 11. Combined Group Survey Responses 20% 40% 60% 80% 100% 98 Table 4. Descriptive Statistics for Screening and Confirmatory Measures a Failed screening group (n=19) 13 male (68.42%) Passed screening group (n=146) 70 male (47.95%) Area tested Measure Subtests Morphosyntax/ Verbal memory RSRa (screening) N/A 70 (5.27)b 100.49 (10.56) Concepts & Following Directions Word structure Recalling Sentences Formulating Sentences 74.89 (12.31)b c N/A N/A 98.32 (11.51)b N/A a Language CELF-4 (confirmatory) Nonverbal intelligence NNATa (confirmatory) RSR = Redmond Sentence Recall Screener Protocol; CELF-4 = Clinical Evaluation of Language Fundamentals 4th edition. NNAT = Naglieri Nonverbal Ability. b Standard score: M = 100, SD = 15. c Core Language score 99 Table 5. Individual Failed Screening Results Student CELF-4b Findings Service Agea Sex RSRb NNATb e (#) DXc Statusd 1 6;2 F 67 72 90 SLI N 2 6;1 M 69 78 97 SLI N 3 6;0 F 63 50 78 NSLI Y 4 5;9 M 78 82 95 TD N 5 5;4 F 68 70 93 SLI Y 6 6;0 M 71 87 95 TD N 7 5;7 F 67 78 108 SLI N 8 6;0 M 75 78 86 SLI N 9 5;5 M 74 99 116 TD N 10 5;5 M 72 78 87 SLI N 11 5;10 F 67 79 100 SLI N 12 5;5 F 74 70 81 SLI Y 13 5;3 M 66 82 114 TD N 14 6;3 M 80 70 93 SLI N* 15 6;3 M 71 64 105 SLI N* 16 5;9 M 64 73 102 SLI N 17 5;3 M 76 88 116 TD Y 18 6;6 M 60 79 113 SLI N 19 5;5 M 68 46 99 SLI Y a b c years;months. Standard score: M = 100, SD = 15. SLI = specific language impairment; NSLI = non=specific language impairment; TD = typically developing. N = not receiving services at time of testing.Y = receiving services at time of testing. eCore Language score. *Student had previously been receiving services in preschool but had been released prior to starting kindergarten Table 6. Partial Diagnostic Accuracy Values True False Referral Source Positives Positive General Education 7 13 Teachera 9 1 Early Interventiona 11 3 Parenta RSR Screening Task 14 5 a Christopulos & Kean (2018) Positive Predictive Rate .35 .90 .79 .74 100 Table 7. Framework Analysis Chart Code Group Summary Facilitators Barriers Supportive attitudes and Supportive attitudes and willingness to change willingness to change Availability of resources/time constraints Benefits to educators/district Concern for well-being of students Screening equipment Screening equipment - pre-recorded stimuli - need for clipboards - simplicity of mp3 player - pencils wear out - headphones helps to focus - some words on recorded on stimuli stimuli hard to understand - old technology - timing between stimuli sentences difficult to control Need for validated screening tool Opinions of screener and Opinions of screener and screening administration screening administration - succinct to 1 page - print hard to read - easy to score online with - more space needed to little experience record responses - easy to pick up - some words used in - practice items helpful stimuli hard for students to - instructions informative comprehend - stand-alone training not - too many stimuli needed sentences - difficulty for students following recorded instructions - Complex for students Response to intervention (RTI) need Training on screener protocol - required - very thorough - informative - training with student recordings helpful - felt prepared afterwards Training on screener - forgot training by the time actual screening took place - training on adults hard Understanding/education with regard to LI Focus Group 1a x x x x x x x x x x x x x x x x Use of existing policy/structure to facilitate annual screening format x Who best to administer screener a Focus Group 2b b x x DSD Related Services teaching assistants. DSD Related Services administrators, SLPs, and general education kindergarten teache Table 8. Survey Results: Descriptive Data Survey Statement S1: Yearly screenings (e.g. hearing, vision, reading, etc.) are an effective way of identifying children with developmental difficulties. S2: Yearly screening, compared to a referral-based approach, is more effective for identifying school-aged children with language impairment. S3: A yearly screening format for language impairment identification would be wellreceived by district general education teachers. S4: A yearly screening format for language impairment identification would be wellreceived by district speech-language pathologists. S5: A yearly screening format would place an unmanageable burden on district speech-language pathologists. S6: A yearly screening format would reduce the burden on district general education teachers. S7: Resources are limited in the district (e.g. personnel, time constraints, etc.) and therefore would make a yearly screening format for language impairment difficult. Kindergarten Teacher Responses N A SA SD D N 43% 47% 1% 3% 8% 1% SLP Responses A Combined Responses SA SD D N 61% 36% 0% 0% 6% A SA SD D 51% 42% 1% 1% 18% 41% 30% 1% 11% 12% 55% 17% 1% 14% 15% 47% 24% 1% 13% 11% 47% 40% 1% 1% 23% 54% 17% 0% 6% 17% 50% 30% 1% 3% 36% 38% 22% 0% 5% 30% 41% 9% 1% 19% 34% 39% 16% 1% 11% 40% 27% 7% 1% 25% 35% 32% 12% 0% 22% 38% 29% 9% 1% 24% 26% 36% 31% 0% 7% 13% 58% 12% 1% 16% 20% 46% 22% 1% 11% 20% 42% 15% 1% 22% 4% 0% 7% 1% 15% 64% 25% 13% 52% 19% 101 Table 8 continued Survey Statement S8: Current caseload sizes of district speech-language pathologists would prohibit the district moving to a yearly screening format for language impairment identification. S9: Parents are a good choice to administer a language screening measure to students. S10: District teaching assistants are a good choice to administer a language screening measure to students. S11: District speech-language technicians are a good choice to administer a language screening measure to students. S12: District general education teachers are a good choice to administer a language screening measure to students. S13: Yearly screenings for language impairments could result in an increase in referrals for assessments by district speech-language pathologists. Response to intervention (RTI) programs represent an effective method for reducing this increase. Kindergarten Teacher Responses N A SA SD D 33% 39% 13% 1% 17% SLP Responses N SA SD 15% 25% 35% 19% 0% 7% 0% 25% 48% 16% 20% 45% 8% 50% 47% A Combined Responses D N SA SD D 22% 29% 37% 15% 1% 18% 3% 0% 28% 54% 17% 5% 20% 20% 28% 7% 0% 0% 12% 52% 23% 7% 9% 24% 22% 28% 15% 64% 13% 1% 7% 3% 19% 39% 7% 13% 54% 19% A 5% 0% 26% 50% 7% 38% 20% 38% 8% 6% 28% 1% 12% 51% 36% 1% 5% 7% 12% 32% 20% 34% 1% 7% 13% 14% 59% 15% 10% 27% 1% 10% 102 Table 8 continued Kindergarten Teacher Responses Survey Statement S14: (For SLPs only. GE teachers select "does not apply to me"): I would be willing to see an increase to my caseload due to screening for language impairment in order to better identify students with language impairment. S15: I am aware of research reporting the short and long-term impact of unidentified language impairment in school-aged children. S16: I am confident in my ability to identify students with a language disorder using evidence-based assessments. S17: I am confident in my ability to help/treat students with a language disorder using evidence-based interventions. S18: I feel I received adequate training regarding language impairment in my formal education (e.g. undergraduate, master’s, or PhD program). S19: Children with language impairment in DSD have been an underserved population as compared to other groups (e.g. Autism, speech/articulation, ADHD, etc.) S20: I would support DSD in adopting a yearly screening format over the current referral-based format for the identification of children with language impairment. SLP Responses N A SA SD D 1% 0% 0% 0% 0% 30% 38% 5% N A Combined Responses SA SD D N A SA SD D 12% 61% 10% 3% 14% 6% 27% 4% 1% 6% 1% 26% 10% 52% 30% 0% 7% 21% 44% 16% 1% 18% 16% 50% 10% 2% 19% 3% 49% 46% 0% 1% 10% 50% 26% 1% 11% 30% 36% 5% 3% 23% 3% 52% 42% 0% 3% 18% 43% 21% 2% 14% 17% 14% 3% 22% 41% 16% 39% 28% 3% 14% 17% 25% 14% 13% 29% 35% 32% 10% 1% 22% 26% 38% 22% 1% 13% 31% 34% 15% 1% 18% 10% 52% 38% 0% 0% 1% 6% 1% 3% 19% 52% 22% 14% 52% 31% N = Neutral. A = Agree. SA = Strongly Agree. SD = Strongly Disagree. D = Disagree 103 CHAPTER 6 DISCUSSION This study used a hybrid implementation approach to examine potential barriers and/or facilitators to the systematic uptake of universal screening in the context of one school district in Utah (i.e., Davis School District – DSD). This particular school district, like many school districts in the United States, has historically relied on a referral-based format for identifying cases of language impairment. This type of format relies on referral sources, such as general education teachers, to shoulder the primary responsibility of implementing their IDEA child-find obligations. The preliminary analysis that served as the basis for this study demonstrated that teacher referrals in DSD were suboptimal in correctly referring cases of language impairment. Given the prodigious amount of burden placed on general education teachers, coupled with the limited training they receive pertaining to language development and disorders during their formal education, it may simply be unrealistic to burden this group of valued and underrecognized professionals with the complex task of identifying children at risk for language impairment. The results of this study indicate a better option would be universal screenings for language impairments, similar to those routinely used to identify vision, hearing, and reading deficits in students. However, adopting an annual screening format in a particular school district requires more than just applying empirically vetted screening measures. It 105 requires consideration of factors, some particular to a given school district and others common across school districts, that have not yet been considered in previous research. A hybrid implementation design represents a particularly apt method for addressing these factors. Specifically, this type of design aims to considerably reduce the amount of time it takes to realize the systematic uptake of research findings into clinical practice by concurrently considering facilitators and barriers specific to DSD with regard to the uptake of an annual screening format for language impairment identification, while at the same time, assessing the validity and reliability of a particular screening measure. This study utilized a multimethod approach that relied on the use of focus group and survey formats to pin-point facilitators and barriers to DSD. It also assessed the usefulness of the Redmond Sentence Recall (RSR) measure for use by DSD as a valid and reliable screening tool. The particular screening measure used in this study was selected for several reasons. First, it targets specific phenotypic markers identified as areas of weakness for children with SLI such as morphosyntax and verbal working memory. Second, it is a relatively quick measure to administer with minimal training requirements, making it appealing for use in school districts. Third, the RSR measure has been shown in past studies to have adequate levels of sensitivity when identifying early elementary school-age children with language impairment (see Redmond, Ash, Christopulos, & Pfaff, 2019). Overall, the results of this study suggest that adopting sentence recall as a universal screener for language impairments is feasible given the resources available to the targeted school district. Key barriers and facilitators were identified through the collection of focus group and survey data. Although the results of this study may be 106 considered limited to the context of the particular school district targeted, the process of arriving at what other school districts might need can be generalized through the exploration of individual districts’ factors. Six specific areas will be covered in this chapter. 1) Review of the prominent findings associated with each individual aim. Aims 1 and 2 will be reviewed individually and aims 3 and 4 jointly. 2) Qualitative research impact review of the study. 3) Implementation impact review of the study. 4) Practical and clinical implications. 5). Limitations and future directions. 6) Conclusion and presentation of key summary findings in bullet-point format. Review of Findings Aim 1: Research Question 1 Does the RSR screener protocol identify language impairments at a higher rate than that associated with the current system in DSD utilizing general education teacher referrals when using a standard score cutoff of <80 on the CELF-4? Specifically, does it provide an occurrence rate that is more consistent with prevalence expectations based on epidemiological reports? This aim looked to add to the growing body of research regarding the identification of cases of language impairment with unknown etiology. The preliminary study data that anchor this project were a record review of all referrals to special education for language impairment across four elementary school sites that utilized a referral-based identification format (Christopulos & Kean, 2018). Data from that study showed an estimated rate of successful referrals at 1.38%. This project, utilizing a 107 screening-based format, produced a higher rate at almost six times of that produced under a referral-based format. More importantly, this rate is consistent with frequency expectations based on previous prevalence studies. Table 9 summarizes the prevalence estimates of the most widely cited epidemiological studies pertaining to language impairment in kindergarten-aged children compared to the results of this study. It is noted the comparison reports were large-scale population studies with sample sizes in the thousands compared to this study sample of 182. This study, mirroring previous studies, suggests that roughly two children in every kindergarten classroom will have a clinically significant idiopathic language disorder that pejoratively impacts academic learning, social elements, and behavior. Screenings for this project were conducted in relatively affluent areas in DSD. Accordingly, the reported occurrence estimate should be considered the minimum estimate of need and may be elevated in lower SES communities (Hoff & Tian 2005). While this study attempted, through representative sampling, to be as valid as possible, two cautionary points are important to highlight here. First, the absence of a false negative rate compromises estimates of occurrence rate. True positives combined with false negatives divided by the total screened sample size produces a more precise estimate of occurrence rate. This limitation, coupled with the expectations that the screening measure would have missed some of the true cases (i.e., false negatives), suggests a truer occurrence rate would be underestimated. Second, standard cutoff scores used for this project were set at 80. The three comparison epidemiological studies varied in cutoff scores used for confirmatory language testing. Accordingly, occurrence findings across this, and the comparison studies should be interpreted with this in mind. The cut 108 point of 80 was selected for this study as it represents the most frequently used criteria by DSD SLPs to qualify students for special education services. Aim 1: Research Question 2 What is the positive predictive rate of the RSR screener protocol when used by personnel in DSD? A barrier identified by focus group participants was the critical need for a screening measure that is both valid and reliable at identifying early elementary schoolage children with language impairment. Research efforts to distinguish clinical markers of developmental language disorders have suggested sentence recall to be a leading candidate, among others. Several studies have looked at the particular screening measure used for this project (i.e., RSR sentence recall task) as a potential tool for school districts to utilize under an annual screening format for language impairment identification (see Archibald & Joanisse, 2009; Redmond, 2007; Redmond, Ash, Christopulos, & Pfaff, 2019). This study looked to replicate positive predictive values from those studies using a different community-based study sample. This study’s number of screened participants was considerably higher for this age group than the referenced studies making it particularly noteworthy. As shown in Table 10, PPV results provide important replications of both Redmond (2007) and Archibald and Joanisee (2009) that have used the RSR screening measure as the primary screening tool. These independent replication studies continue to add to a growing body of literature that proposes the phenotypic markers of verbal working memory and morphosyntax, as targeted in sentence recall measures, as valid and reliable suggesting school districts can use such measures (e.g., 109 RSR screening protocol), as clinical screening tools for language impairment identification. With regard to reliable reference standards, recently Redmond et al. (2019) demonstrated the use of the CELF-4 to be a stable enough measure school districts could utilize to confirm language screening results. Aim 2: Research Question 3 What barriers and facilitators are reported by Davis School District (DSD) project team members based on their experience in the facilitation and administration of the RSR screener protocol as articulated in focus group settings? Aim 2: Research Question 4 Do DSD speech and language pathologists and general education kindergarten teachers agree/disagree with the barriers and facilitators identified by DSD project team members as indicated by their responses to survey statements? General findings Table 11 provides a concise general description of each theme/code group identified by the focus groups. Key findings Experienced professional burden.The barrier of burden experienced by both SLPs and general education teachers was central across many of the identified themes across both focus groups and was widely supported in participant survey responses. For 110 example, it played a role in the survey statements that asked about the representativeness of how experienced burden may underpin one’s attitude and willingness to change, an objective assessment of resource/time constraints, to what extent the need of the child can drive implementation decisions, to what extent programs such as RTI can reasonably play a mitigating role, and how much time is realistically available to seek outside sources to increase understanding/education relating to language impairments. The identification of widespread burden supports previous findings that have shown variables such as workload, retention, and caseload size as being predictors of job satisfaction that can negatively impact best-practice implementation by school-based SLPs (Hutchins, Howard, Prelock, & Belin, 2010). Studies looking at general education teacher burden impact on evidence-based implementation have also reported similar findings (e.g., Higton et al., 2017). The pervasiveness of this barrier emphasizes the importance of viewing the implementation process of annual screening for language impairment through the lens of burden currently and potentially experienced by general education teachers and SLPs. Role of screening in public-school settings. Survey participants widely supported the positive role current screenings play in public schools (i.e., vision, hearing, and reading) and felt the impact of screening for language impairment would be as constructive. Two important points of impact require mentioning. First, district educators and administrators were identified as being beneficiaries of annual screening implementation. District educators would benefit as a result of the burden of identification being redistributed to other professional positions. Administrators would benefit by better accomplishing their mandate to identify children with disabilities under 111 the Individuals with Disabilities Education Act (IDEA) more effectively and efficiently. In addition, district officials may also indirectly benefit from the reduced burden placed on teachers by seeing lower attrition rates and increased overall job satisfaction. Second, general education teachers surveyed in this study reported personal limitations with regard to their own ability to accurately identify children with language impairment for referral because of little to no training in the areas of language development and disorders during their formal education. The elevation of language impairment identification from a referral- to annual-based format in and of itself may help increase awareness to the area of language disorders. As a result, teachers may have increased self-motivation to learn about key aspects of this field. General increased awareness may lead to school and district level training for educators. The benefits of increased awareness of a particular topic have been experienced in the field of dyslexia, resulting in improved resources and legislation. For example, Youman and Mather (2015) found that as of 2015, 28 states reported statewide dyslexia-awareness laws. Six had resolutions or initiatives relating to dyslexia, and 14 states had some type of resource guide (e.g., handbooks) informing about dyslexia. These laws were implemented to inform educators about proper procedures for students in public education settings. Specifically, these laws placed a high emphasis on dyslexia awareness, pilot programs for screening and interventions, and teacher training. Utah is among the states that legally mandate screening for dyslexia in public schools. Accordingly, important parallels should be drawn between the two fields of reading and language development as they relate to awareness on a school, district, and state level. 112 Role of an RTI format as an important component of language-based screenings. Survey participants acknowledged the reality of increased burden placed on district SLPs as a result of caseload increases due to the implementation of an annual screening format for language impairment identification. A mediating factor to this, however, was the utilization of an RTI (or MTSS) format, which has been implemented recently in DSD. Regarding general education teachers, it is essential that care be taken not to replace one burden (i.e., responsibility of referral identifications) with another (i.e., an increase in classroom-based tier 2 and tier 3 RTI interventions as a result of increased identifications by screenings). While teachers may be able to contribute marginally to an RTI format, this responsibility of RTI implementation should fall on another group of professional educators such as speech-language technicians or teaching assistants. Administration of an RTI format should be managed at a district level as opposed to an individual school level. As Freeman, Miller, and Newcomer (2015) pointed out, key stakeholders in public education believe district leadership and involvement is a vital factor for effective RTI implementation with any targeted area. They emphasize that district leadership with any RTI format plays a key role in providing schools with administrative and political support, training, layered in-service programs, data-driven decision making for ongoing evaluation, and access to interagency relationships for improving student outcomes. Groups best suited to conduct screenings. The identification of a group responsible for routine administration of annual screenings was particularly important to focus group members and supported by survey participants. The groups identified as potential candidates based on the feedback of the focus groups were SLPs, TAs, SLTs, general education teachers, and parents. Parents were widely identified as the least 113 qualified option. This may highlight an experiential component a particular group should have in order to effectively participate in the identification of children with potential language impairment. Although parents have been identified in the research literature as valid for identifying language developmental concerns (a point that is emphasized by the wide use of parent checklists for diagnostic purposes) parents typically are appraising their own children and not others (Bishop & McDonald, 2009; Christopulos & Kean, 2018; Norbury, Nash, Baird, & Bishop, 2004; Redmond et al., 2019). Consequently, parent accuracy seems to be directly connected to the familiar and observational nature of a parent-child relationship. As a result, it may be problematic to assume parents would be able to show high levels of identification sensitivity to children not their own. Due to the barrier of burden, confirmed by survey participants, the SLP and general education teacher groups would not be an optimal choice due to current caseload and workload demands. The TA and SLT groups were identified as the best fit for use in an annual screening format. It could be argued these two groups stand as compelling options for districts. Particularly, the SLT group presents as an intriguing choice due in large part to a type of “Goldilocks” effect they offer that may maximize an annualscreening format. On the one hand, parents represent a group that may be underqualified to conduct routine language screenings due to limited training and restricted high-level exposure to children with language impairment on a regular basis. On the other hand, SLPs and general education teachers represent a group that may be overqualified to regularly administer language screenings due to the fast and routine nature of screenings in general. SLTs are able to provide what parents lack that underqualifies them. At the same time, however, SLTs lack the specific educational background and expertise that 114 accordingly overqualifies SLPs and general education teachers. SLTs lack the specific skills compulsory for SLPs and general education teachers to assess and develop treatment plans required as part of the official Individual Education Plans (IEP) qualification and implementation process. While SLPs and general education teachers are obviously qualified to conduct screenings, these two groups often carry caseloads/workloads that are unmanageable. Specifically, with regard to SLPs, the American Speech-Language-Hearing Association reported the national average caseload size of SLPs in 2016 at 48 (ASHA, 2016). Responses to this study’s survey statement inquiring about current caseload size revealed 84% of DSD SLPs have a caseload size higher than the national average with a range of 50-69. Fourteen percent of those reported sizes in the range of 70-89, and 3% greater than 90. Nationwide SLP shortages were brought to the forefront of research discussion during the mid to late 2000s (Boswell, 2006; Boswell, 2007; Moore-Brown, Nishida, Uranga‐Hernandez, Parker, & Shubin, 2005; Polovoy, 2008). These studies often point to a connection between high caseload/workload demands and high attrition rates. By extension, fewer individuals are entering the professional field of communication sciences and disorders. Accordingly, the increased use of reinforcement contingencies, such as SLTs, has been suggested as important to help augment assessment and treatment requirements of public-school IEPs (Petursdottir & Mellor, 2017). In addition, as a natural response to increased use of SLTs, SLPs may feel threatened and undervalued as professionals. Utilization of SLTs as the primary group to lead screenings for language impairment under an annual-based format may represent an ideal or “just right” scenario that places them in a position of support and assistance to SLPs, rather than in direct competition. 115 Education/awareness. There were conflicting results with regard to the degree general education teachers assessed their own ability to identify and teach children with language impairment. Sixty percent of teachers reported feeling confident in their ability to identify children with language impairment for referral to special education, while only 17% stated they felt they had received adequate training during their formal education. The low PPR rate of general education teachers demonstrated in Christopulos and Kean (2018), however, highlights a disconnect between their perceived and actual ability. This was highlighted by the false positive rate of .65, demonstrating that even when general education teachers made referrals, no clinical language impairment often existed. When asked about their confidence level to help/treat students in their classroom with language impairment, responses were more tempered at 41%. It is noted that a high number of neutral responses (30%) were reported with this statement indicating possibly some level of dissonance between what teachers feel their ability level should be as compared to actual abilities. For example, Higton et al. (2017) hinted many general education teachers feel a large sense of altruistic responsibility to be highly proficient at assessment and treatment of most developmental disorders they encounter in their classroom regardless of training received. This may be compounded by the empathetic and caring characteristics teachers often exhibit in their work (O’Connor, 2008). To complicate things further, a minority of teachers (42%), with a robust neutral response rate of 35%, reported they felt children with language impairment have been an underserved population as compared to other groups (e.g., Autism, speech/articulation, ADHD, etc.). In other words, if students with language impairment are not even on the radar of disorders teachers feel they should be looking for, it may be unrealistic to expect this 116 group to be a reliable source to anchor a referral-based identification format. Regardless, the lack of formal education teachers receive in language development and disorders simply does not provide them with a stable enough foundation to reliably identify children who are not developing at age-appropriate linguistic levels. Focus group participants reported no systematic training or public awareness programs provided to nonspecialized district positions relating to language development or impairment emphasizing a pattern of anemic education and awareness experienced by general education teachers. Qualitative Research Impact Qualitative methods encourage certain means of asking questions and thinking through problems. As a unique, and ultimately effective, approach for this project, qualitative analysis allowed for the exploration of facilitators and barriers from the perspective of primary stakeholders regarding the implementation of an annual screening format for the identification of language impairment in early school-aged children. Specifically, through the use of focus groups and utilizing the known theoretical literature available through the fields of communication sciences and disorders and implementation research, the principal investigator was able to better understand the perspectives and experiences of screener administrators (i.e., TAs), SLPs, general education teachers, and district administrators. More importantly, the use of a qualitative approach permitted the stakeholders to share thorough, complex, and personal descriptions regarding this topic. As a result, this method generated an extensive and rich data set that shed light on themes not easily understood or extracted as a result of an exclusively quantitative methods 117 approach. The focus on stakeholders’ experiences and perspectives allowed them to share features regarding the administration, planning, and strategizing regarding an annual screening format they deemed most important and personal. This included features they valued and understood (i.e., facilitators), features they perceived as difficult to understand or easy to overlook (i.e., barriers), and features identified as a direct result of collaboration with each other regarding a topic not frequently explored or discussed in their work environment. Accordingly, the focus group data served as a strong foundation to explore and assess the identified themes and barriers on a wider scale through the use of a survey. A key point that contributed prominently to the success of this project was the enthusiastic willingness from all of the members to accept invitations to discuss this topic in a focus group format openly. Anecdotal reporting from the majority of members expressed appreciation for the opportunity to contribute and for the willingness of the district to participate in the study. Many reported the topic of screening for language impairment as long overdue. The high interest in participation was likely a result of DSD employees feeling appreciated and valued in their respective positions while at the same time feeling comfortable to express opinions regarding controversial or unpopular pointsof-view that district officials may not necessarily endorse. Regarding the implementation of focus group meetings, Barbour (2008) pointed out the mediation of such settings can present difficulties when participants do not engage members/moderator or monopolize the group discussion. Focus group participants from this study were respectful of other members’ positions, listened intently, and fostered productive back-and-forth exchanges that contributed to a rich data 118 set. Implementation Research Impact The transition of discovery to application represents a “full-return” on any particular research investment ranging from time individuals spent in the execution of a study, to monetary investments, to the direct benefit of a particular population being studied. This important role of “transition” is often a low priority of researchers following the completion of research projects. If findings are able to be worked into mainstream clinical practice, the time it takes to do so can be quite lengthy – on average, 15 years (Brownson et al., 2018). The use of innovated implementation research methods, such as hybrid approaches, looks to drastically cut down on the amount of time it takes to realize the systematic uptake of research findings into clinical practice. This project utilized a hybrid implementation method by concurrently assessing facilitators and barriers specific to DSD with regard to the uptake of an annual screening format for language impairment identification, while at the same time assessing the validity and reliability of the RSR screening protocol. It could easily be argued these two distinct and separate aims, when studied individually, would take considerably more time to execute. Aspects such as university IRB approvals, internal district approvals, participant drift, and overall enthusiasm for research-based projects by public-school actors and stakeholders would need to be dealt with in duplicate thus lengthening the planning and execution phases considerably. Thoughtfulness regarding DSD employees’ time was paramount for this project. As a result, the use of a hybrid approach for this project contributed substantially to expedited and efficient project planning and data collection. 119 This study also sought to engage in practice-based research. As Gagliardi et al. (2014) aptly reported, collaboration among academics and stakeholders (i.e., administrators, policy makers, clinicians, etc.) improves the relevance and practice of research thus leading to better policy making, practice, and education outcomes for students, educators, and parents. Practice-based research endeavors to draw on the strengths of the unique parties involved in the development and execution of a particular project. For example, researchers bring methodological and theoretical knowledge to projects vital for evidence-based outcomes. Stakeholders, on the other hand, bring a wealth of experience and knowledge regarding setting, population, and practice context (Cunningham, Daub, & Cardy, 2019). As a result, both sides bring equally unique positions, values, and points-of-view to a study. Kothari and Wathen (2017) reported that when “in-the-trenches” professionals’ capabilities are integrated, evidence-based outcomes stand to be more relevant and impactful to clinical practice. Support and encouragement were widespread from DSD in participation with this study, particularly at the individual school site and district administrator level. This was especially encouraging as these groups are ultimately responsible for any decisions made on students’, educators’, and parents’ behalf to implement policy change regarding how language impairment is identified within their own district. Additionally, anecdotal reporting from participants clearly stated a position of appreciation for their opinions and viewpoints being valued to the point of inclusion in a research study by academics who are often perceived as condescending and out of touch with the everyday clinician or educator. The facilitators and barriers identified in this project are relevant, specifically to 120 DSD. They also stand to be useful for participants of the broader research community in the fields of communication sciences and disorders as well as implementation research. Those who have an interest in language impairment identification and how to best approach this in a public-school setting may find the results of this aim helpful to inform the development of practice- or implementation-based studies in clinical settings. It is not the intent of this project to assess if the identified facilitators and barriers are generalizable outside of the DSD environment. Further implementation work still needs to be conducted regarding this topic within DSD. Utilization of the QUERI structure requires multiple steps to understand access to systematic uptake in a particular setting fully. Others, however, can build on these findings by appraising the extent to which the facilitators and barriers identified in this study may be present in other settings or contexts. By doing so, public schools can begin to develop and implement individualized policies that better serve their students affected by language impairments. Practical and Clinical Implications Findings from this study contribute to best practices with the identification of children at risk for language impairments. Five specific points will be reviewed in this section. 1) Prevalence support for justification of an alternative identification approach to the prevalently utilized referral-based system. 2) Availability of valid and reliable measures to special education professionals that can be used for language impairment identification. 3) The pivotal role RTI formats play in the direct preventing of potential increased caseload sizes to SLPs as a result of transitioning to an annual screening format. 4) Identification of a professional group best suited to regularly administer 121 screenings across all schools in DSD. Points 3 and 4 will be reviewed jointly. 5) The importance of increased awareness through education and training to public educators directly impacted by children with language disabilities. Prevalence Support for Increased Identification Efforts The justification used by local, state, or federal organizations to dedicate substantial resources to the identification and treatment of language impairment is ultimately rooted in the size of a disordered population. This, coupled with the clear results of many outcome-based studies reporting on the negative academic impact of children with language disorders, should play a motivating role in convincing policy makers to increase identification efforts in the public-school setting. This study identified that at minimum, 7.88% of students screened were confirmed to be performing below expected language levels for early school-aged students in DSD. As a result, these impairments may inhibit this group from fully engaging in and progressing through the general curriculum. Another way of looking at this is from the standpoint of the current referral-based format of identification. Under this format, general education teachers may deal with, on average, two students in every classroom they are not aware of as having language deficits. As a result, teachers are at a disadvantage when attempting to fully connect with these students, thus leaving their students vulnerable to academic and social difficulties. When compared with prevalence rates for Autism (equivalent to approximately two students in every grade) coupled with the amount of funding and awareness sustained by local, state, and federal sources, the prevalence of the population with language disorders warrants increased efforts of identification. 122 Availability of Effective Measures School-based clinicians have been hindered in the past with subpar measures available to screen for and confirm the presence of language impairment. Significant progress has been made in the identification of stable clinical markers of language impairment in early school-aged children in the areas of morphosyntax and verbal working memory, among others. Screening measures targeting these areas with the use of sentence recall tasks have shown to be valid and reliable. Additionally, the CELF-4 has been shown to be useful as a confirmatory measure for language impairment identification in early-school age children. Data from this study add to the growing body of research looking at these issues by replicating the use of sentence recall, specifically the RSR measure, and the CELF-4 for confirmatory testing. The RSR measure is particularly appealing for supportive professionals or practicing clinicians. For example, the TAs reported in the focus groups that training for the RSR protocol was streamlined, administration was fast, and the task was easy to pick up on and understand. These features are particularly imperative to professionals who are tasked with the identification of children with developmental disabilities as workload and caseload burdens are prodigious. The Impact of RTI on Burden Reduction and SLTs or TAs as an Initial Good-Fit for Regular Screening Administration The identification of RTI as a facilitative mediator to the barrier of increased burden on SLPs as a result of the implementation of an annual screening format has particular clinical significance. This, however, represents half of the equation for the 123 reduction of burden. Under the current referral-based system, general education teachers assume a large burden of responsibility for the identification and referral of students with possible language impairment to SLPs for assessment. The move to an annual screening format would merely transfer the burden to SLPs, at a more-than-likely increased intensity level compared to what general education teachers experience, primarily because the number of children identified under this type of format would yield increased “hits.” Utilization of an RTI format by professionals other than general education teachers and SLPs is key to reducing burden on both teachers and SLPs. As school districts grapple with the ubiquitous problem of larger than recommended caseloads carried by SLPs coupled with high assessment and meeting demands, the utilization of technician and/or assistant positions to absorb the majority of the responsibility of guiding children who have failed language screening through the middle and upper tiers of an RTI format stands to directly reduce the burden of the aforementioned groups. At the same time, it also carves out needed professional space for these skilled groups that is not in direct competition with SLPs. The Need for Increased Efforts to Bolster Awareness and Education of Language Impairment This study’s documentation of general education teachers’ difficulty with identifying children with language impairments as a result of lack of awareness and education in the areas of language development and disorders should stand as a call for public education decision-makers and policy administrators to make available access to educational sources on a wider scale. This is particularly important if school districts 124 decide not to adopt an annual screening format and continue to rely on a referral-based system for language impairment identification, or in other words – the continued reliance on general education teachers to be the primary identification source. Regardless of formats, increased efforts to bolster awareness of short- and long-term academic and social impact of unidentified language impairment to general education teachers should be of high priority especially given the prevalence of the disorder in school-aged children. General education teachers will always have a high rate of involvement with this population regardless of how they are identified simply as a function of their professional role. Recently, Adolf and Hogan (2019) proposed that schools should include educator training programs as an important step that would play a central role in direct measurement of children’s language abilities in school assessment frameworks. They emphasize the implementation of any such training programs would need a clear directive from school administrators with regard to the depth and breadth of coursework because history has shown such training programs tend to take on the views of the faculty, regardless of scientific backing (DeMonte, 2013; Rickenbrode & Walsh, 2013). As such, general education teachers play a prominent role in bridging the typically developing and impaired populations with which they work. Intensified efforts to continually help them strive to stay current and up-to-date regarding evidence-based research in language development and impairment should be of high priority to those in administrative positions. 125 Limitations and Future Directions This study has several limitations that must be taken into consideration when interpreting its results. The following section will outline these limitations within each of the two aims of the project. Aim #1: Screening and Confirmatory Testing The DSD placed limits on the scope of this project. For example, confirmatory testing of students who passed the RSR screening task was not permitted. Apart from this, due to limited resources and time, it would have been difficult to do so with only the principal investigator executing all phases of a study that was already substantially broad in scope for a project of this nature. By not doing so, however, important data regarding false and true negatives are absent. Had these data been provided, informative diagnostic values such as sensitivity/specificity, and positive/negative likelihood ratios could have been calculated thus providing a clearer picture of the effectiveness of the RSR screening measure, identification rates, and the nature of the language-impaired population in the kindergarten grades in DSD. It is noted, however, that the comparison studies used for this portion of the project (i.e., Archibald & Joanise, 2009; Redmond, 2007; Redmond et al., 2019) were able to provide data for these diagnostic values. It would be likely that the values from this study would be in line with these reports given the similarities in this project’s true and false positive rates. This study utilized an “opt-in” format to recruit participants for screening. With participation rates high at over 90%, there still was a small portion of the target population that was not represented in this project’s data set. It is noted however this 126 issue would be reduced should DSD decide to adopt an annual screening format because district decision and policy makers would more than likely utilize an “opt-out” format as it does with hearing, vision, reading, etc. screenings. A follow-up study, however, would be advisable to confirm this expectation. Another potential limitation was the use of the CELF-4 as the reference standard for language impairment. While the use of this measure for confirmatory purposes of language impairment screening is justified in the research literature (see Redmond et al., 2019 for the most recent evidence), it was not exhaustive of the range of clinical or empirical options available. The CELF, however, is the most widely used standardized test by school-based SLPS (e.g., Betz, Eicjhoff, & Sullivan, 2013; Frinestack & Satterlund, 2018). Future research may reveal a disconnect with screening measures that target sentence recall, past tense probes, and nonverbal working memory tasks and other standardized measures frequently used in public schools. Future studies should look more closely at standardized measures frequently used by DSD SLPs such as the Test of Language Development (TOLD), Oral and Written Language Scales (OWLS), and the Clinical Evaluation of Language Fundamentals – Fifth Edition (CELF-5). The results of this study are limited to kindergarten-aged school children and caution is noted in an attempt to extend findings from this study to older children. The RSR sentence recall protocol was normed on kindergarten through third grade children and therefore can be assumed valid and reliable for use on these grades. It is still yet to be determined, however, how well it would perform on older elementary and secondary school-aged children. Future extensions of this study may look to include these older populations. 127 A barrier that was highlighted in the focus group was the importance of a screening measure that can be used across diverse populations. Outcomes from this study apply primarily to monolingual English-speaking children. Studies looking at sentence recall as valid and reliable on children from nonmainstream dialects and communities have been in limited number, but their outcomes support their use in these groups (Oetting, McDonald, Seidel, & Hegarty, 2016). Aim #2: Focus Groups and Survey Use of focus groups has, at times, been criticized as being a quick, easy, and suspicious method for data collection and analysis that does not link clearly to more rigorous quantitative methods (e.g., Kamberelis & Dimitriadis, 2013; Merton, 1987). While this position is strongly opposed by the principal investigator of this study, an objective analysis of limitations warrants mention. Qualitative methodology plays a meaningful role in research and challenges certain quantitative (i.e., positivist) ontological and epistemological positions. While it is beyond the scope of this project to explore these positions with regard to qualitative and quantitative methodology, it is important to highlight one of the basic principles that starkly distinguishes these two types of methodological approaches, namely, the relationship between the researcher and researched. A positivist approach puts the researcher and researched on distinctive planes throughout the entire development and implementation of a research project. In other words, they are dichotomous, and a rigid division must always exist between the two groups. A qualitative approach is distinct in that the researcher and research are placed on equal, or bidirectional planes. It specifically acknowledges that a researcher’s 128 philosophical foundation (i.e., epistemology) impacts every aspect of the research process. This includes, but is not limited to, what research questions are asked, how they are asked, whom they are asked to, and what framework will be most useful to carry out a project. This project’s success was uniquely based on the philosophy that barriers and facilitators to the systematic implementation of an annual-based screening format for language impairment are rooted in the distinct experiences of DSD TAs, SLPs, general education teachers, and district administrators. It is the progress of knowledge-building that allowed the principal investigator of this study to be observant to all parts of the research process, including the construct of the study, the interconnections between each stage of the research process, and the impact the principal investigator had on the process (Hesse-Biber, & Leavy, 2010). Accordingly, an exclusively quantitative approach simply would not have allowed for collection and analysis of this type of data. An alternative method of identifying facilitators and barriers to an annual screening format for language impairment identification in DSD to that of focus groups was the use of semistructured interviews. The latter potentially could have yielded a deeper data set regarding these themes because guiding questions could have been personalized more to the interviewee due to the one-on-one nature of interviews. Focus group guiding questions for this study had to be written with a broader participant audience in mind. The particularly limiting nature, however, of both of these approaches (focus groups and semistructured interviews) lies in the ultimate small sample sizes. It was felt that the use of focus groups best combated this issue. The project conducted two focus groups with six participants, each totaling 12 members. Use of a multifocus group format, as opposed to a single-focus group, allowed for heightened perspectives from 129 multiple professional positions in the district (i.e., TAs SLPs, kindergarten teachers, and Related Services administrators). Given the limited time and resources for this project, twelve in-depth semistructured interviews with the corresponding transcriptions and data analyses would have been untenable. The sample of focus group participants was homogenous. For example, all members were female and relatively of the same age group. Male members and variability in age groups most likely would have yielded differing viewpoints with regard to screening administration, or background in assessing, identifying, and treating children with language impairment. Future studies utilizing focus groups should seek to include participants from a broader gender and age background in order to create a more diverse and generalizable data set. Future research would greatly benefit from more varied experiences and opinions across different professional segments of public education. To build on the data generated from the small sizes of the focus groups, use of a multimethod approach by means of survey data to confirm focus group findings proved beneficial. This was a cost-effective and time-saving way of assessing the opinions and to a large extent, the experiences of DSD SLPs and kindergarten teachers. There potentially would have been fewer sampling issues and bias had full survey participation been achieved. It is possible the SLPs and kindergarten teachers who did not participate in the survey would have produced differing results than those who completed the survey. Given the high participation rate, however, it is believed there would not be a significant impact on the findings. Use of a forced-choice answer format as opposed to open-ended survey statements may have limited additional discovery of barriers and facilitators not 130 otherwise realized by the focus groups. It is possible that in designing the survey, certain facilitators and barriers identified by the focus groups were not given their appropriate weight. To circumvent this limitation, the vast majority of themes identified by the focus groups were mapped multiple times onto survey statements (See Table 8). As was the case with the focus groups, racial and gender homogeneity existed among the survey participants, but this is quite representational of DSD’s professional demographics. Moreover, it is representational of racial and gender demographics nationally for both fields of speech-language pathology and elementary education (Speech-Language Pathology: Data USA, 2016). Future QUERI Stages of This Study This project implemented the first phase of the QUERI model framework. It looked to execute a pilot project to develop an implementation program and assess the basic feasibility of universal screening in DSD. At its core, QUERI phase 1 projects are to be performed at one or two school sites over the course of a single school year. It is meant to be low-cost and generate an informative list of “lessons learned” for use in a future phase 2 project. A phase 2 extension of this particular project would still operate under a hybrid model with expansion to more school sites applying the barriers and facilitators gathered in phase 1, while continuing to refine and better understand ways to effectively screen for, and confirm, the presence of language impairment. SLTs or TA’s were identified as potential groups best suited to handle the regular administration of an annual screening format for DSD as compared to SLPs, general education teachers, or parents. This question needs to be systematically explored in a phase 2 project under a 131 hybrid-format where the accuracy of these groups is analyzed while at the same time expanding screenings to four to eight schools in the district. The timeframe for a project of this nature would still be conducted within one school year but would require an active research team comprised of DSD employees and university research team members. Results from a study of this nature, by use of quantitative methods, would help better inform DSD which group is best suited to administer screening regularly. Summary and Conclusion The purpose of this study was to engage in implementation research by applying a hybrid framework that looked to examine barriers and/or facilitators to the systematic uptake of annual screening for language impairment in DSD while at the same time determine the usefulness of the RSR screening measure as a valid and reliable tool for use in regular screenings that identifies children with language impairment at a higher rate than the current referral-based format. Moreover, this was a feasibility study that looked to guide and inform policy and decision-makers in the Davis School District regarding the practicability of universal screening for language impairment. As such, the findings from this study contribute appreciably to better understanding the weaknesses of the predominantly utilized referral-based format in public schools, what groups contribute to the problematic nature of such a format, measures that stand to help build a solid foundation for an annual screening format, and the barriers and facilitators to implementing such a format. The following points conclude this report by providing a summary of its key findings. 132 • Kindergarten general education teachers and SLPs in DSD widely agree that utilization of an annual screening format would be more beneficial to their students with language impairments than the current referral-based format that relies primarily on general education teacher referrals. Such a format would aid these groups in better fulfilling their professional responsibilities as public educators. • Professional burden hinders the current referral-based system specifically with general education teachers. If not diffused, transfer of the burden from teachers to SLPs stands as a detrimental barrier to implementation of an annual-based format. • Speech-language technicians and/or teaching assistants represent recommended groups to conduct regular language screenings. Due to the prodigious amount of burden currently on general education teachers and SLPs, it is not recommended these groups be responsible for regular administration of annual language screenings. Parents as well would represent a suboptimal choice for use in regular administration as they lack the required education and experience. • Response to intervention (RTI) format program (i.e., MTSS) is still a relatively new system in DSD. If cultivated and supported properly, RTI stands to play a pivotal role in the reduction of the burden to general education teachers and SLPs by placing the formidable role of execution of tier 2 and tier 3 intervention onto teaching assistants or speech-language technicians. Because these professional positions receive specialized training for augmenting language intervention under the guidance of SLPs, the actual numbers of children initially identified by a screening format that would ultimately require language-based goals on an IEP 133 would greatly be reduced and thus be manageable by SLPs due to tiered intervention. • Under either a referral- or annual-based format for language impairment identification, public schools at the local site and district levels need to increase awareness of the academic impact of unidentified and untreated language impairment in children among general education teachers. This could be achieved through targeted in-service trainings, for example. • The Redmond Sentence Recall (RSR) language screening protocol represents a valid and reliable, and therefore, beneficial tool for routine use in DSD as the primary screening measure for an annual screening format. Likewise, the CELF-4 is recommended for use in confirmatory testing by SLPs. • The RSR language measure identifies children with language impairments 2x more accurately than the leading referral source of general education teachers. • Under an annual-screening format, DSD can expect to identify kindergarten students with language impairments at a rate closer to prevalence expectations when using the RSR measure for screening and the CELF-4 for confirmatory testing. 134 Table 9. Prevalence Estimates for Specific Language Impairment Source Population Grade; Ages Prevalence Language findings confirmatory measure cutoff Present Community-based sample Kindergarten; 7.88% > 80 (-1.3 SD) study from public schools in 4-6 yrs Davis County, Utah, U.S. Norbury et Population-based sample Kindergarten; 7.58% > 77 (-1.5 SD) al. (2016)a from public schools in 4-5 yrs Surrey, England Tomblin et Population-based sample Kindergarten; 7.4% > 81 (-1.25 al. (1997)b from public schools in the 5-6 yrs SD) upper midwest, U.S. Beitchman Population-based sample Kindergarten; 8.04% > 70 (-2 SD) et al. from public schools in the 5 yrs (1986)c Ottawa-Charleton municipal region in Canada a b n = 7,267. n = 7,218 .cn = 1,655 Table 10. Positive Predictive Rates Source Population Present study Community-based sample from public schools in Davis County, Utah, U.S. Redmond Community-based sample (2007)a from public schools in Salt Lake County, Utah, U.S. Archibald Community-based sample & Joanisse from public schools in (2009)b Ontario, Canada a b n = 87. n = 108 Grade; Ages Positive Predictive Value Kindergarten; .74 4-6 yrs Language confirmatory measure cutoff > 80 (-1.3 SD) Kindergarten; .68 5-6 yrs > 85 (-1.0 SD) Kindergarten; .79 5-6 yrs > 80 (-1.3 SD) 135 Table 11. Theme/Code Groups Theme Code group Facilitator Supportive attitudes and & Barrier willingness to change Barrier Facilitator Facilitator Facilitator Barrier Facilitator Barrier Description The attitudes of key actors and stakeholders and their overall willingness to change (or resist change) to an annual screening format. Availability of Current and new burdens as a result of limited resources/time resources and time placed on key actors and constraints stakeholders as a result of the implementation of an annual screening format. Benefits to Reduction of burden on teachers by not being educators/district the primary source of referrals for LI under an annual screening format. Teachers would see a reduction of behavioral problems in students otherwise not identified under the current referral-based system. Benefit to academics and social aspects of students with LI. District officials would better accomplish “child find” mandate under IDEA. Concern for the wellThe needs of the students are paramount. being of students Actors and stakeholders would be willing to see an increase in burden in order to identify and serve the LI population. Response to Intervention RTI is key to distributing the increased burden (RTI) need placed on potential actors and stakeholders as a result of an annual screening format. Understanding/education GE teachers lack the necessary training and with regard to LI experience to reliably identify children with LI for referral to SLPs Use of existing Using existing policy/structure such as RTI to policy/structure to reduce burden. TAs or SLTs could be an initial facilitate annual group in charge of regular screening. This screening would help minimize impact of switch-over to an annual screening format to teachers and SLPs. Who best to administer Parents, GE teachers, SLPs, TAs, and SLT screener were identified as potential groups to be responsible for regular administration. Of these, TAs and SLTs were identified as initial good fits. 136 Table 11. Theme/Code Groups continued Theme Facilitator & Barrier Code group Training on screener protocol Barrier Need for validated screening tool Facilitator & Barrier Facilitator & Barrier Description Training on the RSR screening protocol was identified as important to successful screening implementation. Training was important and helped TAs learn how to administer RSR task effectively. Time gap between the training and screening sessions was large and created concern with TAs. Identification for the need of a valid and reliable screening tool that provides better outcomes than referral-based sources was important. Opinions of screener and Feedback given regarding the mechanics of screening administration screening administration (scoring sheet, stimuli content, etc.) Screening equipment Feedback given regarding the equipment used during screening administration (e.g. mp3 player, headphones, writing utensils, etc.) APPENDIX A REDMOND SENTENCE RECALL MEASURE APPENDIX B 32-POINT COREQ CHECKLIST Domain 1: Research team and reflexivity Personal Characteristics 1. Focus group facilitator 2. Credentials 3. Occupation 4. Gender 5. Experience and training Relationship with participants 6. Relationship established COREQ guide question/description Response Which author/s conducted the interview or focus group? What were the researcher’s credentials? E.g. PhD, MD What was their occupation at the time of the study? Was the researcher male or female? What experience or training did the researcher have? The principal investigator (PI) will conduct the focus groups 4th year PhD candidate Was a relationship established prior to study commencement? Screening Personnel FG: Initial contact gauged interest in participation. 1-hour training for RSR task. DSD Project Team Member FG: Initial pitch of project prior to prospectus development. Regular contact via email or in person throughout duration of project leading up to FG. PhD student, Speech and Language Pathologist Male Coursework in qualitative research during PhD. Qualitative research experience in grounded theory, semi-structured interviews, coding, theme identification/saturation 139 7. Participant knowledge of the interviewer What did the participants know about the researcher? e.g. personal goals, reasons for doing the research 8. Interviewer characteristics What characteristics were reported about the interviewer/facilitator? e.g. Bias, assumptions, reasons and interests in the research topic Domain 2: Study design Theoretical Framework 9. Methodological orientation Participant selection 10. Sampling 11. Method of approach 12. Sample size All screening personnel and DSD project team members where provided information on PI regarding PhD school history, areas of study/interest, information regarding PI’s goals of this project and general reasons for conducting this project for my dissertation. PI purposely left these discussions at a very high level to minimize my bias influencing individuals. PI withheld personal biases regarding this project from all screening personnel and DSD project team members so as to not influence discussion during the FGs. What methodological orientation was stated to underpin the study? e.g. grounded theory, discourse analysis, ethnography, phenomenology, content analysis Content analysis is the methodological orientation used for this project. The PI sought to understand better the culture of DSD and how participants experienced participation in the screening. Specifically, the PI wanted to understand facilitators/barriers to universal screening from their cultural characteristic points of view. The analysis of content produced from FGs allowed the PI to identify perceived barriers and/or facilitators for universal screening of language impairment. How were participants selected? e.g. purposeful, convenience, consecutive, snowball How were participants approached? e.g. face-to-face, telephone, mail, email How many participants were in the study? Purposeful sampling Primary: face-to-face Secondary: email Kindergarten students: 165 Focus group members: 12 Survey participants: 157 140 13. Non-participation Setting 14. Setting of data collection 15. Presence of nonparticipants 16. Description of sample Data Collection 17. Interview guide 18. Repeat interviews 19. Audio/visual recording 20. Field notes 21. Duration 22. Data saturation 23. Transcripts returned How many people refused to participate or dropped out? Reasons? Kindergarten students: 17 (Did not opt in to participate) Focus group members: 1 (scheduling conflict) Survey participants: 66 (Unknown) Where was the data collected? e.g. home, clinic, workplace Screening data: Individual school sites Focus group data: DSD Related Services offices Survey data: Online No Was anyone else present besides the participants and researchers? What are the important characteristics of the sample? e.g. demographic data, date Were questions, prompts, guides provided by the authors? Was it pilot tested? Were repeat interviews carried out? If yes, how many? Did the research use audio or visual recording to collect the data? Were field notes made during and/or after the interview or focus group? What was the duration of the interviews or focus group? Was data saturation discussed? Were transcripts returned to participants for comment and/or correction? See methods section for description. See Appendix C for questions/prompt guide to be used during FC facilitation. The FC questions were not be pilot tested on participants but vetted with guidance by dissertation committee and DSD team members. The information generated from the FG was used to inform the survey statements and were piloted on four SLP’s in the district prior to full distribution for all district SLPs and general education teachers. No Yes Yes 60 minutes Yes Yes 141 Domain 3: Analysis and findings Data Analysis 24. Number of data coders 25. Description of the coding tree 26. Derivation of themes 27. Software 28. Participant checking Reporting 29. Quotation presented 30. Data and findings consistent 31. Clarity of major themes 32. Clarity of minor themes How many data coders coded the data? Did authors provide a description of the coding tree? Were themes identified in advance or derived from the data? What software, if applicable, was used to manage the data? Did participants provide feedback on the findings? 1 (principal investigator) Were participant quotations presented to illustrate the themes / findings? Was each quotation identified? e.g. participant number Was there consistency between the data presented and the findings? Were major themes clearly presented in the findings? Is there a description of diverse cases or discussion of minor themes? Yes – See results section Yes – See methods section for description. Yes – See methods section for description. MAXQDA 2018 version 18.2.0 Yes Yes Yes Yes APPENDIX C REDMOND SENTENCE RECALL (RSR) ADMINISTRATION AND SCORING PROTOCOL PURPOSE: The RSR measures a child’s ability to repeat sentences. Each child will repeat each of the 16 sentences after s/he hears the sentence on the headphones. MATERIALS: MP3 Player Score sheet Headphones Headphone splitter AGES: 5-9 TIME TO ADMINISTER: Approximately 7-10 minutes BEFORE BEGINNING TEST: Split the audio file and state alpha code and test SAMPLE ITEMS: § Say, “You are going to listen to some sentences on the headphones. I want you to listen to the directions and then she will say some sentences. Repeat exactly what she says.” § For the practice sentence, provide children with feedback and encouragement. § The practice sentence may be re-administered if children perform in a way that suggests they do not understand the task or if during the protocol it seems they need to be re-directed to providing imitative responses. TEST ITEMS: § Administer the remaining sentences by pressing play on the MP3 player. § There is a 6 second pause in between each track, if they child needs more time to answer then press the pause button to give them more time. § One repetition of each sentence is allowed if children specifically request it or if the examiner considers it appropriate (e.g., child seems to need to be redirected to the task or couldn’t hear the stimulus). Repetitions ideally should not be given on more than one or two items per test. § Do not count as an error false starts, self-repairs, or the inclusion of fillers (e.g., “um, oh”) as errors. § Administer all items. 143 SCORING During the Assessment • Score each item by comparing the student’s response to the stimulus sentence. • If the response is not an exact repetition of the stimulus sentence, mark an “x” through the word that was not repeated correctly. • Record the child’s incorrect response(s) above the stimulus word/sentence. • Examples of types of errors: omissions, repetitions, additions, transpositions, and substitutions. After the Assessment • The sample will be listened to in order to verify all the changes that child made during the assessment • After the assessment, carefully consider the following scoring procedures in order to classify the student’s response (e.g., 2 points, 1 point, or 0 points): a. Look at the whole sentence. Count any word that is changed, added, substituted, or omitted as one error. b. Count each transposition that changes the meaning of the sentence as two errors. c. Count each transposition that does not change the meaning of the sentence as one error. d. Using an appropriate contracted form (or non-contracted form) is acceptable and should not be counted as an error (was not or wasn’t are both acceptable). e. Responses that contain regional and cultural patterns or variations that reflect dialectal differences from Mainstream American English are acceptable if they are part of the student’s language system. f. Do not count repetitions of words (as in dysfluency or stuttering) as errors. g. Count the number of errors in the response and classify it according to the following rules: OK (0 Errors) <3 Errors Sentence is repeated verbatim. Write 2 in the “0 Errors” column Write 1 in the “<3 Errors” column Any response with four or more words changed, added, >4 Errors substituted, omitted, or reversed; omission or re-sequencing of phrases containing four or more words. Circle 0. h. Add up the totals for each column. i. Add up the total number of points earned and write that number over the printed 32 at the bottom of the score form. APPENDIX D FOCUS GROUP SEMISTRUCTURED CONVERSATION FACILITATORS Focus Group #1 • (Broad) I’m interested in finding out more about your experience with the administration of the screener. • (Specific) I’m interested in finding out more about how you felt about the screening administration. For example, was it difficult to administer? • (Broad) I’m interested in finding out more about your experience with the training of the screener. • (Specific) I’m interested in finding out what your experiences were with the training for the RSR task. Was it too long, too in-depth, not enough in-depth, etc. What would you change? What would you keep the same? • (Broad) What kind of facilitators and/or barriers do you think may exist in the school district with doing annual screening for language impairment from your perspective? • (Specific) Who (what personnel) do you think should be involved in administering annual screening for language impairment? • (Specific) What kind of support do you think annual screening for language impairment might gain from district SLPs? Focus Group #2 • (Broad) What kind of barriers do you think may exist in the school district with doing annual screening for language impairment from your perspective? • (Broad) What kind of facilitators do you think may exist in the school district with doing annual screening for language impairment from your perspective? 145 • (Specific) I’m interested in finding out what your thoughts are regarding how annual screening may be received by parents of students in DSD. • (Specific) Who (what personnel) do you think should be involved in administering annual screening for language impairment? For example, District TA’s, SLTs, GA teachers, parents, etc. • (Specific) What kind of support do you think annual screening for language impairment might garner from team members at the school and/or district level? APPENDIX E REFLEXIVITY STATEMENT The majority of my studies during graduate school have focused on the well-being of children from a language development perspective. I firmly believe that all children deserve the chance to grow up in an environment that affords them access to resources that can place them on a path to a healthy, happy, and productive life. As a speech and language pathologist interested in child language development, I am attentive to how language difficulties negatively impact a child on a personal and social level. These children are often overlooked, or not identified, as they progress through the publicschool system because the signs and symptoms of language impairment are not as obvious as other developmental disorders. This project, then, is one way I hope to improve the chances for children with language impairment. What allows me to take on this project, and what do I bring to it? I am a White male, upper middle-class, well-educated, opinionated, stubborn, and a son of two welleducated parents who taught me that life can be anything I want it to be if I am willing to work hard enough for it. This privilege allowed me to be raised in an environment that removed many of the obstacles life can put in the way of young people such as abuse, drugs, violence, unhealthy relationships, poor health, and lack of family, social, or economic support. As a result, I entered adulthood as a focused individual who has the 147 ability to see things that need to be done, set goals to better a situation, and systematically achieve desired outcomes. I recognize that my privileged life circumstances showed me what I could achieve, supported me, and gave me credit for what I did. I acknowledge however that a part of this privilege is derived from my being male. I cannot escape this. It is magnified by the fact that the field of speech and language pathology is primarily a female dominated field. The Davis School District – the setting in which I work as a speech and language pathologist, and the setting in which I will conduct this dissertation project, is no different. Through the course of my master’s level training and current position working alongside fellow SLPs in the district, I am sensitive to my “maleness” in a sea of females. I imagine most males in my position would feel a level of discomfort being a “token-male” in a female dominated field, but while thoughts such as these occasionally occupy my mind, I am comfortable in this role. I have grown used to this status and embrace it. I acknowledge that because I am a male, people with whom I work at the district level may automatically assume me to be a leader, to be credible, and knowledgeable. So as an important part of this reflexive piece, I am forced to ask myself the question: is my ability to conduct a project like this – meant ultimately to improve the lives of children, in a setting as described – a function of me thinking I am someone who can work hard for a desired outcome, or, at least at some level, a function of the opportunity afforded me because I am a male? It may be impossible to separate these two factors as they often work together in powerful ways. I am not sure if I am able to answer this question presently as I have only begun on this path of reflecting on things not normally thought about in my world. The only thing I can do is acknowledge it; reflect on it; be aware of it; be sensitive to it; observe 148 how it impacts how I interact with project members from the district and how they interact with me. This reflective process pushes me to observe and acknowledge the way in which I enter this project, and how the individuals who I will work with, enter into my story of who I am. In the end, I am changed by the process of seeking to help improve the lives of children, while interacting with professionals, who I am different from, on a fundamentally deep level. REFERENCES Adlof, S. M., & Hogan, T. P. (2019). If we don’t look, we won’t see: Measuring language development to inform literacy instruction. Policy Insights from the Behavioral and Brain Sciences, 6(2), 210-217. Alonzo, C. N., Yeomans-Maldonado, G., Murphy, K. A., & Bevens, B. (2016). Predicting second grade listening comprehension using prekindergarten measures. Topics in Language Disorders, 36(4), 312-333. American Speech-Language-Hearing Association. (2016). 2016 Report: SLP caseload characteristics. Available from http://www.asha.org/uploadedFiles/2016-SchoolsSurvey-SLP-Caseload-Characteristics.pdf Archibald, L. M., & Joanisse, M. F. (2009). On the sensitivity and specificity of nonword repetition and sentence recall to language and memory impairments in children. Journal of Speech, Language, and Hearing Research, 52(4), 899-914. Armstrong, M. J., Gronseth, G. S., Dubinsky, R., Potrebic, S., Murray, R. P., Getchius, T. S., ... Gagliardi, A. R. (2017). Naturalistic study of guideline implementation tool use via evaluation of website access and physician survey. BMC Medical Informatics and Decision Making, 17(1), 9. Ash, A. C., & Redmond, S. M. (2014). Using finiteness as a clinical marker to identify language impairment. Perspectives on Language Learning and Education, 21(4), 148-158. Ash, A. C., Redmond, S. M., Timler, G. R., & Kean, J. (2017). The influence of scale structure and sex on parental reports of children’s social (pragmatic) communication symptoms. Clinical Linguistics & Phonetics, 31(4), 293-312. Baddeley, A. D., & Hitch, G. (1974). Working memory. In G.H. Bower (Ed.), Psychology of learning and motivation (Vol. 8, pp. 47-89). Cambridge, MA: Academic Press. Barbour, R. (2008). Doing focus groups. Thousand Oaks, CA: Sage. Beitchman, J. H., Nair, R., Clegg, M., & Patel, P. G. (1986). Prevalence of speech and language disorders in 5-year-old kindergarten children in the Ottawa-Carleton region. Journal of Speech and Hearing Disorders, 51(2), 98-110. 150 Beitchman, J. H., Wilson, B., Johnson, C. J., Atkinson, L., Young, A., Adlaf, E., ... Douglas, L. (2001). Fourteen-year follow-up of speech/language-impaired and control children: Psychiatric outcome. Journal of the American Academy of Child & Adolescent Psychiatry, 40(1), 75-82. Betz, S. K., Eickhoff, J. R., & Sullivan, S. F. (2013). Factors influencing the selection of standardized tests for the diagnosis of specific language impairment. Language, Speech, and Hearing Services in Schools, 44(2), 133-146. Bishop, D. V. (2010). Which neurodevelopmental disorders get researched and why?. PLoS One, 5(11), e15112. Bishop, D. V., & McDonald, D. (2009). Identifying language impairment in children: Combining language test scores with parental report. International Journal of Language & Communication Disorders, 44(5), 600-615. Bishop, D. V., North, T. O. N. Y., & Donlan, C. H. R. I. S. (1996). Nonword repetition as a behavioural marker for inherited language impairment: Evidence from a twin study. Journal of Child Psychology and Psychiatry, 37(4), 391-403. Bossuyt, P. M., Reitsma, J. B., Bruns, D. E., Gatsonis, C. A., Glasziou, P. P., Irwig, L. M., ... Lijmer, J. G. (2003). The STARD statement for reporting studies of diagnostic accuracy: Explanation and elaboration. Clinical Chemistry, 49(1), 718. Boster, J. B., & McCarthy, J. W. (2018). Designing augmentative and alternative communication applications: The results of focus groups with speech-language pathologists and parents of children with autism spectrum disorder. Disability and Rehabilitation: Assistive Technology, 13(4), 353-365. Boswell, S. (2006). Seven states begin work on personnel shortages, qualifications. The ASHA Leader, 11(10), 1-33 Boswell, S. (2007). California clinicians win school salary gains: Districts use different strategies to address personnel shortages. The ASHA Leader, 12(2), 1-17. Boyle, C. A., Boulet, S., Schieve, L. A., Cohen, R. A., Blumberg, S. J., Yeargin-Allsopp, M., ... Kogan, M. D. (2011). Trends in the prevalence of developmental disabilities in US children, 1997–2008. Pediatrics, 127(6), 1034-1042. Brady, N., Skinner, D., Roberts, J., & Hennon, E. (2006). Communication in young children with fragile x syndrome: A qualitative study of mothers' perspectives. American Journal of Speech Language Pathology, 15(4), 353-364. Breiman, R. F. (2016). Immunization and vaccine related implementation research (IVIR) advisory committee. Thousand Oaks, CA: Sage. 151 Briscoe, J., Bishop, D. V., & Norbury, C. F. (2001). Phonological processing, language, and literacy: A comparison of children with mild-to-moderate sensorineural hearing loss and those with specific language impairment. The Journal of Child Psychology and Psychiatry and Allied Disciplines, 42(3), 329-340. Bronfenbrenner, U., & Morris, P. A. (1998). The ecology of developmental processes. In W. Damon & R. M. Lerner (Eds.), Handbook of child psychology: Theoretical models of human development (pp. 993-1028). Hoboken, NJ: John Wiley & Sons Inc. Brown, J. (1958). Some tests of the decay theory of immediate memory. Quarterly Journal of Experimental Psychology, 10(1), 12-21. Brownson, R. C., Colditz, G. A., & Proctor, E. K. (Eds.). (2018). Dissemination and implementation research in health: Translating science to practice. New York, NY: Oxford University Press. Bujang, M. A., & Adnan, T. H. (2016). Requirements for minimum sample size for sensitivity and specificity analysis. Journal of Clinical and DiagnosticRresearch: JCDR, 10(10), YE01. Cabana, M. D., Rand, C. S., Powe, N. R., Wu, A. W., Wilson, M. H., Abboud, P. A. C., & Rubin, H. R. (1999). Why don't physicians follow clinical practice guidelines?: A framework for improvement. JAMA, 282(15), 1458-1465. Caldwell, J. T., Ford, C. L., Wallace, S. P., Wang, M. C., & Takahashi, L. M. (2017). Racial and ethnic residential segregation and access to health care in rural areas. Health & Place, 43, 104-112. Canedo, J. R., Miller, S. T., Schlundt, D., Fadden, M. K., & Sanderson, M. (2018). Racial/ethnic disparities in diabetes quality of care: The role of healthcare access and socioeconomic status. Journal of Racial and Ethnic Health Disparities, 5(1), 7-14. Catts, H. W., Fey, M. E., Tomblin, J. B., & Zhang, X. (2002). A longitudinal investigation of reading outcomes in children with language impairments. Journal of Speech, Language, and Hearing Research, 45(6), 1142-1157. Chandler, S., Charman, T., Baird, G., Simonoff, E., Loucas, T. O. M., Meldrum, D., ... Pickles, A. (2007). Validation of the social communication questionnaire in a population cohort of children with autism spectrum disorders. Journal of the American Academy of Child & Adolescent Psychiatry, 46(10), 1324-1332. Christopulos, T. T., & Keen, J., (2018). General education teachers’ contribution to the identification of children with language impairment. Unpublished data. 152 Clegg, J., Hollis, C., Mawhood, L., & Rutter, M. (2005). Developmental language disorders–a follow‐up in later adult life. Cognitive, language and psychosocial outcomes. Journal of Child Psychology and Psychiatry, 46(2), 128-149. Conti-Ramsden, G. (2003). Processing and linguistic markers in young children with specific language impairment (SLI). Journal of Speech, Language, and Hearing Research, 46(5), 1029-1037. Conti-Ramsden, G., & Botting, N. (2004). Social difficulties and victimization in children with SLI at 11 years of age. Journal of Speech, Language, and Hearing Research, 47(1), 145-161. Conti‐Ramsden, G., & Botting, N. (2008). Emotional health in adolescents with and without a history of specific language impairment (SLI). Journal of Child Psychology and Psychiatry, 49(5), 516-525. Conti‐Ramsden, G., Botting, N., & Faragher, B. (2001). Psycholinguistic markers for specific language impairment (SLI). Journal of Child Psychology and Psychiatry, 42(6), 741-748. Corbin, J., Strauss, A. L., & Strauss, A. (2015). Basics of qualitative research. Thousand Oaks, CA: Sage. Creswell, J. W., & Creswell, J. D. (2017). Research design: Qualitative, quantitative, and mixed methods approaches. Thousand Oaks, CA: Sage. Cronin, P., Reeve, R., Mccabe, P., Viney, R., & Goodall, S. (2017). The impact of childhood language difficulties on healthcare costs from 4 to 13 years: Australian longitudinal study. International Journal of Speech-Language Pathology, 19(4), 381-391. Cunningham, B. J., Daub, O. M., & Cardy, J. O. (2019). Barriers to implementing evidence-based assessment procedures: Perspectives from the front lines in pediatric speech-language pathology. Journal of Communication Disorders, 80, 66-80. Curran, G. M., Bauer, M., Mittman, B., Pyne, J. M., & Stetler, C. (2012). Effectivenessimplementation hybrid designs: Combining elements of clinical effectiveness and implementation research to enhance public health impact. Medical Care, 50(3), 217. Daneman, M. (1991). Individual differences in reading skills. In R. Barr, M.M Kamil, P. P. Mosentha1, & P.D. Pearson (Eds.), Handbook of reading research Volume II (pp. 471-504). New York & London: Longman. Davis School District: Retrieved March 07, 2018, from https://www.davis.k12.ut.us/Page/93472. 153 Demakis, J. G., McQueen, L., Kizer, K. W., & Feussner, J. R. (2000). Quality Enhancement Research Initiative (QUERI): A collaboration between research and clinical practice. Medical Care, 38(6), I-17. DeMonte, J. (2013). Who is in charge of teacher preparation? Center for American Progress. Retrieved from https:// cdn.americanprogress.org/wpcontent/uploads/2013/06 /DeMonteTeacherPrep-brief-1.pdf Di Rezze, B., Rosenbaum, P., Zwaigenbaum, L., Hidecker, M. J. C., Stratford, P., Cousins, M., ... Law, M. (2016). Developing a classification system of social communication functioning of preschool children with autism spectrum disorder. Developmental Medicine & Child Neurology, 58(9), 942-948. Domitrovich, C. E., Bradshaw, C. P., Greenberg, M. T., Embry, D., Poduska, J. M., & Ialongo, N. S. (2010). Integrated models of school‐based prevention: Logic and theory. Psychology in the Schools, 47(1), 71-88. Dollaghan, C. A. (2007). The handbook for evidence-based practice in communication disorders. Baltimore, MD: Paul H Brookes Publishing Company. Dollaghan, C., & Campbell, T. F. (1998). Nonword repetition and child language impairment. Journal of Speech, Language, and Hearing Research, 41(5), 11361146. Dollaghan, C. A., Campbell, T. F., Paradise, J. L., Feldman, H. M., Janosky, J. E., Pitcairn, D. N., & Kurs-Lasky, M. (1999). Maternal education and measures of early speech and language. Journal of Speech, Language, and Hearing Research, 42(6), 1432-1443. Dunn, M., Flax, J., Sliwinski, M., & Aram, D. (1996). The use of spontaneous language measures as criteria for identifying children with specific language impairment: An attempt to reconcile clinical and research incongruence. Journal of Speech, Language, and Hearing Research, 39(3), 643-654. Eadie, P. A., Fey, M. E., Douglas, J. M., & Parsons, C. L. (2002). Profiles of grammatical morphology and sentence imitation in children with specific language impairment and Down syndrome. Journal of Speech, Language, and Hearing Research, 45(4), 720-732. Eaves, L. C., Wingert, H. D., Ho, H. H., & Mickelson, E. C. (2006). Screening for autism spectrum disorders with the social communication questionnaire. Journal of Developmental & Behavioral Pediatrics, 27(2), S95-S103. Eccles, M. P., & Mittman, B. S. (2006). Welcome to implementation science. Implementation Science, 1(1), 1. 154 Eisenberg, S. L., & Guo, L. Y. (2013). Differentiating children with and without language impairment based on grammaticality. Language, Speech, and Hearing Services in Schools, 44(1), 20-31. Entwislea, D. R., & Astone, N. M. (1994). Some practical guidelines for measuring youth's race/ethnicity and socioeconomic status. Child Development, 65(6), 15211540. Fenson, L., Bates, E., Philip D. S., Marchman V. A., Reznick S. J., & Thal D. J. (2007) MacArthur-Bates communicative development inventories. Baltimore, MD: Paul H. Brookes Publishing Company. Feuerstein, J. L., Olswang, L. B., Greenslade, K. J., Dowden, P., Pinder, G. L., & Madden, J. (2018). Implementation research: Embracing practitioners' views. Journal of Speech, Language, and Hearing Research, 61(3), 645-657. Finestack, L. H., & Satterlund, K. E. (2018). Current practice of child grammar intervention: A survey of speech-language pathologists. American Journal of Speech-Language Pathology, 27(4), 1329-1351.= Flay, B. R., Biglan, A., Boruch, R. F., Castro, F. G., Gottfredson, D., Kellam, S., ... Ji, P. (2005). Standards of evidence: Criteria for efficacy, effectiveness and dissemination. Prevention Science, 6(3), 151-175. Flores, G., Abreu, M., Olivar, M. A., & Kastner, B. (1998). Access barriers to health care for Latino children. Archives of Pediatrics & Adolescent Medicine, 152(11), 1119-1125. Forman, S. G., & Barakat, N. M. (2011). Cognitive‐behavioral therapy in the schools: Bringing research to practice through effective implementation. Psychology in the Schools, 48(3), 283-296. Fourie, R. J. (2009). Qualitative study of the therapeutic relationship in speech and language therapy: Perspectives of adults with acquired communication and swallowing disorders. International Journal of Language & Communication Disorders, 44(6), 979-999. Fowler Jr, F. J. (2013). Survey research methods. Los Angeles, CA: Sage Publications. Freeman, R., Miller, D., & Newcomer, L. (2015). Integration of academic and behavioral MTSS at the district level using implementation science. Learning Disabilities--A Contemporary Journal, 13(1), 59-72. Freemantle, N., Eccles, M., Wood, J., Mason, J., Nazareth, I., Duggan, C., ... Walley, T. (1999). A randomized trial of Evidence-Based Outreach (EBOR): Rationale and design. Controlled Clinical Trials, 20(5), 479-492. 155 Fruth, S. J., Van Veld, R. D., Despos, C. A., Martin, R. D., Hecker, A., & Sincroft, E. E. (2010). The influence of a topic-specific, research-based presentation on physical therapists' beliefs and practices regarding evidence-based practice. Physiotherapy Theory and Practice, 26(8), 537-557. Gagliardi, A. R., Webster, F., Brouwers, M. C., Baxter, N. N., Finelli, A., & Gallinger, S. (2014). How does context influence collaborative decision-making for health services planning, delivery and evaluation? BMC Health Services Research, 14(1), 545. Gathercole, S. E., & Baddeley, A. D. (1990). Phonological memory deficits in language disordered children: Is there a causal connection? Journal of Memory and Language, 29(3), 336-360. Girolamo, T. M. (2017). A national survey: Teacher identification of specific language impairment (Doctoral dissertation), University of Kansas. Glascoe, F. P. (2005). Screening for developmental and behavioral problems. Mental Retardation and Developmental Disabilities Research Reviews, 11(3), 173-179. Glasgow, R. E., Lichtenstein, E., & Marcus, A. C. (2003). Why don’t we see more translation of health promotion research to practice? Rethinking the efficacy-toeffectiveness transition. American Journal of Public Health, 93(8), 1261-1267. Gray, S. (2003). Diagnostic accuracy and test–retest reliability of nonword repetition and digit span tasks administered to preschool children with specific language impairment. Journal of Communication Disorders, 36(2), 129-151. Grela, B. G., & Leonard, L. B. (2000). The influence of argument-structure complexity on the use of auxiliary verbs by children with SLI. Journal of Speech, Language, and Hearing Research, 43(5), 1115-1125. Grol, R., Wensing, M., Eccles, M., & Davis, D. (Eds.). (2013). Improving patient care: The implementation of change in health care. Oxford, England: John Wiley & Sons. Guest, G., Bunce, A., & Johnson, L. (2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18(1), 59-82.] Guest, G., Namey, E., Taylor, J., Eley, N., & McKenna, K. (2017). Comparing focus groups and individual interviews: Findings from a randomized study. International Journal of Social Research Methodology, 20(6), 693-708. Hagedorn, H., Hogan, M., Smith, J. L., Bowman, C., Curran, G. M., Espadas, D., ... Sales, A. E. (2006). Lessons learned about implementing research evidence into clinical practice. Journal of General Internal Medicine, 21(2), S21. 156 Hart, B., & Risley, T. R. (1995). Meaningful differences in the everyday experience of young American children. Baltimore, MD: Paul H Brookes Publishing. Hesse-Biber, S. N., & Leavy, P. (2010). The practice of qualitative research. Thousand Oaks, CA: Sage. Higton, J., Leonardi, S., Choudhoury, A., Richards, N., Owen, D., & Sofroniou, N. (2017). Teacher workload survey 2016. Retrieved from: https://warwick.ac.uk/fac/soc/ier/people/dowen/publications/tws_2016_final_rese arch_report_feb_2017.pdf Hoff, E. (2003). The specificity of environmental influence: Socioeconomic status affects early vocabulary development via maternal speech. Child Development, 74(5), 1368-1378. Hoff, E., & Tian, C. (2005). Socioeconomic status and cultural influences on language. Journal of Communication Disorders, 38(4), 271-278. Hoff‐Ginsberg, E. (1994). Influences of mother and child on maternal talkativeness. Discourse Processes, 18(1), 105-117. Hoover, J. R., Storkel, H. L., & Rice, M. L. (2012). The interface between neighborhood density and optional infinitives: Normal development and specific language impairment. Journal of Child Language, 39(4), 835-862. Howlin, P., Mawhood, L., & Rutter, M. (2000). Autism and developmental receptive language disorder—A follow-up comparison in early adult life. II: Social, behavioural, and psychiatric outcomes. The Journal of Child Psychology and Psychiatry and Allied Disciplines, 41(5), 561-578. Hutchins, T. L., Howard, M., Prelock, P. A., & Belin, G. (2010). Retention of schoolbased SLPs: Relationships among caseload size, workload satisfaction, job satisfaction, and best practice. Communication Disorders Quarterly, 31(3), 139154. Huttenlocher, J., Haight, W., Bryk, A., Seltzer, M., & Lyons, T. (1991). Early vocabulary growth: Relation to language input and gender. Developmental Psychology, 27(2), 236. Jessup, B., Ward, E., Cahill, L., & Keating, D. (2008). Teacher identification of speech and language impairment in kindergarten students using the Kindergarten Development Check. International Journal of Speech-Language Pathology, 10(6), 449-459. Joesch, J. M., Sherbourne, C. D., Sullivan, G., Stein, M. B., Craske, M. G., & Roy-Byrne, P. (2012). Incremental benefits and cost of coordinated anxiety learning and management for anxiety treatment in primary care. Psychological Medicine, 42(9), 1937-1948. 157 Johnson, C. J., Beitchman, J. H., Young, A., Escobar, M., Atkinson, L., Wilson, B., ... Wang, M. (1999). Fourteen-year follow-up of children with and without speech/language impairments: Speech/language stability and outcomes. Journal of Speech, Language, and Hearing Research, 42(3), 744-760. Kamberelis, G., & Dimitriadis, G. (2013). Focus groups. London: Routledge. Kamhi, A. G. (2014). Improving clinical practices for children with language and learning disorders. Language, Speech, and Hearing Services in Schools, 45(2), 92-103. Kelly, J. G., Ryan, A. M., Altman, B. E., & Stelzner, S. P. (2000). Understanding and changing social systems: An ecological view. In J. Rappaport & E. Seidman (Eds.), Handbook of community psychology (pp. 133-159). New York, NY: Kluwer Academic/Plenum. doi:10.1007/978-1-4615-4193-6_7 Kirchner, J.E., Ritchie, M., & Curran, G. (2011). “Facilitating: Design, using, and evaluating a facilitation strategy” Enhancing implementation science meeting sponsored by Department of Veterans Affairs Quality Enhancement Research Initiative, Phoenix: AZ. Kothari, A., & Wathen, C. N. (2017). Integrated knowledge translation: Digging deeper, moving forward. Journal of Epidemiology Community Health, 71(6), 619-623. Landmark Associates. (n.d.). Retrieved July 14, 2019, from https://www.thelai.com/ Law, J., Boyle, J., Harris, F., Harkness, A., & Nye, C. (1998). Screening for primary speech and language delay: A systematic review of the literature. International Journal of Language & Communication Disorders, 33(sup1), 21-23 Law, J., Reilly, S., & Snow, P. C. (2013). Child speech, language and communication need re‐examined in a public health context: A new direction for the speech and language therapy profession. International Journal of Language & Communication Disorders, 48(5), 486-496. Law, J., Roulstone, S., & Lindsay, G. (2015). Integrating external evidence of intervention effectiveness with both practice and the parent perspective: development of ‘What Works’ for speech, language, and communication needs. Developmental Medicine & Child Neurology, 57(3), 223-228. Law, J., & Roy, P. (2008). Parental report of infant language skills: A review of the development and application of the Communicative Development Inventories. Child and Adolescent Mental Health, 13(4), 198-206. . 158 Laws, G., & Bishop, D. V. (2003). A comparison of language abilities in adolescents with Down syndrome and children with specific language impairment. Journal of Speech, Language, and Hearing Research, 46(6), 1324-1339. Leonard, L. B. (2014). Children with specific language impairment. Cambridge, MA: MIT press. Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 55. Lloyd, H., Paintin, K., & Botting, N. (2006). Performance of children with different types of communication impairment on the Clinical Evaluation of Language Fundamentals (CELF). Child Language Teaching and Therapy, 22(1), 47-67. Locke, J., Beidas, R. S., Marcus, S., Stahmer, A., Aarons, G. A., Lyon, A. R., ... Mandell, D. S. (2016). A mixed methods study of individual and organizational factors that affect implementation of interventions for children with autism in public schools. Implementation Science, 11(1), 135. Loucas, T., Charman, T., Pickles, A., Simonoff, E., Chandler, S., Meldrum, D., & Baird, G. (2008). Autistic symptomatology and language ability in autism spectrum disorder and specific language impairment. Journal of Child Psychology and Psychiatry, 49(11), 1184-1192. Lue, M. S. (2001). A survey of communication disorders for the classroom teacher. Boston, MA: Allyn & Bacon. Lyons, R., & Roulstone, S. (2018). Well-being and resilience in children with speech and language disorders. Journal of Speech, Language, and Hearing Research, 61(2), 324-344. MacDonald, M. C., & Christiansen, M. H. (2002). Reassessing working memory: Comment on Just and Carpenter (1992) and Waters and Caplan (1996). Psychological Review, 109, 35–53. Malterud, K. (2012). Systematic text condensation: A strategy for qualitative analysis. Scandinavian Journal of Public Health, 40(8), 795-805. Marchman, V. A., Wulfeck, B., & Weismer, S. E. (1999). Morphological productivity in children with normal language and SLI: A study of the English past tense. Journal of Speech, Language, and Hearing Research, 42(1), 206-219. Marshall, J., Ralph, S., & Palmer, S. (2002). 'I wasn't trained to work with them': Mainstream teachers' attitudes to children with speech and language difficulties. International Journal of Inclusive Education, 6(3), 199-215. Marton, K., Kelmenson, L., & Pinkhasova, M. (2007). Inhibition control and working memory capacity in children with SLI. Psychologia, 50(2), 110-121. 159 Massey, O. T., Armstrong, K., Boroughs, M., Henson, K., & McCash, L. (2005). Mental health services in schools: A qualitative analysis of challenges to implementation, operation, and sustainability. Psychology in the Schools, 42(4), 361-372. McLoyd, V. C. (1998). Socioeconomic disadvantage and child development. American Psychologist, 53(2), 185. Merriam, S. B., & Tisdell, E. J. (2015). Qualitative research: A guide to design and implementation. San Francisco, CA: Jossey-Bass. Merton, R. K. (1987). The focussed interview and focus groups: Continuities and discontinuities. The Public Opinion Quarterly, 51(4), 550-566. Moats, L. C. (1994). The missing foundation in teacher education: Knowledge of the structure of spoken and written language. Annals of Dyslexia, 44(1), 81-102. Montgomery, J. W. (1995). Sentence comprehension in children with specific language impairment: The role of phonological working memory. Journal of Speech, Language, and Hearing Research, 38(1), 187-199. Moore-Brown, B., Nishida, B., Uranga‐Hernandez, Y., Parker, M., & Shubin, J. (2005, November). Finding them and keeping them: Coping with SLP shortages. In Presentation at ASHA Convention. Morgan, P. L., Hammer, C. S., Farkas, G., Hillemeier, M. M., Maczuga, S., Cook, M., & Morano, S. (2016). Who receives speech/language services by 5 years of age in the United States? American Journal of Speech-Language Pathology, 25(2), 183199. Naglieri, J. A. (2001). Do ability and reading achievement correlate? Journal of Learning Disabilities, 34(4), 304. Naglieri, J.A. (2003). Naglieri Nonverbal ability test-individual administration. San Antonio, TX: Harcourt Assessment Inc. Nelson, H. D., Nygren, P., Walker, M., & Panoscha, R. (2006). Screening for speech and language delay in preschool children: Systematic evidence review for the US Preventive Services Task Force. Pediatrics, 117(2), e298-e319. Neta, G., Glasgow, R. E., Carpenter, C. R., Grimshaw, J. M., Rabin, B. A., Fernandez, M. E., & Brownson, R. C. (2015). A framework for enhancing the value of research for dissemination and implementation. American Journal of Public Health, 105(1), 49-57. Newacheck, P. W., Hughes, D. C., & Stoddard, J. J. (1996). Children's access to primary care: Differences by race, income, and insurance status. Pediatrics, 97(1), 26-32. 160 Newcomer, P. L., & Hammill, D. D. (2008). TOLD-P: 4: Test of Language Development. Primary. Austin, TX: Pro-Ed. Nilsen, P. (2015). Making sense of implementation theories, models and frameworks. Implementation Science, 10(1), 53. Norbury, C. F., Gooch, D., Wray, C., Baird, G., Charman, T., Simonoff, E., ... Pickles, A. (2016). The impact of nonverbal ability on prevalence and clinical presentation of language disorder: Evidence from a population study. Journal of Child Psychology and Psychiatry, 57(11), 1247-1257. Norbury, C. F., Nash, M., Baird, G., & Bishop, D. V. (2004). Using a parental checklist to identify diagnostic groups in children with communication impairment: A validation of the Children's Communication Checklist—2. International Journal of Language & Communication Disorders, 39(3), 345-364. O'Callaghan, A. M., McAllister, L., & Wilson, L. (2005). Barriers to accessing rural paediatric speech pathology services: Health care consumers’ perspectives. Australian Journal of Rural Health, 13(3), 162-171. O’Connor, K. E. (2008). “You choose to care”: Teachers, emotions and professional identity. Teaching and Teacher Education, 24(1), 117-126. Oetting, J. B., & Horohov, J. E. (1997). Past-tense marking by children with and without specific language impariemnt. Specific language impairment. Journal of Journal of Speach, Language, and Hearing Research), 62-74. Oetting, J. B., McDonald, J. L., Seidel, C. M., & Hegarty, M. (2016). Sentence recall by children with SLI across two nonmainstream dialects of English. Journal of Speech, Language, and Hearing Research, 59(1), 183-194. Office for Educational Review. (2003). Revised kindergarten development check. Hobart, IN: Office for Educational Review. Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful sampling for qualitative data collection and analysis in mixed method implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 42(5), 533-544. Pascale, C. M. (2010). Cartographies of knowledge: Exploring qualitative epistemologies. Newbury Park, CA: Sage Publications. Patsopoulos, N. A. (2011). A pragmatic view on pragmatic trials. Dialogues in Clinical Neuroscience, 13(2), 217. Pawłowska, M. (2014). Evaluation of three proposed markers for language impairment in English: A meta-analysis of diagnostic accuracy studies. Journal of Speech, Language, and Hearing Research, 57(6), 2261-2273. 161 Petursdottir, A. I., & Mellor, J. R. (2017). Reinforcement contingencies in language acquisition: Implications for language intervention. Policy Insights from the Behavioral and Brain Sciences, 4(1), 25-32. Plante, E., & Vance, R. (1995). Diagnostic accuracy of two tests of preschool language. American Journal of Speech-Language Pathology, 4(2), 70-76. Podell, D. M., & Soodak, L. C. (1993). Teacher efficacy and bias in special education referrals. The Journal of Educational Research, 86(4), 247-253. Poduska, J., Kellam, S., Brown, C. H., Ford, C., Windham, A., Keegan, N., & Wang, W. (2009). Study protocol for a group randomized controlled trial of a classroombased intervention aimed at preventing early risk factors for drug abuse: Integrating effectiveness and implementation research. Implementation Science, 4(1), 56. Poll, G. H., Betz, S. K., & Miller, C. A. (2010). Identification of clinical markers of specific language impairment in adults. Journal of Speech, Language, and Hearing Research, 53(2), 414-429. https://doi.org/10.1044/1092-4388(2009/080016) Polovoy, C. (2008). Telepractice in schools helps address personnel shortages. The ASHA Leader, 13(9), 22-24. Powell, B. J., Proctor, E. K., Glisson, C. A., Kohl, P. L., Raghavan, R., Brownson, R. C., ... Palinkas, L. A. (2013). A mixed methods multiple case study of implementation as usual in children’s social service organizations: study protocol. Implementation Science, 8(1), 92. Prelock, P. A. (2006). Autism spectrum disorders: Issues in assessment and intervention. Austin, TX: Pro-Ed. Rabin, B. A., & Brownson, R. C. (2017). Terminology for dissemination and implementation research. Dissemination and Implementation Research in Health: Translating Science to Practice, 2, 19-45. Racine, D. P. (2006). Reliable effectiveness: A theory on sustaining and replicating worthwhile innovations. Administration and Policy in Mental Health and Mental Health Services Research, 33(3), 356-387. Redmond, S. M. (2005). Differentiating SLI from ADHD using children's sentence recall and production of past tense morphology. Clinical Linguistics & Phonetics, 19(2), 109-127. Redmond, S. M. (2007). Redmond sentence recall manual. Department of Communications Sciences and Disorders. University of Utah, Salt Lake City, Utah. 162 Redmond, S. M. (2011). Peer victimization among students with specific language impairment, attention-deficit/hyperactivity disorder, and typical development. Language, Speech, and Hearing Services in Schools. Redmond, S. M., & Ash, A. C. (2014). A cross-etiology comparison of the socioemotional behavioral profiles associated with attention-deficit/hyperactivity disorder and specific language impairment. Clinical Linguistics & Phonetics, 28(5), 346-365. Redmond, S. M., Ash, A. C., Christopulos, T. T., & Pfaff, T. (2019). Diagnostic accuracy of sentence recall and past tense measures for identifying children's language impairments. Journal of Speech, Language, and Hearing Research, 62, 1-17. Redmond, S. M., Thompson, H. L., & Goldstein, S. (2011). Psycholinguistic profiling differentiates specific language impairment from typical development and from attention-deficit/hyperactivity disorder. Journal of Speech, Language, and Hearing Research, 54(1), 99-117. Reetzke, R., Zou, X., Sheng, L., & Katsos, N. (2015). Communicative development in bilingually exposed Chinese children with autism spectrum disorders. Journal of Speech, Language, and Hearing Research, 58(3), 813-825. Reschovsky, J. D., Hadley, J., & Landon, B. E. (2006). Effects of compensation methods and physician group structure on physicians' perceived incentives to alter services to patients. Health Services Research, 41(4p1), 1200-1220. Rice, M. L., & Blossom, M. (2013). What do children with specific language impairment do with multiple forms of DO? Journal of Speech, Language, and Hearing Research, 56(1), 222-235. Rice, M. L., Tomblin, J. B., Hoffman, L., Richman, W. A., & Marquis, J. (2004). Grammatical tense deficits in children with SLI and nonspecific language impairment. Journal of Speech, Language, and Hearing Research, 47(4), 816834. Rice, M. L., & Wexler, K. (1996). Toward tense as a clinical marker of specific language impairment in English-speaking children. Journal of Speech, Language, and Hearing Research, 39(6), 1239-1257. Rice, M. L., Wexler, K., & Cleave, P. L. (1995). Specific language impairment as a period of extended optional infinitive. Journal of Speech, Language, and Hearing Research, 38(4), 850-863. Rice, M. L., Wexler, K., & Hershberger, S. (1998). Tense over time: The longitudinal course of tense acquisition in children with specific language impairment. Journal of Speech, Language, and Hearing Research, 41(6), 1412-1431. 163 Rickenbrode, R., & Walsh, K. (2013). Lighting the way: The reading panel report ought to guide teacher preparation. American Educator, 37(2), 30-35. Ritchie, J., & Spencer, L. (2002). Qualitative data analysis for applied policy research. In A. Bryman & R.G. Burgess (Eds.), Analyzing qualitative data (pp. 187-208). New York, NY: Routledge. Ruben, R. J. (2000). Redefining the survival of the fittest: Communication disorders in the 21st century. The Laryngoscope, 110(2), 241-241. Sackett, D. L., Haynes, R. B., Tugwell, P., & Guyatt, G. H. (1985). Clinical epidemiology: A basic science for clinical medicine. Boston, MA: Little, Brown. Sadler, J. (2005). Knowledge, attitudes and beliefs of the mainstream teachers of children with a preschool diagnosis of speech/language impairment. Child Language Teaching and Therapy, 21(2), 147-163. Schuele, C. M., & Dykes, J. C. (2005). Complex syntax acquisition: A longitudinal case study of a child with specific language impairment. Clinical Linguistics & Phonetics, 19(4), 295-318. Schwartz, N. H., Wolfe, J. N., & Cassar, R. (1997). Predicting teacher referrals of emotionally disturbed children. Psychology in the Schools, 34(1), 51-61. Semel, E. M., Wiig, E. H., & Secord, W. (2004). CELF 4: Clinical evaluation of language fundamentals 4 screening test. New York, NY: Pearson, PsyhCorp. Skahan, S. M., Watson, M., & Lof, G. L. (2007). Speech-language pathologists' assessment practices for children with suspected speech sound disorders: Results of a national survey. American Journal of Speech-Language Pathology, 16(3), 246-259. Smith, K. B., Humphreys, J. S., & Wilson, M. G. (2008). Addressing the health disadvantage of rural populations: How does epidemiological evidence inform rural health policies and research? Australian Journal of Rural Health, 16(2), 5666. Solomon, D. H., Brookhart, M. A., Polinski, J., Katz, J. N., Cabral, D., Licari, A., & Avorn, J. (2005). Osteoporosis action: Design of the healthy bones project trial. Contemporary Clinical Trials, 26(1), 78-94. Spaulding, T. J., Plante, E., & Farinella, K. A. (2006). Eligibility criteria for language impairment. Language, Speech, and Hearing Services in Schools, 37(1), 61-72. Speech-Language Pathology: Data USA (2016). Retrieved from https://datausa.io/profile/cip/ speech-language-pathology 164 Sperry, D. E., Sperry, L. L., & Miller, P. J. (2019). Reexamining the verbal environments of children from different socioeconomic backgrounds. Child Development, 90(4), 1303-1318. https://doi.org/10.1111/cdev.13072. Stanovich, K. E. (2009). Matthew effects in reading: Some consequences of individual differences in the acquisition of literacy. Journal of Education, 189(1-2), 23-55. Starr, E. M., Martini, T. S., & Kuo, B. C. (2016). Transition to kindergarten for children with autism spectrum disorder: A focus group study with ethnically diverse parents, teachers, and early intervention service providers. Focus on Autism and Other Developmental Disabilities, 31(2), 115-128. Stetler, C. B., Damschroder, L. J., Helfrich, C. D., & Hagedorn, H. J. (2011). A guide for applying a revised version of the PARIHS framework for implementation. Implementation Science, 6(1), 99. Stetler, C. B., Mittman, B. S., & Francis, J. (2008). Overview of the VA quality enhancement research initiative (QUERI) and QUERI theme articles: QUERI series. Implementation Science, 3(1), 8. Stothard, S. E., Snowling, M. J., Bishop, D. V., Chipchase, B. B., & Kaplan, C. A. (1998). Language-impaired preschoolers: A follow-up into adolescence. Journal of Speech, Language, and Hearing Research, 41(2), 407-418. Szczepura, A. (2005). Access to health care for ethnic minority populations. Postgraduate Medical Journal, 81(953), 141-147. Sullivan, G., Craske, M. G., Sherbourne, C., Edlund, M. J., Rose, R. D., Golinelli, D.... Roy-Byrne, P. P. (2007). Design of the Coordinated Anxiety Learning and Management (CALM) study: Innovations in collaborative care for anxiety disorders. General Hospital Psychiatry, 29(5), 379-387. Sullivan, P. M., & Knutson, J. F. (2000). Maltreatment and disabilities: A populationbased epidemiological study. Child Abuse & Neglect, 24(10), 1257-1273. Tabak, R. G., Khoong, E. C., Chambers, D. A., & Brownson, R. C. (2012). Bridging research and practice: Models for dissemination and implementation research. American Journal of Preventive Medicine, 43(3), 337-350. Tipton, L. A., & Blacher, J. (2014). Brief report: Autism awareness: Views from a campus community. Journal of Autism and Developmental Disorders, 44(2), 477483. Tomblin, J. B. (1996). Genetic and environmental contributions to the risk for specific language impairment. In M. L. Rice (Ed.), Towards a genetics of language (pp. 191–210). Mahwah, NJ: Erlbaum. 165 Tomblin, J. B., Freese, P. R., & Records, N. L. (1992). Diagnosing specific language impairment in adults for the purpose of pedigree analysis. Journal of Speech, Language, and Hearing Research, 35(4), 832-843. Tomblin, J. B., Records, N. L., Buckwalter, P., Zhang, X., Smith, E., & O’Brien, M. (1997). Prevalence of specific language impairment in kindergarten children. Journal of Speech, Language, and Hearing Research, 40(6), 1245-1260. Tomblin, J. B., Records, N. L., & Zhang, X. (1996). A system for the diagnosis of specific language impairment in kindergarten children. Journal of Speech, Language, and Hearing Research, 39(6), 1284-1294. Tong, A., Sainsbury, P., & Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6), 349-357. U.S. Department of Education. (n.d.). Retrieved March 09, 2018, from https://www.ed.gov/. U.S. Department of Veterans Affairs. (n.d.). Retrieved March 09, 2018, from https://www.va.gov/. U.S. Office of Special Education. (n.d.). Retrieved November 07, 2017, from http://nirn.fpg.unc.edu/resources/case-example-us-office-special-education. Verdon, S., Wilson, L., Smith-Tamaray, M., & McAllister, L. (2011). An investigation of equity of rural speech-language pathology services for children: A geographic perspective. International Journal of Speech-Language Pathology, 13(3), 239250. Wang, F., & Luo, W. (2005). Assessing spatial and nonspatial factors for healthcare access: towards an integrated approach to defining health professional shortage areas. Health & Place, 11(2), 131-146. Weiler, B., Schuele, C. M., Feldman, J. I., & Krimm, H. (2018). A multiyear populationbased study of kindergarten language screening failure rates using the Rice Wexler Test of Early Grammatical Impairment. Language, Speech, and Hearing Services in Schools, 49(2), 248-259. Weismer, S. E., Tomblin, J. B., Zhang, X., Buckwalter, P., Chynoweth, J. G., & Jones, M. (2000). Nonword repetition performance in school-age children with and without language impairment. Journal of Speech, Language, and Hearing Research, 43(4), 865-878. Wells, G. (1985). Language development in the pre-school years. Cambridge, England: Cambridge University Press. 166 Wells, G. (1986). The meaning makers: Children learning language and using language to learn. Portsmouth, NH: Heinemann Educational Books Inc. Wells, K. B. (1999). Treatment research at the crossroads: The scientific interface of clinical trials and effectiveness research. American Journal of Psychiatry, 156(1), 5-10. Wetherell, D., Botting, N., & Conti-Ramsden, G. (2007). Narrative skills in adolescents with a history of SLI in relation to non-verbal IQ scores. Child Language Teaching and Therapy, 23(1), 95-113. Whitehouse, A. J., Watt, H. J., Line, E. A., & Bishop, D. V. (2009). Adult psychosocial outcomes of children with specific language impairment, pragmatic language impairment and autism. International Journal of Language & Communication Disorders, 44(4), 511-528. Wright, P., & Wright, P. (2007). The child find mandate: What does it mean to you? Retrieved on August 17, 2019. Retrieved from https://www.wrightslaw.com/info/child.find.mandate.htm Yew, S. G. K., & O’Kearney, R. (2013). Emotional and behavioural outcomes later in childhood and adolescence for children with specific language impairments: Meta‐analyses of controlled prospective studies. Journal of Child Psychology and Psychiatry, 54(5), 516-524. Youman, M., & Mather, N. (2015). Dyslexia laws in the USA: An update. Perspectives on Language and Literacy, 41(4), 10. Young, A. R., Beitchman, J. H., Johnson, C., Douglas, L., Atkinson, L., Escobar, M., & Wilson, B. (2002). Young adult academic outcomes in a longitudinal sample of early identified language impaired and control children. Journal of Child Psychology and Psychiatry, 43(5), 635-645. Zhang, X., & Tomblin, J. B. (2000). The association of intervention receipt with speechlanguage profiles and social-demographic variables. American Journal of SpeechLanguage Pathology, 9(4), 345-357. Zubrick, S. R., Taylor, C. L., Rice, M. L., & Slegers, D. W. (2007). Late language emergence at 24 months: An epidemiological study of prevalence, predictors, and covariates. Journal of Speech, Language, and Hearing Research, 50(6), 15621592.