Is "superheroes social skills" an evidence-based approach to teach social skills to children in a clinical setting? A pilot study examining an evidence-based program for childrem with autism

The current study assessed the Superhero Social Skills program as an evidencebased practice for teaching social skills to elementary children with Autism Spectrum Disorder (ASD) in a clinical out-patient setting. The program consists of many research validated components, including peer mediation, video-modeling, and social stories. There were 4 participants with ASD and 4 "peer buddies," all between the ages of 5 and 10. Intervention sessions took place at an outpatient clinical setting over 8 weeks. One lesson was taught per week and incorporated components from the program's typical two lesson per week format. After each session, analog free play observations were conducted and coded by the researcher and another graduate student to achieve interrater reliability. Parents reported the number of spontaneous uses of skills at home to measure generalization. Effect size and percentage of nonoverlapping data points were calculated to determine changes in social engagement and generalization. There were also pre- and postmeasures of social behaviors completed by parents and consumer satisfaction measures completed after the intervention by parents and children. The results of this study indicate increased levels of social initiations, social responses, and social engagement during free play observations. For most participants, there was also an increase in generalized use of the skills. Parents and children reported high levels of satisfaction with the program. Overall, results suggest that the "superhero social skills" program is effective for children with ASD.

IS "SUPERHEROES SOCIAL SKILLS" AN EVIDENCE-BASED APPROACH TO TEACH SOCIAL SKILLS TO CHILDREN IN A CLINICAL SETTING? A PILOT STUDY EXAMINING AN EVIDENCED-BASED PROGRAM FOR CHILDREN WITH AUTISM by Julia Ann Kelly Hood A thesis submitted to the faculty of The University of Utah in partial fulfillment of the requirements for the degree of Master of Science Department of Educational Psychology The University of Utah December 2010 Copyright © Julia Ann Kelly Hood 2010 All Rights Reserved Th e Un i v e r s i t y o f Ut a h Gr a d u a t e S c h o o l STATEMENT OF THESIS APPROVAL The thesis of Julia Ann Kelly Hood has been approved by the following supervisory committee members: Elaine Clark , Chair 3/11/2010 Date Approved William Jenson , Member 3/11/2010 Date Approved Peter Nicholas , Member 3/11/2010 Date Approved and by Elaine Clark , Chair of the Department of Department of Educational Psychology and by Charles A. Wight, Dean of The Graduate School. ABSTRACT The current study assessed the Superhero Social Skills program as an evidence-based practice for teaching social skills to elementary children with Autism Spectrum Disorder (ASD) in a clinical out-patient setting. The program consists of many research-validated components, including peer mediation, video-modeling, and social stories. There were 4 participants with ASD and 4 "peer buddies," all between the ages of 5 and 10. Intervention sessions took place at an outpatient clinical setting over 8 weeks. One lesson was taught per week and incorporated components from the program's typical two lesson per week format. After each session, analog free play observations were conducted and coded by the researcher and another graduate student to achieve interrater reliability. Parents reported the number of spontaneous uses of skills at home to measure generalization. Effect size and percentage of nonoverlapping data points were calculated to determine changes in social engagement and generalization. There were also pre- and postmeasures of social behaviors completed by parents and consumer satisfaction measures completed after the intervention by parents and children. The results of this study indicate increased levels of social initiations, social responses, and social engagement during free play observations. For most participants, there was also an increase in generalized use of the skills. Parents and children reported high levels of satisfaction with the program. Overall, results suggest that the "superhero social skills" program is effective for children with ASD. TABLE OF CONTENTS ABSTRACT...................................................................................................................iii LIST OF TABLES.........................................................................................................vii LIST OF FIGURES.......................................................................................................viii ACKNOWLEDGMENTS.............................................................................................x Chapters 1. INTRODUCTION……………………………………………………..………...1 Evidence-Based Practice………………............….……………....5 Meta-Analyses………………………………...............………....10 Meta-Analyses of General Interventions for ASD……………....12 Meta-Analysis of Social Skills Interventions for ASD…….……13 Current Social Skills Programs…………....…………...………..16 Social Skills Programs for Autism Spectrum Disorders…….......19 Video-Modeling Interventions…………………………..……….23 Peer Mediated Interventions…………………………….…….....24 Self-Management Interventions…………………….…………...25 Social Stories………………………………....………………….25 Social Skills Training at Home and School...................................27 Superhero Social Skills………………………...………………...28 Summary…………………………………..…………….………30 Statement of Purpose……………….……………………………31 Research Questions………….…………………………………..31 2. METHOD………………………………...……………………………..…......…….33 Participants……………………..……………..………………....34 Setting............................................................................................38 Dependent Measures……………….…………………….…....…39 Design……………………………...………………………….....45 Procedures……………………….….…………………….……..47 Baseline……………………………….….………..…..………...48 Superhero Social Skills ..............……….…………………..…...48 v Superhero Social Skills Implementation......................................50 Observation and Data Collection of Social Engagement..….......50 Data Analysis…………………………………………….….…..51 3. RESULTS………………………………………....................………………………55 Treatment Integrity........................................................................56 Reliability of Observations...........................................................56 Research Question 1…………………………………………….57 Research Question 2……………………………………………..71 Research Question 3……………………………………………..76 Research Question 4……………………………………………..78 Research Question 5……………………………………………..86 Research Question 6……………………………………………..88 Research Question 7…………………………..……………........99 Research Question 8……………………………………………100 Research Question 9……………………………………………101 4. DISCUSSION……………………………………………..................…….....…….108 Limitations and Future Research……………………………….113 Implications for Practice……………………………………......116 Appendices A. OVERVIEW OF THE "SUPERHEROES" SOCIAL SKILLS MANUAL...117 B. SAMPLE LESSON: GENERALIZED IMITATION....................................121 C. OBSERVATION SYSTEM............................................................................128 D. ADAPTATION OF THE PARENT DAILY REPORT...................................132 E. ADAPTATION OF THE SOCIAL VALIDITY SCALE...............................134 F. CHILD CONSUMER SATISFACTION SURVEY.......................................136 G. AUTISM SOCIAL SKILLS PROFILE.........................................................138 H. ADVERTISEMENT POSTER.......................................................................144 I. PARENT PERMISSION.................................................................................146 J. CHILD ASSENT FORM................................................................................151 vi K. UTAH STATE OFFICE OF EDUCATION AUTISM CLASSIFICATION CRITERIA..............................................................................................154 L. PLACEMENT CHECKLIST........................................................................158 M. TREATMENT FIDELITY CHECKLIST.....................................................161 N. SKILLS AND THEIR STEPS.......................................................................163 REFERENCES………………………………..………………………………………..165 LIST OF TABLES Table Page 1. Caldarrella and Merrell's Taxonomy of Pro-Social Behaviors......................................4 2. Common Social Skills Training Programs for Youth...................................................20 3. Social Skills Programs for Youth with Autism Spectrum Disorder.............................22 4. Participant Characteristics............................................................................................37 5. Correlation of Power Charges, Scooter Cards, and Blackhole Cards with the Observed Rates of Social Interactions..................................................................................79 6. Average Parent Ratings on the Behavior Intervention Rating Scale............................87 7. Average Pre- and Postintervention Ratings on the Social Responsiveness Scale........89 8. Social Responsiveness Scale Ratings for Participant 1...............................................91 9. Social Responsiveness Scale Ratings for Participant 2...............................................93 10. Social Responsiveness Scale Ratings for Participant 3.............................................95 11. Social Responsiveness Scale Ratings for Participant 4.............................................97 LIST OF FIGURES Figures Page 1. Baseline and intervention measures of social initiations for participant 1………..59 2. Baseline and intervention measures of social responses for participant 1….……..60 3. Baseline and intervention measures of total social interactions for participant 1…61 4. Baseline and intervention measures of social initiations for participant 2………...62 5. Baseline and intervention measures of social responses for participant 2…………63 6. Baseline and intervention measures of total social interactions for participant 2…64 7. Baseline and intervention measures of social initiations for participant 3………...65 8. Baseline and intervention measures of social responses for participant 3…………66 9. Baseline and intervention measures of total social interactions for participant 3.…67 10. Baseline and intervention measures of social initiations for participant 4………..68 11. Baseline and intervention measures of social responses for participant 4…………69 12. Baseline and intervention measures of total social interactions for participant 4…70 13. Baseline and intervention measures of the parent daily report for participant 1….72 14. Baseline and intervention measures of the parent daily report for participant 2….73 15. Baseline and intervention measures of the parent daily report for participant 3.....74 16. Baseline and intervention measures of the parent daily report for participant 4.…75 17. Average number of Scooter and Blackhole Cards earned each session.........…….80 18. Scooter and Blackhole Cards earned by participant 1 during each session.....……81 ix 19. Scooter and Blackhole Cards earned by participant 2 during each session……….83 20. Scooter and Blackhole Cards earned by participant 3 during each session……….84 21. Scooter and Blackhole Cards earned by participant 4 during each session……….85 22. Parent ratings on the SRS for participant 1………………........................…….....92 23. Parent ratings on the SRS for participant 2…………………………………….....94 24. Parent ratings on the SRS for participant 3…………………………………….....96 25. Parent ratings on the SRS for participant 4…………………………………….....98 26. Parent ASSP change scores for all participants……………………………....…..102 27. Parent ASSP change scores for participant 1…………………………….………103 28. Parent ASSP change scores for participant 2………………………………….…104 29. Parent ASSP change scores for participant 3………………………………….....105 30. Parent ASSP change scores for participant 4.........................................................106 ACKNOWLEDGMENTS I would like to thank my family. Without all of you, I would not have been able to achieve my goals. Mom, Tom, Susan, Mike, Sean, Nicolle, Steve, and Dad - I cannot tell you how much your support has meant to me, academically and otherwise. I could not be where I am today without you. Thank you to Dr. Clark and Dr. Jenson for giving me the amazing opportunities that have helped me to achieve so much. You have been and continue to be amazing mentors. CHAPTER 1 INTRODUCTION Autistic Disorder is a pervasive developmental disorder characterized by impairment in social interaction, communication, and restricted, repetitive, and stereotyped patterns of behavior, interests, and activities (American Psychiatric Association, 2000). While there is great variation in the symptomology and severity of the disorder, all children with this diagnosis suffer from impairment in social interactions. However, attempts have been made to help children compensate for these impairments in social interactions by the development of social skills interventions. Autism Spectrum Disorder (ASD) is considered a multifactorial disorder (Rutter, 2005). Rutter, like other researchers, has found evidence of genetic links. Twin studies have found concordance rates of 60% in monozygotic twins and 5% in dizygotic twins. Genetic studies have also revealed the rate of ASD in the general population to be about 0.5%, whereas the rate of ASD in siblings is around 6%. This further substantiates the claim that ASD has genetic origins. Other possible causes or contributors that have been identified but which lack the research to support them include prenatal factors, postnatal factors, and immunizations. Autistic Disorder and Asperger's Disorder, a less severe form of Autistic Disorder but with many of the same characteristics, are more recently referred to as Autism 2 Spectrum Disorders (ASD) because of the wide variation in the severity of symptoms. Asperger's Disorder is considered to be a milder form of autism, with characteristics of impairment in social interaction and restricted, repetitive, and stereotyped behavior, interests, and activities (American Psychiatric Association, 2000). The primary difference between the two diagnoses is the presence of the third characteristic of impaired communication in the diagnostic criteria for Autistic Disorder, but not for High Functioning Autism (HFA). Asperger's Disorder is also typically marked by higher intelligence. While ASD can vary greatly in the symptomology and severity, to receive a diagnosis, there must be the presence of impaired social interaction. This is manifested in difficulties, such as the use of nonverbal behaviors (eye contact, facial expression, body posture, and gestures), ability to develop and maintain reciprocal relationships, and the ability to spontaneously share interests or things of importance with others, to name a few. Even with higher cognitive ability and functional communication skills, there is a noticeable impact on social relatedness for people with ASD. This impairment further impacts areas in academic, behavioral, and emotional functioning (Bellini, Peters, Benner, & Hopf, 2007). Without the acquisition of these skills, children can experience detrimental effects in multiple areas of life functioning. While those with ASD can experience a range of deficits in various areas of functioning, many consider the social impairments and inability to relate to others as the central characteristic of ASD (Fein, Pennington, Markowitz, Braverman, & Waterhouse, 1986). The first description of autism provided by Kanner (1943) described the core deficit of the disorder as being social impairment. It also remains in the current 3 diagnostic criteria as a core deficit for both autism and Asperger's disorder (American Psychiatric Association, 2000). The impairments in social interaction can be manifested both verbally and non-verbally. Verbally, impairments can be seen in the inability to understand abstract language and have meaningful conversations. Nonverbally, they can be seen in lack of eye contact, inability to read social cues, and joint attention. Both verbal and nonverbal impairments strongly impact the ability to effectively interact socially and relate to others. As children with ASD reach school age, they often experience negative effects in many areas of functioning due to their social deficits. Children have been identified as demonstrating a lack of awareness of others, having impaired friendships, and a lack of imaginative play (Stone, Hoffman, Lewis, & Ousley, 1994). Due to these deficits in social skills, many children with ASD have been found to be more lonely than non-ASD peers and also have less awareness and understanding of loneliness (Bauminger & Kasari, 2000). This can greatly affect the child's mental health, but also inhibit the opportunity to interact with others and gain skills needed for normal development, including language development and intelligence that is based on experience and exposure. It has been found that deficits in social skills can lead to poor school achievement, cognitive deficiencies, mental health problems, and higher rates of unemployment in adulthood (Howlin, Mawhood, & Rutter, 2000; Strain & Schwartz, 2001). There are many social behaviors that can have an effect on an individual's level of functioning. Caldarrella and Merrell (1997) identified broad behavioral dimensions that 4 include social skills that children, including those with ASD, may have deficits in and should be used for the identification and treatment of children who are lacking some of these socially appropriate behaviors. These broad dimensions are the following: 1) peer relational skills, 2) self-management skills, 3) academic skills, 4) compliance skills, and 5) assertion skills (Caldarrella & Merrell, 1997). Table 1 provides examples of the social skills included in each of the five broad dimensions identified by Caldarrella and Merrell (1997). Table 1 Caldarrella and Merrell's Taxonomy of Pro-Social Behaviors _______________________________________________________________________ Specific Skills within Each Dimensional Area of Pro-Social Behaviors _______________________________________________________________________ Peer relations Complimenting peers, providing needed assistance, initiating social interactions Self-management Controlling emotional states, following rules, compromising, receiving feedback appropriately Academic Assignment completion, independence, adherence to teacher direction Compliance Following rules and directions Assertion Beginning conversations, accepting compliments, initiating play, establishing friendships, self-confidence _______________________________________________________________________ 5 Evidence-Based Practice Due to the wide range of detrimental effects that social skills deficits can have on a child with ASD or any child with social deficits, there has been a large focus from researchers on developing social skills interventions that are effective. Social skills interventions are widely used in schools and clinical programs for children with social deficits as an attempt to improve their levels of functioning. It is necessary to further study and develop programs that are effective in improving the skills necessary for individuals to thrive in society, socially, and academically. The National Association of School Psychologists (NASP) supports the need for use of evidence-based practice (EBP) by school psychologists. Hoagwood and Johnson (2003) define evidence-based practice as "a body of scientific knowledge, defined usually by reference to research methods or designs, about a range of service practices" (p. 4). Cournoyer and Powers (2002) recommend that the way school psychologists make decisions and provide services be based on the use of evidence-based practices. This means that practitioners use services that have research indicating that the intervention is likely to be beneficial to the person you are using it for and that the practitioner will measure the effects of the intervention on the individual throughout treatment. By doing this, the intervention is likely to produce predictable effects that are beneficial for the individual. Kratochwill and Shernoff (2003) identified five things that are needed to effectively utilize evidence-based practice. The first is that there is collaboration between researchers, trainers, and practitioners to ensure that the interventions being developed are effective in practice environments. The second need is for practitioners to use manualized treatments to increase the treatment fidelity and the likelihood of efficacy 6 when transferring intervention implementation from a research setting into practice. Along with the need for practitioners to use a manual for implementation, it is suggested that more specific practical guidelines be provided to make treatments even more effective. The fourth consideration is the need for professional development for graduate students, trainers, and practitioners to help them make better applications of the interventions to specific practical settings. Finally, it is recommended that a scientist-practitioner model is most effective in supporting the development and research of interventions in practical settings. The American Psychological Association (APA) also provides guidelines for the development, evaluation, and use of evidence-based practice. APA's Presidential Task Force on Evidence-Based Practice (2006) defined evidence-based practice as "the integration of the best available research with clinical expertise in the context of patient characteristics, culture, and preferences" (p. 273). This definition is also very similar to the definition of evidence-based practice as defined by the Institute of Medicine (2001). The American Psychiatric Association also developed similar guidelines to help physicians with decision-making about the best form of treatment for patients. There is consistency between the definitions and this common language may foster a higher level of integration between the medical and mental health communities. Also, this definition and the guidelines set into place by APA identifies specific goals to make mental health services more cost effective and to make practitioners more accountable for their actions and treatments. Similarly to NASP, APA recognizes that collaboration between researchers and practitioners is essential to developing and implementing evidence-based practices. 7 APA Division 12 (Clinical Psychology) and Chambless et al. (1998) have developed criteria for evidence-based practice by delineating between well-established treatments and probably efficacious treatments. Well-established treatments can be determined as such in one of two ways. The first requires at least two good between-group design experiments demonstrating efficacy in that it is more effective than a psychological placebo, other treatment, or an already well-established treatment. The second way in which a well-established treatment can be identified is through a large series of single-case design experiments, with 9 or more participants, which demonstrate efficacy. These single-case design experiments must have used good experimental designs and provide evidence of change by comparing the intervention to another treatment. Both of these methods to identify well-established treatments require treatment manuals, clearly specified client sample characteristics, and effects demonstrated by at least two investigators. Probably efficacious treatments require two experiments that indicate the treatment is superior to a waiting-list control group or a small series of single-case design experiments with 3 or more participants and otherwise meeting the criteria for a well-established treatment. The guidelines set up by Division 12 give very specific criteria for practitioners to determine the efficacy of interventions for their patients and researchers to develop well-established interventions. While APA Division 12 and Chambless et al. (1998) provided very clear guidelines for the classification of evidence-based practice, O'Donohue and Ferguson (2006) have identified some weaknesses in this system of classification. In the criteria previously defined, the determination of evidence-based practice is based on statistical significance rather than clinical significance. Statistical significance is determined by how much 8 chance affects a difference in the results, but clinical significance refers to how meaningful a change is to the client. The second weakness argument against this determination of evidence-based practice is that the decisions are based on efficacy (the treatment is beneficial for patients) rather than effectiveness (if an efficacious treatment will not only be effective in research settings, but also in community settings/private practice). The third weakness found in the EBP criteria is concerned with the issue of heterogeneity versus homogeneity. Most studies that meet this criterion exclude subjects who present with comorbid diagnoses, although in community and private practice settings, the patients are often presenting with multiple conditions. The fourth and last weakness identified by O'Donohue and Ferguson recognizes the bias against inclusion of single-subject, withdrawal, and multiple-baseline research designs due to the requirement of inferential statistics and comparison to a control group. Many groups specific to education have also provided information in regards to evidence-based practice. The Council for Exceptional Children (CEC; Odom et al., 2005) is aiming to help identify some criteria specifically to assist teachers in the identification of research-based practice. Currently, there is not a lot of research on practices that meet EBP criteria and are also relevant for use in an educational setting. The Department of Education has attempted to provide more educationally relevant criteria to determine if practices are evidence-based. The Institute of Educational Sciences (IES) (2003) has recognized criteria for interventions to be considered as having "strong" evidence or "possible" evidence of effectiveness. In order to meet the criteria for having "strong" evidence, an intervention must have been effective in well-designed and implemented randomized controlled trials in two or more typical school settings. 9 "Probable" evidence of effectiveness is found in studies with randomized controlled trials, but may not be able to meet the stringent requirements for having "strong" evidence. The IES places a great deal of importance on randomized controlled trials as a research method. Aspects of the randomized controlled trials that the IES also discusses with strong emphasis are accurate outcome measures, long-term outcomes, detailed descriptions of treatment groups, indication that the intervention groups are systematically equal, and that the results are statistically significant. These criteria should all be met before an educational intervention is considered to have "strong" research evidence of effectiveness. The American Speech-Language Hearing Association (ASHA, 2004) has developed criteria similar to those developed by APA Division 12 as a way of determining the level and amount of research supporting an intervention. They categorize interventions into levels ranging from Level I to Level IV. Level I includes interventions that have been studied through a meta-analysis with one or more studies having randomized designs, Level II can include controlled studies and quasi-experiments, Level III includes case studies and other nonexperimental designs, and Level IV would include interventions without research, but with expert support. Similar to the guidelines developed by the APA, these guidelines give a clear division between the necessary level of research support for interventions and a means to gauge the likely degree of expected outcomes. This is very helpful to educators as they attempt to make decisions about how to best serve their students. Specific to children with autism, the National Autism Center (NAC, 2009) has defined three categories of interventions, Established, Unestablished, and Emerging. 10 Some of the criteria for Established treatments include having research providing evidence of beneficial effects, the expectation of long-term beneficial effects, and evidence that the treatment does not produce harmful effects. While these treatments have been shown to be effective, they should not be expected to be effective for every child and multiple types of treatments may need to be tried before the most effective treatment for that individual is found. Unestablished treatments differ from Established treatments in that there is little or no evidence to support them. Unestablished treatments also may have been found to produce little positive effect or negative effects in the research. Emerging treatments are those that do not have enough research to support effectiveness or lack of effect as an intervention. These should be used with caution as they do not have enough empirical support to determine how effective or detrimental they may be. These guidelines require the practitioner to take a lot of responsibility and use good professional judgment when choosing treatments for patients. Meta-Analyses Meta-analyses are used as a means to combine the results of several studies in order to better determine the degree of effectiveness of similar interventions. According to Glass (1976), they are used as a "statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings" (p. 3). Meta-analyses are used because it can be very difficult to detect statistically significant results from individual studies. Oftentimes, there are not enough participants in individual studies to provide the statistical power needed to show large effects. As suggested by Collins et al. (1992), in order to prove that a drug is designed to reduce the risk of disease 11 by 10%, a sample size of 10,000 would be needed in each of the treatment groups to detect any effect with 0% accuracy. By combining individual studies into a meta-analysis, the ability to detect statistical significance is increased. Blimling (1988) identified four main purposes of meta-analyses: to describe existing studies of a treatment, to determine overall effectiveness of the treatment, to determine influences in the outcome of the treatment, and to quantify the outcome in terms of magnitude and significance. Davis and Crombie (2001) also pointed out some advantages of using meta-analytic research. Using meta-analyses allows people to see the average effects from multiple applications of similar interventions by producing a larger number of participants than the individual studies. This process typically reflects more accurate effects because of the larger sample being used. Another benefit is that meta-analyses are typically more objective than traditional studies and reviews that can often be biased by the researcher or reviewer. In order to maintain the integrity of the results obtained from meta-analyses, it is essential to follow the process that is defined for conducting a meta-analysis. The process starts when the researcher develops a question and defines inclusion criteria for the studies that will be used. By developing the inclusion criteria at the beginning, the researcher is unable to later exclude studies based on personal preference, thus increasing the objectivity of the studies used. The studies that are chosen should have methodological soundness and enough data provided to compare between the studies. Meta-analyses are an objective and highly effective way to evaluate the efficacy of interventions. They provide more accurate information based on the results of multiple research studies and they provide more guidance toward possible areas for future 12 research. Meta-Analyses of General Interventions for ASD Meta-analyses have been considered optimal in the medical research literature for quite a while and are now also being considered as such in the psychological research literature. There are many meta-analyses that have been focused on various interventions for children with ASD. One recent meta-analysis by Eldevik et al. (2009) focused on the effectiveness of a behavioral intervention for young children with ASD, the Early Intensive Behavioral Intervention (EIBI). While behavioral interventions have been considered an effective intervention for children with autism for many years (Eikeseth, 2009; Lovaas, 1987; Rogers & Vismara, 2008), EIBI has also been found to have a large effect size for changes in multiple areas of functioning. The meta-analysis by Eldevik et al. expanded previous research by concluding that EIBI produced a large effect size (1.103) for changes in IQ and moderate effect size (0.660) for changes in adaptive behavior. Another meta-analysis by Spreckley and Boyd (2009) evaluated the effectiveness of an early intervention, Applied Behavioral Intervention (ABI), in many areas of functioning, including cognitive, language, and adaptive behavioral skills. This study concluded that ABI produced moderate effect sizes for cognitive skills (0.38), language skills (0.37), and adaptive behavior skills (0.30). This meta-analysis was only based on the results from four individual studies and a more comprehensive meta-analysis may produce more conclusive results. Hourmanesh (2006) was able to conduct a meta-analysis that included 16 studies on 13 early interventions for children with ASD. Results of this more comprehensive meta-analysis found large effect sizes for cognitive skills (0.64), language skills (0.61), and adaptive behavior skills (0.68). This study was able to support the effectiveness of early behavioral interventions and produce even larger effect sizes with the inclusion of more studies. Backner (2009) also conducted a meta-analysis on early comprehensive behavioral interventions for children with ASD. In addition to the previous 16 research studies used by Hourmanesh, Backner included nine more articles. Backner found that the studies included in this meta-analysis produced a moderate effect size for cognitive skills (0.64), a large effect size for language skills (0.80), and an effect size of 0.28 for adaptive behavior skills, which was much smaller that the results from the Hourmanesh meta-analysis. Overall, early comprehensive behavioral interventions have been found to be effective for children with ASD. While there is some discrepancy between the size of the effect for different skill areas, the interventions would largely be considered as producing moderate effect sizes in general. Meta-Analyses of Social Skills Interventions for ASD Some research has indicated that existing social skills programs are not effective for the majority of children with ASD and the effects may not be generalizable to multiple settings. DuPaul and Eckert (1994) found that many social skills programs were ineffective because the skills being taught were not generalizable into natural situations where use of the skill would be beneficial to the child. DuPaul and Eckert also found that 14 performance deficits impeded the generalization of the skills because the knowledge of the skills was acquirable, but the self-control and impulsivity levels of the children kept them from being able to utilize the skills appropriately in actual situations. This would imply that social skills would need to be taught to children who were able to overcome the performance deficits that might be present in order for there to be positive and generalizable effects. The meta-analysis by Bellini, Peters, Benner, and Hopf (2007) focused on school-based social skills programs for children with ASD. Their study measured the effects of social skills training on children's group play, social initiations, and social responses. The results indicated that the interventions implemented in the schools produced moderate maintenance effects and low generalization effects of group play, social initiations, and responding behaviors for the participants. Bellini also found that the social skills training was less effective when taught outside of the natural setting. Bellini and Akullian (2007) conducted a study that focused on the effects of video-modeling and video self-modeling when used to teach social skills to children with ASD. Their meta-analysis included studies that measured the effect of modeling training on social-communication skills, functional skills, and behavioral functioning. Overall, the studies produced moderate effects for the three variables, with the Percentage of Non- Overlapping Data Points (PND) being 80%. Specifically for effects in the dependent variables, functional skills had the highest PND of 89%, social-communication skills had a PND of 77%, and behavioral functioning had a PND of 76%. Unlike previous research, maintenance effects had a PND of 83% and generalization effects had a PND of 74%, indicating moderate effects in these areas. Little difference was found between the 15 effects of video-modeling and video self-modeling. Zhang (2008) conducted research on the effects of using peers to mediate social skills interventions for children with ASD. Peer mediation as an intervention was found to have a large effect size of 1.46, follow-up results also had a large effect size of 1.49, as did generalization with an effect size of 1.51. Miller (2006) also found peer mediation to be an effective form of social skills intervention. This meta-analysis found peer mediation had a large effect size of 3.27, as did collateral skill interventions (ES=2.37) and child-specific interventions (ES=2.19). Both of these studies provide support for use of peer-mediated interventions as an evidence-based practice when used to teach social skills to children with ASD. A meta-analysis by Lee, Simpson, and Shogren (2007) evaluated the effectiveness of many self-management techniques. The techniques included in this study were self-monitoring, self-assessment, self-evaluation, self-observation, self-recording, self-instruction, and self-reinforcement. Results combined for all forms of self-management produced a PND of 81.9%. The results of this study imply that self-management may be an effective intervention for children with ASD. The current research in social skills programming for children with ASD indicates that there is a lack of effective interventions and a great need for the development of more effective social skills interventions. There are many different methods that have been used to attempt to make social skills interventions more effective. In recent research, self or peer video-modeling (Bellini & Akullian, 2007; Bellini, Akullian, & Hopf, 2007; Charlop-Christy, & Danshevar, 2003; LeBlanc et al., 2003; MacDonald, Clark, Garrigan, & Vangala, 2005; Sherer et al., 2001), peer mediation (Miller, 2006) and 16 social stories (Hagiwara & Myles, 1999; Thiemann & Goldstein, 2001) have been used to increase the efficacy of the social skills programs. Overall, social skills programs for children with ASD are ineffective and not generalizable. Up to this point, there have been some advancements made in the research concluding the efficacy of video-modeling, peer mediation, and social stories as being important in increasing the effectiveness of various social skills training programs. Video-modeling, peer mediation, and social stories are found in the research as being helpful to some children in learning, generalizing, and maintaining social skills (Bellini & Akullian, 2007; Bellini, Akullian, & Hopf, 2007; Charlop-Christy, & Danshevar, 2003; Hagiwara & Myles, 1999; Miller, 2006; Thiemann & Goldstein, 2001). The use of these components can be effective for children learning social skills in a group setting. While there is not a complete consensus in the literature that these interventions are helpful to all children, it is encouraging that they are effective for some when used alone and could be even more effective when combined with other evidence-based approaches. Current Social Skills Programs Social skills are an important part of development for children and there are many populations that do not naturally acquire them, such as people with ASD, depression, or conduct disorder. Programs targeted at helping children develop functional social skills have been developed and used for many groups, including children with depression, behavior disorder, anxiety, and ASD, although the research does not always indicate positive or neutral results of social skills training. 17 Many social skills programs have been developed, but in current research, they have not been found to be effective in increasing social skills or helpful in generalizing skills across settings (Arnold & Hughes, 1998; DuPaul & Eckert, 1994). There are also many meta-analyses that have been conducted to determine the effect size of social skills training on children. Many studies have found the effect size to be small according to Cohen's (1988) measurement for effect sizes, which indicates that below 0.20 is a small effect size. One study aimed at determining the effectiveness of social skills training for children with conduct disorders not only failed to show benefits from group social skills training, but actually showed evidence of detrimental effects (Arnold & Hughes, 1998). In social skills groups for children with behavior disorders, the undesirable behaviors were actually enhanced due to the encouragement and experience sharing between the individuals in the group. Thus, social skills interventions for children with behavior disorders are not deemed effective unless there are neutral peers without behavior disorders in the groups as well. However, another study by Beelman, Pfingsten, and Losel (1994) found that social skills programs taught with children who have Externalizing Disorders had a moderate effect size of 0.48. These effects were not maintained over time, indicating that while a higher effect size was produced initially, the long-term effects of the social skills training was not beneficial for the participants. Quinn, Kavale, Mathur, Rutherford, and Forness (1999) found the effect size of social skills instruction for children with emotional and behavioral disorders to be 0.199. However, the same study did find higher effect sizes for children with anxiety, suggesting that social skills training may be effective for some groups of children. The moderate 18 effect size of 0.422 for social skills training for children with anxiety in this meta-analysis was based on eight individual studies. A more recent meta-analysis conducted by Spence, Donovan, and Brechman-Toussaint (2000) also provides support for use of social skills for children with anxiety. Their study paired social skills training with cognitive-behavioral therapy to reduce school-related anxiety. The results of this study indicate that this treatment was effective and was able to be maintained at 12 months after the completion of treatment. Forness and Kavale (1996) conducted a meta-analysis on social skills programs for children with learning disabilities and found small effects. This meta-analysis included 83 independent studies done to determine the effect of social skills training for children with learning disabilities. The average effect size for all of these studies was 0.21. In a later review of this and other meta-analyses, Forness (2001) found that another meta-analysis (Quinn et al., 1999) also reported a small average effect size of 0.20 for the 35 studies that had met inclusion criteria. Many of the social skills programs that are currently used in schools and clinical settings share commonalities in their focus and their method of instruction. McConnell (2002) divided the current social skills programs into five categories. The first category is environmental modification strategies. The focus of these types of interventions is on making changes to the environment in order to encourage social interactions. The second category of interventions is collateral skills interventions that teach skills, such as play and language, to improve social interaction. Another type of intervention is peer-mediated interventions that use trained typically developing peers to teach skills and encourage social interaction. Child-specific intervention is another type of intervention 19 that teaches specific social skills to children for them to use in their social interactions. The last type of intervention described is comprehensive interventions that combine two or more types of the interventions previously discussed. There are a large number of programs that are marketed for use as social skills curriculum and intervention programs for various populations. Some of the programs have been developed based on research and some have no empirical basis. Table 2 identifies some popular social skills programs and curriculum that are available commercially to be used with multiple populations. The current research on the use of general social skills programs for children with various disabilities does not seem to provide a large amount of evidence for their effectiveness. While some of the groups showed positive effects, many did not. There are many programs that may be effective for various groups of children, but have not yet been the focus of research at this point. This is an area in great need of more research to support decisions being made for programs developed for and conducted with children. Social Skills for Children with ASD Due to the fact that social skills are considered the core deficit of children with ASD, there has been an attempt to develop a social skills curriculum specifically for this population. There are many programs currently available that vary in content and design, but typically all have an adult who didactically teaches the content to a group of children with ASD. The current research in social skills programming for children with ASD indicates that there is a lack of effective interventions and a great need for the development of more effective social skills interventions (Bellini, 2007). 20 Table 2 Common Social Skills Training Programs for Youth _______________________________________________________________________ Current Social Skills Programs _______________________________________________________________________ The ACCEPTS Program Walker, McConnell, Holmes, Todis, Walker, & Golden,1983 ASSET Hazel, Schumaker, Sherman, & Sheldon-Wildgen, 1981 Skill Streaming Goldstein & McGinnis, 1984 Prepare Curriculum Goldstein, 1988 Aggression Replacement Training Goldstein & Glick, 1986 Cool Kids Fister-Mulkey, Conrad, & Kemp, 1998 Tough Kids Social Skills Book Sheridan, 1995 Social Competence Intervention Guli, Wilkinson, & Semrud-Clikeman, 2008 Program _______________________________________________________________________ 21 Currently, there are many manualized social skills programs that are used in schools and clinical settings. The majority of these programs are research-based. Table 3 lists a few of the programs that are widely used in schools and clinics. All of the programs listed are widely used by practitioners and many of them have research supporting their programs. Many of the components that have been found to be effective in these programs have been incorporated into the Superhero Social Skills program, which is the focus of this study. Gray (1994) has published multiple books about social stories, which are now widely used in social skills curriculum. Social stories are developed and used by writing a story that incorporates use of the target skill in a specific situation. The child then learns how to use the skill by reading the story or having it read to them. Social stories are incorporated in the Superhero Social Skills program in the form of comic books featuring the characters from the curriculum. Madrigal and Winner (2008) have developed a social skills curriculum that incorporates a "superhero" theme, but with a focus on social thinking. They utilize specific characters that have to learn skills to overcome a certain social skill deficit. The program in this study uses two superheroes and a sidekick to teach the specific social skills to children. Bellini (2007) found video-modeling to be an effective means to teach social skills to children. By using peer video-modeling, children are able to learn the social skills better than if an adult teaches the lessons in a didactic format. This program also uses video-modeling with peers who are shown using the skills being taught in multiple situations. All of these components and others were incorporated into the Superhero Social Skills program as a way to attempt to produce an effective curriculum for children 22 Table 3 Social Skills Programs for Youth with Autism Spectrum Disorder ______________________________________________________________________ Manualized Social Skills for Children with ASD _______________________________________________________________________ Navigating the Social World: A Curriculum for Individuals with Asperger's Syndrome, High Functioning Autism, and Related Disorders (McKinnon & Krempa, 2005) Building Social Relationships: A Systematic Approach to Teaching Social Interaction Skills to Children and Adolescents with Autism Spectrum Disorders and Other Social Difficulties (Bellini, 2006) Social Skills Training for Children and Adolescents with Asperger Syndrome and Social Communication Problems (Baker & Myles, 2003) Social Skills Solutions: A Hands-on Manual for Teaching Social Skills to Children with Autism (McKinnon & Krempa, 2002) S.O.S. Social Skills in Our Schools: A Social Skills Program for Children with Pervasive Developmental Disorders, Including High-Functioning Autism, and Asperger Syndrome and Their Typical Peers (Dunn, 2006) Think Social: A Social Thinking Curriculum for School-aged Students (Winner, 2006) Designing Comprehensive Interventions for Individuals with High-Functioning Autism and Asperger Syndrome: The Ziggurat Model (Aspy & Grossman, 2008) The New Social Story Book (Gray, 1994) Superflex: A Superhero Social Thinking Curriculum Package (Madrigal & Winner, 2008) _______________________________________________________________________ 23 with ASD. Video-Modeling Interventions Video-modeling was incorporated into the Superhero Social Skills program as one component to help increase its effectiveness. Video self-modeling is implemented by having the target child watch a video of themselves performing the desired skill without error, whereas video-modeling is the process of watching a video of a peer demonstrating appropriate use of the skill or behavior. The child is shown the videos repeatedly and this has resulted in changes in behavior, maintenance, and generalization. Charlop-Christy and Daneshvar (2003) concluded that generalization increased when using video-modeling and believe that the video stimulus is reinforcing and possibly helpful in controlling overstimulation for children with ASD because the video presentation helps to focus attention on one stimulus. Research has found video-modeling is more effective than in-vivo modeling and it is also a cost effective alternative to other forms of training (Bellini & Akkulian, 2007; Miller, 2006). Bellini et al. (2007) found that children with ASD demonstrated increased social engagement that was maintained over time as the result of video self-modeling. Bellini, Akullian, and Hopf (2007) also found that video self-modeling not only increased the effectiveness of social skills training, but that the effects were maintained after the intervention was completed. The maintenance of positive results following treatment, as well as generalization to other individuals and settings was also found by Sherer et al., (2001). Nikopoulos (2007) found increased interaction time and generalization of play skills to new toys and settings for children with ASD after viewing video-modeling of 24 typically developing peers. The generalization of play was also maintained for up to 3 months. While there are many studies that have indicated high effects of video self-modeling, there have also been studies that compare video self-modeling to video-modeling. Results indicate that both forms of modeling produce moderate to large effect sizes, suggesting there is little difference in effectiveness between them (Bellini et al., 2007; Sherer et al., 2001). Thus, video-modeling, either self or peer, is now considered an effective and important component of social skills training for children with ASD. Peer Mediated Interventions The use of peer mediation in social skills interventions for children with ASD has been used to counteract the poor generalization of social skills taught through didactic instruction delivered by adults (Rogers, 2000). Studies have concluded that peer mediated programs are an effective way to teach social skills; however, researchers have found the effects are difficult to maintain because children tend to rely on the peer cues and prompts (McConell, 2002; Rogers, 2000). Miller's (2006) meta-analysis indicated that peer-mediated interventions are the most effective for school age children with ASD when learning social behaviors. Many of the social skills programs currently available do not use peer modeling as part of their instruction and this may prove to be a component that is useful in the development of future social skills training programs. 25 Self-Management Interventions Self-management is used to teach children to monitor and record their own behavior by increasing their awareness of the behavior and their use of the behavior in multiple and unsupervised settings. Stahmer and Shreibman (1992) implemented self-management interventions to children with ASD in order to increase appropriate play behaviors. They found that self-management increased the use of appropriate play, decreased self-stimulatory behaviors, and that the results were maintained and generalized to unsupervised settings. Koegel, Koegel, Hurley, and Frea (1992) found similar results when teaching self-management to children with ASD in an attempt to increase social responsiveness and decrease disruptive behavior. The self-management training had the desired effects on the individuals and it generalized to multiple settings (school, home, and community) without the treatment provider present. Self-management is another component that has been found to be effective for children with ASD and may prove to be an essential component of social skills training programs. Social Stories Social stories have also been studied as an effective component of social skills training. Social stories are stories created to reflect realistic situations that might require the use of skills being taught and demonstrating how these skills can be used appropriately in various social situations. Social stories also often include pictures, which can be helpful for children with ASD who benefit from the use of visual formats. Quirmbach, Lincoln, Feinberg-Gizzo, Ingersoll, and Andrews (2009) found that the use of social stories significantly improved play behavior for children. Hagiwara and 26 Myles (1999), however, did not find consistent and significant results for the participants in their study; rather, the effects were only found for outlier participants. However, for the participants that Hagiwara and Myles found benefited from the intervention, the effect generalized to other situations and could be linked to the skills. Social stories have been shown in the research to be an effective strategy for teaching social skills, but may not be as effective when used as the only form of intervention (Crozier & Tincani, 2007; Sansoti, Powell-Smith, & Kincaid, 2004). It is likely helpful to combine this intervention technique with others when developing social skills programs. Due to their lack of social skills and stereotyped or repetitive behaviors, children with ASD are often bullied by their peers. There have been very few published reports on the rates of bullying among children with ASD. One report by Little (2001) indicated that up 75% of adolescents with Asperger's Disorder in general education settings were bullied and these children are bullied four times more than typically developing peers. Van Roekel, Scholte, and Didden (2009) found the rates of victimization among adolescents with ASD in special education settings to be between 7 and 30%, which is much lower than the rates of victimization in general education settings. These rates were based on teacher and peer ratings of bullying experienced by the children with ASD. They also found that 26% of children with ASD have been identified as bullies who victimize other children. The same study evaluated the perception of bullying for adolescents with ASD and found that the participants with ASD were able to identify bullying situations as accurately as the typically developing peers. Children with ASD were also more likely to identify positive or neutral interactions as bullying interactions. This study is important 27 to the literature on ASD because it identifies a distinct need for more research in the area of the bullying experiences of children with ASD. It also identifies a need for future programming for children with ASD to incorporate skills to help children recognize and respond to bullying as a component of what is taught. Social Skills Training at Home and School For many years, there has been a shift from clinic-based treatment of Autism to home- or school-based treatments. Howlin et al. (1973) promoted the use of parents as the person delivering the intervention to their children rather than a therapiet, as well as basing the treatment in the home rather than in a clinical setting. There can be many benefits to treatment taking place in a naturalistic setting in which the child would be expected to use the skills they are learning. Krasny, Williams, Provencal, and Ozonoff (2003) identified essential aspects of treatment for children with ASD. They include generalization as an essential component of social skills, which can be encouraged through community outings, skill practice in more naturalistic settings, and collaboration with parents and teachers to work on skills outside the group intervention. By incorporating these aspects into treatment, it may still be possible to provide effective social skills training in a clinical setting. Barry et al. (2003) recognized that there are many out-patient clinic settings in which social skills are being taught to children with ASD, but very little research to support this practice. In this study, 4 children with ASD were taught specific social skills and then observed to identify any improvements in greeting, conversation, and play skills. Results from this study indicate that there was an improvement in greeting and 28 play skills during play observations with typical peers, but little improvement in conversation skills. The typical peers were only present during the play observations and not during the social skills training. Self-report from the children with ASD indicated that they felt more social support from peers at school after participating in the social skills training. Parent reports indicated that only greeting skills had improved in the generalized setting. Research indicates that teaching social skills in a naturalistic setting can be beneficial to children and that there is generalization of the social skills in settings where they would use the skills they have learned. There is also evidence that by supplementing clinic-based treatment with opportunities to generalize social skills and incorporating parent and teacher participation in providing opportunities to practice these skills, clinic-based treatment can also be effective for children's acquisition of social skills. Superhero Social Skills Superhero Social Skills was developed based on past research in order to incorporate many previously discussed evidence-based components of existing social skills programs into one program (see Appendix A, Program Overview). Some of the components that are used in this program include video modeling with an optional video self-modeling component, peer mediation through the inclusion of typically developing "peer buddies," self-management of the child's use of the learned skills, and social stories in the form of comic books. The program was developed to teach social skills to children with Autism, Asperger's Syndrome, or Pervasive Developmental Disorder-Not Otherwise Specified. Many existing programs are effective, but lack maintenance effects and 29 generalization of the skills that were taught. One of the main goals of this program is not only to effectively teach social skills to children, but for the skills to be maintained and generalized. Superhero Social Skills includes 18 skills separated into foundational, intermediate, or advanced skills based on the complexity of the skill. Each of the 18 skills is typically taught twice per week, but were combined into a longer session taught once per week for this study (see Appendix B, Sample Lesson). Skills are introduced by the superheroes (The Initiator and Interactor Girl) and their sidekick (Scooter the robot) in an animated video at the beginning of the lessons. The lesson format also includes role-playing social scenarios by pairing children with ASD and typically developing peers. Participants then watch a digital comic social story with a hard copy provided to the participants at the end of the lesson. The video animation and comic books make this program of high interest for children, but while still incorporating evidence-based components to encourage skill acquisition, maintenance, and generalization by the children. In addition to these components, the lessons include social games that reinforce the skills being taught in an enjoyable format. Reinforcement strategies to encourage rule-following behavior and compliance are used throughout the lessons. The goal in the development of Superhero Social Skills was to incorporate multiple components that have met the criteria for evidence-based practice. According to the EBP criteria for the National Autism Center (2009), this study has incorporated several evidence-based practices including modeling and video-modeling, peer mediation, self-management procedures, social stories, and direct instruction. The use of multiple evidence-based practices into one program makes the potential for efficacy 30 favorable when compared to other social skills programs. These same components are also likely to overcome some of the shortcomings of other existing programs, including generalization of social skills. Summary In summary, there are many programs that have been developed to aid children in learning and using social skills, but many have been found to have little, if any, effect. Despite the small effects of social skills programs currently being taught, many schools and clinical settings still provide social skills training due to the negative outcomes associated with poor social skills. It is necessary for programs to be developed and research conducted to identify evidence-based social skills programs for children. Superhero Social Skills has been developed to meet these criteria by combining many of the components of other programs that have been proven effective in the research. Along with evidence-based practices, the Superhero Social Skills Program also uses video animation and other high-interest media to increase the children's level of interest and attention to the material. The present study was conducted to evaluate if the Superhero Social Skills program is an evidence-based approach to teaching social skills when delivered once per week in a clinical outpatient setting for 4 children with ASD. Efficacy will be determined by calculating effect size and percentage of nonoverlapping data points for each participant. 31 Statement of Purpose This study was designed to evaluate the use of the Superhero Social Skills program as an evidence-based practice to teach social skills to children with ASD in a clinical out-patient setting. The purpose of this program is to provide children with the appropriate social skills necessary for participation in pro-social interactions with peers and adults. Another goal of this study is to measure generalization of the skills to multiple situations and settings, such as home and school. The program is based on a superhero theme with animation and comic books as high interest media to appeal to participants and maintain attention to program content throughout the intervention. The program also incorporates research-validated components, such as video-modeling, social stories, peer mediation, and self-management strategies. The program effectiveness was determined by increased use of social skills during observation periods, increased pro-social behaviors in a generalized setting, acquisition of Power Charges on cards during sessions and at home for appropriate use of skills, acquisition of Blackhole and Scooter Cards during sessions, and completion of checklists including the BIRS, SRS, and the Children's Consumer Satisfaction Survey. Research Questions The following research questions were addressed in this study: 1. What is the effectiveness of the social skills intervention as measured by the spontaneous use of social skills when participants are observed during free play in analog free time as measured by an adaptation of a developed observation scale (Bellini, 2007) (see Appendix C)? 32 2. What is the effectiveness of the social skills intervention's generalizability as measured by the spontaneous use of social skills as observed by the parents and reported through an adapted version of the Parent Daily Report (PDR) (Chamberlain & Reid, 1987) (see Appendix D) and reported through a parent telephone interview? 3. What is the effectiveness of the social skills intervention as measured by self-recording using the Power Cards and the number of Scooter and Blackhole Cards earned during intervention sessions? 4. What is the improvement in rule-following behavior during training as measured by the participants earning Scooter Cards and Blackhole Cards over time? 5. What is the consumer satisfaction with the intervention as reported by the parents rating on the Behavior Intervention Rating Scale (BIRS)? 6. What is the effectiveness of the intervention based on the results of the Social Responsiveness Scale (SRS) completed as a pre- and posttest measure? 7. What is the social validity of this intervention as rated by an adaptation of the Social Validity Scale (Bellini, 2006) (see Appendix E)? 8. What was the participant satisfaction with the intervention based on a child consumer satisfaction survey (see Appendix F)? 9. What is the effectiveness of the intervention based on the results of the Autism Social Skills Profile (ASSP) (Bellini, 2007) (see Appendix G)? CHAPTER 2 METHOD This study was designed to evaluate the use of the Superhero Social Skills program as an evidence-based practice to teach social skills to children with ASD in a clinical outpatient setting. The purpose of this program is to provide children with the appropriate social skills necessary for participation in pro-social interactions with peers and adults. Another goal of this study is to measure generalization of the skills to multiple situations and settings, such as home and school. The program is based on a superhero theme with animation and comic books as high interest media to appeal to participants and maintain attention to program content throughout the intervention. The program also incorporates research-validated components, such as video-modeling, social stories, peer mediation, and self-management strategies. The program effectiveness was determined by increased use of social skills during observation periods, increased pro-social behaviors in a generalized setting, acquisition of power charges on cards during sessions and at home for appropriate use of skills, acquisition of Blackhole and Scooter cards during sessions, and completion of checklists including the BIRS, SRS, and the Children's Consumer Satisfaction Survey. Prior to recruitment of participants, consent to conduct the research study at the University Neuropsychiatric Institute (UNI) was obtained. Approval by the University of 34 Utah Institutional Review Board (IRB) was also obtained by the researcher. Participants This study was conducted with 4 children between the ages of 5 and 10 who were recruited from the Utah Autism Research Project, University Neuropsychiatric Institute (UNI), and Salt Lake area elementary schools. The researcher recruited participants by placing flyers and posters (see Appendix H) in Salt Lake area elementary schools, the University Neuropsychiatric Institute (UNI), and at the Utah Autism Research Project. These facilities were considered high traffic areas for families with children who have ASD. Interested parents contacted the researcher and were given more detailed information by phone. If the parent wanted to have their child participate in the program, they met with the researcher to complete the parental consent (see Appendix I) and child assent forms (see Appendix J) and provide past testing required for inclusion criteria. The parents also completed the GADS, BIRS, and SRS during this initial meeting. Each participant was required to provide a peer to attend all sessions with them. All of the children with ASD were required to meet the following inclusion criteria. In order to be included as a participant, children had to meet the following criteria: 1. Have a current medical diagnosis of Autism Disorder, Asperger's Disorder, or Pervasive Developmental Disorder - Not Otherwise Specified by a physician, psychologist, or psychiatrist or an educational classification of Autism based on the Utah State Guidelines (see Appendix K). 2. Obtain scores on the Autism Diagnostic Observation Schedule (ADOS) that 35 meet or exceed the cut-off for Autism Spectrum Disorders 3. Obtain a score on the Gilliam Asperger's Disorder Scale (GADS) that meets or exceeds the cut-off for Autism Spectrum Disorders 4. Obtain a score on the Social Responsiveness Scale (SRS) that meets or exceeds the cut-off for Autism Spectrum Disorders. 5. Obtain a verbal IQ score of 70 or higher on a standardized intelligence test, administered within the past 3 years by a qualified administrator. 6. Possess and demonstrate use of sufficient expressive and receptive language so as to be able to participate in the social skills group. In addition to meeting these criteria, a placement checklist (see Appendix L) designed for this study to screen participants was administered to teachers and parents to aid in the selection of participants. Posters advertising the research study (see Appendix H) were hung at UNI, the Utah Autism Research Project, and at Salt Lake area schools for the purpose of recruiting participants. Parents who were interested contacted the researcher who provided more detailed information about the program to ensure Superhero Social Skills met the child's needs. Once the participants were selected, the researcher obtained parent permission (see Appendix I) and assent from the children (see Appendix J), and parents were given questionnaires to complete in order to determine if the child met inclusion criteria and would benefit from the intervention. The parents of the participants were required to bring a peer or sibling without a diagnosis of ASD to complete the sessions with the target children in order to provide a component of peer mediation and to increase the efficacy of the intervention. The 36 peer/siblings were also between the ages of 5 and 10 and they participated in the sessions with the ASD children. Parents of the participants attended an initial parent training session to be informed about the homework, monitoring of skill use at home, and the weekly parent interviews to be completed. Table 4 provides a summary of the participant characteristics and is followed by a more detailed description of the individual participant characteristics. Participant 1 is an 8-year-old caucasian male with a diagnosis of Pervasive Developmental Disorder - Not Otherwise Specified from a licensed psychologist and an educational classification of Autism. His cognitive ability was assessed in his school using the Woodcock Johnson Tests of Cognitive Abilities, Third Edition. He earned a GAI score of 94 and a Verbal Ability score of 90. Participant 1 is at grade level academically, but is below grade level socially. He reportedly becomes very fixated on particular interests, has limited social and emotional reciprocity, and has difficulty recognizing social cues. Participant 2 is a 10-year-old caucasian male without a diagnosis or educational classification of Autism. The school district is currently completing an assessment in order to give him an educational classification of Autism. On the Weschler Intelligence Scale for Children, Fourth Edition, he earned a Full-Scale IQ score of 94 and a Verbal Comprehension score of 85. The researcher and another graduate student administered the Autism Diagnostic Observation Schedule, Module 3, to Participant 2 because he had not completed this assessment prior to involvement in this study. He earned a combined Communication and Reciprocal Social Interactions Score of 14. Participant 2 has reportedly attended other social groups before, but is unable to make friends. His parents 37 Table 4 Participant Characteristics ______________________________________________________________________ Demographic Information for Participants _______________________________________________________________________ Participant 1 Participant 2 Participant 3 Participant 4 Average ____________________________________________________________ Child's Age 8 10 7 9 8.5 ADOS Total Score 10 14 16 17 14.25 FSIQ Score 94 94 116 62 91.5 GADS Total 97 95 82 112 96.5 SRS (Pre-) 69 72 71 90 75.5 ASSP (Pre-) 114 126 141 91 118 _______________________________________________________________________ report that he does not have social and emotional reciprocity. Participant 3 is a 7-year-old caucasian female with a medical diagnosis of Pervasive Developmental Disorder - Not Otherwise Specified by a licensed psychologist and an educational classification of Autism. Participant 3 was administered the Stanford- Binet Intelligence Scales, Fourth Edition. She earned a Full-Scale IQ score of 116 and a Verbal Reasoning Score of 98. She was also given Module 2 of the Autism Diagnostic Observation Schedule (ADOS) at Valley Mental Health. She earned a Communication score of 8, a Reciprocal Social Interaction score of 8, and a combined score of 16, all of which are above the Autism cut-off. She reportedly has difficulty with imaginative play, 38 she does not have social and emotional reciprocity, and she is not able to communicate effectively despite having a large vocabulary. Participant 4 is a 9-year-old caucasian male with a medical diagnosis by a licensed psychologist and an educational classification of Autism. He was administered the Autism Diagnositc Observation Schedule and received a combined Communication and Reciprocal Social Interaction score of 17, which is above the Autism cut-off. He was administered the Wechsler Intelligence Scale for Children, Fourth Edition and earned a Verbal Comprehension Score of 71 and a Full-Scale IQ of 62. Participant 4 reportedly has difficulty following multistep directions and he has difficulty relating to others socially. He also lacks social and emotional reciprocity. Setting The sessions of the social skills program were conducted at the University Neuropsychiatric Institute (UNI) located in the University of Utah Research Park. All sessions took place in a room used for group meetings and there was a waiting room for parents during sessions. The room had one large table, a television, and a d.v.d player. During sessions, foam squares were placed on the floor for the children to sit on. A second group room was used as an area for free time play. Toys available to the children during free play included LEGOS (LEGO), Ants in the Pants Spongebob Squarepants Edition (Hasbro), Don't Break the Ice (Hasbro), toy cars with a track (Mattel), Transformers (Hasbro), and Jenga (Parker Brothers). The 10-minute observation periods during free play were videotaped for coding and reliability purposes. Two graduate students from the Educational Psychology Department at the 39 University of Utah assisted the researcher with implementation of the intervention and coding of data. One graduate student attended most sessions and assisted by video recording all sessions and free play sessions, passing out Scooter and Blackhole cards for rule-following and rule-breaking behavior, and helped the participants with role-playing and games during the sessions. The second graduate student coded 25% of analog free play observations (13 observations) to measure interrater reliability. Dependent Measures Observation System Bellini's Social Observation System (2007, see Appendix C) was used during the 10-minute free play periods during baseline and following each treatment session to determine the amount of social engagement displayed by each participant with ASD. Bellini's Social Observation System provides codes for the areas of social initiations, social responses, the combined total social engagement, and play behaviors. Social initiations are defined as requesting assistance or information; joining in a play activity or interaction; giving a greeting or compliment; giving, sharing, or showing an object; and requesting interaction or participation. In order for a behavior to be coded as a social initiation, the behavior must be the beginning of a new behavioral sequence. This may be accomplished through changing play activities, partners, or discontinuing a behavior for at least 5 seconds. Social responses are defined as responding to a request for assistance or information, joining an activity upon request, accepting an object when offered, and appropriately continuing an interaction. Play codes were not used in the data analysis for this study. 40 The observation system uses a 10-second time sampling method of observing behaviors, in which the observer watches the behavior for 5 seconds, and then records the social initiation or social response behavior during the next 5 seconds. The observations were all videotaped and then the observations were reviewed and coded by the researcher and by another graduate student separately. The coding was then compared after both observers had completed their coding to determine interrater reliability. Generalization Measure Parents were given an adapted version of the Parent Daily Report (PDR) (see Appendix D) (Chamberlain & Reid, 1987) to record the frequency of skill use in the home setting. The form listed each skill that would be taught in the intervention sessions: Get Ready, Following Directions, Anxiety Reduction, Participate, Generalized Imitation, Body Basics, Recognizing and Expressing Wants and Needs, and Joint Attention. The parents were also given a written description of the skills and their steps. Any questions about the skills or how to monitor them at home were addressed at the parent meeting. At the beginning of the study, the researcher contacted the parents of each participant the evening before the session to collect the data. When it was difficult for the researcher to reach parents by phone, the parents were asked to bring their forms to the sessions and the information was discussed with parents during the free play periods. Social Responsiveness Scale The Social Responsiveness Scale (SRS; Constantino, 2005) is a 65-item questionnaire that assesses general social behaviors (social impairments, social 41 awareness, social information processing, ability for reciprocal social communication, and anxiety and avoidance of social situations) and behaviors specific to autism (autistic traits). The items on the SRS are rated on a scale of 1(not true) to 4 (almost always true). The SRS was completed at pre- and posttest by parents to determine the severity of the autistic symptoms of social impairments and the effects of the intervention over time. Constantino (2005) uses the standard error of measurement (SEM) as a means of determining significant change between scores to interpret treatment effects. SEM "provides a mathematical estimate of how widely scores may tend to vary above or below a given specific result" (p.16). Any scores on the posttest of the SRS that fall below the SEM score are considered to be a significant change. This assessment has been well-researched as a measure that is able to detect treatment effects (Constantino et al., 2004). Power Cards Children participating in the program were given Power Cards each time a new skill was taught. The card has a picture of one of the program's superhero characters and the steps for the skill. There are circles for the child to fill in when they have demonstrated use of the skill throughout the week to be used as a self-monitoring procedure and to encourage generalization of the skill use. When they check-in at the next session, their Power Charges are transferred to a Power Poster on the wall as a way of public posting. Each child earned an average of three Power Charges during the sessions for using the skill in the role play activity and then they tracked their use of the skill when not at the sessions. The number of power charges earned were correlated with the frequency of social engagement during free play sessions as measured by Bellini's 42 Social Observation System. Scooter and Black Hole Cards The number of Scooter and Black Hole Cards earned during each session was tracked to determine the frequency of rule-following and rule-breaking behaviors. Scooter Cards are used as reinforcement for following the group rules (Get Ready, Follow Directions, Be Cool, and Participate) and Black Hole Cards are given for not following group rules. The number of Scooter and Black Hole Cards earned were tracked for each session. Behavior Intervention Rating Scale (BIRS) The BIRS is considered to be a valid measure of treatment acceptability and effectiveness. The BIRS was administered to the parents of participants following the completion of the intervention. Parents rated questions about the effectiveness of the treatment on a six-point scale. Ratings from 1 to 6 ranged from strongly disagree to strongly agree. Social Validity Scale The social validity of the intervention was evaluated using a social validity scale that has already been developed and tested for its psychometric properties. The Social Validity Checklist was developed by Bellini (unpublished) and was adapted for use in this study (see Appendix E). Parents completed the checklist after completion of the intervention by responding to five questions about the program's effectiveness. Possible 43 answers on the scale range from strongly disagree to strongly agree. Answers are then given a numerical value (Strongly Disagree =1, Disagree = 2, Agree = 3, Strongly Agree = 4). The total possible score for each item is 4 and the total possible score for the scale is 20. Child Consumer Satisfaction Survey A child consumer satisfaction survey will be administered following the intervention to determine the acceptability of the treatment to the participants. The Child Consumer Satisfaction Survey (CCSS) was developed for use in this study (see Appendix F). Questions were read out loud to the children and they circled their answer. Responses to the questions were on a five-point scale ranging from strongly disagree to strongly agree. Answers were used to determine the participants' perceptions of the Superhero Social Skills program. Autism Social Skills Profile The Autism Social Skills Profile (ASSP) (see Appendix G) is currently unpublished by Bellini, but has been addressed in his research (Bellini & Hopf, 2007) and was used with his permission. This measure is designed to assess the social interaction of children and may also be used to measure the effectiveness of treatment. Items on the ASSP are answered 1 (never/almost never), 2 (sometimes/occasionally), 3 (often/typically), or 4 (very often/always). Most items on the ASSP are scored by using the number indicated by the rater, but there are a few items that are considered negative items and are reverse scored. The item scores are used to obtain a total score, as well as 44 three subscale scores of Social Reciprocity, Social Participation/Avoidance, and Detrimental Social Behaviors. On the ASSP, higher scores suggest less impairment in social functioning. The ASSP was completed by parents of participants with ASD prior to beginning intervention, as well as after the intervention was completed. Other Measures The Gilliam Asperger's Disorder Scale (GADS) The GADS questionnaire for parents has 40 items and addresses different domains of behaviors, as well as developmental history. The GADS can be used as a screening tool, to document behavioral progress, and for research purposes. The GADS provides documentation about the behavioral characteristics of Asperger's Disorder. Parents completed this assessment prior to treatment. Treatment Fidelity Checklist A checklist was created in order to assess the level of fidelity in implementation of the program. Each step of the lesson implementation was listed on a form (see Appendix M). Following each session, the researcher and graduate student assistant had to indicate which steps were implemented by marking the checklist. A percentage of step implementation was then calculated by dividing the number of steps implemented by the number of total steps for each lesson. All of the treatment fidelity forms were totaled after the completion of all sessions to obtain a mean treatment fidelity percentage. 45 Design Data analysis will be completed using a replicated AB single-subject design. Participants were observed during analog free play periods during four baseline sessions and after the eight intervention sessions. Single-subject research has been used to study the effectiveness of various interventions. Kazdin (1992) stated that single-subject research can be used to draw inferences about interventions as long as continuous observations are completed prior to treatment, during the baseline phase, and throughout the treatment phase. Baseline observations are used to determine a trend in the baseline and establish stability. This trend can be compared to treatment observations to determine if the intervention had an effect on the projected trend. Internal threats of validity exist in this type of study, including maturation, testing effects, and history threats. Historical confounding could also be a possible threat, but is minimized with more than one subject and frequent observations. According to Kratochwill (1978), threats of maturation are minimized if repeated measurement is used, threats of history can also be minimized, and threats of testing effects are minimized if there is not repetitive exposure to a pretest. Specifically AB designs with replication are found to control for historical threats to internal validity if subjects are exposed to multiple and variable environments during the treatment period (Harris & Jenson, 1985). There are many threats to internal and external validity that can be problematic in a single-subject study without any comparison group. Some threats are minimized by manipulating variables in the study design. Replicated AB design research has been found to be effective if there are sudden changes in the participants' behavior that correlate and occur simultaneously with the treatment. 46 Kazdin (1982) stated that single-subject designs are valid if they meet certain criteria. According to Kazdin, a study must include the following to be valid: 1. The data are objective 2. Assessments occur on multiple occasions 3. The target behavior being treated is stable 4. Participants form a heterogeneous group 5. The intervention produces immediate and marked effects Kratochwill (1992) expanded the criteria presented by Kazdin to include the following: 1. The study must be planned 2. There must be a high level of integrity 3. The treatment must be standardized 4. It must produce large effect sizes. Based on the criteria established by Kazdin and Kratochwill, this study is considered to be a valid replicated AB research study. The data in this study are objective in that the behaviors are well-defined and the system used for coding is an impartial means of collecting the data. The second criteria requiring that assessments take place on multiple occasions is met by the observations being conducted 12 times during the course of the study. The target behavior for this study is stable, as ASD is considered to be a stable trait. This study is conducted with a heterogeneous group of children of varying ages, genders, diagnoses, intellectual abilities, and language levels. This study has been well-planned and includes a manualized treatment that was implemented by trained graduate students. Results from this study would suggest that there were large changes in 47 behavior and results produced large effect sizes, which would also imply that this study meets criteria for a valid single-subject research study. Procedures The researcher recruited participants by placing flyers and posters in Salt Lake area elementary schools, the University of Utah Neuropsychiatric Institute (UNI), and at the Utah Autism Research Project. Interested parents contacted the researcher and were given more detailed information by phone. If the parent wanted to have their child participate in the program, they met with the researcher to complete the parental consent and child assent forms and provide past testing required for inclusion criteria. The parents also completed the GADS, BIRS, and SRS during this initial meeting. Once all participants were recruited, parents attended a parent orientation meeting. The researcher provided information about the intervention and lessons, but also explained how to help the child complete the homework and properly check the power cards for reliability of the child's self-monitoring. The researcher provided each parent with a binder containing the Parent Daily Report and a list of the skills and their steps to provide parents with a guideline of how to fill out the forms. Two rooms were used for the social skills intervention and the analog observations. Both rooms had large windows, two large bulletin boards, chairs that were not used by participants, but lined the walls, and a video camera on a tripod recording sessions and analog observations. The room that was used for treatment sessions had one large table that the researcher used to place a 17" computer monitor, laptop, speakers, the reinforcement 48 materials, and the treatment manual during the sessions. There was one large bulletin board on either side of the room that was used to hang the Power Posters correlating to the previous week's skill and the skill being taught during the current session. Foam squares were place on the carpet for the children to sit on. The room that was used for the analog observation periods was similar to the room used for the treatment sessions. The six toys used for free-play (LEGOS, Ants in the Pants Spongebob Squarepants Edition, Don't Break the Ice, toy cars with a track, Transformers, and Jenga) were spread out throughout the room on the floor. Baseline The first baseline analog observation was videotaped by a research assistant during the parent orientation meeting. One participant was ill during the first observation, resulting in there being four total baseline observations in order to have at least three baseline observations for each participant. Each observation was 10 minutes in duration and an audio track was added to each video with cues of when to watch the behavior and when to record for the 10-second time sampling intervals. During the observations, six toys (LEGOS, Ants in the Pants Spongebob Squarepants Edition, Don't Break the Ice, toy cars with a track, Transformers, and Jenga) were set up, all of which could be used for solitary play or for interactive play. Superhero Social Skills An overview of the program is provided (see Appendix A). Also, a sample lesson from the manual (see Appendix B) is included and all lessons follow the format outlined 49 for the 18 skills to be taught (see Appendix N). This program includes 18 lessons and lessons are generally taught twice per week for 18 weeks. Each week, a new skill is taught during the two weekly lessons. This procedure was modified for this study with only one longer lesson being taught each week due to the clinical outpatient setting and difficulty for parents and children to attend twice per week. The two sessions for each lesson were combined into one longer session each week to ensure the participants were still being exposed to all of the program material. The social skills are presented in a video by animated superheroes-The Initiator and Interactor Girl, and their sidekick Scooter the Robot. The superheroes introduce the skill, provide rationale for use of the skill, and outline steps for correct demonstration of the skill. The superheroes then introduce a video with children demonstrating the skill. After viewing several video-modeling scenarios of the skill, the facilitator role-plays a nonexample and a correct example of the use of the skill. The participants and their peer buddies then role-play the skill. After role-playing, children then watch a social story in the form of a digital comic book. After that, the children play a social game that incorporates the skill they have just learned. The second weekly lesson (not used in this study) reviews the acquisition of the skill through a repeated viewing of the entire social skills video and additional role-playing. A DVD of the animation, video modeling, and digital comic book are provided for the home in order to increase generalized use of the skills. In addition to the use of DVDs to present social skills, Power Cards are used. Children fill in a circle on the Power Cards every time they use the skill on the card as a way to self-monitor their use of the skills. The children receive a different power card for each skill during the first 50 lesson of the week. Children bring their cards back each lesson and fill in their Power Poster with the number of Power Charges they earned, as a public posting procedure. Social Stories in the form of a printed comic book that match the digital comic books on the video are also given as homework. Superhero Social Skills Implementation This study was conducted in a clinical outpatient setting one evening per week. Due to the difficulty of children being able to attend sessions twice per week, the lessons were combined into one longer session each week for 8 weeks. Each session was approximately 1 hour long and only the Foundational Skills were taught during this study. Lessons presented included Introduction to the Social Skills Group, Get Ready, Following Directions, Anxiety Reduction, Participating, Generalized Imitation, Body Basics, Expressing Wants and Needs, and Joint Attention. Observation and Data Collection of Social Engagement After every social skills lesson, each participant with ASD was observed during the eight free play periods of 10 minutes. All of the typical peers who attended the treatment session also participated in the free play period. The same six games (Jenga, toy cars with a track, Don't Break the Ice, Legos, Transformers, SpongeBob Squarepants Ants in the Pants) were set up and dispersed throughout the room during each session. All of the free play periods were filmed so that coding could be completed at a later time. Social engagement was coded using Bellini's Social Interaction Codes (Appendix C; Bellini, 2007). During each 10-second interval, one of seven social initiation codes or 51 one of six social response codes were given to label the child's behavior. While coding was completed using all of these codes, only the general classification of social initiation or social response was used for the data analysis. There were also six play codes that could be used, but play codes were not used in the data analysis either. Other sources of data that were collected by the researcher were the number of Power Charges earned between sessions and the number of Scooter and Black Hole Cards earned by each participant during the sessions. Individually, the number of Power Charges and the number of Scooter and Black Hole Cards earned were correlated to the amount of free time spent in social engagement. After the last social skills session, participants with ASD and their peer buddies were given the Child Consumer Satisfaction Survey (CCSS). The researcher read all of the items to the children and explained what each possible answer meant to ensure that the children understood the question and how they were answering. Parents of children with ASD were given the BIRS, the SRS, and the social validity checklist. These measures were collected by the primary researcher and scored. Effect sizes and percentage of nonoverlapping data points were also calculated by the primary researcher using data collected through the Bellini Observation System. Data Analysis Computation of effect sizes (ES) for each subject were computed to determine the effectiveness of the social skills intervention. This method is based on Glass' (1972) ES and is computed by dividing the difference between the baseline means and treatment means by the standard deviation of the baseline for each subject. Cohen (1988) identified 52 a classification system for effect size. An effect size of .2 is considered a small ES, .5 is considered a medium ES, and .8 is considered a large ES. Effect sizes should only be compared to studies of the same design; thus, the effect sizes obtained from this study should only be compared to effect sizes of other single-subject design studies. The use of effect size can have limitations, but Jenson, Clark, Kircher, and Kristjansson (2007) stated, "Rather than simply rejecting a null hypothesis, effect sizes emphasize a difference between groups that is not confounded by sample size" (p. 491). The Busk and Serlin (1992) No Assumptions Model was used to calculate effect sizes and Cohen's (1988) standards for interpreting effect sizes was used to determine the magnitude of the effect. ES was calculated by determining the percentage of intervals during which the participant was engaged in social initiation and social response during baseline and treatment. The difference between the means of baseline and the means of intervention were then divided by the pooled standard deviations from baseline and treatment. Cohen defines a small effect size as one that falls between 0.1 and 0.3, a medium effect size as those falling between 0.3 and 0.8, and a large effect size as those 0.8 and above. Another approach to determining the effectiveness of an intervention in single-subject research designs is to calculate the percentage of nonoverlapping data points (PND; Scruggs & Mastropieri, 1998). This method is used to compute the percentage of nonoverlapping data between the baseline and treatment conditions. This method can be inaccurate if there are outliers found in the baseline phase or when treatment has a detrimental effect; however, it can be an effective form of data analysis when conducting single-subject research. PND is calculated by dividing the number of data points in the 53 treatment phase that exceed the highest or lowest point in the baseline phase by the total number of data points in the treatment phase, yielding a percentage (Scruggs & Mastropieri, 1998). Scruggs and Mastropieri (1998) found that PND is a useful way to assess the efficacy of interventions and to use as a common measurement in order to be able to compare research. Scruggs and Mastropieri also identify a way of determining the impact of interventions based on the PND score. They indicate that PND scores of over 90 (i.e. 90% of treatment observations exceed the highest baseline observation) can be interpreted as very effective, scores between 70 and 90 can be considered effective, scores of 50 to 70 should be considered questionable, and scores below 50 should be interpreted as ineffective treatments. This provides a means for classifying and comparing interventions done in single-subject research. In this study, PND was calculated by identifying the highest data point in the baseline data. The number of treatment data points that fell above the highest baseline data point were divided by the total number of treatment data points. The number of Power Charges filled in on the participants' Power Cards was correlated with the percentage of free time spent in social interaction using a Pearson Correlation Coefficient. The number of Scooter and Black Hole Cards earned during each session was also correlated to the percentage of free time spent in social interaction using a Pearson Correlation Coefficient. Pre- and posttest scores obtained from SRS were compared using Constantino's (2002) suggested method for comparison. Using the formula provided in the SRS manual, the SEM scores were calculated. Constantino recommends changes in the T- 54 scores by one to two SEM should be considered significant treatment effects, but more conservatively, changes by two or more SEM. For this study, conclusions were based on the more conservative method of determining significant treatment effects. Descriptive statistics were used to analyze the results of the BIRS, Bellini's Social Validity Measure, and the CCSS. CHAPTER 3 RESULTS This study was designed to evaluate the use of the Superhero Social Skills program as an evidence-based practice to teach social skills to children with ASD in a clinical out-patient setting. The purpose of this program is to provide children with the appropriate social skills necessary for participation in pro-social interactions with peers and adults. Another goal of this study is to measure generalization of the skills to multiple situations and settings, such as home and school. The program is based on a superhero theme with animation and comic books as high interest media to appeal to participants and maintain attention to program content throughout the intervention. The program also incorporates research-validated components, such as video-modeling, social stories, peer mediation, and self-management strategies. The program effectiveness was determined by increased use of social skills during observation periods, increased pro-social behaviors in a generalized setting, acquisition of Power Charges on cards during sessions and at home for appropriate use of skills, acquisition of Blackhole and Scooter Cards during sessions, and completion of checklists including the BIRS, SRS, and the Children's Consumer Satisfaction Survey. 56 Treatment Integrity Based on the results of the treatment fidelity checklists completed by the researcher and the research assistant, the Superhero Social Skills intervention was delivered with 99% integrity. Reliability of Observations Interrater reliability was assessed to assure consistency, minimize biases, and to ensure that the target social responses and initiations were well-defined. An acceptable level of interrater reliability is defined by Forehand and McMahon (1981) as 80%, therefore, the researcher and another graduate student coder practiced coding child interactions until 80% agreement was reached. The operational definitions for social initiations and responses and examples of these behaviors were a modified version of the observational system developed by Bellini (2007). Interrater agreement was calculated in a sample of 25% of the observations (13 total observations) for social initiations, social responses, and total social interactions. Reliability was calculated by dividing the number of agreements by the number of agreements and disagreements. Interobserver agreement was calculated to be 81.54% for 13 (25%) of the observations. Kappa was also calculated as a method of determining both occurrences and nonoccurrence of behavior (Sattler, 2006). Kappa is used to determine the proportion of observer agreements while correcting for chance agreements. Kappa was calculated using the formula presented by Uebersax (1982). Kappa was calculated at 0.66 for the observer agreement, which is indicative of a substantial agreement (Sim & Wright, 2005). 57 Research Question 1 What is the effectiveness of the social skills intervention as measured by the spontaneous use of social skills when participants are observed during free play in analogue free time as measured by an adaptation of a developed observation scale? The efficacy of the social skills instruction was measured by determining the number of 10-second intervals in a 10-minute observation period during which the participants engaged in social initiation or social response and the total number of social interactions. The intervals were calculated for baseline and treatment phases. The social initiation and social response data were also used to calculate the percentage of nonoverlapping data points (PND; Scruggs, Mastropieri, & Casto, 1987). PND is calculated by identifying the number of data points in the intervention phase that are higher than any of the baseline data points and then dividing that number by the total number of intervention data points. Overall, participants initiated social interaction during an average of 6.09% of baseline intervals and initiated interactions during 9.48% of treatment intervals. Participants responded socially to others during an average of 18.55% of baseline intervals and an average of 30.46% during treatment intervals. Average total social engagement for the participants was 24.68% during baseline intervals and an average of 39.96% during treatment intervals. Based on Cohen's criteria for interpreting effect sizes, a large effect size was observed for the group's average social initiations (ES=0.82), a moderate effect size was observed for the group's average social responses (ES=0.68), and a large effect size for the group's average total social engagement (ES=1.07). Average Percentage of Nonoverlapping Data Points (PND) for participants was 37.23% 58 for social initiations, 39.47% for social responses, and 54.29% for total social engagement, indicating ineffective to questionable treatment effect. Participant 1 attended all sessions of the program (4 baseline and 8 instructional sessions). Participant 1 initiated social interaction an average of 7.49% of the intervals during baseline and an average of 6.44% of the intervals during intervention. He responded to social interaction during an average of 26.64% of the baseline intervals and during 57.29% of the intervention intervals. Total social engagements during baseline were an average of 34.14% and 63.75% during intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 1 for social initiations was a small effect size (ES= -0.0738), for social responses, it was a large effect size (ES=1.412), and for total social engagements, the effect size was also large (ES=1.469). Percentage of nonoverlapping data points was calculated at 0% for social initiations, 62.5% for social responses, and 62.5% for total social engagements, indicating ineffective treatment for social initiations and questionable treatment for social responses and total social engagements. Graphs of this participant's use of social initiations, social responses, and total social interactions during the analog free play observations are found in Figures 1, 2, and 3, respectively. Participant 2 attended only 3 baseline sessions due to illness and only 5 treatment sessions due to the family suddenly needing to relocate to another state. Participant 2 initiated social interaction an average of 3.3% of the intervals during baseline and an average of 6.96% of the intervals during intervention. He responded to social interaction during an average of 15.53% of the baseline intervals and during 21.99% of the intervention intervals. Total social engagements during baseline were an average of 59 Figure 1: Baseline and intervention measures of social initiations for participant 1. Particip.mt 1 - Socia l Initiations " , , , , , , , , n Sessions 60 Figure 2: Baseline and intervention measures of social responses for participant 1. Particip;lnt 1 - Social Responses 1 w - j -'---- , , • • , " • u Sessions 61 Figure 3: Baseline and intervention measures of total social interactions for participant 1. Participant 1 - Total Social Interactions ,OO ~-- , , , • • • u Sessions 62 18.89% and 29% during intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 2 for social initiations was a large effect size (ES=0.8523), for social responses it was a medium effect size (ES=0.434), and for total social engagements it was a medium effect size (ES=0.7359). Percentage of nonoverlapping data points was calculated at 40% for social initiations, 40% for social responses, and 60% for total social engagements, indicating social initiations and social responses were unaffected by the intervention, but there were questionable effects for total social engagement. Graphs of this participant's use of social initiations, social responses, and total social interactions during the analog free play observations are found in Figures 4, 5, and 6, respectively. Figure 4: Baseline and intervention measures of social initiations for participant 2. Participant 2 - Sociiliinitiiltions , .g 10 ~ , , , • , • , • s.. •• IoM 63 Figure 5: Baseline and intervention measures of social responses for participant 2. Particip;lnt 2 - Social Responses --/--- , , • • , • Sessions 64 Figure 6: Baseline and intervention measures of total social interactions for participant 2. Participant 3 attended only 3 baseline sessions due to illness and only 7 treatment sessions due to miscommunication about group scheduling. Participant 3 initiated social interaction during an average of 2.73% of the baseline intervals and an average of 6.18% of the intervals during intervention. He responded to social interaction during an average of 23.3% of the baseline intervals and during 32.36% of the intervention intervals. Total social engagements during baseline were an average of 26.11% and 38.55% during intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 3 for social initiations was a large effect size (ES=1.24), for social Participant 2 - Total Social lnttraction5 , , • • , • Sessions 65 responses it was a large effect size (ES=0.8479), and for total social engagements it was a large effect size (ES=1.017). Percentage of nonoverlapping data points was calculated at 71.43% for social initiations, 42.86% for social responses, and 57.14% for total social interactions, indicating it was an effective intervention for social initiations, ineffective for social responses, and questionable for total social engagement. Graphs of this participant's use of social initiations, social responses, and total social interactions during the analog free play observations are found in Figures 7, 8, and 9, respectively. Figure 7: Baseline and intervention measures of social initiations for participant 3. Participant 3 - Social Initiations , .g 10 ~ , , • • , " • s.. •• IoM 66 Figure 8: Baseline and intervention measures of social responses for participant 3. .,., " flO , , Particip;l nt 3 - Social Responses • • Sessions , • • 67 Figure 9: Baseline and intervention measures of total social interactions for participant 3. Participant 4 attended all sessions of the program (4 baseline and 8 instructional sessions). Participant 4 initiated social interaction during an average of 10.82% of the baseline intervals and an average of 18.32% of the intervals during intervention. He responded to social interaction during an average of 8.73% of the baseline intervals and during 10.19% of the intervention intervals. Total social engagements during baseline were an average of 19.57% and 28.53% during intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 4 for social initiations was a large effect size (ES=1.265), for social responses it was a small effect size (ES=0.024), and for total social engagements it was a large Participant 3 - Total Social 1nteractions ro , , • , , • , Seulons 68 effect size (ES=1.0419). Percentage of nonoverlapping data points was calculated at 37.5% for social initiations, 12.5% for social responses, and 37.5% for total social interactions, indicating that none of them were effective. Graphs of this participant's use of social initiations, social responses, and total social interactions during the analog free play observations are found in Tables 10, 11, and 12, respectively. Figure 10: Baseline and intervention measures of social initiations for participant 4. , ,i 20 t----­• .~ " , , , Participant 4 - Socia l Initiations • , , , , n Sessions 69 Figure 11: Baseline and intervention measures of social responses for participant 4. Particip;lnt 4 - Social Responses , f-- , , • • , , " Sessions 70 Figure 12: Baseline and intervention measures of total social interactions for participant 4. Overall, the data collected suggest that the Superhero Social Skills program is effective in increasing the percentage of social initiations, social responses, and total social engagement. Three out of the 4 participants increased the use of social initiations, all of the participants increased their use of social responses, and all of the participants increased their use of total social engagements. Only small to moderate PND was calculated for the participants. This research question was satisfied with the data collected. Participant 4 - Total Social Interactions , , • • , • • u Sessions 71 Research Question 2 What is the effectiveness of the social skills intervention's generalizability as measured by the spontaneous use of social skills as observed by the parents and reported through an adapted version of the Parent Daily Report (PDR) and reported through a parent telephone interview? The adapted version of the Parent Daily Report consists of the parent recording on a daily basis the child's use of eight social skills (get ready, following directions, anxiety reduction, participation, generalized imitation, body basics, recognizing and expressing wants and needs, and joint attention) being taught in the weekly lessons. At a parent orientation prior to the intervention sessions, parents were given the steps and definitions for each skill. All of the skills and their steps were also explained and demonstrated for the parents during the parent orientation. The parents completed the form daily for three baseline weeks and seven intervention weeks. All social interactions were totaled per week to provide the total number of times the child had used pro-social behaviors outside of the group. The parents recorded the number of times the child used the skill each day. After calculating the total number of social interactions used each week, effect sizes and percentage of nonoverlapping data points were calculated for each subject. Overall, participants participated in an average of 105.4 social interactions per week during baseline and an average of 145.84 social interactions per week during treatment. Based on Cohen's criteria for interpreting effect sizes, a large effect size was observed for the number of social interactions (ES=1.13). PND was calculated at 52.09%, which indicates the intervention is considered to have produced questionable 72 treatment effects. Participant 1 interacted with the skills being taught an average of 158.33 times during the baseline weeks and an average of 174.67 times during the weeks of intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 1 for social interactions was a small effect size (ES=0.25). Percentage of nonoverlapping data points was calculated at 16.67%, also indicating an ineffective result. A graph of the Parent Daily Report results for participant 1 is found in Figure 13. Figure 13: Baseline and intervention measures of the parent daily report for participant 1. ' 00 T'--- , '00 • ~ ••• ~ 100 , , Participant 1 - PDR Tota l • , , , , Weeks 73 Participant 2 interacted with the social skills being measured an average of 36.25 times during the baseline weeks and an average of 41.17 times during the weeks of intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 2 for social interactions was a medium effect size (ES=0.49). Percentage of nonoverlapping data points was calculated at 16.67%, which indicates a small effect for this participant. A graph of the Parent Daily Report results for participant 2 is found in Figure 14. Figure 14: Baseline and intervention measures of the parent daily report for participant 2. Pilrticipilnt 2 - PDR Tota l ,,, " •••"• " ~ " " , ~ , , • , • , " • " Weeks 74 Participant 3 interacted with pro-social behaviors an average of 11 times during the baseline weeks and an average of 22.25 times during the weeks of intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 3 for social interactions was a medium effect size (ES=0.72). Percentage of nonoverlapping data points was calculated at 75%, which is considered to be a moderate effect. A graph of the Parent Daily Report results for participant 3 is found in Figure 15. Figure 15: Baseline and intervention measures of the parent daily report for participant 3. Pilrtici pilnt 3 - PDR Tota l 75 Participant 4 interacted with the social skills being measured an average of 216 times during the baseline weeks and an average of 345.25 times during the weeks of intervention. Based on Cohen's criteria for judging effect sizes, the effect size calculated for participant 4 for social interactions was a large effect size (ES=3.05). Percentage of nonoverlapping data points was calculated at 100%, which indicates that the treatment is very effective. A graph of the Parent Daily Report results for participant 4 is found in Figure 16. Figure 16: Baseline and intervention measures of the parent daily report for participant 4. Pilrticipilnt 4 - PDR Tota l ow 000 ,•••,, ,','00ww00 ~ •• ~ ,w '00 w , , , o • • , u " B Weeks 76 All of the participants increased the number of pro-social behaviors they used weekly from baseline to treatment and overall, the participants increased the number of social interactions they participated in outside of treatment. Effect sizes for participants ranged from moderate to large, and the overall average effect size calculated was large, but the PND is considered only mildly effective overall. The PND calculated for 1 participant determined the intervention was ineffective and it was considered very effective for another. This research question was satisfied based on the data collected from this study. Research Question 3 What is the effectiveness of the social skills intervention as measured by self-recording using the Power Cards and the number of Scooter and Blackhole Cards earned during intervention sessions? The Pearson Correlation Coefficient was used to determine if there was a correlation between the self-recording of Power Charges on the Power Cards and the percentage of time that the children were engaged in social interactions during the analog free-play observations. The same statistic was used to determine a correlation between the amount of social engagement and the number of Scooter and Blackhole Cards earned during the treatment sessions. Data were averaged for all participants and then correlated. There were no significant correlations between total social engagements and the earning of Scooter Cards (r=-o.3395, n=8, p=0.4106), the earning of Blackhole Cards (r=0.4269, n=8, p=0.2915), or the self-recording of Power Charges (r=-0.4177, n=8, p=0.3031). 77 None of the correlations between the total social interactions and the Power Charges earned between sessions for any of the participants were significant. Results from the Pearson Correlation for participant 1 were (r=0.3055, n=8, p=0.4160). Data analysis also did not produce significant results for the correlation between Scooter Cards received during treatment and social engagement during free play (r=-0.4307, n=5, p=0.2867). The correlation between the number of Blackhole Cards earned and the amount of time spent in social engagement was not significant for participant 1 (r=0.4042, n=8, p-0.3206). Participant 2 also had results that were not significant for the correlation between the number of Power Charges earned and the amount of time spent in social engagements during free play (r=0.527, n=5, p=0.3615). The correlation between the number of Scooter Cards earned during lessons and the amount of time spent in social engagement during free play was also not significant for participant 2 (r=-0.068 n=5, p=0.9134). There was not a significant correlation found for participant 2 between social engagement and Blackhole Cards earned, as there were not enough data available to produce a correlation coefficient because this participant had not received any Blackhole Cards during treatment sessions. Participant 3 did not have a significant correlation between Power Charges and social engagement (r=0.0514, n=7, p=0.9128) or between Scooter Cards earned and social engagement (r=0.4374, n=7, p=0.3264). Similar to participant 2, there were no Blackhole Cards given to participant 3 during the treatment session, resulting in the inability to calculate a correlation coefficient. Participant 4 had a correlation between Power Charges and amount of social 78 engagement that was not significant (r=0.1637, n=8, p=0.6986). The correlation between Power Charges and social engagement was not significant for participant 4 either (r=0.088, n=8, p=0.8358). A correlation coefficient was calculated for the number of Blackhole Cards earned and the amount of time spent in social engagement for participant 4, but the correlation was not significant (r=-0.0847, n=8, p=0.842). Participant 4 was the only child who brought his Power Card back to the sessions consistently every week with a total of seven data points. Participant 1 returned with his Power Card four times and participants 2 and 3 brought their Power Cards back to three of the seven opportunities they had to return them. Only 2 of the participants received any Blackhole Cards during the treatment sessions. The small number of data points may have contributed to the results in this correlation. The results of the correlations between total social initiations and Blackhole Cards, Scooter Cards, and Power Charges are found in Table 5. Overall, there were no significant correlations between free time play behaviors and the number of Scooter or Blackhole Cards earned during sessions and the Power Charges earned between sessions. This research question was not satisfied with the data produced from this study. Research Question 4 What is the improvement in rule-following behavior during training as measured by the participants earning Scooter Cards and Blackhole Cards over time? Overall, there were no changes in rule-following behaviors observed across the participants based on the number of Scooter Cards and Blackhole Cards that were earned 79 Table 5 Correlations of Power Charges, Scooter Cards, and Blackhole Cards with the Observed Rates of Social Interactions ________________________________________________________________________ Pearson Correlations _______________________________________________________________________ Power Charges Scooter Cards Blackhole Cards r p r p r p Participant 1 0.3055 0.4618 -0.4307 0.2867 0.4042 0.3206 Participant 2 0.527 0.3615 -0.068 0.9134 N/A N/A Participant 3 0.0514 0.9128 0.4374 0.3264 N/A N/A Participant 4 0.1637 0.6986 0.088 0.8358 -0.0847 0.842 Total -0.4177 0.3031 -0.3395 0.4106 0.4269 0.2915 _______________________________________________________________________ during each session. The number of Scooter Cards slightly increased in the fourth, fifth, and sixth session, and then decreased again. It is difficult to determine if the changes in cards earned by participants is due to changes in behavior or absences of group members during four of the sessions. The number of Blackhole Cards distributed to the group during treatment sessions was consistently one or less. The graph showing the average number of Scooter and Blackhole Cards received per session by all participants is in Figure 17. Participant 1 did not have much variation in the number of Scooter Cards earned (Figure 18). Participant 1 did not receive any Blackhole Cards during the first six sessions, but earned one Blackhole Card during each of the last two sessions. Participant 80 Figure 17: Average number of Scooter and Blackhole Cards earned each session. Average Across Partic.,ants , C- •. , • " ! , •l•i ", r=- u " / ~ / • ", , , , • •, • , , Session 81 Figure 18: