Autism spectrum disorder (ASD) refers to a group of complex neurodevelopmental disorders that share a core set of clinical symptoms that includes impairments in socialization, abnormal language development, and a restricted repertoire of behaviors and interests (APA, 2000). Affected individuals may also present with a wide range of other features and co-morbid conditions including emotional and behavioral disorders (EBD; e.g., see DeBruin et al, 2006). In addition, ASD may be prevalent in certain other developmental disabilities such as Down syndrome (see Reilly, 2009 for a review). Wide variability in symptom profiles, rate of development, and level of functioning is observed across affected individuals and even within the same individuals over time.
This wide heterogeneity in clinical features can challenge clinicians to accurately identify an ASD and/or co-occurring conditions so that appropriate treatment can be provided in a timely manner. This situation is made more difficult because autism and autism spectrum disorders are behaviorally defined syndromes and there is not yet a biological marker for early detection and monitoring of changes in symptoms over time. The complex symptom profiles of individuals with an ASD therefore require clinicians and researchers alike to be knowledgeable in selecting appropriate assessment instruments for a wide variety of purposes. These include screening and diagnosis, subject selection and characterization for research protocols, clinical progress monitoring, and measurement of outcomes in treatment research.
Although several ASD screening measures are commercially available, they vary greatly with respect to test development procedures and extent to which their psychometric properties have been evaluated. Most measures require more research to fully evaluate their psychometric adequacy, particularly as it relates to their usefulness in the assessment of subgroups within the ASD population (e.g., youth with autism and a co-occurring EBD and/or co-occurring intellectual disability, etc.). This gap in evidence-based assessment is recognized by the Interagency Autism Coordinating Committee (2009) and is listed as a priority area for autism research funding.
To advance our understanding of the psychometric adequacy of several common measures in autism spectrum assessment, a series of studies is being conducted by investigators at the University of Rochester, Rochester Institute of Technology, and Hofstra University. Findings from these studies will contribute to the evidence-based assessment literature, which can advance practitioner and researcher knowledge in the selection of psychometrically sound measures for their clinical and research purposes. Below is a brief summary of some recently completed studies and some that are in progress.
Screening Measures for Autism and Autism Spectrum Disorders
Magyar & Pandolfi (2007) investigated the factor structure of the Childhood Autism Rating Scale (CARS; Schopler, Reichler, & Renner, 1988), an early autism screening measure that predates the current DSM-IV conceptual model. This was the third independent study of the constructs underlying the CARS. Magyar & Pandolfi (2007) identified correlated Social-Communication, Social Interaction, Stereotypies and Sensory Abnormalities, and Emotional Regulation factors. These constructs represent core and associated features of autism that are conceptually consistent with DSM-IV. These results differed somewhat from two previous construct validity studies (DiLalla & Rogers, 1994; Stella et al., 1999). However, all three studies indicated that the CARS measured constructs consistent with DSM-IV core symptoms and associated features of autism. Differences in the specific results appeared related to differences in administration procedures used, the settings, and sample characteristics. Although its development predates the DSM-IV, and many newer measures are available, the CARS’ psychometric properties, conceptual relevance, and flexible administration procedures support its continued use as an autism screening measure.
In another study, Pandolfi, Magyar, & Dill (in press) examined the construct validity of the Gilliam Autism Rating Scale-2nd Edition (GARS-2; Gilliam, 2006). The GARS-2 includes three conceptually derived subscales that contribute to the norm-referenced Autism Index, which indicates the probability that a person has autism. The subscales include Stereotyped Behaviors, Communication, and Social Interaction. However, no studies have evaluated the measure’s factor structure and therefore, exploratory and confirmatory factor analyses of the standardization sample data were used to empirically identify factors underlying the GARS-2. Results did not support the conceptually-derived three subscale structure. In fact, four factors were identified and labeled: stereotyped/repetitive behavior, stereotyped/idiosyncratic language, word use problems, and social impairment. The results indicated that the conceptually-derived GARS-2 subscales and Autism Index should not be used for screening. Additional scale development is necessary to determine the future contribution of the GARS-2 in autism screening.
Several other studies are currently underway examining the reliability, construct, and discriminant validity of two other widely used screening measures: the Social Communication Questionnaire (Rutter, Bailey, & Lord, et al., 2003) and the Pervasive Developmental Disorders in Mental Retardation Scale (Kraijer & de Bildt, 2005). Data were collected from children with Down syndrome (DS) and children with DS and co-occurring ASD.
Measures for Evaluating Co-Occurring Symptoms or Disorders
Children and youth with an ASD often present with co-occurring emotional and behavioral disorders (EBD), however, developmental characteristics associated with the ASD such as language and cognitive impairment, poor insight, and problems with accurately reporting changes in thinking, behavior, and mood may make it difficult to identify co-occurring EBD that require specific treatment. Third party report, therefore, is an essential component of a multimethod assessment of children and youth with an ASD. However, EBD screening measures need to be validated specifically for the ASD population. As was the case for the ASD measures described above, evaluating the internal structure (e.g., number and nature of the constructs underlying the scale, scale reliability) and convergent/discriminant validity of EBD measures are necessary to inform clinical and research protocols and practice. This information will help us understand what the instrument measures, how well it identifies those individuals with a co-occurring EBD, how well it discriminates between various clinical and non-clinical subgroups, and its utility for clinical progress monitoring.
Pandolfi, Magyar & Dill examined the factor structure of the Child Behavior Checklist 1.5-5 (CBCL; Achenbach & Rescorla, 2000) and the Child Behavior Checklist 6-18 (CBCL; Achenbach & Rescorla, 2001). These are both widely used and well researched measures for EBD. The scales are constructed similarly in that they contain several norm-referenced scales derived through factor analysis of data from the general pediatric population. Each measures emotional and behavioral syndromes that have also been observed in the ASD population, such as anxiety, depression, aggressive behavior, and attention problems.
In these studies, confirmatory factor analysis of archival data evaluated the adequacy of the CBCL 1.5-5 and 6-18 factor models in well-characterized samples of preschoolers (18 months to 5 years; N=128) and school-age youth (6 to 18 years; N=122) with an ASD. Psychometric results from both the CBCL 1.5 to 5 (Pandolfi, Magyar, & Dill, 2009) and CBCL 6-18 (Pandolfi, Magyar, & Dill, in preparation) supported the factorial validity of each measure. Preliminary analyses indicated that the CBCL 6-18 can discriminate those with an ASD from those with an ASD and a co-occurring EBD. Discriminant analyses are still required for the CBCL 1.5-5. Although more validity studies are needed, these results suggest that practitioners can use these measures to screen for EBDs in children and youth with ASD in conjunction with other clinical data.
Summary
Given the variability in clinical symptom profiles between individuals with an ASD and within the same person over time, the clinical and research communities will require a variety of reliable and valid measures from which to choose. Most of the ASD measures that are commercially available require additional comprehensive psychometric evaluations, such as the ones described here, in order to advance our understanding of the extent to which a measure can be considered evidence-based. The clinical use of psychometrically sound instruments can assist in the well-being of individuals affected with ASD. In addition, such instruments are needed to further our understanding of the nature of the autism spectrum disorder(s). Multidimensional scales may prove to be invaluable in research on autism subtypes. Future work (e.g., taxometric research) utilizing psychometrically sound measures can be used to examine individual variation within the autism spectrum and investigate whether we can distinguish between differences in type of ASD and differences in degree of ASD. Such work has implications for research on cause, risk factors, course, and treatment.
Caroline I. Magyar, PhD is an Associate Professor of Pediatrics in the Strong Center for Developmental Disabilities, Division of Neurodevelopmental and Behavioral Pediatrics, University of Rochester School of Medicine & Dentistry, and Director of the Rochester Regional Center for Autism Spectrum Disorders. Vincent Pandolfi, PhD is an Assistant Professor in the School Psychology Department at the Rochester Institute for Technology and Charles A. Dill, PhD is an Associate Professor in the Clinical Psychology Program at Hofstra University. Drs. Magyar, Pandolfi, & Dill have published on the psychometric evaluation of various screening measures in autism assessment. Drs. Magyar and Pandolfi have a long history of working together on clinical and applied research projects in the area of autism spectrum and related disorders and Dr. Dill’s expertise in quantitative psychology advances these collaborative efforts.
References
Achenbach, T.M. & Rescorla, L.A. (2000). Manual for the ASEBA Preschool Forms & Profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, and Families.
Achenbach, T.M. & Rescorla, L.A. (2001). Manual for the ASEBA School-Age Forms & Profiles. Burlington, VT: University of Vermont, Research Center for Children, Youth, and Families.
American Psychiatric Association (2000). Diagnostic and statistical manual of mental disorders: Fourth Edition Text Revision. Arlington, VA: Author.
DeBruin, E.I., Ferdinand, R.F., Meester, S., deNijs, P.F.A., & Verheij, F. (2006) High rates of psychiatric co-morbidity in PDD-NOS. Journal of Autism and Developmental Disorders, 37, 877-886.
DiLalla, D.L. & Rogers, S.J. (1994). Domains of the Childhood Autism Rating Scale: Relevance for diagnosis and treatment. Journal of Autism and Developmental Disorders, 24(2), 115-128.
Gilliam, J.E. (2006). Gilliam Autism Rating Scale: Second Edition. Austin, TX: Pro-Ed.
Interagency Autism Coordinating Committee (2009). The Interagency Autism Coordinating Committee Strategic Plan for Autism Spectrum Disorder Research (August 15, 2008). http://www.nimh.nih.gov/research-funding/scientific meetings/recurring-meetings/iacc/strategic-plan/index.shtml.
Kraijer, D. & de Bildt, A. (2005). The PDD-MRS: An instrument for identification of autism spectrum disorders in persons with mental retardation. Journal of Autism and Developmental Disorders, 35(4), 499-513.
Magyar, C.I. & Pandolfi, V. (2007). Factor structure evaluation of the Childhood Autism Rating Scale. Journal of Autism and Developmental Disorders, 37, 1787-1794. doi: 10.1007/s10803-006-0313-9.
Pandolfi, V., Magyar, C.I., & Dill, C.A. (2009). Confirmatory factor analysis of the Child Behavior Checklist 1.5-5 in a sample of children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 39, 986-995. doi: 10.1007/s10803-009-0716-5.
Pandolfi, V., Magyar, C.I., & Dill, C.A. (in press). Constructs assessed by the GARS-2: Factor analysis of data from the standardization sample. Journal of Autism and Developmental Disorders.
Pandolfi, V., Magyar, C.I., & Dill, C.A. (in preparation). Confirmatory factor analysis of the Child Behavior Checklist 6-18 in a sample of youth with autism spectrum disorders.
Stella, J., Mundy, P., & Tuchman, R. (1999). Social and nonsocial factors in the Childhood Autism Rating Scale. Journal of Autism and Developmental Disorders, 29(4), 307-317.
Rutter, M., Bailey, A., Lord, C. et al. (2003). Social Communication Questionnaire. Los Angeles: Western Psychological Services.
Reilly, C. (2009). Autism spectrum disorders in Down syndrome: A review. Research in Autism Spectrum Disorders, 4, 829-839. doi: 10.1016/j.rasd.2009.01.012.
Schopler, E., Reichler, R.J., & Renner, B.R. (1988). The Childhood Autism Rating Scale (CARS). Los Angeles: Western Psychological Services.
