Evidence based treatment and the assessment of treatment effectiveness are dependent upon the collection of data during the evaluation process providing information about symptoms, impairment and abilities in children with ASD. Such an assessment allows for a seamless transition from diagnosis to effective treatment. Evaluating the effectiveness of a treatment strategy or program is important for interventions designed to address a broad range of ASD symptoms. The validity of the entire process is closely related to the tools used during the evaluation and diagnostic process. More specifically, the reliability (ability to measure the same construct accurately over time) and validity (what is actually being measured) of the tools used will be directly related to the psychometric qualities of the instrument. As in all areas of evaluation, what is learned depends upon the quality of the data generated and the manner in which the findings are interpreted. Tests and rating scales developed to provide valid and reliable information about children with ASD better informs researchers and clinicians. Tools used for diagnostic decision making and treatment planning have a profound impact on the information obtained and the conclusions reached. The better the tools, the more valid and reliable the decisions and, most importantly, the more helpful the information gathered will be in developing a treatment plan monitoring progress and documenting treatment effectiveness.
Determining the effectiveness of any treatment program for individuals with ASD should be accomplished using methods that reflect specific behaviors as well as larger conceptualizations of the condition (e.g., social, communication and atypical behavior problems). Four key questions must be addressed in this process. They are:
- How are these behaviors identified?
- How are these behaviors measured?
- How do these behaviors change with intervention?
- To what reference point or points will behavior change be calibrated?
In this article we present a way to evaluate symptoms related to ASD on both global and specific levels, identify areas for treatment and evaluate the effects of treatment. To do so we will illustrate using information from the Autism Spectrum Rating Scale (ASRS; Goldstein and Naglieri, 2009, Naglieri and Goldstein, 2013, in print). We choose to illustrate using this tool because it is nationally normed and provides several different types of global scores as well as measures of specific behaviors. In addition, the reliability of the scales is well documented and guidelines for assessing treatment change are also provided.
The ASRS is a rating scale for assessing behaviors associated with ASD in children two through eighteen years of age. The scale was developed based on a comprehensive review of both current theory and literature on the assessment of ASD and diagnostic criteria of the condition in clinical manuals. The ASRS scale structure includes three factorially defined scales (social/communication, unusual behaviors, self-regulation), eight content derived treatment scales (peer socialization, adult socialization, social/emotional reciprocity, atypical language, stereotypy, behavioral rigidity, sensory sensitivity, attention) and a scale based on the currently accepted diagnostic criteria for ASD. A total score is also generated.
The first step in creating a treatment plan once a diagnosis is made is to clarify the specific area or areas of need. We suggest that any nationally normed standard score from a scale like the ASRS that is above one standard deviation (e.g., in the top 16% relative to problem behavior) indicates that the child rated has many behavioral characteristics similar to youth diagnosed with ASD. Taking all the ratings by parents and teachers as a whole, the next step is to identify an intervention plan based on the profile of scores for the ASRS treatment scales. For example, in the case of Donny (see Table 1) we begin by prioritizing the areas of need based on the magnitude of the T-scores. Donny’s highest T-scores were on the Social/Emotional Reciprocity scale rated by parents and teachers. This scale involves specific behaviors such as looking at others appropriately while talking with them, understanding the feelings of others, recognizing social cues, responding appropriately to other people’s statements, interests or feelings and enjoying interacting with others. In order to have a more precise understanding of the exact behaviors that contributed to this high score or a high score on any of the treatment scales, we conduct an item level analysis to identify the greatest need relative to behaviors to improve. This can be accomplished by determining when an item rating is substantially higher than the item average from the normative group. Analysis of the treatment scales and the items included on those scales can be used to identify which specific behaviors warrant intervention. In our text (Goldstein and Naglieri, 2013) we provide a Quick Solution Guide directly keyed to specific behaviors. This allows us to select interventions associated with each behavioral need. In Donny’s case, the next scale that warrants intervention is Peer Socialization. Behaviors in this scale involve seeking the company of other children, talking with other children, choosing to play with peers and responding when spoken to. These behaviors can then be identified and compiled into a functional treatment plan.
Once treatment is begun, it is important to monitor the effect of the interventions over time. We assume that treatment of children with ASD takes time and therefore progress towards goals should be evaluated as frequently as possible during treatment. This may include traditional methods of evaluating specific behavior change (e.g., applied behavioral analysis) but should also include normative data from a rating scale such as the ASRS to calibrate change from the pre-treatment period. For example, raters who complete the ASRS are informed to evaluate the child based upon the behaviors observed during the previous four weeks. The combination of specific behavioral change and standard scores from a norm referenced measure provides a balanced view of progress. We suggest that evidence of treatment effectiveness is strongest when the pre and post-intervention behaviors related to ASD are evaluated using nationally calibrated scores. The approach we recommend is based on a dual criterion of statistically reliable differences and clinically meaningful change. In order to determine if the differences are related to measurement error or actual change brought about by treatment the statistical difference between the two scores should be determined using sound measures. The values needed for significance in using the ASRS are provided for each treatment scale in the test manual. Table 2 provides an example of the differences needed to demonstrate treatment success over three rating periods. This method helps provide families and treatment providers with a valid and reliable means of assessing difference. Small differences in pre and post ratings with low levels of significance would indicate that the strategies chosen to address specific behaviors were in effective and should be reconsidered.
The process of assessment of ASD requires more than just a diagnostic pronouncement. It requires the collection of well-defined behavioral data facilitating a smooth transition between assessment, treatment planning and the evaluation of treatment effectiveness. It is critical that families, treatment providers and evaluators understand these issues and the sound psychometric procedures necessary to assess effectiveness over time in children with ASD.
*Note: This article is abridged from: Evaluation and Treatment Effectiveness in the Field of Autism: Psychometric Considerations and an Illustration by Naglieri, J.A. & Goldstein, S. (in press), in J.A. Naglieri and S. Goldstein Handbook of Autism Treatment, New York: Springer Publishers
Sam Goldstein, PhD is a Clinical Assistant Professor of Psychiatry at the University of Utah School of Medicine. He can be contacted at info@samgoldstein.com. Jack Naglieri, PhD is Research Professor at the University of Virginia. He can be contacted at jnaglieri@gmail.com.
References
Goldstein, S., & Naglieri, J.A. (2009). Autism Spectrum Rating Scale. Toronto: Multi-Health Systems.
