High quality implementation of evidence-based instructional and intervention strategies has been identified as critical for educators supporting autistic students. Using evidence-based practices (EBP) is required by policy (Every Student Succeeds Act [ESSA], 2015, and the Individuals with Disabilities Act [IDEA], 2004) and leads to better student outcomes (Durlak & DuPre, 2008). At least 28 well-defined EBP have been identified based on research demonstrating improved outcomes in autistic students (Steinbrenner et al., 2020), but they are not consistently used within school programs. Multiple barriers to school-based scale up and implementation limit autistic students’ access to EBP and best outcomes, especially in schools with limited resources. Effective EBP use and sustainment can be improved with provision of supports at multiple levels, including support for providers and school district leaders, and changes at the higher system level. This paper describes barriers and supports identified as part of a systematic state-wide effort to improve scale up and implementation of autism EBP in schools.
A teacher in class with her students
California Autism Professional Training and Information Network
CAPTAIN (California Autism Professional Training and Information Network; Suhrheinrich et al., 2020) is a statewide multi-agency collaborative network focused on dissemination and implementation of EBP for autistic individuals across service systems. CAPTAIN uses an implementation science framework and quality improvement cycles to guide member training and use of implementation strategies. For example, to ensure organizational support and resources, key drivers of strong implementation (Aarons et al., 2017), agency leaders nominate staff to participate in CAPTAIN and provide protected time to attend events.
CAPTAIN is organized into 17 regional teams across the state that develop regional plans for information dissemination about autism and EBPs, promoting effective EBP implementation and cross-agency regional collaboration and coordination. CAPTAIN currently has over 400 members representing special education, developmental disabilities, family support services and university programs.
Because access to high quality training and coaching increases high quality EBP use, CAPTAIN members are expected to provide to at least 3 educators annually at least a) 1 awareness training about autism and EBPs, b) 3 trainings on specific EBPs, and c) EBP-specific coaching. Data collection and monitoring has been shown to improve use of EBP (Aarons et al., 2017), CAPTAIN members use research-supported strategies such as pre/post knowledge assessments, EBP fidelity monitoring and child outcome data collection to support practice sustainment. Last year CAPTAIN members who work in schools (n=333) provided EBP training to almost 20,000 people (including teachers, paraprofessionals, administrators, and parents) and individualized coaching to over 6,500 educators, showing considerable reach. A recent study published in Autism found that CAPTAIN-trained educators reported more positive attitudes toward using EBP, were more likely to use the strategies correctly and more often with students (Suhrheinrich et al., 2022). They had more in-depth knowledge of their primary EBP and reported higher classroom quality than educators who did not get CAPTAIN training. These findings show great promise for CAPTAIN as a model to support statewide use of EBPs for autism.
Factors Impacting EBP Scale-up
Although results indicate good reach and training quality based on educator report, CAPTAIN members differ in their completion of training and coaching requirements. A recent project (IES R324A170063) leveraged the CAPTAIN network to identify factors that support the success of members getting EBP into schools.
Social Networks and EBP Knowledge – Considerable evidence indicates social networks improve EBP practices in education (Frank et al., 2004). Additionally, knowledgeable coaches tend to be better at disseminating EBP than those with limited knowledge (Rodway, 2019). In a recent study (Hassrick et al., 2021), we examined the relationship between CAPTAIN member social networks around EBP knowledge, coaching support and financial resources and the quality of coaching provided by 228 CAPTAIN members. Results confirmed that EBP knowledge was key to increased quality of coaching practices, while job position and years of experience were not. Additionally, CAPTAIN members with larger EBP coaching networks had better training outcomes. Networks appeared more important than location (rural, urban) or organizational structures (serving single vs. multiple school districts).
Leadership & Organizational Support – Positive implementation climate (or the extent EBPs are expected, supported, and rewarded) has been linked to better EBP sustainment, improved child outcomes, and decreased staff burnout (Novins et al., 2013). Implementation leadership, or leader readiness and support for using EBP, also drives the success of EBP use and improves climate (Aarons et al., 2017; Melgarejo et al., 2020). We examined ratings of implementation leadership and climate through surveys with 656 educators and focus groups with 30 CAPTAIN members in 28 regions across California (Melgarajo et al., in review). Generally, implementation climate is low in schools. Schools have relative strengths in their focus on EBP in planning and meetings and hiring educators open to EBP use but rarely provide rewards for EBP use. These findings align with school mental health literature (Lyon et al., 2018). Implementation leadership was also rated in the low to moderate range, suggesting this as an area for potential growth. School leaders were seen as supportive of EBP use however, had a weakness in their ability to be proactive in planning for EBP implementation.
Implications
Scaling up the use of EBP regionally or state-wide is challenging. Educational programs that use implementation strategies at both the organizational and the individual provider level report greater success than those who do not have clear implementation plans (Fixsen et al., 2007). Our statewide data indicate a need for improvement in setting the context for successful implementation through improving implementation climate and leadership, trainer and provider EBP knowledge, and increasing communities of practice or networks to support for EBP training and use. CAPTAIN has utilized data to promote improvements in knowledge, leadership support and networks, and the encouraging outcomes suggest several policy and practice implications.
Support is needed at the organizational level to set the context for successful EBP implementation. Educational systems need to improve the resources, supports and recognition provided to facilitate high quality EBP use. Training is required for leaders at all levels to increase knowledge of implementation and improvement sciences and to understand the change process and the supports required for sustainment of quality EBP use. Simply sending educators to workshops or trainings will not translate into better practices. Educators require training accompanied by ongoing coaching and support from an individual knowledgeable about EBP and recognition and support from their leaders. Technical assistance and training providers should be hired based on their knowledge and expertise with the EBP rather than overall years of experience or job position. Communities of practice can facilitate social networks which provide advice and support to both trainers and educators implementing EBP. These networks can improve high quality EBP use, better coaching and training, and improve teacher attitudes toward EBP which, when accompanied by access to high quality training and coaching, improves EBP fidelity and frequency of use. Schools require a stronger focus on implementation climate. Currently educators receive very little incentive to go through the challenge of implementing complex EBP in classrooms.
Currently, many EBP implementation efforts are initiated reactively to solve a problem such as compliance violation or legal issues rather than proactively through needs assessments or improvement processes. Although event-driven reactivity provides opportunities to secure needed resources, it also makes EBP implementation susceptible to short-term planning and quick initiation, which can negatively impact quality and sustainment (Rubin et al., 2016). Proactive planning across levels, appropriate allocation of resources and involvement of an implementation plan for leaders, trainers and educators will facilitate effective use of EBP and improve student outcomes.
Jessica Suhrheinrich is an Associate Professor of Special Education at San Diego State University. Patricia Schetter is a project coordinator at the UC Davis MIND Institute. Melina Melgarejo is a Research Scientist at San Diego State University. Yue Yu is a post-doctoral fellow at the UC Davis MIND Institute. Elizabeth McGhee Hassrick is an Assistant Professor at the Drexel University, AJ Drexel Autism Institute. Aubyn Stahmer is a Professor of Psychology and Behavioral Science at the UC Davis MIND Institute. For more information, please contact jsuhrheinrich@sdsu.edu.
References
Aarons, G.A., Ehrhart, M.G., Torres, E.M., Finn, N.K. & Beidas, R.S. (2017). The Humble Leader: Association of Discrepancies in Leader and Follower Ratings of Implementation Leadership With Organizational Climate in Mental Health. Psychiatric Services, 68 (2), 115-122. https://doi.org/10.1176/appi.ps.201600062
Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review of research on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41(3), 327-350. https://doi.org/10.1007/s10464-008-9165-0
Every Student Succeeds Act. (2015). Public Law 114 – 95 – GovInfo. https://www.govinfo.gov/app/details/PLAW-114publ95
Fixsen, D. L., Blasé, K. A., Timbers, G. D., & Wolf, M. M. (2007). In search of program implementation: 792 replications of the Teaching-Family Model. The Behavior Analyst Today, 8(1), 96-110. http://dx.doi.org/10.1037/h0100104
Frank, K. A., Zhao, Y., & Borman, K. (2004). Social capital and the diffusion of innovations within organizations: The case of computer technology in schools. Sociology of Education, 77(2), 148-171.
Individuals with Disabilities Education Act (2004), Public Law 101-476, GovInfo. https://www.govinfo.gov/app/details/CREC-2004-11-20/CREC-2004-11-20-pt1-PgS11669
Lyon, A.R.; Cook, C.R.; Brown, E.C., Locke, J.; Davis, C.; Ehrhart, M.; & Aarons, G.A. (2018). Assessing organizational implementation context in the educational sector: Confirmatory factor analysis of implementation leadership, climate, and citizenship. Implementation Science 13 (1), 1-14
Melgarejo, M., Nahmias, A.S., Suhrheinrich, J., Li, J., Schetter, P., & Stahmer, A. (under review). Exploring organizational differences in perceptions of implementation climate and leadership in schools: A mixed methods study. Focus on Autism and Other Developmental Disabilities.
Melgarejo, M., Lind, T., Stadnick, N.A., Helm, J.L., & Locke, J. (2020). Strengthening capacity for implementation of evidence-based practices for autism in schools: The roles of implementation climate, school leadership, and fidelity. American Psychologist, 75(8), 1105-1115. doi: 10.1037/amp0000649
Novins, D. K., Green, A. E., Legha, R. K., & Aarons, G. A. (2013). Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review. Journal of the American Academy of Child and Adolescent Psychiatry, 52(10), 1009–1025.e18. https://doi.org/10.1016/j.jaac.2013.07.012
Rodway, J. (2019). Coaching as a Knowledge Mobilization Strategy: Coaches’ Centrality in a Provincial Research Brokering Network. International Journal of Education Policy and Leadership, 14(5), n5.
Rubin, R. M., Hurford, M. O., Hadley, T., Matlin, S., Weaver, S., & Evans, A. C. (2016). Synchronizing watches: The challenge of aligning implementation science and public systems. Administration and Policy in Mental Health and Mental Health Services Research,43(6), 1023–1028.
Steinbrenner, J. R., Hume, K., Odom, S. L., Morin, K. L., Nowell, S. W., Tomaszewski, B., Szendrey, S., McIntyre, N. S., Yücesoy-Özkan, S., & Savage, M. N. (2020). Evidence-Based Practices for Children, Youth, and Young Adults with Autism. FPG Child Development Institute.
