Autism CARES OMB Supporting Statement B

Supporting Statement B

Health Resources and Services Administration

Maternal and Child Health Bureau

Autism CARES Act Evaluation

OMB Control No. XXXX-XXXX

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Representatives from all grantee sites receiving Health Resources and Services Administration (HRSA) grant support under the Autism Collaboration, Accountability, Research, Education, and Support (CARES) Act of 2019 (attachment A) constitute the respondent universe for the collection of data. Grantee representatives may include principal investigators, project directors, and/or research assistants. The following are the grantee sites.

Research Programs (17)

Two types of research programs are in place: networks and individual grants.

Research Networks: Four Research Networks advance the evidence base on effective interventions for individuals with autism/DDs, develop guidelines, and disseminate information. The Autism Intervention Research Network on Physical Health (AIR-P) focuses on improving the physical health and well-being of individuals with autism across the lifespan. Through its interdisciplinary, multisite network, AIR-P supports innovative life course intervention research that promotes optimal health and well-being of individuals with autism. The areas of focus include primary care services and quality; community-based lifestyle interventions; gender, sexuality, and reproductive health; health systems and services; neurology; and genetics.

The Autism Intervention Research Network on Behavioral Health (AIR-B) focuses on improving the behavioral, mental, social, and/or cognitive health of individuals with autism/DDs and developing evidence-based interventions, treatments, and tools. The AIR-B network conducts multisite research to develop evidence-based autism interventions, treatments, and tools. This program uses community-based, participatory research methods to ensure parents and providers can effectively implement evidence-based interventions at home and at school. AIR-B recruits participants from diverse racial/ethnic, rural, and socioeconomic groups and conducts studies in underresourced settings.

The Developmental-Behavioral Pediatrics Research Network (known as DBPNet) promotes and coordinates research activities in developmental, behavioral, and psychosocial aspects of pediatric care. The network focuses efforts on (1) providing an infrastructure to support rapid scientific discovery; (2) conducting ongoing, interactive, and collaborative activities among network members; (3) leveraging innovative research projects and network capacity to compete for grant opportunities from other Federal agencies or private foundations; and (4) fostering research and mentorship opportunities for young or new investigators.

The Research Network on Promoting Healthy Weight Among Children With Autism/DDs (HW-RN) works to advance the evidence base for the prevention and treatment of obesity and enhance healthy weight in this high-risk group. HW-RN is an interdisciplinary research network that leads and promotes coordinated research activities related to promoting healthy weight among children and youth with autism/DDs, with an emphasis on reaching underserved and underrepresented communities.

Individual Research Grants: MCHB supports 11 research grants composed of the following:

• Two Autism Single Investigator Innovation Programs (Autism SIIP): the Autism Transitions Research Project and Autism Longitudinal Data Project

The goal of the Autism SIIP program is to support empirical research and advance the evidence base on interventions and investigate barriers to screening, diagnosis, and receipt of evidence-based autism/DD interventions to improve the health and well-being of children and adolescents with autism, with a special focus on addressing the needs of underserved populations.

• Two Autism Field-Initiated Innovative Research Studies Program (Autism FIRST) grantees

The goal of the Autism FIRST program is to support research on interventions intended to develop and test ways to effectively tailor services, supports, and interventions to individual and family strengths, needs, and challenges and identify services and supports that mediate or moderate relations between family stresses and outcomes for children, adolescents, and young adults with autism/DDs.

• Seven Autism Secondary Data Analysis Research Program (Autism SDAR) grantees (funded for a period of 1 year)

The purpose of the MCH SDAR program is to support applied maternal and child health (MCH) research that uses secondary analysis of available national datasets and/or administrative records to improve the health and well-being of MCH populations.

State Systems (6)

State Innovations in Care Coordination Grant: Five States were awarded Innovations in Care Coordination Grants in September 2019, and a new cohort of five grantees is expected to be awarded in 2023. The State Systems grants provide grantees with the funding needed to implement family navigation and provider training in medically underserved areas to improve access to coordinated and integrated care for children with autism/DDs.

The State Systems grantees are supported by the State Public Health Coordinating Center for Autism. Funded through a cooperative agreement with MCHB, the Center provides technical assistance to the State System grantees and other State entities such as Title V programs and serves as a comprehensive web-based resource center for the State programs.

Training Programs (73)

Two types of training programs are in place: Leadership Education in Neurodevelopmental and Related Disabilities (LEND) and Developmental-Behavioral Pediatrics (DBP) training programs.

Sixty LEND programs have been established, with the following primary objectives:

• Provide high-quality interdisciplinary training to child health professionals from diverse disciplines.

• Prepare those professionals to assume leadership roles in their respective disciplines.

• Provide interdisciplinary services and care to children with special healthcare needs and their families.

LEND funding was designed primarily to support the training of professionals who can provide screening, diagnostic evaluation, and evidence-based intervention for individuals with autism/DDs. Several LEND programs also contribute to the evidence base for autism identification and treatment through participation in research. Consistent with MCHB’s mission, LEND grantees have the related objective of improving the overall system of services available to children with autism/DDs and their families.

Twelve DBP programs receive Autism CARES funds to enhance the behavioral, psychosocial, and developmental aspects of pediatric care. The primary objectives of DBP follow:

• Support and prepare developmental-behavioral pediatric fellows for leadership positions in teaching, research, and clinical care.

• Provide pediatric practitioners, residents, and medical students with essential biopsychosocial knowledge and expertise.

The training programs are supported by the Interdisciplinary Training Resource Center (ITRC) on Autism and Developmental Disabilities. One grantee received funding to serve as the technical assistance and resource center to the LEND and DBP grantees.

Because of the small number and unique nature of each grant program, a census of all programs will be conducted. We expect a 100 percent response rate because participation in the evaluation is part of the grant agreement between HRSA and the grantees. The findings will not be generalized beyond the respondents.

2. Procedures for the Collection of Information

The study is designed to measure the collective impact of MCHB’s Autism CARES Act initiative and assess grantees’ progress in meeting the objectives of HRSA’s Maternal and Child Health Bureau under the legislation. These objectives include increasing awareness of autism and other developmental disabilities (DDs); training professionals to use valid, reliable screening and diagnostic tools; providing evidence-based interventions for individuals with autism/DDs; supporting research to advance evidence-based or consensus-based guidelines for interventions; improving the overall system of services for individuals with autism/DDs; and reducing barriers to screening and diagnosis.

Information in the form of qualitative data will be gathered from a one-time semi-structured interview with the principal investigator or project director from each grant program.

The semi-structured interviews will provide data pertaining to the grantees’ implementation activities (e.g., training of maternal and child health providers) and the intermediate results of those activites (e.g., screening/evaluation of underserved children by trained fellows or faculty), which are expected to support the long-term goals of early screening, diagnosis, and treatment of individuals with autism/DDs.

General Data Collection Procedures

Primary data collection will involve a survey for the Research programs, a survey for the State Systems program, and in-depth interviews with all grantees across all three program areas (Research, Training, and State Systems). In fall 2023, HRSA Project Officers will notify all grantees that the evaluation team will be contacting them via email about the survey and the interviews.

Research and State Systems Surveys (attachments B and D). After HRSA Project Officers have sent a notification email to the Research and State Systems grantees, the evaluation team will follow up with an email containing instructions for survey completion. This email will explain the purpose of the study, instructions on how to access the web survey, and requested time to return the completed survey (10 business days). Grantees will receive the evaluation team’s email and a phone number they can call if they encounter technical problems while completing the survey.

After the 10th business day, the evaluation team will send up to two reminder emails to any grantees who have not completed their surveys. The reminder emails will include survey information, a link to the web survey, contact information for the help desk, and a reminder of the web survey due date.

Two weeks before the end of the survey data collection period, a member of the evaluation team will follow up with a final telephone call with grantees who have not completed the survey to encourage web survey completion, answer questions, and offer an opportunity to complete the survey over the telephone. If any grantees elect to complete the survey via telephone, a trained member of the evaluation team will administer the survey and record survey responses using the grantee’s unique survey link.

At the conclusion of the survey data collection period, a thank-you email will be sent to all participating grantees. HRSA anticipates all Research and State Systems grantees will complete the survey once during the second year of the evaluation (fall 2023).

Semi-Structured Interviews (attachments C, E–H). In the second year of the evaluation, all grantees funded between fiscal year (FY) 2021 through FY 2022 will be invited to participate in a semi-structured interview. Interviews will be conducted after fielding of the surveys is complete.

One week after HRSA Project Officers have sent the invitation email, a trained interviewer from the evaluation team will begin contacting grantees via telephone to confirm receipt of the email, answer questions, and schedule a semi-structured interview. Two weeks after sending the invitation email, the interviewer will send a reminder email to those grantees who have not scheduled their semi-structured interviews. For these remaining grantees, a second telephone call will be made to attempt to schedule an interview. Within 1 week of the scheduled interview, the interviewer will send an email to the grantee to confirm the interview date and time.

The interviews will be conducted over the telephone using the appropriate semi-structured guide. With the permission of the respondent(s), the interviews will be recorded using a conference call interface with recording capability, such as Microsoft Teams. The interviews are expected to average between 60 and 75 minutes, depending on respondent type. At the conclusion of the entire data collection, a thank-you email will be sent to all grantees who participated in the semi-structured interview.

Table B.1 provides a crosswalk of each of the evaluation questions and objectives by data source. Each instrument referenced in this table is included in full as attachments B–H. Not all evaluation questions may be completely answered by these instruments; however, these data will be supplemented by the Discretionary Grant Information System performance measures data.

Table B.1. Evaluation Question and Objective by Data Source

Evaluation Question	Data Collection Instrument
	Research Program Interview Guide (Attachment C)	Research Program Survey (Attachment B)	State Systems Interview Guides (Attachments E and F)	State Systems Survey (Attachment D)	Training Programs Interview Guides (Attachments G and H)
Research
R1.1. How have the autism investments (AIR-B, AIR-P, DBPNet, HW-RN, SIIP, FIRST, and SDAR awards) accomplished the following processes: Advanced interdisciplinary research on effective interventions and guidelines to improve the health and well-being (physical and mental health) of autistic individuals across the lifespan Incorporated lived experience in research priorities and implementation Conducted original research studies on effective interventions designed to improve the health and well-being of individuals with autism across the lifespan Conducted research using existing national datasets and/or administrative records designed to improve the health and well-being of individuals with autism across the lifespan Disseminated research to reach broad audiences including researchers, clinicians, educators, policymakers, and individuals with autism and their families	X	X
R1.2. How have the research findings from autism research investments accomplished the following priorities of the Division of Research in the Office of Epidemiology and Research: Promoted health equity (e.g., through the engagement of community partners, selection of research topics and methods, use of analytic methods, dissemination/translation of research findings) Incorporated lived experience in research (e.g., through engagement of partners, research prioritization and implementation, dissemination/translation of findings)	X	X
R1.3. What new evidence have autism research investments generated in the following focus areas? Transitions Risk factors and life course factors Health and well-being, including clinical improvements, healthy weight, and quality of life	X	X
R1.4. How have research findings from autism research investments been translated to promote changes in practice, programs, and policies? Toolkits, guidelines, policy beliefs, factsheets, etc.	X	X
T1.1. What specific research opportunities are the training programs offering to long-term, medium-term, and short-term trainees?					X
Awareness
R2.1. How have research programs increased awareness of autism and developmental disabilities? Content/topics covered in dissemination Modes of dissemination/products: peer-reviewed articles, guidelines, presentations, website, other Reach of research dissemination Engagement of key partners in translating findings into practice (health professionals, researchers, policymakers, family members, public)	X	X
T2.1. How have training programs increased awareness of autism and developmental disabilities? Increased awareness of autism and developmental disabilities resources and systems of care Promoted screening and evaluation tools, including those that can be used via telehealth Promoted evidence-informed and evidence-based dissemination of new/revised curricula, products, training approaches, research findings, and project results Increased understanding of lived experiences of persons with autism and developmental disabilities and their families Targeted specific populations in training programs’ awareness activities					X
S2.1. How have the State Systems programs increased awareness of autism and developmental disabilities? How has increased awareness improved the well-being of children and youth with autism and developmental disabilities?			X
S2.2. What initiatives were developed or promoted for family navigation?			X
S2.3. What initiatives were developed or promoted for the following other aspects of services? Improving timely provider referrals to comprehensive developmental screening before age 3 Ensuring access to community-based services and supports for children before age 3 Improving care coordination for persons with autism or those with increased likelihood of autism and/or developmental disabilities			X	X
Systems of Care
T3.1. How have the training programs monitored changes in access to screening, diagnosis, and community-based services and supports? Existing gaps in screening, diagnosis, and treatment of autism and developmental disabilities Existing gaps in services for individuals with autism and developmental disabilities (e.g., insurance, housing, employment, transportation) Most efficient ways to address service gaps How services gap changes over time					X
T3.2. To what extent have training programs reduced barriers to autism and developmental disabilities screening, diagnosis, and community-based supports through the following activities? Technical assistance provided to State and local partners Continuing education provided to practicing providers Other activities the training programs conducted to reduce barriers					X
T3.3. Have the training programs seen changes in the age at diagnosis? Age changes and direction of changes Period of time over which the changes occurred Suspected reasons (environmental, systemic, COVID-19, budgetary, etc.) for change					X
S3.1. What activities did the State Systems programs conduct to improve the health and well-being of individuals with increased likelihood for autism and developmental disabilities in underserved areas (areas that have barriers to access and services) and populations (including populations with different cultures and languages)?			X	X
S3.2. What activities and support did the State Systems grantees and the 59 Title V programs receive from the State Public Health Coordinating Center for Autism to positively influence statewide systems of care and innovation?			X
Equity
R4.1. Describe each research program’s study populations, including the inclusion of populations that have been historically underserved and/or marginalized	X
R4.2. In what ways have the research programs addressed health disparities?	X
T4.1. What strategies did the training programs use to engage trainees and faculty from populations that have been historically underserved or marginalized, including those from different racial and ethnic groups and geographic areas? Raised awareness of training and employment (faculty recruitment, employment for former trainees, etc.) opportunities Recruitment and retention of trainee and faculty cohorts representing a diversity of backgrounds (race/ethnicity, geography, disability, etc.)					X
T4.2. What strategies did the training programs use to reduce disparities in access to screening, diagnosis, and entry into services among populations that have been historically underserved or marginalized?					X
T4.3. How did the training programs support and promote diversity, equity, inclusion, and accessibility (DEIA) and health equity through their training (e.g., incorporating principles from Universal Design for Learning, intersectionality, environmental adaptations)?					X
T4.4. What support did the LEND and DBP programs receive from the ITRC in promoting DEIA and health equity in their training programs?					X
S4.1. What strategies did the State Systems programs use to improve access to community-based services and interventions for underserved populations?			X
S4.2. What support did the State Systems programs receive from the State Public Health Coordinating Center to address the needs of underserved communities and populations? In what way did this influence improvement in services for these populations?			X
S4.3. Are there other strategies/activities the State Systems programs use to address health equity?			X
Transitions/Lifespan
R5.1. How did the different research programs include transition topics and populations in their activities? Transitions research topics studied Transition-age populations Partners engaged in the research (across all aspects of research, from planning to dissemination)	X
T5.1. How did training grantees address transition and lifespan-related topics in their curricula? What strategies have the training programs used to incorporate adult-focused providers to improve transition and adult services?					X
T5.2. How has incorporating people with lived experience as trainees and faculty improved understanding of transition and lifespan-related topics?					X
S5.1. How have the State Systems programs used family navigation services to help facilitate transitions across settings and providers, and from pediatric to adult systems of care?			X
S5.2. How have the State Systems programs engaged with families?			X
S5.3. What support did the States Systems grantees receive from the State Public Health Coordinating Center for Autism to address transition to adulthood (including healthcare) in its activities?			X
Resource and Coordinating Centers
T6.1. How did the Interdisciplinary Training Resource Center conduct the following activities? Including support related to health equity and transitions. Provide technical assistance to the LEND and DBP programs Identify and disseminate resources and tools Conduct data analysis and management Advance systems development strategies Promote involvement of people with lived experience (i.e., family members and self-advocates) Support forums for collaboration, sharing of effective practices and opportunities to address emerging issues (e.g., support LEND and DBP workgroups) Form partnerships with Federal agencies and organizations related to the community with autism and developmental disabilities (CDC, ACL, etc.)					X
S6.1. What technical assistance and resources did the State Public Health Coordinating Center for Autism provide to support the State Systems autism awardees and their partners in accomplishing the following: Increase awareness and knowledge of (1) shared resources, (2) telehealth, and (3) family navigators Build capacity to address newly identified implementation barriers and/or emerging issues impacting the target populations Share information across Autism CARES Act awardees to ensure coordinating center’s work informs and influences MCHB autism investments in research, training, and State public health Develop and/or sustain partnership to support contributions to autism and developmental field Develop and disseminate evidence-based or evidence-informed policies, resources, and strategies broadly to children and youth with special health needs (CYSHCN) systems, relevant MCHB awardees, and CYSHCN community Focus effort and lessons learned on development of measures and data collection systems of care to facilitate improved assessment and evaluation of autism and developmental disabilities initiative			X
S6.2. What technical assistance and resources were provided to other entities other than the State Systems autism awardees to address any areas on the list above?			X
COVID-19
R7.1. For programs that were operating before COVID-19, what changes, pivots, or adaptations did the research programs have to make because of COVID-19?	X
R7.2. What positive and negative effect(s) did these changes have on the research program?	X
R7.3. Will the research programs continue with any changes made because of COVID-19, or revert back to the original proposed plan? Describe.	X
T7.1. For programs that were funded before COVID-19, what changes did the training programs make in response to the pandemic that impacted training of autism/DD providers?					X
T7.2. What positive and negative effect(s) did these changes have on the training programs?					X
T7.3. How, if at all, are the training programs planning to continue or revise any of the changes made because of COVID-19?					X
T7.4. How has COVID-19 positively or negatively impacted access to screening, diagnosis, and entry into services?
S7.1. For programs that were operating before COVID-19, what changes did State Systems programs have to make because of COVID-19?					X
S7.2. What effect(s) (both positive and negative) did these changes have on the State Systems programs?			X
S7.3. Are the State Systems programs planning to continue any of the changes they made because of COVID-19?			X
S7.4. What changes to the planned technical assistance, resources, and support did the State Public Health Coordinating Center for Autism make because of COVID-19?			X

ACL = Administration for Community Living; AIR-B = Autism Intervention Research Network on Behavioral Health; AIR-P = Autism Intervention Research Network on Behavioral Health; CDC = Centers for Disease Control and Prevention; DBP = developmental behavioral pediatrics; DBPNet = Developmental-Behavioral Pediatrics Research Network; DD = development disability; FIRST = Autism Field-Initiated Innovative Research Studies Program; HW-RN = Research Network on Promoting Healthy Weight Among Children With Autism/DDs; ITRC = Interdisciplinary Training Resource Center; LEND = Leadership Education in Neurodevelopmental and Related Disabilities; SDAR = Secondary Autism Data Analysis Research Program; SIIP = Autism Single Investigator Innovation Program

3. Methods to Maximize Response Rates and the Issue of Nonresponse

The evaluation team presented the evaluation design to the Autism CARES grantees March 22, 2023, describing the plans to conduct interviews and administer surveys. In July 2023, the evaluation team will also engage grantees about the data collection during a presentation of preliminary findings from secondary data analysis from the first year of the evaluation. During this meeting, the evaluation team will provide updates to primary data collection activities, timeline, and expectations. HRSA grantees who receive support through Autism CARES are required, as a condition of their grants, to provide the requested information to HRSA and to participate in the evaluation with the evaluation team. Therefore, nonresponse is not anticipated as an issue.

4. Tests of Procedures or Methods to be Undertaken

The Autism CARES Act evaluation has been conducted over several years, where grantees were engaged via surveys and interviews many times. Each iteration of primary data collection involved the steps described in this document. These procedures have yielded full participation from grantees in previous rounds of data collection and therefore are expected to be successful.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The evaluation’s subject matter experts (SMEs) reviewed the surveys and semi-structured interview guides. These SMEs were well versed in research and the grant programs themselves. They were asked about the ease of comprehension (e.g., confusing phrasing or layout, confusing concepts) and length of time to complete. All instruments were revised to incorporate SME feedback including clarifying statements, removing repetitive questions, and adding definitions of terms throughout the surveys.

Table B.2 presents a summary of individuals consulted on study design, data collection, and/or analysis.

Table B.2. Individuals Consulted on Study Design, Data Collection, or Analysis

Name	Title	Telephone Number	Email
Primary Contractor, 2M Research
Katherine Flaherty	Project Director	508.613.5990	KatherineFlaherty@westat.com
Carla Bozzolo	Deputy Project Director	571.758.5036	cbozzolo@insightpolicyresearch.com
Claire Wilson	Senior Advisor	703.504.9484	cwilson@insightpolicyresearch.com
External Subject Matter Experts
Margaret Bauman	Child Neurologist and Associate Professor	617.733.2992	mbauman@iccdpartners.org
Karen Kuhlthau	Associate Professor of Pediatrics and Associate Sociologist	617.724.2842	kkuhlthau@mgh.harvard.edu
HRSA Staff
Leticia Manning	Public Health Analyst and Contracting Officer’s Representative	301.443.8335	LManning@hrsa.gov

Table B.3 presents a summary of evaluation team staff participating in study design, data collection, and/or analysis.

Table B.3. Evaluation Team Staff Participating in Study Design, Data Collection, or Analysis

Name	Title	Telephone Number	Email
Primary Contractor, 2M Research
Katherine Flaherty	Project Director	508.613.5990	KatherineFlaherty@westat.com
Carla Bozzolo	Deputy Project Director	571.758.5036	cbozzolo@insightpolicyresearch.com
Claire Wilson	Senior Advisor	703.504-9484	cwilson@insightpolicyresearch.com
Debra Wright	Lead for Training Programs	703.504.9486	dwright@insightpolicyresearch.com
Zarina Fershteyn	Lead for Research Programs	678.708.1353	ZarinaFershteyn@westat.com
Shannon Griffin	Co-lead for Research Programs	571.758.5007	sgriffin@insightpolicyresearch.com
Lyndsay Huey	Analyst for Training Programs	571.451.2926	lhuey@insightpolicyresearch.com
Paul Reischmann	Analyst for Training Programs	571.758.5015	Preischmann@insightpolicyresearch.com
Rikki Welch	Lead for State Systems	571.758.5027	rwelch@insightpolicyresearch.com
Victoria Martin	Analyst for Training and State Programs	571.347.1032	vmartin@insightpolicyresearch.com
Lizzie Nelson	Analyst for Training Programs	571.758.8024	lnelson@insightpolicyresearch.com
Shristi Tiwari	Analyst for Training and Research Programs	336.406.3069	ShristiTiwari@westat.com
Olivia Iles	Data Visualization Analyst	703.914.3762	OIles@insightpolicyresearch.com

Attachments to Supporting Statement B

Appendix A. Autism CARES Act of 2019 (P.L. 116–60)

Appendix B. Research Grantee Survey

Appendix C. Research Grantee Interview Guide

Appendix D. State Systems Grantee Survey

Appendix E. State Systems Grantee Interview Guide

Appendix F. SPHARC Interview Guide

Appendix G. LEND/DBP Interview Guide

Appendix H. ITRC Interview Guide