Detailed Description of AWARE TISS Evaluation Components

Detailed Description of Project AWARE-TISS Cross-Site Evaluation Components

A detailed description of each evaluation component, associated study, relevant evaluation questions, and data collection activities are described in the sections that follow.

Process Evaluation Studies

Implementation and Sustainability Study

As noted, the literature is beginning to demonstrate the impact and value of providing mental health supports and services in schools. The National Center for School Mental Health has summarized the positive outcomes resulting from implementation of comprehensive school mental health systems that address the full array of these services and supports, including mental health promotion, prevention, early identification, and treatment. Such outcomes include a positive impact on social-emotional outcomes and academic performance, improved school climate and safety, enhanced early identification and intervention, and improved access to services (Hoover et al., 2019). As a result, public interest in integrating related mental health programs into schools has grown over the past two decades, along with the recognition that significant guidance is needed about effective delivery and implementation models. Schools have reported struggling with implementation challenges such as gaining teacher and administrator buy-in, limited school personnel time and resources, lack of training and administrative support, school accountability emphasizing academic rather than social-emotional outcomes, and limited parent involvement (National Center on Safe and Supportive Learning Environments [NCSLE], 2021; Forman et al., 2009; Langley et al., 2010; Baffsky, 2023). In addition, while a range of evidence-based practices are available for implementation, little is known about the supports required to optimize implementation (Baffsky, 2023). By investing in the AWARE and TISS programs, national leaders are responding to calls for resources to expand and sustain mental health supports and services in schools, helping to improve access to culturally appropriate school-based mental health programs for a wider range of children, youth, families, and communities. Broader implementation of these programs also provides an opportunity to build the evidence base related to implementation of school-based mental health programs, including improving understanding of how these programs work, why they work, for whom, and under what conditions.

The Implementation and Sustainability study is designed to assess AWARE and TISS program implementation and sustainability overall including in high-need subpopulations and under-resourced communities. Key study goals include gaining an in-depth understanding of how program activities and strategies are implemented, whether they were implemented as intended, and how the programs will be sustained beyond the federal funding period. This study will document the critical factors including barriers and facilitators that influence AWARE and TISS program delivery. Additionally, this study will attempt to contextualize these two programs and assess each program and their partners staff, school personnel, children, youth, and other stakeholder perceptions of the program and experience. The proposed instruments and measures collect both quantitative and qualitative data to gather detailed, descriptive information about implementation models and processes and local adaptation. To reduce the burden, ICF will use existing data when possible, including local evaluation findings, grantee reports, and extant data from SAMHSA’s Performance Accountability and Reporting System (SPARS). The findings will help SAMHSA assess whether the AWARE and TISS program activities are occurring as intended, how they may have been adapted to fit diverse contexts, and areas that need improvement to reach expected program outcomes.

The Implementation and Sustainability Study focuses on the evaluation questions listed in Exhibit 1.

Exhibit 1. Implementation Evaluation Questions

AWARE

A1. Is AWARE program implemented as intended across the sites?

A1.a. What are the challenges to partnership development, collaboration, and implementation, including training across grantees?

A1.b. What innovative strategies were developed and for which communities and populations?

A2. What are the variations in program implementation across sites and the underlying factors for these variations in implementation?

A2.a. What other resources (implementation science TTA, tool kits) are needed to successfully implement the grant goals?

A2.b. What other funding sources are grantees using to implement the grant?

A2.c. To what extent are grantees implementing the three-tiered approach that is culturally competent, trauma-informed, developmentally appropriate, evidence-based, or evidence-informed?

A3. How do participants of the grant program (school personnel, students, parents, state/local agency, and community partners) describe their experiences in the program?

A3.a. Do such experiences vary by grantee demographics, socioeconomic factors, region, urban vs. rural, existing culture of school personnel regarding evidence-based intervention and existence of minimum workforce?

A9. To what extent did AWARE grantees effectively tailor their programs?

A9.a. How do the grantees plan to sustain their programs after federal funding ends?

A9.b. What are some of the barriers and facilitators to sustaining their programs?

A9.c. How are some grantees and their partners developing and implementing a school-based suicide awareness and prevention training policy that is evidence-based and culturally and linguistically appropriate?

A9.d. What evidence-based trainings are provided to students?

TISS

T2. What are some of the site-specific challenges and benefits (schools vs. partner agency sites) where mental health services were provided?

T6. Do grantees make sustainability plans that describe how the project activities can be continued/sustained?

T8. What are some of the allowable activities grantees implemented and grantee evaluation findings including challenges and benefits of such?

T9. What are some of the innovative activities and evidence-based practices grantees implemented?

T9.a. Is the TISS program implemented as intended across the sites?

T9.b. What are the variations in program implementation and the underlying factors influencing the variations?

T9.c. What collaborative efforts have contributed to the improvement in identification, referral, early intervention, treatment, and support services for students?

T9.d. How do the grantees engage families and communities to increase the awareness of trauma impact on children and youth?

This study will use both primary and secondary data collection to provide a comprehensive assessment of current grantee implementation and sustainability planning and inform whether the type and frequency of activities, services, or products planned have been delivered as intended based on grantee work plans. Exhibit 2 provides an overview of each data collection activity, relevant evaluation questions, indicators/data elements, method, respondents, and timeline/data collection frequency.

Exhibit 2. Implementation and Sustainability Data Collection Activity, Methods, and Timeline

Data Collection Activity	Evaluation Questions*	Indicators/Data Elements (Pertains to both AWARE & TISS unless otherwise specified)		Method	Respondents	Timeline
IS	A1, A2, A9, T2, T6, T8, T9	Policy implementation (AWARE only) Implementation of pyramid model (AWARE only) Referral system/pathways (AWARE only) Additional detail on trauma-informed services (TISS only)	Innovative strategies Barriers/challenges Sustainability plan and challenges Suicide awareness or prevention policies (AWARE only)	ATODS—web-based survey	AWARE: SEA, LEA/TEA, SMHA project coordinators (n = 128) TISS: project coordinators (n = 15), other program or partner staff (n = 15)	Annually (Years 1-3)
IKII (Enhanced probing will be used for the TISS Case Studies)	A1, A2, A9, T2, T6, T8, T9	Coordination and decision-making Implementation challenges Contextual, systems, or other factors that affect implementation Sustainability		TTA site visits/qualitative interviews Virtual/qualitative interviews with grantees where TTA site visits are not conducted Case study site visits with selected grantees	Six KIIs per grantee site 1 state project coordinator 1 local coordinator 1 community mental health provider partner 2 licensed mental health professionals in schools 1 school administrator	Annually (Years 1-3)
YFFG-Y YFFG-F	A2, A3, A9	Awareness of school-based programs or resources to promote mental health literacy and meet mental health needs Overall program experience Satisfaction with the program Perspectives related to school climate and positive supports Youth or parent engagement		Focus groups conducted during TTA site visits Virtual focus groups with grantees where TTA site visits are not conducted	Focus groups to include up to 10 youth/parent/ family participants in each; focus groups for 12 AWARE grantees per year and 6 TISS grantees per year such that all grantees have focus groups at least once during the evaluation	Annually (Years 1-3)

*Evaluation questions include the main question and all associated sub-questions listed in Exhibit 1 unless otherwise noted.

Systems Change Study

Collaboration and changes to practices and policies across systems play a central role in both AWARE and TISS. To understand the impacts of both projects and the challenges and barriers faced, it will be essential to understand the characteristics of these partnerships and what changes to infrastructure and policies were successfully made. Training and workforce development are key methods to build the capacity of school and other systems to support student mental health and support trauma-informed practice, making the evaluation of training another important element of understanding systems change.

The purpose of the Systems Change Study is to collect in-depth information on system changes related to Project AWARE and TISS. The Systems Change Study will assess the extent to which the AWARE and TISS programs facilitate better partnerships and collaboration between SEAs, LEAs, and mental health systems to increase student access to evidence-based and trauma-informed mental health services. Additionally, this study will examine the extent to which the schools impacted by the programs have an improved school climate that promotes safe and stable learning environments. The Systems Change Study also will include an assessment of training and workforce development activities that aim to increase awareness, screening, referral, and connection to mental health services for school-aged youth and children. To assess training and workforce development, a pre-post design will be used to assess changes in knowledge, attitudes, and self-efficacy immediately following the trainings offered, with an additional follow-up survey allowing for the assessment of use of knowledge and skills from the training. The Systems Change Study addresses the evaluation questions listed in Exhibit 3.

Exhibit 3. Systems Change Evaluation Questions

AWARE

A4. What are the significant infrastructure plans/changes grantees made in the communities across the sites in improving, expanding, and/or sustaining mental health services for school-aged children when the grants end?

A4.a. How many organizations entered into formal written inter/intra-organizational agreements to improve mental health–related practices/activities that are consistent with the goals of the grant?

A4.b. How many and what type of policy changes were completed because of the grant?

A5. To what extent did the workforce development plan across the grantee sites help increase mental health awareness and literacy?

A5.a. How does training in mental health literacy impact the ability of school personnel to identify and refer youth to mental health services?

A5.b. How many individuals received training in prevention or mental health promotion?

A5.c. What percentage of those who have received training demonstrated improvement between pre- and post-test in knowledge/attitudes/beliefs related to prevention and/or mental health promotion?

TISS

T1. How many collaborative partnerships with local/community trauma-informed support and mental health service systems were developed across and engaged across the sites?

T3. Do grantees develop and implement school-based, trauma-informed support and mental health services plans?

T3.a. Do grantees identify barriers to accessing mental healthcare and include a plan to remove those barriers?

T3.b. Do grantees provide evidence-based/informed/best practice trauma-informed supports and mental health services to children, youth, and their families?

T4. To what extent do grantees develop and implement training plans for teachers, teacher assistants, school leaders, specialized instructional support personnel, and mental health professionals to foster safe and stable learning environments and prevent/mitigate the effects of trauma (including through social and emotional learning)?

T5. To what extent do grantees develop and implement family and community engagement plans to increase awareness of the impact of trauma on children and youth, including sharing best practices with law enforcement, regarding trauma-informed care; involving mental health professionals to provide interventions; and longer-term coordinated care within the community for children, youth, and their families who have experienced trauma?

T7. Do grantees establish local interagency agreements among LEAs, agencies responsible for early childhood education programs, Head Start agencies, juvenile justice authorities, mental health agencies, child welfare agencies, and other relevant agencies in the community?

CROSS-PROGRAM

AT1. How do the factors contributing to effective collaboration translate to effective systems changes?

AT2. What is the type and extent of systems change?

AT2.a. Is there a change in quality of mental health service delivery for school-aged children and youth and increase in the number of quality programs/interventions accessible and available?

AT3. Is there a change in the comprehensiveness of services provided in schools as a result of system change?

AT3.a. Do grantees ensure that there is a mix of services available to meet the needs of subpopulations and under-resourced communities?

Exhibit 4 provides an overview of each data collection activity, evaluation questions, indicators/data elements, respondents, the data collection method, and timeline/data collection frequency.

Exhibit 4. Systems Change Study Data Collection Activity, Methods, and Timeline

Data Collection Activity	Evaluation Questions*	Indicators/Data Elements	Respondents and Data Sources	Methods	Timeline
CPS	A4, T1, T7	Characteristics of the partnership between partnering agencies and grantees Changes to policies and infrastructure of partnering agencies that improve, expand, or sustain mental health services for school-aged children	AWARE SEA, LEA/TEA, SMHA project coordinators (n = 128) Local community mental health provider agency staff (n = 32) AWARE school administrators (n = 32) TISS project coordinators (n = 15) TISS local community mental health provider agency staff (n = 15) TISS school administrators (n = 15)	Web-based survey	Annually (Years 1-3)
TSF	A5, T4	Type of training Number of training participants by field and role Setting of training ZIP code of training Content of training Date and time of training	Grant program staff	Web-based survey	Ongoing as grantees deliver training and educational program (Years 1-3)
PFF	A5, T4	Training experience Perceived feasibility of using information from training	75 participants from each of the AWARE grantees, per year 25 participants from each of the TISS grantees, per year	Web-based survey	Ongoing as grantees deliver training and educational program (Years 1-3)
APPTS TPPTS	A5, T4	Knowledge of mental health resources and supports Mental health awareness and literacy Behavioral intentions to provide mental health supports and referrals Ability to identify and respond to symptoms of mental, behavioral, or emotional needs	125 participants from each of the AWARE grantees, per year 50 participants from each of the TISS grantees, per year	Web-based survey	Ongoing as grantees deliver training and educational program (Years 1-3)
WFS	A5, T4	Self-efficacy and confidence in applying new knowledge Identification and referral of at-risk populations since training Change in practices related to mental health service delivery in schools	50% of pre- and post-survey respondents; approximately 63 participants from each of the AWARE grantees per year and 25 participants from each of the TISS grantees per year	Web-based survey	Ongoing as grantees deliver training and educational program (3- and 12-months post-training in Years 1-3)
STCSS PCSS SSCSS	AT2	Environment supportive of mental health Perceptions of safety (physical and emotional) Relationships within the school Availability of resources for mental health and promoting social and emotional learning	Students (n = 6 per grantee) Parents (n = 6 per grantee) School personnel (n = 6 per grantee) LEA administrator staff (n = 4 per grantee)	Web-based survey	Once in Year 1, once in Year 3

*Evaluation questions include the main question and all associated sub-questions listed in Exhibit 3 unless otherwise noted.

Outcome Evaluation Studies

Identification and Referral Study

As concerns about student safety and mental health continue to rise, schools are increasingly called upon to lead prevention efforts and connect students to necessary care. Both AWARE and TISS work to enhance the array of mental health and substance use services available to students by embedding them within the school or establishing linkages with community-based providers and programs. To effectively accomplish this goal, it is essential that AWARE and TISS grantees develop a strong system for recognizing mental health needs in students and ensuring that they receive appropriate support services. By studying the ways in which school-aged youth are identified and connected to mental health, substance use, or trauma-related support services, the Identification and Referral Study furthers understanding of mental health support needs among youth and the factors that contribute to effective care engagement systems. The purpose of the Identification & Referral Study is to assess the extent to which grantees improve their capacity to identify students in need of support and link them to appropriate trauma-informed, evidence-based mental health services and other resources.

The Identification and Referral Study examines the effectiveness of systems established by AWARE and TISS grantees to identify school-aged youth in need of mental health or other support and connect them to appropriate trauma-informed and evidence-based services. This study will trace the pathway of youth as they are identified, referred to services, and receive supportive care within the 3 months following identification. In doing so, this study seeks to understand referral patterns and identify changes or potential gaps in the identification, referral, and services connection system. The goal of this study is to assess the extent to which youth identified through an AWARE or TISS program are referred to and receive services during the 3-month period following their identification, as well as the factors that best support effective referral systems. Specifically, the Identification and Referral Study will be used to understand 1) the process of identification and referral supported by the AWARE and TISS programs; 2) the primary referral and service needs of youth who are identified and referred; 3) the support received by youth who are identified and referred; and 4) the program or contextual characteristics that facilitate recognition of youth mental health, substance use, or trauma-related needs and enable successful linkages to support services.

Evaluation questions addressed by the Identification and Referral Study are listed in Exhibit 5.

Exhibit 5. Identification and Referral Study Evaluation Questions

AWARE

A6. To what extent was implementation of the three-tiered public health, pyramid model of intervention successful in school settings?

A6.a. How did it help improve the mental health outcomes for subpopulation (age groups, demographics, urban/rural/ family structure, etc.)?

A6.b. How many individuals were screened for mental health or related interventions?

A6.c. How many individuals were referred to mental health or related services?

A6.d. What is the percentage of individuals receiving mental health or related services after referral (access)?

A7. To what extent did existence of Memorandums of Understanding (MOUs) support clear referral pathways and ensure that school-aged children and youths who need more assistance than brief intervention are referred to and receive necessary school-based and/or community-based mental health services?

A11. How many individuals were trained by AWARE grantees to recognize and intervene in signs of suicidal thoughts and behavior, and how does this vary by age and grade level?

A11.a How do identification and referral volumes vary based on implementation of a suicide awareness and prevention training policy?

A11.b To what extent are grantees implementing suicide awareness and prevention training policy successful in increasing help-seeking reports among students?

TISS

T11. To what extent does TISS improve school capacity to identify, refer, and provide services to students in need of trauma support or behavioral health services?

T11.a. To what extent did TISS reflect the best practices for trauma-informed identification, referral, and support developed under section 7132 for the Interagency Task Force for Trauma-Informed Care?

T11.b. How many individuals were identified as in need of trauma support or behavioral health services?

T11.c. How many individuals were referred to evidence-based and culturally relevant trauma support services and mental healthcare?

T11.d. What is the percentage of individuals receiving mental health or related services after referral?

Cross-Program

AT4. Does the effectiveness of identification and referral systems vary based on grantee implementation of practices meant to address behavioral health disparities?

AT4.a. How does the proportion of identified students who receive follow-up services vary based on the characteristics of the youth identified (e.g., race, ethnicity, sexual identity, gender identity) or identifying staff (e.g., role type, training)?

AT4.b. To what extent is the proportion of students receiving services after referral reflective of overall school demographic profile?

This study will draw upon extant data as well as primary data collection activities to assess how grantees are developing and implementing identification and referral systems that support youth in need of mental health, substance use, and trauma-specific support services. This includes collection of identification and referral data from grantees, process study data abstraction, and extant grantee performance data. As shown in Exhibit 9 below, data collection for this study is primarily centered on the Student Identification and Referral Form (SIRF), which builds upon existing grantee Government Performance and Results Act (GPRA) reporting (i.e., IPP indicators submitted in SPARS), to obtain additional information on the nature, setting, source, and amount of identification and referral activity occurring in grantee communities. In addition, the Identification and Referral Study will incorporate other extant process evaluation and GPRA data (e.g., NOMS for TISS grantees, other IPP indicators) to better understand the activities and contextual factors that influence the effectiveness of referral systems. All grantees who are participating in the evaluation will participate in Identification and Referral Study data collection activities.

Exhibit 6 provides an overview of each data collection instrument, evaluation questions, indicators/data elements, respondents, the data collection method, and timeline/data collection frequency.

Exhibit 6. Identification and Referral Study Data Collection Activity, Methods, and Timeline

Data Collection Activity	Evaluation Questions*	Indicators/Data Elements	Respondents and Data Sources	Methods	Timeline
SIRF	A6, A7, T11, AT4	Demographics of students identified Referrals to services Referral sources and locations Receipt of mental health services	Grantee program staff	Web-based form with data entry into ATODS-based survey for up to 100 youth identified annually	Ongoing for each youth identified (Years 1-3)

*Evaluation questions include the main question and all associated sub-questions listed in Exhibit 5 unless otherwise noted.

Youth Resilience and Outcomes Study

AWARE required activities are designed to help grantees develop a sustainable infrastructure for school-based mental health programs and services by building collaborative partnerships, implementing evidence-based programs, and connecting students to mental health resources and services. Relatedly, the TISS program is designed to increase student access to evidence-based and culturally relevant trauma support services and mental healthcare services including local trauma-informed mental healthcare systems. The Youth Resiliency and Outcomes Study will examine the impact of each program on the desired outcomes as defined by the National Outcome Measures. The study will also assess the extent to which the required activities influenced school climate and school safety, student resiliency and coping skills, and awareness of school-based mental health supports. Ultimately, the study seeks to examine how the programs contributed to the social and emotional development of school-aged youth and to the availability and awareness of school- and community-based supports and services.

We understand data collection efforts impose a burden on the grantees and on participants themselves. As such, we will draw on extant data to examine the experiences of program participants. The study will rely on secondary data obtained annually from grantee reporting systems, LEAs, and/or state or national organizations, and through a series of open-ended questions included on the YFFG-Y in year three of the data collection period assessing student resiliency and coping skills. ICF will adhere to all local and state procedures and policies for secondary data requests. Evaluation questions are listed in Exhibit 7.

Exhibit 7. Youth Resiliency and Outcomes Study Evaluation Questions

AWARE

A8. To what extent did AWARE grants increase coping skills and resiliency among students across the sites, especially when faced with the challenges of the COVID-19 pandemic?

A8.a. What is the effect of AWARE grants on school safety across grantee sites?

A10. What are the overall impacts and/or significant outcomes of this program and how effective was this program across the sites based on the analysis of National Outcomes measures (IPP and other performance data) reported through SPARS and information collected through the review of grantee reports, site visits, and focus groups?

A10.a. To what extent do they describe improvements in resiliency and other school- and individual-level outcomes over the course of their schools’ funding?

TISS

T10. What are the overall impacts and/or significant outcomes of this program and how effective was this program across the sites based on the analysis of National Outcomes measures (IPP and other performance data) reported through SPARS and information collected through the review of grantee reports, site visits, and focus groups?

T10.a. To what extent do they describe improvements in resiliency and other school- and individual-level outcomes over the course of their schools’ funding?

T10b. To what extent do students identified by TISS grantees report improvements in functioning, social connectedness, and other mental health outcomes following engagement in mental health services? How does this vary by grantee (e.g., activities, partnerships) and participant (e.g., race, ethnicity, sexual identity, gender identity) characteristics?

As previously described, this study will rely on secondary data. See Exhibit 8 for primary data sources and indicators.

Exhibit 8. Youth Resiliency and Outcomes Study Data Collection Activity, Methods, and Timeline

Data Collection Activity	Evaluation Questions*	Indicators/Data Elements	Respondents and Data Sources	Methods	Timeline
YFFG-Y	A8, A10	Resiliency Coping School safety Awareness of school- and community-based resources and services Barriers and facilitators	Students/youth in AWARE/TISS programs	Virtual or in-person focus groups	Once in Year 3

*Evaluation questions include the main question and all associated sub-questions listed in Exhibit 7 unless otherwise noted.

Program Specific Sub-Studies

TISS Case Studies

The TISS program is a relatively new program designed to address the youth mental health crisis by providing a stronger continuum of supports to meet student mental health needs including trauma-informed mental healthcare. As noted, the program responds to increased exposure to traumatic experiences among youth during the pandemic such as losing a family member or caregiver (U.S. Surgeon General, 2021) combined with disconnection to in-person school supports and services due to social distancing imperatives just as reported problems with anxiety, depression, and suicidal thoughts reached unprecedented prevalence levels among students (NCSLE, 2021). Even before the pandemic, youth exposure to traumatic experiences in the United States was common, with more than two-thirds of children reporting at least one traumatic event by age 16 (SAMHSA, 2023). Potentially traumatic events include psychological, physical, or sexual abuse; community or school violence; witnessing or experiencing domestic violence; national disasters or terrorism; commercial sexual exploitation; sudden or violent loss of a loved one; refugee or war experiences; military family-related stressors (e.g., deployment, parental loss or injury); physical or sexual assault; neglect; and serious accidents or life-threatening illness (SAMHSA, 2023).

Adverse childhood experiences (ACEs)—commonly defined as traumatic events that occur in childhood—are a complex population health problem with substantial detrimental outcomes. The groundwork for the study of ACEs stems from the seminal Centers for Disease Control and Prevention (CDC)/Kaiser Permanente ACEs Study, one of the largest investigations of childhood abuse and neglect and how it impacts health and well-being later in life. This research initially identified 10 potentially preventable ACEs, including physical and emotional abuse and neglect, as well as growing up in a household with substance use, mental health problems, domestic abuse, or instability due to parental separation or incarceration of a household member (Felitti et al., 1998). The study found that exposure to ACEs was related to a range of negative outcomes in adulthood, including increased risk of alcohol and drug use, mental health problems, poor physical health, and risky behaviors (Felitti et al., 1998; Freeman, 2014; Felitti, 2002). Additional research corroborated these findings, suggesting that ACEs often occur together; can result in toxic stress; and are associated with a wide range of adverse behavioral, health, and social outcomes, including premature mortality, alcoholism, drug abuse, depression, suicide, heart disease, obesity, cancer, and chronic lung disease, among others (Felitti et al., 1998; Brown et al., 2009; Dube et al., 2003; Chapman et al., 2004; Williamson et al., 2002). Moreover, ongoing research finds that community-level factors can compound the risk for negative impacts that occur outside of the home. CDC’s National Center for Injury Prevention and Control summarizes in its FY2021–FY2024 Prevention Strategy that conditions such as living in under-resourced or racially segregated neighborhoods, frequently moving, being subjected to homelessness, or experiencing food insecurity can be traumatic and exacerbate the effects of other ACEs (CDC, 2021). In addition, historical and ongoing traumas due to systemic racism and discrimination or the impacts of multigenerational poverty resulting from limited educational and economic opportunities intersect and exacerbate the experience of other ACEs, leading to disproportionate effects in certain populations (CDC, 2021).

The TISS case studies will provide SAMHSA with critical information needed to deepen understanding of implementation processes and effects of the TISS program, including the extent to which this program improves access to trauma-informed mental health services for school-aged youth in need of support and intervention. These studies also provide a unique opportunity to understand how grantees have adapted the TISS program to local conditions, organizational dynamics, and the social and cultural contexts of the community and school environment to ensure access to services for high-need subpopulations and under-resourced communities.

Building on the data collected for the previously described activities, ICF will design and conduct two in-depth case studies to enhance understanding of the implementation and outcomes of the TISS program with a deeper focus on two grantees, including one tribal grantee and one non-tribal grantee. The case study approach will allow us to build an in-depth understanding of the critical factors that affect implementation of the TISS program, including how contextual factors influence program implementation and outcomes. The data from the case studies will seek to tell the grantee’s story, describing key policy junctures, contextual factors, and factors that enabled or impeded the pathways between policies, implementation, and outcomes.

ICF will work with the SAMHSA COR to identify focus areas and associated evaluation questions (from Exhibit 2). Areas of focus, evaluation questions, and grantee selection for the first two case studies will be informed by ICF’s review of grantee materials including grantee documents (applications, progress reports). This review will enable ICF to prepare a bulleted list of key takeaways that may inform the selection of sites for the case studies. As grantee progress reports become available, ICF will develop a matrix including tribal and non-tribal sites and indicate sites that have documented success and sites that have not had success. Such a matrix will guide the selection of grantees to participate in a case study focusing on factors contributing to success. ICF will also consider the School Health Assessment and Performance Evaluation (SHAPE) system (National Center for School Mental Health, 2023) as a foundational model for the evaluation that may support case study site selection. Potential focus areas for the two TISS case studies may include 1) identifying and documenting innovative practices for linking students to local trauma-informed support and mental health systems; 2) understanding how the availability of culturally responsive, trauma-informed support and mental health systems influences outcomes related to TISS program services; and 3) the impact of COVID may be a potential area of focus as part of the case studies.

The case study will use the data sources described in the previous sections in several ways; (1) we will review information from grantees and other extant data sources (as available) to inform the selection of two grantees, with consideration of the selected focus areas for the case study; (2) in the case of the two grantees selected, we will incorporate data from a variety of sources generated through the evaluation to inform the development of comprehensive grantee profiles; and (3) we will analyze data and observational activity and triangulate data sources, including document review and quantitative data from the outcome studies to develop explanatory narratives illustrating each grantee’s TISS program implementation process.

Suicide Awareness and Prevention Sub-Study

Childhood mental health concerns and suicide rates have risen steadily from 2010 to 2020; and as of 2020, suicide is the second leading cause of death for youths under 18 years of age (American Academy of Pediatrics, 2021; CDC WISQARS, 2022). According to the 2020 NSDUH, youth aged 12-17, 12% or 3.8 million had serious thoughts of suicide, 5.3% made a suicide plan, and 2.5% attempted suicide in the past year (SAMHSA NSDUH, 2021). Early identification of suicidal ideation among youth remains a key area of focus for SAMHSA as the agency works to improve treatment and recovery (SAMHSA FY2019–FY2023 Strategic Plan, Priority 2). AWARE grantees opting to implement or plan a student suicide awareness and prevention training policy in secondary schools throughout the course of their projects were prioritized for funding through the award of additional points on FY2022 applications.

The AWARE Suicide and Prevention Sub-Study will include additional subgroup analyses for AWARE grantees who implement a suicide awareness or prevention training policy. ICF will identify grantees meeting these criteria using findings from a comprehensive review of FY2022 AWARE-TISS grantee applications that was conducted in March 2023. ICF will also use data collected from the IS which is part of the Implementation and Sustainability Study.

This examination aims to better understand the structure and efficacy of suicide awareness or prevention training policies. ICF will review grantee documents and catalogue the presence and characteristics of awareness/training plans and use these to examine differences in key outcomes (i.e., screening and referral numbers, help-seeking reports among students, numbers trained). This sub-study is guided by evaluation questions as presented in Exhibit 9 to understand policy structure, outputs and proximal outcomes related to suicide prevention awareness, knowledge, identification and referral skills, and student help-seeking.

Exhibit 9. Suicide and Prevention Awareness Sub-Study Evaluation Questions

AWARE

A6. To what extent was implementation of the three-tiered public health, pyramid model of intervention successful in school settings?

A6.a. How did it help improve the mental health outcomes for subpopulation (age groups, demographics, urban/rural/ family structure, etc.)?

A6.b. How many individuals were screened for mental health or related interventions?

A6.c. How many individuals were referred to mental health or related services?

A6.d. What is the percentage of individuals receiving mental health or related services after referral (access)?

A6.e. To what extent did participation in a suicide awareness and prevention training affect students’ awareness, knowledge, skills, and self-report behavior?

A11. How many individuals were trained by AWARE grantees to recognize and intervene in signs of suicidal thoughts and behavior, and how does this vary by age and grade level?

A11.a How do identification and referral volumes vary based on implementation of a suicide awareness and prevention training policy?

A11.b To what extent are grantees implementing suicide awareness and prevention training policy successful in increasing help-seeking reports among students?

As shown in Exhibit 10, this study will draw upon the TSF used as part of the Systems Change Study.

Exhibit 10. Suicide Awareness and Prevention Sub-Study Data Collection Activity, Methods, and Timeline

Data Collection Activity	Evaluation Questions*	Indicators/Data Elements	Respondents and Data Sources	Methods	Timeline
TSF	A6	Type of suicide awareness and prevention trainings Number of suicide awareness and prevention trainings Number of student trainings Type of training delivery method	Grantee program staff (AWARE grantees)	Web-based form with data entry into ATODS-based survey	Ongoing for each training conducted for up to 10 trainings annually (Years 1-3)

*Evaluation questions include the main question and all associated sub-questions listed in Exhibit 9 unless otherwise noted.

Cross-Cutting Impact Analyses

Cross-Program Impact Analysis

Children and youth in the U.S. experience high rates of mental health conditions and low rates of treatment Bitsko et al. (2022). The AWARE and TISS programs were designed to address the youth growing mental health needs by developing comprehensive school mental health systems and innovative initiatives, activities, and programs to link school systems with local trauma-informed support and mental health systems. The goal of Cross-Program Impact Analysis is to measure the impact of the AWARE/TISS programs’ strengthening of the continuum of supports through evidence-based prevention practices and trauma-informed mental healthcare on mental health outcomes of children and youth. Mental health outcomes of interest include depression, suicidal thoughts and behaviors, mental health service utilization, and deaths by suicide.

The Cross-Program Impact Analysis will rely on comparing the change in mental health outcomes of children ages 12-17 in counties exposed to the AWARE/TISS interventions with the change observed in counties that were not exposed to those interventions during the same period. The analyses will provide a quasi-experimental assessment of the AWARE/TISS Programs’ impact by examining whether outcomes in communities exposed to the AWARE/TISS Programs are different from what would have been observed in the absence of the programs.

While we lack the ability to randomize youths to the AWARE/TISS Programs, we have methodological and analytic tools available to develop a rigorous, valid, and defensible counterfactual condition. Employing and extending methods previously reported in research literature (Garraza et al., 2015; Walrath et al., 2015; Godoy Garraza et al., 2019), we will develop a quasi-experimental approach that minimizes baseline differences in pre-intervention mental health outcomes and relevant population characteristics between the AWARE/TISS counties and control counties.

Cross-Program Impact Analysis will examine the impact of the AWARE/TISS programs through the establishment and enhancement of school-based mental health supports on mental health outcomes, while taking into consideration various contextual factors. The analysis contributes to answering evaluation question 10 (see Exhibit 11) by introducing the control group via quasi-experimental assessment to provide more rigorous examination of programs’ impact.

Exhibit 11. Cross-Cutting Impact Analyses Evaluation Questions

AWARE

A10. What are the overall impacts and/or significant outcomes of this program and how effective was this program across the sites based on the analysis of National Outcomes measures (IPP and other performance data) reported through SPARS and information collected through the review of grantee reports, site visits, and focus groups?

A10.a. To what extent do they describe improvements in resiliency and other school- and individual-level outcomes over the course of their schools’ funding?

TISS

T10. What are the overall impacts and/or significant outcomes of this program and how effective was this program across the sites based on the analysis of National Outcomes measures (IPP and other performance data) reported through SPARS and information collected through the review of grantee reports, site visits, and focus groups?

T10.a. To what extent do they describe improvements in resiliency and other school- and individual-level outcomes over the course of their schools’ funding?

T10b. To what extent do students identified by TISS grantees report improvements in functioning, social connectedness, and other mental health outcomes following engagement in mental health services? How does this vary by grantee (e.g., activities, partnerships) and participant (e.g., race, ethnicity, sexual identity, gender identity) characteristics?

The analysis will rely on secondary data sources. We will explore several county-level datasets, including SAMHSA’s NSDUH, the Centers for Disease Control and Prevention’s (CDC’s) Detailed Mortality Files, CDC’s 500 Cities and PLACES, the Healthcare Cost and Utilization Project (HCUP), and Medicaid. NSDUH provides nationally representative data on mental illness, depression, suicidal thoughts and behaviors, psychological distress, mental health service utilization, and treatment for depression among population aged 12 or older. The CDC Detailed Mortality Files provide data on deaths by suicide derived from the death certificates of residents recorded in the United States. CDC’s 500 Cities and PLACES provides mental health and substance use data for small areas across the country. The HCUP is the largest all-payer ED database publicly available in the United States. It provides data on suicide attempts and deaths by suicide. Medicaid data include service utilization information, mental health diagnoses, suicide attempts, and deaths by suicide.

Behavioral Health Equity Cross-Study Analysis

All individuals, regardless of race, ethnicity, gender, sexual orientation, socioeconomic status, or disability deserve access to high-quality health services and supports. Yet, social determinants of health continue to confer advantages to some and disadvantages to others. Recent findings indicate that, annually, 20% of all children have an identified mental health condition, but youths living in poverty and youths who are members of racial and ethnic minority populations fare worse than their peers with respect to identifiable risk factors, prevalence of certain mental health conditions, and access to care. Coupled with high rates of mental health conditions, suicidal behaviors among high school students have increased significantly. Between 2009 and 2019, 19% of students reported seriously considering attempting suicide—a 36% increase—and 16% reported making a suicide plan in the prior year—a 44% increase (U.S. Surgeon General, 2021). The COVID-19 pandemic has intensified this crisis. Beginning in April 2020, the proportion of children’s mental health–related ED visits among all pediatric ED visits increased and remained elevated through October 2020. Compared with 2019, the proportion of mental health–related ED visits for children aged 5–11 and 12–17 years increased approximately 24% and 31%, respectively (Leeb et al., 2020). The pandemic’s negative impacts most heavily affected individuals already vulnerable to suicide, such as youths with disabilities; racial and ethnic minority youths; lesbian, gay, bisexual, transgender, and questioning (LGBTQ+) youths; low-income youths; youths in rural areas; youths in immigrant households; youths involved with the child welfare or juvenile justice systems; and homeless youths (U.S. Surgeon General, 2021).

The Behavioral Health Equity Cross-Study analysis will address evaluation questions that cut across studies. We will integrate existing data sources and findings from the process and outcome evaluations to understand how the AWARE/TISS grantees invest in and accomplish improvements in behavioral health disparities, especially among racial/ethnic minority, LGBTQ+, and low-income children.

The Behavioral Health Equity Cross-Study analysis will create an understanding of the cultural acuity of mental health supporting activities and assess disparities in mental health outcomes among specific subgroups of children and youth (e.g., racial/ethnic minorities, LGBTQ+, or low-income children). The study will provide additional information on the cultural adaptations employed by grantees including if and how those adaptations vary by specific subgroups; how characteristics of vulnerable populations are taken into consideration when implementing various aspects of AWARE/TISS interventions; which social determinants of health grantees are addressing and how; and how cultural adaptations and attention to subgroup characteristics and social determinants of health affects mental health outcomes.

The Behavioral Health Equity Cross-Study analysis aims to enhance the understanding of AWARE/TISS programs effectiveness by investigating whether specific subpopulations benefitted from the programs and will contribute to answering the following evaluation questions with the cross-study focus (Exhibit 12).

Exhibit 12. Behavioral Health Equity Cross-Study Analysis Evaluation Questions

TISS

T10b. To what extent do students identified by TISS grantees report improvements in functioning, social connectedness, and other mental health outcomes following engagement in mental health services? How does this vary by grantee (e.g., activities, partnerships) and participant (e.g., race, ethnicity, sexual identity, gender identity) characteristics?

CROSS-PROGRAM

AT4. Does the effectiveness of identification and referral systems vary based on grantee implementation of practices meant to address behavioral health disparities?

AT4.a. How does the proportion of identified students who receive follow-up services vary based on the characteristics of the youth identified (e.g., race, ethnicity, sexual identity, gender identity) or identifying staff (e.g., role type, training)?

AT5. Do specific grantee approaches have differential impacts on reducing disparities in mental health outcomes over time?

As an overarching analysis for the process and outcome evaluations, the Behavioral Health Equity Cross-Study analysis will use the data sources described in the previous sections. Primary data sources will include the following instruments: IS, KII and focus group guides (YFFG-Y and YFFG-F), and SIRF. Secondary data sources will include ED’s CRDC, SPARS, NSDUH and Medicaid data. Archival records (e.g., grantee annual reports) will be used to extract information about the types of activities implemented, including any cultural adaptations made and health equity practices employed.

9