NOW IS THE TIME (NITT) – HEALTHY TRANSITIONS (HT) EVALUATION
SUPPORTING STATEMENT
The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for new data collection activities for SAMHSA’s Evaluation of the Healthy Transitions program, part of the Now Is The Time (NITT) initiative. SAMHSA’s NITT initiative aims to better protect U.S. children from violence by making schools safer and increasing access to mental health services. Programs included are NITT Project AWARE, Healthy Transitions and two Minority Fellowship Programs (Youth and Addiction Counselors). While SAMHSA’s NITT initiative includes separate programs, these programs are united by their focus on capacity building, system change and workforce development. The NITT national evaluation is scheduled through March 1, 2020 and will conduct a national cross program evaluation of all three grant programs.
The data collection activities described in this package pertain exclusively to the NITT-HT program component of the national evaluation. NITT-HT data collection strategies include telephone interviews, site visits with guided interviews and focus groups, and web-based assessments. SAMHSA is requesting approval for data collection for the following instruments:
Community Support for Transitions Inventory (CSTI), to collect information about local community support for systems change efforts designed to improve outcomes for youth/young adults with serious mental health conditions. (Attachment 1)
State Support for Transition Inventory (SSTI), to collect information about state support for systems change efforts designed to improve outcomes for youth/young adults with serious mental health conditions. (Attachment 2)
Collaborative Member Survey, to collect information on collaborative functioning and specific activities completed by the NITT-HT leadership team and stakeholders including progress towards accomplishing NITT-HT program goals. (Attachment 3)
Collaborative Self-Assessment, to collect information on the collaborative functioning and specific accomplishments of the NITT-HT grantee Advisory Team. (Attachment 4)
Project Director Web Survey, to measure planning, coordination, and leadership efforts, including efforts around fiscal planning. (Attachment 5)
Project Director Telephone Interview, to provide more in depth information on leadership efforts, planning, coordination and implementation processes to complement information gathered in the project director web survey. (Attachment 6)
Core Staff Web Survey, to assess the degree to which core staff (transition support providers) affiliated with NITT-HT communities are implementing person-centered planning and case management services in their work with youth/young adults. (Attachment 7)
Multi-Media Project Young Adult Probes, a data collection method where young adults share experiences working with or being served by NITT-HT grantee communities through videos, blogs, poems or other media outlets in a youth/young adult-driven fashion. (Attachment 8)
Supplemental Youth and Young Adult Interview (SYAI), to collect information on outcome measures related to the NITT-HT youth/young adult participants’ behavioral health and functioning. (Attachments 9 and 10)
Services and Supports Inventory, to characterize the types of services (including evidence-based practices) being implemented within the NITT-HT grant community. (Attachment 11)
Grantee Visit In-Person Interviews and Focus Group Guides, To elicit information from key project staff (e.g., youth coordinators, providers), stakeholders, youth and young adults, and family member allies about the implementation of the NITT-HT program and youth/young adult program needs and resources. (Attachment 12)
SAMHSA’s NITT-HT program is authorized under Section 520A of the Public Health Service Act, as amended, and addresses the Healthy People 2020 Mental Health and Mental Disorders Topic Area (HP 2020-MHMD) and Substance Abuse Topic Area (HP 2020-SA). NITT-HT also supports two of SAMHSA’s Strategic Initiatives: (1) Prevention of Substance Abuse and Mental illness, and (2) Recovery Support. Finally, this effort supports the fourth component of the President’s NITT Plan, to increase access to mental health services.
Significant numbers of transition-age youth ages 18 to 26 experience serious mental health conditions, approximately 2.4 million (Government Accountability Office, 2008). The most common age of onset for schizophrenia and substance use disorders is during young adulthood, and onset for major depressive disorder is during young adulthood for about 1 in 3 cases. Challenges associated with these mental health problems extend to other aspects of life. Transition-age youth experience disproportionately higher levels of unemployment (Sum et al., 2014). Even when employed, nearly half of young adults with jobs hold positions that do not offer health insurance (Callahan & Cooper, 2004), which hampers access to needed services. Youth/young adults also experience the highest rates of victimization (Menard, 2009). Young people with mental health conditions are often stigmatized, and are at high risk of being socially excluded or bullied (United Nations, 2013).
Despite high need, young adults are less likely to use mental health services than those older or younger (Kessler et al., 2005; Pottick et al., 2014), and are more likely to drop out of treatment than mature adults (Edlund et al., 2002). Findings from the National Survey of Drug Use and Health (NSDUH) demonstrate that more than 6 in 10 young adults with the highest mental health needs did not receive outpatient mental health services in the last year. Similarly, the Greater Smokey Mountain epidemiological study found that among young people with mental health disorders, service utilization dropped sharply from adolescence to young adulthood (Institute of Medicine & National Research Council, 2013), with only 16% of these 21 year olds accessing services (Miller et al., 2013).
Transition-age youth may be harder to engage in treatment. They may view adult services as unattractive, and be especially concerned about the stigma associated with seeking treatment. Young people may also be left out of treatment planning, and have different goals than parents or staff. Fortunately, promising models have been developed to detect and treat youth/young adults with early psychosis (Penn et al., 2005; Menehan, 2013). These and other models underscore the need for developmentally relevant programs to engage youth/young adults in services.
The purpose of the Healthy Transitions (HT) program is to improve access to treatment and support services for youth and young adults 16–25 years that (1) either have, or are at risk of developing a serious mental health condition, or 2) are at high risk of developing a mental illness or substance use disorder, and are at high risk of suicide. The NITT-HT program reflects a public health model that stresses the importance of increasing youth/young adult access to mental health services as a cornerstone component of increasing community safety and reducing victimization and violence. NITT–HT programs have 3 populations of focus: the community at large, 16–25 year olds at risk of developing a serious mental health condition (who might otherwise be under identified), and 16–25 year olds who have already been identified as experiencing a serious mental health condition. Program purposes are slightly different for each target population: for the community, to raise awareness about mental health and substance use early identification; for youth/young adults at risk, to promote outreach and engagement; and for youth/young adults with a serious mental health condition, to improve access to developmentally appropriate, relevant and evidence-based services and supports.
In the fall of 2014, SAMHSA awarded 17 NITT-HT grants to 14 states (Connecticut, Delaware, Florida, Kentucky, Maine, Maryland, Massachusetts, New Mexico, Oklahoma, Pennsylvania, Rhode Island, Tennessee, Utah, and Wisconsin), the Research Foundation for Mental Hygiene in New York, the District of Columbia, and the Fairbanks Native Association in Alaska. Each grantee proposed various activities to improve the system of care for the target population of interest. These activities included the creation of collaborations, networks, councils, and stakeholder involvement, particularly with family and youth/young adults. All NITT-HT grantees aim to provide person-centered care planning and case management services to all youth/young adult participants. Then, dependent upon specific needs, youth/young adult participants are also provided tailored individual interventions, services and supports to address specific behavioral health and functional outcome goals.
The NITT-HT grantee communities include a layered structure stemming from the state or local jurisdiction and flowing down to community-based providers. Every NITT-HT grant includes at least two learning laboratories; many communities have three learning laboratories. Across the NITT-HT grantees, there are an estimated 43 learning laboratories. These learning laboratories are the local communities of practice responsible for implementing the NITT-HT approach. Learning laboratories often include multiple providers or provider organizations. Some learning laboratories are a single town, others include multiple counties.
The NITT-HT national evaluation will capture key elements of service provision that increase youth/young adult’s access and engagement in services. The national evaluation will also examine factors associated with positive outcomes in youth/young adults at risk for or experiencing serious mental health conditions. Ultimately, answers to the NITT–HT specific evaluation questions (see Table 1) will feed into the integrated evaluation model to address the success of the full SAMHSA NITT initiative in achieving its prime mission of making schools safer and increasing access to mental health care.
The NITT-HT process evaluation will include both qualitative and quantitative components supported by multiple types of data collection—web-based data collection from grantee and learning laboratory advisory team members and staff, telephone interviews, a one-time grantee visit (that will include focus groups), and document review. The process evaluation is also designed to give special attention to the role that both youth and family voice play in NITT-HT grantee program development, implementation, and local evaluation.
The NITT-HT outcome evaluation includes two levels of primary outcomes: systems and client. The outcome evaluation is supported by web-based surveys of state/grantee and local community leaders and supplemental youth/young adult interviews.
Special attention in the outcome evaluation will be given to creating data positioned to make national comparisons to youth/young adults aged 18-25 who are not involved with NITT-HT grantee programs. The outcome evaluation plan compares NITT-HT youth/young adults with similarly aged youth across the United States on select indicators of emotional/behavioral health, functioning (employment, education, housing, and health), and mental health service use. The evaluation will include questions administered to youth/young adults in the supplemental youth/young adult interviews that match items included in National Survey of Drug Use and Health [NSDUH]; OMB No. 0930-0110) for young adults ages 18–25 years. The NSDUH is an annual, nationally-representative public-use data set with measures of mental health status, functioning, and mental health and substance abuse service use. The service use items in the NSDUH include an assessment of both outpatient and inpatient mental health service use as well as use of psychotropic medications. Matched national comparisons will be specific to young adults 18 and older, excluding NITT-HT participants who are 16 and 17 years. While the NSDUH survey is conducted with adolescents, the NSDUH survey instrument differs substantially from those NSDUH questions asked of participants 18 and older. Consequently, we do not propose to make matched NSDUH to NITT-HT comparisons for 16 and 17 year old participants.
The national evaluation will also include data submitted by NITT-HT grantee communities as required by SAMHSA through the Common Data Platform ([CDP]; OMB No. 0930-0346). SAMHSA requires that the 17 NITT-HT programs submit grantee-level CDP data semiannually. The required grantee-level CDP indicators include: (1) the number of organizations that entered into formal written inter/intra-organizational agreements (e.g., MOUs/MOAs) to improve mental health-related practices/activities that are consistent with the goals of the grant, and (2) the number of policy (e.g., infrastructure and organizational) changes completed as a result of the grant. SAMHSA also requires NITT-HT grantee communities provide data on several systems-level outcomes including the number of individuals screened for mental health treatment, the number contacted through outreach efforts, the number and percentage referred for services, the number and percentage who received services, and the number and percentage receiving mental health services after a referral. Finally, SAMHSA requires that NITT-HT programs collect client-level data via the CDP every 6 months for all clients currently being served through the NITT-HT program. The client-level services measures include indicators of planned services, drug and alcohol use, family and housing, mental and physical health, recovery/self-help/peer support, and services received. This systems-, grantee-, and client-level information will be made available to the national evaluation team and will not need to be collected through the surveys and interviews at the center of this OMB application.
The core NITT-HT evaluation questions to assess program processes and outcomes fall into four broad areas: 1) collaboration and coordination, 2) mental health service system capacity and infrastructure, 3) mental health service access for individuals in need, and 4) individual resilience and functioning and school/community safety. Table 1 describes the NITT-HT evaluation questions organized by these priority areas.
These evaluation questions will be addressed in the data collection procedures section as described in Section A.2.
Table 1. Evaluation Questions for the Now Is the Time (NITT) Healthy Transitions (HT) Program
Collaboration and Coordination |
|
EQ1. |
Did NITT-HT lead to enhanced collaboration and coordination across youth-serving systems for youth/young adults ages 16 to 25? |
EQ2. |
What were the barriers/facilitators to state-/local-level collaboration, partnership development, and shared decision-making? How were they addressed? |
EQ3. |
How was youth/young adult (Y&YA) voice integrated into aspects of NITT-HT program planning, implementation, and local evaluation efforts? |
EQ4. |
How was family and adult ally voice integrated into aspects of NITT-HT program planning, implementation and local evaluation efforts? |
Mental Health Service System Capacity and Infrastructure |
|
EQ5. |
To what extent were services and supports for youth/young adults with serious mental health conditions developed, improved and expanded as a result of NITT-HT? |
EQ6. |
What were the effects on service capacity and access to care in communities implementing NITT-HT? |
EQ7. |
What NITT-HT strategies were effective in developing, improving, and expanding services and supports for youth/young adults with serious mental health conditions? |
EQ8. |
What were effective NITT-HT strategies to increase access to care? |
EQ9. |
What are effective awareness outreach and engagement strategies for youth/young adults ages 16 to 25 for behavioral health disorders? |
Mental Health Service Access for Individuals in Need |
|
EQ10. |
To what extent did NITT-HT identify and serve young people who would have otherwise fallen through the cracks? |
EQ11. |
To what extent did NITT-HT increase mental health service access for Y&YAs in need? |
Individual Resilience and Functioning: School and Community Safety |
|
EQ12. |
What are the outcomes associated with NITT-HT in the areas of education, employment, housing, primary care, and mental health? |
EQ13. |
To what extent did NITT-HT lead to decreased involvement with juvenile and criminal justice systems? |
EQ14. |
What is the effect of the NITT-HT intervention compared to a national sample of similar young adults not participating in the NITT-HT intervention? |
The NITT-HT national evaluation will use a pre/post, longitudinal follow-up design with matched comparison groups where relevant and possible (discussed further in Section A.16, Analysis). It will combine qualitative and quantitative data and methodologies to fully address SAMHSA’s objectives for the national evaluation and will include process as well as outcome components. Feedback on the NITT-HT evaluation design and data collection instruments (see Section A.8 and Section B.5) has been solicited from internal SAMHSA staff, grantee-level project directors and local evaluators, and an expert advisory panel. After careful review, revisions were made to streamline the instruments, reduce burden, simplify wording, increase variation in response options, improve coherence of scales for summing, and create consistency in assessing infrastructure at the learning laboratory and grantee/state levels.
The following section describes, each NITT-HT national evaluation instrument for which SAMHSA is requesting approval.
Local community support will be assessed using the web-based CSTI, which was designed specifically to assess systems change for communities implementing comprehensive, community-based approaches to improving outcomes for emerging adults with serious mental health conditions (Walker et al., 2015). The CSTI is organized around seven themes (community partnership, collaborative action, transition planning quality assurance and support, workforce, fiscal policies and sustainability, access to needed support and services, and accountability). Each theme includes between four and ten specific item activities that are rated by the respondent from “fully developed” to “least developed.” The CSTI is completed by stakeholders in the local community who are knowledgeable about the process of developing and delivering services for young adults. This will include members of the local advisory or steering committee, staff of the NITT-HT program, staff of agencies providing portions of the services, and young adult and family members’ advocates. Young people, family members, and other allied adults who serve on advisory boards or who are significantly involved with planning and evaluating services for emerging adults will also be asked to respond to the CSTI. The CSTI will be administered at two points during the five-year NITT-HT grant cycle. Time 1 (T1) occurs when the NITT-HT grants are in their second year of funding and Time 2 (T2) occurs toward the end of the fourth year of grant funding. Comparisons between T1 and T2 will be used to examine service and systems change in communities. The same instrument is administered at both time points.
State support for systems change will be assessed using the web-based SSTI. The SSTI is organized around six themes (partnership, collaborative action, workforce, fiscal policies & sustainability, access to needed supports & services, and accountability) that each include three to ten specific item activities that are rated from “fully developed” to “least developed.” The SSTI is designed to be administered to individuals at the state level who are involved with planning and funding services for emerging adults with mental health needs. This will include administrators or staff from state agencies for mental health, child welfare, education, and vocational rehabilitation, as well as other state agencies responsible for some aspects of services to youth/young adults. The SSTI is also completed by youth/young adult and adult allies who are active in promoting, planning, or overseeing services at the state level, as well as other members of state-level advisory groups or governing bodies. Comparable to the CSTI, the SSTI will be administered at two points during the five-year grant cycle. The same instrument is administered at the two time points. Research using the CSTI and SSTI provided evidence of significant change across various system-change subscales in communities and states funded between 2009 and 2014 under the Healthy Transitions Initiative (Walker et al., 2015).
The Collaborative Member Survey is designed to assess specific team processes that contribute to collaboration outcomes at the systems level. The Collaborative Member Survey will be administered to a subset of CSTI respondents who participate in a NITT-HT grantee’s Advisory Team. The Collaborative Member Survey instrument emphasizes aspects of Advisory Teams’ climate (participatory decision-making, structure, management of conflict, reflexivity). This measure is based on standardized measures of collaborative functioning used in multi-sectoral community initiatives (Nowell, 2009; Nowell & Foster-Fishman, 2011). Unlike the SSTI/CSTI instruments, which assess contributions of the initiative to development throughout the system, this instrument focuses specifically on how the Advisory Team functions. To minimize burden, the Collaborative Members Survey will be administered to representatives of the NITT-HT Advisory Team in the third and fourth years of NITT-HT grant program funding, years they are not asked to complete the CSTI.
The Collaborative Self-Assessment was adapted from a previously validated instrument for self-assessment by child and family collaboratives (Emshoff et al., 2007). The survey assesses collaborative functioning and accomplishments, and specific tasks completed by NITT-HT grantee stakeholders and the leadership team including progress in each of the primary “functions” for the NITT-HT grantees (i.e., specific, discrete achievements or steps toward strategic and fiscal planning, expansion of services, early identification outreach, and reduction of barriers to access). This instrument asks respondents to report upon discrete outputs related to collaboration, such as specific expansions of representation on the collaborative, progress in collaboration in training, co-locating, or providing services, and supporting services through redirecting or obtaining additional funds. To minimize burden, the Collaborative Self-Assessment will be completed by one representative of each NITT-HT Advisory Team within the NITT-HT grantee community in the third and fifth years of NITT-HT grant program funding, years they are not asked to complete the CSTI. In order to complete the measure, this representative may have to gather information from other members of the advisory team. It is expected that each NITT-HT grantee will have 2-3 Advisory Teams.
This brief web-based survey will be administered to each NITT-HT Project Director prior to each Project Director Telephone Interview. The Project Director Web Survey will be conducted at three time points (Times 1, 2, and 3). The Time 1 web survey will be completed at the end of the planning phase (NITT-HT grantee’s Year 2 funding). The Time 2 web survey will be completed at mid-project shortly before the scheduled grantee visits associated with the national evaluation (NITT-HT grantee Year 3 or 4 funding). The Time 3 web survey will be completed at the termination/sustainability phase (Year 5 of NITT-HT program grant funding). The Time 1, 2, and 3 web surveys are slightly different to reflect varying annual changes in program implementation emphasis. All three web surveys include prompts designed to assist the Project Director in gathering and recalling information to be discussed during the subsequent Project Director Telephone Interview. The web surveys also include questions on planning, coordination, and leadership processes and fiscal planning. Only the Time three web-based project director survey contains a sustainability module, assessing progress toward sustainability goals, and evidence-based predictors of the sustainability of community initiatives. Sustainability will be measured using the Instrument for Sustainability of Changed Work Practices (Slaghuis, Strating, Bal, & Nieboer, 2011).
The Project Director Telephone Interview will be conducted at the same three time points as the Project Director Web Survey (NITT-HT grantee Years 2, Year 3 or 4, and Year 5). Project Directors will first complete the web survey and upon completion will be asked to schedule a telephone interview. The telephone interview will focus on gathering more in depth information to complement information gathered via the previous web survey. The time 2 Project Director Telephone Interview will occur shortly before the planned one time grantee visit associated with the national evaluation. The Project Director Telephone Interview includes information on state/local implementation, fiscal planning, coordination and organizational challenges, workforce development, quality assurance procedures, sustainability planning, and leadership and political issues.
The Core Staff Web Survey will be administered one time to core NITT-HT staff shortly before the grantee visit in NITT-HT program years 3 (for eight grantees) or year 4 (for nine grantees). “Core staff” are defined as staff members serving as primary providers of planning, case management and coordination services to youth/young adults (“life coaches,” “transition facilitators,” or “transition specialists”). The Core Staff Web Survey uses brief, previously validated instruments to assess characteristics of person-centered practice and barriers to this practice (Tondora & Miller, 2009; Malloy, Haber, Burgess, LaPorte, & Corral, 2015).
The Multi-Media Project Young Adult Probes offer a youth/young adult driven data collection method. Youth/young adult involvement is a priority both for the NITT-HT national evaluation and for NITT-HT grantee communities. Consequently, it will be important to offer youth/young adults opportunities to participate in national evaluation activities in developmentally-appropriate and engaging ways. In the Multi-Media Project, youth/young adults will be invited to voluntarily provide information about their experiences working with or being served by NITT-HT grantee communities using multi-media outlets. These outlets could include videos, photos, blogs, or poems (at the choice of the participating youth/young adult). Youth/young adults will be given informational probes in grantee Years 2, 3, and 4. These probes include questions such as: Who are the most supportive people in your life? What memory sticks out the most when you think about your involvement with (insert program name)? What keeps you coming back to services? What keeps you involved in NITT-HT activities? This multi-method component will result in an especially rich description of youth experiences at NITT-HT learning laboratories.
The Supplemental Youth and Youth Adult Interviews (SYAI) will be conducted with 90 service recipients per NITT-HT grantee at program enrollment (baseline) and 12- and 24-months after enrollment. These 90 cases will be evenly distributed across the grantee’s 2-3 learning laboratories. The SYAI is designed for administration as an audio computer-assisted self-interviewing (ACASI) survey. This mode was selected to offer participating young and young adults maximum privacy while completing the interview and to present minimal survey administration burden to NITT-HT grantee staff. The SYAI includes standardized instruments as well as project-developed items to assess key outcomes of interest for the NITT-HT program. Outcomes assessed within the SYAI include: school/home/daily living functioning, emotional/behavioral health, vocation and education status, housing stability, criminal or juvenile justice involvement, psychotic symptoms, substance use/abuse, trauma symptoms, victimization experiences and propensity to commit violent acts. In addition to primary outcomes of interest, the SYAI interview also assesses intermediate outcomes thought to be critical in influencing change in behavioral health and functioning. Intermediate outcomes assessed within the SYAI include self-efficacy (mental health, school, career and social), perceptions of social support, and perceptions of person-centered care and service alliance.
Additional detail about the specific modules and associated instruments included in the SYAI is included in Table 2 below. The table offers detailed information on the number of items included in the Program Enrollment (baseline) version of the SYAI. The follow-up instrument version is almost identical with a few exceptions—Module 12 (Psychotic Experiences) is not included in the follow-up instrument and Modules 15 (Mental Health Service Use) and 20 (Perceptions of Care and Alliance) have more items in the follow-up instrument to reflect a longer period of potential service engagement. The two separate SYAI instruments are included in Attachments 9 and 10.
Table 2. Supplemental Young Adult Interview Program Enrollment Items and Source Materials
Module 1: Audio Computer Assisted Self-Interviewing (ACASI) Tutorial (6 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
6 |
Practice items to teach respondent the ACASI interview system |
|
Module 2: Demographic Characteristics (up to 6 items) |
||
# of Item(s) |
Construct: Source(s) |
|
Baseline |
12 & 24-Month |
|
5 |
1 |
Demographic characteristics (gender, race/ethnicity, sexual orientation) |
1 |
1 |
Date of birth |
Module 3: Living Situation (30 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
2 |
Primary living location: Project developed item |
|
25 |
Number of people in the household, relationship to household members: NSDUH items |
|
1 |
# moves in the past 12 months: NSDUH item |
|
1 |
# nights homeless in past 30 days: Project developed item |
|
Module 4: Education (up to 10 items) |
||
# of Item(s) |
Construct: Source(s) |
|
Baseline |
12 & 24-Month |
|
1 |
0 |
Highest grade completed: NSDUH item |
5 |
5 |
School enrollment: NSDUH items |
1 |
1 |
School enrollment expectations: Project-developed item |
3 |
0 |
Highest grade completed for father and mother: Project-developed items |
Module 5: Employment (9 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
8 |
Employment status and hours worked: NSDUH items |
|
1 |
Employment expectations: Project-developed item |
|
1 |
Job satisfaction: Project developed item |
|
Module 6: Y/YA Income (4 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
2 |
Supplemental Security Income (SSI) and SSI intentions: NSDUH and project developed item |
|
2 |
Hourly wage: Project developed items |
|
(continued)
Table 2. Supplemental Young Adult Interview Program Enrollment Items and Source Materials (continued)
Module 7: General Health (1 item) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
1 |
Overall health: NSDUH item |
|
Module 8: Health Insurance (2 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
2 |
Insurance status: NSDUH items |
|
Module 9: Arrest/Criminal Justice (up to 4 items) |
||
# of Item(s) |
Construct: Source(s) |
|
Baseline |
12 & 24-Month |
|
2 |
0 |
Arrest and jail/detention history (ever): NSDUH items |
2 |
2 |
Past year arrest and jail/detention: Project-developed items |
Module 10 & 11: General Mental Health and Functional Impairment (19 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
6 |
Nonspecific psychological distress: Kessler-6 (K6: Kessler et al., 2003) |
|
5 |
Feelings in the past 30 days: NSDUH items |
|
8 |
Disturbance in social adjustment and behavior: World Health Organization Disability Assessment Schedule (WHO-DAS, NSDUH version; Novak, Colpe, Barker, & Gfroerer, 2010; Substance Abuse and Mental Health Services Administration, 2010) |
|
Module 12 (Program Enrollment only): Psychotic Experiences (16 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
16 |
Psychosis risk: Prodromal Questionnaire (PQ-16; Ising et al., 2012) |
|
Module 13: Behavior Disorders and Substance Use (10 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
10 |
Externalizing and substance use impairment: Global Appraisal of Individual Needs–Short Screener, External and Substance Use Disorder scales (GAIN-SS; Dennis, Chan, & Funk, 2006) |
|
Module 14: Trauma Symptoms (6 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
6 |
Post-Traumatic Stress Disorder Symptoms: Post-Traumatic Stress Checklist—Civilian (PCL-C; National Center for PTSD, 2012) |
|
(continued)
Table 2. Supplemental Young Adult Interview Program Enrollment Items and Source Materials (continued)
Module 15: Mental Health Service Use (up to 21 items) |
||
# of Item(s) |
Construct: Source(s) |
|
Baseline |
12 & 24-Month |
|
8 |
8 |
Outpatient mental health service use: NSDUH items |
8 |
8 |
Inpatient mental health service use: NSDUH items |
2 |
1 |
Psychotropic medication use: NSDUH items |
0 |
3 |
Mental health treatment: NSDUH items |
Module 16: Victimization and Violence (16 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
16 |
Violence perpetration and victimization: MacArthur Community Violence Screening Instrument (MCVSI; Steadman et al., 1998) |
|
Module 17: Major Depressive Episode (50 items) |
||
# of Item(s) Baseline, 12 & 24-Month |
Construct: Source(s) |
|
50 |
Depression: NSDUH items |
|
Module 18: Self-Efficacy (up to 32 items) |
||
# of Item(s) |
Construct: Source(s) |
|
Baseline |
12 & 24-Month |
|
7 |
7 |
General self-efficacy: DCI |
8 |
8 |
Academic self-efficacy: Bandura’s Multi-dimensional Scales of Perceived Self-efficacy (MSPSE) (select items; Bandura, 1989a, 1989b) and “What Happens at School” subscale of the American Institutes of Research Self-determination Scale (AIR-SDS; select items, Wolman, Campeau, Dubois, Mithaug, & Stolarski, 1994) |
10 |
10 |
Career self-efficacy: Job Observation and Behavior Scale: Opportunity for Self-determination (JOBS: OPS) (Brady, Rosenberg, & Frain, 2008, n.d.) |
7 |
7 |
Social self-efficacy: Project-developed items |
Module 19: Social Support (up to 40 items) |
||
# of Item(s) |
Construct: Source(s) |
|
Baseline |
12 & 24-Month |
|
36 |
40 |
Social support and connectedness: Assessment of Social Connectedness (select items; Cook & Kilmer, 2010) |
(continued)
Table 2. Supplemental Young Adult Interview Program Enrollment Items and Source Materials (continued)
Module 20: Service Perceptions and Alliance (up to 21 items) |
||
# of Item(s) |
Construct: Source(s) |
|
Baseline |
12 & 24-Month |
|
3 |
2 |
NITT-HT services received: Project developed items |
5 |
14 |
Alliance with service provider: Working Alliance Inventory-Short (select items; Tracey & Kokotovic, 1989) |
0 |
5 |
Person-centered care: Select items adapted from Tondora & Miller (2009) and Malloy, Haber, Burgess, LaPorte, & Corral, (2015) |
Note: NSDUH=National Survey of Drug Use and Health
The Services & Supports Inventory will be administered one time by telephone to an individual nominated by the NITT-HT Project Director. The Project Director will be asked to nominate a key informant best positioned to reflect on services being implemented within the target NITT-HT grant community. This telephone interview, to be administered prior to the grantee visit, is designed to identify specific providers and other stakeholders to participate in the 1 time grantee visit. Interviewers will solicit information about specific services, especially evidence-based and evidence-informed practices being provided to youth/young adults through NITT-HT associated behavioral health or other professional agencies. The respondent will also provide a preliminary assessment of the frequency and quality of implementation of the practice(s) to (1) characterize the number of evidence-based practices being implemented within the NITT-HT grantee community as well as the overall level and variability in the quality of evidence-based practice implementation throughout the system; and (2) enable selection of the providers who are well-positioned to describe experiences with evidence-based practice implementation within the NITT-HT grantee communities.
Grantee visits are planned to occur one time per NITT-HT grantee over the course of the NITT-HT grantees’ funding period (either in grantee Year 3 or 4). During these one-time grantee visits, several in-person interviews will be conducted with NITT-HT staff along with two client-oriented focus groups. Staff participants will be identified by grantee leaders during the Project Director Telephone Interview and completion of the Services and Supports Inventory (which are completed prior to the grantee visit). If key individuals identified by grantee leaders are not available during the in-person site visits, telephone interviews may be conducted in lieu of in-person interviews. If telephone interviews are needed for select individuals, the interview guide will be identical to that used for the in-person interviews. The grantee visit will also include a document review process.
The following section describes the types, respondents, purpose and content of the Grantee Visit In-Person Interview and Focus Group Guides.
Youth Coordinator In-Person Interview Guide. The Youth Coordinator In-Person Interview will consist of qualitative prompts designed to elicit youth coordinator staff experiences working with their NITT-HT grantees. The interview prompts focus on the Youth Coordinator functions including participation in planning and coordination, outreach, mentoring, and other activities. Interview prompts are informed by a previously piloted survey of statewide peer mentor training (Legere & Nemec, n.d.).
Provider In-Person Interview Guide. In-person Provider Interviews will be conducted during NITT-HT grantee visits. Providers are individuals who provide behavioral health services/treatment directly to youth and youth adults served within the NITT-HT community, other than the transition facilitators. These individuals will likely come from NITT-HT partner organizations. Interview prompts focus on two areas: (1) questions related to perceptions of organizational support by the collaborative (see web-based collaborative survey), and (2) questions regarding implementation of evidence-based practices (Aarons, 2004), including general attitudes, types of practices being used, implementation supports (e.g., manuals or toolkits used and access to these), and use of fidelity assessments.
Other Stakeholder In-Person Interview Guide. Stakeholders making important contributions to developing service and support systems do not always easily fall into conventional categories like service provider or administrator. Often, such stakeholders (e.g., board members for agencies, leaders or liaisons for advocacy groups, leaders or advocates with religious or charitable organizations) make critical contributions to the development of a system without which the system’s development goals would not be reached. In-person interviews will also be conducted during grantee visits with these other key stakeholders, as identified by grantee leadership. The in-person interview will consist of qualitative prompts designed to elicit experiences contributing to systems development, including the history of their involvement, their specific contributions to the systems development effort, and strategies, barriers and facilitators to making these contributions.
Young Adult Focus Group Guide and Information Sheet. Two focus groups of young adults will be conducted during the grantee visit—one for youth/young adults directly involved in NITT-HT system change efforts, and one for youth and youth adults who are recipients of NITT-HT services. Two different guides offer prompts for each focus group. Prompts are designed to elicit perceptions based on youth/young adult lived experience. Prompts will ask about resources to support successful youth/young adult transition at their NITT-HT sites, whether site practices are well aligned to address needs and cultivate resources, and their ideas about how to build on these achievements in the future. An information form without any personally identifiable information will be completed by each focus group participant. The purpose of this information form is to gather general background information (e.g., demographics, extent of experience with the mental health system and grantee community) about focus group participants.
Family/Adult Ally Focus Group and Information Sheet. Two family/adult ally focus groups will be conducted during the NITT-HT grantee visit—one focused at the client-level (for family members of youth/young adults service recipients), and one focused at the systems level (for family members involved in NITT-HT grantee planning and systems change efforts). The Family/Adult Ally focus group guides include prompts to gather information about family member perceived needs and resources to support youth/young adults at their NITT-HT sites. Focus group prompts are based on the Youth and Family Involvement in Teams (Y-FIT) instrument (LaPorte et al., 2014). This instrument focuses on family member and adult ally experiences in participating in and supporting person-centered planning with youth/young adults in NITT-HT. An information form without any personally identifiable information will be completed by each focus group participant. The purpose of this information form is to gather general background information (e.g., e.g., demographics, extent of experience with the mental health system and grantee community) about focus group participants.
Grantee Visit Document Review. Files or charts of a subset of youth/young adults participating in the SYAI will be reviewed during the grantee visit. This document review will be designed to ascertain types of standard documentation routinely completed for youth/young adult clients served as well as the consistency of completion of these documents. Information extracted from client charts will be programmatic only; there will be no identifying or personal information extracted from these client charts.
The multiple data collection methods and sources included within the NITT-HT national evaluation are necessary to address the core NITT-HT evaluation questions. These questions require that information be gathered from individuals representing various layers of the NITT-HT grantee community (state, learning laboratory, provider organizations). The evaluation questions also require that multiple types of individuals report upon key outcomes of interest (project directors, key grantee staff, service providers, youth and young adults and family members). Table 3 provides an overview of the NITT-HT evaluation questions, main constructs of interest and the data sources and modules/sections of the data collection instruments used to measure them.
Table 3. Now Is the Time-Healthy Transitions (NITT-HT) National Evaluation-Evaluation Questions, Constructs, Data Sources and Instrument Module/Sections
EQ1. Did NITT-HT lead to enhanced collaboration and coordination across youth-serving systems for youth/young adults ages 16 to 25? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Collaboration |
CSTI |
Community Partnership (Theme 1); Collaborative Action (Theme 2) |
|
SSTI |
Partnerships (Theme 1); Collaborative Action (Theme 2) |
|
CDP Grantee-level outcomes* |
# organizations that entered into formal MOUs or MOAs; # of policy changes (infrastructure and organizational) completed as a result of the grant |
Coordination |
CSTI |
Access to Needed Support and Services (Theme 6) |
|
SSTI |
Access to Needed Support and Services (Theme 5) |
EQ2. What were the barriers/facilitators to state-/local-level collaboration, partnership development, and shared decision-making? How were they addressed? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Barriers/facilitators to Collaboration |
Project Director Telephone Interview |
Composition/Functioning of Advisory Group; Cross Sectoral Coordination (Systems-Level); Cross Sectoral Coordination (Services-Level) |
|
Collaborative Member Survey |
Council Effectiveness; Council Vision |
|
Collaborative Self-Assessment |
Collaborative Participation |
|
Youth Coordinator In-Person Interview |
Composition/Functioning of Advisory Group; |
(continued)
Table 3. Now Is the Time-Healthy Transitions (NITT-HT) National Evaluation-Evaluation Questions, Constructs, Data Sources and Instrument Module/Sections (continued)
EQ2. What were the barriers/facilitators to state-/local-level collaboration, partnership development, and shared decision-making? How were they addressed? (continued) |
||
Barriers/facilitators to partnership development |
Project Director Telephone Interview |
State/Local Partnership/Coordination; Contextual Political/Leadership Issues/Strategies; |
|
Collaborative Member Survey |
Council Functioning; Managing Conflict; Member Satisfaction |
|
Collaborative Self-Assessment |
Team Development |
|
Youth Coordinator In-Person Interview |
State/Local Partnership/Coordination; Other Contextual Political/Leadership Issues/Strategies; |
Barriers/facilitators to shared decision making and person-centered planning |
Core Staff Web Survey |
Barriers to Person-Centered Planning |
|
Youth Coordinator In-Person Interview |
Youth and Family Voice (Systems Level) |
EQ3. How was youth/young adult (Y&YA) voice integrated into aspects of NITT-HT program planning, implementation, and local evaluation efforts? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Y&YA Voice in Planning |
Y&YA Focus Groups (Systems-Level) |
Initial Engagement; Personal Engagement; Youth Involvement |
|
Multi-Media Project Young Adult Probes |
Youth Voice/Youth Involvement |
|
Youth Coordinator In-Person Interview |
Youth and Family Voice (Systems Level) |
|
Project Director Telephone Interview |
Youth and Family Voice (Systems Level) |
|
Collaborative Self-Assessment |
Family Engagement |
Y&YA Voice in Program Implementation |
Y&YA Focus Groups |
Service Improvement; Youth/Family Engagement in Services |
|
Multi-Media Project Young Adult Probes |
Youth Voice/Youth Involvement |
|
Youth Coordinator In-Person Interview |
Youth and Family Voice (Systems Level) |
Y&YA Voice in Local Evaluation |
Y&YA Focus Groups (Systems-Level) |
Service Improvement |
|
Youth Coordinator In-Person Interview |
Youth and Family Voice (Systems Level) |
(continued)
Table 3. Now Is the Time-Healthy Transitions (NITT-HT) National Evaluation-Evaluation Questions, Constructs, Data Sources and Instrument Module/Sections (continued)
EQ4. How was family and adult ally voice integrated into aspects of NITT-HT program planning, implementation and local evaluation efforts? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Family and Adult Ally Voice in Planning |
Family and adult ally focus groups (Systems-Level) |
Role in Systems Change; Initial Engagement; Personal Engagement |
|
Project Director Telephone Interview |
Youth and Family Voice (Systems Level) |
|
Youth Coordinator In-Person Interview |
Youth and Family Voice (Systems Level) |
|
Collaborative Self-Assessment |
Family Engagement |
Family and Adult Ally Voice in Program Implementation |
Family and adult ally focus groups (Systems-Level) |
Family Engagement in Services; Service Improvement |
|
Project Director Telephone Interview |
Youth and Family Voice (Systems Level) |
Family and Adult Ally Voice in Local Evaluation |
Youth Coordinator In-Person Interview |
Youth and Family Voice (Systems Level) |
|
Project Director Telephone Interview |
Youth and Family Voice (Systems Level) |
EQ5. To what extent were services and supports for youth/young adults with serious mental health conditions developed, improved and expanded as a result of NITT-HT? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Y&YA General Services and Supports |
Services and Supports Inventory |
Prompts to identify grantee community service and support array |
|
Project Director Web Survey |
Basic Program Description; Measurable Progress |
|
Project Director Telephone Interview |
Description of Core Programs and Staff; Workforce Development |
|
Youth Coordinator In-Person Interview |
Description of Core Programs and Staff; Workforce Development |
Evidence-Based Practices |
Services and Supports Inventory |
Prompts to identify grantee community evidence-based practice use |
|
Project Director Web Survey |
Basic Program Description |
|
Project Director Telephone Interview |
Description of Core Programs and Staff; Workforce Development |
|
Provider In-Person Interview |
Implementation; Organizational Capacity |
Early Intervention Services and Supports for Early Psychosis |
Services and Supports Inventory |
Prompts to identify grantee community use of early intervention services and supports |
|
Project Director Telephone Interview |
Description of Core Programs and Staff |
(continued)
Table 3. Now Is the Time-Healthy Transitions (NITT-HT) National Evaluation-Evaluation Questions, Constructs, Data Sources and Instrument Module/Sections (continued)
EQ6. What were the effects on service capacity and access to care in communities implementing NITT-HT? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Service Capacity |
CSTI |
Transition Planning Quality Assurance and Support (Theme 3); Fiscal Policies and Sustainability (Theme 5); Accountability (Theme 7) |
|
SSTI |
Fiscal Policies and Sustainability (Theme 4); Accountability (Theme 6) |
Workforce Training and Support |
CSTI |
Workforce (Theme 4) |
|
SSTI |
Workforce (Theme 3) |
Access to Care |
SYAI |
Mental Health Service Use (Module 15) |
|
CDP System-Level Outcomes* |
# of individuals referred for services; # of individuals who received services |
|
CSTI |
Access to Needed Support and Services (Theme 6) |
|
SSTI |
Access to Needed Supports and Services (Theme 5) |
|
DCI * |
Recovery, Self-Help, and Peer Support (Section F2) |
EQ7. What NITT-HT strategies were effective in developing, improving, and expanding services and supports for youth/young adults with serious mental health conditions? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Y&YA Services and Supports |
Services and Supports Inventory |
Prompts to identify grantee community service and support array |
|
Project Director Web Survey |
Basic Program Description; Measurable Progress |
|
Project Director Telephone Interview |
Description of Core Programs and Staff |
|
Youth Coordinator In-Person Interview |
Description of Core Programs and Staff; Quality Assurance/Other Implementation Drivers |
Person-Centered Planning and Case Management |
Core Staff Web Survey |
Person-Centered Planning; Barriers to Person-Centered Planning |
Evidence-Based Practice Implementation |
Services and Supports Inventory |
Implementation Drivers; Supervision; Collaboration; Drift |
|
Project Director Telephone Interview |
Description of Core Programs and Staff; Quality Assurance/Other Implementation Drivers; Sustainability/Dissemination |
(continued)
Table 3. Now Is the Time-Healthy Transitions (NITT-HT) National Evaluation-Evaluation Questions, Constructs, Data Sources and Instrument Module/Sections (continued)
EQ7. What NITT-HT strategies were effective in developing, improving, and expanding services and supports for youth/young adults with serious mental health conditions? (continued) |
||
Evidence-Based Practice Infrastructure and Support |
Provider In-Person Interview |
Organizational Capacity; EBP Attitudes Scale |
|
Other Stakeholder In-Person Interview |
Organizational Capacity; EBP Attitudes Scale |
|
Project Director Telephone Interview |
Description of Core Programs and Staff; Quality Assurance/Other Implementation Drivers; Sustainability/Dissemination |
Perceptions of Care Quality and Service Alliance |
SYAI |
Service Perceptions and Alliance (Module 21) |
|
Y&YA Focus Groups (Client-Level) |
Initial Engagement; Enrollment Process; Goal Development; Goal Implementation; Transition; Support System |
|
Family and Adult Ally Focus Groups (Client-Level) |
Ongoing Communication; Balance of Involvement; Family Support |
EQ8. What were effective NITT-HT strategies to increase access to care? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Y&YA Services and Supports |
Services and Supports Inventory |
Prompts to identify grantee community service and support array |
Person-Centered Planning and Case Management |
Core Staff Web Survey |
Person-Centered Planning; Barriers to Person-Centered Planning |
Evidence-Based Practice Implementation |
Services and Supports Inventory |
Prompts to identify grantee community evidence-based practice use |
|
Project Director Telephone Interview |
Description of Core Programs and Staff; Quality Assurance/Other Implementation Drivers |
|
Provider In-Person Interview |
Implementation Drivers; Supervision; Collaboration; Drift |
Perceptions of Care Quality and Service Alliance |
SYAI |
Service Perceptions and Alliance (Module 21) |
|
Y&YA Focus Groups (Client-Level) |
Initial Engagement; Enrollment Process; Goal Development |
|
Family and Adult Ally Focus Groups (Client-Level) |
Barriers to Involvement; Ongoing engagement |
(continued)
Table 3. Now Is the Time-Healthy Transitions (NITT-HT) National Evaluation-Evaluation Questions, Constructs, Data Sources and Instrument Module/Sections (continued)
EQ9. What are effective awareness outreach and engagement strategies for youth/young adults ages 16 to 25 for behavioral health disorders? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Y&YA Services and Supports |
Services and Supports Inventory |
Prompts to identify grantee community service and support array |
|
Project Director Web Survey |
Basic Program Description; Measurable Progress |
|
Project Director Telephone Interview |
Description of Core Programs and Staff |
|
CDP System-Level Outcomes* |
# individuals screened for mental health treatment; # individuals contacted through outreach efforts |
Outreach and Engagement Strategies |
Services and Supports Inventory |
Prompts to identify grantee community outreach and engagement strategies |
|
Project Director Telephone Interview |
Description of Core Programs and Staff |
|
Y&YA Focus Groups (Client-Level) |
Initial Engagement; Enrollment Process; Goal Development |
|
Family and Adult Ally Focus Groups (Client-Level) |
Barriers to Involvement; Ongoing engagement |
|
Youth Coordinator In-Person Interview |
Description of Core Programs and Staff |
|
Multi-Media Project Young Adult Probes |
Youth Engagement/Factors for Success; Relationships |
EQ10. To what extent did NITT-HT identify and serve young people who would have otherwise fallen through the cracks? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Mental health |
SYAI |
General Mental Health (Module 10); Functional Impairment (Module 11); Major Depressive Episode (Module 17) |
Mental health treatment access |
SYAI |
NSDUH items from Mental Health Service Use (Module 15) |
EQ11. To what extent did NITT-HT increase mental health service access for Y&YAs in need? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Mental health service access |
SYAI |
Mental Health Service Use (Module 15) |
|
DCI* |
Recovery, Self-Help, and Peer-Support (Section F2) |
|
CSTI |
Access to Needed Support and Services (Theme 6) |
|
SSTI |
Access to Needed Supports and Services (Theme 5) |
(continued)
Table 3. Now Is the Time-Healthy Transitions (NITT-HT) National Evaluation-Evaluation Questions, Constructs, Data Sources and Instrument Module/Sections (continued)
EQ12. What are the outcomes associated with NITT-HT in the areas of education, employment, housing, primary care, and mental health? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Education & education expectations |
SYAI |
Education (Module 4) |
|
DCI* |
Education, Employment and Income (Section D) |
Employment & employment expectations |
SYAI |
Employment (Module 5) |
|
DCI* |
Education, Employment and Income (Section D) |
Housing |
SYAI |
Living Situation (Module 3) |
|
DCI* |
Family and Housing (Section C) |
Primary Care |
SYAI |
General Health (Module 7) |
|
DCI |
Disability Measures (Section A3); Mental and Physical Health (Section F1—physical health items) |
Mental health |
SYAI |
General Mental Health (Module 10); Functional Impairment (Module 11); Psychotic Experiences (Module 12); Behavior Disorders and Substance Use (Module 13); Trauma Symptoms (Module 14) |
|
DCI* |
Drug and Alcohol Use (Section B); Mental and Physical Health (Section F1—mental health items) |
EQ13. To what extent did NITT-HT lead to decreased involvement with juvenile and criminal justice systems? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Arrest (Ever and past 12 months) |
SYAI |
Arrest/Criminal Justice (Module 9) |
Arrest (past 30 days) |
DCI* |
Crime and Criminal Justice Status (Section E) |
Jail/Juvenile Detention |
SYAI |
Arrest/Criminal Justice (Module 9) |
Violence |
SYAI |
Victimization and Violence (Module 16) |
EQ14. What is the effect of the NITT-HT intervention compared to a national sample of similar young adults not participating in the NITT-HT intervention? |
||
Construct |
Data Source |
Instrument Module/Section or Item(s) |
Mental Health |
SYAI |
General Mental Health (Module 10); Functional Impairment (Module 11); Major Depressive Episode (Module 17) |
Functional Outcomes (education, employment, housing, general health) |
SYAI |
NSDUH items from Education (Module 4), Employment (Module 5), Housing (Module 3) and General Health (Module 7) |
Mental Health Treatment Access |
SYAI |
NSDUH items from Mental Health Service Use (Module 15) |
Note. Asterisks (*) indicate data not collected as part of the NITT-HT National Evaluation, but that will be made available to the National Evaluation team for analysis.
Note. CSTI, Community Support for Transitions Inventory; SSTI, State Support for Transitions Inventory; CDP, Common Data Platform; DCI, Data Collection Instrument; SYAI, Supplemental Youth and Youth Adult Interview.
The following are self-administered, web-based surveys to be completed through the NITT-HT online data collection system.
Community Support for Transition Inventory (CSTI)
State Support for Transition Inventory (SSTI)
Collaborative Member Survey
Collaborative Self-Assessment
Project Director Web Survey (T1, T2 and T3)
Core Staff Web Survey
Supplemental Youth and Young Adult Interview (SYAI)
Using a web instrument allows for automated data checks as well as for skip procedures which will reduce the burden among respondents and possibility of data entry error, thereby increasing the efficiency of data entry and improving data quality. The automated data checks will help respondents give valid responses (e.g. restricting the range of responses when asked about “number of times events occurred in the past year” to 0 to 365), and also ensure that responses follow the expected format (e.g. numbers or dates where those are expected). Responses will generate skip patterns for later questions in the instrument, where the respondents only complete relevant sets of questions and do not see others (e.g. in the SYAI, respondents are not administered questions about behaviors in the past 12 months if the event never occurred in response to a previous “lifetime” question).
Using a web-based system will provide the capability to send automatic email reminders to grantees if and when surveys have not been completed.
A dashboard and other reports will also be available to SAMHSA and the NITT-HT team, as well as the grantees, to monitor the overall status of data collection. Grantees will have access to their own data and will also be able to review online descriptive findings for their associated learning laboratories.
The qualitative instruments include telephone and in-person interviews, including focus group moderator guides administered as part of the NITT-HT data collection. Respondents will be read questions by a NITT-HT evaluation team liaison while a NITT-HT note taker records each response. With respondent consent, interviews will be recorded as a back-up to the note taker. After the interview, the interviewer and note taker will review the completed interview notes for accuracy; any areas of discrepancy will be validated with the recording. Once the interview responses are considered final, the recording will be deleted. Until they are deleted, the recordings will be kept by the note taker on a secure, password protected computer.
Telephone
Project Director Telephone Interview
Services & Supports Inventory
Site Visit In-person Interviews and Focus Groups
Youth Coordinator In-Person Interview
Provider In-Person Interview
Other Stakeholder In-Person Interview
Young Adult Focus Group Guide & Information Form
Family/Adult Ally Focus Group Guide & Information Form
SAMHSA monitors the performance of NITT-HT programs by requiring the grantees to collect and submit data through the Client-Level Services Measures for Discretionary Programs - Data Collection Instrument (DCI) (OMB No. 0930-0346). A detailed cross-walk was developed to outline areas, instruments and items included within the DCI and those included within the SYAI to ensure that the SYAI items did not duplicate what will be available via the DCI. The SYAI – baseline, 12-month follow-up, and 24-month follow-up cover a few of the same overarching topics as the client-level DCI (e.g., education, employment, living situation) but there are no duplicate items or measures collected from both the SYAI – baseline and the 12- and 24-month follow-up and the client-level DCI . Additionally, the DCI questions cover a previous 30-day timeframe which is not robust enough to accurately assess the impact of treatment services on outcome measures or establish best practices, which are both primary objectives of the national evaluation. The SYAI – baseline, 12-month follow-up, and 24-month follow-up questions are unique from the DCI questions in that timeframes are extended from assessing the previous 30-days to assessing the previous 6 or 12 months (depending upon the indicator). There are also several topics included in the SYAI that are not covered at all within the DCI. These include perceptions of academic and career self-efficacy, social support, psychotic symptoms, propensity to commit violent acts, and reports on past year outpatient and inpatient mental health service and psychotropic medication use. The SYAI also uniquely includes items to assess youth and young adult perceptions of the quality of services received through the NITT-HT grant program.
Additionally, the DCI is administered to all clients receiving NITT-HT services while the SYAI will be administered on a voluntary basis to a subset of 90 clients across the NITT-HT grantee community who receive services. Follow-up administrations of the DCI are only required for those youth/young adults still actively engaged in services. For the purposes of a national evaluation, relying solely on DCI follow-up data would be biased. Consequently, the SYAI will be administered at 12- and 24-months, regardless of whether or not the client is still receiving NITT-HT services. At the 12- and 24-month SYAI interviews, fewer youth/young adult participants will need to complete both the DCI and SYAI.
For all other NITT-HT instruments, this evaluation is collecting information unique to NITT-HT programs that is otherwise not available for these grantees because of the scale and breadth of the initiative’s implementation: nationwide, across a spectrum of provider settings, and across a broad cross-section of youth/young adult service recipients. A literature review prepared by the evaluation team in the spring of 2015 confirmed that the information being collected by these data collection instruments cannot be obtained through other sources.
Participation in this evaluation will not impose a significant impact on small entities. NITT-HT grantees and learning laboratories may include state agencies, tribal organizations and other jurisdictions, and community service providers. Some of the community service providers may be small entities; however, the NITT-HT data collection instruments are designed to include only the most pertinent information needed to be able to carry out the evaluation effectively, and their impact will not be significant.
The Supplemental Youth and Young Adult Interview (SYAI) – baseline, 12-month Follow-up, and 24-month Follow-up will be administered on a voluntary basis to clients who receive services under the NITT-HT grant. Only those clients who complete the initial baseline interview will be asked to complete 12- and 24-month follow-up interviews. Data collection at these follow-up points is necessary to measure the short- and longer-term outcomes of the NITT-HT program.
Following up at 12- and 24-months is optimal for producing useful outcome data. Waiting until 12-months after the initial receipt of services allows enough time for effects of the NITT-HT program to develop, including changes in mental health symptoms, substance use behavior, stable housing, criminal justice involvement, education, employment, and perceptions of care. The contractor will be able to make strong 12- and 24-month comparisons by conducting the SYAI with a total of 90 clients from each of the NITT-HT grantee (distributed across all grantee affiliated learning laboratories) and applying the contractor’s extensive experience studying longitudinal outcomes among youth/young adults.
The multiple data collection points for the CSTI, SSTI, Collaborative Member Survey, Collaborative Self-Assessment, Project Director Web Survey and Project Director Telephone Interview are necessary to track and evaluate grantees’, learning laboratories’ and grantees’ progress and change over time. Without multiple data collection points, SAMHSA would be unable to adequately evaluate the NITT-HT initiative and measure progress and change over time associated with these grantees and learning laboratory programs and services for this youth/young adult population. The NITT-HT evaluation has made every effort to ensure that data are collected only when necessary and that extraneous collection will not be conducted.
This information collection fully complies with the guidelines in 5 CFR 1320.5(d)(2).
The notice required by 5 CFR 1320.8(d) was published in the Federal Register on November 10, 2015 (80 FR 69689). No comments were received.
SAMHSA has made extensive use of experts in the area of young adult mental health research and current grantee staff (e.g., project directors, local evaluators), including family members and young adults with lived experiences, to provide guidance on the design and analysis of the NITT-HT evaluation. The following Healthy Transitions in-person and virtual meetings were held to review the various aspects of the NITT-HT evaluation, including the preliminary evaluation plan and data collection procedures which were distributed prior to each meeting to give participants the opportunity to review and formulate feedback.
Introductory Webinar (June 2015)
Grantee Meeting (August 2015)
Evaluator Virtual Meeting (August 2015)
Expert Panel Meeting (August 2015)
Grantees and local evaluators have had several opportunities to provide feedback on the preliminary evaluation plan and data collection procedures. Feedback was provided on all aspects of the evaluation and participants’ comments and suggestions were incorporated into the development of the data collection procedures and instruments. Feedback included:
Create opportunities to provide NITT-HT grantees with feedback from the program evaluation throughout the evaluation period.
Perform a cross-walk of the SAMHSA DCI tool with the SYAI to ensure a lack of duplication between the two youth/young adult instruments.
Create the SYAI to be administered electronically (as opposed to site interviewer-administered) in 45 minutes or less.
Offer youth/young adults incentives for their participation in the national evaluation interviews.
The list of expert panel members is provided in Table 4.
Table 4. Now is the Time – Healthy Transitions Evaluation Expert Panel
Expert |
Affiliation |
Contact Information |
MaJosé Carrasco, M.P.A |
Director |
Email: majose@nami.org |
Lynda Gargan, Ph.D., M.Ed. |
Interim Executive
Director |
Phone: 240-403-1901 Email: lgargan@ffcmh.org |
Susan Keys, Ph.D. |
Associate Professor
and Senior Researcher |
Phone: 541-322-2046 Email: susan.keys@osucascades.edu |
Benjamin Le Cook, Ph.D., M.P.H. |
Assistant Professor of
Psychiatry |
Phone: 617-503-8449 Email: bcook@cha.harvard.edu; bcook@charesearch.org |
Pamela Orpinas, Ph.D. |
Professor |
Phone: 706-542-4372 Email: porpinas@uga.edu |
Mark Salzer, Ph.D. |
Professor and
Chair |
Phone: 215-204-7879 Email: mark.salzer@temple.edu |
Supplemental Youth and Young Adult Interview (SYAI): NITT-HT clients are transition age youth/young adults with mental health disorders who can typically be a hard-to-reach, and sometimes transient, population. To increase response rates, all clients who agree to participate in the SYAI at baseline will receive a gift card respondent payment worth a $20 value. Clients who agree to participate in the 12- and 24-month follow-up will receive a gift card respondent payment worth a $25 value. Participating clients will not be penalized if they wish to skip questions or stop the interview before completing the baseline or either of the follow-up interviews. Survey research literature suggests that monetary payments to respondents have a strong positive effect on response rates and no known adverse effect on reliability. Research has shown improved response rates when remuneration is offered to respondents. Results from the 2001 National Household Survey on Drug Abuse (NHSDA) respondent payment experiments were reported by Wright, Bowman, Butler, & Eyerman (2005); key conclusions from their analyses are summarized below:
The $20 and $40 respondent payments each produced about a 10-point gain in overall response rates when compared with the $0 control group. The overall response rate was significantly higher for $40 than the $20 respondent payment within many of the subgroups addressed in the analysis. Both payment groups more than paid for themselves due to decreased costs of follow-up and more productive screening resulting from the improved response rates. Respondent payments motivate (or obligate) respondents to admit to substance use that they might not have admitted without the payment.
Young Adult Focus Group: Focus groups will be conducted with youth/young adults directly involved in NITT-HT system change efforts, and with youth/youth adults who are recipients of NITT-HT services. To increase response rates, all youth/young adults who agree to participate in the focus groups will receive a gift card respondent payment worth a $25 value as appreciation for their time and efforts. Where available, participants will also be provided bus passes to help facilitate their transportation to the focus group. Participants will not be penalized if they choose not to respond to or answer any questions raised during the group discussion or leave the discussion before its conclusion.
Family/Adult Ally Focus Group: Focus groups will be conducted with family/adult allies involved in NITT-HT grantee planning and systems change efforts, and with family/adult allies of youth/youth adults who are recipients of NITT-HT services. To increase response rates, all family/adult allies who agree to participate in the focus groups will receive a gift card respondent payment worth a $25 value as appreciation for their time and efforts. Where available, participants will also be provided bus passes to help facilitate their transportation to the focus group. Participants will not be penalized if they choose not to respond or answer any questions raised during the group discussion or leave the discussion before its conclusion.
No other incentives or gifts will be given to any respondents.
Concern for privacy and protection of respondents’ rights will play a central part in the implementation of all study components. The NITT-HT evaluation team members developing the instruments and surveys and analyzing the data have extensive experience protecting and maintaining the privacy of respondent data. All NITT-HT data will be securely stored on a protected server. A SORN/PIA is currently being sought.
Supplemental Youth and Young Adult Interview (SYAI)
The process of a self-administered web-based SYAI survey with audio capabilities on headphones is designed to protect client privacy, reduce discomfort and burden, and ensure that the data collected are of the highest quality. Though the SYAI will be self-administered by the client, the contractor is asking the NITT-HT grantee learning laboratory staff to assist with administering the SYAI at baseline and again at the 12- and 24-month follow-up if the client prefers to return to the learning laboratory to use the designated NITT-HT evaluation tablet to complete the interview. Initially it will be important to have someone familiar with the youth or young adult to explain the purpose of the NITT–HT evaluation to bolster participant engagement in the supplemental interview. The contractor will hold training sessions with all grantee learning laboratory selected staff to detail the steps involved in initiating the administration of the client interview and the procedures to follow to ensure protection of respondent’s rights and safeguarding of client data. The client interview consent form and frequently asked questions (FAQ) will be accessible via the survey web-based program and provided to the grantee learning laboratory staff as a handout to reference and give to clients for their records.
To begin the SYAI baseline, the learning laboratory selected staff will provide the client with a brief introduction to the interview and ask the client if they will agree to hear more. If the client agrees to proceed, the staff will read the consent form for the interview aloud to the client, answer any questions and ask if the client understands and agree to its contents. If the client understands and agrees, the staff will instruct the client to click the consent button on the consent page of the web-based program to acknowledge review and submit consent. A participant cannot complete the web-based instrument until they have submitted formal consent. The consent form will explain the purpose of the NITT-HT evaluation and the interview, describe the interview length and procedures, describe risks or benefits and steps the evaluation is taking to protect the client’s privacy, inform the client of the 12- and 24-month follow-up interviews, inform them of the incentive, and explain that the interview is voluntary and they may refuse to answer a question or stop the interview at any point without penalty. The consent form will also include the OMB approval expiration dates, the statement of survey burden, and the statement that the study is federally sponsored. The client will then be given the consent information page for their records. The staff will enter the NITT-HT site number and the client’s DCI ID number on the first screen of the interview.
For programs that serve minors or adults with legal guardians, learning laboratory staff will collect consent from parents and legal guardians as well as obtain “assent” from clients prior to starting the supplemental interview. The client will utilize the web-based program to acknowledge review and submit consent.
All clients who complete the SYAI Baseline will be asked to participate in the SYAI 12- and 24-month follow-up. The client will receive a consent form prior to each follow-up interview outlining the same content as the baseline consent form – participation is voluntary and they will not be penalized for non-participation.
To reduce burden on NITT-HT grantee learning laboratory staff, the 12- and 24-month contacting will be conducted by the contractor. In preparation for the follow-up interviews, the contractor will email clients regarding the interview, including a link to the web-based interview to complete on a personal computer or to return to the learning laboratory and complete on the designated NITT-HT evaluation tablet. The email will also include an evaluation team contact to call toll-free or e-mail with questions. Clients who have not completed a follow-up interview after four weeks of the contractor sending the email will receive a call from the contractor’s trained interviewer staff and given the option to complete the interview on the phone with a trained interviewer.
Client interviews will be identified with the client DCI number which will be necessary to link the baseline data with the 12- and 24-month follow-up data and to link the DCI data with the SYAI baseline, 12- and 24-month follow-up data. Client’s names, phone numbers, email addresses and contact information for someone who knows how to contact the client will be collected during the SYAI baseline interview for the purposes of re-contacting the client for the 12- and 24-month follow-up interviews. Clients’ names or other personally identifying information will not be linked to data collected.
Youth Coordinator In-Person Interview, Other Stakeholder In-Person Interview and Provider In-Person Interview. Prior to beginning a site visit interview, the participants, including both project staff and key informants, will be read, asked to sign and provided a copy of the consent form. The consent form informs participants of their rights, including the right to not answer any question; participants must provide written consent to participate in the site visit discussion before the interview begins. If a participant does not provide written consent to participate then the interview will not take place. Participants will also be asked for their consent to record the discussion. Recordings will be used to ensure that information is correctly captured from multiple interviews, information has been consistently captured across site visitors, and to correct and clarify brief written notes as needed and as part of data quality assurance procedures. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed. No individual-level or personal data will be collected by the NITT-HT evaluation from grantee or learning laboratory staff, who will only provide information about their organizations, NITT-HT activities, and implemented interventions, rather than information about themselves personally.
Young Adult and Family/Adult Ally Focus Groups. Prior to beginning the Young Adult and Family/Adult Ally Focus Group discussions, conducted during the Grantee Site Visit, the participants will be read, asked to sign and provided a copy of a consent form. The consent form informs participants of their rights, including the right to not answer any question or participate in any discussion; participants must provide written consent to participate before the focus group discussion begins. If a participant does not provide written consent to participate then that individual will be excused from the group and provided with the incentive in appreciation for their time. Participants will also be asked for their consent to record the discussion. Recordings will be used to ensure that information is accurately captured and to correct and clarify written notes as needed as part of data quality assurance procedures. Recordings will only be accessible to the contractor and will be stored on password-protected secure servers and destroyed once de-identified notes are completed.
Participants will also be asked to complete an information form the conclusion of the focus group. The form will not include any identifiable or personal information, but includes basic background and demographic information for analysis purposes only.
During the grantee visit, files or charts of a subset of youth/young adults participating in the SYAI will be reviewed. This document review will be designed to ascertain types of standard documentation routinely completed for youth/young adult clients served as well as the consistency of completion of these documents. Information extracted from client charts will be programmatic only; there will be no identifying or personal information extracted from these client charts.
All information collected during the site visit will be reported in aggregate only and individual respondents will not be identified.
All other data collection instruments, aside from the SYAI and Grantee Visit Interviews, collect programmatic data (i.e., information about the organizations and implemented interventions) at the grantee and learning laboratory levels along with aggregated, non-identifying participant-level data (e.g., estimated percent of individuals receiving specific types of intervention approaches). Sensitive respondent information, such as birthdates and social security numbers, will not be collected.
To create a profile for grantees and learning laboratory staff to log in to the NITT-HT online data collection system, the system will collect names, telephone numbers, mailing addresses and email addresses of grantee and learning laboratory staff. This identifying information will be accessible only to select NITT-HT evaluation staff and Grantee Project Officers at SAMHSA. No other personal information will be collected from participants as the focus of this data collection is on programmatic characteristics, including the processes and outcomes of the NITT-HT grantees and learning laboratories.
The NITT-HT systems development team takes responsibility for ensuring that the web and data system is properly maintained and monitored. Server staff will follow standard procedures for applying security patches and conducting routine maintenance for system updates. Data will be stored on a password-protected server, and access to data in the system will be handled by a hierarchy of user roles, with each role conferring only the minimum access to system data needed to perform the necessary functions of the role.
For all data collection activities, the contractor will use passwords to safeguard all project directories and analysis files containing completed survey data to ensure that there is no inadvertent disclosure of study data. Contractor staff has been trained on handling sensitive data and the importance of privacy. All contractor staff will sign a privacy pledge. In addition, interviews and focus group guides and all consent forms will be reviewed and approved by the contractor’s Institutional Review Board ([IRB]; Federal Wide Assurance #3331). In keeping with 45 CFR 46, Protection of Human Subjects, the NITT-HT procedures for data collection, consent, and data maintenance are formulated to protect respondents’ rights and the privacy of information collected. Strict procedures will be followed for protecting the privacy of respondents’ information and for obtaining their informed consent.
Data from the NITT-HT client interviews will be safeguarded in compliance with the Privacy Act of 1974 (5 U.S.C. 552a). The privacy of data records will be explained to all respondents during the consent process and in the consent forms.
Supplemental Youth and Young Adult Interview (SYAI): The client interviews, by necessity, will collect sensitive information about measures of key functional outcomes including mental health, trauma, psychotic experiences, self-efficacy, and perceptions of care as these are all outcomes of interest to SAMHSA and necessary to fully achieve the objectives of the NITT evaluation. All three of the SYAI – Baseline, 12- and 24-month follow-up interviews will be self-administered with audio capabilities on headphones to eliminate discomfort a client may feel in sharing sensitive information with learning laboratory staff. If these self-administered questions cause any distress for the client, the staff member overseeing the administration of the SYAI will connect them with someone from the grantee learning laboratory who they can speak with. Sensitive information of this nature is always regarded as private, and privacy for clients in federally assisted treatment programs is assured through strict adherence to Federal Regulation 42 CFR, Part 2. All SYAI will be conducted in a private space and the administrator will first review and answer any questions the client may have about consent for participation. Clients will be informed about the purpose of the data collection and that their participation is voluntary. They will be assured that they may stop taking the interview at any time without forfeiting the incentive and without penalty from the grantee or learning laboratory program. Informed consent will be submitted electronically by the client on the tablet before starting the interview. In addition, specific assurances will be provided to clients concerning the safety and protection of data collected from them. Clients’ names or other personally identifying information will not be linked to data collected.
Grantee Visit Document Review. A document review will be conducted on the files or charts of a subset of youth/young adults participating in the SYAI. Information extracted from client charts will be grantee and learning laboratory programmatic only; there will be no identifying or personal information extracted.
No questions of a sensitive nature will be asked on any of the following web-based, telephone or in-person data collection instruments:
Community Support for Transition Inventory (CSTI)
State Support for Transition Inventory (SSTI)
Collaborative Member Survey
Collaborative Self-Assessment
Project Director Web Survey
Core Staff Web Survey
Project Director Telephone Interview
Services & Supports Inventory
Grantee Site Visit In-person Interviews & Focus Group Guides
The estimated number of respondents, responses per respondent, and burdens hours are described for each NITT-HT data collection activity below. There are no direct costs to respondents other than their time to complete the instrument. Table 5a provides annual respondent, number of responses, burden, and cost details for each instrument, and Table 5b presents a summary of the instrument annualized burden hours. Respondent costs are calculated as total burden hours × the average hourly wage for associated job categories (as reported in the 2014 Occupational Employment Statistics [OES] by the Bureau of Labor Statistics [BLS]). The test procedures used to determine burden hours are detailed in Section B.4.
Community Support for Transition Inventory (CSTI). A maximum of 1,075 community leaders (15-25 community leaders per 43 learning laboratories) will complete the CSTI survey. The survey will be completed once during Year 2 and once during Year 4 of the grant period. The CSTI takes 24 minutes on average to complete per response; including time for reading the survey instructions and consent (3 minutes) and time to complete the web-based survey (21 minutes). This time represents actual averages taken across three randomly selected communities where the survey was previously fielded.
State Support for Transition Inventory (SSTI). A maximum of 425 state leadership members (20-25 state leaders per 17 grantees) will complete the SSTI survey. The survey will be completed once during Year 2 and once during Year 4 of the grant period. The SSTI takes 19 minutes on average to complete per response; including time for reading the survey instructions and consent (3 minutes) and time to complete the web-based survey (16 minutes). This time represents actual averages taken across three randomly selected communities where the survey was previously fielded.
Collaborative Member Survey. This survey will be completed by a subset of respondents to the CSTI and SSTI who participate in Advisory Teams. A maximum of 1,075 respondents (15-25 advisory team members per 43 learning laboratories) are expected to complete the web-based Collaborative Member Survey. The survey will be completed once during Year 3 and once during Year 5 of the grant period. The Collaborative Member Survey is estimated to take 15 minutes to complete per response. The estimated burden time is based on paper-and-pencil surveys completed by evaluation staff members who have experience working with grantees.
Collaborative Self-Assessment Survey. The Collaborative Self-Assessment Survey will be completed by 43 total respondents (one advisory team member per 43 learning laboratories). The Collaborative Self-Assessment Survey will be completed once in Year 3 and once in Year 5 of the grant period. The Collaborative Self-Assessment Survey is estimated to take 50 minutes to complete per response; this includes time to look up and compile information (10 minutes) and time to complete the web-survey (40 minutes). The estimated burden time is based on paper-and-pencil surveys completed by evaluation staff members who have experience working with grantees.
Project Director Web Survey. All NITT-HT grantees (n=17) are expected to complete the Project Director Web Survey. The survey will be completed once during each of Years 2, 3, and 4. The Project Director Web Survey is estimated to take 20 minutes to complete with one response per grantee. The estimated burden time is based on paper-and-pencil surveys completed by evaluation staff members who have experience working with grantee project directors.
Project Director Telephone Interview. All NITT-HT grantees (n=17) are expected to complete the Project Director Telephone Interview, once during each of Years 2, 3, and 4. The Project Director Telephone Interview is estimated to take 120 minutes to complete with one response per grantee; this includes time to look up and compile information (10 minutes) and time to complete the telephone interview (110 minutes). The estimated burden time is based on evaluation team’s expectations of the time required to conduct these interviews. Actual interview lengths may vary, depending on the learning laboratory, but they will not exceed the burden estimate.
Core Staff Web Survey. A maximum of 430 core staff (no more than 10 core staff per 43 learning laboratories) are expected to complete the Core Staff Survey once during the grant period. The Core Staff Web Survey is estimated to take 20 minutes to complete per response. The estimated burden time is based on paper-and-pencil surveys completed by evaluation staff members with grantee staff experience.
All NITT-HT grantees (n=17) will be visited once during the 5-year grant period. All activities associated with the grantee visit (i.e., interviews, focus groups, and document review) are described below.
Services & Support Inventory. A representative from each of the NITT-HT grantees (n=17) is expected to complete the Services & Support Inventory once during the grant period. The Services & Support Inventory is estimated to take 40 minutes to complete with one response per grantee; this includes time to look up and compile information (10 minutes) and time to complete the telephone interview (30 minutes). The estimated burden time is based on evaluation team’s expectations of the time required to conduct these interviews. Actual interview lengths may vary, depending on the learning laboratory, but they will not exceed the burden estimate.
Youth Coordinator In-Person Interview. A maximum of 129 staff are expected to participate in the Youth Coordinator Interview during the grantee visit, once during the grant period. This includes three staff members (one youth coordinator and up to two peer workers) per learning laboratory (n=43). The Youth Coordinator In-Person Interview is estimated to take 60 minutes to complete per response; the participant will not need to look up or compile any information for this interview, this time is to complete the in-person interview. The estimated burden time is based on evaluation team’s expectations of the time required to conduct these interviews. Actual interview lengths may vary, depending on the learning laboratory, but they will not exceed the burden estimate.
Provider In-Person Interview. A maximum of 85 key provider informants (five key providers per 17 grantees) will participate in the Provider In-Person Interview during the grantee visit, once during the grant period. The Provider Grantee Visit Interview is estimated to take 45 minutes to complete per response; the participant will not need to look up or compile any information for this interview, this time is to complete the in-person interview. The estimated burden time is based on evaluation team’s expectations of the time required to conduct these interviews. Actual interview lengths may vary, depending on the learning laboratory, but they will not exceed the burden estimate.
Stakeholder In-Person Interview. Approximately 51 community stakeholders total (3 stakeholders per 17 grantees) will participate in the Stakeholder In-Person Interview during the one grantee visit during the grant period. The Stakeholder In-Person Interview is estimated to take 45 minutes to complete per response; the participant will not need to look up or compile any information for this interview, this time is to complete the in-person interview. The estimated burden time is based on evaluation team’s expectations of the time required to conduct these interviews. Actual interview lengths may vary, depending on the learning laboratory, but they will not exceed the burden estimate.
Young Adult Focus Group. A maximum of 860 youth/young adult client participants (20 participants per 43 learning laboratories) will participate in the Young Adult Focus Group during the grantee visit, once during the grant period. Each focus group will take 105 minutes (an hour and forty-five minutes) to complete; this includes time for introductions, directions and informed consent (15 minutes), time for the focus group discussion (80 minutes), and time to complete a paper-and-pencil information form (10 minutes). The estimated burden time for the focus group discussion is based on evaluation staff member’s expectations of the time required to moderate this group discussion. Actual focus group may vary, depending on the group, but they will not exceed the burden estimate. The estimated burden time for the information form is based on paper-and-pencil forms completed by the evaluation team’s Young Adult Board (YAB) members with lived experiences.
Family/Adult Ally Focus Group. A maximum of 860 family/adult allies (20 participants per 43 learning laboratories) will participate in the Family/Adult Ally Focus Group once during the grant period. Each focus group will take 105 minutes (an hour and forty-five minutes) to complete; this includes time for introductions, directions and informed consent (15 minutes), time for the focus group discussion (80 minutes), and time to complete the paper-and-pencil information form (10 minutes). The estimated burden time for the focus group discussion is based on evaluation staff member’s expectations of the time required to moderate this group discussion. Actual interview lengths may vary, depending on the learning laboratory, but they will not exceed the burden estimate. The estimated burden time for the Information Form is based on paper-and-pencil forms completed by YAB members with lived experiences.
Document Review. All NITT-HT learning laboratories (n=43) are expected to make available the files/charts for a subset of youth/young adults participating in the SYAI for a Document Review. Staff hours to make these files/charts available is estimated to take 15 minutes to complete total; this includes time to look up and compile the files/charts (10 minutes) and time to provide them to the evaluation team members to conduct the document review (5 minutes). The estimated burden time is based on evaluation team members with grantee staff experience.
Multi-Media Project Young Adult Probes. A maximum of 510 youth/young adults are expected to participate in the Multi-Media Project Young Adult Probes in grantee Years 2, 3, and 4. This includes 30 youth/young adults per grantee (n=17). The Multi-Media Project Young Adult Probes is estimated to take 20 minutes to complete per response. The estimated burden time is based on evaluation team’s expectations of the time required to log information about their experiences using multi-media outlets. The actual length may vary, but should not exceed the burden estimate.
Supplemental Youth & Young Adult Interview (SYAI). A total of 1,530 youth/young adult clients (90 clients per 17 grantees) will complete the SYAI three times during the grant: at baseline, 12-month follow-up, and 24-month follow-up. The SYAI is estimated to take 40 minutes to complete per response; this includes time for reviewing instructions and informed consent (5 minutes) and time to complete the in-person interview (35 minutes). The estimated burden time is based on paper-and-pencil interviews completed by the YAB, young adults with lived experiences.
Table 5a. Annualized Data Collection Burden
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Burden Hours |
Average Hourly Wage |
Total Respondent Costa |
Community Support for Transition Inventory |
1,075 |
1 |
1,075 |
0.40 |
430 |
$32.56 |
$14,001 |
State Support for Transition Inventory |
425 |
1 |
425 |
0.32 |
136 |
$32.56 |
$4,428 |
Collaborative Member Survey |
1,075 |
1 |
1,075 |
0.25 |
269 |
$32.56 |
$8,759 |
Collaborative Self-Assessment Survey |
43 |
1 |
43 |
0.83 |
36 |
$32.56 |
$1,172 |
Project Director Web Survey |
17 |
1 |
17 |
0.33 |
6 |
$32.56 |
$195 |
Project Director Telephone Interview |
17 |
1 |
17 |
1.5 |
26 |
$32.56 |
$847 |
Core Staff Web Survey |
430 |
1 |
430 |
0.33 |
142 |
$22.03 |
$3,128 |
Grantee Visits |
|
|
|
|
|
|
|
Services & Supports Inventory |
17 |
1 |
17 |
0.67 |
11 |
$32.56 |
$358 |
Youth Coordinator In-Person Interview |
129 |
1 |
129 |
1 |
129 |
$15.32 |
$1,976 |
Provider In-Person Interview |
85 |
1 |
85 |
0.75 |
64 |
$35.59 |
$2,278 |
Stakeholder In-Person Interview |
51 |
1 |
51 |
0.75 |
38 |
$32.56 |
$1,237 |
Young Adult Focus Group |
860 |
1 |
860 |
1.75 |
1,505 |
$11.90 |
$17,910 |
Family/Adult Ally Focus Group |
860 |
1 |
860 |
1.75 |
1,505 |
$20.10 |
$30,251 |
Document Review |
43 |
1 |
43 |
0.25 |
11 |
$22.03 |
$242 |
Multi-Media Project Young Adult Probes |
510 |
1 |
510 |
0.33 |
168 |
$11.90 |
$1,999 |
Supplemental Youth & Young Adult Interview |
1,530 |
1 |
1,530 |
0.67 |
1,025 |
$11.90 |
$12,198 |
TOTAL |
5,522 |
|
7,167 |
|
5,501 |
|
$100,979 |
a Total respondent cost is calculated as total burden hours x average hourly wage.
Table 5b. Annualized Burden Summary
Instrument |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Total Burden Hours |
Community Support for Transition Inventory |
1,075 |
1 |
1,075 |
430 |
State Support for Transition Inventory |
425 |
1 |
425 |
136 |
Collaborative Member Survey |
1,075 |
1 |
1,075 |
269 |
Collaborative Self-Assessment Survey |
43 |
1 |
43 |
36 |
Project Director Web Survey |
17 |
1 |
17 |
6 |
Project Director Telephone Interview |
17 |
1 |
17 |
26 |
Core Staff Web Survey |
430 |
1 |
430 |
142 |
Grantee Visits |
|
|
|
|
Services & Supports Inventory |
17 |
1 |
17 |
11 |
Youth Coordinator In-Person Interview |
129 |
1 |
129 |
129 |
Provider In-Person Interview |
85 |
1 |
85 |
64 |
Stakeholder In-Person Interview |
51 |
1 |
51 |
38 |
Young Adult Focus Group |
860 |
1 |
860 |
1,505 |
Family/Adult Ally Focus Group |
860 |
1 |
860 |
1,505 |
Document Review |
43 |
1 |
43 |
11 |
Multi-Media Project Young Adult Probes |
510 |
1 |
510 |
168 |
Supplemental Youth & Young Adult Interview |
1,530 |
1 |
1,530 |
1,025 |
TOTAL |
5,522* |
|
7,167 |
5,501 |
*This is an unduplicated count of total respondents.
There are no respondent costs for capital or start-up or for operation or maintenance.
The annualized cost to the government is approximately $752,023.The estimated 5-year total cost to the government for the data collection is $3,760,117. This includes approximately $$740,166 per year (or $3,700,832 total) for developing the instruments; programming and maintaining the online data collection system; providing data collection training to grantees and learning laboratories; contractor labor for managing data collection; processing, cleaning, and housing data; and analyzing and reporting data. Approximately $11,857 per year (or $59,285 total) represents SAMHSA costs to manage/administer the data collection and analysis for 10% of one employee (GS-14-4, $118,570 annual salary).
This is a new data collection.
Table 6 outlines the key time points for the study and for the collection of information.
Table 6. Time Schedule for Data Collection
Activity |
Time Schedule |
Prepare for data collection, including programming web system |
October 2015-April 2016 |
Obtain OMB approval for data collection |
April 2016 |
Collect data |
April 2016-July 2019 |
Collect Time 1 Grantee PD Web Surveys |
April 2016-May 2016 |
Collect Time 1 Grantee PD Interviews |
May 2016-June 2016 |
Collect Time 1 CSTI/SSTI |
May 2016-July 2016 |
Collect SYAI Baseline Interviews |
June 2016-June 2017 |
Collect Multi-Media Young Adult Probe data |
August 2016 – February 2019 |
Collect Time 1 Collaborative Member Survey |
May 2017-July 2017 |
Collect Time 1 Collaborative Self-Assessment |
May 2017-July 2017 |
Collect Time 2 Grantee PD Web Surveys prior to first group of grantee visits (8 grantees total) |
November 2016-October 2017 |
Collect Time 2 Grantee PD Interviews prior to first group of grantee visits (8 grantees total) |
December 2016-November 2017 |
Conduct first group of Services and Supports Inventory prior to grantee visits (8 grantees total) |
January 2017-December 2017 |
Collect first group of Core Staff Web Surveys prior to grantee visits (8 grantees total) |
January 2017-December 2017 |
Conduct first group of grantee visits (8 grantees total) |
March 2017-February 2018 |
Collect SYAI 12 month Interviews |
June 2017-June 2018 |
Collect Time 2 Grantee PD Web Surveys prior to second group of grantee visits (9 grantees total) |
November 2017-October 2018 |
Collect Time 2 Grantee PD Interviews prior to second group of grantee visits (9 grantees total) |
December 2017-November 2018 |
Conduct second group of Services and Supports Inventory prior to grantee visits (9 grantees total) |
January 2018-December 2018 |
Collect second group of Core Staff Web Surveys prior to grantee visits (9 grantees total) |
January 2018-December 2018 |
Conduct second round of grantee visits (9 grantees total) |
March 2018-February 2019 |
Collect SYAI 24 month Interviews |
June 2018-June 2019 |
Collect Time 2 CSTI/SSTI (Time 2) |
May 2018-July 2018 |
Collect Time 2 Collaborative Member Survey |
May 2019-July 2019 |
Collect Time 2 Collaborative Self-Assessment |
May 2019-July 2019 |
Collect Time 3 Grantee PD Web Surveys |
May 2019-June 2019 |
Collect Time 3 Grantee PD Interviews |
June 2019-July 2019 |
Analyze data |
August 2016-January 2020 |
Disseminate
findings |
October 2016-February 2020 |
Note. CSTI, Community Support for Transitions Inventory; SSTI, State Support for Transition Inventory; OMB, Office of Management and Budget; PD, Project Director; SYAI, Supplemental Young Adult Interview.
The NITT-HT evaluation will help SAMHSA reach its diverse stakeholders through targeted products and innovative dissemination venues. The evaluation’s objective for all reports and dissemination products is to provide user-friendly documents and presentations that help SAMHSA successfully disseminate and explain the findings. The dissemination plan includes products in a variety of formats for a variety of target audiences. Audiences for these reports will include Congress, SAMHSA Centers, the evaluation’s SAMHSA Contracting Officer’s Representatives (CORs), NITT-HT grantees and participants, and the broader mental health and substance abuse fields (e.g., academia, researchers, policy-makers, providers). The NITT-HT evaluation recognizes that different audiences are best reached by different types of report formats. For example, reports to Congress will require materials that are concise but offer policy-relevant recommendations. Reports created for SAMHSA Centers and the CORs will require more in-depth information, such as substantive background and discussion sections, to supplement the analytic approach. Reports created for NITT-HT grantees will be concise handouts with helpful and easy-to-read graphics on performance data rather than lengthy text. The NITT-HT evaluation will develop an assortment of disseminations products, including short and long analytic reports, congressional briefings, annual evaluation reports, research and policy briefs, ad hoc analytic reports, journal articles, best practice summaries, and conference or other presentations.
Process evaluation analyses will occur in three phases, which collectively will be used to answer the following process questions:
What were the barriers/facilitators to state-/local-level collaboration, partnership development, and shared decision-making? How were they addressed?
What are effective awareness outreach and engagement strategies for youth/young adults ages 16 to 25 for behavioral health disorders?
What NITT-HT strategies were effective in developing, improving, and expanding services and supports for youth/young adults with serious mental health conditions?
What were effective NITT-HT strategies to increase access to care?
The practices used to address these NITT-HT objectives and their associated barriers and facilitators will first be characterized for each grantee and its learning laboratories. Then, “clusters” of learning laboratories will be identified using process variables (i.e., presence or absence of practices, barriers, facilitators, and their combinations). Finally, these process variables will be used to predict change in youth outcomes assessed in the NITT-HT outcome analyses. Thus, process analyses will be conducted in three phases: (1) single grantee case studies, describing process characteristics of each grantee and its learning laboratories, with reporting of preliminary feedback from these case study analyses to grantees; (2) cross-grantee/learning laboratory, multiple case study analyses, examining presence and levels of strategies, barriers, and facilitators at the learning laboratory level and identifying discrete clusters or subgroups of learning laboratories using these data; and (3) process-outcome analyses, including: (a) comparisons of youth outcomes by cluster (learning laboratory level, group-focused analysis); (b) examination of relationships between learning laboratories’ process variable scores and youth outcomes (learning laboratory level, variable-focused analysis); and (c) testing of relationships between scores from the process section of the youth supplementary interview and youth outcomes (mediational analysis).
Phase 1 analysis (single case study). In the first, single case study phase (P1), to be conducted following grantee visits in Years 3 and 4, the unique case of each NITT-HT grantee will be described, by (a) identifying common a priori or “template” themes related to practices, facilitators, and barriers derived from prior demonstration reports and literature on best practices for the population; (b) using an inductive, constant comparison approach to develop themes capturing unique observations of respondents on facilitators or barriers and/or unique practices; and (c) providing descriptive summaries of quantitative data on process dimensions related to collaborative functioning assessed in the local area and state (e.g., leadership, shared-decision making, management of conflict).
Phase 2 analysis (cross-grantee study). The Phase 2, cross grantee analysis (P2) will employ a mixed methods approach involving qualitative characterization of subgroups or clusters of grantees and comparison of these grantee clusters on dimensions of collaborative functioning.
General approach. In the final year, groups of grantees with shared practices, barriers, and facilitators will be characterized qualitatively, using axial coding (see following section). These groups will be compared on dimensions of collaborative functioning, assessed through advisory group measures (i.e., the web-based local and state collaborative rating form, and the annual team self-assessment). This comparison provides an important enhancement to the evaluation question, “What are barriers and facilitators to collaboration, partnership, and shared decision-making?”, by allowing identification of configurations of practices, barriers, and facilitators that were associated with better functioning of local and state advisory groups, the central vehicle for collaborative processes in NITT-HT projects.
Axial coding method. The qualitative method for characterizing clusters in the cross-grantee phase will involve axial coding, or identification of second-order relationships among practice, barrier, and facilitator codes identified in Phase 1. Once identified, these codes will be used to characterize NITT-HT activities generally (i.e., ways in which practices, barriers, and facilitators were consistently linked, irrespective of the grantee or learning laboratory involved). Then, codes limited to specific clusters of learning laboratories (rather than generally across the demonstration) will be identified. The minority of laboratories sharing a particular axial code will be identified as a cluster for that code. For example, one might find certain types of practices tend to be implemented by particular providers (e.g., evidence-based interventions focusing on specific problems or populations) and be associated with barriers found when implementing practices in isolated specialized settings (e.g., limited resources). Learning laboratories where this barrier was prominent could be classified as “specialist” laboratories. Other grantees might focus more on practices purveyed through multigrantee or statewide implementation efforts, and face barriers related to multigrantee implementation (e.g., difficulties in adapting practices to local conditions). These could be referred to as a “generalist” cluster. Note that this example is merely for illustration; the actual clusters will be identified through the previously described inductive coding process.
Phase 3: Process-outcome analyses. In the final phase (P3), also conducted in the final year, learning laboratory clusters will be used in process-outcome analyses described below in the Outcome Evaluation Analysis Section.
Youth and family voice (including Youth Voice Special Study). The approach to understanding infusion of youth and family voice throughout NITT-HT systems will be primarily qualitative, involving characterization of themes from youth and family focus groups on youth and family voice in services (service level), and youth and family voice in systems change and development (systems level). These themes will initially be characterized separately, both by grantee and also across grantees, to ensure that these important topics receive adequate emphasis in interpretation and reporting of process evaluation data. However, themes related to youth and family voice from focus groups will also be incorporated in analyses for Phases 1 through 3, because in addition to being of value in their own right, specific ways in which youth and family voice are present (or absent) in systems can facilitate or impede other NITT-HT objectives, including development of outreach and engagement, improved services, and reduction of barriers to access. The Youth and Family Voice analyses will address the following two evaluation questions:
How was youth/young adult voice integrated into aspects of NITT-HT program planning, implementation and local evaluation efforts?
How was family and adult ally voice integrated into aspects of NITT-HT program planning, implementation and local evaluation efforts?
The overall quantitative analysis plan involves examining variation in key NITT-HT outcome measures and, in some cases, cross-logic model processes by which HT achieved these outcomes. These analyses involve basic descriptive terms and approaches to modeling and describing variance components among key outcomes.
The NITT team has identified several features of the full evaluation design and evaluation questions that guided the selection of the analysis frameworks, including:
repeated measures (i.e., data collected at multiple time points);
data from youth/young adult nested within learning laboratories/grantees;
nonrandomized comparison groupings; and
nonrandom selection of intervention components that occurred in combination.
Each of these features led to the selection of a series of complex analysis frameworks to be used primarily in the later years of the evaluation which include:
multilevel latent growth models;
item response theory;
propensity score weighting; and
finite mixture modeling/latent class modeling.
A brief overview of each framework follows:
Multilevel latent growth models. Multilevel models in general are geared toward partitioning variability across multiple levels of aggregation, such as individual-level summary data collected repeatedly (as in the baseline, 12-month, and 24-month SYAI), and clustered within learning laboratories who are themselves clustered within NITT-HT grantees. Analyses can then incorporate predictors of changes in outcomes over time to assess which factors are related to increases in positive outcomes, decreases in negative outcomes, or both—both within and across grantees. When fit in the structural equation modeling framework, multiple trajectories (e.g., activities, outputs, outcomes) can be structured for multiple outcomes to test for mediation (Cheong, MacKinnon, & Khoo, 2003), which can include individual- , laboratory-, and/or grantee-level mediation effects.
Item response theory. Item Response Theory methods will be used to address issues regarding construct scaling. For constructs that reflect multiple-item scales, confirmatory factor analysis models will be employed to assess which items load on which factors. Factor/scale scores will then be derived via item response theory models, which weight each item according to how common (or rare) a response is and how correlated the item is with other items making up the factor.
Propensity scoring approaches. Propensity scoring is a statistical approach used to balance measured covariates that influence the probability of selection into two or more nonexperimental groups and also influence treatment outcomes (Rosenbaum & Rubin, 1983; Shadish, Cook, & Campbell, 2002; West, Biesanz, & Pitts, 2000). For the purposes of NITT-HT evaluation, the probability of individual communities being included as active intervention communities (versus nonrandomized units from the NSDUH comparison dataset) will be examined. After controlling for the propensity score, covariate distributions should be equal across conditions, which will mimic random assignment to the conditions of interest in the particular evaluation question. These scores can then be used to weight outcome analyses to produce unbiased estimates of the treatment effect (Harder, Stuart, & Anthony, 2010; McCaffrey, Ridgeway, & Morral, 2004; Rosenbaum & Rubin, 1983; Shadish, 2010).
Finite mixture modeling/latent class modeling. Whereas propensity scoring approaches are designed to optimize causal inferences when isolation of intervention types is of interest, latent class models would be used to cluster learning laboratories or grantees into groups on the basis of various combinations of intervention types. Class membership is a latent variable that distinguishes groups on the basis of estimated differences in either the probability of endorsement of an item (e.g., yes/no to a particular intervention domain) or mean differences on continuous measures (e.g., the intensity of implementation of a particular intervention domain). For example, among multiple service approaches, the latent class analysis will be used to determine a small number of common service activity patterns across subgroups of communities.
The approaches to be used in answering each of the four NITT-HT client-level outcome evaluation question are shown in Table 7.
To compare NITT-HT young adults with similarly aged young adults in NSDUH (18–25), propensity scoring, integrative data analysis (IDA) and weighted multilevel longitudinal growth models (MLLGM) to address most of the evaluation questions will be used. To estimate the impact of NITT-HT on individual-level outcomes, the analysis team will first construct propensity scoring models of the probability that a “grantee” was an NITT-HT “grantee” or a comparison “site” (i.e., from NSDUH). In a separate process, the analysis team will use IDA to estimate Item Response Theory (IRT) scale scores for measures of mental health and substance use outcomes across NITT-HT, NSDUH, and NHIS. After scoring of the mental health and substance use indicators, the IDA/IRT scores will be modeled under the primary framework of MLLGM. A series of weighted MLLGM models will be fit for each focal outcome to determine whether NITT-HT has an impact on changes over time in key outcomes. In addition to the incorporation of propensity score weighting, the analyses will be structured to include differences in variance components across the NITT-HT and comparison groups. A series of longitudinal mediation models will also be structured where NITT-HT status is the predictor, intervening variables serve as mediators, and mental health/substance use are primary outcomes.
Table 7. NITT-HT Client-Level Outcome Evaluation Questions and Analytic Techniques
Evaluation Question |
Multi-Level (Growth) Modeling |
Item Response Theory |
Propensity Score Weighting |
Latent Class Analysis |
To what extent did NITT-HT identify and serve young people who would have otherwise fallen through the cracks? |
X |
|
|
|
What are the outcomes associated with NITT-HT in the areas of education, employment, housing, primary care, and mental health? |
X |
|
X |
X |
What is the effect of the NITT-HT intervention compared to a national sample of similar young adults not participating in the NITT-HT intervention? |
X |
X |
X |
X |
There are three outcome evaluation questions designed to examine systems-level change in NITT-HT grantee communities. These are:
Did NITT-HT lead to enhanced collaboration and coordination across youth-serving systems for youth/young adults ages 16 to 25?
To what extent were services and supports for youth/young adults with serious mental health conditions developed, improved and expanded as a result of NITT-HT?
What were the effects on service capacity and access to care in communities implementing NITT-HT?
These evaluation questions will be addressed primarily through the analysis of the CSTI and SSTI web surveys. This evaluation will use multilevel models to examine the effects of individual-level variables and the data collection grantee. Since the CSTI and SSTI data have a hierarchical structure with respondents nested within grantees, multilevel techniques will be used. The analysis will use confirmatory factor analyses to confirm the factor structure and compute factor scores. Subsequently, null models will be created to examine between-group effects for each dependent variable (entire scale scores and individual theme scores). Covariates (e.g., race and role in the project) and the predictor of interest, time, as well as possible interactions, will be added to these null models. More detail on each of these steps is provided below. Change over time for the two subscales focused on collaboration and coordination, as well as the overall CSTI/SSTI scores, will be most important in addressing this particular research question.
Data will be cleaned and multiple imputations will be performed to handle missing data. (Respondents with extensive missing data will be dropped from the analyses.) Cronbach’s alpha for the overall assessments and the subscales will be computed for both time points. At T1, overall, subscale, and item means for the entire dataset and for the individual communities/states will be computed. Each state/community will receive a report describing its own outcomes. At T2, change over time will be assessed using multilevel modeling. The first step in this process will be to assess the model fit using confirmatory factory analyses and examining fit statistics for each assessment. Assuming the model shows acceptable fit, factor scores for the subscales and the overall scales will be determined for the imputed dataset and uses in the subsequent analyses.
For the multilevel analyses, null models will be computed for the community and state data. Models will also be computed using covariates and, most importantly, time. Models with interaction terms will also be examined. Intercepts and slopes for time will be calculated for subscales and the entire CSTI/SSTI using factor scores and imputed data.
Process-Outcome Analyses. Learning laboratory clusters developed with information gathered during the process evaluation will be used in process-outcome analyses. Individual subgroup types (learning laboratory clusters) will be entered individually as a dummy coded predictor variable in multilevel (youths nested within grantees) models of change in youth/young adult functioning over time. In mediational models, grantee subgroup membership will be used to predict change in mediator variables (e.g., youth perceptions of person-centered process) and assess the degree to which relationships between subgroup membership and outcomes can be explained by these youth-level process mediators. These analyses will help answer process evaluation questions related to strategies or practices, including the “what are effective strategies” questions in awareness, outreach and engagement, in developing, improving and expanding services and supports, and in increasing access to care.
OMB approval expiration dates will be displayed.
There are no exceptions to the certification statement. The certifications are included in this submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Snaauw, Roxanne |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |