OMB Ss-a 3-30-15

National System of Care Expansion Evaluation

Supporting Statement

1. JUSTIFICATION

1. CIRCUMSTANCES OF INFORMATION COLLECTION

The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Mental Health Services (CMHS) is requesting approval from the Office of Management and Budget (OMB) for the collection of data from SAMHSA’s Comprehensive Community Mental Health Services for Children and their Families Program also known as the Children’s Mental Health Initiative (referred to hereafter as CMHI) System of Care (SOC) Expansion Planning Grants and the SOC Expansion Implementation Cooperative Agreements (hereafter referred to as implementation grants). This request represents a proposed new data collection effort for 8 instruments.

The proposed National SOC Expansion Evaluation described here reflects the transition of CMHI from an innovative community-based demonstration program to one that is expanding its scope or geographic reach to larger regions such as states, multi-county areas, tribes and territories (hereafter referred to as jurisdictions). This evolution reflects SAMHSA’s Theory of Change Model that takes the development of an intervention through the phases of innovation, translation, dissemination, implementation and wide scale adoption¹. This initiative includes two grant funding efforts: SOC Expansion Planning grants funded for one year, and SOC Expansion Implementation grants funded for four years.

In FY 2011, in order to stimulate wide scale adoption, SAMHSA began awarding one-year SOC Expansion Planning grants to jurisdictions to develop comprehensive strategic plans for providing and expanding services provided by the SOC. These planning grants were designed to help jurisdictions develop and complete comprehensive short- and long-term strategic plans with action steps to improve, implement, expand and sustain SOCs across their jurisdiction. Following the first 11 planning grants funded in FY 2013 and 9 funded in FY 2014, SAMHSA funds 14 new planning grants each year.

The four-year SOC Expansion Implementation grants are intended to help jurisdictions to plan and carry out implementation of SOCs jurisdiction-wide. Following the first 15 expansion implementation grants funded in FY 2013 and 22 funded in FY 2014, SAMHSA funds 23 new expansion implementation grants each year.

1. Background

It is critical to address the mental health (MH) needs of children, youth, and young adults along with their families and caregivers. It is estimated that more than 20 percent of children and youth under the age of 18 have a serious MH condition either currently or have had one at some point during their lifetime. Approximately eleven percent of children have significant functional impairments that limit their participation in typical daily activities of childhood (Merikangas, et al., 2010). The Centers for Disease Control and Prevention (2012) indicates that suicide is the third leading cause of death for youth and young adults age 15 to 24 years, and approximately 16 percent of youth in grades 9 through 12 had seriously considered suicide in the 12 months preceding a nationally representative survey². MH conditions impact children and youth of all ages throughout their development, and often negatively affect their functioning at home, in school, and in the community. Given the scope and complexity of the challenges faced by children, youth, and young adults with serious mental health challenges, many are involved with multiple child-serving systems, including MH, substance use, education, juvenile justice, child welfare, and primary care. To address the needs of children and youth with serious MH needs, caregivers and other family members are often faced with the challenge of navigating multiple child-serving systems.

The SOC model was developed in response to the need to improve the organization, coordination and delivery of children’s MH services and to improve clinical and functional outcomes of children, youth, and young adults with significant MH needs. SOC is an organizational model that involves collaboration across agencies that serve children, youth and families to provide an array of effective, community-based, culturally- and linguistically-appropriate services and supports for children, youth, and young adults with or at risk for behavioral health challenges and their families (Stroul, Blau, & Friedman, 2010).

Although the SOC approach continues to evolve and adapt to the needs of children, youth, and families, the Stroul, Blau, and Friedman issue brief cited above lists the core values of a SOC as being:

• Family-driven and youth-guided, with the strengths and needs of the child and family determining the types and mix of services and supports provided;

• Community-based, with the locus of services as well as system management resting within a supportive, adaptive infrastructure of structures, processes, and relationships at the community level; and

• Culturally and linguistically competent, with agencies, programs, and services that reflect the cultural, racial, ethnic, and linguistic differences of the populations they serve to facilitate access to and utilization of appropriate services and supports and to eliminate disparities in care.

The purpose of the original CMHI community grantees was to provide Federal support through grants to States, political subdivisions within States, the District of Columbia, tribal areas, and territories to develop integrated home and community-based systems and supports for children and youth with severe emotional disturbances (SED) and their families (Huang et al., 2005). CMHI was shaped by the Child and Adolescent Service System Program (CASSP), which was implemented in 1984 to assist States and communities in building a comprehensive, community-based SOC. CMHI was funded as a Federal demonstration grant program at the initial level of $4.9 million, and 22 communities were awarded grants through CMHI in 1993 to 1994. In the last 20 years, CMHI grants have funded children’s MH services in all 50 states, Puerto Rico, Guam, and American Indian/Alaska Native territories and authorities.

Since its inception, CMHI has served more than 103,000 children and youth and their families, who have shown a variety of clinical and functional improvements during their involvement with CMHI. Examples of clinical and functional improvements have included increases in behavioral and emotional strengths, reductions in suicide attempts, improvements in academic outcomes, decreased criminal justice involvement, reductions in reliance on inpatient care, and more stable living situations (Annual Report to Congress, 2011).

The goals of the CMHI program are to:

• Expand jurisdictions’ capacity to serve children and adolescents with SED and their families;

• Provide a broad array of accessible, clinically effective and fiscally-accountable services, treatments and supports;

• Serve as a catalyst for broad-based, sustainable systemic change inclusive of policy reform and infrastructure development nationwide;

• Create a care management team with an individualized service plan for each child;

• Deliver culturally and linguistically competent services with special emphasis on racial, ethnic, linguistically diverse and other underrepresented, underserved or emergent cultural groups; and

• Encourage and facilitate full participation of families and youth in planning, evaluation and sustainability of local services and supports and in overall system transformation activities.

The National SOC Expansion Evaluation (hereafter referred to as the Evaluation) is driven by the SOC program theory model (illustrated in the Evaluation logic model (see Attachment 1).

b. The Need for Evaluation

Section 565(c)(1) of the Public Health Service Act of Public Law 102-321 mandates annual evaluation activities of SAMHSA programs. Basic requirements are the documentation of: characteristics of the children and families served by the grant program; the type and amount of services they receive; how these services are coordinated; and the associated costs. Equally important is the need to assess whether the program was implemented and the services were experienced as intended as well as the geographic distribution of providers and clients. It is also critical to assess whether the children served by the program experience improvement in clinical and functional outcomes, whether family life is improved, and whether improvements endure over time.

Further evaluation requirements under Sections 564(f) and 565(c)2 of PL102-321 include:

• Annual reports to the Secretary of Health and Human Services (HHS) that include a description of the number of children served, child demographic characteristics, types and costs of services provided, availability and use of third-party reimbursements, estimates of the unmet need for services within grant jurisdictions, how the grant was expended to establish a jurisdiction-wide SOC, and other information as required by the Secretary.

• Annual reports to Congress that provide information on longitudinal evaluations of outcomes of services provided by the funded SOC, the effect of activities conducted under funded SOC on the utilization of hospital and other institutional settings, barriers to the achievements of establishing interagency collaboration within the SOC, and parental assessment of the effectiveness of the SOC.

3. Clearance Request

This submission requests OMB clearance for (1) data collection to evaluate the CMHI Expansion Planning and Implementation grants and (2) the estimated burden for collecting data under this protocol. The request estimates burden for data collection in a total of 51 planning grants and 106 implementation grant jurisdictions.

3. Overview of the Proposed Evaluation

In this section, we summarize the Evaluation design, data collection activities, and evaluation questions to provide to provide an overview of the Evaluation. The primary goal of the National SOC Expansion Evaluation is to describe efforts to plan, implement and expand systems of care. CMHS will also examine how implementation is related to child, youth and family outcomes. The Evaluation will make use of an array of quantitative and qualitative data and analytic strategies, while attending carefully to the need to minimize grantee burden and maximize data quality.

CMHS is planning a longitudinal, multi-level, cross-site evaluation that incorporates assessment at the jurisdiction, local system, and child/youth and family levels. Definitions and evaluation aims for each level are presented in Table 1.

Table 1. Definition and aims of the three levels of the Evaluation

Level	Definition	Aims
Jurisdiction	Grantee: usually a state but can be another broad area such as a multi-county region or tribal area	Describe how the SOC is organized and financed Identify and describe mechanisms and strategies to implement and expand the SOC Assess SOC development over time Depict inter-agency linkages at top administrative levels and how they change over time
Local System	Counties, municipalities, multi-city areas within jurisdictions that organize to provide services directly to children and families	Explore funding mechanisms and arrangements Describe geographic reach of expansion efforts Depict local inter-agency linkages at service level and how they change over time Assess local SOC implementation and expansion over time
Child & Family	Children, youth, and young adults who access services, navigate multiple agencies and organizations, and receive direct services (aka “clients”) ages 5-21 and their families	Describe geographic spread of population served Describe the children, youth, young adults and families served, and measure their outcomes over time Describe services used Assess satisfaction with services

Jurisdictions should support good SOC practice at the local system level, and good SOC practice should lead to improved outcomes for children and families. It is widely acknowledged that broad system change at the jurisdictional level alone cannot be expected to result directly in differences in child and family outcomes (Bickman & Heflinger, 1995). It is also noted that local systems face greater difficulty enacting change when they do not have sufficient support at the jurisdiction level. For example, Fixsen, Blase, Metz, & Van Dyke (2013) emphasized that State-level support is necessary for local systems to provide faithfully implemented evidence-based treatments (EBTs). Therefore, the Evaluation must examine factors at all three levels that facilitate or impede success.

The data collection for the Evaluation is most easily described in terms of four core components. Each component is comprised of a set of analyses that cluster together substantively, though some span more than one level.

The four core components with their corresponding data collection activities are as follows:

1. The Implementation Assessment will document the development and expansion of the SOC, with a focus on expansion planning.

Data collection activities: Stakeholder Interviews, Self-Assessment of Implementation Survey (SAIS), and the SOC Expansion Assessment (SOCEA) to measure strategies and mechanisms implemented.

NOTE. This is the only evaluation component that collects data from planning grantees (in addition to implementation grantees). Planning grantee participation will be sought for the Stakeholder Interviews and the SAIS, but not the SOCEA.

2. The Network Analysis of jurisdictions and local systems will describe inter-sector collaboration. The Geographic Information System (GIS) Component will measure the geographic coverage of the SOC.

Data collection activities: Network Analysis Surveys at the jurisdiction and local system levels; work addresses of attendees at meetings and training sessions, and other

implementation and planning events at the jurisdiction and local system levels; and census block group designations associated with home addresses of clients receiving services.

3. Financial Mapping involves a biennial review (i.e., twice per grantee) of implementation grantees’ progress in utilizing multiple funding sources and developing financial sustainability to support the SOC. The Benchmark Component will compare relative rates of access, utilization, and expenditures for children’s MH services.

Data collection activities: Financial Mapping Interview; and Benchmark Tool (information requests from financial administrators and personnel working with the Medicaid Agency and the MH Authority reporting and payment systems).

4. The Child and Family Outcome Component will collect demographic and background information, and longitudinal data on child clinical and functional outcomes, family outcomes, and child and family satisfaction with services.

Data collection activity: The child and family outcome instruments include the Columbia Impairment Scale, Pediatric Symptom Checklist-17, and the shortened Caregiver Strain Questionnaire. These instruments will be added to the CDP Client-Level Services Measures for Discretionary Programs, CMHS PROGRAM ONLY instrument within the program-specific designated Section H..

Evaluation activities and their timing will differ by level of the Evaluation (i.e., jurisdiction, local system, or child/youth and family) and type of grant (planning or implementation). Table 2 summarizes the planned evaluation questions by level of the Evaluation, data collection activities, and grant cohort. In that their funding period is limited to one year and are not expected to deliver services during their funding period, planning grantees will participate in fewer aspects of the Evaluation. In addition, respondents for most data collection activities will be stakeholders involved with grantee planning and expansion efforts. Child and family outcome instruments items and a portion of the SOCEA will include client (i.e., served children, youth, and young adults) and caregiver respondents.

Table 2. Evaluation levels, data collection activities, and questions by grant cohort

Evaluation Levels, Data collection activities, & Questions	*Grant Type
Jurisdiction Level
Stakeholder interviews	Planning and implementation grants
Q1. What are the stated goals of the funded planning and/or implementation grants? Q2. What strategies and mechanisms did grantees plan to meet those goals? Q3. In the process of implementing and expanding the SOC, what factors facilitated and what factors impeded implementation? Q4. To what extent are SOC values embodied in the planning and implementation efforts?
Self-Assessment of Implementation Survey (SAIS)	Planning and implementation grants
Q1. What mechanisms and strategies are utilized to implement and expand the SOC? Q2. For each strategy/ mechanism, what is the current stage of implementation (e.g., fully implemented, partially implemented, initial implementation)? Q3. To what extent do SOCs develop over time? Q4. What facilitators and impediments to SOC expansion do jurisdictions report?
Network Analysis Survey: Jurisdiction	Implementation grants
Q1. Are all key jurisdiction-level agencies/organizations collaborating to implement and expand the SOC? Q2. What is the nature of the collaboration among child-serving jurisdiction-level agencies/organizations? Q3. How cohesive or fragmented is the overall system at the jurisdiction level?
GIS Component: Jurisdiction level	Implementation grants
Q1. To what extent to are administrators and policymakers from partner agencies and organizations spread across the jurisdiction? Q2. Do some areas of the jurisdiction appear to be disproportionately represented (compared to population) in SOC expansion efforts?
Financial Mapping Interview	Implementation grants
Q1. What funding sources support the SOC expansion efforts and how do they change over time?
Benchmark Tool	Implementation grants
Q1. How are children’s MH funds used and how do they change in all or selected jurisdictions?
Local System Level
SOCEA	Implementation grants: 1 local system per grant
Q1. What strategies and mechanisms do local systems employ to implement and expand the SOC? Q2. How fully implemented are those strategies? Q3. To what extent does SOC implementation and expansion efforts embody SOC principles? Q4. What barriers impede implementation and expansion efforts? What factors facilitate implementation and expansion?
Network Analysis Survey: Local service system level	Implementation grants: 1 local system per grant
Q1. Which agencies and organizations are collaborating to implement and expand the local SOC? Are all key child-serving agencies involved? Q2. What is the nature of the collaboration among child-serving agencies and organizations at the local system level? Q3. How cohesive or fragmented is the local service system?
GIS Component: Local level	Implementation grants
Q1. To what extent are agencies, organizations, and service providers involved in local service system implementation and expansion efforts spread proportionately throughout the local catchment area?
Child and Family Level
Child and family outcome instruments	Implementation grants: 1 local system per grant
Q1. What are the characteristics (i.e., demographics, family attributes, and clinical features) of children, youth, and families served through the CMHI SOCs? Q2. Do children/youth and families served experience improvements in outcomes? Q3. To what extent do youth and families appraise their service experiences as consistent with SOC principles? Do they report being satisfied with the services they receive?
Cross-Level Evaluation Questions
Q1. To what extent is jurisdiction-level implementation quality related to: (1) changes in local SOC expansion assessment scores; and (2) changes in local system network integration? Q2. To what extent is local service system of care development related to improved youth and family functioning?

NOTE. *All data collection activities are conducted with implementation grants. Only the first two data collection activities listed, Stakeholder interviews and SAIS, also include planning grants.

2. PURPOSE AND USE OF THE INFORMATION

At its core, the purpose of the Evaluation is to assess the success of the SOC planning and implementation grant initiatives. This section describes how, and for what purpose, the information collected will be used by SAMHSA, CMHS, grantees, and the practice community.

To justify continued support for the SOC expansion grants, SAMHSA/CMHS requires evidence that grantees can support SOCs that are effective, cost-effective and sustainable in communities throughout the country. SAMHSA/CMHS can use Evaluation findings to examine: 1) the extent to which the program results in true expansion and sustainability of SOC service delivery; 2) which mechanisms and strategies are most effective for realizing broad program goals; and 3) whether the program results in improvement in child and family outcomes.

SAMHSA/CMHS

Evaluation findings will be useful to SAMHSA, CMHS directors, and Grant Project Officers (GPOs) by: (1) fulfilling the program’s legislatively mandated requirements to evaluate its programs; (2) supporting several of SAMHSA’s Strategic Initiatives; and (3) providing essential program management and development information to CMHS leadership.

Legislative Requirements. As described under The Need for Evaluation (Section A.1b), the Evaluation was designed to respond to the legislatively-mandated requirement to evaluate the CMHI and to report program evaluation findings in annual reports to Congress and to the Secretary of HHS. Table 3 maps proposed data collection activities to public law and to required activities delineated in the implementation grant request for application (RFA) (based on the FY 2013 application). In addition, the Evaluation addresses the RFA requirement that grantees conduct Data Collection & Performance Measurement as well as Performance Assessment.

Table 3. Purpose of data collection activities relative to public law and activities required of grantees

Data collection	Public Health Services Act Requirements	Implementation Grant RFA Required Activities
Stakeholder interviews	562(a)(3): Seek collaboration among all public human service agencies in the community including MH, educational, child welfare, and juvenile justice services. 562(2)(a-e): Provide culturally appropriate services within the SOC without discrimination and in the least restrictive, most normative environment. Provide outreach regarding available services and identify children with Serious Emotional Disturbance (SED) early.	Use SOC values throughout planning and implementation including meaningful involvement by families and youth in governance and planning and implementation activities, establishing policies, administrative, and regulatory structures that support ongoing SOC implementation, provisions to ensure that SOC services, policies and programs are culturally and linguistically accessible.
Network Analysis Surveys	562(a)(3): as cited for Stakeholder interviews	Collaborate across child serving agencies and among critical providers of programs including those serving youth nearing adulthood.
Self-Assmt. of Implementation Survey (SAIS) & GIS	564(b): Plan for the development of a jurisdiction-wide SOC for community based services for children with an SED…and address current gaps in community services.	Comply with HHS action plan to reduce racial/ethnic health disparities. Program Purpose: Expand services across geographic areas and population groups within the jurisdiction.
Financial Mapping Interview Benchmark Tool	564(f) & 565(c): Submit report to Secretary describing types and costs of services provided, availability and use of third-party reimbursements, estimates of unmet service needs, and the effect of activities on hospital/institution utilization rates.	Develop financing approaches that promote a cross-agency service delivery system, create flexible funds, and develop fiscally accountable approaches to care review. Seek third-party reimbursement.

Data collection	Public Health Services Act Requirements	Implementation Grant RFA Required Activities
Systems of Care Expansion Assessment (SOCEA)	564(f) & 565(c)(1): Assess barriers  and achievements resulting from interagency collaboration in providing community-based services to children with a SED. Submit report to the Secretary assessing how the grant worked to establish a jurisdiction-wide SOC. 562(e)(1)(A): Child services provided through the system are coordinated and child’s needs are periodically reassessed. 562(e)(2)(A): Services are provided in the cultural context that is most appropriate for the child and family involved. 565(c)(1): Parental assessment of the SOC’s effectiveness.	Deliver SOC services within a family driven, youth guided framework, with families and youth as partners in planning and implementing activities, and with culturally/linguistically competent and evidence supported services.
Common Data Platform (CDP)	565(c)(1): Submit a report to the Secretary describing grant activities and the number and demographics of children served by the SOC. Assess the SOC’s effectiveness, including longitudinal service outcome assessments and parents’ assessment of SOC effectiveness.	Serve children with a SED; comply with the HHS Action Plan to Reduce Racial and Ethnic Disparities Data Collection & Performance Measurement: Assess longitudinal outcomes.

Note. Public Law refers to Public Health Service Act Title V, Part E Public Law 102-321, Section 561-565, 42 U.S.C. 290ff-4. Assmt. = Assessment.

SAMHSA’s Strategic Initiatives. In 2010, SAMHSA identified eight Strategic Initiatives with input from stakeholders including Federal, State and local leaders; constituency groups; advisory council members; members of Congress; people in recovery; and family members. These initiatives are designed to focus SAMHSA’s work on improving lives and capitalizing on emerging opportunities. The Evaluation is relevant to the following strategic initiatives in the following ways:

• Trauma and Justice, by assessing client outcomes and services available and provided to individuals who are (1) in need of trauma-informed services and (2) involved with or at-risk for involvement with the juvenile or criminal justice systems;

• Recovery Support for individuals recovering from MH and substance use disorders. Evaluation findings will show SAMHSA the extent to which it is engaging consumers and their families in self-directed care, shared decision-making, and person-centered planning;

• Health Reform, which emphasizes the need for integrated and coordinated care such as that promoted by the SOC model and assessed by the Evaluation; and

• Data, Outcomes and Quality, as Evaluation findings are intended to help improve the quality of SAMHSA’s CMHI program.

CMHS Leadership. SAMHSA/CMHS directors and GPOs can use Evaluation findings to address program management priorities including accountability (i.e., legislative requirements, as described in Section 1.b), program and policy planning, and program justification. For example, Evaluation findings can be used by CMHS leadership to:

• Monitor the progress of funded activities, which is essential for program monitoring, providing program TA, and program justification.

• Inform both intra- and interagency program and policy planning.

• Develop policies and provide guidance regarding SOC development.

• Support TA activities to help grantees best meet program goals.

• Support the many partners that work on CMHI in collaboration with CMHS, including the National Federation of Families for Children’s MH and Youth M.O.V.E. National in their efforts to help build SOC for children's MH services.

Some Evaluation findings will be of use to both CMHS leadership and grantees such as (but not limited to) the following:

• Illustrating the development of SOCs as they move toward offering integrated and comprehensive services;

• Providing detailed information on how to successfully bring SOCs to scale and sustain them;

• Identify funding sources used by states to sustain or expand SOCs services;

• Describing experiences and implementation practices (across all grantees);

• Identifying best practices and effective strategies;

• Understanding barriers and facilitators to successful implementation;

• Comparing access, utilization and expenditure patterns for children’s MH services across states and other types of jurisdictions;

• Documenting savings from reducing high-cost, out-of-home services that illustrate the business case for SOCs;

• Showing whether there are observable differences in child and family outcomes that can be plausibly linked to the SOC approach; and

• Describing how children and families experience the service system and how they use services and supports (i.e., utilization patterns).

Grantees and the Practice Community

Grantees and the practice community can use Evaluation findings to:

• Improve the implementation of their SOC and achieve the goals of the CMHI;

• Improve the quality of the services they provide;

• Identify additional opportunities for financing SOC services and obtaining matching funds and adopt the financing strategies and funding sources associated with the most successful implementations of SOC services, all with the goal of SOC sustainability;

• Identify opportunities to further reduce the use of restrictive and expensive out-of-home services by strengthening SOC and community-based MH care;

• Learn what barriers to treatment and other essential services children or youth and their families perceive and work to eliminate such barriers;

• Learn whether families experience services as the grantees intended and identify their programs’ strengths and weaknesses;

• Help identify gaps in system development and barriers to collaboration;

• More effectively allocate personnel and funding and prioritize activities; and

• Provide summary reports to their local steering committees or other advisory boards, support statewide expansion efforts, develop interagency partnerships, and obtain resources to sustain systems with interagency agreements.

3. USE OF INFORMATION TECHNOLOGY

Web-Based CMHI Portal

A web-based CMHI portal will be developed to collect and manage all data obtained and submitted by grantees at the jurisdiction, local system, and child and family levels. The CMHI portal will be designed to receive jurisdiction- and local system-level data. For the child and family outcome component, the CMHI portal will incorporate the Client-Level Services Measures for Discretionary Programs, CMHS PROGRAM ONLY from the web-based Common Data Platform (CDP)³ data collection and reporting system and will manage data at the client level. The CDP tool will operate under the identifier, CDP, and will be accessed and utilized via the CMHI portal.

The use of web-based surveys and forms decreases respondent burden, as compared to that required for alternative methods, such as a paper format, by allowing for direct transmission of the survey or form. Respondents can complete the survey at a time and location that is convenient for them. In addition, the data entry and quality control mechanisms built into the web-based format reduce errors that might otherwise require follow-up, thus reducing burden compared to that required for a hardcopy data collection.

Finally, SAMHSA and its contractors, together termed the National Evaluation Team (NET), strive to ensure that all web-based solutions are fully compliant with Section 508 of the Rehabilitation Act. This includes ensuring that all posted documents are compliant or have a compliant alternative. The NET utilizes Adobe products that are capable of producing compliant PDF files per the SAMHSA-recommended process. The NET has a thorough knowledge of Section 508 standards and employs accessibility specialists with experience in Section 508 compliance verification, including assessment with a variety of assistive technologies, including screen readers, screen magnifiers, and voice recognition software.

All Stakeholder Interviews and SOCEA data collection will be conducted by telephone, Skype, video-conferencing, etc. The Network Analysis Survey and SAIS will be conducted online. Child and Family Outcome Study data will be collected at the service site by local staff. Therefore, there is no need to use distance data collection methods. However, we anticipate that many communities will use the CDP computer-assisted personal interview software to collect data.

Common Data Platform (CDP).  Instruments for the CMHI Child and Family Outcome Component will be inserted into in Section H of the CDP and data will be entered via the web-based CMHI Portal. This OMB submission includes the CDP only insofar as it involves (1) Evaluation-related additions to CDP Section H; (2) for clients age 11 to 17, both the youth and caregiver will be asked to complete the CDP tool, whereas SAMHSA grant requirements include completion of the CDP by either the youth or the caregiver (i.e., overall grantee burden increases); and (3) collecting the CMHI Evaluation version of Section K (Services Received).  The CDP is described here only to provide context and a more comprehensive description of the Evaluation.

In compliance with the Government Performance Reporting Act (GPRA), SAMHSA requires all implementation grantees to provide National Outcome Measures (NOMs) reported through the CDP for each client served. SAMHSA’s CDP system is managed and operated through a separate contract with a different contractor.

CDP is an integrated, Internet-based data input, management, and dissemination system, seeking OMB approval. CDP will allow systematic data input, immediate validation to identify data input flaws, and monitoring of data entry and evaluation in real time. It will reduce processing time and provide the capability of creating interactive reports. The CDP will be secure and will provide of different levels of password-protected access to grant and program data. Within CDP, centralized data management systems will facilitate automatic notifications to alert site evaluators when an interview is due, thereby eliminating the need for site-level duplication of effort and expense in the design of local tracking materials.

• Grantee Training and TA. Grantees are trained and receive TA on CDP data collection, data entry, and use under the CDP GDTA contract.

• Data Input. Child and family data from all clients served by grantees will be entered directly into CDP. CDP is designed to be administered, data entered, and used by intake staff or case managers often located at various agencies rather than at a central evaluation office. (See Assurance of Confidentiality, Section 10.) The primary goal of this web-based data entry is to maximize the capture of descriptive information on all children served in SOC programs while minimizing burden.

• Data Monitoring, Management and Dissemination. CDP will monitor data quality and provide feedback to grantees. Basic validations are completed during the data entry process. In addition, every month missing data reports will be provided to grantees that detail any potential data errors or issues. Reporting features will support sites’ abilities to use their data for quality assurance monitoring and system improvement purposes. The CDP will make grantee-specific data available to grantees to help meet their local evaluation needs. Grantees will be able to edit their data and will immediately be able to see their data aggregated in reports or download it.

The federally-required descriptive information includes the number of persons served by age, gender, race and ethnicity. This information is gathered using the Client-Level Services Measures for Discretionary Programs, CMHS PROGRAM ONLY. Other information collected by CDP includes the following:

• Demographic information

• Education level and attendance

• Employment

• Military status of the client and/or family members

• Family structure and living situation

• Housing stability

• Social connectedness

• Overall physical health

• Overall MH

• Substance use

• Arrests

Augmentations to CDP (Items added by Evaluation). The Client-Level Services Measures for Discretionary Programs, CMHS PROGRAM ONLY CDP instrument includes a blank, place-holding section (Section H) where SAMHSA can add items to assess SAMHSA grant initiatives. This NET proposes to add a limited number of items to the CDP’s Section H for the purpose of this Evaluation.

Other Data Collected via the CMHI Portal. The Self-Assessment of Implementation Survey (SAIS) and the Network Analysis Surveys (at the jurisdiction and local levels) will be completed via the CMHI portal.

Training and TA

The NET will provide training and TA support to sites to facilitate implementation of the Evaluation protocol and the use of evaluation results at the site level. Although site personnel will be trained on the CDP tool under a different contract, the NET will use webinars and TA to train grantees in the Section H component specific to the Evaluation. In addition, a site liaison will be assigned to every site providing SOC services that is also collecting CDP data. To help ensure that data are in keeping with evaluation standards and to address any questions or concerns of the participating sites, site liaisons will work closely with local sites.

4. EFFORTS TO IDENTIFY DUPLICATION

This Evaluation collects data that were not been collected previously in the National Evaluation of CMHI involving community-level grantees. Instead, this evaluation provides information specific to the current CMHI SOC expansion program, which federal legislation requires to be collected from the funded jurisdictions themselves. The Evaluation will serve as a primary mechanism through which the expansion of SOC will be understood, improved, and sustained.

The Stakeholder Interviews and SOCEA data collection will all be conducted over telephone, Skype, video-conferencing, etc. The Network Analysis Survey and SAIS will be conducted online. Child and Family Outcome Study data will be collected at the service site by local staff. Therefore, there is no need to use distance data collection methods. However, we anticipate that many communities will use the CDP computer-assisted personal interview software to collect data.

5. INVOLVEMENT OF SMALL ENTITIES

Individual grantees vary from small entities through large provider organizations. Every effort has been made to minimize the number of data items collected to the least number required to accomplish the objectives of the effort and to meet evaluation reporting requirements and therefore, there is no significant impact involving small entities.

6. CONSEQUENCES IF INFORMATION IS COLLECTED LESS FREQUENTLY

The Evaluation was designed to keep the burden of data collection to a minimum. Table 4 summarizes the maximum number of times each data collection activity is proposed to be conducted. In some circumstances, data collection activities will be conducted less frequently than indicated. (More information on the data collection intervals is presented in Table 8.)

Table 4. The maximum number of times each data collection activity is proposed to be administered

Maximum Number of Administrations	Data Collection Activity
Once	GIS data (child and family level)
Twice	Stakeholder interviews Network Analysis Surveys SOCEA Financial Mapping Interview Benchmark Tool
Up to three times	Child and family outcome data
Up to four times	SAIS
Up to 12 times (quarterly or after each meeting event per grantee preference)	GIS data (jurisdiction and local system levels)

A single collection is adequate for the GIS data. These data are descriptive and are not expected to change significantly over the course of the Evaluation. GIS data for clients and families will be abstracted from client admission records.

Other data collections proposed for the Evaluation need to be collected more than once to assess change over time. Successful programs are expected to expand SOCs and improve child and family outcomes.

As shown in Table 4, some proposed data collections are twice: within the first 12 to 18 months of the grant, with a follow-up collection 2 to 3 years later. These data collections are as follows: Network Analysis Surveys, SOCEA, Stakeholder Interviews, Financial Mapping Interviews, and the Benchmark Tools. Documenting change over time is essential to evaluating grantees’ (and the grant program’s) effectiveness at expanding SOCs in their jurisdiction. Two data collections is the minimum necessary to assess such change. SOCs change slowly, so collection of data at this interval is sufficient to provide information on implementation, organizational involvement and relationships, and financing. If these data were collected less frequently, change could not be detected.

Data collection of the Child and Family Outcome Component is planned for a maximum of three data collection points. All clients will be interviewed at intake. Clients who are still receiving services at 6- and 12-months post-intake will also be interviewed at those intervals, unless the client is discharged before either time point; in that case a discharge interview will be collected. Clinicians who work with this population of children suggest that once children enter services, they are likely to experience detectable improvements within the first 6 months of services. Waiting 12 months to collect outcome data would miss important changes that are likely to happen in children who are still developing. In addition, whether improvement is sustained is important to demonstrate. Assessing outcomes at 6 and 12 months allows us to understand the course of improvement over time, and documenting such improvement (if evident) is important to demonstrating the effectiveness of the grant program.

The NET will request GIS data quarterly for jurisdiction and local system level implementation and planning events. However, because these GIS data amount to little more than rosters of meeting attendees and their addresses, grantees can submit the information as the events occur, or quarterly whichever is most convenient to them. This form will be administered more than once to capture meeting information close to when the meeting occurs to ensure more accurate data and for the grantee’s convenience. In many instances, we expect grantees to indicate that a new meeting has not taken place there will be no data to report.

7. CONSISTENCY WITH GUIDELINES IN 5 CFR 1320.5(d) (2)

The data collection fully complies with the requirements of 5 CFR 1320.5(d) (2).

8. CONSULTATION OUTSIDE THE AGENCY

The 60-Day FRN was published on October 2, 2014 (79 FRN 59498). No Comments were received.

Both external and internal stakeholders were consulted in the development of these indicators, the data collection methodology, and the associated burden. SAMHSA obtained feedback and consultation regarding the availability of data, methods and frequency of collection, and the appropriateness of data elements. Section B.5 lists the consultants with their contact information and area of expertise based on which they provided consultation for the relevant evaluation component. More information describing outside consultations follows.

Federal Consultation. SAMHSA requires the Center for Substance Abuse Treatment (CSAT), the Center for Substance Abuse Prevention (CSAP), and the Center for Behavioral Health Statistics and Quality (CBHSQ) review of the Annual Report to Congress on the Evaluation of the Comprehensive Community MH Services for Children and Their Families Program, for which the Evaluation must provide a supplemental chapter each year.

Expert Consultation. The NET includes and has consultation agreements with experts in areas relevant to the Evaluation, including child MH services research, child and family psychology, SOCs, program evaluation, measurement, quantitative and qualitative analysis, economics, web site development and usability testing, and research within Native American communities. In addition, the evaluation team includes individuals who were involved with prior National Evaluation of CMHI and prior collection of GPRA data.

Grantee Consultation. Previously funded grantees have provided input for the Evaluation. Grantees were involved in the pilot-testing of the Stakeholder interviews, SAIS, Network Analysis Surveys, the System of Care Expansion Assessment (SOCEA), and the Financial Mapping as described in Section B4. Additional input regarding evaluation processes and data utilization will be sought from grantees through conference calls with the NET and regular contacts with individual site liaisons.

Youth and Family Consultation. The National Federation of Families forms an integral part of the NET and actively participated in the development of all instrumentation. The Federation also helped develop data collection procedures and training resources. Similarly, Youth M.O.V.E. has been involved in all aspects of planning the Evaluation, including development of instruments, procedures and training materials. These contributions helped ensure sensitivity to parent and youth issues and concerns, maximized clarity of meaning, and strengthened the feasibility of administering the questionnaires. Both groups will continue to be involved in all aspects of this Evaluation.

9. PAYMENT TO RESPONDENTS

No monetary incentives are provided to respondents.

10. ASSURANCE OF CONFIDENTIALITY

SAMHSA is seeking a SORN and PIA to protect participants. In addition, has already obtained Westat IRB approval to pilot test the evaluation tools, as described in Section B.4, and to conduct the Evaluation, with one exception. The NET is in the process of obtaining approval from Westat’s IRB to collection data from children and families using the SOCEA. (Further, the NET will conform to all requirements of the Privacy Act of 1974, under the System of Records: Alcohol, Drug, and MH Epidemiological, and Biometric Research Data, DHHS, #09–30–0036; the most recent publication in the Federal Register occurred on January 19, 1999 (64 FR 2914).)

All members of the NET shall receive general awareness training and role-based training, commensurate with the responsibilities required to perform the tasks of the project. The NET will be responsible for ensuring that each member of the team has completed the SAMHSA Security Awareness Training as required by the agency, as well as Human Subjects Research Training prior to performing any project work or accessing any system, and on an annual basis thereafter, throughout the period of the project. The NET will maintain a list of all individuals who have completed these trainings and shall submit this list to the Project Officer upon request. As a part of this training, the NET shall ensure that all staff read, agree to, and sign the HHS Rules of Behavior. The NET shall also ensure that all staff have the required level of security clearance commensurate with the sensitivity of the information being stored, processed, transmitted or otherwise handled by the System or required to perform the tasks of the project. At the minimum, all members of the team shall be subjected to a Public Trust background check and be granted a Public Trust clearance before access to the System or other HHS resources is granted.

The NET shall make efforts to guard the names of respondents, all information or opinions collected in the course of interviews, and any information about respondents learned incidentally during the project. Hard copies of survey data and notes containing personal identifiers shall be kept in a locked containers or a locked room when not being used. Reasonable caution shall be exercised in limiting access to data to only those persons who are working on the project and who have been instructed in appropriate Human Subjects requirements for the project.

Only authorized users, which include grantees, GPOs, Branch Chiefs, Division Directors, the Contract Officer’s Representative (COR) and a small number of the NET will have authorized access to the main modules of the web-based CMI Portal. To enter the restricted sections of the site, users must successfully login with their credentials. The NET and grantees are responsible for entering, reviewing, and modifying performance data in the Data Entry section. The NET’s System Administrators have additional system rights, which include posting announcements, setting up programs and grants, and adding/updating users’ accounts. Account and administrative sections houses information specific to Grantees and Programs; this information may be sensitive, and is therefore password-protected. All Evaluation team members having access to system components or data are authorized for such access. Access to system information is controlled by creating/removing accounts and access groups, assigning rights to accounts and access groups, assigning accounts to access groups, granting access through physical access controls, and granting permission for access, transport or storage of information.

Serial numbers shall be assigned to respondents prior to creating a machine-processible record and identifiers such as name, dates of birth and addresses, which shall not ordinarily be a part of the machine record. When identifiers are part of the machine data record, the NET’s data processing manager, in consultation with the project director, will determine adequate measures to limit who had access to them.

Electronic files and audio files will be accessible only to project staff and under password protection. Access to network-based data files is controlled through the use of Access Control Lists or directory- and file-access rights based on user account ID and the associated user group designation, which is maintained by the system administrator. Access control on the PC is achieved for the most part by sound file management procedures by each user. Staff is instructed on the proper use of PCs for the storage, transfer, and use of sensitive information and the tools available, such as encryption.

This data collection involves three general sources of data: 1) clients and families; 2) other stakeholders asked to respond based on their professional roles, not their personal thoughts or feelings; and 3) administrative data. Informed consent forms and/or scripts are included in the attachments along with the corresponding instrument.

Clients and families. Client records at the sites are also covered under the aforementioned Privacy Act System of Records. Client and family data will be collected for the 1) child and family outcome component, 2) SOCEA, and 3) GIS Component at the child and family level.

Child and family outcome component. Access to the CMHI web-based portal will be password protected and data encryption will be used to enhance security. No information that can potentially be used to identify a client will be included in these data files other than the child/youth/young adult’s unique evaluation identification number (referred to hereafter as the unique ID). No member of the NET will ever have access to information that can link the unique ID to personal identification information. Further, the project will operate under an ADP/IT security plan approved by CMHS for project data.

The Evaluation requires collecting descriptive and outcome data from children, youth, young adults and caregivers. Each grantee will be strongly encouraged to obtain local IRB approval for the informed consent and assent procedures and data collection activities they perform for this Evaluation with children and their families. In addition, grantees will be encouraged to obtain a Certificate of Confidentiality, authorized by Section 301(d) of the Public Health Service Act. This certificate will keep the data private to the extent provided by law, protecting the investigator(s) from civil and criminal subpoena to identify participants in court. As noted previously, consent forms and/or scripts are included in the attachments along with the corresponding instrument.

Each grantee will develop and implement an active informed consent procedure that informs the participants of the purpose of the Evaluation, describes what their participation entails, and addresses the security measures described above. In addition, respondents will be informed that their participation is voluntary, that they have the right to discontinue participation at any time without impacting services they receive, and of the risks and benefits of participation. Informed assent will be obtained from participating older children and adolescents (age 11–17 years). In addition, informed consent will be obtained from young adults age 18 and older. Written informed consent or assent will be obtained from children and families at the point of entry into services and before the collection of evaluation data. Grantees are instructed to determine whether updates to consents are required at each data collection point, as the legal custody of a child may change, a child may become old enough to participate in a youth interview, a youth may become an emancipated minor or age up into adult status, and local IRBs may have requirements for regular updates.

In all grantee sites, child/youth and family outcome component data are collected by site staff. These staff members are responsible for developing procedures to guard Evaluation data during data collection, storage of data, and reporting of all information obtained through data collection activities. (Please recall from Section A.3, these client-level data will be transmitted to the client via the secure CMHI Portal online system.) These procedures include limiting the number of individuals who have access to identifying information, using locked files to store hardcopy forms, assigning unique IDs to each participant to ensure anonymity, and implementing guidelines pertaining to data reporting and dissemination.

GIS Component at the child and family level. Site staff will use the clients’ address obtained from their clinical record to determine the Census block group of their residence. (The NET will provide sites with a program to convert home addresses to Census block group designation.) The NET will only receive Census block group data, not home addresses.

Systems of Care Expansion Evaluation (SOCEA) with clients and families. A limited number of clients and families who are actively receiving services through the SOC will be interviewed by the NET using the SOCEA. NET staff will administer the SOCEA by telephone, Skype, or video-conference. The interviewer will conduct the informed consent process before beginning the interview seeking respondent permission to conduct and audio-record the interview, and parental permission for youth respondents. Audio-recordings will be used for the purpose of clarification and possible transcription. Verbal consent will be obtained to 1) participate in the interview and 2) for the interview to be audiotaped. Audio recordings and hard copies of interviews will be destroyed at the end of the Evaluation, in consultation with the Westat IRB.

Other Stakeholders in their Professional Roles. Most types of data collection activities in the Evaluation involve professional stakeholders, including child and family advocacy representatives, as respondents. Data collection will be conducted and/or coordinated by NET staff. These data collection activities will be conducted by 1) telephone interviews or 2) web-based surveys (via the CMHI portal). Specifics related to each of these data collection activities are discussed here.

Telephone Interviews. Respondents’ identities will be known (for all but youth and caregiver respondents), so to ensure participants’ rights, an active informed consent process will occur. NET staff will obtain verbal consent for telephone interviews using the consent scripts included in the attachments with the corresponding instrument. Data collection activities falling into this category include the following:

• Stakeholder interviews,

• Financial Mapping Interview, and

• SOCEA with other stakeholders (i.e., stakeholders other than youth and caregivers).

Some telephone interviews will be audio recorded. Separate informed consent will be obtained to audiotape these interviews. As mentioned previously, audio recordings will be destroyed at the end of the Evaluation, in consultation with the Westat IRB. Web-based Surveys. As described previously, the CMHI Portal will be used for web-based data collection. Data collection activities falling into this category include the following:

• SAIS and

• Network Analysis Surveys.

Respondents’ identities will be known, so to ensure participants’ rights, an active informed consent process will occur. Potential participants will be contacted by mail, email, or telephone to explain the survey. Survey explanation will include the voluntary nature of survey completion, treatment of responses, and the risks, benefits, and rights as respondents. Participants will be asked to indicate, by checking a box on the Web survey that they agree to participate in the Evaluation before they complete and submit the survey. Information about the Evaluation and participant rights will be presented in the Web survey prior to the check box indicating consent to participate. The letter and the Web survey will also provide contact information if the survey recipient has questions or desires clarification prior to participation. If the individual does not have Internet access, alternative administration methods will be used such as 1) a packet sent by regular mail containing a cover letter, an informed consent form, a survey, and a return envelope (the cover letter will indicate that the respondent is to return the informed consent form and the completed survey in separate envelopes enclosed in the packet) or 2) the survey will be administered by telephone interview following the procedures detailed previously.

Administrative Data. When electronic data are transferred to the NET, data files will be encrypted to make the information indecipherable during electronic transfer. Data will be transmitted securely and all caution will be used, as described in Section A.3, Use of Information Technology. The term, administrative data, is used loosely here to refer to 1) Financial Mapping and Benchmark Data and 2) GIS data from meeting participants.

Financial Mapping and Benchmark Data. For these data analysis activities, NET staff will request deidentified financial service and other cost data. Westat IRB approval will be obtained for all data collection activities, including these, prior to commencing data collection.

GIS Data from Meeting Participants. NET staff will request rosters of professional meeting and training participants and their addresses to calculate Census block group. Hard copies of the forms used to collect GIS data from meeting participants will be stored in locked cabinets or rooms. Reasonable caution shall be exercised in limiting access to data to only those persons who are working on the project and who have been instructed in the applicable Human Subjects requirements for the project. Address lists will be destroyed by shredding hard copies of forms once Census block group is determined.

11. QUESTIONS OF A SENSITIVE NATURE

In that this project pertains to services to children with SED and their families, it is necessary to ask questions that are potentially sensitive as part of the Child/Youth and Family Outcome Component. However, only information that is central to the Evaluation is being sought. Some questions asked of children, young adults, and caregivers address dimensions such as child emotions, behavior, social functioning. In addition, young adults age 18 to 21 will be asked about their experience with violence and trauma, but no other respondents. (The violence and trauma items were taken from the TRAC adult instrument already approved by OMB (OMB No. 0930-0285). The answers to these questions will be used to determine baseline status and to measure change in these areas experienced after receiving SOC services. Each grantee must keep data on child and family status and service use, as well as treatment records and other related information. For these reasons, the data collection required for the Evaluation is not introducing new, sensitive domains of inquiry, but is paralleling standard procedures in the field of children’s MH.

In addition to information on child clinical status and social functioning, other questions of a sensitive nature will be asked of families. These include questions related to caregiver strain associated with raising a child with SED. These questions are included in order to assess the extent to which caregiver strain is reduced after the child/youth and family receive SOC services. Moreover, family representatives who have consulted with the NET consistently identify a lack of information on family outcomes as a weakness in previous evaluations and studies.

The NET will train grantees to include specific language in their consent and assent forms to describe the data collection activities and information that will be accessed through the client’s and family’s records and shared with the NET. Although grant communities may work with personally identifiable information to extract and link electronic records, no personally identifiable information will be included in any data transferred to the NET, other than the client’s unique ID.

Before collecting data, each grantee will obtain active consent from caregivers and young adults age 18 to 21. In addition, assent will be obtained from children and adolescents. In that process, respondents will be made aware that the information they provide will be strictly guarded and that they can withdraw their participation at any time without any adverse effects on the services they receive. Similarly, respondents can freely choose to refrain from answering any questions they find objectionable. Consent forms and scripts are included in the attachments along with the corresponding instrument.

Only aggregated information will be disseminated. The primary dissemination vehicles are the Report to Congress and the Report to the Secretary of HHS, though there may be other dissemination efforts as well. SAMHSA will be careful to avoid disseminating small numbers with demographic information that might be used to deduce the identity of individual respondents. Specific rules used to avoid such dissemination will be determined in consultation with NET statisticians, but typically involve suppressing data where a small number of respondents would otherwise appear in a table shell or text.

12. ESTIMATES OF ANNUALIZED HOUR BURDEN

SAMHSA is requesting an estimated average annual total burden of 11,958 hours for this submission. Table 5 displays the annual burden estimates for the Evaluation. Table 6 summarizes the combined burden associated with the initial three years of data collection for the National SOC Expansion Evaluation, summarized across planning and implementation grants.

Table 5. Estimated average annual burden

Instrument/ Data Collection Activity	Respondent	Number of Respondents	Responses per Respondent	Total Number of Responses	Hours per Response	Total Annual Burden Hours	Hourly Wage	Total Cost
Implementation Assessment
Stakeholder Interviewsa	Project Director	57	1	57	1.6	90	$26.44	$2,375
	Family Organization Representative	57	1	57	1.6	90	$21.55	$1,936
	Youth Organization Representative	57	1	57	1.6	90	$21.55	$1,936
	Core Agency Partnersb	287	1	287	1.3	358	$26.44	$9,474
SAISa	Grant leadership	1,540	1.93	2,970	0.82	2,426	$26.44	$64,130
SOCEA	Project Director & Representatives from Family & Youth Organizations	143	1	143	1.5	215	$26.44	$5,671
	Core Agency Rep, Service Providers	429	1	429	1.0	533	$26.44	$14,079
	Care Coordinators	95	1	95	1.7	162	$21.55	$3,493
	Caregivers	95	1	95	0.75	106	$11.47	$1,216
	Clients 14-21	95	1	95	0.5	48	$7.25	$346
Network Analysis Survey
Jurisdiction	Grant leadership	353	1	353	0.4	147	$26.44	$3,893
Local system	Local providers of direct services	707	1	707	0.4	294	$21.55	$6,345
GIS Component: Group Collaborative Events for GIS Analysis Form
Jurisdiction	Grant administrator/Project Director	106	4	424	0.25	106	$21.55	$2,803
Local system	Local administrator/Project Director	106	4	424	0.25	106	$21.55	$2,803
Financial Mapping and Benchmark Components
Financial Mapping Interview	Financial administrators at: Medicaid Agencies & MH Authorities	97	1	97	2.0	217	$26.44	$5,737
	Financial administrators at: Trade associations & Family organizations	32	1	32	1.5	52	$26.44	$1,384
Benchmark Tool	Payment/reporting personnel at: Medicaid Agencies & MH Authorities	24	1	24	40.0	960	$21.55	$20,688
Child and Family Outcome Component
Background Information (CDP)c	Caregivers of clients age 11-17d	631	2.12 e	1,337	0.37	491	$11.47	$5,628
	Clients age 11-17	631	2.12	1,337	0.37	491	$7.25	$3,557
Family/Living Information	Caregivers of clients age 5-17f	3,172	2.12	6,725	.05	336	$11.47	$3,857
	Clients age 18-21g	650	2.12	1,377	.05	69	$7.25	$499
Caregiver Strain Questionnaire - Short Form	Caregivers of clients age 5-17	3,172	2.12	6,725	0.12	807	$11.47	$9,257
Columbia Impairment Scale	Caregivers of clients age 5-17	3,172	2.12	6,725	0.08	538	$11.47	$6,171
	Clients age 11-21h	1,911	2.12	4,051	0.08	324	$7.25	$2,350
Pediatric Symptom Checklist-17	Caregivers of clients age 5-17	3,172	2.12	6,725	0.05	336	$11.47	$3,857
	Clients age 11-21	1,911	2.12	4,051	0.05	203	$7.25	$1,469
Client record review	Site staff	28	407	11,261	0.21	2,365	$21.55	$50,964
Total Annual Burden
All	All	9,365		56,664		11,958		$235,915

1. Burden includes planning and implementation grantees.

2. Core agency partners include (1) representatives from MH, child welfare, and juvenile justice and (2) CMHI quality monitors.

3. OMB clearance sought for CDP is limited to the added burden for a second respondent (Caregiver OR Client age 11 to 17). For clients age 11 to 17, CDP only collects information from either Caregivers OR youth. In addition, clearance is requested for the burden only as OMB approval of CDP has been sought separately.

4. Assumes 33% of clients will be age 11 to 17 and that the additional CDP interview for clients age 11 to 17 and their caregiver will be evenly split between clients and caregivers. Evaluation design requires all participating clients age 5 to 17 to have a caregiver participating in the evaluation.

5. Accounts for attrition.

6. Assumes 83% of clients will be age 5 to 17.

7. Assumes 17% of clients will be age 18 to 21.

8. Assumes 50% of clients will be age 11 to 21.

Table 6. Total estimated annual burden

Instrument / Data Collection Activity	Number of Respondents	Total Number of Responses	Average Annual Burden (Hours)
Stakeholder Interviews	459	459	628
SAIS	1,540	2,970	2,426
SOCEA	858	858	1,063
Network Analysis Survey	1,060	1,060	442
GIS	212	848	212
Financial Mapping Interview	129	129	269
Benchmark Tool	24	24	960
Child and family instruments (respondent & staff burden)	5,083	50,316	5,959
Total	9,365	56,664	11,958

13. ESTIMATES OF ANNUALIZED COST BURDEN TO RESPONDENTS

There are neither capital nor startup costs, nor are there any operations or maintenance costs.

14. ESTIMATES OF ANNUALIZED COST TO THE GOVERNMENT

SAMHSA has planned and allocated resources for the management, processing, and use of the collected information in a manner that shall enhance its utility to agencies. The contract award to cover evaluation of this project is $5,599,119 over a 48-month period. Thus, the annualized contract cost is $1,399,779.

Additional costs will be incurred indirectly by the government in personnel costs of staff involved in oversight of data collection. It is estimated that one SAMHSA employee will be involved for 15 percent of their time. Cost of staff time will approximate $13,500 annually.

The estimated annualized total cost to the government will be $1,413,279.

15. CHANGES IN BURDEN

This is a new data collection.

16. TIME SCHEDULE, PUBLICATION, AND ANALYSIS PLANS

a. Time Schedule

The time schedule for implementing the Evaluation and publishing findings in the supplement to the Annual Report to Congress is summarized in Table 7. A 3-year clearance is requested for this project.

Table 7. Schedule of Evaluation activities

Activity	Date
Receive OMB clearance for data collection	XXX
Begin data collection 2 months after OMB clearance	XXX
Stop data collection	June 30, 2018
Begin processing and analyzing data	6 months after OMB clearance
Produce supplement to Annual Report to Congress	Every October from 2014 to 2018

NOTE. *Data will be collected from planning grants only once, as their funding period lasts for only one year.

b. Data Analysis Plan

Data collection and analytic strategies are linked to the Evaluation questions listed in Table 9. Data analyses are described for each Evaluation component followed by a description of cross-level analyses.

Implementation Assessment

Stakeholder interviews. Using a pre-established analysis frame, coders will document descriptive information about what implementation and expansion strategies are being used by each grantee and how well or fully they have been implemented. Multiple coders will be trained, and inter-rater reliability will be tested to ensure consistency in identifying and recording strategies.

Qualitative analysis of interview data will be conducted to comprehensively describe implementation and expansion plans and efforts. SAMHSA will describe: planned and implemented strategies and mechanisms; participants’ involvement in the planning and implementation process; role of child-serving sectors, youth groups, and family organizations; funding mechanisms; efforts to reach vulnerable and hard-to-reach populations; support of local systems in direct service delivery; and implementation barriers and facilitators encountered. Finally, findings will be compared across jurisdictions to identify commonly employed strategies, as well as those that are potentially innovative. The SOC principles most and least often included in the development of implementation and expansion efforts will also be described.

Self-Assessment of Implementation Survey (SAIS). Analysis of SAIS data will focus on grantees’ efforts to implement and expand the SOC in their jurisdictions. Areas described in the Stakeholder Interviews will be assessed quantitatively in the SAIS. Analyses will also explore what facilitators and impediments to SOC implementation and expansion were encountered. Measurement quality of the SAIS will be examined using estimates of reliability (e.g., internal consistency) and factor analysis. Descriptive statistics (e.g., mean, standard deviation) will be calculated for individual items, and multi-level longitudinal analyses (e.g. hierarchical mixed models and/or structural equation modeling) will be used to examine change over time across all dependent variables, with respondents nested within jurisdiction. To the extent that groups of items appear to assess the same construct, summary scores may be calculated (e.g., means or sums). Jurisdictions will receive their own scores, as well as the combined scores of all other jurisdictions for comparison purposes.

SOCEA. The SOCEA (i.e., SOC Expansion Assessment) interview will be used to describe and assess grantee efforts to implement and expand the SOC at the local system level. The primary focus is on direct service delivery processes and management. Qualitative analysis of SOCEA data will be used to describe (1) SOC implementation and expansion mechanisms and strategies used to implement and expand SOC service delivery; (2) progress made toward implementing the expansion strategies; (3) facilitators and impediments to implementing strategies; and (4) how well management and operation and direct service delivery embody SOC principles. From the qualitative data, SAMHSA will generate numerical ratings of the extent to which SOC implementation is effective, as well as adherence to SOC principles. In addition to calculating descriptive statistics (e.g., mean, standard deviation) at the time level, summary scores will be calculated for each system of care principle. Multi-level longitudinal analyses (e.g. hierarchical mixed models and/or structural equation modeling) will be used to examine change over time across all dependent variables, with respondents nested within jurisdiction. The reports will follow a standard outline to ensure comparability across grantees. To inform program improvement, the site report will be provided to local systems including the community’s own scores and the mean scores of the other assessed communities combined.

Network Analysis and GIS

Increasing connections among agencies and organizations, and spreading SOCs to new geographic areas are key goals of SOC implementation and expansion efforts. Network analysis provides an assessment of relationships among agencies and organizations. GIS will provide a description of geographic coverage of the SOC.

Network Analysis. The Evaluation will identify the network of agencies that work together at the top administrative level of jurisdictions and local systems to implement and expand the SOC. It will describe how, and the extent to which, key agencies and organizations collaborate and coordinate their efforts. For the two levels involved, the focus of network analyses are as follows:

• Jurisdiction. Interactions among leaders at the top administrative levels responsible for a jurisdiction-wide SOC implementation and expansion governance, policy development, system oversight, and quality improvement efforts.

• Local system. Cross-agency interactions related to direct service provision to children/youth and their families including management, operations, practice, and care monitoring.

Network analysis generates graphic representations depicting the relationships between and among agencies and organizations across the local system and the jurisdiction. Network analysis generates numerical computations of the strength of these cross-agency relationships or links between and among participants. These data will be used to describe the network in terms of density (i.e., how sparsely or closely are participants connected), centralization (i.e., whether there are one or more partner agencies around which most of the others tend to gather), fragmentation (i.e., whether many system participants appear isolated or whether the system made up of small clusters that are unconnected to each other), and coordination.

Specifically, these data will be used to assess collaborations among stakeholders in implementing and expanding the SOC. As social network data are relational data rather than traditional data of independent observations, data entry and analysis are necessarily specialized. In particular, social network data will be entered and stored in rectangular matrices rather than lists. These matrices will be created in the social network analysis programs UCINET and NodeXL for storage and analysis. UCINET is utilized to quantify social network data. NodeXL is used to create network graphics or “maps” that visually portray the observed network thereby providing companion graphs for network analytics.

GIS. SAMHSA will consider estimates of population density of the jurisdiction to understand what types of areas (e.g., frontier, rural, suburban, urban) are covered by what types of expansion efforts (e.g., training, governance, planning). Specifics related to each level are described next.

• Jurisdiction. Business and office addresses will be used in GIS analyses to map areas of the jurisdiction where attendees (either in-person or virtual) to important planning and implementation events (e.g., governance body meetings, strategic planning meetings, training sessions, summits) work. This will provide an indication of how widely distributed participants are across the jurisdiction. Although the offices of most stakeholders involved in planning and expansion efforts to be clustered around State capitals and county seats, seeing broader distribution may be an indication of broader involvement and buy-in across the jurisdiction.

• Local system. GIS efforts will focus on events related to planning and expansion efforts related to direct service provision and program development. The business and office addresses of individuals who attend these events, in person or virtually, will be recorded and used in GIS analyses. These events may include training of direct service providers and supervisors, training on SOC values, multi-agency care review meetings, and management meetings. GIS analyses will map how well event participants are spread across the local system catchment.

• Child and family. Residential Census block groups of clients will be collected at baseline and used in GIS analyses. This will provide a representation of the areas where SOC clients live, and to what extent the entire area is being reached. Particular attention will be directed at assessing how well the local SOC reaches residential areas likely to house hard-to-reach populations.

In addition to analyzing GIS data within each level, GIS data will be considered together to discover relationships across levels. GIS data across levels will be overlaid to depict the extent to which coverage across levels correspond. For example, SAMHSA will assess whether the children and families served live in the same areas where participants in clinical training seminars practice.

Financial Mapping and Benchmark Component

Financial Mapping. For each state, county, or tribe included in the analysis cohort, SAMHSA will compare the number and types of state and federal funding sources for state children’s MH and SOC services by level of MH care during the first or second implementation grant year to the funding sources used two years later. (Levels of MH care include inpatient, residential, emergency care, outpatient, care management, rehabilitative services, peer and family organizations, other supports, flexible funds, and so forth.) Specifically, from the information collected through the Financial Mapping Interview and review of administrative data, the NET will create a map. The map will be in the form of a matrix showing for each level of care in the children’s MH system that identifies (1) any applicable income or clinical eligibility criteria for the children’s MH services; (2) the continuum of services; (3) the federal, state, county, tribal, or commercial health plan funding source or sources of funds; and (4) the State, county or tribal agency through which the services are funded. These sources of funds in the map will include State funds, county funds, Medicaid, Medicaid managed care plans, Medicaid waivers, commercial health plans (which may include Exchange-related health plans), and other sources. Respondents may also identify the services and resources provided by other state and local agencies, including juvenile justice, child welfare, and education sources when relevant. Finally, SAMHSA will also seek to understand grantees’ plans for future funding of SOC services.

The first interview will establish a baseline of the funding sources used at the beginning of the grant. The second interview conducted two years later will identify any changes in the structure of financing over the years of the grant and its impact on behavioral health services for children in the jurisdiction. The map will also document any relevant expansion of benefits to broaden coverage for SOC services such as wraparound planning, intensive care coordination, family and youth peer-support, and flexible funds.

Benchmark Component. The voluntary benchmark component will use MH Authority and Medicaid Agency data to compare states’ rate of penetration, utilization and expenditures for children’s MH services by level of MH care. All indicators will be calculated with detailed specifications for numerators (i.e., children served, utilization and costs by type of service) and denominators (i.e., populations covered). SAMHSA plans to compare across states within the same year. For grantees with at least 2 years of data, SAMHSA will compare changes in penetration, utilization, expenditures, and psychiatric hospital readmission rates between the years. Analyses will focus on the relative use of services and financial resources by level of MH care. SAMHSA will identify spending patterns that may indicate shifts in costs, and cost savings or cost offsets, particularly in inpatient and residential treatment services. Examples of variables that will be calculated from requested information include penetration rates, inpatient days, residential days, emergency room use, outpatient visits per 1000, users of MH care coordination FTEs per 1000, 30-day readmission rates, data on the use of family and youth partners or mentors, and in-home service utilization. The denominator for penetration and rates per 1000 for Medicaid will be health plan members in the relevant age group and for MH Authorities will be US census estimates of child population in the jurisdiction. Data will be collected from the MH Authority and for Medicaid funds.

Child and Family Outcome Component

Data elements available for analyses at the child and family level from CMHI web portal are listed in Section B.2 (for data collection added by the Evaluation) and in Section A.3 (for data already collected by the CDP). SAMHSA will use univariate descriptive analyses to characterize children, youth, and families being served through CMHI implementation grants. Descriptive information of Evaluation participants will include score ranges, means, and medians. Results will be reported for each jurisdiction as well as for all implementation grantees combined. Bivariate analyses will be performed to find correlations between child, youth, and family characteristics (e.g., relationships between family income and caregiver strain at intake).

The longitudinal design assesses whether individual children and families experience meaningful improvements in outcomes during treatment. Changes over time in child and caregiver satisfaction with services will be tested using descriptive analyses (e.g., frequencies and percentages) as well as univariate and bivariate analyses (e.g., are there differences in satisfaction by clinical severity). Change in child and family outcomes over time will be tested using a variety of techniques. Repeated measures analysis of variance (ANOVA) will be used to test the significance of change over time within and between groups, both across and within jurisdictions. Repeated measures analysis of covariance (ANCOVA) will be conducted as well, to examine the influence of covariates. Path analysis and other structural equation modeling techniques will be used to investigate the direct and indirect effects of independent variables (such as, type and amount of services received, and demographic variables) on dependent outcome measures (such as symptom severity, social functioning, and caregiver strain). Structural equation modeling will prove particularly useful for sub-analyses of data from older youth in which latent constructs (e.g. youth functioning) have ratings from both youth and caregivers). Multi-level modeling (MLM) will be used to estimate growth curves (e.g., changes in the level of symptomatology) at the individual level based on repeated observations.

Cross-Level Analyses

Two sets of additional analyses are planned that cut across the different levels of the Evaluation (i.e., the jurisdiction, local system, and child and family levels). The first set of analyses will focus on how the quality of the SOC expansion implementation at the jurisdiction level is related to changes at the local system level. Specifically, changes in (1) SOCEA scores and (2) local system network integration such as network density, centrality, fragmentation, and coordination (based on data from the Network Analysis: Local System) will be analyzed relative to the jurisdiction’s implementation quality (based on data from the web-based SAIS), using MLMs.

The second set of analyses will use a series of MLMs to evaluate child, youth, young adult, and family outcomes based on the extent to which system-level expansion quality (as indicated by SOCEA scores) is related to improved child, youth, young adult, and family functioning. SAMHSA will explore whether particular domains on the SOCEA are related to specific child and family outcomes.

17. DISPLAY OF EXPIRATION DATE

All data collection instruments will display the expiration date of OMB approval.

18. EXCEPTIONS TO THE CERTIFICATION STATEMENT

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.

1 Hyde, P.S. Leading change in an era of health reform. Presented at National Child Traumatic Stress Network Grantee Meeting. Baltimore, MD • March 1, 2011. Cited July 8, 2014 at http://store.samhsa.gov/shin/content//SMA11-PHYDE030111/SMA11-PHYDE030111.pdf .

2 National Center for Injury Prevention and Control, Centers for Disease Control. Suicide facts at a glance 2012. Cited July 8, 2014 at http://www.cdc.gov/violenceprevention/pdf/suicide_datasheet-a.pdf

3 The CDP is currently under review by OMB. The CDP has been designed to replace the Transformation Accountability (TRAC) system and tool (OMB No. 0930-0285, Exp. 5/312015).

15