Children’s Mental Health Initiative National Evaluation
Supporting Statement
The Substance Abuse and Mental Health Services Administration (SAMHSA), Center for Mental Health Services (CMHS) obtained approval from the Office of Management and Budget (OMB) for data collection for the National System of Care (SOC) Expansion Evaluation, OMB No. 0930-0349, approved on 4/28/2015 and with an expiration date of 4/30/2018. This approval covered the collection of data from SAMHSA’s Comprehensive Community Mental Health Services for Children and their Families Program also known as the Children’s Mental Health Initiative (referred to hereafter as CMHI) System of Care (SOC) Expansion Planning Grants and the SOC Expansion Implementation Cooperative Agreements and a total of eight data collection instruments. These instruments will continue to be used for SOC Implementation Grants initiated in Fiscal Years 2013 and 2014.
This submission is a request for approval of revised versions of these data collection instruments for the newly awarded CMHI National Evaluation contract, which will evaluate SOC Expansion and Sustainability Cooperative Agreements issued in or after FY2015. These revisions are responsive to the evaluation of specific goals and objectives SAMHSA CMHI requires of this new Grantee cohort. The remainder of this document will address this new submission.
Background
The SOC model was developed in response to the need to improve the organization, coordination and delivery of children’s MH services and to improve clinical and functional outcomes of children, youth, and young adults with significant MH needs. SOC is an organizational model that involves collaboration across agencies that serve children, youth and families to provide an array of effective, community-based, culturally- and linguistically-appropriate services and supports for children, youth, and young adults with or at risk for behavioral health challenges and their families (Stroul, Blau, & Friedman, 2010).
The purpose of the original CMHI community grantees was to provide Federal support through grants to States, political subdivisions within States, the District of Columbia, tribal areas, and territories to develop integrated home and community-based systems and supports for children and youth with severe emotional disturbances (SED) and their families (Huang et al., 2005). CMHI was shaped by the Child and Adolescent Service System Program (CASSP), which was implemented in 1984 to assist States and communities in building a comprehensive, community-based SOC. CMHI was funded as a Federal demonstration grant program at the initial level of $4.9 million, and 22 communities were awarded grants through CMHI in 1993 to 1994. In the last 20 years, CMHI grants have funded children’s MH services in all 50 states, Puerto Rico, Guam, the Northern Mariana Islands, and American Indian/Alaska Native territories and authorities.
Since its inception, CMHI has served more than 103,000 children and youth and their families, who have shown a variety of clinical and functional improvements during their involvement with CMHI. Examples of clinical and functional improvements have included increases in behavioral and emotional strengths, reductions in suicide attempts, improvements in academic outcomes, decreased criminal justice involvement, reductions in reliance on inpatient care, and more stable living situations (Annual Report to Congress, 2011). The goals of the CMHI program are to:
Expand grantees’ capacity to serve children and adolescents with SED and their families;
Provide a broad array of accessible, clinically effective and fiscally-accountable services, treatments and supports;
Serve as a catalyst for broad-based, sustainable systemic change inclusive of policy reform and infrastructure development nationwide;
Create a care management team with an individualized service plan for each child;
Deliver culturally and linguistically competent services with special emphasis on racial, ethnic, linguistically diverse and other underrepresented, underserved or emergent cultural groups; and
Encourage and facilitate full participation of families and youth in planning, evaluation and sustainability of local services and supports and in overall system transformation activities.
The CMHI National Evaluation (hereafter referred to as the Evaluation) is driven by the SOC program theory model (illustrated in the Evaluation logic model (see Attachment 1)).
b. The Need for Evaluation
Section 565(c)(1) of the Public Health Service Act of Public Law 102-321 mandates annual evaluation activities of SAMHSA programs. Basic requirements are the documentation of: characteristics of the children and families served by the grant program; the type and amount of services they receive; how these services are coordinated; and the associated costs. Equally important is the need to assess whether the program was implemented and the services were experienced as intended as well as the geographic distribution of providers and clients. It is also critical to assess whether the children served by the program experience improvement in clinical and functional outcomes, whether family life is improved, and whether improvements endure over time.
Further evaluation requirements under Sections 564(f) and 565(c)2 of PL102-321 include:
Annual reports to the Secretary of Health and Human Services (HHS) that include a description of the number of children served, child demographic characteristics, types and costs of services provided, availability and use of third-party reimbursements, estimates of the unmet need for services, how the grant was expanded, and other information as required by the Secretary.
Annual reports to Congress that provide information on longitudinal evaluations of outcomes of services provided by the funded SOC, the effect of activities conducted under funded SOC on the utilization of hospital and other institutional settings, barriers to the achievements of establishing interagency collaboration within the SOC, and parental assessment of the effectiveness of the SOC.
Clearance Request
This submission requests OMB clearance for (1) data collection to evaluate the CMHI Expansion and Sustainability Cooperative Agreements and (2) the estimated burden for collecting data under this protocol. The estimated burden for data collection is a total of 54 Cooperative Agreements.
Overview of the Proposed Evaluation
The evaluation is designed using a strategic framework (adapted from Stroul and Friedman 2011 and Stroul, Dodge, Goldman, Rider and Friedman, 2015) that provides analytic dimensions (i.e., policies, services/supports, financing, training/workforce, and strategic communications). These dimensions cut across the State System, Local System and Service Delivery levels and together link to a range of proximal and distal outcomes. The evaluation will identify and assess the mechanisms and strategies employed to implement and expand systems of care, and explore the impact on system performance and child and family outcomes. Evaluation activities are framed by the five strategic areas (1. Policy/ Partnerships; 2. Services/Supports; 3. Financing; 4. Training/Workforce; and 5. Strategic Communications) to examine whether specific mechanisms and strategies lead to proximal and distal outcomes. System of care principles are woven throughout the framework at both the State and Local levels. The evaluation tools are designed to allow analysis across levels.
2. Purpose and Use of the Information
At its core, the purpose of the Evaluation is to assess the success of the SOC expansion and sustainability grant initiatives. This section describes how, and for what purpose, the information collected will be used by SAMHSA, CMHS, grantees, and the practice community.
SAMHSA/CMHS requires evidence that grantees can support SOCs that are effective, cost-effective and sustainable in communities throughout the country. SAMHSA/CMHS can use Evaluation findings to examine: 1) the extent to which the program results in true expansion and sustainability of SOC service delivery; 2) which mechanisms and strategies are most effective for realizing broad program goals; and 3) whether the program results in improvement in child and family outcomes.
SAMHSA/CMHS
Evaluation findings will be useful to SAMHSA, CMHS directors, and Grant Project Officers (GPOs) by: (1) fulfilling the program’s legislatively mandated requirements to evaluate its programs; (2) supporting several of SAMHSA’s Strategic Initiatives; and (3) providing essential program management and development information to CMHS leadership.
Legislative Requirements. As described under The Need for Evaluation (Section A.1b), the Evaluation was designed to respond to the legislatively-mandated requirement to evaluate the CMHI and to report program evaluation findings in annual reports to Congress and to the Secretary of HHS. Table 23 maps proposed data collection activities to public law and to required activities delineated in the grant request for application (RFA) (based on the FY 2015 application). In addition, the Evaluation addresses the RFA requirement that grantees conduct Data Collection & Performance Measurement as well as Performance Assessment.
Table 2. Purpose of data collection activities relative to public law and activities required of grantees
Data collection |
Description of Tools |
Implementation Grant RFA Required Activities |
Key Partner Interviews
System of Care Expansion and Sustainability Survey (SOC-ESS) |
Qualitative data collection from administrators, youth and family representatives, and child agencies.
Self-report survey administered to representatives from grantee, youth and family, provider, and advocacy organizations and child serving sectors. |
Use SOC values throughout planning and implementation including meaningful involvement by families and youth in governance and planning and implementation activities, establishing policies, administrative, and regulatory structures that support ongoing SOC implementation, provisions to ensure that SOC services, policies and programs are culturally and linguistically accessible. |
Network Analysis Surveys |
Self-report survey designed to measure depth and breadth of collaboration across agencies and organizations. |
Collaborate across child serving agencies and among critical providers of programs including those serving youth nearing adulthood. |
Child and Family Study
GIS |
Longitudinal data collection includes child and family background information, and clinical and function outcomes. Tools include: Caregiver Strain Questionnaire, Columbia Impairment Scale, Pediatric Symptom Checklist, background data from the SAMHSA National outcomes Measures(NOMS), the Baby and Pre-School Pediatric Symptom Checklist and the Brief Infant and Toddler Emotional Assessment.
Will measure geographic coverage and spread using census block groups derived from home addresses. |
Comply with HHS action plan to reduce racial/ethnic health disparities. Program Purpose: Expand services across geographic areas and population groups.
|
Financial Mapping Interview Benchmark Tool |
Interviews conducted with financial administrators, provider trade and family organizations. |
Develop financing approaches that promote a cross-agency service delivery system, create flexible funds, and develop fiscally accountable approaches to care review. Seek third-party reimbursement. |
|
|
|
Note. Public Law refers to Public Health Service Act Title V, Part E Public Law 102-321, Section 561-565, 42 U.S.C. 290ff-4. Assmt. = Assessment.
SAMHSA’s Strategic Initiatives. In 2014, SAMHSA defined six Strategic Initiatives with input from stakeholders including Federal, State and local leaders; constituency groups; advisory council members; members of Congress; people in recovery; and family members. These initiatives are designed to focus SAMHSA’s work on improving lives and capitalizing on emerging opportunities. The Evaluation is relevant to the following strategic initiatives:
Prevention of Substance Abuse and Mental Illness, as the Evaluation determines the relationship between systems change and client care practices, and mental health and substance abuse prevention outcomes;
Health Care and Health Systems Integration, which emphasizes the need for integrated and coordinated care such as that promoted by the SOC model and assessed by the Evaluation;
Trauma and Justice, by assessing client outcomes and services available and provided to individuals who are (1) in need of trauma-informed services and (2) involved with or at-risk for involvement with the juvenile or criminal justice systems;
Recovery Support for individuals recovering from MH and substance use disorders. Evaluation findings will show SAMHSA the extent to which it is engaging consumers and their families in self-directed care, shared decision-making, and person-centered planning;
Health Information Technology, as Evaluation findings will assess and inform how data is used in client care and coordination, and program and client outcome monitoring, and
Workforce Development, one of the key elements of the strategic framework guiding the evaluation, Evaluation findings will assess and formalize best practices in this area.
CMHS Leadership. SAMHSA/CMHS directors and GPOs can use Evaluation findings to address program management priorities including accountability (i.e., legislative requirements, as described in Section 1.b), program and policy planning, and program justification. For example, Evaluation findings can be used by CMHS leadership to:
Monitor the progress of funded activities, which is essential for program monitoring, providing program TA, and program justification.
Inform both intra- and interagency program and policy planning.
Develop policies and provide guidance regarding SOC development.
Support TA activities to help grantees best meet program goals.
Support the many partners that work on CMHI in collaboration with CMHS, including the National Federation of Families for Children’s MH and Youth M.O.V.E. National in their efforts to help build SOC for children's MH services.
Some Evaluation findings will be of use to both CMHS leadership and grantees such as (but not limited to) the following:
Illustrating the development of SOCs as they move toward offering integrated and comprehensive services;
Providing detailed information on how to successfully bring SOCs to scale and sustain them;
Identify funding sources used by states to sustain or expand SOCs services;
Describing experiences and implementation practices (across all grantees);
Identifying best practices and effective strategies;
Understanding barriers and facilitators to successful implementation;
Comparing access, utilization and expenditure patterns for children’s MH services across states and other grantees;
Documenting savings from reducing high-cost, out-of-home services that illustrate the business case for SOCs;
Showing whether there are observable differences in child and family outcomes that can be plausibly linked to the SOC approach; and
Describing how children and families experience the service system and how they use services and supports (i.e., utilization patterns).
Grantees and the Practice Community
Grantees and the practice community can use Evaluation findings to:
Improve the implementation of their SOC and achieve the goals of the CMHI;
Improve the quality of the services they provide;
Identify additional opportunities for financing SOC services and obtaining matching funds and adopt the financing strategies and funding sources associated with the most successful implementations of SOC services, all with the goal of SOC sustainability;
Identify opportunities to further reduce the use of restrictive and expensive out-of-home services by strengthening SOC and community-based MH care;
Learn what barriers to treatment and other essential services children or youth and their families perceive and work to eliminate such barriers;
Learn whether families experience services as the grantees intended and identify their programs’ strengths and weaknesses;
Help identify gaps in system development and barriers to collaboration;
More effectively allocate personnel and funding and prioritize activities; and
Provide summary reports to their local steering committees or other advisory boards, support statewide expansion efforts, develop interagency partnerships, and obtain resources to sustain systems with interagency agreements.
Web-Based CMHI Portal
A web-based portal, The CMHI National Evaluation Portal, will be developed to collect and manage all data obtained from and submitted by grantees. The use of web-based surveys and forms decreases respondent burden, as compared to that required for alternative methods, such as a paper format, by allowing for direct transmission of the survey or form. Respondents can complete the survey at a time and location that is convenient for them. In addition, the data entry and quality control mechanisms built into the web-based format reduce errors that might otherwise require follow-up, thus reducing burden compared to that required for a hardcopy data collection.
Finally, the CMHI National Evaluation Team will ensure that all web-based solutions are fully compliant with Section 508 of the Rehabilitation Act. This includes ensuring that all posted documents are compliant or have a compliant alternative. SAMHSA will utilize Adobe products that are capable of producing compliant PDF files per the SAMHSA-recommended process. The CMHI National Evaluation Team has a thorough knowledge of Section 508 standards and employs accessibility specialists with experience in Section 508 compliance verification, including assessment with a variety of assistive technologies, including screen readers, screen magnifiers, and voice recognition software.
All Key Partner Interviews and Financial data collection will be conducted by telephone, Skype, video-conferencing, etc. The Network Analysis Survey and SOCESS will be conducted online. Child and Family Outcome Study data will be collected at the service site by local staff. Therefore, there is no need to use distance data collection methods.
Training and TA
The CMHI National Evaluation Team will provide training and TA support to sites to facilitate implementation of the Evaluation protocol and the use of evaluation results at the site level. In addition, a site liaison will be assigned to every Grantee, to help ensure that data are in keeping with evaluation standards and to address any questions or concerns.
This evaluation provides information specific to the current CMHI SOC expansion program. The Evaluation will serve as a primary mechanism through which the expansion of SOC will be understood, improved, and sustained. This data is not collected through any other mechanism.
Individual grantees generally are States or major municipal systems, not small entities. Every effort has been made to minimize the number of data items collected to the least number required to accomplish the objectives of the effort and to meet evaluation reporting requirements and therefore, there is no significant impact involving small entities.
The Evaluation was designed to keep the burden of data collection to a minimum. Table 2 summarizes the maximum number of times each data collection activity is proposed to be conducted. In some circumstances, data collection activities will be conducted less frequently than indicated.
Table 2. The maximum number administrations for each data collection activity
Maximum Number of Administrations |
Data Collection Activity |
Once |
|
Twice |
|
Up to three times |
|
Up to four times |
|
A single collection is adequate for the GIS data. These data are descriptive and are not expected to change significantly over the course of the Evaluation. GIS data for clients and families will be abstracted from client admission records.
Other data collections proposed for the Evaluation need to be collected more than once to assess change over time. Successful programs are expected to expand SOCs and improve child and family outcomes.
The data collection fully complies with the requirements of 5 CFR 1320.5(d) (2).
8.a. Federal Register Notice
The 60-Day FRN was published on December 22, 2016 (81 FR 93946). No comments were received.
8.b. Outside Consultations
Both external and internal stakeholders were consulted in the development of these indicators, the data collection methodology, and the associated burden. SAMHSA obtained feedback and consultation regarding the availability of data, methods and frequency of collection, and the appropriateness of data elements. Section B.5 lists the consultants with their contact information and area of expertise based on which they provided consultation for the relevant evaluation component. More information describing outside consultations follows.
Federal Consultation. SAMHSA requires the Center for Substance Abuse Treatment (CSAT), the Center for Substance Abuse Prevention (CSAP), and the Center for Behavioral Health Statistics and Quality (CBHSQ) review of the Annual Report to Congress on the Evaluation of the Comprehensive Community MH Services for Children and Their Families Program, for which the Evaluation must provide a supplemental chapter each year.
Expert Consultation. The CMHI National Evaluation Team includes and has consultation agreements with experts in areas relevant to the Evaluation, including child MH services research, child and family psychology, SOCs, program evaluation, measurement, quantitative and qualitative analysis, economics, web site development and usability testing, and research within Native American communities. In addition, the evaluation team includes individuals who were involved with prior National Evaluation of CMHI and prior collection of GPRA data.
Grantee Consultation. Previously funded grantees have provided input for the Evaluation. Grantees were involved in the pilot-testing of the Key Partner interviews, SOCESS, Network Analysis Surveys, and the Financial Mapping as described in Section B4. Additional input regarding evaluation processes and data utilization will be sought from grantees through conference calls with the CMHI National Evaluation Team and regular contacts with individual site liaisons.
Youth and Family Consultation. The National Federation of Families forms an integral part of the CMHI National Evaluation Team and actively participated in the development of all instrumentation. The Federation also helped develop data collection procedures and training resources. Similarly, Youth M.O.V.E. has been involved in all aspects of planning the Evaluation, including development of instruments, procedures and training materials. These contributions helped ensure sensitivity to parent and youth issues and concerns, maximized clarity of meaning, and strengthened the feasibility of administering the questionnaires. Both groups will continue to be involved in all aspects of this Evaluation.
No monetary incentives are provided to respondents.
Westat has already obtained IRB approval of all data collection tools and to conduct the Evaluation. Further, SAMHSA will conform to all requirements of the Privacy Act of 1974, under the System of Records: Alcohol, Drug, and MH Epidemiological, and Biometric Research Data, DHHS, #09–30–0036; the most recent publication in the Federal Register occurred on January 19, 1999 (64 FR 2914).
All members of the CMHI National Evaluation shall receive general awareness training and role-based training, commensurate with the responsibilities required to perform the tasks of the project. SAMHSA will be responsible for ensuring that each member of the team has completed the SAMHSA Security Awareness Training as required by the agency, as well as Human Subjects Research Training prior to performing any project work or accessing any system, and on an annual basis thereafter, throughout the period of the project. SAMHSA will maintain a list of all individuals who have completed these trainings and shall submit this list to the Project Officer upon request. As a part of this training, SAMHSA shall ensure that all staff read, agree to, and sign the HHS Rules of Behavior. The CMHI National Evaluation Team shall also ensure that all staff have the required level of security clearance commensurate with the sensitivity of the information being stored, processed, transmitted or otherwise handled by the System or required to perform the tasks of the project. At the minimum, all members of the team shall be subjected to a Public Trust background check and be granted a Public Trust clearance before access to the System or other HHS resources is granted.
The CMHI National Evaluation Team shall make efforts to guard the names of respondents, all information or opinions collected in the course of interviews, and any information about respondents learned incidentally during the project. Hard copies of survey data and notes containing personal identifiers shall be kept in a locked containers or a locked room when not being used. Reasonable caution shall be exercised in limiting access to data to only those persons who are working on the project and who have been instructed in appropriate Human Subjects requirements for the project.
Only authorized users, which include grantees, GPOs, Branch Chiefs, Division Directors, the Contract Officer’s Representative (COR) and a small number of the CMHI National Evaluation Team will have authorized access to the main modules of the CMHI National Evaluation Portal. To enter the restricted sections of the site, users must successfully login with their credentials. The CMHI National Evaluation Team and grantees are responsible for entering, reviewing, and modifying performance data in the Data Entry section. The CMHI National Evaluation Team’s System Administrators have additional system rights, which include posting announcements, setting up programs and grants, and adding/updating users’ accounts. Account and administrative sections houses information specific to Grantees and Programs; this information may be sensitive, and is therefore password-protected. All Evaluation team members having access to system components or data are authorized for such access. Access to system information is controlled by creating/removing accounts and access groups, assigning rights to accounts and access groups, assigning accounts to access groups, granting access through physical access controls, and granting permission for access, transport or storage of information.
Identifying information such as individual names and addresses will not be part of any machine data record. Electronic files and audio files will be accessible only to project staff and under password protection. Access to network-based data files is controlled through the use of Access Control Lists or directory- and file-access rights based on user account ID and the associated user group designation, which is maintained by the system administrator. Access control on the PC is achieved for the most part by sound file management procedures by each user. Staff is instructed on the proper use of PCs for the storage, transfer, and use of sensitive information and the tools available, such as encryption.
This data collection involves three general sources of data: 1) clients and families; 2) other stakeholders asked to respond based on their professional roles, not their personal thoughts or feelings; and 3) administrative data. Informed consent forms and/or scripts are included in the attachments along with the corresponding instrument.
Clients and families. Client records at the sites are also covered under the aforementioned Privacy Act System of Records. Client and family data will be collected for the 1) child and family outcome component and 2) GIS Component at the child and family level. In addition, there will be a short network analysis survey administered to children aged 11 and over and their caregivers as part of the child and family interviews conducted by agency administrative staff. These questions will be elective, not required.
Child and family outcome component. Access to the CMHI National Evaluation portal will be password protected and data encryption will be used to enhance security. No information that can potentially be used to identify a client will be included in these data files other than the child/youth/young adult’s unique evaluation identification number (referred to hereafter as the unique ID). No member of the CMHI National Evaluation Team will ever have access to information that can link the unique ID to personal identification information. Further, the project will operate under an ADP/IT security plan approved by CMHS for project data.
The Evaluation requires collecting descriptive and outcome data from children, youth, young adults and caregivers. Each grantee will be strongly encouraged to obtain local IRB approval for the informed consent and assent procedures and data collection activities they perform for this Evaluation with children and their families. In addition, grantees will be encouraged to obtain a Certificate of Confidentiality, authorized by Section 301(d) of the Public Health Service Act. This certificate will keep the data private to the extent provided by law, protecting the investigator(s) from civil and criminal subpoena to identify participants in court. As noted previously, consent forms and/or scripts are included in the attachments along with the corresponding instrument.
Each grantee will develop and implement an active informed consent procedure that informs the participants of the purpose of the Evaluation, describes what their participation entails, and addresses the security measures described above. In addition, respondents will be informed that their participation is voluntary, that they have the right to discontinue participation at any time without impacting services they receive, and of the risks and benefits of participation. Informed assent will be obtained from participating older children and adolescents (age 11–17 years). In addition, informed consent will be obtained from young adults age 18 and older. Written informed consent or assent will be obtained from children and families at the point of entry into services and before the collection of evaluation data. Grantees are instructed to determine whether updates to consents are required at each data collection point, as the legal custody of a child may change, a child may become old enough to participate in a youth interview, a youth may become an emancipated minor or age up into adult status, and local IRBs may have requirements for regular updates.
In all grantee sites, child/youth and family outcome component data are collected by site staff. These staff members are responsible for developing procedures to guard Evaluation data during data collection, storage of data, and reporting of all information obtained through data collection activities. These procedures include limiting the number of individuals who have access to identifying information, using locked files to store hardcopy forms, assigning unique IDs to each participant to ensure anonymity, and implementing guidelines pertaining to data reporting and dissemination.
There will be an optional a short network analysis survey administered to children aged 11 and over and their caregivers as part of the child and family interviews conducted by agency administrative staff. These questions will be elective, not required.
GIS Component at the child and family level. Site staff will use the clients’ address obtained from their clinical record to determine the Census block group of their residence. (The CMHI National Evaluation Team will provide sites with a program to convert home addresses to Census block group designation.) The CMHI National Evaluation Team will only receive Census block group data, not home addresses.
Other Partners in their Professional Roles. Most types of data collection activities in the Evaluation involve professional partners, including child and family advocacy representatives, as respondents. Data collection will be conducted and/or coordinated by CMHI National Evaluation staff. These data collection activities will be conducted by 1) telephone interviews or 2) web-based surveys (via the CMHI National Evaluation portal). Specifics related to each of these data collection activities are discussed here.
Telephone Interviews. Respondents’ identities will be known (for all but youth and caregiver respondents), so to ensure participants’ rights, an active informed consent process will occur. CMHI National Evaluation staff will obtain verbal consent for telephone interviews using the consent scripts included in the attachments with the corresponding instrument. Data collection activities falling into this category include the following:
Key partner interviews,
Financial Mapping Interview, and
Some telephone interviews will be audio recorded. Separate informed consent will be obtained to audiotape these interviews. As mentioned previously, audio recordings will be destroyed at the end of the Evaluation, in consultation with the Westat IRB.
Web-based Surveys. As described previously, the CMHI National Evaluation Portal will be used for web-based data collection. Data collection activities falling into this category include the following:
Network Analysis Surveys.
SOCESS
Respondents’ identities will be known, so to ensure participants’ rights, an active informed consent process will occur. Potential participants will be contacted by mail, email, or telephone to explain the survey. Survey explanation will include the voluntary nature of survey completion, treatment of responses, and the risks, benefits, and rights as respondents. Participants will be asked to indicate, by checking a box on the Web survey that they agree to participate in the Evaluation before they complete and submit the survey. Information about the Evaluation and participant rights will be presented in the Web survey prior to the check box indicating consent to participate. The letter and the Web survey will also provide contact information if the survey recipient has questions or desires clarification prior to participation. If the individual does not have Internet access, alternative administration methods will be used such as 1) a packet sent by regular mail containing a cover letter, an informed consent form, a survey, and a return envelope (the cover letter will indicate that the respondent is to return the informed consent form and the completed survey in separate envelopes enclosed in the packet) or 2) the survey will be administered by telephone interview following the procedures detailed previously.
Administrative Data. When electronic data are transferred to the CMHI National Evaluation, data files will be encrypted to make the information indecipherable during electronic transfer. Data will be transmitted securely and all caution will be used, as described in Section A.3, Use of Information Technology. The term, administrative data, is used loosely here to refer to Financial Mapping and Benchmark Data.
Financial Mapping and Benchmark Data, Financial Planning Interviews. For these data analysis activities, CMHI National Evaluation staff will request deidentified financial service and other cost data. Westat IRB approval will be obtained for all data collection activities, including these, prior to commencing data collection.
11. Questions of a Sensitive Nature
In that this project pertains to services to children with SED and their families, it is necessary to ask questions that are potentially sensitive as part of the Child/Youth and Family Outcome Component. However, only information that is central to the Evaluation is being sought. Some questions asked of children, young adults, and caregivers address dimensions such as child emotions, behavior, social functioning. In addition, young adults age 18 to 26 will be asked about their experience with violence and trauma, but no other respondents will be asked these questions. (The violence and trauma items were taken from the TRAC adult instrument already approved by OMB (OMB No. 0930-0285). The answers to these questions will be used to determine baseline status and to measure change in these areas experienced after receiving SOC services. Each grantee must keep data on child and family status and service use, as well as treatment records and other related information. For these reasons, the data collection required for the Evaluation is not introducing new, sensitive domains of inquiry, but is paralleling standard procedures in the field of children’s MH.
In addition to information on child clinical status and social functioning, other questions of a sensitive nature will be asked of families. These include questions related to caregiver strain associated with raising a child with SED. These questions are included in order to assess the extent to which caregiver strain is reduced after the child/youth and family receive SOC services. Moreover, family representatives who have consulted with the CMHI National Evaluation consistently identify a lack of information on family outcomes as a weakness in previous evaluations and studies.
Only aggregated information will be disseminated. The primary dissemination vehicles are the Report to Congress and the Report to the Secretary of HHS, though there may be other dissemination efforts as well. SAMHSA will be careful to avoid disseminating small numbers with demographic information that might be used to deduce the identity of individual respondents. Specific rules used to avoid such dissemination will be determined in consultation with CMHI National Evaluation statisticians, but typically involve suppressing data where a small number of respondents would otherwise appear in a table shell or text.
12. Estimates of Annualized Burden Hours and Costs
Exhibit 1 shows the estimated annualized burden hours for the respondents’ time to participate in the instrument/data collection activity. Across the instruments, the total annualized burden is estimated to be 4,330 hours. Exhibit 2 shows the estimated annualized cost burden associated with the participants’ time to take part in this research. The total cost burden is estimated to be $228,525.
Exhibit 1. Estimated annualized burden hours
Instrument / Data Collection Activity |
Number of Respondents |
Total Number of Responses per Respondent |
Average Annual Burden (Hours) |
Key Partner Interview |
462 |
924 |
339 |
SOCESS |
1422 |
5688 |
948 |
Network Analysis Survey |
690 |
1380 |
230 |
Financial Mapping Interview |
225 |
450 |
95 |
Benchmark Tool |
12 |
24 |
320 |
Financial Planning |
54 |
162 |
32 |
Child and family instruments |
9242 |
27,726 |
2366 |
Total |
12,107 |
36,354 |
4,330 |
Exhibit 2. Estimated annualized cost burden
Instrument/ Data Collection Activity |
Respondent |
Number of Respondents |
Responses per Respondent |
Total Number of Responses |
Hours per Response |
Total Annual Burden Hours |
Average Hourly Wagea |
Total Cost |
Implementation Assessment |
|
|
|
|
|
|
|
|
Key Partner Interviews |
Project Director |
84 |
2 |
168 |
1.5 |
252 |
$29.83 |
$7,517 |
Family Organization Representative |
54 |
2 |
108 |
1.5 |
162 |
$22.47 |
$3,640 |
|
Youth Organization Representative |
54 |
2 |
108 |
1.5 |
162 |
$22.47 |
$3,640 |
|
MH Agency Director |
54 |
2 |
108 |
1.5 |
162 |
$29.83 |
$4,832 |
|
Core Agency Partnersb |
162 |
2 |
324 |
0.75 |
243 |
$29.83 |
$7,249 |
|
Quality Monitor |
54 |
2 |
108 |
0.33 |
36 |
$29.83 |
$1,063 |
|
SOCESS |
Project Director |
84 |
4 |
336 |
0.5 |
168 |
$29.83 |
$5,011 |
Family Organization Representative |
108 |
4 |
432 |
0.5 |
216 |
$22.47 |
$4,854 |
|
Youth Organization Representative |
108 |
4 |
432 |
0.5 |
216 |
$22.47 |
$4,854 |
|
Core Agency Partners |
432 |
4 |
1,728 |
0.5 |
864 |
$29.83 |
$25,773 |
|
Practitioners |
690 |
4 |
2,760 |
0.5 |
1,380 |
$22.47 |
$31,009 |
|
Network Analysis Survey |
|
|
|
|
|
|
|
|
Network Analysis Survey |
Key Agency Partners |
690 |
2 |
1,380 |
0.5 |
690 |
$29.83 |
$20,583 |
Financial Mapping and Benchmark Components |
|
|
|
|
|
|
|
|
Financial Mapping Interview |
Financial administrators at: Medicaid Agencies & MH Authorities |
108 |
2 |
216 |
0.75 |
162 |
$29.83 |
$4,832 |
Financial administrators at: Trade associations & Family organizations |
108 |
2 |
216 |
0.5 |
108 |
$29.83 |
$3,222 |
|
Tribal Financial Administrators |
9 |
2 |
18 |
0.75 |
14 |
$29.83 |
$403 |
|
Benchmark Tool |
Payment personnel at Medicaid Agencies & MH Authorities |
12 |
2 |
24 |
40 |
960 |
$22.47 |
$21,571 |
Financial Plan Interviews |
Financial Planning Directors |
54 |
3 |
162 |
0.6 |
97 |
$29.83 |
$2,899 |
Exhibit 2. Estimated annualized cost burden (continued) |
||||||||
Child and Family Outcome Component |
|
|
|
|
|
|
|
|
Administrative Measures |
Caregivers of clients age 0-17c |
4,136 |
1 |
4,136 |
0.05 |
207 |
$11.60 |
$2,399 |
Clients age 11-26 |
1685 |
1 |
1,685 |
0.05 |
84 |
$7.25 |
$611 |
|
Client Functioning |
Caregivers of clients age 0-17c |
4,136 |
3 |
12,408 |
0.15 |
1,861 |
$11.60 |
$21,590 |
Clients age 11-26d |
970 |
3 |
2,910 |
0.15 |
437 |
$7.25 |
$3,165 |
|
Caregiver Strain Questionnaire |
Caregivers of clients age 0-17c |
4,136 |
3 |
12,408 |
0.15 |
1,861 |
$11.60 |
$21,590 |
Columbia Impairment Scale |
Caregivers of clients age 5-17e |
2,859 |
3 |
8,577 |
0.08 |
686 |
$11.60 |
$7,959 |
Clients age 11-26d |
2,655 |
3 |
7,965 |
0.08 |
637 |
$7.25 |
$4,620 |
|
Pediatric Symptom Checklist-17 |
Caregivers of clients age 5-17e |
2,859 |
3 |
8,577 |
0.05 |
429 |
$11.60 |
$4,975 |
Clients age 11-26d |
2,655 |
3 |
7,965 |
0.05 |
398 |
$7.25 |
$2,887 |
|
New Tools in 2015 |
|
|
|
|
|
|
|
|
Brief Infant and Toddler Emotional Assessment (BITSEA) |
Caregivers of children and youth 0 to 5 years of agef |
1,277 |
3 |
3,831 |
0.08 |
306 |
$11.60 |
$3,555 |
Baby Pediatric Symptom Checklist (BPSC) |
Caregivers of children and youth for ages 1 month to 18 monthsf |
638 |
3 |
1,914 |
0.05 |
96 |
$11.60 |
$1,110 |
Preschool Pediatric Symptom Checklist (PPSC) |
Caregivers of children and youth for ages 18 months to 66 monthsf |
639 |
3 |
1,917 |
0.05 |
96 |
$11.60 |
$1,112 |
Total Annual Burden |
|
|
|
|
|
|
|
|
All |
All g |
12,107 |
|
36,354 |
|
12,990 |
|
$228,525 |
Based on the average hourly wages for Community and Social Service Specialists, All Other (21-1099; $22.47) and Social Workers (21-1020; $29.83) from the May 2015 National Industry-Specific Occupational Employment and Wage Estimates, 621330 - Offices of Mental Health Practitioners; the Federal minimum wage of $7.25; and an estimated average hourly wage of $11.60 for a family of four living 25% below poverty level.
Core agency partners include (1) representatives from MH, child welfare, and juvenile justice and (2) CMHI quality monitors.
Assumes 81% of clients will be age 0 to 17.
Assumes 52% of clients will be age 11 to 26.
Assumes 56% of clients will be age 5 to 17.
Assumes 25% of clients will be age 0 to 5, with 12.5% of clients age 0 to 2.5, and 12.5% age 2.6 to 5).
Sums shown indicate unduplicated respondents and responses per respondent.
There are neither capital nor startup costs, nor are there any operations or maintenance costs.
SAMHSA has planned and allocated resources for the management, processing, and use of the collected information in a manner that shall enhance its utility to agencies. The contract award to cover evaluation of this project is $6,661,621 over a 60-month period. Thus, the annualized contract cost is $1,332,324.
Additional costs will be incurred indirectly by the government in personnel costs of staff involved in oversight of data collection. It is estimated that one SAMHSA employee will be involved for 15 percent of their time. Cost of staff time will approximate $13,500 annually.
The estimated annualized total cost to the government will be $1,345,824.
This is a new data collection.
a. Time Schedule
The time schedule for implementing the Evaluation and publishing findings in the supplement to the Annual Report to Congress is summarized in Table 3. A 3-year clearance is requested for this project.
Table 3. Schedule of Evaluation activities
Activity |
Date |
Receive OMB clearance for data collection |
February or March 2017 |
Begin data collection 2 months after OMB clearance |
February or March 2017 |
Stop data collection |
June 30, 2020 |
Begin processing and analyzing data |
6 months after OMB clearance |
Produce supplement/Annual Report to Congress |
Every October from 2016 to 2020 |
b. Data Analysis Plan
Data analyses are described for each Evaluation component followed by a description of cross-level analyses.
Implementation Assessment
Key Partner Interviews. Using a pre-established analysis frame, coders will document descriptive information about what implementation and expansion strategies are being used by each grantee and how well or fully they have been implemented. Multiple coders will be trained, and inter-rater reliability will be tested to ensure consistency in identifying and recording strategies.
Qualitative analysis of interview data will be conducted to comprehensively describe implementation and expansion plans and efforts. SAMHSA will describe: planned and implemented strategies and mechanisms; participants’ involvement in the planning and implementation process; role of child-serving sectors, youth groups, and family organizations; funding mechanisms; efforts to reach vulnerable and hard-to-reach populations; support of local systems in direct service delivery; and implementation barriers and facilitators encountered. Finally, findings will be compared across grantees to identify commonly employed strategies, as well as those that are potentially innovative. The SOC principles most and least often included in the development of implementation and expansion efforts will also be described.
System of Care Expansion and Sustainability Survey (SOCESS). Analysis of SOCESS data will focus on grantees’ efforts to implement and expand the SOC. Areas described in the Key Partner Interviews will be assessed quantitatively in the SOCESS. Analyses will also explore what facilitators and impediments to SOC implementation and expansion were encountered. Measurement quality of the SOCESS will be examined using estimates of reliability (e.g., internal consistency) and factor analysis. Descriptive statistics (e.g., mean, standard deviation) will be calculated for individual items, and multi-level longitudinal analyses (e.g. hierarchical mixed models and/or structural equation modeling) will be used to examine change over time across all dependent variables, with respondents nested within grantees. To the extent that groups of items appear to assess the same construct, summary scores may be calculated (e.g., means or sums). Grantees will receive their own scores, as well as the combined scores of all other grantees for comparison purposes.
Network Analysis
Increasing connections among agencies and organizations, and spreading SOCs to new geographic areas are key goals of SOC implementation and expansion efforts. Network analysis provides an assessment of relationships among agencies and organizations
Network Analysis.
The network analysis survey is designed to reach administrative leaders and directors of key agencies and organizations within a system of care who are responsible for SOC implementation and expansion efforts. SAMHSA will assess the composition of partners within the system and assess their cross-agency interactions related to direct service provisions to children/youth and their families.
Network analysis generates graphic representations depicting the relationships between and among grantee partner agencies and organizations. Network analysis generates numerical computations of the strength of these cross-agency relationships or links between and among participants. These data will be used to describe the network in terms of density (i.e., how sparsely or closely are participants connected), centralization (i.e., whether there are one or more partner agencies around which most of the others tend to gather), fragmentation (i.e., whether many system participants appear isolated or whether the system made up of small clusters that are unconnected to each other), and coordination.
Financial Mapping and Benchmark Component
Financial Mapping. For each state, county, or tribe included in the analysis cohort, SAMHSA will compare the number and types of state and federal funding sources for state children’s MH and SOC services by level of MH care during the first or second grant year to the funding sources used two years later. Specifically, from the information collected through the Financial Mapping Interview and review of administrative data, the CMHI National Evaluation Team will create a map. The map will be in the form of a matrix showing for each level of care in the children’s MH system that identifies (1) any applicable income or clinical eligibility criteria for the children’s MH services; (2) the continuum of services; (3) the federal, state, county, tribal, or commercial health plan funding source or sources of funds; and (4) the State, county or tribal agency through which the services are funded. Respondents may also identify the services and resources provided by other state and local agencies.
Benchmark Component. The voluntary benchmark component will use MH Authority and Medicaid Agency data to compare states’ rate of penetration, utilization and expenditures for children’s MH services by level of MH care. All indicators will be calculated with detailed specifications for numerators (i.e., children served, utilization and costs by type of service) and denominators (i.e., populations covered). SAMHSA will identify spending patterns that may indicate shifts in costs, and cost savings or cost offsets, particularly in inpatient and residential treatment services. Examples of variables that will be calculated from requested information include penetration rates, inpatient days, residential days, emergency room use, outpatient visits per 1000, users of MH care coordination FTEs per 1000, 30-day readmission rates, data on the use of family and youth partners or mentors, and in-home service utilization. The denominator for penetration and rates per 1000 for Medicaid will be health plan members in the relevant age group and for MH Authorities will be US census estimates of child population in the grantee area. Data will be collected from the MH Authority and for Medicaid funds.
Financial Planning Interviews. Three new interview tools have been developed to collect data on the strategic financial planning that grantees are required to perform as a condition of the grant. In Year Two, a brief interview will collect information about how grantees are approaching this task. In Year Three, after the Financial Plans are submitted, a longer and more detailed interview about the planning process will be conducted. Some of the information requested may be provided in the planning document itself. These plans will be reviewed and will prepopulate the interviews with available data to avoid requesting information that has already been provided. Finally, in the last year of the grant, grantees will be interviewed about the achievement of the goals of the plan and what has facilitated goal achievement and what has created barriers.
Child and Family Outcome Component
The CMHI National Evaluation Team will use univariate descriptive analyses to characterize children, youth, and families being served through CMHI SOC grants. Descriptive information of Evaluation participants will include score ranges, means, and medians. Results will be reported for each grantee as well as for all grantees combined. Bivariate analyses will be performed to find correlations between child, youth, and family characteristics (e.g., relationships between family income and caregiver strain at intake).
The longitudinal design assesses whether individual children and families experience meaningful improvements in outcomes during treatment. Changes over time in child and caregiver satisfaction with services will be tested using descriptive analyses (e.g., frequencies and percentages) as well as univariate and bivariate analyses (e.g., are there differences in satisfaction by clinical severity). Change in child and family outcomes over time will be tested using a variety of techniques. Repeated measures analysis of variance (ANOVA) will be used to test the significance of change over time within and between groups, both across and within grantees. Repeated measures analysis of covariance (ANCOVA) will be conducted as well, to examine the influence of covariates. Path analysis and other structural equation modeling techniques will be used to investigate the direct and indirect effects of independent variables (such as, type and amount of services received, and demographic variables) on dependent outcome measures (such as symptom severity, social functioning, and caregiver strain). Structural equation modeling will prove particularly useful for sub-analyses of data from older youth in which latent constructs (e.g. youth functioning) have ratings from both youth and caregivers). Multi-level modeling (MLM) will be used to estimate growth curves (e.g., changes in the level of symptomatology) at the individual level based on repeated observations.
Cross-Level Analyses
Two sets of additional analyses are planned. The first set of analyses will focus on how the quality of the SOC expansion implementation as measured by one set of instruments relates to that measured by others, e.g., changes in (1) SOCESS scores and (2) system network integration such as network density, centrality, fragmentation, and coordination will be analyzed relative to the Grantee’s implementation quality (based on data from the web-based SOCESS), using MLMs.
The second set of analyses will use a series of MLMs to evaluate child, youth, young adult, and family outcomes based on the extent to which system-level expansion quality (as indicated by SOCESS scores) is related to improved child, youth, young adult, and family functioning. SAMHSA will explore whether particular domains on the SOCESS are related to specific child and family outcomes.
Impact of System of Care Implementation and Child and Family Outcomes. In addition to the longitudinal and descriptive analyses discussed for the Child and Family Component, SAMHSA also propose using MLM to explore whether system performance, as indicated by the SOCESS, is related to child and family outcomes. It allows time-varying assessments (e.g., if the three-month data collection is skipped but the six-month follow-up data is collected). SAMHSA will use standard model-building approaches focusing on creating parsimonious models. This general approach will provide flexibility for testing a variety of important research questions. SAMHSA will explore whether particular dimensions assessed with SOCESS (e.g., quality of individualized service provision) are related to specific child and family outcomes (e.g., reduction in child symptom severity). Of particular interest will be examining whether scores on implementation of EBTs are related to improvements in child symptoms and functioning.
All data collection instruments will display the expiration date of OMB approval.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Wendy Kissin |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |