Project LAUNCH MSE_OMB-SSA_100317v2 clean

Request for Clearance for

the Multi-Site Evaluation of Project LAUNCH

Supporting Statement A

OMB Information Collection Request

OMB #0970-0373

Submitted by:

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

U.S. Department of Health and Human Services (HHS)

330 C Street, SW

Washington, DC 20201

Project Officers:

Laura Hoard (ACF/OPRE) and Kelley Smith (SAMHSA)

October 2016

Updated June 2017 and October 2017

TABLE OF CONTENTS

A. Justification 5

A1. Necessity for the Data Collection 5

A2. Purpose of Survey and Data Collection Procedures 6

A3. Improved Information Technology to Reduce Burden 18

A4. Efforts to Identify Duplication 19

A5. Involvement of Small Organizations 19

A6. Consequences of Less Frequent Data Collection 19

A7. Special Circumstances 20

A8. Federal Register Notice and Consultation 20

A9. Incentives for Respondents 22

A10. Privacy of Respondents 26

A11. Sensitive Questions 28

A12. Estimation of Information Collection Burden 29

A13. Cost Burden to Respondents or Record Keepers 32

A14. Estimate of Cost to the Federal Government 32

A15. Change in Burden 32

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication 32

A17. Reasons Not to Display OMB Expiration Date 34

A18. Exceptions to Certification for Paperwork Reduction Act Submissions 35

LIST OF EXHIBITS

EXHIBIT 1: Project LAUNCH MSE Study Structure

Exhibit 2: Project LAUNCH MSE Instruments, Annual Number of Respondents, Type of Administration, and Periodicity

EXHIBIT 3: Status of Previously Approved Data Collection Instruments in New OMB Request

Exhibit 4: Crosswalk of Project LAUNCH Research Questions and Instruments/Sources

Exhibit 5: Conceptual Elements Measured in the Parent Survey and the Corresponding Survey Instrument from which They Were Drawn

Exhibit 6: DATA COLLECTION EFFORTS AND OUTCOMES of interest CROSSWALK

Exhibit 7: mEMBERSHIP OF project LAUNCH Consultant Cadre

EXHIBIT 8: Proposed Structure for INCENTIVES for the Multi-site Evaluation of Project LAUNCH

Exhibit 9: Estimated Annual Response Burden and Cost for Newly Requested Data Collection

LIST OF ATTACHMENTS

Attachment A: crosswalk of mse Part A aND cse surveyS

Attachment B: Project LAUNCH Direct Services Survey

Attachment C: Project LAUNCH Systems Activities and Outcomes Survey

Attachment D: Project LAUNCH School Survey

Attachment E: PROJECT launch PARENT SURVEY, 4 WEEKS-18 MONTHS

Attachment F: pROJECT launch Parent survey, 18 months-3 years

Attachment G: pROJECT launch Parent survey, 3-5 years

Attachment H: pROJECT launch Parent survey, 5-13 years

Attachment I: Teacher survey (edi)

Attachment J: Project LAUNCH Key Informant Interview on Systems Change Discussion Protocol, LAUNCH Community

Attachment K: Project LAUNCH Key Informant Interview on Systems Change Discussion Protocol, Comparison Community

Attachment L: Project LAUNCH Research questions, constructs, and operational definitions

ATTACHMENT M: PROJECT LAUNCH 30 DAY FEDERAL NOTICE

Attachment N: Project LAUNCH recruitment materials for school districts

Attachment N.1: Project LAUNCH School District Recruitment Letter: LAUNCH

Community

Attachment N.2: Project LAUNCH School District Recruitment Letter: Comparison

Community

Attachment N.3: Project LAUNCH Fact Sheet: Surveys to be Collected (for Schools)

Attachment N.4: Project LAUNCH Teacher Survey (EDI) Fact Sheet

Attachment N.5: Project LAUNCH Program Information for Recruitment Packet

Attachment N.6: Project LAUNCH Parent Information Flyer for Schools

Attachment N.7: Project LAUNCH School District Recruitment Script

Attachment O: project launch recruitment materials for schools

Attachment O.1: Project LAUNCH School Recruitment Letter: LAUNCH Community

Attachment O.2: Project LAUNCH School Recruitment Letter: Comparison Community

Attachment O.3: Project LAUNCH Fact Sheet: Surveys to be Collected (for Schools) Attachment O.4: Project LAUNCH Teacher Survey (EDI) Fact Sheet

Attachment O.5: Project LAUNCH Program Information for Recruitment Packet

Attachment O.6: Project LAUNCH Parent Information Flyer for Schools

Attachment O.7: Project LAUNCH School Recruitment Script

Attachment O.8: Project LAUNCH School Coordinator Recruitment Script

Attachment P: project launch recruitment materials for eces

Attachment P.1: Project LAUNCH ECE Recruitment Letter: LAUNCH Community

Attachment P.2: Project LAUNCH ECE Recruitment Letter: Comparison Community

Attachment P.3: Project LAUNCH Fact Sheet: Surveys to be Collected (for ECEs)

Attachment P.4: Project LAUNCH Program Information for Recruitment Packet

Attachment P.5: Project LAUNCH Parent Information Flyer for ECEs

Attachment P.6: Project LAUNCH ECE Recruitment Script

Attachment P.7: Project LAUNCH ECE Coordinator Recruitment Script

ATTACHMENT Q: COMMUNICATION WITH PROJECT LAUNCH SCHOOL SURVEY RESPONDENTS

Attachment Q.1: Project LAUNCH School Survey: Key Information Requested

Attachment Q.2: Project LAUNCH School Survey Initial Email

Attachment Q.3: Project LAUNCH School Survey Nonresponse Prompt

ATTACHMENT R: parent information collection template for school/ece coordinators

Attachment S: COMMUNICATION WITH PROJECT LAUNCH PARENTS

Attachment S.1: Project LAUNCH Parent Information Flyer for Schools

Attachment S.2: Project LAUNCH Parent Information Flyer for ECEs

Attachment S.3: Project LAUNCH Parent Survey Flyer Follow-Up Email

Attachment S.4: Project LAUNCH Parent Survey Initial Email

Attachment S.5: Project LAUNCH Parent Survey Nonresponse Prompt

Attachment S.6: Project LAUNCH Parent Survey Thank You Email

Attachment S.7: Project LAUNCH Parent Survey Check-In Email

Attachment S.8: Project LAUNCH Teacher Survey (EDI) Parental Consent Letter

ATTACHMENT T: Teacher Survey (EDI) DEMOGRAPHIC DATA EXPORT INSTRUCTIONS

ATTACHMENT U: Teacher Survey (EDI) DEMOGRAPHIC DATA EXPORT FORM

ATTACHMENT V: COMMUNICATION WITH PROJECT LAUNCH TEACHER SURVEY (EDI) RESPONDENTS

Attachment V.1: Project LAUNCH Teacher Survey (EDI) Fact Sheet

Attachment V.2: Project LAUNCH Teacher Survey (EDI) Thank You Email

ATTACHMENT W: PROJECT LAUNCH RECRUITMENT MATERIALS FOR KEY INFORMANT INTERVIEWS ON SYSTEMS CHANGE

Attachment W.1: Project LAUNCH Key Informant Interview on Systems Change

Recruitment Letter: LAUNCH Community

Attachment W.2: Project LAUNCH Key Informant Interview on Systems Change

Recruitment Letter: Comparison Community

Attachment W.3: Project LAUNCH Program Information for Recruitment Packet

Attachment W.4: Project LAUNCH Key Informant Interview on Systems Change

Recruitment Script

Administration for Children & Families (ACF)
Multi-Site Evaluation of Project LAUNCH

Supporting Statement

A. Justification

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (DHHS) seeks approval for collection of data to conduct the multi-site evaluation (MSE) of Project LAUNCH (Linking Actions for Unmet Needs in Children’s Health). OMB has approved a previous data collection package for Project LAUNCH (OMB Control # 0970-0373). This current request is intended to: 1) modify selected previously approved items, and 2) introduce new measures and methods.

The Project LAUNCH Multi-Site Evaluation (MSE) was initially funded by an Interagency Agreement between the Substance Abuse and Mental Health Services Administration (SAMHSA) and ACF. This contract, which SAMHSA now funds directly, was awarded to NORC at the University of Chicago and includes the design of and data collection for the MSE. While we propose to utilize some items that are similar to those that have already been approved, the research proposed in this package seeks to evaluate the impact of Project LAUNCH at the community level—the intended focus of the program—and relies on instruments and measures that differ from those used in the previous stage of the evaluation.

Study Background

The purpose of Project LAUNCH is to promote healthy development and wellness in children from birth to eight years of age. Project LAUNCH addresses issues in the child services system by enhancing systems coordination, integrating children’s behavioral health services with other health services, and implementing evidence-based programs to promote children’s healthy development. To accomplish these goals, Project LAUNCH utilizes five core strategies: 1) mental health consultation in early care and education settings; 2) developmental assessments across service settings; 3) integration of behavioral health into primary care; 4) family strengthening and parenting skills training; and 5) enhanced home visitation. Project LAUNCH grantees are organized into cohorts as a function of the timing of their program funding. Thus far, grantees include the following:

• 6 grantees in Cohort 1 funded in 2008

• 12 grantees in Cohort 2 funded in 2009

• 6 grantees in Cohort 3 funded in 2010

• 11 grantees in Cohort 4 funded in September 2012

• 5 grantees in Cohort 5 funded in September 2013

• 15 grantees in Cohort 6 funded in September 2014

The grantees focus on systems-level development at the state or tribal level and in one designated community. Additionally, within the designated or “local” community, all grantees are required to implement promotion and prevention activities within each of the five core strategies. Project LAUNCH is authorized under Section 520A of the Public Health Service Act (42 U.S.C. 290bb–32) and addresses the Healthy People 2020 Topic Area 18 for Mental Health and Mental Disorders.

The information to be collected during Project LAUNCH MSE is designed to examine the impact of Project LAUNCH at the community level and the mediators of that impact. The MSE will include both Project LAUNCH and comparison communities.

Legal or Administrative Requirements That Necessitate the Collection

There are no legal or administrative requirements that necessitate the data collection activities. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

From within each of the five core LAUNCH strategies, Project LAUNCH allows funded grantees to select evidence-based interventions that will best serve their communities based on the specific community’s population and needs. Project LAUNCH grantees also engage in systems-focused work at the local and state levels, including the coordination of systems and services for young children and their families as well as policy development efforts. The Project LAUNCH MSE seeks to determine the impact of LAUNCH on communities with respect to: 1) child and family outcomes, and 2) systems-level outcomes.

The prior cross-site evaluation (CSE) of Project LAUNCH, conducted by Abt Associates, was a related study that collected primarily process information. The MSE data collection effort will include analyses of both existing CSE data that grantees entered in the previous data portal and future MSE data. The current MSE aims to provide outcome information that will extend and supplement the previous data as well as provide an examination of comparison communities and families that are not exposed to LAUNCH.

Research Questions

The data collection activities and instruments for the MSE have been designed to address the following overarching research questions:

Child and Family Outcomes

1. How does Project LAUNCH affect specific individual-level child and family outcomes?

   1. Are children in LAUNCH communities in better social-emotional health than those in comparison communities?

   2. Do families in LAUNCH communities exhibit stronger parent-child relationships than those in comparison communities?

   3. Do parents in LAUNCH communities have higher reported levels of social support than do parents in comparison communities?

   4. Do parents in LAUNCH communities have lower reported levels of depression than do parents in comparison communities?

   5. Do families in LAUNCH communities engage more frequently in specific family activities (e.g., playing games, reading books, arts and crafts) than those in comparison communities?

   6. Are children in LAUNCH communities more likely to be “school ready” than those in comparison communities, as measured on the following domains?

      1. physical health and well-being,

      2. social competence,

      3. emotional maturity,

      4. language and cognitive development, and

      5. communication skills and general knowledge in relation to developmental benchmarks

   7. Are children in LAUNCH communities less likely to be suspended and/or expelled from school than those in comparison communities?

   8. [SECONDARY RESEARCH QUESTION] Are children in LAUNCH communities in better physical health than those in comparison communities after program implementation*?

   9. [SECONDARY RESEARCH QUESTION] Do children in LAUNCH communities use more health-care services such as primary care, mental-health services, preventive care/screening, dental care, and vision care than those in comparison communities?

   10. [SECONDARY RESEARCH QUESTION] Do children in LAUNCH communities have better access to specific health-care services such as primary care, mental-health services, dental care, and vision care than those in comparison communities?

2. Does Project LAUNCH’s impact on the individual-level outcomes in Question 1 vary across the interventions supporting each of the program’s core strategies? These include:

   1. home visiting

   2. mental health consultation in school and Early Childhood Education Center (ECE) settings

   3. integrating behavioral health into primary health care

   4. screening

   5. family strengthening

3. Does the impact of LAUNCH vary by family and community socioeconomic factors?

   1. family education level

   2. race/ethnicity

   3. income

   4. target children’s age

4. Do families in LAUNCH communities fare better on the outcomes included in Question 1 over time (i.e., as a function of time since program implementation)?

Systems-Level Outcomes

5. Are LAUNCH coalition-building activities (e.g., wellness councils) associated with other systems outcomes (such as advocacy activities, funding levels, and others)?

6. Are LAUNCH public information campaigns associated with other systems outcomes (such as advocacy activities, funding levels, and others)?

7. Are LAUNCH advocacy activities associated with other systems outcomes (such as public information campaigns, funding levels, and others)?

8. Are LAUNCH activities that are designed to increase funding associated with other systems outcomes (such as public information campaigns, advocacy activities, and others)?

9. Does the impact of LAUNCH vary by community socioeconomic characteristics?

   1. family education level

   2. race/ethnicity

   3. income

10. What are the facilitators that contribute to improving systems-level outcomes?

    1. Are LAUNCH communities in areas with co-occurring child and family programs more likely than communities without them to achieve improved systems outcomes?

    2. Are LAUNCH communities with schools that emphasize child and family well-being (either through formal programs or other otherwise) more likely than communities without them to achieve improved systems outcomes?

11. What are the barriers that impede improvement of systems-level outcomes?

    1. Are LAUNCH communities in states with larger uninsured populations less likely to achieve improved systems outcomes?

    2. Are LAUNCH communities in states with specific fiscal challenges less likely to achieve improved systems outcomes? (These may include, for instance, large unemployed populations, divided government with respect to party control of executive and legislative branches, and budgetary issues, among others.)

12. What are the economic costs of LAUNCH Program Implementation and how do these compare to the economic benefits related to improvements in child and family outcomes?

    1. How much does LAUNCH cost to implement overall and per child in the designated target population—with the target population identified as a combination of geography (defined ZIP codes for inclusion) and demographics (children living beneath a designated income level)?

    2. What percentage of resources on average do grantees spend on each LAUNCH strategy?

    3. If LAUNCH resulted in changes in child and family outcomes, what are published estimates of the dollar value of these improvements?

    4. What is the total cost of LAUNCH per child in the designated target population after accounting for these cost offsets?

       1. What is the cost per child served by LAUNCH?

       2. What is the cost per child who saw developmental improvement?

       3. If the cost offsets exceed the implementation costs of LAUNCH, what is the return on investment (ROI) of LAUNCH with ROI defined as:

Study Design

To address the research questions detailed above, the MSE will consist of two parts: Part A and Part B. Exhibit 1 presents the overall MSE study structure and timing.

Exhibit 1. Project LAUNCH MSE Study Structure (Start dates dependent on OMB approval)
	Summer 2016	Fall 2016	Winter 2017	Spring 2017	Summer 2017	Fall 2017	Winter 2018	Spring 2018	Summer 2018	Fall 2018
MSE Part A	MSE Portal Design and Survey Programming	Portal Reporting	Analysis	Portal Reporting	Analysis	Portal Reporting	Analysis	Portal Reporting	Analysis	Portal Reporting and Analysis
MSE Part B	OMB Clearance Received
	LAUNCH Site Selection
	Comparison Community Analysis and Matching
			District and School Initial Recruitment			District and School Commitment Renewal
					School Survey				School Survey
						School Survey Analysis				School Survey Analysis
			Parent Recruitment				Parent Recruitment
			Parent Survey Data Collection				Parent Survey Data Collection
				Parent Survey Analysis				Parent Survey Analysis
						Teacher Recruitment
							Teacher Survey (EDI) Data Collection
								Teacher Survey (EDI) Analysis
				Key Informant Recruitment				Key Informant Recruitment
				Key Informant Interviews				Key Informant Interviews
					Key Informant Interview Analysis				Key Informant Interview Analysis

In Part A, all 31 Project LAUNCH grantees in Cohorts 4, 5, and 6 will enter direct-service activities (semi-annually) and systems-level activities and outcomes (annually) into a Web-based data portal. The information collected in this manner will relate to: state, tribal, and community systems development; implementation of evidence-based services in local communities; and service system outcomes for children and families.

Part B will consist of additional data collection conducted in ten Project LAUNCH sites and ten comparison communities. This portion of the MSE involves gathering information directly from parents, teachers, school administrators, and local/state key informants in the selected LAUNCH and comparison communities to examine child, family, and systems-level outcomes.

Exhibit 2. Project LAUNCH MSE Instruments, Annual Number of Respondents, Mode of Administration, and Periodicity
Instrument	Annual Number of Respondents		Mode of Administration	Fall 2016	Winter 2017	Spring 2017	Summer 2017	Fall 2017	Winter 2018	Spring 2018	Summer 2018	Fall 2018
MSE Part A
Direct Services Survey	31		Web-based	X		X		X		X		X
Systems Activities and Outcomes Survey	31		Web-based	X				X				X
MSE Part B
School Survey	120		Web-based				X				X
Parent Survey	1800		Web-based		X	X	X		X	X	X
Teacher Survey (EDI)	160		Web-based						X
Key Informant Interviews on Systems Change	70		Telephone			X	X			X	X

Tribal and territorial grantees will participate in MSE Part A, but will not be included in the sampling frame for MSE Part B. MSE Part B will collect data only from state LAUNCH grantees due to challenges with instrumentation and prior testing in similar communities, sensitivities around collecting individual-level data from tribal members, the lack of appropriate comparison communities for tribal and territorial communities, and budgetary realities.

Strengths of the Proposed Design. The proposed design aims to address the complexities of the LAUNCH program, including differences in grantee start times, intervention choices, and age groups of specific focus. The design captures the spirit and intent of the LAUNCH program insofar as interventions and implementation approaches are tailored to fit the needs of a specific community, and tests the average impact of these diverse interventions on a uniform set of child, parent, and child mental-health system outcomes. An additional strength of the design is its approach to selecting: 1) LAUNCH program areas for inclusion, 2) eligible Early Childhood Education Centers (ECEs) and schools within each LAUNCH location for data collection, and 3) families for study inclusion drawn from lists of individuals who volunteer to participate. The study uses a quasi-experimental matching design to select comparison communities and (for the Parent Survey) the longitudinal measurements of family and child outcomes at two time periods, which is considered the best possible study approach to isolate programmatic effects while avoiding spurious conclusions about program effectiveness.

In addition, the sample size for the parent survey respondents was determined based on the statistical power required to detect cross-sectional differences between the intervention and comparison communities. As the study also collects data over time, the size of the sample will allow for a large number of possible comparisons between LAUNCH and comparison communities as well as statistical assessments of changes among individuals over time, mitigating the risk of failure to detect any programmatic impacts. Additional strengths of the proposed design include the comprehensiveness of programmatic implementation data that will be collected, the breadth and diversity of child and family outcomes measured, and the inclusion of parallel surveys to measure child and family behavioral and emotional well-being as well as school readiness.

Limitations of the Proposed Design. Our study is limited by the following external factors that are beyond our control: the lack of randomization of communities at the outset of LAUNCH program implementation; the different starting times and lack of uniformity in core strategy implementation across LAUNCH program locations; and the inability of LAUNCH communities to provide lists of individuals who received LAUNCH direct services. Each of these limitations stems from the difficulties of evaluating an existing program in varying stages of the implementation process or from the design of the LAUNCH program itself. The MSE design is also limited by its selection of parent respondents from a convenience frame of those who chose to volunteer to participate, as opposed to randomized recruitment from all parents whose children are enrolled at a specific ECE or school. In addition, our design is ambitious in its goal of recruiting 20 LAUNCH and comparison communities for participation, in light of the time and budgetary resources available. However, because the study is powered to detect cross-sectional differences in parent survey data, we will still be able to detect any programmatic effects after two waves of data collection, even if fewer than 20 communities are recruited. The Teacher Survey (EDI), on the other hand, has never been fielded in the manner or scale proposed so there is no precedent upon which to base predictions of success.

Universe of Data Collection Efforts

Previously Approved Data Collection Instruments

The instruments from the previously approved information collection request (OMB #0970-0373) include those listed in Exhibit 3. Exhibit 3 also includes the status of these previously approved instruments in this current OMB request.

Exhibit 3. Status of Previously Approved Data Collection Instruments in New OMB Request

Previously Approved Data Collection Instrument	Status in New Data Collection Request
Site Visit and Telephone Interview Guides	Discontinued
Electronic Data Reporting: Systems Measures	Revised (see Attachment A for crosswalk)
Electronic Data Reporting: Services Measures	Revised (see Attachment A for crosswalk)
Outcomes Data Tables in End of Year Reports	Discontinued

Current Request for Data Collection Instruments

Additional information on the testing and review of instruments is found in Supporting Statement B and the specific research questions that each is tailored to address are presented below in Exhibit 4.

Exhibit 4. Crosswalk of Project LAUNCH Research Questions and Instruments/Sources
RESEARCH QUESTION	INSTRUMENT OR SOURCE
	MSE Part A		MSE Part B				Additional (Secondary) Data Sources*
	Direct Services Survey	Systems Activities and Outcomes Survey	Parent Survey	Teacher Survey (EDI)	School Survey	Key Informant Interviews on Systems Change
Child and Family Outcomes
How does Project LAUNCH affect specific individual-level child and family outcomes? Compared to those in comparison communities:
Are children in LAUNCH communities in better social-emotional health?			X
Do families in LAUNCH communities exhibit stronger parent-child relationships?			X
Do parents in LAUNCH communities have higher reported levels of social support?			X
Do parents in LAUNCH communities have lower reported levels of depression?			X
Do families in LAUNCH communities engage more frequently in specific family activities?			X
Are children in LAUNCH communities more likely to be “school ready”?				X
Are children in LAUNCH communities less likely to be suspended/expelled from school?					X
Are children in LAUNCH communities in better physical health after program implementation?			X
Do children in LAUNCH communities use more health-care services?			X
Do children in LAUNCH communities have better access to specific health-care services?			X
Does Project LAUNCH’s impact on the individual-level outcomes in Question 1 vary across the interventions supporting each of the program’s core strategies?	X
Does the impact of LAUNCH vary by family and community socioeconomic factors?			X
Do families in LAUNCH communities fare better on the outcomes included in Q1 over time?			X	X	X
Systems-Level Outcomes
Are LAUNCH coalition-building activities associated with other systems outcomes?		X				X
Are LAUNCH public information campaigns associated with other systems outcomes?		X				X
Are LAUNCH advocacy activities associated with other systems outcomes?		X				X
Are LAUNCH activities designed to increase funding associated with other systems outcomes?		X				X
Does the impact of LAUNCH vary by community socioeconomic characteristics?							X
What are the facilitators that contribute to improving systems-level outcomes?
Are LAUNCH communities in areas with co-occurring child and family programs more likely than communities without them to achieve improved systems outcomes?	X
Are LAUNCH communities with schools that emphasize child and family well-being more likely than communities without them to achieve improved systems outcomes?					X
What are the barriers that impede improvement of systems-level outcomes?
Are LAUNCH communities in states with larger uninsured populations less likely to achieve improved systems outcomes?							X
Are LAUNCH communities in states with specific fiscal challenges less likely to achieve improved systems outcomes?							X
What are the economic costs of LAUNCH Program Implementation and how do these compare to the economic benefits related to improvements in child and family outcomes?
How much does LAUNCH cost to implement overall and per child in the designated target population—with target population defined by geography and demographics?	X
What percentage of resources on average do grantees spend on each LAUNCH strategy?	X
If LAUNCH resulted in changes in child and family outcomes, what are published estimates of the dollar value of these improvements?	X
What is the total cost of LAUNCH per child in the designated target population after accounting for these cost offsets?	X		X

* The MSE will rely on pre-existing, publicly available sources such as the American Community Survey, the Current Population Survey, and the Bureau of Labor Statistics for data on demographic measures, the share of the population with no health insurance coverage, and state-level fiscal characteristics.

MSE Part A – All Project LAUNCH Grantees

All Project LAUNCH grantees are required to complete two Web-based surveys listed in Section A2 as a condition of their funding from SAMHSA: the Direct Services Survey and the Systems Activities and Outcomes Survey (see Attachments B and C for these surveys). In developing the MSE Direct Services and the Systems Activities and Outcomes surveys, we carefully reviewed each previously approved CSE survey item in order to determine topics that should be present in both the CSE and MSE surveys, maintain continuity in the requested data, and minimize burden to respondents. To facilitate the process of streamlining the CSE survey items, we created crosswalks that compare the CSE survey items with those found in the MSE Direct Services Survey and the Systems Activities and Outcomes Survey. These crosswalks can be found in Attachment A. The team worked with SAMHSA/ACF to make the survey items as consistent as possible with the CSE data, but also to clarify and improve upon them where necessary. This involved some modification to the wording and content of the questions. The team agreed that the benefits of these changes would offset any analytical challenges stemming from discontinuities in the data, would not prevent the tracking of LAUNCH activities by strategy (or the number of participants reportedly reached by each activity), and would not generate any additional burden for respondents.

Relative to the surveys administered for the CSE, these surveys will:

• facilitate categorization of program activities;

• limit opportunities for free-text descriptions that are exceedingly burdensome to summarize;

• link each activity to the LAUNCH strategy it addresses, its target population, the frequency of its occurrence, the number of individuals served, and the location of the intervention; and

• link each group of activities to its respective level of LAUNCH funding.

All of these improvements will make the LAUNCH program easier to monitor and evaluate.

MSE Part B – Sites Randomly Selected for the MSE

School Survey: The School Survey asks school administrators and ECE directors to respond to questions related to rates of child suspension and expulsion from preschool and elementary school. This instrument was developed by the MSE team based on research on child suspension and expulsion conducted by Gilliam (2005)¹ and preliminary results from the Preschool and Childcare Expulsion Study (Martin, Bosk, and Bailey, n.d.)². See Attachment D for the School Survey.

Parent Survey: The Parent Survey consists of several proposed instruments (or subscales of instruments and, in two cases, individual items) that cover a range of constructs and domains and that have all been used previously in large data collections with strong and well-documented reliability and validity (see Exhibit 5). See Attachments E, F, G, and H for the Parent Survey.

Exhibit 5. Conceptual Elements Measured in the Parent Survey and Corresponding Survey Instrument from Which They Were Drawn

Constructs and Domains	Instruments
Demographics	Project-developed and based on the American Community Survey (ACS)3 Child-Level Variables: Age Sex Ethnicity Race Language spoken at home Birth order Health insurance status Health insurance type Preschool age group/school grade Parent-Level Variables: Education level Employment status Relationship to child
Child Health Status Health care access and quality Child health and well-being	Items from the National Survey of Children’s Health (NSCH)4
Child Social-Emotional Health Attachment/relationships Initiative Self-regulation Behavioral concerns Social awareness Self-management Goal-direction	Devereux Early Childhood Assessment (DECA)5,6,7 Devereux Student Strengths Assessment (DESSA)8
Parent-Child Relationship Attachment Discipline practices Involvement Parenting confidence Relational frustration	Parenting Relationship Questionnaire (PRQ)9
Parental Depression	Center for Epidemiologic Studies Depression Scale – short version (CES-D-10)10
Family Home Environment Child development/literacy Home environment/activities	Items from the Early Childhood Longitudinal Survey (ECLS)11
Parental Social Support	Functional Social Support Questionnaire (FSSQ)12

To support the development of new or modified questions for the Parent Survey, we cognitively tested the instrument among parents of children (with a high-school education or less) who had received LAUNCH services. The team recruited these participants with the help of two LAUNCH sites and the testing involved nine one-hour interviews with respondents from Washington, DC (n=4) and Massachusetts (n=5). This testing was designed to gain insight into respondents’ understanding of the questions, to assess their validity in light of any modifications, and estimate the time burden associated with responding to the survey. We received and implemented two minor suggested clarifications to the question wording, but there were no major areas of confusion identified.

Teacher Survey (EDI): The Early Development Instrument (EDI) is a validated, population-based survey completed by kindergarten teachers, with items focused on each of the students in their classrooms. These items capture measures of early child development and school readiness in five key domains: 1) physical health; 2) emotional maturity; 3) social competence; 4) language and cognitive skills; and 5) communication skills and general knowledge. To conduct this survey, we are partnering with the Center for Healthier Children, Families and Communities at UCLA, which is licensed to administer the survey. Primary among the goals of collecting these data is to produce valid and reliable neighborhood- and community-level data representing children’s health and development across multiple domains.¹³ See Attachment I for the Teacher Survey (EDI).

Key Informant Interviews on Systems Change: This interview guide was developed to explore systems change outcomes related to Project LAUNCH in light of the community-level focus of both the program and this evaluation. Specifically, the interview guide includes, but is not limited to the following topics:

• significant systems changes achieved, including: changes in funding or reimbursement for child and family mental-health services; the establishment of institutional policies (e.g., within a school district) or protocols (e.g., within a health system) that would lead to improvements in mental-health service delivery; the creation of public information campaigns; and coalition-building and organizational efforts between government and non-governmental organizations;

• other systems changes not captured in the portal or due to factors other than Project LAUNCH;

• future plans for systems activities;

• the relationship between systems outcomes and initial LAUNCH plans; and

• coalition-building activities, public information campaigns, advocacy activities, and activities to build funding (see Attachments J and K for interview protocols).

We have designed the interview guide to establish awareness and agreement regarding each systematic component, solicit information on activities designed to address each component, and collect information on progress toward achieving system-level changes. See Attachments J and K for the Key Informant Interview on Systems Change Discussion Protocols for both LAUNCH and comparison communities.

Exhibit 6 depicts at a high level the types of outcomes captured by each of the MSE data collection efforts. For a more detailed table mapping each research question to specific survey items, see Attachment L.

Exhibit 6. Data Collection Efforts and Outcome of Interest Crosswalk

Data Collection Effort	Outcomes of Interest
	Child and Family Outcomes	Systems-Level Outcomes
MSE Part A
Direct Services Survey	X	X
Systems Activities and Outcomes Survey		X
MSE Part B
Parent Survey	X
Teacher Survey (EDI)	X
School Survey	X	X
Key Informant Interviews on Systems Change		X

Future Information Collection Requests

At this time, there are no planned future information collection requests for the MSE.

A3. Improved Information Technology to Reduce Burden

Web-based data collection platforms will be used to reduce burden for grantees in both Part A (i.e., the Direct Services Survey and the Systems Activities and Outcomes Survey) and Part B for teachers (i.e., the Teacher Survey (EDI)), parents (i.e., the Parent Survey), and school personnel (i.e., the School Survey). The data provided by all Project LAUNCH grantees in Part A will be collected through Liberty, a Web-based platform through which the current CSE surveys are administered. Upon OMB approval for the MSE, the new surveys will be programmed into a user-friendly, Web-based data portal. To minimize burden across reporting periods, some information entered at the first data collection time point will be pre-populated for grantees in subsequent reporting periods, with grantees given the opportunity to revise the pre-populated information as needed. Examples of data that can be pre-populated include program descriptions, the locations in which services occur, and the types of services documented in the previous reporting periods.

In Part B, the Teacher Survey (EDI) will be administered to teachers through the EDI Portal software. This software is owned and managed by the UCLA Center for Healthier Children, Families, and Communities, which is licensed to administer the EDI.

Data for the Parent Survey and School Survey will be collected through Liberty, the aforementioned Web-based tool. Data for the Key Informant Interviews on Systems Change will be collected through telephone interviews.

A4. Efforts to Identify Duplication

SAMHSA requires grantees to conduct local evaluations of their Project LAUNCH programs and gives them the flexibility to design their own evaluations, selecting measures and research questions that meet the goals of their particular programs. The MSE does not duplicate these efforts. We have designed the MSE to provide a more comprehensive examination of Project LAUNCH activities and community-level outcomes across all grantees. The study design proposed here will enable comparison of LAUNCH-specific measures with national trends in early childhood priority areas, equivalent data gathered in comparison communities, community-wide impacts, and additional outcomes that are outside the scope of the local grantee evaluations (e.g., changes in child and family outcomes across sites over time).

We will ensure that the data are collected in a standardized and consistent manner over time and across grantees. We will also ensure that data instruments and associated analyses will be disseminated to all grantees for incorporation into their local evaluation and programmatic improvement efforts.

A5. Involvement of Small Organizations

Elementary schools and ECE programs are the only small organizations that would be eligible for inclusion in the MSE. In these and all sites, we will minimize burden by using Web-based data collection tools to allow participants to respond at their convenience.

A6. Consequences of Less Frequent Data Collection

For Part A, completion of the Direct Services Survey will be required biannually to correspond with SAMHSA’s progress-reporting periods and facilitate tracking of the evolution of the LAUNCH program and the services it provides. Completion of the Systems Activities and Outcomes Survey will be required annually. Less-frequent data collection would limit the insights to be gathered from the MSE and could pose challenges for the management of the LAUNCH grant program itself.

For Part B, data for the Parent Survey, School Survey, and Key Informant Interviews on Systems Change will each be collected twice over the three year approval period. The Teacher Survey (EDI) will only be collected once during the data collection period. As mentioned above, delivery of services to children and families in Project LAUNCH communities is thought to be cumulative with increasing effects as children age. Collecting longitudinal data on an annual basis will allow the MSE to capture changes over time and report on these changes, while also reducing attrition of the sample that could result from less-frequent data collection.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This was published in the Federal Register Vol. 80, No. 54, Page 15016, on March 20, 2015 and provided a 60-day period for public comment. A copy of this notice is included as Attachment M. During the notice and comment period, the government received no comments.

It should be noted that we have revised the burden table submitted in the original Federal Register notice to include the modest burden associated with collecting student demographic information for the Teacher Survey (EDI). Since publishing the original Federal Register notice, an entity licensed to administer the Teacher Survey informed us that collection of this information is necessary to complete EDI data collection. The additional row in the burden table has been included as “Collection of Student Demographics for Teacher Survey (EDI).” We also revised the burden table submitted in the original Federal Register notice to include burden associated with recruitment. This includes the burden placed on superintendents, principals, and directors in our efforts to recruit school districts, schools, and ECEs, respectively, as well burden placed on school coordinators and ECE coordinators in serving as the liaison between the research team and their school or ECE and helping to recruit participants.

Consultation with Experts Outside of the Study

Many individuals and organizations, including the Project LAUNCH grantees and the Consultant Cadre, were contacted for advice on aspects of the evaluation design and data collection instruments. Their feedback was obtained through a series of webinars, during which we presented and discussed our proposed study design and methods. Members of the Project LAUNCH Consultant Cadre are (listed in Exhibit 7), have experience in the fields of child development, child health and wellness, mental health, tribal health, health policy, school readiness, early childhood programs, systems change, program implementation, and evaluation research. The study design and sample size requirements were created in partnership with senior staff in NORC’s Statistics and Methodology Department. The recruitment methodologies were developed in collaboration with senior staff in NORC’s Education and Child Development Department, who provided expertise on the level of effort and time required to recruit school districts, schools, and parent survey participants. The data collection methodologies and Web-based tools have been designed in collaboration with NORC’s Information Technology Services Department. Methods regarding recruitment and implementation of the EDI were created based on the input and direction of UCLA’s Center for Healthier Children, Families, & Communities.

Exhibit 7. Members of Project LAUNCH Consultant Cadre
Consultant	Title and Affiliation
Peg Burchinal, PhD	Senior Scientist, Frank Porter Graham Child Development Institute Adjunct Professor, Department of Education, University of California, Irvine
Christina Bethell, PhD	Professor, Department of Pediatrics, School of Medicine, Oregon Health and Science University
Catherine Walsh, MPH	Owner/Founder, Results for Children™
Nancy Whitesell, PhD	Associate Professor, Community and Behavioral Health Department, Colorado School of Public Health, University of Colorado at Denver
Bob Goerge, PhD	Senior Research Fellow, Chapin Hall at the University of Chicago
Katherine E. Grimes, MD, MPH	Associate Clinical Professor of Psychiatry and Child Psychiatrist, Department of Psychiatry, Harvard University Medical School
Stephanie M. Jones, PhD	Assistant Professor, Center on the Developing Child, Harvard University
Michelle Christensen Sarche, PhD	Associate Professor, Community and Behavioral Health Department, Colorado School of Public Health, University of Colorado at Denver
David M. Chavis, PhD	Principal Associate/CEO, Community Science
Ruth Perou, PhD	Child Development Studies Team Leader, National Center on Birth Defects and Developmental Disabilities (NCBDDD), Centers for Disease Control and Prevention (CDC)
Aleta Meyer, PhD	Senior Social Science Research Analyst, Office of Planning, Research, and Evaluation (OPRE), Administration for Children & Families (ACF)
Robin Harwood, PhD	Health Scientist, Maternal and Child Health Research Program, Maternal and Child Health Bureau (MCHB), Health Resources and Services Administration (HRSA)
Mary Kay Kenney, PhD	Health Statistician, Maternal and Child Health Bureau (MCHB), Health Resources and Services Administration (HRSA)
Lara Robinson, PhD, MPH	Behavioral Scientist, Child Development Studies Team, National Center for Birth Defects and Developmental Disabilities (NCBDDD), Centers for Disease Control and Prevention (CDC)

A9. Incentives for Respondents

Participation in the MSE of Project LAUNCH will place some burden on grantees, school coordinators, teachers, parents, and childhood education program directors and other relevant leaders. To offset this burden, we have developed a structure for respondents to receive incentives based on the effective use of incentives in prior studies and our desire to acknowledge respondents’ efforts in a respectful way. The proposed structure is presented in Exhibit 8 below.

MSE of Project LAUNCH Component	Respondent/Entity	Minutes to Complete	Winter 2017	Spring 2017	Summer 2017	Fall 2017	Winter 2018	Spring 2018	Summer 2018
Parent Survey	Parents	30	$25			-	$0
Teacher Survey (EDI)	Kindergarten Teachers	600	-	-	-	-	$50	-	-
	School for Substitute Teachers	-	-	-	-	-	≤$300	-	-
School Survey	School Coordinators / ECE Coordinators	60	-	-	$100	-	-		$100
Key Informant Interviews on Systems Change	Experts on Child Health and Well-Being	60	-	$0		-	-	$0

Incentives will be offered to the following individuals:

• Teachers who participate in the Teacher Survey (EDI) will be offered $50, dependent on the school’s approval of such an incentive;

• School coordinators, who will assist with the coordination of the Parent Survey and Teacher Survey (EDI) as well as completion of the School Survey will be offered $100 per school year, up to two times (again, provided the school approves); and

• Substitute teachers, who schools may need to hire to cover kindergarten teachers’ classes on the day they receive training and complete the Teacher Survey (EDI) – the school will receive up to $300 to help cover the cost of a substitute teacher (with this amount based on the estimated maximum daily rate for a substitute).

• Parents who participate in the Parent Survey will be offered $25 for completing the survey.

No incentives will be offered to grantees. Nor will incentives be offered to respondents of the key informant interviews, because as childhood education program directors and other relevant leaders they have a vested interest in supporting programs that promote child health and wellness and, therefore, in the overall success of the study.

Published literature supports our decision to provide incentives for parents, teachers and school coordinators in this study. Previous research has found that there are challenges with conducting research in schools, including limited school resources.¹⁴ In addition to building relationships with key stakeholders – particularly with teachers and principals – providing incentives for participation has been shown to be a successful strategy to motivate participation of teachers, administrators, and parents.¹⁵ Research suggests that incentives will hold particular promise for the parents targeted by Project LAUNCH given that the communities touched by the program are primarily lower-income, less-educated, and in some states, include a higher percentage of racial and ethnic minorities than does the general population. Previous studies indicate that the inclusion of incentives improves responsiveness and representativeness among survey samples in similarly vulnerable populations (e.g., Beebe, et al., 2005; Singer, Van Hoewyk, and Maher, 2000; Mack, 1998; Dykema, et al., 2012).

The MSE study involves multiple, inter-related recruitment efforts for Part B. To adhere to the evaluation design, we must secure participation from school districts, schools, and early childhood education programs (ECEs) in a site before we can start recruiting parents. In addition, since the target number of completes for the Parent Survey was determined based on the statistical power required to detect cross-sectional differences between the intervention and comparison communities, ensuring that we reach this number will be of utmost importance to the integrity of this component of the evaluation as a whole. The size of the sample will also allow for a large number of comparisons between LAUNCH and comparison communities as well as statistical assessments of changes among individuals over time, mitigating the risk of failure to detect any programmatic impacts. Maintaining the sample from the first year of data collection will be especially critical in the second year of data collection since our longitudinal design requires continued participation from the same parents who participated in the initial year. It is important to note that if we are unable to recruit a sufficient number of parents from the original schools and ECEs that agreed to participate in the study, we will likely not have the resources to reach out to new districts, schools, and ECEs to start the process anew.

Previous research has found that there are often challenges with conducting research in schools, including limited resources and difficulties acquiring parental consent. In Jaycox, et al. (2006), the authors discuss three evaluations conducted in schools on programs at various stages of implementation, similar to Project LAUNCH. They note that “recruiting schools to participate in the evaluation of such programs can be difficult, and ample time is required (Cline, Schafer-Kalkhoff, Strickland, & Hamann, 2005; Horowitz et al., 2003; Lytle et al., 1994; Peterson, Kealey, Mann, Marek, & Sarason, 2000).” The authors also acknowledged that, because of these challenges, “[i]ncentives for students, parents, teachers, administrators, and support staff members are extremely important (Cline et al., 2005).”

The amounts of the proposed incentive for school/ECE coordinators were determined in consultation with in-house NORC experts based on previously conducted studies with similar recruitment strategies, respondent populations, methods, and burden. For example, the Healthy Communities Study, which was conducted for the National Heart, Lung, and Blood Institute (NHLBI), offered $50 for school coordinators. The project was completed five years ago so, when adjusted for inflation, the amount is comparable to those we are proposing, especially considering the significant role coordinators will play in this proposed study.

The $25 incentive for parents will help ensure that we obtain completed surveys from parents who are representative of the population in selected LAUNCH and comparison communities as the goal of the MSE is to evaluate the impact of Project LAUNCH interventions on families in the communities where the interventions took place. As noted above, these communities are primarily lower-income, less-educated, and in some states, include a higher percentage of racial and ethnic minorities than does the general population. Several previous studies have demonstrated not only the effectiveness of incentives in hard-to-reach populations similar to those served by Project LAUNCH (and therefore critical to the MSE), but also that the inclusion of incentives improves the representativeness of survey samples (e.g., Beebe, et al., 2005; Singer, Van Hoewyk, and Maher, 2000; Mack, 1998; Dykema, et al., 2012).

Update October 2017: To date, we have compared information from the first 224 parent respondents on their race/ethnicity, education level, employment status, and income range to the averages for these variables using the American Community Survey (ACS) data on these communities. Thus far, Parent Survey respondents are more likely to be white, college-educated, employed full-time, and from higher-income categories than are the parents in their communities at large. This response bias will pose significant risks to the validity of the survey results unless we take measures to increase the representativeness of the sample. As noted above, the best strategy for ensuring that we obtain survey responses from a representative set of parents in the available timeframe is to offer a modest monetary incentive to parents.

While no specific published information is available on amounts of the incentives that have been used in the administration of the Teacher Survey (EDI), NORC has conducted other studies with teachers and early childhood education providers for which we successfully utilized incentives for participants’ efforts. This includes the NAGB Content Alignment Study, in which teachers were trained on the use of an instrument over a period of several days and received, on average, $118 per day for their participation. The incentive for teachers and school coordinators as well as for substitute teachers in our study is based on consultation with entities that have administered the EDI, including UCLA’s Center for Healthier Children, Families, and Communities and the local evaluator for the Project LAUNCH grantee in New York. UCLA advised us that compensation is typically given to teachers who complete the EDI outside of school time or to cover the cost of substitutes if the teachers complete the EDI during the school day. The teachers filling out the EDI will be doing so during their school day, while substitute teachers oversee their classrooms. However, completing the EDI for each of their students will take teachers approximately 10 hours total, which is longer than their typical school day. Also, the process of completing the survey will be remarkably different from a typical day in the classroom insofar as the EDI requires that teachers give significant consideration to each student’s physical well-being, language and cognitive skills, as well as social and emotional development. Since the approach we propose includes teachers’ time both during and after school hours, we propose offering teachers a $50 incentive for the additional time the EDI will take outside of normal school hours and the cognitive burden of completing the survey for each of their students in addition to reimbursing schools for the cost of substitute teachers.

The substitute teacher will be needed for one full school day while the kindergarten teacher is receiving training on and completing the EDI. Based on the extensive experience of UCLA’s Center for Healthier Children, Families and Communities, which is licensed to administer the EDI, offering a sufficient amount for schools to pay substitute teachers boosts school participation rates. UCLA’s leadership informed us that substitute teacher rates vary by district and are based on seven to eight hours of the teacher’s time. They advised that, while some rates may be as low as $200, the standard district rate is closer to $350. Based on UCLA’s advice, which was informed by prior experience administering the EDI, we set the school reimbursement rate for substitute teachers at up to $300. However, this amount is a ceiling; we will reimburse schools for the actual cost, which we understand may be less than $300.

The amounts of the incentives we are proposing are comparable to – and in some cases slightly higher than – those used in the aforementioned recent NORC studies. However, the complexity of our design and burden of participation on respondents justify our amounts. The EDI has never been fielded on this scale before; it has only been conducted in one community at a time, and we aim to field it in 20 sites (10 LAUNCH and 10 comparison communities). Further, the burden is considerable for school coordinators, who will be asked to recruit participants, relay information, and handle logistics for the Parent Survey or the Teacher Survey (EDI), and teachers, who will be asked to complete Teacher Survey (EDI) for every student in his/her classroom.

A10. Privacy of Respondents

For each component of the data collection, there will be specific procedures in place to ensure respondents’ privacy to the fullest extent of the law. For the LAUNCH MSE, all data collection efforts for both Part A and Part B will be submitted to and approved by NORC’s Institutional Review Board (IRB) prior to their initiation. The NORC team has also allocated time to meeting school district requirements for district-level IRB approval, a process with which NORC has extensive experience from multiple previous studies that have involved combining uniform project and data collection descriptions with customized responses to individualized district concerns.

Direct Services Survey and the Systems Activities and Outcomes Survey: The evaluation will collect information for these surveys using a Web-based data portal. The data collection and analysis modules include https certificate-based authentication and transaction encryption processes approved for HIPAA data transactions on federal health-care projects. It will not be possible for any individuals who are not part of the research team to access these data. Grantees may access their own data with passwords provided by the MSE team. Large portions of the data that grantees will report through the Web-based data portal are already being collected by grantees for their local evaluations. All information on services will be reported to the MSE in the aggregate; thus, no information collected can be linked to individual families or agencies.

All data collected via the Web-based data portal will be stored electronically through NORC’s password-protected secure network system. Project directories and databases are protected at NORC by assigned group memberships, passwords, and other techniques (e.g., ACLs), which prohibit access by unauthorized users. In addition to the issue of protection of privacy, data security encompasses backup procedures and other file management techniques to ensure that files are not inadvertently lost or damaged. All of NORC’s systems run on computers with automatic full-disk FIPS-compliant encryption system software to protect against loss of sensitive data. NORC secures data transfer via the Internet by means of FIPS 140-2 compliant VPN. Data at rest on mobile computers or portable media are encrypted by means of using FIPS 140-2 compliant whole full-disk encryption technology.

NORC requires the use of internal network data storage services to store all project-related data files. Internal network storage is provided so as to mitigate the potential of data loss due to accidents, computer equipment malfunction, failure, or human error, as well as to administer access rights surrounding the support of privacy issues that may be related to both legal and contractual obligations. Wide arrays of network security precautions are undertaken by NORC to ensure the proper storage of all project data.

These include:

• All production file servers are equipped with fault-tolerant disk arrays and redundant power supplies so as to minimize the risk of losing valuable project data. These data are also protected by both surge suppression and Uninterruptible Power Supplies (UPS) as an added protective measure.

• All operating system vendors are routinely monitored by a designated NORC Information Technology (IT) infrastructure resource monitor for security patches with updates applied as necessary. Data transfers to removable media for purposes of client delivery or archival are performed on all documentation by only the IT department in order to control data formatting and ensure readability. This also gives the IT department the opportunity to scan the deliverables for viruses while maintaining detailed shipping manifests and receipts for all deliveries.

• All NORC-authorized network users are issued a FIPS-compliant encrypted, challenge-response user-id and password, which must be used to sign into each of the project applications and data areas located on the network. The user-id/password system restricts the user's access to only their specific project accounts, thereby further restricting the type of data access allowed for each individual user.

• Employees are required to change their server passwords on a regular basis to ensure greater project security.

• The use of any software by NORC employees requires a thorough review and approval process prior to installation.

• Remote access to the NORC network is performed through NORC’s firewall using both Virtual Private Network (VPN) technology and FIPS-compliant encrypted challenge-response technology. The primary tools used to provide this secure remote access include Juniper SSL Gateway or Citrix, depending upon the access required. A series of firewalls and packet filtering routers have been configured by the Infrastructure team so as to protect each NORC Internet Access Point, and NORC employs a dedicated IT infrastructure resource for the purpose of monitoring the LAN and WAN for signs of intrusion and other security violations. Host-based applications such as SFTP and web servers are run only on servers inside NORC's data center, and are separate from the servers that are designated to store and collect client data.

• Connectivity among all NORC sites is protected by dedicated data circuits. There are also dedicated NORC IT resources in place for the purpose of monitoring software that proactively searches for security holes, allowing for timely corrective action.

• NORC routinely engages third parties to conduct network security audits. A typical audit includes comprehensive attempts at network penetration from undisclosed sources and a review of policies and procedures.

School Survey, Parent Survey, and Teacher Survey (EDI): Each respondent in the School Survey, Parent Survey, and Teacher Survey (EDI) will be asked to complete a Web-based consent form that will appear at the start of the respective survey (see individual surveys for consent forms). Respondents will not be permitted to continue with the Web-based survey if they have not consented. The consent form will explain the study procedures for ensuring that respondents’ answers will remain private to the fullest extent of the law. The consent form will also explain that: a) completion of the survey is voluntary, and that there are no penalties for refusing to participate or ending participation at any time during the survey; b) the respondent can refuse to answer any question for any reason; c) data will be stored in de-identified files (e.g., data will be kept separate from respondents’ contact information and the information linking the individual to their data will be destroyed after the final round of data collection); and d) no identifying information will be used in any evaluation reports and all findings will be reported in the aggregate. Respondents will provide consent each time they participate.

Key Informant Interviews on Systems Change: At the start of the telephone interviews with key informants, respondents will be read a consent script that explains the study procedures for ensuring that all responses will remain private to the full extent of the law (see interview discussion protocols for consent forms). The respondent must provide verbal consent before the interview begins. The consent explains that: a) participation in the interviews is voluntary, and there are no penalties for refusing to participate or ending participation at any time during the interviews; b) the respondent can refuse to answer any question for any reason; c) data will be stored in de-identified files and d) no names of individuals will be used in any evaluation reports. Respondents must also consent to the interview being recorded to ensure that their responses are captured accurately. Recordings will be saved using identification numbers rather than identifying information and will be destroyed at the conclusion of the study. Respondents will be offered a written copy of the consent script to keep for their records.

In all MSE documents and reports, we will not indicate the individual names and titles of anyone participating in data collection on behalf of a Project LAUNCH grantee or comparison community. Their titles may, however, be presented in the aggregate and vignettes of activities at grantee sites may be included. We will inform all study participants about this during our consent process prior to initiating each interview.

A11. Sensitive Questions

To achieve the goal of describing the families potentially reached by Project LAUNCH services, the MSE will ask parents some questions about family and child risk factors that some participants may find sensitive, such as parental depression, aspects of the child’s home environment, and questions related to the child’s social and emotional well-being. All sensitive questions included in the Parent Survey have been fielded widely in instruments used in national studies, and to our knowledge no serious adverse events have ever occurred from their use. However, sensitive questions about child development, mental health, and a family’s home environment may make some respondents uncomfortable. To mitigate any discomfort among respondents, the Web-enabled survey will include a button labeled Feeling Upset? that will be visible and accessible from each page of the survey. If clicked, the system will generate a pop-up window with text reminding the individual that they can take a break and finish the survey later if they want and providing instructions for how to log off the survey and log back on when they are ready to complete it. Although it is highly unlikely that respondents will need it as a result of this survey, the pop-up will also contain a link to the website and the hotline phone number for The Crisis Call Center, an independent, not-for-profit, non-denominational crisis counseling service that offers free counseling to anyone at any time.

As part of the consent process, participating parents will be informed that they might find some questions sensitive and will be asked to indicate their consent prior to beginning the survey, acknowledging that their participation is voluntary. All respondents will be informed that their identity will be kept private to the full extent of the law, that they can skip any questions they do not want to answer, and that they can choose not to participate or end the survey at any time with no adverse consequences.

All information collected at the individual child/family level in the Parent Survey and Teacher Survey (EDI), as well as information from the School Survey, will be de-identified and reported only in the aggregate for the MSE.

A12. Estimation of Information Collection Burden

Burden Remaining from Previously Approved Information Collection Burden Hours

As illustrated above under Previously Approved Data Collection Instruments, the electronic/web-based surveys designed for the CSE have been revised under this OMB request. Once approved, the revised instruments (Direct Services Survey and Systems Activities and Outcomes Survey) will replace the current ones and be completed by the 31 active LAUNCH grantees instead. All related burden is included in the following section: “Newly Requested Information Collection.”

Newly Requested Information Collections

The requested data collection does not impose a financial burden on respondents nor will respondents incur any expense other than the time spent completing the surveys and interviews.

The estimated annual burden for study respondents is identified in Exhibit 9. Estimates for the Web-based data portal surveys in Part A (the Direct Services Survey and Systems Activities and Outcomes Survey) were calculated based on an assumption of 31 Project LAUNCH grantees. Estimates for MSE Part B (the School Survey, Parent Survey, Teacher Survey (EDI), and Key Informant Interviews on Systems Change) were based on the sample size calculations detailed in Supporting Statement B. Values for annual burden hours in Exhibit 9 were calculated as the total burden hours divided by the number of rounds in which each instrument will be implemented, representing an average annual burden.

We are requesting clearance for a 3-year study period.

	Total Burden Requested Under this Information Collection Exhibit 9. Estimated Annual Response Burden and Cost for Newly Requested Data Collection
Data Collection Instrument		Number of Respondents	Annual Number of Responses Per Respondent	Average Burden Hours Per Response	Total Burden Hours	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
MSE Part A
Direct Services Survey		31	2	8.5	1,317.516	439	$18.78	$8,244.42
Systems Activities and Outcomes Survey		31	1	8	74417	248	$18.78	$4,657.44
MSE Part B
Recruitment of School Districts, Schools, ECEs, and Participants		340	1	1.912	1,300	433	$25.52	$11,050.16
Parent Survey		1800	1	0.5	1,800	600	$16.87	$10,122
Teacher Survey (EDI)18		160	1	10	1,600	533	$25.40	$13,538.20
Collection of Student Demographics for Teacher Survey (EDI)19		20	1	2	40	13	$26.42	$343.46
School Survey		120	1	1	240	80	$25	$2,000
Key Informant Interviews on Systems Change		70	1	1	140	47	$31.61	$1,485.67
						2,393		$51,441

The row titled “Recruitment of School Districts, Schools, ECEs, and Participants” reflects burden placed on study participants through two distinct efforts: 1) the burden placed on superintendents, principals, and directors during our recruitment of school districts, schools, and ECEs, respectively, and 2) the burden placed on ECE coordinators in recruiting parent volunteers to complete the Parent Survey, ensuring that the proper individual(s) complete the School Survey, relaying information to various parties on behalf of the research team as needed, and 3) for school coordinators, helping to coordinate logistics for the day teachers will complete the EDI. To calculate the total burden of these distinct recruitment efforts, we determined the number of respondents, the estimated burden hours, and average hourly wage for each effort separately. Determining the number of respondents across these efforts involved two steps. First, we considered the recruitment of school districts, schools, and ECEs. Our target recruitment is six ECEs and two schools per site in 20 sites. Accounting for nonresponse and those that decline to participate, we estimate that we will attempt to recruit administrators at nine ECEs, three schools, and two school districts per site (280 individuals). Second, we considered the number of school and ECE coordinators. Once we have recruited the schools and ECEs, we will have 120 ECE coordinators and 40 school coordinators working as liaisons for the study. In estimating their annual burden hours, we considered the tasks we will ask each to do (see above), the fact that in most ECEs the administrator will serve as the coordinator, and that the second year will be less burdensome because they will only need to remind the prior year’s respondents (to both the Parent Survey and the School Survey) about the study and gain their continued participation. Using the appropriate labor categories (described below), we calculated the total annual cost of each effort. By summing these annual costs and dividing by the total annual burden hours across these efforts, we calculated an average hourly wage. We divided the annual burden hours by the annual number of respondents to determine the average burden hours per response.

Total Annual Cost

The total annual cost of this newly requested collection is $51,441, based on 2,393 annual burden hours.

The mean hourly wage is based on several categories as reported in the 2013 Bureau of Labor Statistics, Occupational Employment Statistics. For superintendents, principals, and ECE directors, we used the average hourly wage of two categories – “Education Administrators, Preschool and Childcare Center/Program” and “Education Administrators, Elementary and Secondary School.” We used “Educational Administrator, Early Childhood” for the school coordinator and ECE coordinator as well as the administrator who will fill out the School Survey. We used the labor category of “Average Worker, All Occupations” for parent respondents, “Kindergarten Teachers, Except for Special Education” for teacher respondents, “Computer Support Specialist” for the school district IT manager, “Social and Community Service Manager” for respondents to the Key Informant Interviews on Systems Change, and “Miscellaneous Community and Social Service Specialists” for respondents to the Direct Services Survey and the Systems Activities and Outcomes Survey.

To avoid inflating the total annual cost, we were mindful of instances where burden would only occur in one year of the study. For example, as the note below Exhibit 9 indicates, the Teacher Survey (EDI) will only be administered once. As a result, the rows for this survey and the collection of demographic information for it reflect annual numbers that will apply in only one year, which thus also represent the total burden for these two activities.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for data collection activities will be $2,006,086. Annual costs to the Federal government will be $668,695 for the proposed data collection under this OMB clearance number (0970-0373).

A15. Change in Burden

This is an additional information collection request under OMB #0970-0373.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Analysis Plan

Part A

Data entered into the Web-based portal by all Project LAUNCH grantees will be analyzed for multiple purposes, including:

1. Providing descriptive information about the programs and systems change activities being implemented across all LAUNCH grantees; and

2. Providing covariate information about the number and types of activities conducted and number of children served by each LAUNCH grantee for the cross-sectional and longitudinal analyses of instruments in Part B.

Part B

School Survey. We will use suspension and expulsion rates collected by the School Survey to describe rates of suspension and expulsion in elementary schools and ECEs in both LAUNCH and comparison communities as well as to examine differences in these rates between LAUNCH and comparison sites. Additionally, these data will be used in the regression models for child and family outcomes if applicable. As the MSE will collect secondary information on school expulsion and suspension policies, we will be able to consider these data in their specific local contexts.

Parent Survey. The Parent Survey will be used to analyze many of the primary child and family outcomes of interest for this study over time and in different age groups of children. Prior to analysis, all data collected through the Parent Survey will be cleaned for out-of-range responses and missing data will be replaced using multiple imputation when possible. Psychometric scores for each scale measured in the Parent Survey will be calculated based on published methodologies. To support analysis and comparisons to other studies, individual parent responses for each DECA instrument will be converted into nationally normed percentile scores, which are used for communication of DECA results, as well as standardized scores, which can be used in statistical analyses to measure programmatic impact. Information regarding power calculations and study design choices based on the properties of the DECA can be found in Supporting Statement Part B, Section B-1.

Cross-Sectional Analyses. We will perform cross-sectional analyses to test the programmatic impact of LAUNCH on the population mean standardized DECA score for responses collected among data collected within the LAUNCH sites as compared to the data collected in the comparison communities. Paired sample t-tests will be used to compare differences in the mean standardized DECA scores between intervention and comparison communities, with the magnitude of such differences categorized as ‘no meaningful difference’ (<2), small (2-4), medium (5-7), or large (8 or greater).²⁰ Although such a simple analytic case cannot be used for Project LAUNCH (due to confounding from other variables), this approach will be used to capture basic outcomes of interest. The more complex statistical models the study will use to determine programmatic impact more holistically are described below.

The MSE’s use of a quasi-experimental design will minimize any demographic differences (e.g., with respect to race, parent education, or household income) between LAUNCH and comparison communities that may be associated with systematic differences in DECA scores. As overall differences in demographics and other factors will persist even with the most accurate selection of comparison communities, a statistical model must be used to correct for these differences so that they do not confound estimates of the LAUNCH program effect. In addition, within-community differences in both the LAUNCH and comparison communities—and different collection locations within communities—will result in expected clustering of responses, which must also be adjusted for to isolate the effect of the program. As described in detail in Supporting Statement Part B, Section B-1, we will apply a Generalized Estimating Equation (GEE) regression model to estimate the incremental impact of LAUNCH on mean DECA Total Protective Factors Scores after controlling for potentially confounding variables of interest. We will include interaction terms between the program effect variables as well as alternative specifications of functional form of the continuous program effects variables to test for nonlinearities.

Changes or Improvements in Outcomes over Time. We will conduct longitudinal analyses that assess average change in within-person DECA scores over time in the LAUNCH sites compared to the comparison sites through use of a hierarchical linear model (HLM). The HLM model controls for clustering associated with time, site, and data collection location. Variables included in the model will be similar to the GEE model.

Teacher Survey (EDI). We will use data from the EDI to measure the effect of LAUNCH on school readiness at the school and community level. For the target communities, we will receive a cleaned, scored, and de-identified data file from UCLA, an entity licensed to administer the EDI. As the unit of observation for the EDI is the classroom, the analyses will focus on school and community-level results rather than results for individual teachers or children. We will conduct cross-sectional analyses similar to those conducted in the Parent Survey. For example, within the given year of Teacher Survey (EDI) data collection, we can compare within LAUNCH sites and to comparison communities the scores on the five domains of the EDI (physical health and well-being; social competence; emotional maturity; language and cognitive development; and communication skills and general knowledge) as well as children’s overall level of school readiness. We will account for as many modifiers as possible to make a reasonable estimation of what is contributing to observed differences in EDI scores.

Key Informant Interviews on Systems Change. As noted above, the key informant interviews will provide additional contextual information to complement the quantitative data collected through the portal and will allow the team to probe qualitatively on local community and system dynamics that may impact LAUNCH implementation—and thus systems activities and the achievement of systems outcomes. Using the discussion protocols and approach detailed above, this information will support the interpretation of data from the other instruments (i.e., Parent Survey, School Survey, and Teacher Survey (EDI)) as appropriate.

Time Schedule and Publication

The schedule for fielding, analyzing, and reporting the data findings for the multi-site evaluation of Project LAUNCH is as follows, pending OMB approval:

Part A

• Direct Services Survey Biannually, Fall 2016-Fall 2018

• Systems Activities and Outcomes Survey Annually, Fall 2016-Fall 2018

Part B

• School Survey Summer 2017, Summer 2018

• Parent Survey Winter/Spring/Summer 2017, 2018

• Teacher Survey Winter 2018

• Interviews on Systems Change Spring/Summer 2017, 2018

Analysis and Reporting (MSE Parts A and B)

• Data Reports Annually, 2016-17, 2017-18

• Data Analysis Annually, 2016-17, 2017-18

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 Gilliam, W. (2005). Prekindergartners Left Behind: Expulsion Rates in State Prekindergarten Systems. Accessed from http://www.hartfordinfo.org/issues/wsd/education/NationalPreKExpulsionPaper.pdf

2 Martin, K. A., Bosk, E., & Bailey, D. (n.d.). Preliminary Report from the Preschool and Child Care Expulsion Study. Accessed from http://sites.lsa.umich.edu/kamartin/wp-content/uploads/sites/41/2014/01/Preschool-Expulsion-Brief-for-Participants-1.pdf

3 U.S. Census Bureau. (2015) American Community Survey, American Community Survey 2015; http://www2.census.gov/programs-surveys/acs/methodology/questionnaires/2015/quest15.pdf.

4 Child and Adolescent Health Measurement Initiative (CAHMI). (2012). 2011/12 National Survey of Children’s Health; http://www.cdc.gov/nchs/data/slaits/2011NSCHQuestionnaire.pdf.

5 Mackrain, M., LeBuffe, P., & Powell, G. (2007). Devereux Early Childhood Assessment for Infants and Toddlers. Lewisville, NC: Kaplan Early Learning Company.

6 LeBuffe, P.A. & Naglieri, J.A. (2003). Devereux Early Childhood Assessment—Clinical Form. Lewisville, NC: Kaplan Early Learning Company.

7 LeBuffe, P.A. & Naglieri, J.A. (2012). Devereux Early Childhood Assessment for Preschoolers, second edition. Lewisville, NC: Kaplan Early Learning Company.

8 LeBuffe, P.A., Shapiro, V.B., & Naglieri, J.A. (2014). Devereux Student Strengths Assessment. Charlotte, NC: Apperson SEL +.

9 Kamphaus, R. W., Reynolds, C. R., & Pearson. (2006). PRQ: Parenting Relationship Questionnaire. Minneapolis, MN: Pearson Assessments.

10 Radloff LS. (1977). ‘The CES-D Scale: A Self-Report Depression Scale for Research in the General Population.’ Applied Psychological Measurement 1:385-401.

11 U.S. Department of Education. (2011). National Center for Education Statistics. Early Childhood Longitudinal Study, Kindergarten Class of 2010-11 (ECLS-K: 2011). https://nces.ed.gov/ecls/kinderinstruments.asp#questionnaires.

12 Broadhead WE, Gehlbach SH, de Gruy FV, et al. (1988). ‘The Duke-UNC Functional Social Support Questionnaire: Measurement of Social Support in Family Medicine Patients.’ Med Care 26:709-23.

13 For more information, see UCLA’s Center for Healthier Children, Families, and Communities’ website at http://www.healthychild.ucla.edu/ourwork/edi/.

14 Jaycox, L. H., McCaffey, D. F., Ocampo, B. W., Shelley, G. A., Blake, S. M., Peterson, D. J., et al. (2006). Challenges in the evaluation and implementation of school-based prevention and intervention programs on sensitive topics. American Journal of Education, 27(3), 320–326. doi:10.1177/1098214006291010

15 Kelly, M. S., Harrison, J., Schaughency, E., & Green, A. (2014). Establishing and maintaining important relationships in school mental health research. School Mental Health, 6, 112-124.

16 Active Project LAUNCH grantees in Cohorts 4, 5, and 6 will complete the Direct Services Survey once in 2016 (fall) and twice in 2017 and 2018 (spring and fall). There is a total of five data reporting periods. So, the total burden hours is 31*8.5*5=1,317.5.

17 All active Project LAUNCH grantees in Cohorts 4, 5, and 6 will complete the Systems Activities and Outcomes Survey in the spring of 2016, 2017, and 2018 for a total of three data reporting periods.

18 The Teacher Survey (EDI) will be conducted only once over the course of the three-year approval period. This rows shows the total burden associated with this effort as well as that number divided by three to reflect the annual burden hours.

19 Because the Teacher Survey (EDI) will be administered only once over the course of the three-year approval period, the demographic data for the EDI will also be collected only once. This row shows the total burden hours associated with this effort as well as that number divided by three to reflect the annual burden hours.

20 Devereux Center for Child Resilience. Calculating DECA Change Scores. Accessed 1/18/2016. http://www.centerforresilientchildren.org/infants/calculating-deca-it-change-scores/