Project LAUNCH SSB Template_052013_rev

Project LAUNCH Cross-Site Evaluation

OMB Information Collection Request

0970 - 0373

5.21.13

Supporting Statement

Part B

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officers: Laura Hoard

B1. Respondent Universe and Sampling Methods

The cross-site evaluation collects information from all Project LAUNCH grantees (6 grantees funded in September 2008, 12 grantees funded in September 2009, 6 grantees funded in September 2010, and an additional 11 grantees funded in September 2012). As part of their grant requirements, grantees are required to participate in cross-site data collection activities, including annual interviews conducted during a site visit in each cohort’s second year and by telephone in years one, three, and four; semi-annual Web-based data reporting on systems development and services delivery; and reporting outcome data from local evaluations. Accordingly, the study represents the universe of funded grantees, and results will be representative of all grantees.

Site Visits and Telephone Interviews

Purposeful identification will be used to select respondents both for the telephone interviews and for the site visits. Telephone interviews are conducted in years one, three, and four of each cooperative agreement with site visits conducted in the grantee’s second year. Purposeful selection is appropriate, because information can come only from individuals with particular roles or knowledge of the community and its systems.

Semi-Annual Web-Based Data Reporting: Systems Measures

Project LAUNCH staff for each grantee will complete the systems measures for that site. No sampling is required, as this information will be reported by the universe of Project LAUNCH grantees.

Semi-Annual Web-Based Data Reporting: Services Measures

Data on the universe of children, families, and providers participating in Project LAUNCH services will be collected semi-annually through the Web-based reporting system. Grantees will report aggregate information on children and families served under different service models (e.g., home visitation), including demographic information. They will report aggregate information on providers who participate in training or other activities offered by Project LAUNCH, and on provider practice and provider setting changes.

Review of End of Year Reports

Additional grantee-specific aggregate child and family outcomes will be reported by the grantees as part of their local evaluations. Grantees will select the outcomes they wish to examine based on the evidence-based practices they are implementing at the local level and will report aggregate outcome data on children and families who receive a particular service(s) and provider changes in practice and knowledge in their end-of-year evaluation reports.

B2. Procedures for Collection of Information

Site Visits and Telephone Interviews

The evaluation will include telephone interviews and site visits to all Project LAUNCH grantees. One or two-person teams from the cross-site evaluation senior staff will conduct the site visits, which are expected to take between 1.5 – 2 days to complete during the second year of the cooperative agreement. Telephone interviews are conducted in years one, three, and four of each cooperative agreement.

During the site visits and telephone interviews, senior researchers will speak with key staff involved in implementing and overseeing Project LAUNCH activities. Likely respondents for site visits include: the State Child Wellness Coordinator, the Local Child Wellness Coordinator, the Local Child Wellness Council Chair, State Child Wellness Council Chair, and local service providers/stakeholders. Respondents for the telephone interviews include: state and local coordinators for Cohorts 1, 2, and 4 and the local coordinator for Cohort 3 grantees, the local Child Wellness Council Chair, and local service providers/stakeholders.^b

Information collected during the site visits/telephone interviews cover the following broad categories:

• General Program Information

• Community/Tribal Context

• Strategic Planning Process

• Child Wellness Council and Program Oversight

• Project Outreach

• Project LAUNCH Service Delivery

• Workforce Development and Capacity Building

• System Coordination

• Cultural Competence

• Quality Monitoring and Improvement

• Sustainability and Replication

The interviews will follow interview guides, which will have been populated with information gathered from existing sources, to the extent possible, in advance of the site visits/telephone interviews. During site visits/telephone interviews, the cross-site evaluation team will verify information about the target population and the population served; implementation of Project LAUNCH activities at the state, tribal, and local levels; infrastructure development and systems changes; collaboration and coordination across child-serving systems including partnerships established; strategies developed and adopted to promote implementation of evidence-based promotion and prevention programs; staff training procedures; barriers encountered and how they were addressed; achievements, and sustainability of services and systems changes.

Semi-annual Web-based Data Reporting

Grantee staff will participate in semi-annual Web-based data reporting activities. These data will be used to describe the Project LAUNCH activities that have occurred over the most recent six month reporting period and correspond to SAMHSA’s project reporting periods (e.g., April 1st-September 30th, October 1st-March 31st). Grantees will provide data through a secure website that will be run using the Decision Support 2000+ platform. Although the evaluation will not dictate which Project LAUNCH staff member will be tasked with providing these data, we anticipate that the State Child Wellness Coordinator, the Local Child Wellness Coordinator, and/or the local evaluator at each grantee site will assume primary responsibility for this effort. Grantees will be asked to report on systems development activities at the state/tribal and local levels and on services delivery at the local level.

Data to be reported in the Web-based data reporting system include the following:

Systems Development Activities at the State, Tribal, and Local Levels:

• The degree to which Project LAUNCH has assisted in the development and implementation of an integrated system of care for children and families; collaboration and coordination with other agencies; workforce development; and sustainability of LAUNCH activities over time,

• Activities undertaken to improve the local service system (e.g., trainings conducted, collaboration/infrastructure development activities, quality monitoring activities), and

• Activities undertaken to improve cultural competence in local service systems and service delivery.

Service Delivery at the Community Level:

• For each service model being implemented (e.g., home visitation):

  • Program model and core components of the model

  • Proportion of program funded by LAUNCH

  • Whether the program is targeted or universal and, if targeted, characteristics of the target children or families

  • Number of eligible families and children served in a given reporting period

  • Number of staff in the community who are delivering the service

  • Number of children/family members screened

  • Number of children/family members referred

  • Demographics of children and families in the service

  • Fidelity of implementation of the program model

  • Type and frequency of training provided

  • Provider and provider setting changes

Review of End of Year Reports

One set of research questions for the LAUNCH cross-site evaluation concerns the effects of LAUNCH-supported services on three samples in the LAUNCH communities: (a) the provider workforce in child and family services, (b) parents/families, and (c) children birth to 8 years old. The research questions ask about outcomes for these groups within each of the five LAUNCH promotion and prevention strategies, as well as across the five strategies. The analyses to address these cross-site research questions will be based on data culled from reports of the LAUNCH local evaluations, which are required as part of the SAMHSA funded grants. Each LAUNCH project is required to conduct a local evaluation of the implementation and outcomes of the LAUNCH-supported services. Annual local evaluation reports include analyses of services and systems outcomes from each of the five years of the grant as well as analyses and description of changes in outcomes over the five year period of each grant. The cross-site evaluation will abstract these data from reports and conduct meta-analyses that summarize the results generated and reported by the 35 local evaluations from their outcome studies.

The local LAUNCH evaluations are not required to use a specific design or common measures in their outcome studies. As a result, both within and across the LAUNCH local evaluations, the outcome study designs for different service strategies vary in terms of the groups for whom outcomes are being assessed (providers, parents, and/or children), the specific outcome measures, the data collection schedule, and whether the design includes both pre- and post-measures and or includes a comparison group. This variation in the designs for the outcome studies of the LAUNCH services means that the results from the local evaluations will vary in terms of the strength of the evidence. Our approach to summarizing these results involves categorizing findings in terms of their “strength of evidence.” This concept of strength of evidence underlies the ratings of studies produced by the What Works Clearinghouse (WWC).¹ However, the WWC standards for strength of evidence are relevant only for a certain set of designs, specifically, random assignment designs, quasi-experimental designs, regression discontinuity designs, and single subject designs. Nearly all of the outcome studies being conducted as part of the local LAUNCH evaluations will employ designs that do not include a comparison group and may or may not include pre- and post-measures of the same respondents. Therefore, as part of the cross-site evaluation, we are developing a set of standards for strength of evidence of pre-post designs. A draft set of standards is shown in Appendix H. The cross-site evaluation will track information on the design that produces each outcome finding, as well as the results themselves.

Local evaluators will be provided table shells to describe their outcome data as well as information on their outcome evaluation design. Grantees will be asked to complete the tables (Appendix H) and include them in the evaluation report they submit annually to SAMHSA.

Quality Control

Site Visits/Telephone Interviews

We have instituted a variety of methods to ensure the quality of the data collected in the site visits/telephone interviews. All site visitors/telephone interviewers will attend training before interviews are conducted, which will cover the interview protocols, definitions for terms used in the interview guides, and any questions about the data collection instrument or procedures. Staff will review strategies for working with diverse populations and for ensuring that data are collected in a manner that is culturally sensitive and respects the backgrounds and traditions of Project LAUNCH staff. During site visits, the cross-site evaluation team will not have direct contact with service recipients.

Semi-annual Web-based Data Reporting

Evaluation team members will work closely with Project LAUNCH staff to ensure the quality of the data collected in the secure electronic data capture system. Each Project LAUNCH grantee has been assigned a liaison from the cross-site evaluation team who is trained on the data reporting system. The liaison will be available to provide technical assistance to grantees as they have questions about the data reporting requirements. (The liaisons will also conduct the site visit and telephone interviews with their assigned grantees.)

The evaluation team provides technical assistance to grantees to ensure that the data collected through our electronic reporting system are of the highest quality. Technical assistance helps grantees to identify which Web-based survey to use in reporting each of their Project LAUNCH-supported activities. Technical assistance will also be provided in use and navigation through the Web-based reporting system as well as the method to download their own data for use in the grantee’s local evaluation. Assistance also covers data cleaning and checks, as well as timelines for data submission. A User’s Manual and technical assistance will be provided to grantees on reporting each data element (including definitions) within the electronic surveys.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

Project LAUNCH grantees are required to participate in the cross-site evaluation as a condition of receiving Project LAUNCH funding. Grantees recognize that they are involved in a new and innovative initiative and that results from this initiative could be important for and make a contribution to the field. Additionally, grantees will use the data that they submit to the electronic reporting system for both local reporting and reporting to SAMHSA on grant implementation. Grantees receive Excel files every six months with the cumulative data they have reported, which will allow them to use the data in their own local evaluations and help them track their efforts over time. Our expected response rate for this effort is 100 percent.

Dealing with Nonresponse

Non-response is expected to be minimal.

Maximizing Response Rates

Cross-site evaluation liaisons will provide support and technical assistance to local evaluators to ensure that Electronic Data Reporting for Systems Measures and Services Measures are completed as thoroughly as possible. Technical assistance and training are provided to local evaluators to assist in maximizing response rates.

B4. Tests of Procedures or Methods to be Undertaken

Wherever possible, the evaluation uses measures that have been previously developed and tested and that have demonstrated validity and reliability. The Cross-Site Evaluation Team has vast experience developing instruments and incorporating existing measures where necessary. Systems development questions were selected based on a comprehensive review of existing measures from a number of other studies. Questions on child and family characteristics were derived from existing national surveys, including the Early Head Start Research and Evaluation Study and the SLAITS. To ensure that we have selected the most appropriate measures, the instruments were discussed with project consultants identified in Table B.1 below, local evaluators from the first cohort of six grantees, and federal staff from ACF and SAMHSA. Usability testing of the Web-based data system was conducted in 2009 by evaluators from the first cohort of grantees. In addition, the interview guides were revised and some questions were eliminated based on data collection experiences to date (as discussed in Supporting Statement A).

tABLE B.1
Membership of Project LAUNCH EXPERT CONSULTANT Group

Individual	Affiliation
Beulah Allen, MD	National Indian Youth Leadership Project
David M. Chavis, PhD	Community Sciences
Paul Florin, PhD	University of Rhode Island
Stephanie M. Jones, PhD	Harvard University
Milton Kotelchuck, PhD	Boston University
Deborah Leong, PhD	Metropolitan State College of Denver
Judith S. Palfrey, MD	Harvard Medical School
Michelle Christensen Sarche, PhD	American Indian Alaska Native Programs
Ruth E.K. Stein, MD	Albert Einstein College of Medicine
Joseph Trimble, PhD	Western Washington University
Abraham H. Wandersman, PhD	University of South Carolina

B5. Individual Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Abt Associates Inc. is conducting this project under contract to ACF. The plans for statistical analyses for this study were developed by Abt Associates. The team is led by Deborah Klein Walker, Principal Investigator; Margaret Gwaltney, Project Director; and Barbara Goodson, Evaluation Design Lead. At the federal level, Laura Hoard is the ACF COTR for the Cross-Site Evaluation, and Jennifer Oppenheim and Yanique Edmond are the SAMHSA program leads for Project LAUNCH.

b Cohort 3 grantees do not have a state coordinator because the cooperative agreement was awarded only to local communities.

7