Project LAUNCH Cross-Site Evaluation
Supporting Statement Part A
For OMB Approval
The Administration for Children and Families (ACF) of the U.S. Department of Health and Human Services is requesting Office of Management and Budget (OMB) clearance for instruments to be used in data collection for a cross-site evaluation of a new initiative called Project LAUNCH (Linking Actions for Unmet Needs in Children’s Health). The Substance Abuse and Mental Health Services Administration (SAMHSA) awarded six five-year cooperative agreements under Project LAUNCH in September 2008, 12 additional cooperative agreements in September 2009, and six additional cooperative agreements in September 2010. Through an intra-agency agreement between SAMHSA and ACF, the Office of Planning, Research and Evaluation within ACF is overseeing the development and implementation of a cross-site evaluation of Project LAUNCH.
The purpose of Project LAUNCH is to promote healthy development and wellness in children birth to 8 years of age. Project LAUNCH is intended to address issues in the child service system by enhancing systems coordination, integrating child behavioral health services with other health services, and incorporating evidence-based programs to address children’s healthy development. Each Project LAUNCH grantee is expected to focus on systems-level development at the State or Tribal level and in one designated community. Additionally, within the designated or “local” community, grantees are also required to implement promotion and prevention activities within each of the five categories: 1) mental health consultation; 2) developmental assessments across service settings; 3) integration of behavioral health into primary care; 4) family strengthening and parenting skills training; and, 5) home visitation.
Project LAUNCH is authorized under Section 520A of the Public Health Service Act (42 U.S.C. 290bb–32) and addresses Healthy People 2010 focus area 18 (Mental Health and Mental Disorders). In order to provide SAMHSA, Congress, and the public with information on how this new and comprehensive initiative is implemented and on its outcomes, ACF has contracted with Abt Associates Inc. to conduct a cross-site evaluation. The cross-site evaluation will collect information from Project LAUNCH grantees related to State, Tribal, and local systems development; implementation of evidence-based services in local communities; and service system outcomes for children, families and providers.
This submission to OMB is a revision to the OMB Control #0970-0373, approved on January 8, 2010. The instruments have not changed. There are three changes proposed:
In the previous ICR, the first instrument was entitled, “Site Visit Interview Guide,” as site visits were originally planned each year. We are now proposing telephone interviews – using the same instrument – to be conducted in alternate years with site visits, to minimize burden on grantees. This instrument is, therefore, re-titled “Site Visit and Telephone Interview Guide.”
In the previous ICR, the number of respondents for the second and third instruments was 18 – corresponding with the 18 cooperative agreements funded in 2008 and 2009. We are now proposing to increase the number of respondents from 18 to 24, to include the most recent six cooperative agreements funded in 2010.
Although clearance has been granted until January 2013, the current submission requests clearance for a three-year period of data collection for the 24 Project LAUNCH grantees through summer 2014 (or three years from approval of this package). ACF will then submit an extension request for the continuation of evaluation activities during the final years of the grants.
The cross-site evaluation will describe how Project LAUNCH efforts affected State and Tribal policies pertaining to children and families, led to expansion of existing services or implementation of new services, and increased service coordination. Although this is not an impact evaluation, findings will also document outcomes for children, families, and providers touched by Project LAUNCH.
Data collection methods for the cross-site evaluation will consist of 1) telephone interviews with Project LAUNCH grantees alternating with site visits to grantees, and 2) a web-based data reporting system related to systems development and services implementation and outcomes.
During the site visits, evaluators will conduct interviews with Project LAUNCH administrators, collaborators and service providers in State and Tribal areas and local communities of focus. The site visits have two primary objectives. First, the cross-site evaluation will collect data about Project LAUNCH activities aimed at infrastructure development and systems change at both the State/Tribal and community levels. We will seek to understand how these activities are implemented and document system change outcomes. Second, the cross-site evaluation seeks to understand grantees’ progress in program implementation and service delivery. During site visits, members of the evaluation team will collect information about the implementation of services within each of the five service categories of interest to Project LAUNCH.
Similarly, during the telephone interviews, evaluators will hold conversations with Project LAUNCH state and local coordinators for Cohorts 1 and 2, and with the local coordinator and a key state staff (e.g., the state wellness council lead, the Maternal and Child Health Title V director or the lead state person on the community wellness council) for Cohort 3. The evaluation team will collect information about infrastructure development and systems change at the state and local level as well as implementation of services in the five service categories. Data collected during the telephone interviews and site visits will expand and aid in explaining data reported by grantees in the web-based data reporting system.
Through the web-based data collection system, grantees will be asked to report semi-annually on systems development (at the State, Tribal, and local levels) and on services (including implementation of services or trainings for children, families and providers; services delivered; demographics of children, families and providers receiving Project LAUNCH services; and changes in service delivery settings and provider practices).
Grantee-specific child and family outcomes will be reported by the grantees as part of their local evaluations as per SAMHSA grant requirements and therefore not calculated in the burden requested in this clearance. However, findings reported by grantees using experimental or quasi-experimental designs to assess outcomes of Project LAUNCH services may be utilized in reports from the cross-site evaluation.
Evaluation findings will provide ACF and SAMHSA with ongoing information about program operations, apparent strengths and challenges, and areas where grantees would benefit from technical assistance and support. Accordingly, the evaluation will help guide policy development and program improvement at all levels. Evaluation findings will be routinely shared with Project LAUNCH grantees so that they can learn from each other, and can benefit from innovative strategies they are using in their project designs as well as how they are overcoming challenges to reaching their goals. Finally, cross-site evaluation results will support and enrich program implementation among new cohorts of grantees if Project LAUNCH is expanded.
The proposed data collection will use a web-based data entry system, previously developed for the Center for Mental Health Services (Decision Support 2000+ platform), to collect information in a uniform manner across grantees. This is a NET-based suite of online applications that provides the capacity for rapidly producing secure data capture and project collaboration portals tailored to the needs of SAMHSA evaluation and grantee monitoring. It is user-friendly and facilitates accurate and expeditious data collection.
This on-line data capture system offers the following major features that will support both data collection and program management:
A performance measurement site with state-of-the-art data capture and upload/download features, real-time data quality reports, trend reports, and program comparisons; and
Project management tools providing technical assistance materials, helpdesk access, and a space for clients and project staff to collaborate on documents, share schedules, contact information, and project task lists.
To minimize burden in subsequent data reporting periods, information entered at the first data collection time point will be pre-loaded for each grantee. Grantees will designate individuals needing access to the system.
Because Project LAUNCH is a new initiative, there have not been any similar cross-site data collection efforts. Although SAMHSA requires grantees to conduct local evaluations of their Project LAUNCH programs, the cross-site evaluation is designed to provide a more comprehensive examination of Project LAUNCH activities and outcomes across the communities served. To reduce the possibility of duplication of efforts between local evaluations and the national evaluation, the cross-site evaluation is designed to build upon and complement local data collection plans specified by Project LAUNCH grantees in their grant applications.
There is no plan to auto-populate or carry over data from local evaluation reports. However, we will auto-populate some questions in the data reporting system using previously reported data after grantees have reported in the web-based data reporting system. Examples of data that will be auto-populated include the program description, the target population, and the location in which services occur.
No small businesses are impacted by the data collection for this project.
A.6 Consequences of Collecting the Information Less Frequently
To address the study’s research objectives about how Project LAUNCH is implemented over time and how service provision is affected, telephone interviews or site visits will be conducted with grantees each year. However because site visits are more time-intensive for grantees to coordinate and schedule, telephone interviews will be conducted every other year to minimize grantee burden. Data about services provided, numbers of families served, system change efforts at the State, Tribal, and local levels, and service system outcomes for children, families and providers will be required semi-annually to correspond with SAMHSA’s semi-annual progress-reporting periods.
There are no special circumstances that might require deviation from the guidelines of 5 CFR 1320.5.
The first Federal Register notice for the Cross-Site Evaluation of Project LAUNCH was published in the Federal Register Vol. 76, No. 25, Page 6619 on February 7, 2011. During the notice and comment period, one public comment was received but did not relate to the instruments or data collection. A copy of the 60-day notice is included in Appendix A.
Many individuals and organizations, including the Project LAUNCH Expert Consultant Group, were contacted for advice on aspects of the evaluation design and data collection instruments. Their feedback was obtained through in-person meetings and telephone conversations. Members of the Project LAUNCH Expert Consultant Group are listed in Table A.1.
tABLE A.1
Membership of Project LAUNCH EXPERT CONSULTANT Group
Individual |
Affiliation |
Beulah Allen, MD |
National Indian Youth Leadership Project |
David M. Chavis, PhD |
Community Science |
Paul Florin, PhD |
University of Rhode Island |
Stephanie M. Jones, PhD |
Harvard University |
Milton Kotelchuck, PhD |
Boston University |
Deborah Leong, PhD |
Metropolitan State College of Denver |
Judith S. Palfrey, MD |
Harvard Medical School |
Michelle Christensen Sarche, PhD |
American Indian Alaska Native Programs |
Ruth E.K. Stein, MD |
Albert Einstein College of Medicine |
Joseph Trimble, PhD |
Western Washington University |
Abraham H. Wandersman, PhD |
University of South Carolina |
In addition to the Expert Consultant Group, guidance on the evaluation design was sought from Russell Glasgow, Ph.D., the developer of the evaluation framework that will be used to guide the cross-site evaluation. This framework, called RE-AIM, is described more fully in section A.16.
No payments or gifts will be provided to respondents as part of data collection.
For each of the data collection methods, procedures will be in place to provide assurance of privacy to respondents to the fullest extent of the law.
Site visits and telephone interviews
During the site visits and telephone interviews with grantees, we will speak with a variety of State, Tribal, and local project staff. Each respondent will be asked to sign a consent form to participate in an interview, and the consent form will explain the study procedures for assuring that the answers provided by the respondent will remain private. The consent form will explain that (a) participation in the interviews is voluntary and there are no penalties for refusing to participate at any time during the interviews; (b) the respondent can refuse to discuss any topic; (c) data will be stored in de-identified files and (d) no names will be used in any evaluation reports. At the time of the interviews, respondents will be asked to sign two copies of the consent form. One will be retained by the cross-site evaluator, and the second copy will be given to the respondent for his/her files. A copy of telephone interview and site visit consent forms are provided in Appendix B.
Although we will not indicate the names and titles of individuals at Project LAUNCH sites who provide information during the site visits or telephone interviews, it is possible that we will be drawing comparisons between sites and identifying grantees by name. We will inform all study participants about this during our consent process prior to initiating the interview.
Semi-annual web-based data reporting
The evaluation will collect information using an on-line data collection system: DS2000+. This system has passed internal audits and tests conducted by the Abt Associates Inc. Security Officer and have been reviewed on multiple occasions by SAMHSA IT staff members. The DS2000+ data collection and analysis modules include https certificate-based authentication and transaction encryption processes approved for HIPAA data transactions on Federal healthcare projects. The DS2000+ platform provides a modular permission system allowing data segregation by role, individual, and system object. Accordingly, it will not be possible for any individuals who are not part of the research team to access these data.
A large portion of the data grantees will report through the web-based reporting system is about service delivery and will involve data already being collected by grantees for their local evaluations. All information on services will be reported to the cross-site evaluation in the aggregate; thus no information collected can be linked to individual families or individual agencies. To the extent that the local evaluations involve collecting information on services provided to individual families and children, the cross-site evaluation team will work closely with local grantees and their regional and State Institutional Review Boards (IRBs) to ensure that the necessary IRB approvals are obtained, and that human subject protections are assured. No respondent identifiers will be contained in public use files made available from the cross-site evaluation.
The cross-site evaluation will be requesting information in the aggregate about the demographics of children and families served. Although these data will be reported to the cross-site evaluation only in the aggregate through the web-based reporting system, local program staff will be collecting it on an individual basis from parents as they are enrolled in services and upon completion or termination of services. As part of this data collection, local project staff will collect written consent from each parent for participation in the evaluation. The consent form will describe the procedures to assure privacy for the respondents, and the interviewers will be knowledgeable about privacy procedures and will be prepared to describe them in detail or to answer any related questions raised by respondents. Necessary consent forms and privacy protections will be developed and administered by each grantee. Each grantee will obtain IRB approval from their regional and/or State IRB, as well as the institutional IRB overseeing the local evaluation, and therefore will be subject to the terms and guidelines of their specific IRB. The cross-site evaluation team will provide technical assistance in drafting these documents and responding to IRB concerns, if requested by the grantee. Copies of the consent forms from individual families and children will be stored at the local sites, under lock and key.
The cross-site evaluation will be requesting information on providers in local sites who participated in Project LAUNCH-related activities. These include trainings, mental health consultations services and integration of behavioral health and primary care. Specific questions will ask about how these services have changed their practices in caring for children. Although these data will be reported to the cross-site evaluation only in the aggregate through the web-based reporting system, local program staff will be collecting it on an individual basis from providers as part of their local evaluations. Each grantee has or will obtain IRB approval from their regional, local and/or State IRB, as well as the institutional IRB overseeing the local grantee evaluation, and therefore will be subject to the terms and guidelines of their specific IRB. Copies of the consent forms from individual providers will be stored at the local sites, under lock and key.
All data collected via the web-based portal will be stored electronically through Abt’s password protected secure network system. Project directories and databases are protected at Abt Associates by assigned group memberships, passwords and other techniques (e.g. ACLs), which prohibit access by unauthorized users. In addition to the issue of protection of privacy, data security encompasses backup procedures and other file management techniques to ensure that files are not inadvertently lost or damaged. Project data files are backed up to tape, using fast dump/restore software (backup Exec version 8.5) and DLT-4 tapes that hold up to 70GB of compressed data. The procedures currently utilized at Abt Associates ensure the privacy and security of many various Abt Associates research databases.
To achieve the goal of describing the families participating in local Project LAUNCH services, local program staff will be collecting information from parents that will include some sensitive questions about family and child risk factors. Although this information is sensitive, it is necessary to accurately describe families who receive Project LAUNCH services. The questions employed are from standardized measures or have been used extensively in prior studies with no evidence of harm (for example, in the Fragile Families Study and in the Early Head Start Research Evaluation Project).
The data on individual families and children being collected by the local Project LAUNCH grantees and their local evaluators will be dictated by the guidelines set forth by their IRBs and HIPAA policies to ensure the confidentiality of sensitive information. As part of the consent process, participating parents will be informed that they might find some questions sensitive, and will be asked to sign a consent form to participate, acknowledging that their participation is voluntary. All respondents will be informed that their identity will be kept private and that they do not have to answer questions that make them uncomfortable.
All information collected at the individual child/family level will be de-identified and reported in the aggregate to the cross-site evaluation.
The proposed data collection does not impose a financial burden on respondents nor will respondents incur any expense other than the time spent completing the interviews.
The estimated annual burden for study respondents—State, Tribal, and local Project LAUNCH program staff, evaluators, providers and stakeholders—is identified in Table A.2. These estimates were calculated based on an assumption of 24 Project LAUNCH grantees, and, for annual site visits, 12 respondents per site.
To compute the total estimated annual cost, the total burden hours were multiplied by the average hourly wage for each adult participant, according to the Bureau of Labor Statistics, Current Population Survey, 2009. We assumed that most respondents to the site visit and telephone interviews and web-based data reports (state and local systems data; local services data) would be program directors and program evaluators, and accordingly, used the mean hourly rate for Social Scientists and Related Workers, All Other ($35.31 per hour) to assess the cost of their providing this information.
This estimate is based on both the experience of the site visits as well as the past practices and experiences of the cross-site evaluation team. Additionally, not all questions in the interview guide are relevant to every individual who will be interviewed. For these reasons, we consider this estimate of burden to be accurate.
TABLE A.2
ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST
Instrument |
Annual Number of Respondents |
Number of Responses per Respondent |
Average Burden Hours per Response |
Estimated Annual Burden Hours
|
Average Hourly Wage |
Total Annual Cost
|
Site Visit and Telephone Interview Guide |
240 |
1 |
1.25
|
300 |
$35.31 |
$10,593.00 |
Electronic Data Reporting: Systems Measures |
24 |
2 |
4
|
192 |
$35.31 |
$6,779.52 |
Electronic Data Reporting: Services Measures |
24 |
2 |
8
|
384 |
$35.31 |
$13,559.04 |
Estimated Annual Burden Hours: |
876 |
|
$30,931.56 |
|||
Not applicable.
The total cost to the federal government for the three years of data collection and reporting under the contract with Abt Associates is estimated to be $4,373,400. Thus, annual costs (total costs divided by three) to the government are $1,457,800.
A.15 Explanation for Program Changes or Adjustments
This submission to OMB is a revision to the OMB Control #0970-0373, approved on January 8, 2010. The instruments have not changed. There are three changes proposed:
In the previous ICR, the first instrument was entitled, “Site Visit Interview Guide,” as site visits were originally planned each year. We are now proposing telephone interviews – using the same instrument – to be conducted in alternate years with site visits, to minimize burden on grantees. This instrument is, therefore, re-titled “Site Visit and Telephone Interview Guide.”
In the previous ICR, the number of respondents for the second and third instruments was 18 – corresponding with the 18 cooperative agreements funded in 2008 and 2009. We are now proposing to increase the number of respondents from 18 to 24, to include the most recent six cooperative agreements funded in 2010.
Although clearance has been granted until January 2013, the current submission requests clearance for a three-year period of data collection for the 24 Project LAUNCH grantees through summer 2014 (or three years from approval of this package). ACF will then submit an extension request for the continuation of evaluation activities during the final years of the grants.
The cross-site evaluation of Project LAUNCH is intended to address the following four consolidated questions:
Q1: What are the system level changes at the state/Tribal level, including
Improved coordination and collaboration across agencies serving young children and families;
Sustained implementation of a coordinated, family-centered, culturally competent child-serving system;
Improved infrastructure, legislation and other policies;
Increased public education outreach and awareness; and
Sustained funding and maintenance of child-serving systems?
Q2: What are the system level changes at the community/local level, including
Improved coordination and collaboration across agencies serving young children and families;
Sustained implementation of a coordinated, family-centered, culturally competent child-serving system;
Improved infrastructure, legislation and other policies;
Increased public education outreach and awareness; and
Sustained funding and maintenance of child-serving systems?
Q3: How have the child and family services in the community changed, including
Workforce development;
Increased number of providers trained in the EBP curriculum
Increased provider knowledge about appropriate referrals
Providers with increased knowledge of child development and behavioral health
Setting practices; and
Increased implementation of developmental screening and assessment in a range of primary care and early childhood settings;
Implementation/expansion of integration of mental health into primary care;
Implementation/expansion of mental health consultation for providers in a variety of settings;
Implementation/expansion of evidence-based prevention and wellness-promotion practices, including home visiting and family strengthening and parent training programs;
Implementation of culturally-relevant, family-centered practices in a range of primary care and early childhood settings; and
Increased number of children and families served?
Q4: What is the effect on the overall development and wellness of young children in the Project LAUNCH community, including
Increased number of children reaching physical, social, emotional, behavioral, and cognitive developmental milestones; and
Increased number of children entering school ready to learn (including physical, social, emotional, behavioral, and cognitive readiness)?
To address these questions, the cross-site evaluation will utilize two basic data collection components – site visits/telephone interviews and web reporting on systems and services. Table A.3 shows which data collection components will be used to address each of the evaluation questions. In the site visits and telephone interviews, evaluation staff will interview State, Tribal, and local staff to obtain information on the community context, service coordination and systems infrastructure that support services. The web-based reporting system will have two components. One component will focus on State, Tribal, and local systems-level development. A second component will track aggregate numbers of services and trainings delivered to children, families, and providers. The services component includes aggregate demographics collected on families who receive services and information about provider and provider setting changes.
The data on service delivery will be used to produce a profile for each of the evidence-based practices being implemented under Project LAUNCH. The profile will be based on the RE-AIM framework. RE-AIM conceptualizes the ultimate public health benefit of an intervention as being the combined effect of five “evaluative dimensions”: Reach (proportion of the population that participated in the intervention), Effectiveness (outcomes), Adoption (proportion of settings that adopt the intervention), Implementation (extent to which the intervention is implemented as intended in the real world), and Maintenance (extent to which the intervention is sustained over time). According to its developers, the approach is consistent with systems-based,1,2 community-based, and public health interventions.2 Using RE-AIM, the public health impact is expressed as a single indicator, derived as the multiplicative combination of the five dimensions. The index can be used to evaluate the overall public health impact of an intervention3 or to compare two or more public health interventions.
In the case of the evaluation of Project LAUNCH, each service will be scored on three of the RE-AIM dimensions: Reach, Implementation, and Adoption, thus creating a modified RE-AIM score. Scores will be on a scale of 0 to 100%. These scores will be used to characterize service delivery within each of the five service types that Project LAUNCH grantees are expected to implement: home visiting, family strengthening, developmental assessments, mental health consultation, and integration of mental health into primary care. If a site is implementing multiple services within a category, the scores for Reach, Implementation and Adoption will be averaged across those services.
TABLE A.3
Evaluation Questions and Data SOURCES
Evaluation Question |
Data Source |
||
Site Visits/ Telephone Interviews |
Semi-annual Web-based Reporting |
Grantee Local Evaluations |
|
1. System level changes at the state/Tribal level |
X |
X |
|
2. System changes at the community/local level |
X |
X |
|
3. Changes in child and family services in the community |
X |
X |
|
4. Changes the overall development and wellness of children in the Project LAUNCH community |
|
|
X |
Analyses will employ a variety of methods, including descriptive statistics (means, percentages) and simple tests of differences across subgroups and over time (t-tests, chi-square tests). Most of the evaluation questions call for descriptive analyses, which can be answered by calculating averages and percentages of families, children, providers participating in services; average scores on service delivery outcomes; and comparisons of these averages across grantees and across time. Cross-tabulations of program characteristics and family characteristics will also provide important information about differences across programs in the kinds of families they serve and how different families are served.
The time schedule for fielding, analyzing, and reporting the data findings for the cross-site evaluation of Project LAUNCH is as follows:
Site Visits/Telephone Interviews Fall 2011, 2012, 2013
Semi-Annual Web-based Reporting Spring/Fall 2011 – 2013
Data Analysis Fall 2011 – Summer 2013
Data Reports Annual
To date, there have been no publications from the study. However, regular reports will be produced. Abt will prepare a set of tables for all data collected from the site visits/telephone interviews and the web portal annually. This information will be used in short briefs, reports and presentations requested by ACF/SAMHSA; these may include brief reports for the project website, brochures or fact sheets for providers and/or families, issue briefs, or peer-reviewed journal articles.
A final cross-site evaluation report will be produced at the end of the contract period. The final report will include a description of data collection and analysis methods and a summary of findings.
Not applicable. The OMB number and expiration date will be displayed on the top of the first page of each data collection instrument and consent form including the top of the first web page for the web-based data collection system.
No exceptions are necessary for this data collection.
REFERENCES
1 Stokols, D. Establishing and maintaining health environments: toward a social ecology of health promotion. Am Psych; 1992;47: 6-22.
2 Green, LW & Johnson, JL Dissemination and utilization of health promotion and disease prevention knowledge: theory, research and experience. Can J of Pub Health: 1996 (suppl 2); 11-17.
3Bopp, M. Wilcox, S Laken, M Hooker, SP Saunders, R Parra-Medina, D. Using the RE-AIM framework to evaluate a physical activity intervention in churches. Prev Chronic Dis: 2007; 4(4): 1-9.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Dawn Smith |
| Last Modified By | borson_j |
| File Modified | 2011-06-29 |
| File Created | 2011-06-29 |