Supporting Statement B - LTO GenIC May 2020

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data: Identifying Targets of Opportunity

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

April 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Brett Brown

Part B

B1. Objectives

Study Objectives

The objectives of this study are to understand the current challenges and opportunities for linking employment and child/youth development related evaluations to administrative datasets to examine long-term outcomes and provide the Administration for Children and Families (ACF) with information about the steps and resources needed to pursue long-term follow-up (using admin data) for each of the 8 studies in our sample (mentioned below in Target population).

Generalizability of Results

This study is intended to present reliable descriptions of the parameters and procedures for linking evaluation datasets to administrative data for 8 major evaluations (mentioned in Section B2). The study will not promote statistical generalization to other studies, sites, or service populations, but can provide useful lessons on the various aspects that need to be considered when assessing if it is possible to link evaluation data to administrative data to estimate long-term impacts.

Appropriateness of Study Design and Methods for Planned Uses

To accomplish the study objectives, the project team will be conducting interviews with people who worked on or have significant knowledge of 8 evaluations. These interviews build on data collected in a previous phase of the project (approved under ACF’s generic clearance for formative data collections; OMB #0970-0356). In that phase, the project team surveyed evaluation staff using a data collection template. We are now following up with the evaluation teams to collect more in-depth information. We believe interviews are the best way to collect this information because we will be able to get additional information as needed. Understanding the parameters and procedures for doing long-term follow-up by linking with administrative data is complicated and requires input from multiple stakeholders (e.g. including those who review the legal and ethical parameters of study proposals, provide access to data, and conduct the analyses). Given these considerations and the wide variety of information needed, interviewing multiple informants who understand different aspects of the study for each evaluation was determined to be the best approach.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The target population is staff who were involved in or have knowledge of the following 8 evaluations:

• Evaluation of Accelerated Training for Illinois Manufacturing (ATIM)

• The Employment, Retention, and Advancement Project & Hard-to-Employ Demonstration and Evaluation Project¹

• Florida's Independence Project

• Fragile Families and Child Wellbeing Study

• Los Angeles Jobs-First Greater Avenues for Independence (GAIN) Evaluation

• Welfare Restructuring Project

• WIA Adult and Dislocated Worker Programs Gold Standard Evaluation

• National YouthBuild Evaluation

For each study, we are targeting individuals in the following roles:

• Project Director or Principal Investigator

• Evaluation Data Manager

• Evaluation Legal Representative

• IRB Representative

• Original or Primary Funder

• Administrative Data Source Provider

Data will be collected in two waves. The size of this population for Wave 1 will be 8 individuals and for Wave 2 will be 35 individuals, for a total of 43 individuals.

Sampling

There is no sampling being conducted in this data collection effort.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument(s)

To develop the interview protocols, the project team developed a data collection planning tool. The tool identified broad categories of information and potential informants needed to assess the feasibility of conducting long-term follow-up research. The team then developed specific questions and potential follow-up questions within each of those categories. All of the questions build on information collected in earlier stages of the project, and seek to fill gaps in information or gather more detailed information. From this planning tool, the project team created interview guides (or protocols) for each informant type (which is based on his/her role in the original study). Please see Instrument 1: LTO Task 3 Informant Interview Protocols for the full set of questions.

Given the data collection methods, minimization of measurement error is not applicable.

B4. Collection of Data and Quality Control

MDRC has contracted with the ACF Office of Planning, Research, and Evaluation (OPRE) in the United States Department of Health and Human Services (HHS) and will be the agency collecting the data using interviews. MDRC will be interviewing people who worked on the above-mentioned evaluations.

After gaining OMB approval, members of the project team will contact Wave 1 informants, who completed the data collection template during the original performance period of the project. The project team will contact targets by e-mail and phone to request interviews. During Wave 1 data collection, we will request suggestions for Wave 2 informants. These are people who they perceive to be better suited to answer specific questions about different feasibility aspects.

The study team will use email and phone scripts that explain the project and its goals, including background about the specific information needed (see Appendix C - LTO Outreach Informant Interviews Email). An information memo (Appendix B - LTO Outreach Informant Interviews Cover Memo) will also be attached to e-mails. This project description is intended to provide potential interviewees context on why they are being asked to participate in the data collection.

To ensure quality and consistency across interviews, the project team will meet internally before the interviews are conducted to discuss the primary goals of the interviews and any questions on the content being covered. We will also draw on prior experience conducting similar types of interviews in other evaluations. Further, at least two individuals from the project team will participate in each interview and after each interview, the team will debrief about the information collected. The interview team will record the interviews if given permission from the interviewee to ensure accuracy of the information collected. In instances where the information collected is unclear, the project team will reach back out to the interviewees to clarify the response and will then make corrections, as needed.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Still, we want to minimize nonresponse so that we do not have to rule out studies that may otherwise be good candidates for long-term follow-up. Therefore, we will use several strategies to increase participation:

• Initial outreach to respondents will be done by senior project staff who have connections with members of the evaluation teams. We will also draw on our previous connections with respondents who participated in the data collection effort completed in the previous phases of the project.

• If individuals do not respond to the initial emails that are sent, we will follow-up with them by phone.

  • Wave 1 of data collection involves a request for evaluation project directors and principal investigators to identify key respondents for Wave 2. If Wave 2 respondents do not respond to our initial outreach, we will reach out to Wave 1 respondents to directly connect us to Wave 2 respondents.

• The study team also has numerous relationships with individuals at the federal government and other research organizations. We will ask our current contacts to help identify and put us in touch with individuals involved in the evaluations.

B6. Production of Estimates and Projections

The information collected will be for internal use only, primarily to identify possible future studies. Therefore, no estimation or projection methods are necessary.

B7. Data Handling and Analysis

Data Handling

Data will be collected through semi-structured interviews. The project team will take detailed notes with interviewees responses. The interviews will also be recorded for the purposes of fact checking. Following each interview the project team will listen to the recordings and update the responses as needed. Spelling and grammatical errors within the interview protocols will be checked through reviews by the project team.

Data Analysis

The data collected for this effort will be used in conjunction with data collected during the earlier stages of the project where data was collected. The project team will analyze and summarize the information collected to describe the practical, legal, and ethical parameters and the likely steps required to conduct long-term follow-up for the 8 evaluations in our sample.

Data Use

No documentation from this effort will be released to the public. These data will be used to prepare an in-depth long-term outcome feasibility report that will feature a detailed profile for each of the 8 studies. This report will be for the use of ACF/OPRE only.

B8. Contact Person(s)

The following people from the study team will be responsible for collecting and/or analyzing data:

Ms. Alexandra Pennington (212) 340-8847 alexandra.pennington@mdrc.org

Mr. Jonathan Bigelow (212) 340-8646 jonathan.bigelow@mdrc.org

Ms. Kelsey Schaberg (212) 340-7581 kelsey.schaberg@mdrc.org

Dr. Richard Hendra (212) 340-8623 richard.hendra@mdrc.org

Attachments

Appendix A - LTO Outreach Informant Interviews Cover Memo

Appendix B - LTO Outreach Informant Interviews Phone Script

Appendix C – LTO Outreach Informant Interviews Email

Instrument 1 - Informed Consent and Interview Protocols

1We are collecting information about both evaluations with the intention of assessing the feasibility for conducting a long-term follow-up study that builds on the evidence from both of these studies. The overall level of effort to complete data collection for these studies is assumed to be within the parameters of the agreed upon level of effort associated with data collection for one study.

7