NextGen Project Full ICR SSB_v18

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Next Generation of Enhanced Employment Strategies Project

OMB Information Collection Request

0970 - 0545

Supporting Statement

Part B

AUGUST 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Marie Lawrence

Sarita Barton

Megan Reid

Part B

B1. Objectives

Study Objectives

The Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for data collection activities conducted for the Next Generation of Enhanced Employment Strategies Project (NextGen Project). The objectives of this project are:

1. To identify and rigorously evaluate the effectiveness of innovative programs designed to promote employment and economic security among people with complex challenges to employment

2. To describe the operations, implementation successes and challenges, and lessons learned for each program

3. To estimate the costs of each studied program

This request is for a nonsubstantive change to the currently approved follow-up surveys. We added one question to each survey (first follow-up survey – Instrument 3 and second follow-up survey – Instrument 4) that collects Social Security number if the study participant did not provide it upon study enrollment.

Generalizability of Results

The impact studies, which are randomized controlled trials (RCTs), are intended to produce internally-valid estimates of the programs’ causal impacts, not to promote statistical generalization to other sites or service populations. The descriptive and cost studies are intended to present internally-valid descriptions of the service population, implementation, and cost of the chosen programs, not to promote statistical generalization to other sites or service populations. 

Appropriateness of Study Design and Methods for Planned Uses

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

The study’s purposive selection of programs to evaluate and its impact, descriptive, and costs studies are appropriate for the government’s goal of identifying and rigorously evaluating innovative programs designed to promote employment and economic security among low-income people with complex challenges to employment.

• Impact studies. The results of the impact studies could be used to inform federal, state, and local policymakers about future funding of the tested programs; by program administrators and directors who might consider implementing the tested programs, or something like them, for their own programs; and program developers and technical assistance providers facilitating implementation of evidence-based practices. It is important that rigorous, internally valid methods are used to assess effectiveness. Therefore, as described further under the Study Design heading in Part A2, the project team is conducting a separate impact study for each program. Each impact study uses an RCT design.

• Descriptive studies. Findings from descriptive studies will support interpretation of the impact findings, describe the programs, and could help other programs replicate or refine the program for their own contexts.

• Cost studies. Results of the cost study are important for program funders and for practitioners considering implementing the programs.

One of the limitations of the study for its intended use (informing the design and adoption of future employment programs) is that the impact studies will produce an internally valid estimation of the impact of the program model and its implementation at the time of the study. Impacts are a function of the program model, its implementation, characteristics of the service population, and also of other similar services that are available in the studied community (sufficient treatment/control contrast). This context should be considered when interpreting whether and how findings will apply to program expansion. For this reason, the study will also include descriptive and cost studies of employment programs which the study team will use to interpret the impact estimates. This information will also help other program funders or operators considering implementing the programs.

B2. Methods and Design

Target Population

The target population for the NextGen Project is low-income individuals with complex challenges to employment, including physical and mental health conditions, a criminal history, or limited work skills and experience. The project is working closely with the Social Security Administration (SSA) to incorporate a focus on employment-related early interventions for individuals with current or foreseeable disabilities who have limited work history and are potential applicants for Supplemental Security Income (SSI).

The project is conducting impact evaluations of four programs. Each focuses on participants with one or more employment challenges and is aiming to achieve a sample size of at least 1,000 study participants, equally split into treatment and control groups. This will lead to a total sample size of up to 4,200 participants. The study team will conduct descriptive and cost studies of these four programs. It will also conduct cost studies of up to three others not participating in impact studies and descriptive studies of up to two of those. The project could add programs later if circumstances warrant.

Sampling and Site Selection

The site identification approach was described in detail in two previous Generic ICR submissions, one for stakeholder engagement and one for site assessment, both of which received approval under the generic clearance for Formative Data Collections for ACF Research (OMB #0970-0356). In summary, the programs for the project were selected to meet three general criteria:

1. The program addresses the research priorities of this project.

2. The program is well implemented, or could be after some technical assistance.

3. It is feasible to rigorously evaluate the program using an experimental design, or could be after the program received evaluation technical assistance.

Sampling for impact studies. The sample frame for the impact study is all people who are eligible for and interested in the program and consent to participate in the evaluation during the enrollment period. The project team is collecting survey information from all study participants at three points: (1) at baseline, before random assignment occurs; (2) at about 6 to 12 months after random assignment via the first follow-up survey; and (3) at about 18 to 24 months after random assignment via a second follow-up survey. The project team will attempt to survey the universe of study participants and will also examine administrative data on earnings, benefit receipt, and potentially other outcomes on all study participants. The team anticipates obtaining information from administrative sources for all sample members.

Table B.1 reports program-level minimum detectable differences on earnings outcomes for survey and administrative data by sample size. The target sample size for three of the four impact studies is 1,000 study participants, 500 each in the treatment and control groups. For the fourth impact study the goal is 1,200 study participants, 600 in each study group. For the administrative data, the project team expects nearly 100 percent coverage; they expect about an 80 percent response rate for each of the participant follow-up surveys.

For the administrative data, we expect data for 95 percent of the sample, with 5 percent not reporting their Social Security number or reporting an invalid number. The minimum detectable differences on earnings for a randomly-assigned sample of 1,200 are about $167 for average monthly earnings reported on the survey and $460 for quarterly earnings reported from administrative data. In other words, the study is powered to detect a difference of $167 or greater in monthly earnings between the program and control groups as measured through the follow-up surveys. If the true difference in earnings between the groups is less, the study likely will not detect a statistically significant impact. These minimum detectable differences correspond to minimum detectable effect sizes of 0.16 for the survey sample and 0.15 for the administrative data sample. For other programs with a randomly assigned sample of 1,000, the minimum detectable differences would be about $183 (effect size = 0.18) for average survey monthly earnings and $504 (effect size = 0.16) for administrative data quarterly earnings, respectively. Evidence reviews, such as the What Works Clearinghouse, consider effect sizes of 0.25 standard deviations or larger as substantively important (U.S. Department of Education 2022). However, note that these minimum detectable differences and corresponding effect sizes are only estimates based on data from previous studies on similar populations. Those previous studies provide suggestive evidence that under our assumptions, the projected sample size should generate adequate statistical power for the magnitude of impacts the program can generate. If target sample sizes, standard deviations of the outcomes, or responses rates differ from our expectation, the statistical power is likely to change as well. We also include power calculations for a sample size of 300—the minimum sample size for subgroup analysis.

Table B.1. Minimum detectable difference (MDD) per program by sample size on key outcomes for an RCT

Study sample (program and control)	Monthly earnings measured with survey data		Quarterly earnings measured with administrative data
	MDD of impacts in dollars	MDD of impacts in effect sizes	MDD of impacts in dollars	MDD of impacts in effect sizes
1,200	$167	0.16	$460	0.15
1,000	$183	0.18	$504	0.16
300	$334	0.32	$920	0.30

Assumptions: individuals are randomly assigned; equal random assignment probabilities for program and control groups; $1,034 standard deviation of monthly earnings and $3,102 standard deviation of quarterly earnings; covariates explain 20 percent of the variation in the outcomes; response rate of 80 percent on the survey; match rate of 95 percent for the administrative data; two-tailed test, p-value of 0.05.

Respondent recruitment for descriptive studies. The descriptive studies are based on three types of data collection, some of which involve purposeful respondent recruitment:

1. Semi-structured discussions with program staff, leaders, and, if applicable, partners and employers. Program staff and leaders are recruited purposively for discussions using organizational charts and information on each employee’s role at the organization. Staff from partner organizations and employers, if applicable, are recruited based on their involvement with the program and its participants. Purposeful respondent recruitment is appropriate because particular insights and information available from individuals will depend on their perspectives based on their role at the organization. The results of the descriptive study are not intended to generalize beyond the program being studied.

2. Surveys of program staff and leaders. The universe of all frontline staff and leaders at the selected programs are asked to complete a web-based staff or leadership survey collecting information on their professional backgrounds, skills, experience, and perceptions of the program. This will provide a broader perspective on these topics than can be elicited through the interviews, and thus targeting the universe of staff and leaders is appropriate.

3. In-depth interviews of program participants. The project team is recruiting approximately 20 treatment group members from each program to complete the interviews among treatment group members who have participated in the program. The team selects treatment group members who were randomly assigned at least six months before the interviews so that they include study participants who have potentially participated in the program for six months. These interviews provide narrative, in-depth context and experiences of program participants.

Sampling for cost studies. Leaders from each participating program (or their designees) may submit their accounting records to the project team, who will use them to complete a standardized Excel-based workbook. Or, program staff who are familiar with the program's expenditure and accounting records may directly complete the workbook.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Table B.2 lists the data collection instruments relevant to the extension period that started in April 2023 and links them with the study’s objectives. The data collection instruments were developed to capture essential data for the study’s main research questions that are not readily available from administrative sources.

A description of how each question in the baseline survey, identifying and contact information, and follow-up surveys will be used in the analysis is provided in Appendices B, C, and D. Appendices B and D link each question to its objective in the analysis. These appendices also include references for items that were used in previous studies.

Data for impact studies. The baseline and two follow-up survey instruments for the NextGen Project were developed by content experts at Mathematica and OPRE and informed by reviewing instruments used in similar data collection efforts. Many questions are sourced from prior studies, such as the Parents and Children Together study (OMB #0970-0403), the Evaluation of Supplemental Nutrition Assistance Program (SNAP) Employment and Training Pilots (OMB #0584-0604), Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506), and the National Beneficiary Survey (OMB #0960-0800). Other items come from scales that have been frequently used in large-scale national surveys, such as the SF-12® Health Questionnaire to assess health status. Finally, the surveys were developed in coordination with the OPRE Building Evidence on Employment Strategies for Low-Income Families Project (BEES) study (OMB #0970-0537), with which the NextGen Project is coordinating (as described in Part A). Areas of measurement coordination with the BEES instruments are described in the question-by-question justifications for the baseline and follow-up surveys (Appendices B and D).

The project team used industry best practices to reduce potential sources of measurement error. These practices include:

• Using validated items from previous surveys administered to similar populations to the extent possible.

• Including in the instruments automatically enforced skip patterns, built-in range checks, internal item consistency checks, and required answer fields.

• Pretesting the baseline and follow-up surveys with individuals similar to the populations served by the type of programs being assessed for inclusion in the NextGen Project. The project team timed the interviews and used cognitive interviewing and respondent and interviewer debriefings to assess respondents’ understanding of the survey questions, identify improvements to the flow and structure of the instruments, and to ensure burden was low. The same question was not asked of more than 9 people. The surveys were updated based on the findings.

Data for descriptive studies. The discussion guides for program staff, employers, and partners, the surveys of program staff and leaders, and the in-depth participant interview guide were developed by content experts at Mathematica and OPRE. They were informed by reviewing instruments used in similar data collection efforts. These efforts included the Evaluation of SNAP Employment and Training Pilots (OMB #0584-0604) and the Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506). The guides were also informed by a review of corresponding instruments submitted to OMB by BEES (OMB #0970-0537).

The project team pretested the staff and leadership surveys with staff and leaders, with similar background and work experience to those implementing programs being considered for inclusion in the NextGen Project. The same question was not asked of more than 9 people. As a result of the pretests, the surveys were updated for clarity, flow, and to reduce burden. Used to record service receipt in some programs, RAPTER® is a secure, web-based system that program staff use to administer consent to participants, collect their identifying and contact information, conduct random assignment, and enter information on the services received or activities participated in by study participants. RAPTER was developed by content experts at Mathematica and are tailored to the NextGen programs and the services they provide.

Data for cost studies. The Excel-based cost workbook was developed by Mathematica staff, who selected the cost elements based on cost-collection tools developed for the Evaluation of SNAP Employment and Training Pilots (OMB #0584-0604) and the Workforce Investment Act (WIA) Adult and Dislocated Worker Programs Gold Standard Evaluation (OMB #1205-0504). The project team did not pretest the cost workbook, but instead provides training to program leaders (or their designees) on the cost study and how to complete the workbook, and a designated site liaison works with programs to help them understand the request and complete the workbook accurately.

Table B.2. Crosswalk between the data collection instruments and the study’s objectives  

	Study objectives
Instrument	Estimate the effectiveness of the program	Describe program operations and implementation	Estimate program costs
Instrument 1. Baseline survey – revised	All items	All items	n.a.
Instrument 2. Identifying and contact information – revised	All items (used to match to administrative outcome data and locate study participants for follow-up surveys)	n.a.	n.a.
Instrument 3: First follow-up survey – revised August 2024	Items in Sections A, B, and C	Items in Section D	n.a.
Instrument 4: Second follow-up survey – revised August 2024	Items in Sections A, B, and C	n.a.	n.a.
Instrument 5. Service receipt tracking – revised	All service receipt items	All service receipt items	All service receipt items
Instrument 6. Staff characteristics survey	n.a.	All questions	n.a.
Instrument 7. Program leadership survey	n.a.	All questions	n.a.
Instrument 8. Semi-structured program discussion guide	n.a.	All questions	n.a.
Instrument 9. Semi-structured employer discussion guide	n.a.	All questions	n.a.
Instrument 10. In-depth participant interview guide	n.a.	All questions	n.a.
Instrument 11. Cost workbook	n.a.	n.a.	All items

n.a. = not applicable

B4. Collection of Data and Quality Control

Impact studies. The project team is collecting the data for the impact evaluation via three surveys of study participants as well as administrative records.

• Baseline survey (Instrument 1) and Identifying and contact information (Instrument 2). In each program selected for the study, program staff identify individuals eligible to participate in the program and administer the consent form (Appendix A) to the applicant. If the applicant consents to participate in the study, staff enters the person’s identifying and contact information into the Random Assignment, Participant Tracking, Enrollment and Reporting system, or RAPTER® (Instrument 2). Program staff either administers the baseline survey (Instrument 1) to the program applicant or the applicant self-administers the survey via the web. After study participants have completed the baseline survey, program staff ask for some contact information and enter it into RAPTER®. After this information is collected, RAPTER® randomly assigns each study applicant to the treatment or control group and notify program staff of the assignment. The program staff notify the study participant of his or her assignment.

To ensure quality and consistency in this data collection, the project team:

• Provides a written procedures manual to program staff who will enroll study participants.

• Provides training to all program staff who will conduct study enrollment. The training covers administering consent, collecting identifying and contact information from participants, administering the baseline survey, notifying enrollees about the result of random assignment, and handing them off to the correct post-assignment protocol (treatment or control). Additional trainings are provided for new staff and if issues arise.

• Provides a designated liaison that the program staff can call to answer questions.

• Provides a hotline that the program staff can call if they cannot reach the designated liaison.

The project team monitors for quality and consistency in the data collection by the program staff. They regularly review the data entered into RAPTER® and the survey responses, looking for patterns of missing data and other data quality issues. They work with programs to resolve them quickly.

First follow-up survey (Instrument 3) and second follow-up survey (Instrument 4). The follow-up surveys are available to all study participants to either self-administer via the web or complete using computer-assisted telephone interviewing (CATI).

The project team has and will ensure quality and consistency in the data collected by the surveys by using tactics such as:

• For self-administered web surveys: use clear and straightforward language; include predominantly closed-ended questions; include check boxes, drop-down menus, and response categories; include soft checks to prevent outlier entries; and ensure the layout is compatible with multiple browsers, tablets, and smartphones.

• Recruit qualified interviewers to administer the survey by CATI.

• Train the interviewers in interviewing techniques as well as the intent of each question in the survey.

• Listen to about 10 percent of all CATI interviews to detect inaccurate presentation of information on the study; errors in reading questions; biased probes; inappropriate use of feedback in responding to questions; and any other unacceptable interviewer behavior.

• Examine data on the number of completed interviews, calls made, refusals, refusal conversions as well as time per call, and time per interview by interviewer. Supervisors will provide feedback to interviewers based on these data.

• Debrief with groups of interviewers shortly after the start of a data collection to discuss the respondents’ level of cooperation and ability to understand and answer the survey questions.

• Examine frequencies and cross-tabulations of data collected on a regular basis to pinpoint any unexpected aspects of instrumentation, particularly in skip logic, valid value ranges, the operation of edits and consistency checks, and the recording of data for legitimately skipped items and “don’t know” and refusal responses.

• Examine frequencies and cross-tabulations on data collected, by mode of collection, to look for evidence of mode bias or large differences in responses between self-administered web surveys and interviewer-administered telephone interviews.

Descriptive studies

Service receipt tracking (Instrument 5). Program staff can use RAPTER® to record information about all treatment group members’ participation in the program. To ensure quality and consistency in collection, the project team trains program staff involved with the evaluation on how to use RAPTER® to enter service receipt data with accuracy and in a timely fashion. The project team monitors for quality and consistency the staff entries into RAPTER® by reviewing what they are entering. Approximately twice per month, the team checks that program staff are entering data regularly and in as much detail as needed. If a program already collects data on service receipt through its own database, the project team uses the information the program already collects. Before beginning random assignment, the project team worked closely with the program staff to understand what information is entered into the program’s database, how staff enter information, and if any improvements were needed to ensure that the data will meet study needs and quality expectations. The project team requested a deidentified sample extract before the evaluation began to make sure the program’s information collection met expectations. The project team requests data extracts regularly to ensure that data is received for all enrolled study participants and developed reports to monitor the data entry frequency and quality.

Staff characteristics survey (Instrument 6) and program leadership survey (Instrument 7). The project team asks frontline staff and leaders to complete the appropriate survey via the web. To ensure quality and consistency in collection, the project team:

• Designed the surveys to use clear and straightforward language; include predominantly closed-ended questions; include check boxes, drop-down menus, and response categories; and include program checks to prevent outlier entries.

• Regularly examines the data collected through the surveys, checking for indicators of potential quality issues such as blank open-ended responses or high item nonresponse rates.

Semi-structured program discussion guide (Instrument 8) and semi-structured employer discussion guide (Instrument 9). The project team members are interviewing program staff, partner staff, and employers in person or by telephone. The project team recruits program staff and leaders for discussions using organizational charts and information on each employee’s role at the organization. The team recruits staff from partner organizations and employers, if applicable, to offer perspectives based on their involvement with the program and its participants.

To ensure quality and consistency in data collection, all interviewers are trained in the study research questions, the research approach, the topics to be covered in the data collection, and techniques for data collection (including protecting privacy, preparing post-visit summaries, and ensuring data security). The training also covers the content of the instruments to ensure full understanding of the questions and the collection of comparable, complete, and high quality data across the team. Refresher trainings and trainings for new interviewers are provided, as needed. If respondents consent to being recorded, the interviewer audiorecords discussions with program administrators, supervisors, staff; and key partner staff, including employers. Task leaders for the descriptive study periodically review completed interviews for quality and for missing information.

In-depth participant interview guide (Instrument 10). The project team is conducting in-person, one-on-one interviews with study participants. The project team is working with program staff to recruit participants.

To ensure quality and consistency in data collection, the interviewers are trained in the study research questions, the research approach, the topics to be covered in the data collection, and techniques for data collection. Topics include protecting privacy, using culturally appropriate and trauma-informed interviewing techniques, and ensuring data security. The training also covers the content of the interview protocol to ensure full understanding of the questions and the collection of comparable, complete, and high quality data across the team. The project team monitors for quality and consistency in the data collection. Any necessary refresher training is provided. If respondents consent to being recorded, the interviewer audiorecords discussions with participants.

Cost studies

Cost workbook (Instrument 11). The project team sends an Excel-based workbook for collecting data on program costs to program leaders (or a designee) for each program. The workbook records information on the expenditures associated with the program for a recent 12-month period.

The data collection approach includes two steps. First, the project team asks program leaders for their accounting records or financial reports and obtains as much information as possible from these records. Second, if additional information is needed after review of financial records, the project team asks the programs to complete the workbook in part or in full, depending on the information required.

To ensure quality and consistency in collection, the project team trains the program leaders (or their designees) on the cost study and how to complete the workbook. A designated site liaison works with programs to help them understand the request and complete the workbook accurately. The project team monitors for quality and consistency in the data collection by thoroughly reviewing each completed workbook, checking for completeness and internal consistency. Interviewers ask follow-up questions about the information entered into the workbook as needed.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The project team calculates conditional response rates as the number of completed surveys or other data collection instruments as a percentage of the number of people asked to complete the survey or instrument. If any study enrollees become ineligible for the study after they have been randomly assigned, the project team removes them from the denominator of the response rate calculation. This could happen if, for example, someone died during the course of the study.

Item response rates will be calculated as the number of people who complete an item as a percentage of the number of people who respond to any questions on the survey or other data collection instrument. The project team will exclude from the item response calculation any people who were not offered the question due to a survey skip pattern.

Impact Studies

• Baseline survey and identifying and contact information. Applicants eligible for study participation are only be enrolled in the study and randomly assigned if they consent to participating in the study, complete the baseline survey, and provide their identifying information as part of the intake process. Therefore, the project team anticipates that 100 percent of study participants will provide these data during the extension period, which matches the 100 percent response rate for data collected so far. The project team does not anticipate significant item nonresponse based on prior experience asking similar questions with similar populations and is consistent with the data collected so far. In a similar baseline survey that was used for the Evaluation of Employment Coaching for TANF and Related Populations (OMB #0970-0506), the item nonresponse was low; for example, the nonresponse to the employment status question was less than 3 percent.

• Follow-up surveys. The project team anticipates an 80 percent response rate on the follow-up surveys based on their experiences conducting follow-up surveys with similar populations and studies. The team will attempt to complete both first and second follow-up surveys with the entire sample. The project team has achieved similar response rates in other studies with hard-to-engage populations. For instance, in the evaluation of the Building Nebraska Families program (OMB #0970-0246), the team achieved an 87 percent response rate on the 18-month follow-up survey and an 83 percent response rate on the 30-month follow-up survey for TANF recipients who faced multiple challenges to employment. For the Personal Responsibility Education Program (PREP) evaluation (OMB #0970-0398), the project team achieved an 84 percent response rate on the 12-month follow-up survey and an 82 percent response rate for the 24-month follow-up survey for the Healthy Families San Angelo program, a home visitation program that seeks to engage a low-income population. For the Parents and Children Together follow-up surveys, the project team achieved an 88 percent response rate for the low-income mothers and fathers in the healthy marriage program study (OMB #0970-0403). The project team does not anticipate significant item nonresponse on the follow-up survey based on data collected so far and prior experience asking similar questions with similar populations, as described above.

To maximize response rates on the surveys, the following techniques are being used, which were also employed in the aforementioned efforts:

• Allow respondents to complete the survey in different ways. Respondents can complete the survey either online (using a computer, tablet, or smartphone) or by telephone.

• Send reminder notifications. In addition to notifying the study participant about the follow-up surveys during study intake, the project team uses a combination of letters, emails, texts, and telephone calls to encourage people to participate (Appendix G) throughout data collection efforts.

• Obtain accurate, up-to-date contact information. The project team collects detailed contact information during study intake and the follow-up surveys to aid in locating participants to complete the follow-up surveys. Before the start of the follow-up surveys, the project team also updates participant contact information through online database searches and may request updates from participants via text message or email.

• Use intensive locating methods, as needed. The project team initially notifies participants about the survey by mail and email and asks them to complete it via the web, though they will also be able to complete it via telephone. After four weeks, the project team attempts to contact the participants via telephone, so they can complete the survey via telephone. If the participants cannot be reached by telephone, the project team uses the contacts identified by the participant during the baseline data collection, for help locating them. If the participants still cannot be located, the project team conducts customized, individual searches for contact information using specialized databases. Finally, if study participants still cannot be located, trained field locators go in person to the study participant’s home and neighborhood. If they locate the study participant, the field locators lend him or her a smartphone to complete the survey.

• Offer tokens of appreciation. As discussed in greater detail in Part A, Section A9, the study’s strategy for tokens of appreciation is designed to retain respondents in the longitudinal data collection and decrease the differential response rate between the treatment and control groups, and therefore reduce nonresponse bias on impact estimates.

• Continuous quality improvement. The study collects data on each attempt to contact a respondent, including the mode, time, date, interviewer, and contact results. Examining these paradata helps identify the most effective calling times and interviewers. The project team also uses paradata to determine which methods of contact (letters, emails, texts, or telephone calls) prove the most successful in this study, so that they can adjust the frequency and type of contacts to achieve high response rates.

Descriptive and cost studies. The service receipt tracking, staff and program leader surveys, semi-structured discussions, in-depth interviews with program participants, and cost study workbooks are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

Cost studies

Based on similar research projects, such as the Evaluation of SNAP Employment and Training Pilots (OMB #0584-0604) and the WIA Adult and Dislocated Worker Programs Gold Standard Evaluation (OMB #1205-0504), the project team expects all programs to provide cost data which aligns with progress so far. To maximize responses, the project team is flexible with the data collection approach, asking programs to submit their existing accounting records and tailoring the sections of the cost workbook to fill in any gaps after reviewing those records. The project team also provides technical assistance to the programs as they complete the workbook.

NonResponse

Impact Studies

During survey fielding for the first and second follow-up surveys, the team actively monitors response rates, with an eye to any treatment–control differences. If such differences are observed, the project team intensifies the locating efforts for the group with the lower response rate to minimize differential nonresponse during active data collection.

Following the close of data collection, the project team will analyze nonresponse on the follow-up surveys to assess whether the survey respondents are representative of the full study sample. Using the data on participants’ characteristics collected at baseline, the project team will conduct statistical tests (chi-square and t-tests) to gauge whether the treatment group members who participated in data collection are representative of all the treatment group members, whether the control group members who participated in data collection are representative of all the control group members, and whether there are systematic differences in the treatment and control group members who responded to the survey.

The project team will use two approaches to correct for potential nonresponse bias in the estimation of program impacts. First, the regression models described in Section B7 will adjust for observed differences between the characteristics of treatment and control group respondents. Second, because this regression procedure will not correct for differences between respondents and nonrespondents in each research group, the project team will construct sample weights so that the weighted baseline characteristics of respondents in the treatment and control group in each program are similar to those of the full sample (respondents and nonrespondents). These weights will be constructed using data from the baseline surveys.

To reduce any bias resulting from item nonresponse, the project team will impute values for missing data. Imputation is particularly useful in cases in which data might be systematically missing related to an observable characteristic. For example, if a study participant was not employed, the team knows that his or her wage and salary earnings will be zero. However, many more data items are required to construct a measure of earnings for employed individuals and, thus, it is more likely that employed individuals will have missing earnings. This suggests that, without imputation, estimates of earnings might be biased downward. The imputation approaches used will include logical imputation, predictive mean matching, and hot-deck imputation. The approach used will be determined by the type of data that are missing.

Descriptive and Cost Studies

The data will not be used to generate population estimates, either for internal use or dissemination.

B6. Production of Estimates and Projections

The estimates from this project will be released publicly following ACF review.

Impact studies. The impact studies will estimate the effectiveness of each program in the study in improving outcomes of study participants. Any observed differences in outcomes between the treatment and control group members can be attributed to the effectiveness of the program; in statistical terms, the differences are internally valid estimates of the mean impacts of the program, as delivered, on the corresponding outcomes for similar populations in the same environment.

The project team will use the constructed sample weights described in Section B5 in the impact analysis so that the weighted baseline characteristics of respondents in the treatment and control group in each program are similar to those of the full sample (respondents and nonrespondents). The project team will also address missing responses as described in Section B5.

The baseline data will be used to describe the study participants in each program. The project team will use chi-squared tests of differences in means over all characteristics to assess whether random assignment successfully generated treatment and control groups with similar baseline characteristics, and that the treatment and control group respondents to the follow-up surveys are similar. The project team will also report t-tests of differences for individual characteristics.

Impacts will be estimated for each program. The project team will use regression estimators to control for residual differences between the treatment and control groups and to construct more efficient estimators than the simple difference-in-means estimators (as described in B7).

Descriptive studies. The data will not be used to generate population estimates, either for internal use or dissemination.

Cost studies. The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Survey data. The web survey and the telephone interview software use real-time logic rules, enforce skip patterns, and provide soft and hard checks. Soft and hard checks are displayed for interviewers or respondents if the provided information conflicts with earlier responses or is out of range for expected values. Hard checks require resolution before continuing; soft checks can be suppressed. All CATI interviewers are subject to real-time or recorded monitoring to ensure they are correctly interpreting and entering respondent responses. Following data collection, the project team will conduct comprehensive data reviews and quality assurance reviews to ensure skip patterns are enforced and data are complete and within expected ranges.

During data processing and coding, the project team will conduct quality assurance reviews to ensure consistency and minimize any data processing errors. Specifically, coders will participate in a comprehensive training session, and the project team will monitor their work, perform quality control checks, and conduct quality assurance reviews of all weighting and imputation procedures. Any outliers, skip logic errors, or other recodes of survey data will be recorded in both internal programs and data editing spreadsheets.

RAPTER®. As with the survey software, RAPTER® uses real-time logic rules and validity checks to prevent entry errors. The project team extensively tested all functionality. The project team trains all program staff in the use of the system, provides them with a written procedures manual, and routinely examines the data staff enter into RAPTER® to ensure quality.

Data Analysis

Impact studies. The impact analysis will consist of comparisons of means and distributions using the randomly assigned treatment and comparison group. This will include unadjusted and regression-adjusted means for outcomes including employment, earnings, and benefit receipt. Differences in means or proportions of follow-up outcomes between the treatment and control group will provide unbiased estimates of the impacts of the program. Estimates that are more precise will be obtained using regression models to control for random differences in the baseline characteristics of treatment and control group members. In their simplest forms, these models can be expressed by the following equation: Y_i = α +βX_i + δT_i + ε_i, where Y_i is an outcome (such as earnings) for person i; α is a constant; X_i is a (column) vector of baseline characteristics (such as gender, age, race/ethnicity); β is a vector of coefficient parameters for the extent to which baseline characteristics are predictive of the outcome; T_i is an indicator for whether person i received treatment; δ represents the impact of the program; and ε_i is an error term. These models will be estimated separately for each program.  

 

If the sample is large enough, the project team will conduct a subgroup analysis to examine who benefits most from the program. Subgroup effects will be estimated using the following equation:  

Y_i = α + βX_i + δ₁T_i + δ₂G_i + δ₃T_iG_i + ε_i, where G_i is an indicator for whether person i is part of a subgroup; δ₂ represents the relationship between subgroup status and the outcome; and δ₃ represents the additional effect of treatment for those in the subgroup. The project team will consider subgroups that are appropriate for the program’s target population, such as those defined by disability status, work readiness, employment challenges, or history of TANF receipt. The impact study will be pre-registered on Open Science Framework.

 

Descriptive studies. To analyze the large amount of interview data collected from multiple sources efficiently and accurately, the project team will develop a coding scheme that maps to the implementation framework, research questions, and programs’ theories of change. After coding the data, the team will look for common themes across data elements and respondents and examine the extent to which the programs adhered to the fidelity measures defined during the program selection phase. The analysis will include an assessment of conditions needed to replicate and sustain the program. The descriptive study will be pre-registered on Open Science Framework. 

 

Cost studies. The project team will use information from the cost studies to compute an average cost of the program per participant-month. From this, and using information on the average number of months the program participants in the study were engaged in the program, the average cost of the program per participant will be estimated. The team will use data collected from the surveys and administrative records to estimate the average benefit of the program per participant and compare the benefits and costs. Doing so involves considering the possible benefits and costs of the program from the participants’, government, and rest of society’s perspectives. The cost study will be pre-registered on Open Science Framework.  

Data Use

The project team will publish findings from the project throughout the study in technical reports and briefs. The project team anticipates that reporting on the descriptive and cost studies will continue through 2024; reporting on the intermediate impact findings will begin in 2026 and continue through 2027; and reporting on the final impact findings will begin in 2027 and continue through 2028. In addition to presenting findings, reports and briefs will document the methodologies used to collect, process, and analyze the project’s data across the impact, descriptive, and cost studies; this will assist readers in assessing study quality and interpreting the findings. Study limitations and information about the generalizability of the results will be included when presenting findings. 

 

In addition, the project will prepare final data files and documentation to be available publicly. The provided documentation will improve the understanding of how to properly interpret, analyze, and evaluate the information resulting from the data collection. The project team anticipates that data archives (restricted or public use) would become available starting in 2029 and hosted on a data archive platform such as the Inter-university Consortium for Political and Social Research (ICPSR).  

B8. Contact Persons

Contact information for people who can answer questions about the statistical aspects of the survey: 

• Marie Lawrence: Catherinemarie.Lawrence@acf.hhs.gov

• Annalisa Mastri: AMastri@mathematica-mpr.com

 

Mathematica developed the plans for this data collection. Leaders of the project team from OPRE, Mathematica, and Tree House Economics who designed and/or will collect and analyze the data are as follows: 

• Gabrielle Newell, former social science research analyst, ACF 

• Sarita Barton, social science research analyst, ACF 

• Marie Lawrence, social science research analyst, ACF 

• Megan Reid, social science research analyst, ACF 

• Sheena McConnell, former senior vice president, Mathematica 

• Michelle Derr, senior researcher, The Adjacent Possible 

• David Stapleton, partner, Tree House Economics 

• Annalisa Mastri, senior researcher, Mathematica 

• Jody Schimmel-Hyde, senior researcher, Mathematica 

• Kristen Joyce, senior researcher, Mathematica 

• Ryan Callahan, survey researcher, Mathematica

Attachments

Instruments

Instrument 1. Baseline survey – revised (approved June 2022) 

Instrument 2. Identifying and contact information – revised 

Instrument 3. First follow-up survey – revised August 2024

Instrument 4. Second follow-up survey – revised August 2024

Instrument 5. Service receipt tracking – revised 

Instrument 6. Staff characteristics survey – revised  

Instrument 7. Program leadership survey – revised 

Instrument 8. Semi-structured program discussion guide – revised 

Instrument 9. Semi-structured employer discussion guide – revised 

Instrument 10. In-depth participant interview guide – revised 

Instrument 11. Cost workbook  

Appendices 

Appendix A. Informed consent form – revised

Appendix A.1. Bridges consent forms

Appendix B. Question-by-question justification for baseline survey – revised (approved June 2022)

Appendix C. Question-by-question justification for identifying and contact information – revised

Appendix D. Question-by-question justification for follow-up surveys – revised August 2024

Appendix G. Follow-up survey reminders and notifications – revised

Appendix G.1. NextGen Project recruitment materials

Appendix P. Federal Register Notice

Appendix P.1. Federal Register Notice – 30-day request, published January 2021

Appendix P.2. Federal Register Notice – 30-day request, published March 2022

Appendix P.3. Federal Register Notice – 60-day request, published December 2022

Appendix Q. Summary of requested changes (submitted February 2021)

Appendix Q.1. Summary of requested changes – revised (approved June 2022)

Appendix R. Certificate of Confidentiality

Supporting Statement B References 

U.S. Department of Education. What Works Clearinghouse Procedures and Standards Handbook, Version 5.0. Washington, DC: Institute for Education Sciences, 2022. Available at https://ies.ed.gov/ncee/WWC/Docs/referenceresources/Final_WWC-HandbookVer5_0-0-508.pdf. Accessed June 13, 2024.

15