NonSubstantive Change Memo - NextGen_Phase 2_Dec 2020

MEMORANDUM

To: Jordan Cohen, Office of Information and Regulatory Affairs, Office of Management and Budget (OMB)

From: Hilary Bruck, Gabrielle Newell, and Marie Lawrence

Office of Planning, Research, and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: 12/8/2020

Subject: Non-substantive Change Request – Next Generation of Enhanced Employment Strategies Project [Impact, Descriptive, and Cost Studies] (OMB # 0970-0545)

Background

Type of Request: This memo requests approval of non-substantive changes to data collection instruments for the Next Generation of Enhanced Employment Strategies (NextGen) Project. Specifically, we seek approval to use a subset of Phase 2 instruments with programs selected for inclusion in the project, with non-substantive changes made to all but one of those instruments. These requested non-substantive changes do not require any changes to the burden estimates.

Study Features Salient to Request: The NextGen Project is intended to build on the findings and lessons learned from past and ongoing evaluations by identifying and rigorously evaluating the “next generation” of employment strategies for highly vulnerable populations with complex barriers to obtaining and retaining employment. The project includes impact, descriptive, and cost studies of up to 10 interventions.

In April 2020, OMB approved the new information collection request (ICR) for Phase 1 data collection instruments for the NextGen Project. The first phase submission included instruments that will be uniform across programs selected for inclusion in the NextGen Project and participating in impact evaluations. These include the informed consent form (Appendix A), the baseline survey (Instrument 1), and identifying and contact information (Instrument 2). In the first ICR, we included drafts of Phase 2 data collection instruments and associated burden estimates for initial review, information purposes, and public comment. We indicated that some of the Phase 2 instruments may require revisions to tailor to each program selected for the evaluation and that, as programs are selected, we will submit tailored Phase 2 instruments—as needed—as either a non-substantive change request or a revision with abbreviated public comment time, dependent on the level of changes and guidance provided by OMB.

We are seeking official approval to use a subset of the Phase 2 instruments, with non-substantive changes to all but one of these instruments, across all selected NextGen Project sites. Rather than tailoring instruments to each selected site, as proposed in the first ICR, we intend to use the same Phase 2 instruments across all sites, with skip patterns and instructions to interviewers indicating whether certain items only apply to certain types of respondents or programs. The non-substantive changes proposed to the Phase 2 instruments include revisions to capture how programs responded to COVID-19 and the resulting recession. Below are additional details regarding these requests.

Time Sensitivity: We are planning to begin descriptive and cost data collection in some selected sites in late 2020, to capture information on the effects of COVID-19.

REQUEST DETAILS

We are currently seeking official approval to use a subset of the Phase 2 instruments across all selected NextGen Project sites, with non-substantive changes to all but one of these instruments (described further below). The original ICR submission included burden estimates for each Phase 2 instrument. The burden for the instruments included in this request falls within those original estimates; the proposed non-substantive changes do not change the burden estimates. We will seek approval to administer the remainder of the Phase 2 instruments in a future ICR. The following instruments are part of this request:

1. Instrument 6. Staff characteristics survey - revised

2. Instrument 7. Program leadership survey - revised

3. Instrument 8. Semi-structured program discussion guide - revised

4. Instrument 10. In-depth participant interview guide – revised

5. Instrument 11. Cost workbook

Details about requested changes to Instruments 6, 7, 8, and 10 follow. There are no changes proposed to the Phase 2 Excel-based cost workbook (Instrument 11) to collect cost study data for all NextGen Project sites.

Non-Substantive Change Details

Surveys of Program Staff and Leaders (Instrument 6. Staff characteristics survey – revised; Instrument 7. Program leadership survey – revised)

These changes are necessary to better reflect the activities program staff and leaders might conduct; streamline items where able and appropriate; and capture information about program changes related to COVID-19.

• We added response options to B1 and B2 of Instrument 6 to capture additional staff activities.

• We changed the response options for B4 and B5 of Instrument 6 and B3 of Instrument 7 to make them easier for respondents to complete and to capture more precise information.

• We added a response option to B8 of Instrument 7 to capture additional program leader activities.

• We deleted C1 and C2 of Instrument 6 and C1 of Instrument 7 because we are requesting similar information in Instrument 8. Semi-structured program discussion guide. We revised the Section C. introduction text to reflect removal of these items.

• We added the word “now” to C3 and renumbered it to C1 of Instrument 6. We made the same change to C2 of Instrument C7 and renumbered it to C1.

• We added sub-items to D1 on Instrument 6 and C2 on Instrument 7 to capture the effects of COVID-19 on program operations and clarified that respondents should consider all the statements under this question as of “right now.” We added logic to skip these subitems if the COVID-19 pandemic is no longer applicable at the time of data collection.

Program Discussion Guide (Instrument 8. Semi-structured program discussion guide – revised)

These changes are necessary to understand programs’ responses to contextual factors including the pandemic and recession and the social justice and equity movement.

• We added contact information for the study team to the introductory section of the discussion guide, as recommended by our Institutional Review Board.

• We added interviewer notes and probes to capture changes in programs due to COVID-19.

• We made slight changes to E4 to E7 to ask about refining the intervention model in addition to asking about its development.

• We revised wording and added probes at question I2 to make it easier to answer for participants.

• We added question I3 to capture race and equity issues around program administration.

We also made revisions to the mode for this instrument in Supporting Statement A, to indicate that the discussions may be conducted via telephone or video depending on any COVID-related restrictions that prevent in-person discussions.

Participant Interview Guide (Instrument 10. In-depth participant interview guide – revised)

These changes are necessary to understand programs’ responses to the pandemic and recession.

• We made minor revisions to the introductory text to mention COVID-19. In addition, we added contact information for the study team to this section, as recommended by our Institutional Review Board.

• We made minor revisions to item A1 and added A2 to ask whether respondents are in school. This will help interviewers determine which subsequent questions are appropriate to ask.

• We made minor revisions to item C8c to remove reference to receiving a stipend.

• We added C12 to ask how COVID-19 affected participants’ engagement in the program.

• At the end of section D, we added interviewer notes to confirm respondents’ email or mailing address for the purposes of delivering the interview token of appreciation (a gift card), in cases where the interview is conducted virtually.

We also made revisions to the mode for this instrument in Supporting Statement A, to indicate that the interviews may be conducted via telephone or video depending on any COVID-related restrictions that prevent in-person discussions.