BEES_Supporting Statement A_Sept 2022_3_Clean

Building Evidence on Employment Strategies

OMB Information Collection Request

0970-0537

Supporting Statement

Part A

September 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Megan Reid

• This is a request for an extension to complete data collection for Building Evidence on Employment Strategies (BEES) study. The request is for an additional three years of approval.

• This information collection was originally approved on November 9, 2019 (OMB# 0970-0537).

• BEES is conducting multiple studies of innovative interventions aimed at improving employment outcomes, advances in the labor market, and economic security for individuals and families with low incomes.

• BEES includes impact and implementation studies. Impact studies will include RCTs, when possible. Some programs in earlier stages of development will only involve implementation studies.

• BEES will address gaps in the evidence base about employment services for populations in a number of domains, including individuals struggling with disability or mental health issues, involved in the criminal justice system, or those facing substance or opioid use disorders. Information gathered from BEES is broadly intended to add to the current scientific research in this service area. Policymakers will gain a better understanding of the current innovative efforts to provide employment services to low-income populations and the efficacy of these interventions. This could influence their decision-making about interventions and policies to support. Practitioners will find ideas for their own program improvement and better understand the landscape of employment services for these populations.

Overview

A1. Necessity for the Data Collection

The Office of Planning, Research, and Evaluation within the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval to continue to collect data from sites and participants for the Building Evidence on Employment Strategies (BEES) study. The overall purpose of BEES is to conduct rigorous evaluations of innovative interventions aimed at improving employment outcomes, advances in the labor market, and economic security for low-income individuals and families. Populations of particular interest for BEES include individuals facing substance and opioid use disorders, disability or mental health challenges, and their families; criminal justice-involved populations; or others at increased risk of enrolling in TANF or SSI/SSDI. While the target population and domains may vary, all interventions will include an employment services component. BEES includes impact and/or implementation studies of 20 different programs.

Study Background

There is both a great deal known, and a great deal more to learn, about interventions designed to increase employment and economic security among low-income Americans. Rigorous studies have demonstrated that many types of interventions can improve labor market outcomes for disadvantaged groups. Job search and subsidized employment can increase employment and earnings in the short-term, while earnings supplements can increase both employment and income, at least while supplements remain in place. Career pathways models can produce lasting earnings gains for those who meet program entrance criteria. But there remain many open questions on how employment interventions work for specific populations struggling with employment.

BEES is building ACF’s program of research in employment training and support strategies, providing new perspectives on employment for low-income individuals in a few specific domains. Below is an overview of the scientific research available on employment programs for low-income individuals within these domains: substance and opioid use disorder, criminal justice, and mental health and disability. These domains represent populations who may enroll in TANF or SSI/SSDI. The summaries below indicate where the gaps in research lie about employment services for these examples of populations of interest to ACF, and how the BEES research questions may address these gaps.

1. Substance and Opioid Use Disorder. Introducing vocational services into drug treatment programs has been a widespread policy recommendation for decades. However, most substance use disorder treatment and recovery programs have excluded employment-focused services since most insurers do not reimburse for non-medical services, leading to a lack of research in this area. In addition, studies of employment training or counseling in drug treatment have rarely had adequate comparison or control groups and have therefore not provided reliable results. For the most part, studies have compared voluntary participants in employment activities with people who did not volunteer, introducing selection biases into the comparisons. The range of other predictors of ultimate success in both achieving drug reduction and employment have not been fully accounted for—age, education, prior work history, arrest background, housing status, ancillary problems such as co-occurring mental illness, and readiness or motivation. Promising approaches integrate treatment, employment services, and intensive supports in various combinations, but the evidence base on these models is thin, suggesting that there is a great deal to learn on this subject in BEES.

2. Criminal Justice. Prisoners returning to their communities after incarceration often face obstacles to regular employment, including low levels of education and lack of recent work experience (Pager, 2003; Holzer, Raphael, and Stoll, 2004; Uggen, Wakefield, and Western, 2005). In addition, some employers avoid hiring job applicants with criminal records. Ex-prisoners have low earnings upon reentry; in the twelve months after release, only 55 percent reported any earnings at all, and only 20 percent earned more than $15,000 during the year (Looney and Turner, 2018). The model of transitional jobs for those recently released from prison has had mixed results, with a lack of sustained employment and only one program showing reductions in recidivism rates (Jacobs Valentine, 2012; Redcross et al., 2012; Butler et al., 2012). Sustaining employment for this population continues to be a topic of interest, and one that could be addressed by BEES.

3. Mental Health and Disability. RCTs have found that improved access to and quality of mental health care reduces disability and improves employment outcomes (Wang et al., 2007). Integrating employment interventions with mental health treatment yields additional benefit (Lerner et al., 2015). Among people with mental health conditions, supported employment programs, like the Individual Placement and Support model (IPS), have positive effects on mental health symptoms and impairment (Mueser, Drake, and Bond, 2016). Implementing IPS in settings outside of the community mental health system for people with diverse disabilities is proving to be an especially fruitful area of study. Given these bi-directional relationships, it is important to consider going beyond the simple dichotomy between “treatment first” or “job first”. Instead, as in the substance abuse domain, organized interventions to improve mental health care and promote competitive employment may have synergistic benefits. Overall, there is strong evidence for the efficacy of specific treatment approaches for depression, anxiety and other conditions that can act as barriers to steady employment. The challenge – and a promising area of inquiry for BEES – is to learn how best to promote steady participation in treatment, and how to integrate treatment with employment services.

Building the evidence base in these and other areas furthers ACF’s broad goal of increasing the number of employment-focused interventions supported by rigorous research. The broad intent of BEES is to increase the number of evidence-supported comprehensive interventions and to support the field in moving toward rigorous evaluations. While BEES is prioritizing interventions that have been fully implemented previously, undergone formative testing, or are candidates for an impact evaluation, other programs or interventions are also being considered. The overarching goal is to identify interventions that can move toward more rigorous evaluation. In some cases, this may mean an evaluation focused solely on implementation research.

BEES currently has studies of 20 different programs underway, each focusing on improving employment outcomes for low-income individuals. Program services being evaluated include employment and training services such as job search assistance, job readiness services, vocational education, and coaching; employment barrier removal services such as those targeting substance use disorder and mental health; and other services aimed at promoting and supporting employment and economic security, consistent with the purposes of Temporary Assistance for Needy Families (TANF) and HHS priorities. Randomized control trial designs are being used where an impact study is appropriate and possible. Any impact study will be accompanied by a corresponding implementation study, which will describe the programs and contextualize the impact results. For programs of interest that are not appropriate for an impact study, BEES is conducting an implementation-only study. Collectively, with continued data collection and analysis, these studies help ACF fulfill the goal of adding to the evidence base.

The study team used the generic clearance, Formative Data Collections for ACF Research (OMB #0970-0356), to learn more about programs of interest to inform the design of this larger study. Based on this information, 20 programs are being studied through BEES. See Section A2 for additional information.

Coordination with Current OPRE Project and Social Security Administration

BEES is actively coordinating with another project sponsored by OPRE, the Next Generation of Enhanced Employment Services (NextGen) Project. The NextGen Project includes approximately five rigorous evaluations of employment-focused programs; these do not overlap with programs selected for the BEES project. BEES and the NextGen Project have a common goal to foster stronger understanding of the types of interventions that can improve labor market outcomes for low-income individuals; however, the projects also maintain separate domains of focus. BEES is especially interested in evaluating interventions for individuals struggling with opioid dependency, abuse of other substances, and/or mental health issues, while the NextGen Project is especially focused on evaluating interventions that are market-oriented and/or employer-driven. OPRE is intentionally and strategically coordinating these projects to prevent duplication of effort; fully capitalize on the opportunity the projects afford for large-scale, rigorous evaluation; and advance the knowledge base regarding effective employment strategies for low-income, vulnerable populations.

In addition, both projects are involved in a joint effort with the Social Security Administration (SSA). SSA is providing demonstration program funds to ACF to support the addition of a disability focus in both projects. SSA funds may be used for both evaluation-related expenses and program development in sites selected for evaluation under either project.

To ensure BEES and the NextGen Project produce comparable results, and to meet SSA’s priorities across both projects, the projects’ information collection requests to OMB include several common instruments and/or questions within instruments. We provide a table in Section A.2 that outlines commonality between the BEES and NextGen instruments.

Legal or Administrative Requirements that Necessitate the Collection

Although this specific data collection is not mandated, the Consolidated Appropriations Act of 2017 requires ACF to conduct research to understand the impact of TANF and other programs on employment, self-sufficiency, and other outcomes. The BEES project meets this requirement.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The purpose of BEES is to build the evidence base of innovative approaches designed to boost employment and earnings for low-income individuals receiving or at risk of receiving TANF. BEES includes two broad types of sites, each providing employment services for low-income individuals. Behavioral health sites include those providing substance or opioid use disorder treatment or recovery services, as well as those addressing mental health issues or working with people with disabilities. Non-behavioral health sites are prioritizing innovative approaches to improving employment outcomes for low-income individuals, such as whole family approach interventions.

BEES includes impact and implementation studies. While BEES is prioritizing sites ready for randomized controlled trials (RCTs), the research team identified some sites that are in earlier stages of development and not suitable for rigorous evaluation (discussed below). For these types of sites, the team is conducting implementation-only studies. For sites participating in an impact evaluation, administrative data is also being used to estimate program impacts on key outcomes, such as employment and earnings.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.  

Study Design

The BEES design has three primary components:

1. Site Selection and Recruitment. The recruitment effort of programs into BEES has now been completed.

2. Impact study. Programs enrolled in BEES use rigorous evaluation designs, with an emphasis on randomized control trials (RCT) where possible (see the Research Design and Statistical Power subsection of Supporting Statement B for additional details). Outcomes for program and control group members are compared at follow-up using the instruments described in this package. The internal validity of our estimates of these program impacts will be particularly strong using an RCT design.

BEES is testing a diverse set of interventions designed to improve employment outcomes for low-income individuals, with a focus on specific target populations that have characteristics that may affect their ability to work. The findings are not designed to be generalized to a broader population but may be used to suggest interventions that could be tested further (e.g., with multiple sites or other populations) to support broader generalization. 

As an exploratory study, one limitation is that tests that take place in only one location might not represent the effects in other locations or with other target populations. A second limitation is that some of the interventions may include multiple components and it will be difficult to determine which individual components are effective or whether combinations of components interact with one another to produce larger or smaller effects. These limitations will be clearly stated in written discussions of study findings.

3. Implementation study. All programs will participate in an implementation study. For those participating in an impact study, the corresponding implementation study will document the theoretical underpinnings of the intervention studied in the impact analysis, explore the operational structures that support program implementation, document implementation practices and staff and participant experiences, assess whether staff practices reflect each intervention’s core principles and components, and document control group services. Each intervention’s theory of change (often called a logic model) guides our choices of topics to pursue in understanding how a program operates; whether the program is being implemented with fidelity to the logic model; and whether, how, and why a program does or does not achieve its expected outcomes. Crucially, the analysis will also examine the program-control treatment contrast. Finally, the analysis will contribute to practitioner promising practices or lessons for future replication or adaptation.

As previously noted, other programs will only participate in an implementation study. Programs that are not suitable for an impact study at this time due to their maturity or the small size of their service population could still provide valuable information to inform the field through an implementation study. The study focuses on describing the program’s context within its community, its goals and structure, any partnerships it may have, and how it enrolls participants and delivers services. It does not look at participant outcomes as part of the study. This component has the primary goal of adding to our knowledge about the landscape of employment services for low-income individuals, particularly in cases where rigorous evaluation is not currently feasible due to size or other concerns.

Phased Approach to Data Collection Approval

Using the generic clearance Formative Data Collections for ACF Research (OMB control number 0970-0356), the team identified some programs that, following OMB approval, were formally recruited for the BEES evaluation. The below table briefly summarizes programs that are participating in the BEES project.

Type of Program (Number of Programs)	Intervention	Capacity for Evaluation	Proposed Study
Substance Use Disorder (SUD) Treatment and Employment Programs (6 programs)	Combines employment services with treatment and recovery services	This approach is relatively new to the field; most programs are small. Because of their size and maturity, these programs are not appropriate for a rigorous RCT, but the field can benefit from implementation studies.	Implementation-Only
Substance Use Disorder (SUD) Treatment and Employment Programs (4 programs)	Combines individualized employment services with treatment and recovery services	These programs are larger and more mature and thus participating in a rigorous evaluation.	Impact and Implementation
Whole Family Approach Program (1 program)	Combines more holistic coaching with employment services to improve two-generation outcomes, including employment and family well-being	This intervention provides the opportunity to build on a recently tested two-generation model that has positive early results.	Impact and Implementation
Non-Custodial Parent Program (2 program)	Combines employment services, parenting workshops and intensive case management for non-custodial parents interacting with child support agencies for the first time.	This intervention has been previously piloted and builds on evidence-based approaches for the model components.	Implementation-Only
Health & Disability (3 programs)	Pairs employment services with existing health-related treatment services	These intervention layer evidence-based approaches on other treatment services within well-established programs.	Impact and Implementation
Employment Services for a TANF/SNAP population (3 programs)	Offers employment services in coordination with other existing services to people receiving TANF and select SNAP benefits.	These well-established programs are offering evidence-based employment services to a new target population.	Impact and Implementation
Criminal Justice Involvement (1 program)	Addresses key barriers to employment associated with justice system involvement fines and fees	This issue is of great interest to the field, but the smaller program is not ready for a rigorous evaluation.	Implementation-Only

Data Collections for Impact and Implementation Studies

This extension request allows for the continued data collection to complete the impact and implementation studies of the programs in the table above.

This includes the use of the following data collection instruments listed below. (See the Universe of Data Collection Efforts subsections for additional details).

1. Impact Studies

   1. Baseline Information Form for Participants

• Baseline Information Form for Participants (CCC) (Attachment D-1)

• Baseline Information Form for Participants (2Gen-Chicago) (Attachment D-2)

• Baseline Information Form for Participants (IPS FQHC) (Attachment D-3)

• Baseline Information Form for Participants (IPS SUD) (Attachment D-4)

• Baseline Information Form for Participants (IPS TANF-SNAP) (Attachment D-5)

2. Contact Update Request Form (Attachment E)

3. 12- to 24-Month Follow-Up Participant Survey (Attachment K)

   • 12- to 24-Month Follow-Up Participant Survey (IPS SUD/FQHC) (Attachment K-1)

   • 12- to 24-Month Follow-Up Participant Survey (CCC) (Attachment K-2)

2. Implementation Studies

   1. Program Managers, Staff, and Partners Interview Guide – SUD Programs (Attachment F)

   2. Program Managers, Staff, and Partners Interview Guide – Whole Family Approach Programs (Attachment G)

   3. Program Managers, Staff, and Partners Interview Guide (Attachment L)

   4. In-Depth Case Study of Staff-Participant Perspectives

      • Participant Case Study Interview Guide (Attachment M)

      • Program Staff Case Study Interview Guide (Attachment N)

   5. Program Staff Survey (Attachment O)

This request also included the following supplementary materials:

1. Informed Consent Form for Participants (Attachment H)

2. Welcome Letter (Attachment I)

3. 12- to 24- Month Survey Advance Letters (Attachment P)

4. 12- to 24- Month Survey Email Reminders (Attachment Q)

5. 12- to 2- Month Survey Flyer (Attachment R)

Data Collection Timeline

We expect data collection to take place over an additional three-year period, following OMB approval. Due to COVID-19, the study timeline has been extended. For the implementation-only studies identified, information collection was delayed due to COVID-19 but began in June 2020. For other participating sites identified and recruited using the OMB-approved instruments, baseline data collection began in March 2020. New sites are still launching study activities in 2022, thus extending data collection farther out than originally intended. Due to the changes in timeline to date, the timing of the follow up surveys have been adjusted since original approval. In 2020, ACF adjusted the plan from fielding the follow up survey 12 months after enrollment to a 12-18 month follow up effort through a nonsubstantive change request. At this time, we have updated the plan to extend the fielding period slightly to field the surveys 12- to 24-months after participant enrollment. There are no changes to the actual survey or related materials.

As sites begin enrolling participants in the study, baseline data collection occurs and then the participant surveys are fielded 12- to 24- months after each participant’s enrollment. Pending OMB approval of this extension, we expect this data collection to take place into 2025. Administrative records will supplement the baseline data and follow up surveys and will be collected for a period of approximately three years. Implementation study activities for sites with an impact study – including site visits, staff survey (where applicable), and case studies –begin approximately 6-9 months after study enrollment begins). Pending OMB approval of this extension, we expect these activities will take place through 2023. As needed during the COVID-19 pandemic, we are using virtual approaches such as video conferencing, more frequent phone calls, and remote data collection.

Research Questions

Key research questions for BEES are outlined in the table below. First, we identify a set of specific research questions for the implementation-only studies of SUD treatment/employment programs and whole-family approach programs. Then, we identify broad research questions for all other sites.

These questions allow the researchers to both describe the landscape of employment interventions for populations of interest in BEES and measure the effectiveness of employment programs selected for rigorous evaluation.

Research Question	Impact Study	Implementation Study	Site Recruitment Phase
What are the context and goals for establishing programs?		X
How are programs structured and operated including organizational partnerships, target group and recruitment strategies, employment service provision, and other support services?		X
What are the lessons from implementing the programs, including successes and challenges?		X
Which programs and/or promising practices warrant further study to determine effectiveness?		X
What is the impact of the specific interventions, both on measures of success in the first year and on longer-term measures of employment, well-being1, and economic security measured through survey and administrative data sources?	X
What can be learned about the impact of these interventions for important subgroups within the target populations (such as race, education-level, or socioeconomic status)?	X
Based on prior and emerging research, and current practice, what types of programs and approaches are most promising for the target population of each intervention?	X	X	X
Which specific interventions in each domain are ready to move to the next level of evidence?		X	X
What lessons on program implementation and cost can shed light on the impact results and help facilitate the expansion or replication of successful interventions?		X
What lessons can be drawn across the study about the design and implementation of successful interventions?	X	X

Universe of Data Collection Efforts

As previously discussed, we are requesting an extension to continue data collection for the impact and implementation studies. The site selection and recruitment activities, previously approved, have now been completed. An overview of the materials is provided below.

Site Selection and Recruitment

Site selection instruments were approved under the generic clearance Formative Data Collections for ACF Research (OMB control number 0970-0356) to gather information about potential sites (initially on January 19, 2018, with revised clearance approval on August 14, 2018). We used the discussion guides approved under this ICR to formally recruit programs and for further program identification efforts.

The BEES evaluation team recruited programs with the previously approved materials. This recruitment was completed using a comprehensive stakeholder outreach effort. Site recruitment staff, working in teams of two or three, met in-person and by phone to discuss the evaluation with select informants, state and local administrators, and program staff. These visits and telephone calls were used to collect information to determine which promising interventions are a good fit for the study.

Instruments used²:

1. Discussion Guide for National Policy Experts and Researchers. We consulted with regional and national organizations that work closely with relevant state and local programs in the targeted domains to identify promising interventions and programs. This high-level outreach also included other experts from the fields of interest. Consultations were completed primarily via telephone.

2. Discussion Guide for State and Local Administrators. As promising interventions operated by local and state administrators were identified through discussions with national policy experts and researchers, as described above, further outreach with state and local administrators helped identify promising sites. The study team introduced administrators to the study and the semi-structured discussion guide was used to learn about potential programs and assess the feasibility of launching evaluation activities within each local context overseen by the administrators. The study team also assessed state and local administrators’ willingness to help recruit the program staff involved in operating services at potential sites.

3. Discussion Guide for Program Staff at Potential Sites. Through the consultations listed above or the study team’s existing connections, a series of telephone conversations and 1-2 day in-person, when possible due to Covid-19, recruitment visits took place with the program staff directly operating innovative program models. A discussion guide was used to organize discussions. The semi-structured guide was used to learn about program administrative structures, programmatic experiences, community service contexts, and the feasibility of potential evaluation designs. Meetings were scheduled primarily with sites individually (although not always in person) to understand their program flow, respond to questions and concerns, and discuss research design options.

Impact Study Instruments and Consent

The goal of each piece of information from the participant is to estimate the effects of the interventions or to understand the experiences of individuals who receive program services. In total, the research team expects to enroll 5,000 individuals across multiple programs. Individuals seeking services from one of the sites being studied will be asked to complete the following activities.

Instruments:

1. Baseline Information Form for Participants (Attachment D). These include 15 minutes of questions and are completed by individuals electronically or on paper. Baseline information collected will be used to describe the sample, form subgroups, and increase the statistical precision of the impact analysis. As noted, this form includes the breadth of questions to be asked in the form. Most questions in certain indicated sections of the instrument, such as those about health and substance use, will only be included for the programs serving the relevant population. Similarly, most questions about housing preferences will only be asked for programs using whole family approaches.

2. Contact Update Request Form (Attachment E). This letter requests updated contact information from the participant, to increase likelihood of successful interview completion. We expect the form to take about 6 minutes to complete.

3. Informed Consent Form for Participants (Attachment H). An informed consent form will describe the study and what it means to participate. If willing, participants will sign the form.

4. 12- to 24-Month Follow-Up Participant Survey (Attachment K). This survey will be 30 minutes, using mixed mode fielding. Individual studies will include the relevant modules below depending on the population served. The survey modules include questions in the following areas:

   1. Service receipt. The 12- to 24-month survey will provide quantitative data on service receipt. Similar information will also be collected for the control group to define the treatment contrast. For the program group, these data can help describe how participants view the intervention, how they engage with program staff, and the services and activities they use.

   2. Employment-related outcomes. Although administrative records will provide information on quarterly earnings, the survey would collect information on jobs not covered by the unemployment insurance system and characteristics of jobs (such as, full-time vs. part-time, hourly wages, and so forth).

   3. Public assistance, material hardship, and housing. Individuals will be asked whether they or anyone in their household received benefits of the following types: TANF, SNAP, Supplemental Security Income or Social Security Disability Income, unemployment insurance, workers’ compensation, WIC, energy assistance, and housing vouchers. For each type of assistance, they will also be asked how many months they or anyone in their household received benefits since they were randomly assigned. Questions will also address material hardship and housing status.

   4. Health. This includes an adapted version of the SF-36 health survey about overall health status that has been found to be a strong predictor of use of health care resources and disability benefits.

   5. Behavioral health. Regarding depression, the survey will include the Kessler 6-item questionnaire. Regarding substance use, the survey will include questions from the Addiction Severity Index – Drug/Alcohol Use section, which is a validated screener used to identify those who have substance use problems.

Implementation Instruments

These implementation instruments will be tailored for each program design, target group, and local context, and the instruments presented here represent the universe of questions that could be included. Some of the instruments described below were tailored to programs already identified at the time of the initial information collection request (ICR). The remaining instruments described below include the universe of all questions that may be asked of sites recruited since the initial ICR and any future sites.

Approved Instruments:

1. Program Managers, Staff, and Partners Interview Guide – SUD Programs (Attachment F). Implementation studies are being conducted in SUD treatment or recovery programs that integrate employment services. If it is not possible to hold an in-person visit, interviews will be conducted virtually via phone calls or video conference. Specific topics include: the choice of target groups; participant outreach strategies; employment, training, and support service provided; SUD treatment and recovery services; development or refinement of existing employment-related activities and curricula to serve the target group; how and why partnerships were established; strategies for engaging employers with a SUD population; and promising practices and challenges.

2. Program Managers, Staff, and Partners Interview Guide – Whole Family Approach Programs (Attachment G). Implementation studies are being conducted in programs that integrate employment services with a whole family approach. If it is not possible to hold an in-person visit, interviews will be conducted virtually via phone calls or video conference. Specific topics include: the choice of target groups; participant outreach strategies; employment, training, and support service provided; other services offered; development or refinement of existing employment-related activities and curricula to serve the target group; how and why partnerships were established; and promising practices and challenges.

3. Program Managers, Staff, and Partners Interview Guide (Attachment L). Staff interviews are being conducted during two rounds of implementation study visits for most sites. If it is not possible to hold an in-person visit, interviews will be conducted virtually via phone calls or video conference. During each visit, 90-minute semi-structured interviews of program staff and partners will explore staff roles and responsibilities, the provision of program services, and implementation success and challenges. The number of interviewees will vary by site depending on the organization staffing structure, however, program managers, program staff (i.e. case managers), and key partners will be interviewed, for an average of 10 interviewees. Topics for the interviews include: program model and structure, staffing, program implementation, program components strategies and staff experiences, participant knowledge, awareness, participation, and views of program, use of services and incentives, and counterfactual environment.

4. In-Depth Case Study of Staff-Participant Perspectives: Participant and Program Staff Interview Guides (Attachments M and N). In-depth case studies will be conducted at up to 14 impact and implementation evaluation sites examining selected participants and their corresponding case manager to understand how program staff addressed a specific case, how the participant viewed the specific services and assistance received, and the extent to which program services addressed participant needs and circumstances. For each site, one-on-one interviews will be conducted separately with six participants (each 90 minutes in length) and with their respective case managers (60 minutes in length). Staff interviews differ from those described above because they will focus on how the staff member handled a specific case in contrast to how the program works overall. The case studies will provide examples of how the program worked for specific cases, and will enhance the overall understanding of program operations, successes, and challenges.

5. Program Staff Survey (Attachment O). Online staff surveys will be fielded to 20 staff at each site, as needed, and will cover background and demographics, staff responsibilities, types of services provided by the organization, barriers to employment, program participation, and organizational and program performance. Each survey will take 30 minutes to complete.

Administrative Records

In addition to the instruments outlined above, the study will use administrative records to estimate program impacts on key outcomes such as employment and earnings. These may include records on employment and earnings (National Database of New Hires), public assistance (TANF, Supplemental Nutrition Assistance Program (SNAP)), Medicaid claims, income (IRS), criminal justice records, child support awards and payments, disability benefits from the Social Security Administration, and National Student Clearinghouse data. Exact records to be collected will depend on the nature of the intervention. Respondents will be informed of any records that will be used as part of this study.

National Database of New Hires (NDNH) data will be the main source of information on employment and earnings. In addition, we will use two to three state data sources, depending on the specific program and population being served. These could include the following:

• Public benefits records will provide monthly Temporary Assistance to Needy Families (TANF) and Supplemental Nutrition Assistance Program (SNAP) benefit amounts, both prior to and after random assignment. These data might also provide information on Medicaid eligibility and some kinds of TANF-funded “non-assistance” such as wage subsidies paid to employers.

• Medicaid claims will provide information on Medicaid enrollment and Medicaid-reimbursed health care. This will be an important source of information on behavioral health treatment for interventions targeted at substance users and those with mental health problems.

• Other administrative data sources. Depending on the intervention, we might seek to acquire child support and criminal justice records. We might use child support records for interventions targeted at noncustodial parents or criminal justice records for sites that serve a high proportion of former prisoners. We may additionally use disability benefits from Social Security Administration and education records from the National School Clearinghouse. Finally, we will explore the possibility of matching our sample to IRS tax data to obtain a comprehensive set of information on earnings, income, and tax credits such as the Earned Income Tax Credit and the Child and Dependent Care Credit. MDRC is currently using IRS data in other projects.

Coordination with OPRE’s NextGen Project and SSA

As noted in Section A.1, BEES is actively coordinating with OPRE’s NextGen project. Both projects aim to evaluate employment-focused programs for the same broad populations: low-income and vulnerable populations with complex barriers to employment. Though both projects target similar populations, each have priority focus areas. In terms of interventions, NextGen will prioritize interventions that are market-oriented and/or employer-driven and as such BEES will not evaluate such interventions. In terms of populations of interest, BEES will prioritize evaluating interventions for individuals struggling with opioid or other substance use disorder; NextGen may also include individuals with substance use disorder in evaluations, as populations with complex barriers to employment may face multiple overlapping barriers. Further, both projects have partnered with the Social Security Administration to support the addition of a disability focus in both projects. The projects are coordinating closely in program identification, outreach, and recruitment to ensure efforts are not duplicated and programs are not burdened by being contacted and/or considered by both teams.

To ensure BEES and the NextGen project produce consistent results that can be compared, and to meet SSA’s priorities across both projects, the projects’ information collection requests to OMB will include several common instruments and/or questions within instruments. The below table summarizes common items across both project’s participant surveys and informed consent form.

BEES Instrument/Attachment	BEES items that will be the same in Next Gen materials
Baseline Information Form (Attachment D) Benefit Receipt	BEES items F.1 – F5
Baseline Information Form (Attachment D) Disability Status	BEES items I.1 – I.7
Baseline Information Form (Attachment D) Health	BEES items J.1 – J.9
Informed Consent Form (Attachment H)	Regarding other research uses: The Social Security Administration will do additional research on [how BEES/NEXTGEN program affects] your earnings and receipt of disability benefits. They will do this research through 2040. They will use information such as your name, gender, date of birth, and Social Security Number to try and locate you in their records. They will only use your information to do research. The information will not be used to make decisions about any benefits you receive from the Social Security Administration, now or in the future. The Social Security Administration will not contact you directly.
12- to 24- month Follow-Up Participant Survey (Attachment K) Service Receipt and Participation	BEES items A1a-e, A1k-l; A19
12- to 24- month Follow-Up Participant Survey (Attachment K) Program Satisfaction	BEES item B1, B3, B4
12- to 24- month Follow-Up Participant Survey (Attachment K) Physical and Mental Health	BEES items D8a-f
12- to 24- month Follow-Up Participant Survey (Attachment K) Substance Use	BEES section E
12- to 24- month Follow-Up Participant Survey (Attachment K) Household Information, Income, and Material Hardship	BEES items F1C-F1C.15, F7

A3. Improved Information Technology to Reduce Burden

This study uses information technology to minimize respondent burden and to collect data efficiently.

The baseline information form is administered electronically (using a computer, tablet, or smartphone) when possible. Completing the form electronically moves the participant quickly through the form. We also create a web page that would allow respondents to update their contact information easily, efficiently, and at a time most convenient for them.

For the 12- to 24-month interviews, respondents call a phone center for computer-assisted telephone interviewing (CATI). CATI reduces respondent burden by using skip logic to quickly move to the next appropriate question depending upon a respondent’s previous answer.

The online staff survey is hosted on the Internet via a live secure web-link. This approach is particularly well suited to the needs of these surveys in that respondents can easily stop and start if they are interrupted and review and/or modify responses in previous sections.

A4. Efforts to Identify Duplication

Data being collected for BEES is not available in any other form in a consistent manner across the evaluation’s sites.

Although many employment programs assess participants in some of the outcomes and baseline characteristics being collected in BEES, those assessments differ by local program, and local programs do not collect similar information on control group members. To the extent possible, state and federal administrative data are used to assess outcomes such as employment and receipt of public assistance, so the interviews of study participants are focused on outcomes that are not available from those administrative records sources.

Information that is being collected for implementation research is not expected to be available in any other form. For example, program sites are expected to vary to the extent that they use a management information system, and those systems are likely to not to include a common set of information across sites. However, the team has verified with each site that information being requested is available only through the surveys and qualitative interviews that are proposed.

As mentioned above, BEES is actively coordinating with another project sponsored by OPRE, the Next Generation of Enhanced Employment Services (NextGen) Project (see the Study Background subsection for additional details).

A5. Involvement of Small Organizations

We have some participating sites that are small community organizations.

We use technology to minimize the burden on program staff. Participants will be able to complete the baseline information form and informed consent form on a laptop or tablet where possible rather than relying solely on the use of paper forms, reducing the staff time required for this data collection.

A6. Consequences of Less Frequent Data Collection

The BEES data collection aims to collect information only as frequently as needed to achieve the aims of the study. Eliminating any proposed data collection items would compromise our ability to address key research questions.

1. Baseline Information Form for Participants (Attachment D). The baseline information form is administered once. Without it, we would be unable to verify that random assignment has yielded program and control groups similar in their observable background characteristics and in their baseline measures of outcomes. The baseline information form is also essential for describing the characteristics of the study sample.

2. 12- to 24-Month Follow-Up Participant Survey (Attachment K). For participants in a program with a mental health, disability, or substance use focus, the 12- to 18-month follow-up survey is administered once. The follow-up survey is essential for allowing us to estimate the program impacts across domains of interest.

3. Program Managers, Staff, and Partners Interview Guide (Attachment L). Semi-structured interviews will be conducted with program staff at two points in time, when possible. We are conducting two rounds of implementation study visits to each site to help understand how each program matures and changes over time, thus necessitating data collection at two points. If it is not possible to conduct the visits in-person, visits will be conducted virtually via phone calls or video conferencing. Due to the expected timing of site start up, we have enough time in the evaluation study period to conduct two rounds of site visits in up to 14 sites. These interviews will be critical to understand the implementation of each program and its context.

4. Participant and Program Staff Case Study Interview Guides (Attachments M and N). The case study interviews to be completed at up to 14 sites will provide a clearer understanding of how individuals move through the programs. This will add depth to the information gleaned from the interviews described above. For each site, individual interviews will be conducted with at most six participants (each 90 minutes in length) and with their case managers (60 minutes in length).

5. Program Staff Survey (Attachment O). The staff surveys will be fielded, as needed, once to twenty staff members at up to 14 participating sites. These surveys will provide further staff perspective on topics such as organizational performance and staff responsibilities. It will complement the interviews and case studies described above.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on July 21, 2022 Volume #87, Number 139, page 43527, and provided a sixty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

No outside experts were consulted in designing the data collection package.

A9. Tokens of Appreciation

The BEES study is designed to test the effectiveness of a variety of employment services using a random assignment study design, with longitudinal follow-up. The size of the sample is set at study recruitment, and a high response rate is necessary to maintain statistical power to detect meaningful effects when measuring participant outcomes. In addition, the integrity of the study’s estimates requires maintaining similar response rates for the randomly assigned treatment and control groups and across demographic groups of central interest to the research study. Maintaining high response rates will be especially difficult in BEES because the study may include target populations, such as, individuals facing substance and opioid use disorders, disability or mental health challenges, or others who may enroll in TANF or SSI/SSDI, all of which represent individuals with whom it may be particularly difficult to maintain contact over time. Their circumstances often result in frequent moves, short stays in hospitals or treatment centers, short periods of time living with others, and in some instances, homelessness. Because of the complex design and study population with many unmet needs, it is important to build respondent buy-in early in the study and retain as much of the sample as possible over time.

As previously approved, we have provided or plan to provide the following tokens of appreciation during the outreach, locating, and data collection process.³ These are intended to show study participants that the study team appreciates their ongoing participation in data collection activities and offset any incidental costs of participation. In accordance with OMB guidelines, the team took several factors into consideration when determining whether or not to use tokens.⁴ Specifically, the team took into account design-specific threats to data quality, efforts to reduce non-response bias, the complexity of the study design, and study sample retention over the 12- to 24-month follow-up period. As described in Supporting Statement B, section B3, these are one part of a broader study design intended to build participant buy-in and maintain participation over time.

The team’s logic and the amount approved for specific tokens is as follows:

Longitudinal Surveys

As previously approved, we will offer tokens for study participants at each stage of data collection related to the survey protocol. Three factors informed the study’s choice of the amounts for survey respondents:

1. Respondent burden, both at the time of the interview and over the life of the study;

2. Costs associated with participating in the interview at that time; and

3. Other studies of comparable populations and burden

The following token amounts for the BEES structured interviews will be provided as a gift card upon completion:

• $20 for completing a 15-minute survey at study enrollment/baseline

• $5 for updating contact information at 3 months, 6 months, or 9 months post enrollment.

• $25 for completing a 30-minute 12- to 18-month (post enrollment) survey.

In designing the token structure, the team considered that the BEES population is expected to be highly mobile, thus maintaining good contact data is critical to follow-up contact efforts. This is the logic for the contact update tokens of appreciation. The amount relate to the survey increases commensurate with increased length of the follow-up period and the survey length. We offer a $25 incentive for the 12- to 24-month follow-up survey to reflect the longer survey protocol administered at that time. The team theorizes that tokens will be a particularly powerful tool for maintaining a high response rate in the control group, given that these sample members will not be engaged in program services.

In-depth Interviews:

For the interviews with program participants for the case studies providing the staff/participant perspective, it is important to secure individuals with a range of background characteristics to capture a variety of possible experiences with program services. Although participation in the interviews is voluntary, there are potential burdens placed on study participants. Without offsetting the direct costs incurred by respondents for participating in the interviews, such as transportation and child care, the research team increases the risk that only those individuals able to overcome potential financial barriers will participate in the study.

As such, we are providing $50 for completing a semi-structured 90-minute interview during program participation, to be provided by gift card upon completion. The participants will receive a $50 gift card to account for expenses such as transportation, and/or childcare that may otherwise prevent their participation in the study. We believe $50 is a reasonable amount for the time and cost associated with participation in these data collection activities, but is not so high as to appear coercive for potential participants.

A10. Privacy of Respondents

The study team is committed to protecting the privacy of participants and keeping private the data that are entrusted to us, to the extent permitted by law. Respondents are informed of all planned uses of data, that their participation is voluntary, and that their information is kept private to the extent permitted by law. Study participants are asked to provide a Social Security number (SSN) to facilitate collecting administrative data from state and federal agencies and to help the study team locate them in the future, but individuals do not have to provide an SSN to be in the study. The study’s consent form is included as Attachment H.

The MDRC IRB review each evaluation planned as part of BEES individually. The study is currently in the site identification phase. As programs agree to participate and develop a study design alongside the research team, IRB applications are presented for review and approval.

Due to the sensitive nature of this research (see A11 for more information), the evaluation has obtained a Certificate of Confidentiality. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

The following privacy and data security measures are in place to protect respondents’ privacy, including any personally identifiable information collected about them:

1. All data, including paper files, portable media (e.g., voice/video recordings) and computerized files, are kept in secure areas. Paper files and portable media are stored in locked storage areas with limited access on a need-to-know basis.

2. Merged data sources have identification data stripped from the individual records or encoded to preclude overt identification of individuals.

3. All reports, tables, and printed materials are limited to the presentation of aggregate numbers.

4. Compilations of individualized data are not provided to participating sites.

5. Confidentiality agreements are executed with any participating research subcontractors and consultants who must obtain access to detailed data files. These agreements are corporate forms and are not distributed to respondents.

6. As specified in the evaluator’s contract, the research team shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard.  The research team ensures that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information.

7. Any data stored electronically is secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations.

8. All research staff are trained on appropriate privacy and data security matters.

Information is not maintained in a paper or electronic system from which they are actually or directly retrieved by an individual’s personal identifier.

A11. Sensitive Questions

Some questions in the baseline information form and 12- to 24-month interviews may be sensitive for study participants. Individuals are being asked about their use of alcohol and drugs as well as about their mental health, particularly depression symptoms. These questions are necessary because a goal of the study is to understand the effects of employment interventions for individuals with opioid dependence, other substance abuse, or mental health issues. As noted in section A4, this information will not be available from other data sources.

Across all data collection, respondents are informed by research staff prior to the start of the interviews or surveys that their answers will be kept private to the extent permitted by law, that results will only be reported in the aggregate, and that their responses will not affect any services or benefits they or their family members receive.

At the point of enrollment in the study, the informed consent form (Attachment H) provides an overview of data collection efforts to expect during the course of the study. Staff obtaining consent from participants are trained to answer questions about what it means to participate in the study. Participants who have agreed to be in the study are asked to provide personally identifiable information on the baseline information form (Attachment D) including, for example, date of birth, address, and social security number. The Baseline Information Form likewise asks about current mental health, substance use, and disability. At follow-up, interviews (Attachments F and G) may ask sensitive questions as well. Both ask, for example, about substance use disorder and mental health treatment for relevant populations. The collection of personal identifiers is necessary for participant locating for follow-up interviews and to allow us to access and match administrative records data.

A12. Estimation of Information Collection Burden

Table A.1 shows the annual burden and cost of the completed data collection instruments and activities described in this ICR. Table A.2 shows the annual burden and cost of the ongoing data collection instruments and activities described in this ICR.

Table A. 1: Previously Approved and Completed Data Collection

The time period for this approval spans from November 2019 through November 2022. The assumed wage rate was based on the May 2017 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor Statistics. The rate used for researchers and policy experts, $60.47, was equivalent to management, scientific, and technical consulting services under SOC code 19-3011. The rate used for State and local administrators, $49.12, was equivalent to the local government managers under SOC code 11-1021. The rate used for program staff $23.62 was equivalent to local government workers under SOC code 21-1023. The average hourly wage of participants is estimated from the average hourly earnings ($4.92) of study participants in the Building Strong Families Study (Wood et al., 2010). These average hourly earnings are lower than minimum wage because many study participants were not working. This is also the case for BEES study participants.

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total annual cost
Attachment A. Discussion guide for national policy experts and researchers	10	3	1	1	3	$60.47	$181.41
Attachment B. Discussion guide for state and local administrators	55	18	1	2	36	$49.12	$1768.32
Attachment C. Discussion guide for program staff at potential sites	72	24	1	2.75	66	$23.62	$1558.92
Attachment D1-D5. Baseline information form for participants	2000	667	1	0.25	167	$4.92	$821.64
Attachment E. Contact Update Letter and Form	1000	333	1	0.1	33	$4.92	$162.36
Attachment F. Program managers, staff, and partner interview guide – SUD Programs	50	50	1	1.5	75	$23.62	$1771.5
Estimated Annual Burden Total					380 hours		$6264.15

Table A.2: Burden for Ongoing Data Collection

The below table reflects the amount of burden still remaining for collection during the renewal period. This includes data collection related to the impact studies – baseline and follow up surveys of study participants – and for the implementation studies. There are no updates to the estimated burden, but for this ICR, the remaining burden has been annualized over the new request time period.

The assumed wage rate was based on the May 2021 employment and wages from Occupational Employment Statistics survey from the Bureau of Labor Statistics (http://www.bls.gov/oes/current/oes_stru.htm). The rate used for program staff $23.62 was equivalent to local government workers under SOC code 21-1023. The average hourly wage of participants is estimated from the average of the minimum wages in states where BEES participants are enrolling in programs. This number is adjusted for unemployment to account for the many participants in the BEES study who are not working.

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage		Total annual cost
Attachment D1-D5. Baseline information form for participants	3,000	1,000	1	0.25	250	$10.90		$2725.00
Attachment E. Contact Update Letter and Form	4,000	1,333	1	0.1	133	$10.90		$1449.70
Attachment F. Program managers, staff, and partner interview guide – SUD Programs	30	10	2	1.5	30	$23.62		$708.60
Attachment G. Program managers, staff, and partner interview guide – Whole Family Approach Programs	20	7	2	1.5	21	$23.62		$496.02
Attachment K. 12-month Follow-Up Participant Interview	4300	1433	1	0.5	717	$10.90		$7815.30
Attachment L. Program Managers, Staff, and Partners Interview Guide	200	67	2	1.5	201	$23.62		$4747.62
Attachment N. Program Staff Case Study Interview Guide	84	28	1	1	28	$23.62		$661.36
Attachment M. Participant Case Study Interview Guide	84	28	1	1.5	42	$4.92		$206.64
Attachment O. Program Staff Survey	300	100	1	0.5	50	$23.62		$1181.00
Estimated Annual Burden Total					1,472 hours				$19,991.24

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities is estimated to be $10,120,000. With approval of the extension, annual costs to the Federal government will be $3,373,333 for the proposed data collection. The table below breaks down these costs.

Total Costs
Cost Category	Estimated Costs
Surveys	$5,715,000
Implementation	$435,000
Publications/Dissemination	$3,970,000
Total	$10,120,000

A15. Change in Burden

This is an extension to an approved information collection. There are no changes to burden estimates, but burden has been updated to reflect completed data collection activities and only include burden for data collection that will take place over the next three years. The remaining burden has been annualized over the new request time period.

Additionally, as described in A2, the fielding period for the follow up surveys has been adjusted slightly. As such, the file names for the follow up surveys were updated and reuploaded. There are no changes to the actual survey content or content in the related materials.

Finally, all materials have been updated to reflect current templates and naming conventions used by the ACF Office of Planning, Research, and Evaluation.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Analysis Plan

Estimating program impacts.

With plans for randomized controlled trials (RCT) in many sites, the starting point for the intent-to-treat impact analysis is to compare outcomes for all program group members and control group members. The precision of the estimates will be enhanced by estimating multivariate regression models that control for factors at baseline that may also affect the outcome measures. Such impacts are often referred to as “regression-adjusted” impacts. To increase precision, impact estimates are regression adjusted, controlling for baseline characteristics. Impacts will also be estimated for key subgroups to investigate whether the interventions have larger effects for some groups of participants (Bloom and Michalopoulos, 2011). In the main subgroup analysis, subgroups will be chosen using baseline characteristics, based on each evaluation’s target population and any aspects of the theory of change that suggest impacts might be stronger for some groups. Supporting Statement B details the survey response bias analysis approach, including weighting, if necessary. Statistical analyses will be conducted in SAS and R.

Analyzing Implementation Study data.

Notes from qualitative data collection are imported into Nvivo, MDRC’s mixed-methods analysis software. Notes are coded using a pre-specified coding scheme that accounts for the priorities of research questions and what we hope to learn from the process study. Quantitative data (interviews and surveys, for example) undergoes descriptive statistics analysis in SAS or R. If warranted, quantitative data may also be imported into Nvivo for analysis.

Time Schedule and Publication

As noted earlier, we expect data collection to continue to take place over an additional three-year period, following OMB approval. As sites continue enrolling participants in the evaluation, the interviews occur 12 to 24 months following each participant enrollment.

Implementation study activities for sites where an impact study will occur–including site visits, staff survey, and case studies–will begin after sites begin enrollment in the study. Pending OMB approval, we expect this data collection effort will continue through 2025. For the implementation-only studies identified to date, information collection is scheduled to be completed in 2022. Publication and dissemination of the findings – through briefs, reports, journal articles and conference presentations – will occur on a rolling basis, after data collection and analysis is completed for a given site.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

Attachments

Previously Approved Instruments Currently in Use:

Baseline Information Form for Participants (Attachment D)

• Baseline Information Form for Participants (CCC) (Attachment D-1)

• Baseline Information Form for Participants (2Gen-Chicago) (Attachment D-2)

• Baseline Information Form for Participants (IPS FQHC) (Attachment D-3)

• Baseline Information Form for Participants (IPS SUD) (Attachment D-4)

• Baseline Information Form for Participants (IPS TANF-SNAP) (Attachment D-5)

Contact Update Request Form (Attachment E)

Program Managers, Staff, and Partners Interview Guide – SUD Programs (Attachment F)

Program Managers, Staff, and Partners Interview Guide – Whole Family Approach Programs (Attachment G)

12- to 24-Month Follow-Up Participant Survey (Attachment K)

• 12- to 24-Month Follow-Up Participant Survey (IPS SUD/FQHC) (Attachment K-1)

• 12- to 24-Month Follow-Up Participant Survey (CCC) (Attachment K-1)

Program Managers, Staff, and Partners Interview Guide (Attachment L)

In-Depth Case Study of Staff-Participant Perspectives

• Participant Case Study Interview Guide (Attachment M)

• Program Staff Case Study Interview Guide (Attachment N)

Program Staff Survey (Attachment O)

Previously Approved Instruments No Longer in Use:

Discussion Guide for National Policy Experts and Researchers (Attachment A)

Discussion Guide for State and Local Administrators (Attachment B)

Discussion Guide for Program Staff at Potential Sites (Attachment C)

6-Month Follow-Up Participant Survey (Attachment J)

Supplementary Materials Currently in Use:

Informed Consent Form for Participants (Attachment H)

Welcome Letter (Attachment I)

12- to 24- Month Survey Advance Letters (Attachment P)

12- to 24- Month Survey Email Reminders (Attachment Q)

12- to 2- Month Survey Flyer (Attachment R)

References

Berlin, M., Mohadjer, L., Waksberg, J., Kolstad, A., Kirsch, I., Rock, D., & Yamamoto, K. 1992. “An experiment in monetary incentives.” Proceedings of the Survey Research Section of the American Statistical Association, 393-398.

Bloom, Howard S., and Charles Michalopoulos. 2011. “When is the Story in the Subgroups? Strategies for Interpreting and Reporting Intervention Effects on Subgroups” Prevention Science, 14, 2: 179- 188.

Butler, David, Julianna Alson, Dan Bloom, Victoria Deitch, Aaron Hill, JoAnn Hsueh, Erin Jacobs

Valentine, Sue Kim, Reanin McRoberts, and Cindy Redcross. 2012. What Strategies Work for the Hard-to-Employ? Final Results of the Hard-to-Employ Demonstration and Evaluation Project and Selected Sites from the Employment Retention and Advancement Project. New York, NY: MDRC.

Council of Professional Associations on Federal Statistics. 1993. “Providing Incentives to Survey

Respondents: Final report.” Retrieved January 17, 2019 from http://www.copafs.org/reports/providing_incentives_to_survey_respondents.aspx.

Dillman, D. A., Smyth, J. D., & Christian, L. M. 2009. Internet, mail, and mixed-mode surveys: The

tailored design method. (3rd ed.) New York, NY: John Wiley & Sons.

Gueron, Judith M., and Edward Pauly. 1991. From Welfare to Work. New York, NY: Russell Sage Foundation.

Hamilton, Gayle. 2002. Moving People from Welfare to Work: Lessons from the National Evaluation of

Welfare-to-Work Strategies. Washington, D.C.: U.S. Department of Health and Human Services

and U.S. Department of Education.

Hamilton, Gayle, and Charles Michalopoulos. 2016. Job Search or Basic Education Participation First:

Which Improves Welfare Recipients’ Earnings More in the Long Term? New York, NY: MDRC.

Holzer, Harry J., Stephen Raphael, and Michael A. Stoll. 2004. “Will Employers Hire Former Offenders?

Employer Preferences, Background Checks, and Their Determinants.” Pages 205-243 in Mary Pattillo, David Weiman, and Bruce Western (eds.), Imprisoning America: The Social Effects of Mass Incarceration. New York: Russell Sage Foundation.

Jacobs Valentine, Erin. 2012. Returning to Work After Prison: Final Results from the Transitional Jobs

Reentry Demonstration. New York, NY: MDRC.

Kelly, B., Margolis, M., McCormack, L., LeBaron, P.A., Chowdhury, D. 2017. “What Affects People’s

Willingness to Participate in Qualitative Research? An Experimental Comparison of Five Incentives.” Field Research. Volume: 29 issue: 4, page(s): 333-350

Lerner, Debra, David A. Adler, William H. Rogers, Hong Chang, Annabel Greenhill, Elina Cymerman,

and Francisca Azocar. 2015. "A randomized clinical trial of a telephone depression intervention to

reduce employee presenteeism and absenteeism." Psychiatric Services 66, no. 6: 570-577.

Looney, Adam, and Turner, Nicholas. 2018. Work and Opportunity Before and After Incarceration. Washington, D.C.: The Brookings Institution.

Mercer, Andrew, Andrew Caporaso, David Cantor, and Reanne Townsend. 2015. “How Much Gets You Much? Monetary Incentives and Response Rates in Household Surveys.” Public Opinion Quarterly 79: 105-129.

Moffitt, Robert. 2004.  The Three-City Study Incentive Experiment: Results from the First Two Waves.  Retrieved from http://www.jhu.edu/~welfare.

Mueser, Kim T., Robert E. Drake, and Gary R. Bond. 2016. "Recent advances in supported employment for people with serious mental illness." Current opinion in psychiatry 29, no. 3: 196-201.

Pager, Devah. 2003. “The Mark of a Criminal Record.” American Journal of Sociology 108: 937-975.

Redcross, Cindy, Megan Millenky, Timothy Rudd, and Valerie Levshin. 2012. More Than a Job: Final

Results of the Center for Employment Opportunities. New York, NY: MDRC.

Singer, Eleanor, John Van Hoewyk, Nancy Gebler, Trivellore Raghunathan, and Katherine McGonagle.

“The effect of incentives in telephone and face-to-face surveys.” Journal of Official Statistics. 1999;15: 217–30.

Singer, Eleanor and Richard A. Kulka. 2002. “Paying respondents for survey participation,” in Studies of Welfare Populations: Data Collection and Research Issues, eds. Michele Ver Ploeg, Robert A. Moffitt and Constance F. Citro, National Academy Press, Washington, D.C., pp. 105-127.

Singer, Eleanor, and C. Ye. 2013. The Use and Effects of Incentives in Surveys. The Annals of the

American Academy 645, 112-141.

Uggen, Christopher, Sara Wakefield, and Bruce Western. 2005. “Work and Family Perspectives in

Reentry.” Pages 209-243 in Jeremy Travis and Christy Visher (eds.), Prisoner Reentry and Crime in America. New York: Cambridge University Press.

Wang, Philip S., Gregory E. Simon, Jerry Avorn, Francisca Azocar, Evette J. Ludman, Joyce McCulloch, Maria Z. Petukhova, and Ronald C. Kessler. 2007. "Telephone screening, outreach, and care management for depressed workers and impact on clinical and work productivity outcomes: a randomized controlled trial." Jama 298, no. 12: 1401-1411.

Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, and JoAnn Hsueh. 2010. The Building Strong Families Project: Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families. Washington, D.C.: U.S. Department of Health and Human Services.

1 This covers a few types of questions in the survey, including recovery from substance use disorder, mental and physical health. Some also overlap with what would fall under “economic security” such as housing situation/stability and food insecurity.

2 To review previously approved instruments that are no longer in use, see https://www.reginfo.gov/public/do/PRAICList?ref_nbr=202112-0970-013

3 Please refer to Supporting Statement B of this information collection request for more detail on the team’s plans to maximize response rates and minimize non-response bias.

4 See page 69, questions 75 and 76, https://obamawhitehouse.archives.gov/sites/default/files/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf

20