NonSub Change Memo - 12-18 Month Follow Up Tailored for SUD Progs & Token of App at Baseline

To: Jordan Cohen

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

From: Megan Reid

Office of Planning, Research, and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: July 25, 2022

Subject: Nonsubstantive Change Request – Building Evidence on Employment Strategies (OMB #0970-0537)

This memo requests approval of two nonsubstantive changes for the approved information collection, Building Evidence on Employment Strategies (OMB #0970-0537):

1. Updates to a follow up survey instrument to tailor for a recruited site, and

2. Adjustments to the token of appreciation structure.

Background

The goal of the BEES study is to evaluate the effectiveness of innovative programs designed to boost employment and earnings among Americans with low incomes. As described in the justification package for this study, approved instruments may be tailored to specific sites as they are recruited. Multiple BEES sites focused on delivering employment services to clients of a Substance Use Disorder (SUD) treatment and recovery provider or those receiving behavioral health services at a Federally Qualified Health Center. This request includes a tailored 12-month follow-up survey for participants in these sites.

The BEES study is utilizing a randomized control trial methodology to assess program impacts. As is considered best practice in random assignment studies, impacts will be estimated using an intent-to-treat design where the entire program group will be compared with the entire control group, regardless of whether program group members received services from the study program, and regardless of whether control group members received similar services in the community. The ability of the study to provide precise estimates of the effects of programs will depend in part on the number of participants who are enrolled in the study and for which data are collected. In addition, results will be most relevant to the federal government if a broad sample of individuals can be enrolled.

However, enrollment into BEES has been slower than expected. As shown in Table 1 below for select study sites, we are projecting that we will be unable to meet our study enrollment goal based on the enrollment to date and study enrollment timeline.

Table 1. Study Enrollment Progress to Date

Study/Location	Enrollment Start Date	Total Enrollment to Date (as of 7/15/22)	Total Enrollment Goal	Projected Enrollment	Enrollment End Date
IPS FQHC (Chicago, IL)	April 2021	294	500	354	October 2022
IPS FQHC (Nashua, NH)	May 2021	135	250	175	November 2022
IPS SUD (Tulsa, OK)	July 2021	270	450	370	January 2023
IPS SUD (Toledo, OH)	July 2021	162	250	210	January 2023

Previous Terms of Clearance:

As described in the justification package for this study, approved instruments may be tailored to specific sites as they are recruited. Attachment K: 12- to 18-Month Follow-Up Participant Survey was previously approved.

Currently, BEES study sites have been approved to offer participants a token of appreciation for completing follow-up surveys and contact update requests, but no participant tokens of appreciation were proposed previously for baseline data collection.

Overview of Requested Changes

Tailored Follow-Up Survey

First, we request to tailor the previously approved Attachment K: 12-to 18-Month Follow-Up Participant Survey for the sites noted above and the relevant program content and local context (See Attachment K-1).

Updates include the following:

• Customized language in questions and response categories throughout related to the local context and available program services.

• Added introductory language for the online version of the survey, to be used when applicable.

• Reorganized or grouped individual questions (or series of questions) and added skip logic to improve the administration of the survey to participants.

• Removed section at the end of the instrument that confirmed and/or collected contact information for friends or family members.

Changes to Token of Appreciation Structure

Regarding the change to the token of appreciation structure, the analysis of enrollment to date illustrated in Table 1 raises concerns about the ability of the BEES study to answer the study’s research questions. As a result, we are now requesting a modification of the BEES study token of appreciation structure as part of our ongoing efforts to address lower-than-expected study enrollment. Below we outline the study team’s efforts to date to improve enrollment, details and the hypothesized benefits of offering tokens of appreciation to participants at baseline.

Strategies Implemented to Date

• Expanded eligibility criteria

• Expanded referral sources/partners

• Behavioral science infusion (Infusing recruitment materials and approaches with behaviorally informed insights via MDRC’s Center for Applied Behavioral Science)

• Human-centered design approach (Team has used a human-centered design approach to look for ways to make the study enrollment process more inviting to prospective participants)

• Ensuring warm handoffs between referral sources and study staff

• Connection to leadership (Direct collaboration with site leadership to ensure support for study and program activities at all levels)

• Direct recruitment outreach to potential participants

• Connection to referral sources (facilitated sessions between employment intervention developers and referral sources on the importance of employment and to inform understanding of barriers to referrals)

• Referral system enhancements, such as simplifying referral submission processes

• Data-informed TA (Regular data collected to inform understanding of bottlenecks in the recruitment process and target technical assistance accordingly)

Proposed Intervention and Expected Benefits

As previously approved by OMB, the evaluation’s data collection plan currently includes gift cards to participants for completing different study activities. Participants who complete a 30-minute follow up interview will receive a $25 gift card as a token of appreciation for their participation. In addition, small amounts will be given to sample members to encourage them to update their contact information prior to the follow up interview. Also important to note, a small number of study participants will be asked to participate in longer (90-minute) semi-structured interviews.

We propose adding a similar $25 gift card token of appreciation to participants at baseline. The goal of this additional token of appreciation is to increase the number of people enrolled in the study and provide a large enough sample to answer the study’s research questions. In addition, this might encourage people to participate in future interviews once they have been enrolled. As noted in Supporting Statement A, these are intended to show study participants that the study team appreciates their ongoing participation in data collection activities and offset any incidental costs of participation.