Attachment P. Request to change burden and incentive structure-amount

MEMORANDUM

To: Josh Brammer, Office of Information and Regulatory Affairs, Office of Management and Budget (OMB)

From: Hilary Bruck and Victoria Kabak, Office of Planning, Research, and Evaluation, Administration for Children and Families (ACF)

Date: March 4, 2020

Subject: Request for changes to estimated respondent burden and incentive structure/amount for the Evaluation of Employment Coaching for TANF and Related Populations (OMB Control Number 0970-0506)

Background

Type of Request: Non-substantive change of estimated survey burden in all sites based upon information from fielding to date. Non-substantive change to incentive structure and amount for longitudinal surveys for four of the six sites.

Study Features Salient to Request: The purpose of the Evaluation of Employment Coaching for TANF and Related Populations (OMB control number 0970-0506) is to describe select employment coaching interventions for low-income populations and estimate their effectiveness. ACF will use information from the evaluation to inform policymakers and practitioners interested in funding, designing, or implementing interventions to improve employment outcomes of low-income populations.

The impact evaluation involves a randomized controlled trial in six sites. During baseline study enrollment, people eligible for employment coaching were randomized into a treatment group who were offered employment coaching and a control group who were not offered employment coaching.

Two follow-up surveys will collect data on the outcomes of members of the treatment and control groups. The first follow-up survey is administered between 6 and 12 months after random assignment; the second follow-up survey is administered between 21 and 24 months after random assignment. As the timing of study enrollment varied by site, the survey data collection at each site proceeds on its own timeline.

Progress to Date: At two of the six sites in the study—MyGoals in Baltimore and MyGoals in Houston—survey production is on target to reach 80 percent response with small differences in the response rates between treatment and control groups. We do not request any changes to the use of incentives at these sites.

In four sites—Family Development and Self-Sufficiency (FaDSS), Jefferson County Colorado Works, LIFT, and Work Success—completions of the first follow-up survey are lower than anticipated and the treatment-control response rate differentials are higher than anticipated. This is the case despite lengthening the survey data collection period beyond the originally scheduled three months. We are concerned that observed treatment/control non-response bias and longer-than-expected fielding periods will affect the evaluation’s ability to generate unbiased estimates of whether the programs were effective.

Sample members are released for data collection on a rolling basis based on their date of study enrollment. ACF has secured the following participation in the survey for the sample that has been in the field for at least six months in FaDSS, Jefferson County, and LIFT; more recent enrollees with less than six months in the field are not included in the figures for these sites. At Work Success, where enrollment began later than in the other sites and no sample members have been in the field for six months, the figures include sample that has been in the field for at least three months.

Current response rates for FaDSS, LIFT, Jefferson County, and Work Success sites^a

Site	Total number releasedb	Overall response rates	Response rate in treatment group	Response rate in control group
FaDSS	301	51%	53%	49%
Jefferson County	92	41%	48%	35%
LIFT	173	72%	78%	65%
Work Success	134	43%	54%	32%

^a The survey began in March 2019 in FaDSS, July 2019 in Jefferson County, June 2019 in LIFT, and November 2019 in Work Success.

^b Includes sample members released for at least six months for FaDSS, Jefferson County, and LIFT and for at least three months in Work Success.

Previous Terms of Clearance: The following tokens of appreciation for both the first and second follow-up surveys were approved by OMB under Control Number 0970-0506 (March 2018):

• Participants responding before 4 weeks: $35 gift card

• Participants responding after 4 weeks: $25 gift card

Time Sensitivity: This request has high time sensitivity. Increases in response rates need to occur before the first follow-up survey is closed, which can be no later than the start of the second follow-up survey. Sample members are released for second follow-up survey data collection on a rolling basis based on their date of study enrollment. The second follow-up survey data collection will begin in March 2020 for the earliest enrollees at the FaDSS site.

Request Overview

ACF requests to update the survey’s communication protocols for all sites to reflect data on the average time to complete the survey. ACF also requests a non-substantive modification of the survey incentive structure and amount for four sites, for the first and second follow-up surveys.

These changes are proposed as part of OPRE’s ongoing efforts to address lower-than-expected response rates and observed non-response bias between treatment and control in four RCT sites. ACF is concerned that without these interventions, data will produce biased estimates of program impacts and will underrepresent participants in key analytic groups.

The remainder of this memo highlights ACF’s efforts to date to secure the planned response rates and address emerging non-response bias; describes the proposed changes to the survey fielding protocol; and explains the hypothesized impact of the proposed approach and assessment of the effort.

Mitigation to Date

From the beginning of survey fielding, ACF has actively monitored survey production and response rates. To date, the contract team has taken the following steps to improve respondent location and encourage participation:

Initial Design

• The follow-up surveys were offered to respondents in two modes: web and telephone, allowing study participants to complete the surveys how and when it is convenient for them.

• Survey outreach used multiple channels and multiple contact attempts. Prior to attempted telephone contact, all participants were notified about the data collection through an advance letter. Participants who provided an email address were sent email messages providing login information for the survey web form. Participants with an email address and/or assent to text were sent two email messages and/or two text messages prior to telephone contact attempts.

• Nonresponding participants received multiple telephone calls over several weeks on different days and at different times asking them to complete the survey.

• Non-working telephone numbers or those that did not result in contact were subject to locating, through contact with individuals provided by the participant at the time of study enrollment.

• A specialized letter was sent to participants who make non-adamant refusals, followed by contact from a specially-trained and experienced telephone interviewer.

• If the participants could not be contacted by telephone, field locators made in-person contact with them and provided them with a cell phone to complete the survey with a telephone interviewer.
  
  

Additional Mitigation

As it became apparent that survey production in the four affected sites would likely be insufficient, the contractor took additional steps to address non-response in these sites.

• The survey fielding period was expanded to provide additional time to locate participants and complete surveys.

• Site staff began providing updated participant contact information for nonresponding sample members.

• The contractor debriefed staff at the FaDSS and Jefferson County sites to diagnose why participants were reluctant to complete the follow-up surveys. Staff reported that housing costs in some areas increased levels of mobility and homelessness among study participants.

• The contractor debriefed interviewers and found that the stated 60-minute length of the survey discouraged some participants from starting the survey.

Plans for Future Mitigation

• (The subject of this request) ACF requests a change to the burden estimate for communications with sample members at all sites, and a change to the survey incentive structure/amount in four sites.

• Site staff will continue to provide updated participant contact information for nonresponding sample members.

• Field locating staff will monitor local homeless shelters in areas where nonresponding sample members were last known to be living.

• As described in Supplemental Statement B of the OMB package approved in March 2018, following the conclusion of data collection, we will use demographic information from the baseline survey to produce survey weights that account for the study design and statistically correct for non-response bias on observable factors, to support internally-valid estimates of program impacts for study participants.
  
  

Proposed Intervention for OIRA Approval

We will continue our increased efforts in tracing and field location, described above.

The advance materials for the survey currently inform study participants that the survey will take an average of 60 minutes to complete. Because the average length of the interviews to date is 45 minutes, ACF proposes to change the burden estimate used in communication with study participants across all sites from 60 to 45 minutes. Changing the burden estimate to the lower, more accurate number, could increase the likelihood that sample members agree to complete a survey.

We also propose to replace the currently-approved $35 or $25 incentive with a $50 incentive to all study participants completing a follow-up survey in the FaDSS, Jefferson County, LIFT, and Work Success sites. This increased incentive would be offered for completion of both the first and second follow-up surveys. This amount would be offered irrespective of whether sample members complete the survey within the four-week “early bird” period. The incentive would not be changed in the two sites in which response rates are reaching the targeted levels.

ACF’s proposed change to the incentives is aimed at increasing the likelihood that sample members who are successfully contacted respond to the survey. It is also intended to increase the likelihood that friends and relatives of the respondents who are contacted by the survey team will tell respondents about the opportunity to complete the survey.

Revisions have been made to study materials that reference the length of the survey and amount of incentives, retaining versions and/or language regarding the original incentive amount and structure for use with the two MyGoals sites (see Appendices A, C, I, and N). Work Success is the only site still enrolling participants into the study; therefore, the revisions to the informed consent form are only applicable to that site.

Expected Benefits and Proposed Assessment

We hypothesize that reducing the perceived respondent burden and increasing the incentives will increase response rates conditional on successful contact, particularly for the control group that is disproportionately underrepresented among survey respondents. This would reduce the risk of bias in our experimental impact estimates.

While recent RCTs with similar populations do not provide causal evidence about the impact of incentives on survey participation, several data collections provide observational data suggesting that the amount may be sufficient to support desired outcomes. Recent randomized controlled trials with similar populations have used incentives in the $40 to $50 range to conduct longitudinal surveys of equal or lower respondent burden; these data collection efforts had lower non-response bias than the current study. Examples include the Enhanced Transitional Jobs Demonstration study (OMB control number 0970-0413), which obtained a 70 percent or greater overall response rate in five out of eight sites, with an average treatment-control differential of less than 3 percent. The evaluation of YouthBuild (OMB control number 1205-0503) and the Self-employment Training Demonstration (OMB Control Number 1205-0505) similarly achieved high overall response rates with low treatment-control differentials and offered incentives of $40 to $50. The Supplemental Nutrition Assistance Program (SNAP) Employment and Training Study conducted for the U.S. Department of Agriculture’s Food and Nutrition Service offers respondents $40 for completing the study’s survey, which averages 30 minutes to complete. Given these recent experiences, we propose that a $50 incentive is an appropriate amount for respondents for completing a 45-minute follow-up survey.

This proposal was designed to maximize improvements in data quality given the results of ACF’s efforts to date and the constraints of remaining time and budget for the data collection. ACF is interested in using this opportunity to contribute to the body of evidence on the role of incentives in mitigating non-response bias. We intend to calculate pre-post response rates for treatment and control group members and the study participants as a whole. In addition, we will examine the demographic characteristics of pre- and post-$50 respondents and compare them with the full baseline study sample. We will use this analysis as a measure of the impact of our changes to fielding protocol on observable non-response bias.

Given that we propose to pair the incentive with additional outreach and a change in the communication about the length of the survey, this analysis will not isolate the causal effect of increased incentives alone. ACF anticipates that the information resulting from our survey administration efforts, including the proposed change in incentive amount, will be of sufficient quality to meaningfully contribute to ongoing learning about strategies for improving the quality, efficiency, and effectiveness of our survey data collections.