PACE - Year Up Incentive Change Request Memo

To: Jennifer Park and Josh Brammer; Office of Information and Regulatory Affairs (OIRA); Office of Management and Budget

From: Brendan Kelly, Office of Planning, Research and Evaluation (OPRE); Administration for Children and Families (ACF)

Date: November 21, 2014

Subject: Pathways for Advancing Careers and Education (PACE) (OMB # 0970-0397) – Proposed Experiment to Examine Change to 15-Month Survey Incentives for Year Up

Background

The initial OMB approval of the Pathways for Advancing Careers and Education (PACE)¹ 15-month follow-up data collection was obtained in August 2013 and we started administering the survey in December 2013. Overall, the survey is going well, but participants in one program—Year Up—have a relatively low overall response rate, and the difference between the response of treatment group and control group members is large. Sample is released for the survey in monthly cohorts, at the beginning of the month after the study participant reaches their 15^th month anniversary since random assignment. Year Up has eight different sites in the study and the sample released so far is primarily from one site, the National Capital Region (NCR).

At this time, we are requesting approval for a small experiment to increase the incentive payment from $30 to $40 for control group members at the Year Up NCR site. The effect of the increased incentive on response rates will inform any decision to request a broader approval for an increased incentive for the first follow-up survey (15-month survey) and will inform our preparation for the second follow-up survey scheduled to start in February 2015. The proposed incentive for the second follow-up survey is $40 for all treatments and controls.

Statement of the Problem

The PACE survey data collection is progressing well in most sites. The main exception is the Year Up NCR site. The research team has identified two reasons why the Year Up participants may respond differently than those in other sites. First, the Year Up site targets a younger population than most other study sites and appear to be more mobile and more likely to break survey appointments. Second, Year Up has a more intensive enrollment process than other sites. This may make participants assigned to the control group less enthusiastic about participating in a survey for a study of Year Up given that they invested substantial time meeting the application requirements but were not chosen to be in the program.

The table below shows the response rate for the Year Up NCR site. The response rates are shown by month and year of random assignment, by intervention group and overall. The table was created from response rate data collected by Abt SRBI, the survey firm for the PACE data collection effort. As shown, in the table, the response rate for treatment group members is 63 percent and for control group members is 39 percent, a 23 percentage point differential.² The entire PACE sample has a T-C response rate differential of only 8 percentage points.

We have two main concerns about the differential response rate in Year Up. First, while PACE has robust procedures for analyzing non-response bias and making adjustments that will mitigate any potential non-response bias, a large differential in response rates will affect the face validity of the results and may lead to third party review entities (e.g., What Works Clearinghouse) to judge the findings unfavorably. Second, we are concerned about precision of our estimates and loss of power from a reduced control sample. Year Up is the only PACE site with a T-C random assignment ratio that is not 1. In Year Up, two study participants are assigned to the treatment group for every participant assigned to the control group. This means that is more costly to have a lower response rate for controls in Year Up than at the other sites, because the power of statistical tests is increased more for each control that is part of the survey data analysis than for each treatment.

Proposed Changes and Rationale

We propose to increase the incentive payment for Year Up NCR control group members from $30 to $40 for the 42 non-respondent controls in the field now and for the 159 NCR controls that will be released to the field in December 2014 (n=153) and January 2015 (n=6). The goal of the change is to increase the percent of controls that respond to the survey and to learn whether this is a change to consider for any other Year Up sites that have the same pattern as NCR after they have been released to the field. It will also inform the plans for the second follow-up survey, which has planned on a $40 incentive for both treatment and control group members. By increasing the incentive for non-respondent controls already in the field for various amounts of time, we will obtain information on whether the incentive results in responses from these proven hard to reach participants. By increasing the incentive for the cohorts just being released, we will be able to test the full effect of increasing the incentive on the control response rate.

We are requesting this change because of the low response rate of controls in Year Up NCR and particularly the treatment-control differential in response rates. As mentioned earlier, our concern about the differential response rate is due to third party entities that will judge the validity of the T-C impact estimates based on response rates before any non-response testing and adjustments are done and because of concern about losing power because only one-third of Year Up study participants were assigned to the control group.

We have tried many other adjustments for Year Up NCR that have not substantially reduced the response rate differential. These efforts include an extended field period (27 weeks so far for the Year Up cohort randomly assigned in January 2013); increased effort to locate and interview respondents by hiring an additional field interviewer for NCR, confirming appointments through texting (recommended by Year Up program staff); experimenting with additional locating efforts for a sample of 25 cases; including a more labor intensive manual look-up of contact information using our vendor’s databases; and going back to the site to obtain any updated or different contact information that we did not collect at baseline.

We believe this is the appropriate time to test an increased incentive. The Treatment-Control differential exists even for the sample that has been in the field an extended time, for which we have made intense efforts to locate and survey. Participants from the other seven Year Up sites are starting to be released and, if we observe the same pattern with other Year Up sites, we want to understand whether increasing the incentive by $10 will increase the control response rate by a non-trivial amount. In total, the eight Year Up sites have 2,545 study participants, approximately 850 of which are controls.

1 PACE was initially identified as Innovative Strategies for Increasing Self-Sufficiency.

2 Note that the Year Up NCR sample is still being worked, including some cases that were just released to the field a few ago, so these are not the final response rates for the sample released so far.

2