OMB Memo for Request for change in data collection methodology

u. s. dEPARTMENT OF eDUCATION

institute of education sciences

National Center for Education Evaluation and regional assistance

to: Shelly Martinez, Sharon Mar

from: Marsha Silverberg, Meredith Bachman

subject: Request for change in data collection methodology and incentives to improve response rates for the Evaluation of the DC Opportunity Scholarship Program (OMB# 1850-0800)

date: 07/15/2015

Background:

The Evaluation of the Opportunity Scholarship Program (OSP) is a congressionally-mandated study of the only federally-funded private school voucher program. As approved by OMB (NOA 4/17/13), the evaluation includes a sample of about 1,800 students who participated in lotteries (random assignment). The study will conduct annual testing and surveys with the student sample (in schools) and surveys of their parents and DC public and private school principals (both mixed mode). This request seeks to add a more intensive follow up effort with nonresponding parents and students. This requested methodology change will not impact the number of responses or burden hours for the evaluation.

Parent Survey: Status and Proposed Change

As of July, the response rates for the 2015 parent survey were low for parents in both the treatment and control groups (65% and 54% respectively). More than two-thirds of the sample are 2 or 3 years away from when they applied to the OSP and many non-respondents either did not receive an offer of a voucher or did receive a voucher but declined to use it; they no longer have a connection to the program or feel responsibility to participate in data collection.

In order to increase the response rates for both groups, subsamples were drawn from the set of nonrespondents after the initially planned follow up through email, telephone, and mail. Of the 308 households with at least one control group student for whom a parent survey had not been completed, 154 households were sampled (50%). Of the 190 households with one or more treatment group students for whom a parent survey had not been completed, 95 households were sampled (50%).

The evaluation team would like to use an intensive outreach approach that includes home visits, to survey just over 50% of subsampled control group parents and 30% of the subsampled treatment group parents to achieve weighted response rates of approximately 80% for both groups. This refusal conversion approach was successfully used during the first DC Choice evaluation. We are not asking for a change in the $20 incentive for parents already approved.

Student Testing: Status and Proposed Change

In part because of a dramatically shortened testing window this year, the percent of treatment and control group students tested in school was also low (73 percent and 58 percent, respectively).

Thirty-five of the untested students (19 control and 16 treatment) happen to be part of households that were subsampled for the parent survey conversion effort and have the potential to be included in that activity. These are students who were not tested because they were not at school during any of the three testing days or because their school was never scheduled for testing (see table below).

This proposed change is opportunistic and low cost, for 2 reasons: (1) outreach workers will be visiting these homes anyway, and (2) there are $20 gift cards left over from the previous evaluation of the program (completed in 2011) that could be used for an incentive. Because the students targeted are mostly those not regularly attending school, it is unlikely they will agree to a 1.5 hour assessment and 15-minute survey without an offered incentive. We believe $20 is appropriate in this situation.

Reason not tested	Number of students
	Subsampled students	Students not subsampled
Graduated	2	2
Home School	0	2
Moved	15	50
Parent Refusal	4	11
Student untraceable	66	164
Student Refusal	0	12
IEP Not offered	11	22
Student Not Available	30	102
School Not Scheduled	5	13
Scheduled (KIPP)	12	50

Follow up with OMB

We will report back to OMB in the early fall, when data collection is complete. At that time, we will document the results of the refusal conversion efforts and any proposal to continue or modify it beyond this year. In doing so, we will consider more sophisticated, adaptive methods of subsampling targeting and conversion. At OMB’s suggestion, we will consult with NCES regarding possible options for 2016.

3