ED Response to OMB Comments

Memorandum

To: Chris Boccanfuso, REL COR

From: Dean Gerdeman, REL Midwest Director; Chris Brandt, REL Midwest Deputy Director; Ann-Marie Faria; Nicholas Sorensen; Mindee O’Cummings; Jessica Heppen

Date: 2/20/2014

Re: RE: ICR Reference Number 201311-1850-002 Evaluation of the Early Warning and Intervention Monitoring System—RMO and OIRA Comments

Overview of Memo

The purpose of this memo is to further specify the responses to the questions posed by RMO and OIRA in response to RE: ICR Reference Number 201311-1850-002 Evaluation of the Early Warning and Intervention Monitoring System (EWIMS). In an email dated February 11, 2014, OMB posits: “The response to question 6 -- Education is very generous incentives. Do we really need to be so?” Following please find the original question (question 6) from OMB and our response to OMB’s follow-up question from February 11, 2014 in blue.

Original Question 6: What besides “support for post-study implementation costs” is proposed to be included in the incentive to control schools? Please provide an estimate of the value of the incentive per school.”

Follow-up Question: “The response to question 6 -- Education is very generous incentives. Do we really need to be so?”

The original incentive plan for the EWIMS Impact Study included two incentives for control schools: (1) a stipend for data transfers of student level data files and (2) a stipend for completing a web-based administrator survey. Although incentives are a key strategy for facilitating recruitment of high schools for experimental studies, maintaining a clean treatment contrast during the course of the study, and a safeguard to combat differential missingness in data collection; we propose decreases in the incentives included in the original ICR. Specifically, our revised plan proposes to reduce these incentives by about 65 percent, from $207,360 to $72,000.

• Original student administrative data collection stipend: $720 per data pull, per school, with 4 data transfers, for $2,880 per school and a Total Cost of $207,360

• Reduced student administrative data collection stipend: $250 per year per school, with 4 data transfers, for $1,000 per school and a Total Cost of $72,000.

Our revised plan reduces incentives by $135,360.

The financial incentives are critical for all study schools. Control schools do not receive the intervention until fall 2015. Thus, the incentives ensure that they are motivated to participate in the data collection during the evaluation. The incentives also motivate treatment schools because they are receiving the intervention and actively involved with the study team. Moreover, many of the treatment schools do not have trained data specialists to pull the data without support from the research team. Without adequate incentives for data collection, the contractor’s ability to gather critical data from schools is at risk. For instance, if higher than expected numbers of control schools fail to provide necessary extant data for the impact analyses, missing data (particularly differential missingness by treatment status) will bias the estimates. While we believe it is essential to offer some incentive for the data transfer of student records, we propose to now provide a once a year $500 school-level incentive for sharing student data. This is a reduction by $1,880 per school, for a total reduction of $135,360 for this study.

We maintain our intent to provide a $30 incentive for administrators to complete the Web-based survey in study schools. The $30 incentive will further ensure that the contractor is able to collect primary data for the study. The incentives for data collection were carefully calculated based on estimated labor burden and salary of the staff who will be involved in the data collection. The amount for the school administrators who participate in the annual Web-based survey or the on-site interviews was based on the burden level of the survey. For a high-burden survey, such as the proposed survey, ED recommends an incentive of no more than $30. Therefore, we offered a $30 stipend for these activities. We do not think that reducing the cost of this incentive is feasible, given the burden of the survey and instead suggest retaining this incentive for the administrators who complete the web-based survey.

REL Midwest Memorandum on ICR Reference Number 201311-1850-002

Evaluation of the Early Warning and Intervention Monitoring

System—Response to RMO and OIRA Comments—1