Ed Response to Third Round of OMB Questions

Response to OMB’s Questions of July 27, 2009

Evaluation of Secondary Math Teachers from Two Highly Selective Routes to Alternative Certification

Thank you for your quick response to our memo of July 24, 2009 regarding 200904-1850-004. Below, we present our answers to your questions of July 27, 2009.

1. To be clear, are all students in the study receiving $5 for completing the evaluation? During our conference call, OMB initially raised the concern that 3 incentives (teacher, consent form, evaluation completion) were considered too many.

Less than one third of all students in the study will receive the three incentives. The class incentive (given to the teacher to spend on the class) and student consent form incentives will only be offered in districts that require active consent. The assessment completion incentive will only be given to students who participate in the assessment and we are only administering the assessment to high school students. Table 1 shows the expected percentage of students who would be offered each combination of incentives (in the absence of an experiment).

Table 1: Percentage of Students in Study Who Would be Offered Each Incentive

Incentives	Estimated Percentage of Students
Class, consent form, and assessment incentives	31%
Class and consent form incentives only	27%
Assessment incentives only	20%
No incentive	21%
Total	100%

2. While OMB appreciates the proposed experiment in Section D of your response, it is unclear to us which of the three groups would be the control group. We recommend using three groups, but having one of those three groups receive no teacher incentive and no consent form incentive. The third group could receive whichever incentive (teacher or consent form) IES estimates to be more effective.

While we would prefer an experiment in which all three groups receive some incentive, we are willing to do an experiment in which there is a control group in which the students receive neither a class nor a consent form incentive. This experiment would randomly assign schools to one of three groups:

1. Treatment 1: Class receives a $25 incentive if 95 percent or more of the consent forms are returned and individual students are offered $5 if they return the consent form.

2. Treatment 2: There is no class incentive; individual students are offered $5 if they return the consent form.

3. Control: There is neither a class incentive nor a student financial incentive for returning the form.

3. If IES proposes to pursue the experiment we suggest above, we also request an estimate of power for that experiment.

Table 2 presents the minimum detectable difference (MDD) in the rate at which consent forms are returned for the incentive experiment described above. If we assume an intra-class correlation (ICC) of 0.10, the MDD is 14 percentage points. Based on the evidence that we assembled in our memo of July 24, 2009 (see Table 1 in that memo), we expect that the difference in the consent form response rate between Treatment 1 and Control is likely to be about 40 percentage points (see Thompson 1984). The differences in the form return rate between Treatment 2 and Control and between Treatment 1 and Treatment 2 are likely to be smaller but still much larger than 14 percentage points.

Table 2. The Minimum Detectable Differences in the Rates at Which Consent Forms are Returned for the Incentive Experiment

Districts	Number of Schools	Number of Students per School	ICC	Expected Return Rate with No Incentive	MDD
All active consent	65	160	0.10	0.42	0.14
Note: We assume clustering at school level.

2