passback

Contract No.: ED-01-CO-0039/0010

MPR Reference No.: 6136-600

The Supporting Justification

For OMB Clearance of Information Collection Forms for the Evaluation of Reading Comprehension Interventions

Response to Questions

October, 2007

Submitted to: U.S. Department of Education Institute of Education Sciences 555 New Jersey Ave., NW, Rm. 308 Washington, DC 20208 (202) 208-7078 Project Officer:  Audrey Pendleton

Submitted by: Mathematica Policy Research, Inc. 600 Maryland Ave., SW, Suite 550 Washington, DC 20024-2512 (202) 484-9220 Project Director: Jerry West, Ph.D. Deputy Project Director: Wendy Mansfield, Ph.D.

Question 1

If a large number/percentage of the original schools do not agree to participate in the second year, how will you proceed? Is there a threshold beyond which too many schools have dropped out?

response

For both study components,¹ we recommend having at least 70 percent of original schools in the study sample for each intervention group (and for the control group) in the second year. This threshold of 70 percent is consistent with What Works Clearinghouse guidelines. If the school participation rate falls below 70 percent for a given intervention group, then we advise against calculating impacts for that group or including it in the analysis. If the participation rate falls below 70 percent for the control group, we suggest excluding the control group from the analysis (that is, we would not calculate experimental impacts by comparing an intervention group to the control group, though we would still compare the intervention groups to one another).

There are two reasons for excluding the intervention group from analysis if the participation rate is less than 70 percent. First, a substantially different participation rate may be a result of self-selection on the part of schools—that is, the schools that remain in the intervention group may no longer be comparable to schools in the control group or in the other intervention groups. Second, the loss of statistical power for that intervention group reduces the probability of detecting meaningful impacts of the intervention.

We do not have information on the fraction of schools that might participate in the first component of the study. We do, however, have some preliminary information (received from the school districts) on the share of schools that might participate in the second component. Based on that information, we believe that only one intervention group—Reading for Knowledge—might not have enough schools to support an analysis of impacts (see Table 1). If, after contacting schools, we find that we cannot achieve a participation rate of at least 70 percent (i.e., 11 schools) for this group, we will not calculate impacts for that group.

TABLE 1

COMPONENT TWO: LIKELY PARTICIPATION RATES BY TREATMENT GROUP

Treatment Group	Schools			Districts
	Number Eligible for Component 2	Number Likely to Participate in Year 2	Percentage	Number Eligible for Component 2	Number Likely to Participate in Year 2	Percentage
Control group	21	19	90	10	10	100
Project CRISS	17	15	88	10	10	100
Read About	17	15	88	10	9	90
Read for Real	15	13	87	9	7	78
Reading for Knowledge	16	8	50	9	6	67
Total	86	70	81	N.A.	N.A.	N.A.

Question 2

Will the evaluation look at whether teachers who left the original schools transferred into control schools?

response

Yes, we will examine whether Year 1 treatment teachers transferred into control schools in Year 2. We will do so by using the information obtained through the recruitment of Year 1 schools to participate in Year 2 of the evaluation. Once schools agree to participate, we will send a list of Year 1 teachers to the school principals requesting that they (1) indicate which teachers will be returning in Year 2, (2) update the contact information for those teachers, and
(3) add the names and contact information of any new fifth-grade teachers. Then, when our local field staff contact the schools to initiate student testing, we will reconfirm the names of the fifth-grade teachers and collect transfer information on those teachers who are no longer at the school. This will allow us to track teachers who have moved from one study school to another.

Question 3

In component 1, approximately what percent of students are likely to transfer to other schools? When assessing these transfer students, will ED be "measuring the school environment" in the schools to which these students transfer, even if those schools were not originally part of the study? If so, what are the consent procedures that will be followed?

response

Of the schools that participated in year 1 of the study, 63 percent end at grade five (see Table 2). We therefore expect that at least 60 percent of the students in year 1 will have graduated to a middle (or “feeder”) school for grade 6. Based on other education studies we have conducted, we estimate that another 10 percent of year 1 fifth graders will have transferred to other schools (“transfer” schools) in year 2.

If the feeder schools agree to participate in component 1, we will assess the year 1 students attending these schools in year 2. We will also abstract school-level data from the Common Core of Data (CCD) and SchoolMatters to measure the environment of those schools. We will not assess year 1 students now attending transfer schools if—as expected—the schools have only one or two of these students. Nor will we measure the environment at transfer schools in which we are not assessing students.

With regard to consent procedures, the parent letters disseminated in year 1 requested consent for both the 2006-2007 and 2007-2008 school years (see the attached letter). However, we will resend consent letters to parents if any schools ask us to do so.

TABLE 2

PERCENTAGE OF STUDENTS LIKELY TO TRANSFER TO NEW SCHOOLS IN COMPONENT 1

Year 1 Schools	Number of Schools	Percent	Number of Students	Percent
High grade 5	56	63	3,695	61
High grade 6 or more	33	37	2,397	39
All year 1 schools	89	100	6,092	100

question 4

In component 2, in the evaluation of impact of 5th graders taught in schools with a full year of experience with the intervention, will the new teachers also be trained in the intervention? What is the rationale behind including both original and new teachers in this assessment?

response

Any new fifth-grade teachers in intervention schools participating in component 2 will be trained just as the year 1 teachers were trained. The rationale for including both original and new teachers in component 2 is to answer two research questions. Focusing on the original cohort of teachers will allow us to answer the question of whether interventions are more effective after teachers have had a year of experience with them. Only teachers who participated in year 1 will be included in this analysis. Focusing on all teachers (i.e., both original and new) will allow us to answer the question of whether interventions are more effective after schools have had a year of experience with them.

Question 5

Please let us know how the data collection went last year. Does ED have any preliminary results on the impact evaluation?

response

Data collection in year 1 was successful; we achieved response rates of 89 percent or higher for the student assessments, teacher surveys, classroom observations, and school information forms (see Table 3). To date, we have obtained student records from 8 of the 10 districts; we are still collecting student records from the remaining 2 districts. We are now cleaning the data files and do not have any preliminary results to report at this time.

TABLE 3

PRELIMINARY RESPONSE RATES FOR YEAR 1 DATA COLLECTION INSTRUMENTS

Data Collection	Response Ratea
Teacher survey	97
School information form	92
Student baseline tests	99
Classroom observations	98
Student follow-up tests	89

^aThese response rates are preliminary, as the data are currently being cleaned.

1 Schools originally involved in random assignment could be missing from either component of the second year study. Schools that were originally randomly assigned would not be represented in the first component if none of the students from those schools are present in the sample for the second year. This could happen if, for example, none of the middle schools that are fed by an original school agree to participate in the study or if one of the participating elementary schools that includes sixth-graders refuses to participate. An original school could be missing from the second component of the study if that school refuses to participate in the study for a second year.