ED Response to OMB Comments

Memorandum

To: Joelle Lastica, Institute of Education Sciences

From: R. Dean Gerdeman, Director, REL Midwest

Jim Lindsay, Senior Researcher, REL Midwest

Date: April 8, 2014

Re: Response to questions from OMB regarding ICR for Ramp-Up Implementation Study.

This memo contains REL Midwest’s responses to OMB’s questions (see italicized colored text following each question).

1. Do the later-implementing schools know that they will be getting resources to implement the Ramp-Up program at any point during the study? If so, could there be possible bias in how these schools describe their existing college-readiness activities since they have self-selected to receive something else?

RELMW: The later-implementing schools know that implementation within their school is being delayed by one year. The choice of which schools would implement Ramp-Up earlier versus later was made by the program developer, through a random process. The developer—The College Readiness Consortium—framed the school selection process as either implement Ramp-Up early or continue their present college readiness activities for another year at which point they would have the chance to implement Ramp-Up. In sum, the random assignment of schools to conditions should mitigate any bias in responding among schools.

2. FRPL is not a particularly reliable indicator of socio-economic status in high school. Will there be any other comparisons in measures related to college readiness for other subgroups of students such as by race/ethnicity? If no, why not?

RELMW: We did not propose examining subgroups of students by race/ethnicity because Minnesota schools in general serve predominantly white populations of students (per the latest data on Minnesota Department of Education’s website, 91 percent of the students in the sampled schools are Caucasian). Students of different racial/ethnic groups tend to concentrate in specific schools or counties (i.e., Native American populations in North Central region of the state, African American and Hmong—or Asian—in the Twin Cities). Thus race/ethnicity is confounded with school, making such a comparison less informative.

In addition, this project is being performed in collaboration with a group of stakeholders from different states who all have an interest in using research to improve the college and career success of students. Perhaps it was their knowledge of Minnesota that influenced their preference for the research question to focus on student poverty status, represented by FRPL—as unreliable as it is—for subgroup analysis.

We will acknowledge the potential reliability concerns surrounding FRPL in the “limitations” section of our report.

3. Is there any planned follow up for an impact evaluation of the Ramp-Up program?

RELMW: Much as we would love to track the students into their post-secondary lives, we do not plan any longer-term follow-up with the sample of 22 schools or the students from these schools who may or may not continue on into post-secondary education. There are multiple reasons for why no follow-up is planned with this sample:

1. A sample of 22 schools is insufficient for detecting impact of this intervention;

2. The REL contract is time-limited, with an expectation that reports be completed by December 31, 2016. Another wave of data collection would make that deadline difficult to meet;

3. At present in Minnesota, attempting to track students from secondary level to post-secondary means asking for social security number, which might influence more parents to opt their child out of the implementation study. We did not want to face the possibility of sacrificing students from the implementation study for the chance to obtain a poorly-powered impact estimate.

However, REL Midwest has proposed another study of Ramp-Up’s impact, this one with a larger sample of schools which is completely independent of the implementation study sample.

4. A comment on language: throughout the Supporting Statement, the study team describes the results of this study as “evidence”. OMB is very sensitive to how “evidence” is used, and in OMB’s view, evidence is only produced by studies with rigorous methodological designs that attempt to measure causal impact. OMB would prefer that the study team not use “evidence” in reference to this study since OMB views it as a mischaracterization of the information coming out it.

RELMW: We had made changes in Supporting Statement A and Supporting Statement B so that the term “evidence” is used only in reference to findings produced from studies that meet What Works Clearinghouses group design standards or studies that meet What Works Clearinghouses group design standards, with reservations.

REL Midwest Memorandum on RELMW Responses to OMB—1