Emergency memo

u. s. dEPARTMENT OF eDUCATION

institute of education sciences

National Center for Education Evaluation and regional assistance

to: Rachel Potter, Brian Harris-Kojetin

from: Jonathan Jacobson and Marsha Silverberg

THRU: Kathy Axt

subject: Request for Expedited Review: Forms Clearance Package for the Impact Evaluation of Upward Bound’s Increased Focus on Higher-Risk Students

date: 11/2/2006

We are writing to request expedited review for a clearance package that we plan to submit to OMB in November 2006. This memo provides background on the program we are evaluating and summarizes the reasons for the request and our proposed schedule, should OMB grant our request. We ask for a response as soon as possible.

1. Background

The Upward Bound Program

Upward Bound, initiated under Title IV of the Higher Education Act of 1965, is a federal pre-college program designed to help economically disadvantaged students prepare for, enter and succeed in college. Funding for 800 Upward Bound grantees equaled $278 million in FY 2006.

Upward Bound was one of the first education programs evaluated through a large-scale, multi-site, random assignment evaluation. This evaluation, which began in 1991, was conducted using a sample of 67 Upward Bound grantees. Approximately 2,800 eligible students applying to Upward Bound programs went through random assignment between 1992 and 1994, with 1,500 assigned to the treatment (Upward Bound) and 1,300 to a control group. A series of reports have been released describing findings from the evaluation. The most recent, “The Impact of Regular Upward Bound: Results from the Third Follow-up Data Collection” (Myers et al. 2004, on-line at http://www.ed.gov/rschstat/eval/highered/upward/upward-3rd-report.html) found no statistically significant evidence of impacts of Upward Bound on overall postsecondary attendance rates or college credits earned. At the same time, the evaluation found evidence that certain subgroups of students with lower college expectations experienced positive impacts from Upward Bound.

Pointing to the findings from the recent evaluation, OMB has rated the Upward Bound program a “ineffective” and indicated a need for the program to be better targeted on higher-risk students (http://www.whitehouse.gov/omb/expectmore/summary.10000210.2005.html). Responding to OMB’s assessment, the Office of Postsecondary Education (OPE) has introduced significant changes to Upward Bound for the FY 2007 grant competition. To be funded in 2007, Upward Bound grantees must serve new students starting in Grade 9 or 10, and 30 percent of the new students served must be higher-risk 9^th graders, as indicated by low GPA, low test scores, or failure to take pre-algebra or higher in Grade 8 and algebra or higher in Grade 9 (http://www.ed.gov/legislation/FedRegister/finrule/2006-3/092206b.html).

The New Upward Bound Evaluation

In addition to encouraging OPE to modify the program, OMB has asked ED to initiate a new evaluation to assess the impact of the redesigned program. In response and working with IES, OPE is requiring grantees to participate, if selected, in a new national evaluation of Upward Bound. It is important to note that many Upward Bound grant recipients are unhappy with both the changes in program emphasis and the conduct of a new evaluation. Our ability to conduct the evaluation will depend to a large extent on ED’s ability to hold fast to the requirements and to minimize the burden on grantees.

The new evaluation, like its predecessor, will involve the identification of a representative sample of grantees and the random assignment of eligible new students to treatment or control groups. The scale of the new evaluation is sufficiently large (90 grantees, 1,800 treatment students and 1,800 control students) to permit the detection of impacts on the higher-risk subgroup as well as on eligible students as a whole.

Information about the evaluation was included in the grant notice. Grantees selected for the evaluation must recruit approximately twice as many eligible students as the grantee has openings. Several steps must occur prior to random assignment:

• Each grantee must determine the eligibility and higher-risk status of applicants

• Evaluation consent forms (from both parents/guardians and students), student baseline information forms, and student selection forms must be completed and turned in, in addition to the grantees’ own application materials.

2. Rationale for Request: Challenge of Completing Baseline/Random Assignment Activities

ED finds itself in a difficult position that threatens our ability to implement the evaluation. We had intended to award an evaluation contract in June 2006 but ED’s contract office fell behind and the contract was not awarded until September 15, 2006. Moreover, recent concerns about potential burden and disruption raised by the grantee community have put a greater responsibility on the evaluation to ensure early grantee sampling and coordination of baseline data collection with each grantee’s application process. There are two main challenges:

• Because their probability of selection for the evaluation is relatively low (about 12%) and doing so would interfere with their regular procedures, many grantees will ignore the requirements to recruit twice as many applicants and to defer offering admission to students until after random assignment. It will therefore be a substantial problem for the evaluation if grantees start wrapping up their recruitment before the evaluation can notify grantees of their selection and need to “over-recruit.”

• Grantees already have an extensive application process. Having to go back to students who completed the grantees’ applications and ask them and their parents/guardians to fill out baseline information forms and consent forms will be a burden on grantees, students, and their families and potentially disrupt the application process.

Many Upward Bound grantees enroll new students in summer programs and these grantees will start screening applicants for eligibility several months earlier, even before new Upward Bound grants are announced in April 2007. If the evaluation has to go through a full 120- to 150-day OMB clearance process, our notification to and work with grantees on data collection could not begin until late March or early April 2007, just as many grantees are completing their application periods. This schedule would likely mean that summer programs would need to be delayed or perhaps cancelled for new students, creating hostility toward the evaluation, undermining the validity of the evaluation and harming its ability to draw conclusions about the effectiveness of the redesigned Upward Bound program.

To avoid this risk, IES is requesting that OMB waive the usual 60-day comment period for the package requesting approval of the sampling of grantees and the collection of parental and student consent forms, baseline information forms, and student selection forms. If the plan for sampling and baseline data collection were subject to a 30-day public comment period only (followed by 30 days of review by OMB), then contacting grantees sampled to be in the evaluation could occur in late January or early February 2007, allowing more time for the evaluator to work with grantees selected for the evaluation (contingent on their receipt of funding for 2007), ensure that a sufficient number of eligible students are recruited to form a control group, and obtain the necessary consent and baseline student data required before random assignment can occur.¹

3. Proposed Timeline

November 28, 2006	30-day notice posted for emergency data collection
December 29, 2006 – January 29, 2007	OMB review and approval of emergency data collection
January 2, 2007	First Federal Register notice posted for regular OMB clearance
January 30, 2007	Contractor begins to contact sampled grantees and distribute consent forms and baseline information forms to students and their parents/guardians, and student selection forms to grantees, integrating these forms with grantees’ own application materials and processes where possible
March 9, 2007	Second Federal Register notice posted for regular OMB clearance
April 2007	2007 Upward Bound grants announced
April 2007 – May 2007	Random assignment of eligible students to treatment or control groups following completion of consent forms, baseline information forms, and student selection forms, with priority to random assignment of students applying to Upward Bound projects with summer programs
April 10, 2007 – May 10, 2007	OMB review and approval (under regular clearance procedures) of further baseline data collection
June 2007	Upward Bound summer programs begin for new students
June 2007 – August 2007	Completion of consent process and baseline data collection and random assignment for eligible students at remaining Upward Bound projects (all without summer programs for new students)
September 2007	Upward Bound fall programs begin for new students

Thank you for your consideration of our request.

1 Since this emergency clearance would only last for six months, and baseline data collection and random assignment could continue over a longer period for some grantees, ED would submit data collection plans for an additional 60-day public comment period in order to obtain approval for baseline data collection over a longer period of time.

4