u. s. dEPARTMENT OF eDUCATION
institute of education sciences
National Center for Education Evaluation and regional assistance
Q1. When will the grantee survey, follow-up survey of student, school record abstractions and case study protocols be submitted?
A1. We plan to submit these protocols to OMB in March 2008.
Q2. To date, what percentage of selected students have opted not to participate?
A2. Six percent of the over 900 eligible applicants who have entered admissions lotteries have opted not to participate in the evaluation.
Q3. Given the legislative push to dismantle this evaluation, please give us an update on how implementation of the study design has fared. Have grantees faithfully implemented the RCT protocols? Or is there a sense that they are waiting to see how and if Congress intervenes?
A3. As of 9/28/07, we had conducted random assignment of students applying to Upward Bound at 24 of the 101 host institutions included in the study, which are primarily programs that brought in new students over the summer, and had enrolled about one-quarter of our target sample of students for the evaluation. We expect to enroll three-quarters of our target sample of 3,600 students by the end of December 2007. Only two of the host institutions have not cooperated fully with data collection since their IRBs have expressed reservations about the consent forms, exemption from multi-site IRB review, and possibly negative consequences for control group students. We have responded to each of these concerns. Most UB projects are cooperating with the evaluation because such cooperation is a condition of their FY 2007 grants as long as the Absolute Priority remains in effect.
Q4. How can the evaluation be salvaged if the evaluation prohibitions in HR2669 are enacted? What about the FY2008 House appropriation report language?
A4. If the Absolute Priority is overturned, we expect most grantees to cease cooperation with random assignment and begin to admit preferred control group students to fill openings in the program. The Office of Postsecondary Education has not decided whether to continue to fund the evaluation under such circumstances, and a rigorous examination of impacts would not be possible in any case. Without FY 2008 funding, we expect the evaluation to run out of money in March 2008. Our priority before then is to include as many eligible students as possible in baseline data collection and random assignment.
Q5. The document repeatedly notes that students with be followed for "about 24 months". Are students going to be administered the follow up survey at a specific period of time (the end of 12th grade) or after a specified period of time? Is 24 months a sufficient period of time to detect an impact on outcomes of interest? Will the follow up survey be implemented before or after a student graduates from high school?
A5. The first follow-up survey of students will occur during fall 2009--about two years after random assignment for most students, some of whom will be starting Grade 11, and others of whom will be starting Grade 12. This amount of time will be sufficient to detect impacts on services received, test scores, grades, course taking, and college expectations, but not high school completion or college attendance. Once students turn age 18, we will request student consent for collection of data on postsecondary outcomes. Data collection on postsecondary outcomes would occur between 2013 and 2016 under a separate contract since we are prohibited from having contracts with a period of performance longer than 5 years.
Q6. On page 5, the submission discusses the research questions. We have the following comments:
Question 2: What strategies and approaches do grantees follow to promote academic success for higher-risk students? Shouldn't the evaluation also look at the strategies and approaches that are effective with the overall participating student population and whether or not the effective strategies differ depending on the population served?
Question 3: Does the impact of Upward Bound vary according to grantee practices and characteristics, including grantees' previous experience with the Upward Bound Participant Expansion Initiative? Shouldn't the evaluation also look at whether the impact varies according to student characteristics?
A6 . We plan to study all of these issues by analyzing how impacts vary with student characteristics, with project characteristics, and with the intersection of the two.
Q7. Given the number of estimates planned for calculation, there seems to be a high risk of false positives-especially when looking at the relationship between strategies and impacts. How will the evaluation report address any positive estimates for subgroups?
A7. The issue of multiple comparisons is being addressed on many IES evaluations. We plan to address the possibility of false positives by (a) using the Benjamini-Hochberg procedure to identify which effects are statistically significant after accounting for the number of comparisons and controlling the overall false discovery rate within each outcome domain to be 0.05; (b) estimating impacts on standardized average outcomes within each domain; and (c) performing omnibus tests to see if we can reject the hypothesis that all impacts within a domain are zero.
Q8. On pg. 20 the submission indicates that “in developing hypotheses about why specific practices or groups of practices or structures might lead to better impacts, we will rely first on the theories of program stakeholders. In simplest terms, following the clearance of evaluation findings for public release, we will inform a group of stakeholders of the results of the impact analysis and ask them whether they agree with, or are surprised by, the findings regarding program characteristics associated with better impacts. In either case, we will ask informants why they believe as they do.” Do you plan to follow up on the stakeholder discussion with an analysis of how these results compare to existing research on programs with similar goals or serving similar populations?
A8. The soliciting of reactions from stakeholders was envisioned as occurring in the context of case studies of the implementation of Upward Bound practices associated with positive impacts on students. Before we submit case study protocols for OMB review in May 2008, we will refine our approach for identifying hypotheses regarding these estimated effects. Obviously previous research on similar programs is another source of hypotheses that could explain why certain UB practices are associated with positive impacts.
Q9. Also on page 20, when looking at the experiences of the control group, will the evaluation collect any data on the sources of funding/ support for activities similar to those supported by Upward Bound that were available to the control group?
A9. We can identify funding/support for the Talent Search programs offered at half of the host institutions in the study. While we will collect data from grantees on other services available in the community, and from students on non-Upward Bound services received, given the complexity of funding for community-based services (federal, state, local, foundation, in-kind) we do not expect to obtain detailed funding/support information for all of these non-UB programs.
Q10. Please cite the IES confidentiality statute on all instruments rather than just promising confidentiality.
A10. The IES confidentiality statue is mentioned explicitly on both the parental consent and student assent forms before the respondent is asked to sign: In accordance with the Privacy Act (5 U.S.C. Section 552a) and the Institute of Education Sciences confidentiality statute (20 U.S.C. 9573, Section 183), the information gathered on [you] your child will be kept strictly confidential and will be used by the Department and its contractors ONLY for the purpose of the study, except if required by law. For this reason, we do not see why an additional citation is required on any of the forms.
| File Type | application/msword |
| Last Modified By | jonathan.jacobson |
| File Modified | 2007-10-09 |
| File Created | 2007-10-04 |