Responses to OMB Comments

1850-0858: Feasibility and Conduct of an Impact Evaluation of Title I Supplemental Education Services - Responses to OMB questions (6/4/09 FINAL)

1. Please add some detail to page 1 about the number of students in the sample from the 8 districts.

We have added the following footnote on the first page to describe the sample we anticipate having at this point: “The eight districts have a total of approximately 55,000 eligible SES applicants in 2008-09, with approximately 80% of the sample from Florida.” Not all of these 55,000 students will ultimately be included in the analysis sample, however, for several reasons. Some of the districts have stratified prioritization schemes that will rule some students out of the analysis; some of the districts may not be able to provide outcome data on all of the students; and the analysis will need to exclude applicants with very high and very low prior achievement levels. We won’t know the exact sample size until we have all of the data in hand from the districts.

2. Please provide some information about how NCEE can be sure that recruiting from up to 10 more school [districts] will get to the 50,000 desired sample.

At this time, we anticipate having an adequate sample size for the analyses to be sufficiently powered, although the analysis sample (as described in response to question 1) may be somewhat less than 55,000. The primary reason we plan to recruit additional districts is to increase the geographic diversity included in this study as more than half the currently recruited districts (5 of the 8) and approximately 80% of the student sample are from Florida. We feel that including 4 more non-Florida districts will sufficiently address the concern that so much of the sample is currently from Florida.

3. Has NCEE reached out to these 10 additional districts yet? How likely are they to participate?

Recruiting efforts in the first year suggest that most districts that are eligible will in fact participate (8 of 9 that were eligible in the first year agreed to participate). To be eligible for the study, a district must have an oversubscription for SES (i.e., the district is not able to serve all eligible applicants because of funding constraints), and it must offer services to eligible applicants on a basis consistent with the study design. We anticipate that additional districts with oversubscription will be identified in the fall of 2009-2010. The 10 districts referred to in Table A.1 are districts with potential for oversubscription in 2009-2010, based on information gathered during recruitment this year. It is not possible to know with certainty how many more districts will be eligible until the fall, after schools’ eligibility to offer services is determined and applications from eligible students are received by districts. We anticipate contacting in fall 2009 all potentially eligible districts that are not already participating and that are not in Florida. In recent years, there is evidence that the number of districts providing SES and the number of students receiving SES has been increasing. For example, in 2004-2005 approximately 1,000 of 14,000 districts were required to offer SES to 430,000 students nationwide. This number rose to 529,627 students in 2006-2007. No national data are available on trends in the number of districts oversubscribed for services, but this number has inevitably increased as the number of participating students increases while funding limits remain capped.

4. What were the barriers to recruitment in the first round? Why did 4 of the districts decline to participate?

Recruitment for this study requires that districts are oversubscribed and have an SES service allocation procedure based on quantifiable measures of prior student achievement. The main barrier to recruitment is a lack of oversubscription. Of the 30 originally targeted districts, eight are participating in the study and 22 are not. Of the 22 not participating, 17 reported not being oversubscribed for the 2008-2009 school year, 3 don’t meet the second criteria for participation (that they must use a quantifiable prioritization scheme), one district with minimal oversubscription ultimately would not cooperate with the evaluation effort, and only one district refused to participate outright before we could determine eligibility. Compared to many education studies, this study involves relatively little burden on districts, and we are confident that if there are additional districts that meet the eligibility requirements for the study, we can gain their participation. If we are unable to obtain additional sample, we still feel that the study is sufficiently powered and worthwhile, albeit with findings that are heavily concentrated in Florida districts. The concentration in Florida districts is due in part to the fact that SES participation rates are substantially higher there than in most states, but we anticipate that will not continue to be the case, given changes in non-regulatory guidance to states. Hence we, along with the program office, feel that it is worth expanding the sample if possible.

5. Regarding the burden “included” in the prior submission, please add some burden to this ICR to account for the recruitment of the 4 additional districts (i.e., any initial contact with the districts to establish whether they want to participate).

We have added an estimated 54 burden hours to recruit the additional 4 districts.

6. The prior submission did not contain a request for incentives. Please explain why they are necessary now. Additionally, please explain the extent to which the Upward Bound model is relevant in this case.

The original OMB submission was limited to district-level recruiting efforts for which incentives were not needed. The current request includes outcome data collection activities for this study, which includes the SES Provider Survey for which we are requesting approval for incentives. SES providers are the only source for gathering information on the characteristics and types of services provided by SES provider organizations in each district, but there are no built-in incentives for these SES providers to cooperate with our data collection efforts. Without this information the evaluation will only be able to provide a “black-box” answer about the effectiveness of SES services. A specific contribution of this SES evaluation will be to provide descriptive information about SES and to examine whether specific SES provider characteristics and services are related to impacts on student achievement (research question #2). SES providers are considered hard-to-reach respondents as they are non-district and non-school employees. These district-funded service providers do not receive funds directly from the U.S. Department of Education, so there is no condition, such as the EDGAR requirements, for SES providers to cooperate. We cited the Upward Bound experience to illustrate that, for a similar hard-to-reach group, it was necessary to not only offer an incentive, but to increase the amount when response rates fell short of expectations. We think that the SES providers are likely to be as difficult if not more difficult a group to get to respond (at least the Upward Bound grantees received funding directly from ED).

7. We note that NCEE plans to calculate provider-specific impacts. Could this be part of the incentive to encourage the providers to participate in the study?

While we appreciate your suggestion to share provider-specific analyses with each provider organization, we do not think it would be possible, nor do we think it would be a sufficient incentive to obtain acceptable response rates. Small sample sizes will preclude the estimation of precise impacts for all but the very largest providers, of which there are just a handful. Thus, this is not an incentive for all survey respondents. There is also the concern that results shared with providers may be misinterpreted as being statistically significant or meaningful since in most cases provider-specific analyses will likely not be sufficiently powered to produce reliable provider-specific estimates. In addition, many providers are likely to view the calculation of individual impact estimates as a threat rather than a benefit. We feel that a monetary incentive is important because we have a very short period of time to collect this data (basically June and early July) before the providers are away for the summer and prior to when programming for the new school year and recall problems lead to less reliable answers on the survey. Finally, the incentive provides for the burden on the individual’s time to complete the survey (the respondent), while the provider-specific data is more a direct benefit to the organization.

We have added the following sentence to Part A section 16.a and to Part B section 2.b to clarify the analyses that will be feasible:

“Because the preferred providers will be identified prior to determining the RD cutoff, we will be able to calculate provider-specific impacts for districts served by the largest providers. However, very few providers will have a large enough sample size for sufficient statistical power to produce reliable provider-specific estimates. In most cases, provider-specific estimates will be aggregated across providers based on provider characteristics and practices.” 

8. Regarding Table A.3, please explain what the “Number of Responses/Respondent” column means and how it is used in this table.

Our understanding of “responses” was incorrect, and Table A3 has been changed to reflect OMB’s definition where the number of responses equals the number of record keepers (or respondents).

9. Appendix A - in the instructions sheet (page 2), the materials assert that "responses will be kept confidential." Given the careful statements referring to ESRA and the precautions noted elsewhere in these documents, we want to confirm that NCEE intends to make this assertion.

We have reconsidered this bullet point in the questionnaire instructions and we have dropped the first sentence of the fifth bullet. We believe that we have adequately covered our position on data protections and disclosure on the cover of the questionnaire and in the respondent cover letter. In addition, as we noticed that these precautions are repeated twice on the survey and letters, we have taken out the repeated text and only include the precautions once in the body of letter text and once on the survey cover using consistent language as follows:

Your responses to this data collection are protected from disclosure per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183. Responses to this data collection will be used only for statistical purposes. Mathematica Policy Research (MPR) will present the information collected as part of this survey in an aggregate form, and will not associate responses to any of the individual SES providers who participate. MPR will not provide information that identifies you and your organization to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.