ED-OMB QandA 12-20-07

QUESTION: Regarding question 4, did NCEE do a formal power analysis to determine the level of precision required in key estimates? What are the precision requirements?

RESPONSE: No. We did not base our participant sample size requirements on a formal power analysis, given that our goals in terms of estimating key outcomes are descriptive and that our analysis plans do not call for formal statistical comparisons across Centers, groups of Centers, or within a Center over time.

We will report descriptive information from these data across all sampled projects from each Center, the group of 16 RCCs, the group of 5 Content Centers, and for the system of 21 Centers.

To arrive at our participant sample size computations, we gathered information about expected numbers of participants for sampled projects. We sought to select a sufficient number of participants to obtain a fair reflection of views from a substantial numbers of participants in each project, while also balancing concerns about respondent burden.

We believe that our proposed approach provides face validity with respect to our joint goals of sufficiency and fairness, given that we are sampling:

• all potential respondents for projects with up to 12 participants

• 12 participants for projects with between 13 – 25 participants, representing a minimum of 48 percent of potential respondents [or 40 percent, assuming an 85 percent response rate]

• 48 percent of potential respondents for projects between 26 – 100 participants.

QUESTION: Will the descriptive statistics include estimates by project size?

RESPONSE: No. The Comprehensive Centers are asked to indicate on their Project Inventory Forms whether they consider each project to be of a major, moderate or minor level of effort, relative to the projects that they have undertaken. This strategy ensures that we would sample projects that reflected, for each Center, their most important and resource-intensive activities.

In fact, we used these data in our sampling of projects (already cleared by OMB) for the first year of this evaluation. It permitted us to focus primarily on major projects and review some moderate projects while excluding all minor projects. Given that the definitions of major, moderate and minor projects were developed for these sampling purposes and essentially allow each Center to rank order its own projects, we believe the definitions across Centers of the absolute levels of effort for projects in these categories may not be consistent. Therefore, we do not believe it would be meaningful to report on outcomes by project size.