OMB-ED QandA July 2010

OMB Questions & ED Responses

July 2010

QUESTION 1

Is there anything we can learn from this study about alternative models for selecting high and low-performing schools?

RESPONSE

We propose to address the following secondary research question in the Title I Study:

Can we accurately identify high- and low-performing schools using readily-available school-level performance data and demographic information?

Education policymakers and practitioners seek to identify schools on the high and low ends of performance for distinct purposes. High-performing schools have been identified for studies of schools that, "beat the odds" to offer best-practice models. Low-performing schools have been identified for significant intervention under the No Child Left Behind Act; more recently, the new School Improvement Grants (SIG) program is providing millions of dollars to turn around the lowest performing schools across the country.

This study will identify consistently high- and low-performing schools for the sample using publicly-available information on school proficiency levels on state-administered 3rd grade reading assessments and the percentage of students eligible for free or reduced price lunch. School performance will be measured by overall reading proficiency levels measured across a three-year span and by the extent to which a school exceeds or fails to meet expectations of proficiency levels, conditional on the students' level of economic disadvantage (that is, a residual measure from a regression model). This is similar to the "value added" concept, since high performing schools are not necessarily more effective if they simply have a more economically advantaged student population.

Within the selected schools, we will measure teaching practices and student growth from pre-kindergarten through grade 3 using sensitive, reliable, and consistent measures across schools and districts. The information we collect about students and teaching practices will help examine how consistently low- or high-performing these schools are. Using the student-level measures that are comparable across schools and more sensitive than the state assessment measures used to identify high- and low-performing schools for the study, we will measure how consistent each school's performance is, both across pre-kindergarten through grade 3 on average and within these grades. We will measure the distribution of student growth associated with teachers within a school and with each grade level and analyze the extent to which student growth in each grade can predict whether the school was identified as high or low performing. We will analyze whether the teaching practices that are associated in the study's analyses with greater student growth can predict whether the school is classified as high- or low-performing.

These results will answer questions such as: (1) Do schools tend to have consistently high- or low-performance across grades and across classrooms? (2) Are third grade assessment measures (typically the first year states collect standardized results) indicative of cumulative school effects in earlier grades?

QUESTION 2

Can we use this study as an opportunity to learn more about various approaches to observing teachers and teaching?

We propose to address the following secondary research question in the Title I Study:

RESPONSE

How can researchers measure teaching practices more reliably?

Past research has often shown a weak association between measures of teaching practices collected through intensive observation and student outcomes. This finding has been particularly disappointing in studies of research-based education interventions that have shown impacts on teaching practices but little effect on student achievement (Garet et al. 2008; Jackson et al., 2007; Gamse et al. 2008). Reliability of classroom observation measures, or the ability of the measure to provide stable and consistent judgments about the quality of instruction, has been cited as a key issue that may explain the weak measured relationships between teaching practices and student learning in previous studies.

The study team is developing an observation instrument (protocol for conducting observations and rubrics for assessing the observations), training materials, and training procedures designed to measure teaching practices more reliably than has been done in past studies. The rubric will have more "anchors" explaining clearly how to score each dimension; will include more dimensions of practice to distinguish different aspects of teaching; offer participatory training designed to increase understanding and confidence in using the measure and making scoring judgments; and require high levels of exact agreement for certification in coding videotaped classroom situations and actual classrooms.

Thus, the observation measure and training materials represent an important contribution to the study of teacher quality and student learning. The study will discuss the observational protocol and measure, and its reliability compared with other classroom measures. It will analyze inter-rater reliability as well as the variation across observations for a single teacher during the year. It will discuss how the measure, training materials, and training procedures were all designed to improve reliability, highlighting differences from measures used in previous studies and lessons learned in this study. Recommendations for improving the reliability of teaching practice measures in future studies will be included.