OMB comments - responses for Teacher Quality Distribution Study

Response To OMB Comments for the Study of the Distribution of Teacher Effectiveness

Can ED provide a summary of justification for the changes they made to their model and analysis, particularly going from the use of the D-Index to the use of AEG?

The goal of the distribution measure for this study is to describe the extent to which disadvantaged and non-disadvantaged students have equal access to effective teachers, defined here as teachers with high value-added scores. The distribution measure will be used both to describe the distribution in each district and as the key outcome variable in the analysis of the relationship between the distribution and district policies. We sought a distribution measure that would be a good fit for these purposes and met several criteria that are important to the overall study. Below we compare the Generalized D-Index (D-Index) and the Average Effectiveness Gap (AEG) based on these criteria and explain why we believe the AEG is more appropriate for this study.

1. Ease of interpretation by policymakers. The notion of a gap has long existed in educational policy debates; for example, the black/white test score gap. The AEG builds on the concept of a gap but moves from defining the gap in terms of test score levels (the more common notion) to defining it in terms of access to effective teachers as measured by test scores gains, or value added. The AEG can also be used to describe how the teacher effectiveness gap contributes to the narrowing or increasing of the overall achievement gap between disadvantaged and non-disadvantaged students. By contrast, the D-Index has an interpretation grounded in the number of teachers who need to move across quintiles of school poverty categories to obtain an even distribution of effective and ineffective teachers within each quintile.

Two properties of the AEG make it a more meaningful measure of the distribution for policymakers. First, the AEG provides a direct measure of the gap in teacher effectiveness for disadvantaged and non-disadvantaged students. It is a metric that gives consistent information across districts, based on the assumption that a one standard deviation difference in student performance on test scores is an equivalent measure of a test score gap across districts. By contrast, two districts could have the same D-Index score even if they had very different underlying gaps in the effectiveness of the teachers of disadvantaged and non-disadvantaged students. This is because the D-Index groups teachers into three categories based on their effectiveness (i.e., high, average, and low) and does not measure variation in effectiveness within these three categories, and because the variance of the percentage of student FRL across quintiles of schools can differ by district.

Second, is the AEG is directional, meaning the AEG can distinguish between a distribution of teacher effectiveness that is inequitable (i.e., average teacher effectiveness is greater for disadvantaged students) and compensatory (i.e. average teacher effectiveness is greater for non-disadvantaged students) along a continuum. However, the D-Index is non-directional. For example, the D-Index would assign the same numeric value to a distribution of teachers in which effective teachers were all assigned to non-disadvantaged students and the mirror image of this distribution in which effective teachers were all assigned to disadvantaged students. This makes the D-Index more difficult to interpret than the AEG.

2. Use as the dependent variable in the correlational analysis. A key part of our analysis will involve examining how district policies are correlated with our distribution measure. The two properties described above make the AEG a better fit as a dependent variable in the correlational analysis. Since the D-Index is a less precise distribution measure, we could not examine the extent to which policies are associated with a narrowing or increasing of the teacher effectiveness gap. Also, the non-directional nature of the D-Index means that we could not determine whether policies are associated with an inequitable or compensatory distribution. The AEG does not present these challenges—we can relate policies to changes in the teacher effectiveness gap and the AEG indicates the direction of any inequities in the distribution.

3. No need to base judgments of effective or ineffective teachers on an arbitrary cutoff. This is true of the AEG, but not the D-Index. The D-Index requires cutoffs to establish discrete classifications of teachers (i.e., high, average, and low teacher effectiveness) and establish school disadvantage quintiles.

4. Across- and within-school measures. Used in combination, the ASEG and ATEG provide information on whether an inequitable distribution is driven by differences in teacher effectiveness within and/or across schools. The D-Index measures only the across-school distribution of teacher effectiveness.

5. Extension to multiple years for a cohort of students. By measuring the AEG annually over multiple years, we will be able to examine whether there is a cumulative gap that grows larger each year or whether inequitable distributions in some years are offset by compensatory distributions in others. Although this is possible with the AEG, it is not clear how one would do this with the D-Index.