The Institute of Education Sciences (IES) thanks the National Education Association (NEA) for the thoughtful comments and questions. Below, after each comment, is a response from IES.
(1) There is a significant scientific debate underway over the credibility of value-added studies based on student test scores in measuring teacher quality. Value-added studies ignore both teacher qualifications and broad measures of teacher effectiveness, such as classroom observation, and instead focus on a single measure of teacher effectiveness—standardized student test scores—that were not originally designed to be used for teacher evaluation and do not fully reflect student learning. Value-added studies can produce differing results depending on statistical methodologies, and can also produce inexplicable variations in teacher ratings from year to year within the same statistical methodology. Moreover, there is no evidence that value added analyses can fully take into account differences among the groups of students who are assigned to teachers, or differences in ways students are assigned to teachers.
IES Response: We agree that there is a growing literature and healthy debate about the value-added methodology. To begin to address these concerns, we have revised the planned analyses consistent with advice from our technical working group. The study plan includes conducting various analyses to assess the sensitivity of findings to variations of the value added modeling. In addition, the planned analyses include strategies that are less subject to excessive variation in year to year teacher value added scores. Specifically, the primary analyses do not rely on the precision of the value added estimates of individual teachers, but instead rely on a comparison of the average value added estimates for groups of teachers of different types of students (e.g., compare the teaching experiences as measured by the average value added of their teacher of students who do and do not qualify for free and reduced price lunch). We think with proper written context and analytic rigor, these analyses will provide valuable information for this growing area of policy interest.
The use of value added as a measure for this study is relevant for two additional reasons. First, recent federal initiatives such as the Teacher Incentive Fund and Race to the Top may make use of value added measures of teacher effectiveness. As a result, there is interest in understanding how these policies might be related to the distribution of teachers based on these measures. Second, value added measures can be applied consistently across districts in the study, allowing for a comparison of the distribution of teacher effectiveness across districts.
(2) Given the questions around value-added analyses, it is very important that any study using value-added analyses carefully contextualize the results. It is an overstatement to say that a value-added study truly measures either teacher quality or teacher quality distribution, and the study should thus be more conservatively defined to take into account known study limitations.
IES Response: All study findings will be presented carefully and in context, particularly with respect to the interpretation of the value added estimates. All reports will include information about what the analyses do and do not inform. In addition, the presentation will be sensitive to the level of confidence we have in the patterns based on the totality of sensitivity analyses. Study limitations will be described in all reports, and we will also note that teachers’ value added using student test scores is just one measure of job performance.
(3) It is also important, given the limits of value-added analyses, that teacher privacy be protected in this study so that educators are not unfairly harmed by classifications based on limited data and a limited statistical model. For example, there are already examples of de-contextualized value-added results appearing in newspapers. The study attempts to protect privacy by noting that responses will not be associated with specific schools or individuals, and hopefully extra efforts will be made where the uniqueness of a school or subject would suggest the identity of the educator and students. In addition, we are unsure of the impact of this portion of the “notice of confidentiality” in the proposed District Notification Letter: “We will not provide information that identifies you or your district to anyone outside the study team, except as required by law.” How would this interact, for example, with a public record request at the state or federal level?
IES Response: Districts will remain anonymous in the study reports published by IES, and the reports will not identify specific schools or individuals. The Privacy Act requires that IES publish findings in summary or statistical form only, so that the identity of individual respondents contained in the subject data is not revealed. According to the Education Sciences Reform Act (ESRA; 2002), this protection extends to schools as well. All individually identifiable information about students, their academic achievements, their families, and information with respect to individual schools, will remain confidential. In addition, districts will not receive the value added estimates for individual teachers as part of the study. Finally, the study team would not provide the names of districts participating in the study in response to a public records request.
The contractor and its subcontractors are required to comply with Department security policy requirements as set forth in the Privacy Act of 1974 (P.L 98-579, U.S.C. 552a), ED Handbook for Information Assurance Security Policy, OCIO-01 (March 2006), and the ED Departmental Directive OM: 5-101, Contractor Employee Personnel Security Screenings. The contractor is required to establish procedures at its sites and at subcontractor sites to ensure confidentiality of data. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) confidential data are required to comply with the requirements of the confidentiality standards in section 183 of the ESRA (20 U.S.C. 9573).In addition, the contractor is required to comply with the Department's IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance. The contractor is required to comply with IT security requirements in the Federal Information Security Management Act (FISMA), Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance.
IES and its contractors are attentive and fully adhere to these requirements.
(4) Since value-added analyses of teachers in a district could, if privacy and anonymity are not maintained, be used to judge the effectiveness of individual schools and teachers, we believe that the proposal should include a process whereby teachers and their union representatives are included in the decision over whether the district volunteers for this study. Teachers should also be allowed to verify the accuracy of their data which includes verifying their student rosters.
IES Response: The decision to participate in the evaluation is made by individual school districts and is voluntary. IES follows all processes required by districts interested in participating in the study. In addition, IES as well agrees that it is important that the teacher-student links be of high quality. In fact, data quality, including the quality of the teacher-student links, is an important aspect of the recruitment process for this study. Districts with poor student-teacher links were purposely excluded from consideration for the study. Finally, confidentiality of individual teacher and student information is an important consideration for this study. The contractor is purposefully creating a database that would not permit the identification of individual teachers.
(5) We find the burden hours estimate in the Mathematica Statement (Part A) to be conservative. It estimates at page 10 that a district would spend 20 hours providing student records and 12 hours providing teacher personnel data in the first round of data collection, and less in future collections. Even a small data problem, such as a problem in the data relating teachers to students, could cause those 32 hours to markedly increase. This is particularly true since the study proposes to use districts that have not used value-added analyses in the past, thereby introducing an unnecessary burden on respondents (Appendix B, Section III states: “This is also an opportunity for your district to pilot a value-added model.”)
IES Response: We agree that a consistent problem with the linkages between students and teachers in a district’s data system would be time-consuming to remedy. However, the study team will be specifically recruiting school districts that already have data systems and collections in place in which students are linked to teachers across school years. During recruitment, districts will be asked whether they have existing data that provides a link between students and teachers since this is a requirement for the study. The study team has quite a bit of experience collecting exactly this type of data from school districts and has based the burden estimates in the OMB package on experience working with these districts. The possible benefit mentioned in Appendix B Section III is general guidance on value-added modeling for districts that already have the necessary data but have not used the data in this way. It is not meant to imply that the district will receive support in developing a data system. In addition, while some of the districts may not have experience with value added modeling, recruitment for the study will also include districts that have used value added analyses in the past.
(6) With regard to the study of factors that impact successful distribution of quality teachers, we note that the study proposes to interview districts with regard to policies that might advance equitable distribution of effective teachers. We recommend that teachers in the districts also be interviewed as to measures impacting equitable distribution of teachers, e.g.
on measures that would influence their decision to transfer to another school, as well as measures that would improve teacher effectiveness in low-performing schools.
IES Response: We considered surveying teachers as part of this study. However, IES is conducting two studies of performance-based compensation systems in high-need schools that include teacher surveys and will elicit information about teacher decisions about transferring schools and factors that teachers use in choosing schools to transfer to. Due to the information that will be available from these other studies, IES is purposeful in minimizing teacher burden by not including a similar survey in the distribution study.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Authorised User |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |