The Issue: Teacher
Evaluation Using Alternative Measures of Student Growth
|
In school districts that
have sought to measure the effectiveness of teachers in raising
student achievement, statistical methods that aim to measure
growth or value-added are typically applied to students’
scores on statewide standardized tests. To permit evaluation of
grades and subjects not tested through state exams and to derive
a more comprehensive picture of teacher effectiveness, some
school districts have begun to use alternative student outcome
measures in value-added models (VAMs) and other student growth
models. These alternative measures include end-of-course
curriculum-based assessments and additional standardized as well
as Student Learning Objectives (SLOs)—specific growth
targets for a teacher’s own particular set of students,
typically set by individual teachers and approved by principals.
Various school
districts are adopting alternative student outcomes for measuring
growth in both low-stakes contexts for instructional purposes and
in high-stakes contexts for teacher evaluation. Many more
districts will need to adopt them soon due to changes in
state-level evaluation systems requiring a measure of student
growth. Little is yet known about the features and uses of these
measures, the process of implementation, or the challenges
encountered during implementing.
|
The Study
|
As a partner of the
mid-Atlantic Regional Educational Laboratory (REL Mid-Atlantic),
which is housed at ICF International and funded by the U.S.
Department of Education (ED), Mathematica Policy Research has
been contracted to conduct a study to examine the implementation
of alternative measures for student growth used to evaluate
teachers. We are seeking to identify 9 districts to participate
in the study. The key research questions are
What
student outcome measures other than state standardized test
scores are currently being used in growth measures to assess
teacher performance?
How
have school districts implemented the data collection and
analysis necessary for growth measures based on alternative
student outcomes, and what obstacles have they encountered?
How
are the alternative measures being used for other purposes in
addition to teacher evaluation?
How
much weight does each alternative measure alternative receive in
a teacher’s overall evaluation, and how does the weight
vary by grade and subject? How does the distribution of scores
on the alternate measure compare to the distribution of scores
on growth measures used with state assessments or other
measures of performance?
What are the
perceived benefits and drawbacks of using growth models
(including VAM) based on each type of alternative outcome: SLOs,
end-of-course curriculum-based assessments, and nationally
normed assessments? What costs (notably in terms of time and
effort) do they impose on teachers, principals, and districts?
|
How The Study Works
|
This study aims to fill
the gap in information available to districts and policymakers on
measures of student growth that do not use state standardized
tests via qualitative case studies of up to nine districts that
are using alternative measures of student achievement growth in
teacher performance ratings. The study’s scope will
encompass three categories of alternative student growth
measures: (1) end-of-course curriculum-based assessments used in
growth models, (2) nationally normed assessments such as the Iowa
Test of Basic Skills used in growth models, and (3) Student
Learning Objectives (SLOs).
The case studies will
examine what alternative outcome measures are used, how the
alternative growth measures are implemented, challenges and
obstacles in implementation, how the measures are being used,
and, where possible, the distribution of teacher performance on
the measures, as compared with the distribution of teacher
performance on conventional value-added measures that are based
on state assessments. Districts participating in the study will
not be identified in published reports.
|
Benefits of
Participating in the Study
|
Through their
participation in the study, districts can make an important
contribution to policymaker understanding of alternative measures
of student growth as tools for measuring teacher performance in
multiple contexts. By providing information on the implementation
process, the effectiveness of the measures in differentiating
teacher performance, and the perceived costs and benefits of the
measures from stakeholder perspectives, this study has the
potential to inform states and districts in the REL Mid-Atlantic
region (and throughout the country) in deciding which measures
are promising for use in evaluation, which are of doubtful value,
and how to move forward with implementation. Participation in the
study is voluntary.
|
Study Participation
Requirements
|
In the fall of the 2013-14
school year, participating districts will assist the study team
with scheduling a site visit and/or a series of telephone
interviews with school and district staff. The study team will
conduct one-on-one interviews with up to ten staff
members—including at least one district administrator, two
or three principals, two or three teachers, and one teachers’
union/association representative. During these interviews, the
study team will gather information on district- and school-level
implementation of alternative growth measures, applications of
the measures, the distribution of teacher performance on the
measures, and the perceived costs and benefits of the measures
from the perspectives of those interviewed.
|
The Study Team
|
Mathematica Policy
Research, Inc., a nonpartisan policy research firm, conducts
research and surveys for federal and state governments,
foundations, and private sector clients. Mathematica’s
studies of education initiatives and other programs have been
used to inform national policymakers for more than 35 years.
Mathematica strives to improve public well-being by bringing the
highest standards of quality, objectivity, and excellence to bear
on the provision of information collection and analysis to its
clients. Mathematica has offices in New Jersey, California,
Illinois, Massachusetts, Michigan, and Washington, DC. See
www.mathematica-mpr.com.
|
To Find Out More
|
Contact Mathematica’s
project director, Brian Gill, by phone at (617) 301-8962 or by
email at bgill@mathematica-mpr.com.
|
Data Confidentiality
|
Responses to the data
collection activities will be used for research purposes only.
The reports prepared for the study will summarize findings across
the sample and will not associate responses with a specific
district, school, or individual. We will not provide information
that identifies you or your district to anyone outside the study
team, except as required by law.
Mathematica
follows the confidentiality and data protection requirements of
the Institute of Education Sciences (The Education Sciences
Reform Act of 2002, Title I, Part E, Section 183). We will
protect the confidentiality of all information collected for the
study and will use it for research purposes only. No information
that identifies any study participant will be released.
Information on respondents will be linked to their institution
but not to any individually identifiable information. No
individually identifiable information will be maintained by the
study team. All institution-level identifiable information will
be kept in secured locations and identifiers will be destroyed as
soon as they are no longer required.
|