Alternative student outcomes for growth measures case studies

OMB Supporting Statement Part B: Alternative Student Growth Measures for Teacher Evaluation: Case Studies of Early Adopters

Appendix G: Union Representative Interview Protocol

Instruction to interviewer: when the respondent is unable to respond completely to a question, please always follow up by asking for the names and contact information of other staff in the district who can provide more information.

Introduction and Consent for Audio Recording

The purpose of this section is to introduce the respondent to the study and request consent for audio recording.

1. Thank you for taking the time for the interview today. My name is [name] and I work for Mathematica Policy Research, which is conducting this study for the Mid-Atlantic Regional Educational Laboratory (REL). The Mid-Atlantic REL’s mission is to “build research alliances with education practitioners and policymakers to help states and districts use data to answer important issues of policy and practice.” The results of this study will inform publically available reports aimed at states and districts considerting implementing measure of teacher performance that include student growth. As I mentioned [in my email/when we spoke on the phone], we are conducting case studies of districts implementing measures of teacher performance that include student growth, but that measure growth on outcomes other than state assessments. These alternative outcome measures might include nationally normed assessments, end-of-course curriculum-based assessments, or “student learning objectives” (SLOs) developed by individual teachers for their own students. Student growth might be measured through a simple subtraction of end-of-year achievement from beginning-of-year achievement, through a student growth percentile method, through a value-added model, or implicitly in SLOs that are set based on an understanding of baseline achievement levels in a class at the beginning of the school year. We plan to examine what alternative outcome measures are used, how the growth measures are implemented, challenges and obstacles in implementation, how the measures are being used, and the effectiveness of these measures in differentiating teacher performance. We expect our findings to benefit states and districts considering the use of such alternative growth measures. The purpose of this interview is to gather information on the implementation of [insert specific growth measures used in the district] in the district from your perspective as a district administrator. We will be interviewing teachers, principals, and union representatives in the district to gather information on their perspectives as well. We will not identify you by name in the study reports.We also will not identify the districts included in the study in subsequent reports.

Before beginning, I would like to get your permission to record the interview. This will ensure that we accurately capture information about district policies and will prevent me from having to ask questions and take notes at the same time. The audio recordings from all individuals interviewed within your district will be compiled into a single district summary that is for internal analysis purposes only and will not be published or shared with anyone outside the study team. You will not be identified in the district summary. This summary will then be aggregated with other district summaries and analyzed for trends and variations across multiple districts. Do I have your permission to record the interview?

Type of Student Outcome and Growth Measure Used

1. Are teachers in this district measured based on [identify the alternate outcome growth measure(s) discussed in district interview]?

2. What role did the teachers’ union/association play in the design of the alternative measure(s)? Can you describe the process for rolling out the alternative measure(s) in the district (including SLOs)? What role did the teachers’ union/association play in the implementation? Did the union raise any concerns regarding the alternate measure(s)? If so, how did the district respond to and address those concerns? Were teachers provided with any training or professional development related to the implementation of the measure?

3. Does the district use a VA model? If not, how does the district measure growth (e.g., percentile method, simple subtraction)? How long has the growth/VA model been in place? Has the district made any changes to this approach to measuring growth during that period?

Implementation of Data Collection and Analysis

1. I would like to learn more about the administration of these alternative student assessments in the district. (For each assessment) Are you familiar with how the assessment administered (e.g., timing, frequency, who administers)? What is the process for collecting assessment results?

2. Are you aware of any quality control measures in place in the district for the collection and analysis of the teacher performance data? (If district uses SLOs) For example, what documentation of student performance do teachers need to submit for SLOs? Is this documentation monitored regularly? Is this documentation subject to any audits? (If district uses a VAM/growth model) For example, do teachers administer or score the student assessments or is administration and/or scoring conducted by independent proctors or an external firm?

3. What kinds of obstacles, if any, have teachers encountered in preparing for and administering the student assessment or submitting student assessment results? How much extra time, if any, have teachers had to devote to these tasks? What is your perspective on how to address these obstacles? Have you found the district to be responsive to these concerns?

4. Have there been any changes to the district’s approach to using alternative student growth measures? Have these changes resulted in notable improvements?

Use of Alternative Growth or VA Measure

1. For what purposes is the district using each alternative student outcome measure used in assessing teacher performance? Is it used for evaluating teachers? In what grades and subject areas? Is it used as part of a performance-based compensation system? In what grades and subject areas? Do teachers receive targeted professional development based on their performance? Does the measure affect teacher assignments? Are there any other ways in which the district is using these measures?

2. What role did the teacher’s union/association play in determining how the alternative student outcome measure would be used? Is the measure used in just cause terminations or tenure denial? If so, does the teacher’s union/association support the measure’s use in these decisions?

3. Do teachers receive any feedback regarding their performance on each measure? What type of feedback? When and how is this feedback provided? Who provides this feedback to teachers?

4. If used in teacher evaluation, is the measure combined with any other measures of teacher performance (for example, value-added or other growth measures based on state assessments or classroom observation measures)? How much weight does each performance measure receive in a teacher’s overall evaluation rating? Does the use of the measure in teacher evaluation vary by grade and/or subject area?

5. Does a teacher’s performance rating on the alternative measure affect whether he/she is awarded tenure?

Benefits and Drawbacks

1. Could you summarize some of the main benefits, from your perspective, of implementing this alternative VAM or growth measure of teacher performance in the district? For example, has the district been able to measure the performance of a larger percentage of teachers in the district? How have teachers in the district responded to the use of alternative student outcomes to measure their performance? Do you perceive this alternative measure as an effective way to improve teacher quality in the district?

2. What do you perceive as the primary drawbacks or costs of implementing the measure? Do you think the measure allows for a sufficient level of comparability across teachers on performance? If SLOs are used in the district, do you think all teachers are setting sufficiently high standards for student growth?

3. How do you perceive the district’s communication of the use of the alternative measure in evaluations to teachers?

District Context

1. Have there been any changes in the district administration or policies that may have affected the implementation of these alternative measures of teacher performance (for example, change in superintendent or implementation of a performance-based pay system)?

2. Did the district have the support of the teachers’ union/association for the implementation of the VAM or growth measure of teacher performance using an alternative student outcome? Why or why not?

3. Has the district received funding or resources from external sources (e.g. federal government, foundations) to develop or implement this measure?

Is there anything else that you think might be important to know about the development and implementation of this alternative growth or value-added measure in the district? Are any other key stakeholders in the district with whom you think it would be important for us to speak?

OMB Package: Alternative Student Outcomes for Growth Measures Case Studies 1