TLES District Screener

District-Level Screening Protocol

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is XXXX–XXXX. The time required to complete this voluntary information collection is estimated to average 30 minutes per response. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202–4537. If you have comments or concerns regarding the status of this review of your district’s eligibility for study participation, write directly to: Melanie Ali, U.S. Department of Education, Institute of Education Sciences, Room 502B, 555 New Jersey Avenue, NW, Washington, DC 20208-5500.

Protocol

The telephone call should be made to one of the following people:

• The district person responsible for the system evaluating teachers and leaders.

• A senior human resource person in charge of teacher/leader compensation and/or career development.

These individuals may be identified on the district’s website under Human Resources. In the rare case in which a contact is no longer current, you may need to call the district’s general number for Human Resources and ask for the best person to talk to about teacher and leader evaluation or teacher and principal career development. If the person you reach does not agree that he or she is the right person to provide the necessary information, ask that person to recommend someone else.

Before the call, please send to the district contact(s) a scheduling e-mail with the materials that describe the study. When you’re ready for the call, start with the following conversation:

Good morning {or afternoon}, my name is _____, I am with the American Institutes for Research (AIR), an independent research firm in Washington, D.C. AIR is launching the initial phase of a U. S. Department of Education research project on teacher and leader evaluation systems. As a first step, we are contacting a number of districts to learn more about their teacher and leader evaluation systems, including the different components of the systems. We exchanged e-mails a few days ago, and I sent you materials describing the study.

Do you have time to talk now? My questions will take about 30 minutes. All the information collected will be kept confidential. We follow the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). We will protect the confidentiality of all information collected for the study and will use it for research purposes only. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific program, district, school, or individual. We will not provide information that identifies any study participant to anyone outside the study team, except as required by law.

Before we get started, do you have any general questions about the study? I would be happy to describe the
study briefly. (Refer to the provided two-page description of the study to explain main points.)

PART I: Current Teacher Evaluation System Used by District

First, I would like to ask a few general questions about the teacher evaluation system currently used by your district.

1. Briefly, what goes into a teacher's rating under your district's evaluation system? (Open-ended description, probe using categories below.)

Different categories:

 Rating on classroom practice (e.g., classroom management, specific instructional practices)

 Rating on other aspects of teacher practice (e.g., parent communication, collegiality)

 Rating on student growth (e.g., value-added modeling)

 Other, please specify________________________________

2. What are the different sources of information used for evaluating teachers? (Open-ended description, probe using categories below.)

Different sources of information:

 Classroom Observation  Surveys  Portfolios

 Student growth  Walkthroughs  Other, please specify____________

3. How often do teachers receive a summative rating from the evaluation system? (Open-ended description, probe using categories below.)

 Less than once a year  Once a year

If category Classroom Observation is marked in Question 2, then proceed to Question 4. If category Classroom Observation is not marked, probe: Can you reconfirm that classroom observations are not used as a part of the teacher evaluation system? If reconfirmed, proceed to Question 17.

Next, I would like to ask you some more specific questions about how you determine ratings of teachers’ classroom practice.

4. Is there a form used to record ratings of classroom practice? What is this form called?

 Yes  No

Name of the form: __________________________________________________________

If Yes, proceed to Question 4 a. If No, probe: is there any system for determining ratings of teacher practice? Then proceed to Question 6.

4 a. For how many years has this form been in use?

Number or years: ___________________________________________________________

5. How detailed is the form?

5 a. Specifically, how many aspects of classroom practice are rated? For example, an aspect of classroom practice might be “classroom management”. Can you name few of the aspects? (Assess whether aspects relate to classroom practice; if not, redirect respondent.)

Number of aspects: _________________________________________________________

Aspect 1:_________________________________________________________________

Aspect 2:_________________________________________________________________

Aspect 3:_________________________________________________________________

5 b. How many rating levels are available for these different aspects? For example, an observation tool might have two rating levels for each aspect of a classroom practice (e.g., classroom management could be rated “satisfactory” or “unsatisfactory”). Alternatively, an observation tool might have four rating levels for each aspect of classroom practice (e.g., “1” “2” “3” and “4”) or different rating levels may be used depending aspects rated (Open-ended description, list rating levels.)

Different rating levels used for different aspects:  Yes  No

Rating levels: _____________________________________________________________

5 c. How detailed is the guidance provided to raters to help them determine what ratings to select? I’d like to understand if there is general guidance (e.g., a general definition of “satisfactory” that applies to all aspects that are rated), or if there is guidance specific to each aspect of practice that is rated (e.g., a definition of “satisfactory” that is specific to “classroom management,” another definition of “satisfactory” that is specific to “clarity of explanations,” and so on for each aspect on the form. (Open-ended response, mark categories if appropriate, probe for examples.)

 The same general definition is used for all aspects rated (describe the definition.)

 A specific tailored definition is used for each of the aspects rated (describe the definitions.)

5 d. I’d also like to understand whether the form for rating classroom practice draws on a framework distributed by a national organization of some kind or on a framework that is commercially available? I will read aloud names of some of these frameworks. Can you tell me if the currently used observation tool draws on any of the mentioned frameworks? (Read the list of frameworks in the table and probe, mark down answers.)

Table 1. List of Classroom Observation Frameworks

Name of the observation frameworks	5 d. District form based/drawn on (Yes/No)
Classroom Assessment Scoring System (CLASS)
Framework for Teaching (FFT) a.k.a The Danielson Framework a.k.a Danielson’s Framework for Teaching (FFT)
McREL's Teacher Evaluation System
The Marzano Teacher Evaluation Model
The System for Teacher and Student Advancement a.k.a Teacher Advancement Program (TAP)
Any other commercial evidence-based system (Please identify the system.)

Thank for you answering my questions about your forms for recording ratings of classroom practice. If it is available publicly, could you send me the rating form for my reference, or point me to where to find it on your Website?

Web-site address: ______________________________________________________________________

My remaining questions are about the observations and observers—specifically who observes classroom practice, and how they are trained as well as the district's future plans for evaluating teachers' classroom practice.

6. Who conducts the observations that determine ratings of classroom practice? ______________________________________________________________________________

7. How much training is provided for the observers? (Probe about the number of hours of training.)

_____________________________Hours

8. Are the observers required to reach a specific standard at the end of the training? For example, are the observer trainees observing together with a ‘gold standard’ observer to test whether they have mastered the protocol?

 Yes  No

9. Are those who conduct the observations required to do additional training during the school year, or every few years, to maintain their skills?

 Yes, additional training during school year  Yes, additional training annually

 Yes, additional training every few years  No additional training is required

10. How often does the current evaluation system require that new teachers are observed?

 Less than once a year  Once a year  2 times a year

 3 times a year  4 or more times a year

11. How often does the current evaluation system require that veteran teachers are observed?

 Less than once a year  Once a year  2 times a year

 3 times a year  4 or more times a year

12. We know that for various reasons, from scheduling conflicts to limited resources, the actual number of observations per teacher may vary. Can you tell me how many times most of the veteran teachers are observed?

 Less than once a year  Once a year  2 times a year

 3 times a year  4 or more times a year

13. How many minutes does an observation typically last?

_____________________________Minutes

14. Are teachers given feedback based on the observations conducted?

 Yes  No

15. How often are teachers given feedback based on the observations? For example, are teachers given feedback after each observation?

 Less than once a year  Once a year  2 times a year

 3 times a year  4 or more times a year  After each observation

16.  We’re also interested in your experience with frameworks that your district is currently using or has used for any other purposes, including providing information for a teacher compensation incentive system — not for evaluative purpose.  I will read aloud names of some of these frameworks. Can you tell me if you are currently using or have used in the past an observation tool based on any of the mentioned frameworks? (Open-ended description, mark on the table as appropriate; probe: for what purpose was the tool used and in how many schools; frequency of use and the length of the observation.)

Table 1. List of Classroom Observation Frameworks

Name of the observation frameworks	16. Non-evaluative use of the framework (purpose, frequency, length of observation)	16. Number of schools	16. When used and for how long
Classroom Assessment Scoring System (CLASS)
Framework for Teaching (FFT) a.k.a The Danielson Framework a.k.a Danielson’s Framework for Teaching (FFT)
McREL's Teacher Evaluation System
The Marzano Teacher Evaluation Model
The System for Teacher and Student Advancement a.k.a Teacher Advancement Program (TAP)
Any other commercial evidence-based system (Please identify the system.)

17. For purposes of our study, we would also be interested in knowing if your district is piloting or considering a different approach to giving teachers ratings of classroom practice – either a different rating system, a different approach to selecting or training observers, or changing the number of observations. Are you aware of any initiatives to pilot or develop a new approach to giving teachers ratings of classroom practice? (Open-ended response, probe and mark down answers as appropriate).

Probe: Do you know whether there are plans to pilot the system with any specific grade levels or schools in the district? What kind of plans?

Probe: Regarding what you just described, we would be interested in knowing the approximate timeline and the type and number of schools that would be affected.

Timeline, type and number of schools:

 2012-2013 school year  2013-2014 school year  2014-2015 school year

Elementary Schools Number: _______  All  Most  Few

Middle Schools Number: _______  All  Most  Few

High Schools Number: _______  All  Most  Few

PART II: Current leader evaluation system

Next, I would like to ask questions about the current principal evaluation system used by the district.

18. Does the district have a system in place for principal evaluations?

 Yes  No If the answer is No proceed to Question 28.

19. Is the principal evaluation system aligned with any particular standards (such as the ISLLC standards or state standards)?

 Yes  No

20. Briefly, what aspects of principal performance are used to determine a principal's rating under your district's evaluation system? (Open-ended description, probe using categories below.)

Different aspects:

 Aspects related to instructional leadership (e.g., instructional quality, curriculum rigor)

 Aspect related to other leadership practices (e.g., school culture, parent and community relations)

 Aspects related to accountability (e.g., improving conditions of schools)

 Student growth (e.g., value-added models)

 Other, please specify________________________________

21. What are the different sources of information required for evaluating principals and how often are they required to be used? (Open-ended description, probe using categories below.)

Different sources of information:

 Surveys  Observations  Interviews

 Student growth models  Testimonials  School walkthroughs

How often required to be used:

 Less than once a year  Once a year

 2 times a year  3 or more times a year

Next, I would like to ask you some more specific questions about how you determine ratings for the different components used in principal evaluations.

22. Is there a form or an instrument used to record ratings of principal practice? What is this form called?

 Yes  No

Name of the form/instrument: __________________________________________________________

If Yes, proceed to Question 23. If No, probe: is there any system for determining ratings of principal practice? Then proceed to Question 28.

23. For how many years has this form/instrument been in use?

Number of Years: ___________________________________________________________

24. Who fills out the instrument(s)/form(s) used for principal evaluations?

 Principal  Principal's supervisor  Principal and principal's supervisor

 Parents  Students  Building teachers, principal and principal's supervisor (e.g., 360 degree evaluation)

25. Are principals provided feedback based on the instrument(s)/form(s) used?

 Yes  No

26. Who provides the feedback and how often is the feedback provided? (Open-ended description of the feedback, probe about the frequency of feedback.)

 Less than once a year  Once a year  2 times a year

 3 times a year  4 or more times a year

27. I’d also like to understand whether the form for rating principal practice draws on a framework distributed by a national organization of some kind or on a framework that is commercially available. I will read aloud names of some of these frameworks. Can you tell me if the currently used principal evaluation form is based/drawn on any of the mentioned principal evaluation systems?

Table 2. List of Principal Evaluation Systems

Name of the principal evaluation system	District system based on (Yes/No)
Vanderbilt Assessment of Leadership (VAL-ED)
McREL’s Principal Evaluation System
The Marzano Principal Evaluation Model
The Doug Reeves Model
The System for Teacher and Student Advancement a.k.a Teacher Advancement Program (TAP)
The Marshall System
Any other commercial evidence-based system (Please identify the system.)

Thank for you answering my questions about your principal evaluation system. If it is available publicly, could you send me the rating form for my reference, or point me to where to find it on your Website?

My remaining questions are about implementing the principal evaluation system and your district's future plans for the system.

28. For purposes of our study, we would also be interested in knowing if your district is piloting or considering a different approach to evaluating principals. Are you aware of any initiatives to pilot or develop a new approach for evaluating principals? (Open-ended response, probe and mark down answers as appropriate.)

Probe: Regarding what you just described, we would be interested in knowing the approximate timeline and the type and number of schools that would be affected.

Timeline, type and number of schools:

 2012-2013 school year  2013-2014 school year  2014-2015 school year

Elementary Schools Number: _______  All  Most  Few

Middle Schools Number: _______  All  Most  Few

High Schools Number: _______  All  Most  Few

29. Is the district currently partnering with external organizations, such as local universities or New Leaders for New Schools to improve principal effectiveness? Can you tell me the name of the organization, and the nature of the effort?

 Yes  No

Name of the organization: _______________________________________________________

Nature of the effort: ___________________________________________________________

PART III: Need for Contact with Teacher or Administrator Union

30. If your district meets the requirements of participating and chooses to participate in the study, would we need to seek endorsement from the teacher’s union and, if applicable, the school administrators’ union, to pilot a comprehensive teacher and leader evaluation system in your district?

 Yes  No

31. Do you know who would be the appropriate teacher’s union person or, if applicable, the school administrators’ union person(s) to contact?

 Yes  No

Name__________________________________________________________________

Contact Information: ______________________________________________________

We have some additional questions regarding the district’s data systems. Are you the correct person to answer these questions? If not, could you refer me to the correct person? (If the correct person to ask about district’s data systems, proceed to Question 32; otherwise probe for name and contact information.)

Name: ____________________________________________________

Phone: ___________________________________________________

Email: ____________________________________________________

Thank you for your time. We will carefully review the information you gave us today. If your district qualifies for participation in the study, what are the appropriate channels for further communication? For example, can we call you for further information or should we contact a specific district employee? (Open-ended description.)

If you have any questions for me, you can reach me at [contact information]. Thank you again for your time.

PART IV. Current Use of Student Data System for Teacher and
Leader Evaluation

This portion of the protocol must be answered by a person who is knowledgeable about the district data systems. It is likely that this person is different from the contact person who answered the questions in Parts I, II and III. If a different contact person is required to answer questions about district’s data systems, Part IV interview will only be scheduled after it is clear that information received in Parts I and II will not make the district ineligible to participate.

Good morning {or afternoon}, my name is _____, I am with the American Institutes for Research (AIR), an independent research firm in Washington, D.C. AIR is launching the initial phase of a U. S. Department of Education research project on teacher and leader evaluation systems. As a first step, we are contacting a number of districts to learn more about their teacher and leader evaluation systems, including the different components of the systems. You were referred as the correct contact to ask questions about your district’s data systems by [add name].

Do you have time to talk now? My questions will take about 10 minutes. All the information collected will be kept confidential. We follow the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). We will protect the confidentiality of all information collected for the study and will use it for research purposes only. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific program, district, school, or individual. We will not provide information that identifies any study participant to anyone outside the study team, except as required by law.

Before we get started, do you have any general questions about the study? I would be happy to describe the
study briefly.

32. I would like to ask you some questions about the student data systems in your district and how they are currently used. Before we get started I would like to confirm that your district [does/does not] use student growth (e.g., measured through value-added modeling) to evaluate teachers and/or principals. [Use information from Questions 1, 2 and 20 to determine the correct statement] Is that correct?

 Yes – Use student growth  No – Does not use student growth

If the answer is Yes, proceed to Question 33. If the answer is No, proceed to Question 34.

33. How are student growth scores used as a part of your teacher or leader evaluation systems? (Open-ended description, probe: is the student growth model done by district or are the growth scores provided by the state? How much weight do the student growth scores have in teacher or principal evaluations (e.g., 30% of their effectiveness score is based on student growth)? Is there a website page or other publicly available materials that describes the student growth model used by the district?) Proceed to the end of the protocol.

34. Is your district or state using a student growth model, but the results are not currently used as a part of your teacher or leader evaluation systems? For example, is a student growth model used to provide information for a teacher compensation incentive system?

 Yes  No

If Yes, write down what scores are used for:____________________________________________________________

If the answer is Yes, proceed to Question 43. If the answer is No, proceed to Question 35.

35. Does the district have yearly student assessment test scores in reading and mathematics for grades 3 to 8 (such as scores from state administered tests) since the school year of 2010-2011?

 Yes  No

If the answer is Yes, proceed to Question 36. If the answer is No, proceed to Question 43.

36. In its current data systems, does the district use unique IDs for the following? (Use categories below.)

37. Are student, teacher and principal IDs linked to school IDs? For example, can students and teachers be followed from school to school? (Mark below.)

Students  Yes  No  Linked to school ID

Teachers  Yes  No  Linked to school ID

Principals  Yes  No  Linked to school ID

Schools  Yes  No

If Yes to having unique student and teacher IDs proceed to Question 38, if No to having unique student and/or teacher IDs proceed to Question 43.

38. Has the unique ID system for teachers and students been in place at least since the school year of 2010-2011?

 Yes  No

39. Are the assigned IDs kept the same from year to year?

 Yes  No

40. Can the district link students to teachers and courses?

 Yes  No

41. Are course codes consistent over time and across schools?

 Yes  No

42. What other student-level information is linked to students using unique and consistent student IDs since fall 2010? I will read aloud some categories. Can you let me know if this information is available? (Probe using categories below.)

 An indicator of whether the student is considered to have attended for the full academic year (or entry/exit dates and rules for determining full academic year status).

 Attendance (e.g., days present/absent)

 Student demographic information (e.g., ethnicity, gender.)

 Program participation information (e.g., special education status, English language learner status.)

 Exit/entry dates into schools and/or courses

43. Does the district have any initiatives underway to use student growth for teacher or leader evaluations?

 Yes  No

If the answer is Yes, proceed to Question 44. If the answer is No, proceed to the end.

44. What does the initiative involve, and when would it begin? (Open-ended description.)

Thank you for your time. We will carefully review the information you gave us today. If your district qualifies for participation in the study, what are the appropriate channels for further communication? For example, can we call you for further information or should we contact a specific district employee? (Open-ended description.)

If you have any questions for me, you can reach me at [contact information]. Thank you again for your time.