4 8 5 REL SW Supporting Statement A REVISED KM 121814 clean shortappendix

SUPPORTING STATEMENT A:

REQUEST FOR CLEARANCE OF INFORMATION COLLECTION FORMS FOR

“IMPACTS OF A DETAILED CHECKLIST ON FORMATIVE FEEDBACK TO TEACHERS”

August 2014

Submitted to: Submitted by:

U.S. Department of Education SEDL

Institute of Education Sciences 4700 Mueller Blvd.
555 New Jersey Ave. NW, Rm. 308 Austin, TX 78723

Washington, DC 20208 Phone: (800) 476-6861

4700 Mueller Blvd. Austin, TX 78723

800-476-6861

www.relsouthwest.org

This publication was prepared for the Institute of Education Sciences (IES) under contract ED-IES-12-C-00012 by Regional Educational Laboratory Southwest, administered by SEDL. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.

Contents

OVERVIEW 5

Impact Research Questions 7

Implementation Fidelity Research Questions 7

Exploratory Research Questions 7

A. JUSTIFICATION 9

1. Circumstances Necessitating the Data Collection 9

2. How, by Whom, and for What Purpose the Information Is to Be Used 17

3. Use of Information Technology 18

4. Efforts to Avoid Duplication of Effort 19

5. Sensitivity to Burden on Small Entities 19

6. Consequences of Less Frequent Collection and Obstacles to Burden Reduction 19

7. Circumstances Requiring Special Information Collection 20

8. Federal Register Announcement and Consultation Outside the Agency 20

a. Federal Register Announcement 20

b. Consultations Outside the Agency 20

c. Unresolved Issues 21

9. Payments or Gifts to Respondents 22

10. Assurances of Confidentiality 22

11. Justification for Sensitive Questions 24

12. Estimates of Hour Burden 25

Burden for Study Recruitment 26

Burden for Extant Data Collection 27

Burden for Survey Data Collection 27

Total Cost to Respondents 27

13. Estimated Total Annual Cost Burden to Respondents or Recordkeepers 28

There are no start-up costs associated with this collection. 28

14. Estimated Annualized Cost to the Federal Government 28

15. Program Changes or Adjustments 28

16. Plans for Tabulation, Publication, and Schedule for Project 29

a. Tabulation Plans 29

b. Publication Plans 29

c. Project Time Schedule 31

17. Display of Expiration Date 31

18. Exceptions to the Certification Requirement 32

REFERENCES 33

APPENDIX A – Teacher and Principal Survey Instruments 35

Spring 2015 School Leader Survey 35

Spring 2016 School Leader Survey 35

Spring 2015 Teacher Survey 35

Spring 2016 Teacher Survey 35

APPENDIX B – Principal and Teacher Recruitment Materials 36

Wave 1 Principal Recruitment Script– SPRING 2015 36

Wave 1 Teacher Recruitment Script– – SPRING 2015 36

Wave 2 Principal Recruitment Script– – SPRING 2016 36

Wave 2 Teacher Recruitment Script– – SPRING 2016 36

All Follow Up Scripts to Principals and Teachers 36

APPENDIX C – Secondary Data Elements 37

Teacher and School Administrative Data Variables List 37

NM TEACH Reflect Data Variables List 37

Principal and Teacher Contact Information 37

APPENDIX D – Technical Working Group Suggestions 38

APPENDIX E – Educational Sciences Reform Act 40

APPENDIX F – IRB Approval 41

APPENDIX G –Study Participants’ Consent 46

Principal Wave 1 and Wave 2 Consent 46

Teacher Wave 1 and Wave 2 Consent 46

APPENDIX H - Confidentiality Form and Affidavits 47

OVERVIEW

The U.S. Department of Education (ED) requests clearance for the recruitment materials and data collection protocols under the OMB clearance agreement (OMB Number [IES to complete]) for activities related to the Regional Educational Laboratory Program (REL). ED, in consultation with SEDL, is planning a clustered randomized evaluation in New Mexico to test the effectiveness of materials intended to improve the feedback that principals provide in one-on-one conferences to their teachers about their classroom instruction. The study includes an impact analysis and an implementation analysis.

New Mexico Public Education Department (NM PED) staff have identified the topic of principal feedback to teachers as an area where New Mexico needs assistance. In particular, NM PED staff have received comments from principals that they do not feel adequately prepared to provide teachers with feedback about their instructional practices. The NM PED has committed to ensure that every student has equitable access to an effective principal and teacher every day they are in school. Although NM PED recognizes that the conferences are likely an area in the overall evaluation system in need of improvement, it has limited resources and time to focus on the post-observation conference step in the evaluation cycle. Therefore, any potential interventions to address this issue must be easy to implement and require few resources.

This impact study will examine whether an enhanced feedback guide (Conversation Protocol), relative to business-as-usual guidance to principals and teachers, achieves the following:

1. improves the structure and content of the principal-teacher feedback conversation,

2. improves perceptions that the feedback delivered to teachers is useful

3. better targets guidance to teachers’ regarding professional development that is tied to their formal classroom observation scores

4. improves quality of teacher instruction as measured by subsequent formal observation ratings on the state’s tool, called NM TEACH, and

5. increases student achievement and state standardized tests.

To examine these outcomes, ED contractors will invite all public school principals of charter and non-charter schools in the state to participate in the study. Blocking by school district, a randomly selected half of principals who agree to participate (along with teachers in their school who also agree to participate in the study) will be assigned to the treatment group. The remaining half of principals (along with teachers in their school who also agree to participate in the study) will be assigned to the control group.

In summary, the study will compare principals and teachers in the treatment and control conditions, which are detailed in Table 1.

Table 1. Treatment and Control Conditions

	Treatment group	Control group
Principals
NM PED-sponsored professional development for principals offered during the summer, of which 1.5-2 hours were devoted to feedback to teachers. Training conducted in regional in-person professional development sessions led by the state’s contractor, Southern Regional Education Board (SREB). All principals in the state are required to participate as a part of their job requirements.		
List of documents to bring to each feedback conversation	
24-item checklist to use during each feedback conversation.	
5-minute video in which principal testifies about her experience using the checklist.	
Business-as-usual guide to remind principals about “5 stages of feedback” described in NM PED-sponsored principal professional development to use for feedback conversations		
Teachers
Business-as-usual guide: A document the includes a reminder to teachers about their right to feedback within 10 calendar days of formal observation under the NM TEACH Educator Effectiveness System, and a link to the NM PED provided FAQ on teacher evaluation (http://ped.state.nm.us/ped/NMTeach_FAQ.html)		
List of documents to bring to each feedback conversation	
24-item checklist to use during each feedback conversation	
5-minute video in which teacher testifies about her experience using the checklist	

ED contractors will request that the treatment group principals and control group principals do three things: (1) disseminate the relevant guide (and a link to the video, where relevant) to all school leaders and to all teachers in his/her building, (2) use the guide and view the video, where relevant (along with all other schools leaders in the building who observe teachers) with 100% of the teachers that they formally observe during school year 2015-2016, and (3) complete a baseline survey in fall 2015 and a follow-up survey in spring 2016. ED contractors will request treatment group teachers and control group teachers to do two things: (1) use the guide they received from ED contractors (and view the video, where relevant) in the feedback conversations they have with their school leader(s), and (2) complete a baseline survey in fall 2015 and a follow-up survey in spring 2016.

Impact Research Questions

The evaluation will examine the impact of providing a post-classroom observation feedback checklist and a video to principals and teachers on principal, teacher, and student outcomes. To do so, this evaluation will use a cluster RCT design with random assignment at the school level to address the following impact and implementation questions.

Does provision of the feedback checklist and video relative to the business as usual guidance:

1. improve the content and structure of post-observation conference according to principals and teachers?

2. improve principals’ and teachers’ perceptions of the utility of post-observation conference?

3. increase the amount of time it takes to complete the post-observation conference?

4. cause principals to recommend and teachers to take professional development that is aligned to needs identified in the formal observations?

5. improve the quality of teachers’ subsequent instructional practices as measured by principals’ formal classroom observation scores?

Implementation Fidelity Research Questions

ED contractors will also examine the implementation of the Feedback Checklist relative to the business-as-usual guidance by analyzing responses to survey questions regarding the type of guidance used. The implementation portion of the study will address the following research questions:

6. How extensively do principals and teachers in the treatment and control groups report using the form of guidance they were assigned?

7. How extensively do principals and teachers in the treatment and control groups report using the form of guidance they were not assigned (i.e. cross-overs)?

Exploratory Research Questions

ED contractors will also perform exploratory analyses examining the association of the enhanced feedback protocol on subgroups of teachers and principals.  In addition, we will also examine whether provision of the treatment guide improves subsequent student performance on state standardized tests. Although the design of the study allows for causal analysis of these questions, for reasons of limited sample size within New Mexico and limited time for the treatment or control condition to influence behavior and student achievement, we will likely have insufficient statistical power to detect differences. Nevertheless, these questions have high policy relevance for NM PED, so we include them to provide insights for follow-on studies and for NM PED’s improvement of subsequent iterations of NM TEACH observations and feedback. The exploratory research questions are:

6. Does the provision of the checklist to principals and teachers increase student achievement relative to business-as-usual guidance regarding feedback?

7. Is there suggestive evidence that the effects of providing the checklist varies according to the following subgroups of principals and to the following subgroups of teachers:

1. extensive users of the checklist – i.e., teachers who have received three versus two formal observations and principals who have used the checklist in schools that have a greater than average number of teachers within the given district

2. personal beliefs and attitudes about NMTEACH evaluation system

3. accountability pressure (as measured by prior year’s school accountability grade)

4. those principals who have received training on the NMTEACH Observation Rubric

5. those who work in schools that teachers rate as collaborative and supportive

6. those with greater qualifications, including more years of experience and certification type

7. those who teach core subjects versus non-core subjects

8. those who work in elementary versus middle or high schools

9. teachers in the top and bottom quartile of the NM TEACH summative effectiveness score distribution

10. principals in the top and bottom quartile of the NM TEACH School Leader effectiveness score distribution

Beyond subgroup analyses, ED contractors will also gather information about the perceptions users have of the treatment checklist and the control guide.

6. What are principals’ and teachers’ perceptions of the checklist and videos versus business-as-usual feedback guidance? For example, do they report that it is easy to use, burdensome, formulaic, lengthy, useful for creation of professional development plans, and helpful in causing teachers to commit to instructional changes?

7. What are the main reasons why principals report not using the form of guidance they received? checklist or the business-as-usual guidance?

The project consists of data collection from NM PED and participating principals and teachers who work in New Mexico public schools. Specifically, in this OMB clearance package, ED is requesting clearance for the following data collection approaches:

• Recruitment materials for participating principals and teachers

• Extant administrative records data collections from NM PED

• Two waves of web-based surveys of principals and teachers in treatment and control schools.

ED believes that the data collections for which clearance is being requested represent the bare minimum necessary to assess the impact and implementation of the Conversation Protocol.

A. JUSTIFICATION

1. Circumstances Necessitating the Data Collection

New Mexico is committed to ensure that every student has equitable access to an effective principal and teacher. As a part of this mission, the NM PED has committed to “implementing a rigorous, uniform [classroom] observation protocol, providing immediate constructive feedback [to teachers], using meaningful student data, and other multiple measures [that] will provide valuable information to aid the personal development and growth of each teacher and principal.” NM PED undertook the task of reforming their teacher evaluation system starting in 2011. In 2012-2013, New Mexico piloted a new teacher evaluation system called NM TEACH that is mandated in all public schools starting in the 2013-2014 school year. Under NM TEACH, teachers are evaluated each school year based on three primary components:

1. evidence of student achievement improvement (such as value added scores on state and end of course exams),

2. average scores on the state-developed teacher observation rubric (NM TEACH), and

3. locally selected measures from a menu of options approved by the state (e.g., teacher attendance or student surveys).

These components are combined into a summative evaluation score, with 50% of the summative evaluation score based on improved student achievement, 25% based on the observation of teacher practice, and the final 25% based on the locally selected measures. Regarding the second component of teacher evaluations, NM PED requires that a principal or a designated school leader observe his or her teachers formally two or three times per year (with 20 minute observations), and informally throughout the school year (with several short “walk-throughs” lasting 3-5 minutes each). For formal observations, the school leader must rate the teacher using the NM TEACH Observation Rubric—a single rubric that applies to all grade levels and subjects taught. The combined scores from the 2-3 required formal observations comprise the second element of teachers’ ratings.

As the teacher evaluation cycle is conceived, a critical step to improving instruction through the revised teacher evaluation system is the required conference between the teacher and the school leader that takes places within 10 days after each of the formal classroom observations. In this conference the school leader reviews his or her classroom observation rubric scoring with the teacher and makes recommendations for improvement and professional development. Note that this study will only recruit one type of school leader—principals—and we therefore only refer to principals when describing the conduct of formal observations and feedback below.¹

In the research literature, much attention has been paid to the development of the classroom formal observation rubrics and the extent to which teacher observations align with other measures of student learning.² However, NM PED, like the broader field, has paid much less attention to the next step in the annual cycle of the teacher evaluation: the feedback conversation in which the principal conveys to the teacher the results of the scored observation. There is a dearth of scientifically rigorous research about effective feedback, and there are few resources dedicated to teaching principals how to convey the results of the classroom observations in a manner that helps teachers improve their instruction and leads to improved student outcomes.

The teacher feedback process is perhaps the least developed link in new teacher evaluation systems. The majority of academic attention has been devoted to the mechanics of the value added scores and the relative weight of multiple measures (Mihaly et al. 2013; Bill and Melinda Gates Foundation, 2012; McCaffrey, Lockwood, Koretz, Louis, and Hamilton, 2004), as well as the development of observation rubrics and their alignment with other measures of student learning (Bill and Melinda Gates Foundation, 2012; Pianta and Hamre, 2009; Hill, Kapitula and Umland, 2010). Yet the quality and clarity of the feedback is the primary means by which teacher evaluation systems would allow teachers to recognize the aspect of their practice that needs improvement and could encourage teachers to seek out appropriate professional development and to improve teachers’ instruction. It is therefore a critical point in the teacher evaluation cycle to study.

The research literature provides some limited evidence that teachers improve their instruction when they receive professional learning opportunities that are ongoing and closely connected to teachers’ curriculum and instruction (Supovitz and Turner, 2000; Correnti, 2007; Correnti and Rowan, 2007), as well as those that are aligned with what school districts value (Penuel et al., 2007; Garet et al., 2001). Given that teacher-principal conferences are meant to occur throughout the year and directly reflect what teachers are doing in their classrooms, those conferences could thus be an important mechanism for encouraging both teacher growth and directing teachers to appropriate professional development. Most recently, Hill (2009) provides evidence that professional development remains largely ineffectual for improving practice, often coming in the form of short-term, low-quality training that teachers find difficult to apply in their classrooms. If tightly linked to teachers’ observed classroom practices, recommended professional development could become increasingly relevant to teachers’ instructional practices. (Little, 1993; Ball and Cohen, 1999; Wilson and Berne, 1999).

To date, only two studies have examined the feedback given to teachers by principals, and both are based on the evaluation system in Chicago Public Schools. The University of Chicago Consortium on Chicago School Research (CCSR) found that Chicago Public Schools’ expectations for principal-teacher conferences pre- and post-observation did not align with principal and teachers’ actual practices (Sartain et al., 2011). In that CCSR study, only 10% of the questions principals asked of teachers reflected high expectations and required deep reflection about instructional practice and 65% of questions required limited response and focused on affirmation of the principal’s views. Additionally, principals tended to talk for 75% of the conference, and teachers only 25%. Overall, Chicago Public School principals reported a need for guidance on how to provide actionable feedback, which is precisely what this study aims to do for New Mexico principals. A more recent study by CCSR (Sporte et al., 2013) of the same teacher evaluation system found that administrators list the provision of useful feedback to teachers as an area where principals need professional development.

The theories employed in our intervention comport with the main lessons from a broad literature providing guidance on the most effective methods and features for delivering job performance feedback and for conducting performance appraisals. In the field of clinical supervision, Heckman-Stone (2003) provides some guidance for the proper structure of performance evaluation and feedback based on observations from supervision of students in the clinical practices. In providing feedback, she recommends that feedback standards should be agreed upon and understood by both parties and be provided regularly to build a strong relationship and avoid misunderstanding. Additionally, positive feedback should be provided prior to negative evaluations, and that any feedback should be properly contextualized into specific skill areas so that a supervisee is clear on what skills are being assessed and how they are being assessed. The standard professional development that the New Mexico PED offers in the summer to all principals about NM TEACH covers some, but not all, of these principles. The business-as-usual guidance to principals does not spell out ways to offer positive feedback before negative feedback, does not contextualize feedback so that the supervise is clear on what skills are being assessed (i.e., no reference to the NM Observation Rubric), and does not include a discussion of how teachers are being assessed.

In both health and education research, several simple, low-cost interventions such as the one studied here have proven effective. Most relevant to this study, the distribution of a one-page checklist of tasks before surgery with common-sense items on it such as nurses and surgeons should introduce themselves before surgery and correctly time the use of antiseptics had the extremely large impact of reducing surgical complications by 36% and deaths by 47% (Haynes et al., 2009 in Yeager & Walton, 2011). The author of the surgery checklist, Atul Gawande, claims that the surprisingly large effects of the checklist is due to health care professionals simply (and erroneously) assuming that surgeons and nurses were doing all the items on the checklist without verifying that fact. New Mexico PED staff with whom the SW REL researchers have spoken profess a similar assumption that principals and teachers will carry out common-sense steps in feedback conversations (e.g., review in detail the ratings on the observations on NM TEACH Observation Rubric, offer both positive and negative feedback, mutually develop a set of next steps for the teacher where the professional development explicitly ties to the scores on the Observation Rubric) but admit that they have no way to verify that assumption. The limited research on feedback within these new teacher evaluation systems like NM TEACH suggest that feedback conversations do not adhere to the seemingly common-sense guidelines provided by NM PED-sponsored professional development or by the Carnegie checklist.

Yeager and Walton (2011) review six education studies that have had substantial, lasting impacts on student achievement even though the interventions are short, provide information-only, do not teach academic content but instead seek to influence students’ beliefs and attitudes about school. The authors deduce two lessons from the interventions in the six studies: (1) the interventions use students’ perspectives to influence student behavior, and (2) they use “stealth tactics” like requiring fewer hours of study participants time to receive the session, not stigmatizing students, and not being didactic. Our study builds off of this work on psychological interventions by examining a low-cost, brief intervention tailored to New Mexico that could theoretically improve the quality of feedback for teachers.

Given the critical role of principal feedback in teacher evaluation systems and the limited research about it, the study proposed here is a cluster randomized controlled trial evaluation to estimate the impact of providing principals and their teachers with a detailed checklist and video suggesting how to structure the formative observation feedback conversation—as compared to a reminder to principals about the 5 steps for feedback outlined in the required state-sponsored professional development session —on the content and structure of the post-observation conference, the perceived quality of the feedback itself, the quality and take-up of professional development for teachers, teacher practice as measured by subsequent formal classroom observation scores, and, by extension, student achievement. The combination of the Conversation checklist and the video are intended to be a low-cost intervention—an intervention that primarily consists of information—that capacity-constrained state departments of education could theoretically implement with success.

As stated in the overview, the study proposed here is a cluster randomized controlled trial evaluation to estimate the impact of providing principals and their teachers with about a checklist to follow during the feedback conversations regarding teachers’ instruction relative to the minimal guidance offered currently to principals about effective feedback. Blocking by school district, a randomly selected half of principals who agree to participate in the study (along with teachers in their school who also agree to participate in the study) will receive the Conversation Protocol, which contains an introduction about the importance of feedback, a feedback conversation checklist to be used during the post-observation conference, and a link to a video testimonial in which a principal or a teacher describe their experience using the checklist. The checklist is a customized version of the Teacher Feedback Conversation Protocol provided in “Strategies for Enhancing the Impact of Post-Observation Feedback for Teachers,” developed by coauthors at the Carnegie Foundation for the Enhancement of Teaching (Myung and Martinez, 2013). While this version of the Conversation protocol has not been used in New Mexico, The Vision Network, located in Delaware, has used the Carnegie version of the checklist with principals in the 2013-2014 school year and will again use it in the 2014-2015 school year. The designer of the checklist, Myung, also reports that other schools have piloted the use of the checklist, but we are not aware of its widespread usage as an official tool adopted by districts or state education agencies.

To conduct a placebo randomized controlled trial, ED contractors will also provide a business-as-usual guide to control group principals and control group teachers participating in the study. The control guide for principals will list the five steps for feedback discussed in the summer 2014 professional development attended by NM principals. This will mimic “business-as-usual” for principals in New Mexico, which is professional development only provided to principals and not to teachers about feedback. The control group principals and teachers will not receive a link to the video nor will they receive the checklist. Meanwhile, to mimic “business-as-usual” for teachers, control group teachers will receive a short PDF with content from the state NM PED website reminding them of their right to a post-observation feedback conversation within 10 calendar days of the observation.

The treatment guide (called the Conversation Protocol) will go beyond business as usual by providing a brief introduction explaining the importance of a feedback, providing a video testimonial, and, most importantly, providing a detailed checklist for the feedback session. Checklists have been shown to have large impacts on practice in other settings (Haynes et al., 2009). ED contractors have applied the following four lessons from other effective simple, low-cost, information-only interventions in designing this study:

1. For the treatment group, systematize the information provided to bridge both the temporal gap between in-person professional development and feedback conversations and the conceptual gap between theory and practice by offering an easy-to-follow checklist for both principals and teachers to break down the feedback conversation into discrete steps for use before and during the feedback session.

2. For the treatment group, present a video (one video for principals; and second video for teachers) about the checklist from a principal’s (teacher’s) perspective to minimize their resistance to the message, rather than have professional development consultants or state or district staff persons tell principals and teachers what to do.

3. Avoid stigmatizing either the treatment or control group by framing the study as trying out two alternatives for improving feedback to teachers.

4. Keep the information brief to respect principals’ and teachers’ busy schedules, and because interventions requiring fewer hours of time have proven more effective than longer lasting interventions in several relevant education studies (Yeager & Walton, 2011).

The Conversation Protocol conforms to the schedule of formal observations and the required elements of NM TEACH Observation Rubric and is structured based on the Carnegie Protocol with the following designated sections:

1. Open warmly and get down to business. (Thanks for meeting with me. Could you remind me of the lesson goal? Let’s clarify what we want to accomplish.)

2. Focus on what’s going well. (What worked? In addition, here’s what I saw.)

3. Identify challenges facing the teacher. (What were some things you feel could have done better? It sounds like the challenges are ….)

4. Generate ideas for addressing the challenges. (Let’s approach these one by one…)

5. Prioritize next steps.

6. End positively (Was this conversation helpful? Thanks for your time and insights.)

The theory of change, presented in Figure 1, emphasizes the links in the chain that might be expected to connect principals’ feedback with changes to teacher instruction and, subsequently, to student learning. We envision that the process starts with a principal’s formal observation of a teacher’s classroom lesson. Any expected impacts of that observation on teacher instruction and student learning would then depend upon whether the teacher and principal used the guide both to reflect on the instructional strengths and challenges in the lesson and discuss those strengths and challenges in a way that led to actionable and useful feedback to the teacher during the teacher-principal conference. Then, subsequent links in the chain depend upon the teachers’ positive perceptions of the teacher-principal conference and a desire to seek out the recommended professional development (PD), the quality of the PD itself and the extent to which the teacher is able to utilize the PD to improve his/her instruction.

Figure 1 – Theory of Change

ED contractors propose to collect both primary and secondary data to understand the impact of providing the Conversation Protocol to principals and teachers on principals and teachers’ perceptions about the quality of feedback, teachers’ professional development, teachers’ instructional quality as measured by classroom observation scores and student achievement. The primary data will consist of a principal and a teacher Web-based baseline and follow-up survey (for four total survey instruments). The secondary data consists of formal observation scores by school leaders of teacher practice as well as principal, teacher and school characteristics and student achievement scores collected by NM PED.

Table 2 shows the timeline for all three data collection activities described in this OMB package.

Table 2. Data Collection Timeline

Data Collection	Purpose	Requesting OMB Clearance?	May / June 2015	May/ June 2016
Extant data for recruitment from New Mexico Public Education Department	Email addresses and first and last name for principals and teachers to recruit them into study via email (by April 1, 2015)	Yes	X
Extant administrative school and student data from New Mexico Public Education Department	Teachers’ NM TEACH Observation Rubric scores (RQ5) Student achievement data from spring state standardized test (RQ 9) Description of study schools, principal qualifications, teacher qualifications, and student population (RQ10)	Yes	X	X
Principal surveys	Perceptions of post-observation conferences (RQ1, 2) Behaviors in post-observation conferences with teachers (RQ3, 4, 7, 8, 10) Use of checklist and guidance (RQ 11, 12)	Yes	X	X
Teacher surveys	Perceptions of post-observation conferences (RQ1, 2) Behaviors in post-observation conferences with teachers (RQ3, 4, 7, 8, 10) Use of checklist and guidance (RQ 11, 12)	Yes	X	X

See Appendix A for the teacher and principal survey instruments, and Appendix B for the Recruitment Materials. Appendix C contains the secondary data elements to be provided by NM PED to ED Contractors. Appendix D summarizes the suggestions received from ED’s contractors for this study and the responses to those suggestions. The data collection involved in this study is authorized by the Educational Sciences Reform Act (ESRA) of 2002. Please see Appendix E for the ESRA. Appendix F provides the IRB approval by Human Subjects Protection Committee at the research organization carrying out the study. Appendix G provides the written consent forms for study participants.

2. How, by Whom, and for What Purpose the Information Is to Be Used

ED’s contractor for REL Southwest will analyze the data to be collected through this study using statistical models and procedures that are preapproved by the Institute of Education Sciences (IES). The contractor will then summarize the findings in a report that will undergo review for quality and relevance by the National Center for Education Evaluation and Regional Assistance’s (NCEE’s) external review contractor. After the report has undergone IES review, findings will be presented to members of the NM PED (the primary audience) and published through IES for educators and education researchers (secondary audience).

The primary audience for this study is NM PED, and other state and local agencies considering ways to improve feedback from principals to teachers. The results from all of the research questions will be used by New Mexico to decide whether to adopt the checklist and video as a mandatory or recommended element of teacher evaluation. The study will also offer detailed information to NM PED about teachers’ use of professional development (PD), their perceptions of it, the PD topics they pursue, and PD recommended to them. NM PED currently has no way to gather this information, because PD is arranged for and determined by local school districts in the state. In addition, the study will provide NM PED with detailed feedback from principals and teachers about the state’s teacher evaluation system, supports for it (or lack thereof), and suggestions for improvement. Answers to the exploratory and implementation research questions could help NM PED fill gaps in their knowledge regarding district and school practices in professional development and teacher feedback.

More generally, the results of the study will provide concrete evidence to state policymakers regarding the utility of low-cost, easy to implement conversation checklists, scripts, and testimonial videos if they can encourage teachers both to seek out appropriate professional development opportunities and improve their instruction. Should this New Mexico Guide improve teachers or principals’ description of the structure and content of the feedback conversation and perceptions of the feedback process or subsequent classroom observation scores, it suggests that other localities may benefit from developing customized checklists built around their evaluation system but incorporating the core theories of effective performance feedback. If the checklist and video do not influence then it suggests that the checklist and video do not add value to the pre-existing professional development about classroom observation, and an examination of more intensive interventions is warranted.

To provide this information to the primary audience and the secondary audience, ED’s contractor must collect the four forms of data described in Table 2. Namely, the contractor must collect extant data from NM PED that includes:

• teacher and principal contact information for ED’s contractor to recruit participants into the study;

• principals’ NM TEACH Observation Rubric ratings of teachers’ classroom to determine if classroom ratings improve following distribution of the treatment and control guide (research question 5);

• student-level achievement data to determine if student achievement improves in treatment and control schools following distribution of the guide (research question 9); and

• school-level, teacher-level, and principal-level characteristics to determine if those characteristics moderate take-up and perceptions of the treatment and the control group guide (research question 10).

We estimate that NM PED will devote a total of 20 hours to provide ED’s contractors with contact information and then another 105 hours from NM PED for school-, teacher-, and principal-level data.

The contractor must also administer a principal survey and a teacher survey at two time points to determine if there were changes in study participants’:

• Perceptions of the post-observation conferences following receipt of the treatment and control guide (research questions 1 and 2).

• Behaviors in the post-observation conferences following receipt of the treatment and control guide (research question 3 and 4).

• Perceptions and use of the guidance they received (research questions 6, 7, 8, 11, and 12).

The surveys will take up to 30 minutes to complete, and they will be administered to all study participants in spring 2015 and again in spring 2016. For the purposes of this OMB package, it is assumed that 600 principals and 3,000 teachers will complete the spring 2015 and the spring 2016 surveys.

3. Use of Information Technology

The data collection will be conducted in a manner that utilizes technology to reduce respondent burden and improve accuracy. Where feasible, REL SW researchers will collect all possible data from administrative sources rather than through primary data collection. For example, electronic school records on teachers, principals and schools will be gathered from NM PED and transferred directly to ED contractors using secure file transfer procedures, and thus reduce the number of questions posed to teachers and principals in their surveys. Clear instructions will be provided on the method of data transfer from NM PED to ED Contractors.

For data that can only be obtained directly from principals and teachers, ED contractors will employ MMIC™ (Multimode Interviewing Capability), a comprehensive information system developed by RAND, to administer the surveys.³ MMIC is used to manage the whole data collection process from questionnaire design, sample management, and fieldwork monitoring. Using MMIC, ED contractors will email to study participants a link to online surveys. To reduce the burden on respondents, the MMIC software is flexible, and allows survey respondents to participate using a multitude of devices like computers, PDAs, and Smart phones, and to switch between devices while completing the survey. When requested, questionnaires will be transmitted to and from the respondent by fax. A telephone number to a staffed MMIC help desk and electronic mail address are included in the questionnaire if anyone has questions. These procedures are designed to minimize the survey burden on respondents.

4. Efforts to Avoid Duplication of Effort

In an effort to avoid duplication of effort, this study will use extant administrative records where possible to understand the impact of the Conversation Protocol. As described in Table 2, ED’s contractors will collect principals’ rating scores of teachers on NM TEACH Observation Rubrics from secondary data sources collected by NM PED, rather than ask principals to provide this information in surveys. As a part of this same secondary data request, ED’s contractors will also collect school-level characteristics such as size, level (elementary, middle, high), accountability status, teacher-level characteristics such as degree earned, race, gender, and job title, principal-level characteristics such as degree earned, race, gender, and job title, and student-level characteristics like student achievement to minimize the length of surveys administered directly to principals and teachers. The only data collected that will be unique to this study are the web-based principal and teacher survey data, which are also described in Table 2.

5. Sensitivity to Burden on Small Entities

The use of administrative records will reduce the burden on school educators by ensuring that only the minimum amount of original data is requested from schools in order to meet the objectives of this study. Aside from surveys emailed directly to participating principals and teachers, ED’s contractors will not contact schools to request additional data.

6. Consequences of Less Frequent Collection and Obstacles to Burden Reduction

The Education Science Reform Act of 2002 states that the central mission and primary function of the regional education laboratories is to support applied research and provide technical assistance to state and local education agencies within their region (ESRA, Part D, section 174[f]). If the proposed data were not collected, REL Southwest would not be fulfilling its central mission to serve the states in the region and provide support for evidence-based research. The research questions and the intervention contemplated in this study respond to the concerns raised by NM PED, which is a constituent member of the REL Southwest. If the proposed data were not collected, NM PED and, by extension, other SEAs would not know whether a low-cost informational intervention such as this one has an impact on the quality of principal feedback to teachers, and by extension academic outcomes.

This is a one-time study (i.e., not recurring) and therefore periodicity is not addressed.

7. Circumstances Requiring Special Information Collection

There are no special circumstances.

8. Federal Register Announcement and Consultation Outside the Agency

a. Federal Register Announcement

A 60-day Federal Register Notice was published on October 16, 2014 by IES. There have been no public comments received. A 30-day notice was published on [DATE TO BE COMPLETED BY IES].

b. Consultations Outside the Agency

The evaluation team has consulted the Technical Working Group (TWG) of researchers with expertise in large-scale randomization studies and impact evaluation that was formed to review studies under the Southwest REL. We have consulted the TWG on the overall study design, data collection and survey instruments. Major recommendations from the TWG are included in Appendix D.

Members of the TWG include:

Dan Goldhaber, Ph.D.

Director, CALDER (National Center for Analysis of Longitudinal Data in Education Research)

Vice President, American Institutes for Research (AIR)

Director, Center for Education Data & Research (CEDR), University of Washington Bothell

Co-Editor, Education Finance and Policy

3876 Bridge Way N, Suite 201

Seattle, WA 98103

Ph: 206-547-1562

Fax: 206-547-1641

E-mail: dgoldhab@uw.edu

Geoffrey Borman, Ph.D.

Professor of Education, University of Wisconsin—Madison

Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research Training Program

Senior Researcher, Consortium for Policy Research in Education.

348 Education Building
1000 Bascom Mall
Madison, WI 53706-1326

Ph: 608-263-3688

Fax: 608-265-3135

E-mail: gborman@education.wisc.edu

Johannes M. (Hans) Bos, Ph.D.

Vice President and Program Director, International Development, Evaluation, and Research (IDER) Program

American Institutes for Research

2800 Campus Drive, Suite 200
San Mateo, CA 94403

Ph: 650-843-8100

Fax: 650-843-8200

E-mail: jbos@air.org

W. Steven Barnett, Ph.D.

Board of Governors Professor and Director of the National Institute for Early Education Research

Rutgers University

73 Easton Avenue
New Brunswick, NJ  08901

Ph: 848-932-4350 x23132

Fax: 732-932-4360

E-mail: sbarnett@nieer.org

c. Unresolved Issues

There are no unresolved issues.

9. Payments or Gifts to Respondents

ED contractors propose to provide teachers and principals a $25 gift certificate upon completion of each of two 30-minute surveys (baseline survey before the intervention, and follow-up survey after the intervention), for a total of $50 per person. These amounts were set based on the hourly wage rate of principals ($44.23/hour) and teachers ($32.52/hour) and the estimated length of the survey. Incentives will be distributed electronically (i.e., a link to a gift card) after respondents complete the surveys.

The only other data for the study will be obtained from NM PED, which has already agreed in concept to provide such data and will not be compensated for the time required to draw together the data files for transmission to ED’s contractor. As a constituent member of the REL Southwest, NM PED identified this topic as of interest of them, and thus the study is being done at their request.

10. Assurances of Confidentiality

The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant federal regulations and requirements. The Southwest REL will be following the new policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183 requires “All collection, maintenance, use, and wide dissemination of data by the Institute” to “conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment.

All survey responses will be kept confidential, and will only be used for the purpose of the study. No one at the school, district, or the state will have access to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. The project has been approved by RAND’s Human Subjects Protection Committee, which serves as RAND’s Institutional Review Board (IRB00000051) to review research involving human subjects. RAND is registered with the Office of Human Research Protection (OHRP) as a research institution (IORG0000034). RAND's Federal wide Assurance for the Protection of Human Subjects (FWA00003425, effective until July 1, 2018) serves as our assurance of compliance with federal regulations.

In addition, for student information, the data collection efforts will ensure that all individually identifiable information about students, their academic achievements, their families and information with respect to individual schools, shall remain confidential in accordance with section 552a of Title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act. The study will also adhere to requirements of subsection (d) of section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony.

ED contractors will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released publicly. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution type (e.g., elementary schools vs. middle schools) but not to any individually identifiable information. No individually identifiable information will be maintained by the study team upon study completion.

Respondents will be advised that all information identifying them and the school will be kept confidential, to the extent possible. All members of the study team have obtained their certification on the use of human subjects in research. The following safeguards are routinely employed at RAND, the contractor that will execute this study, to carry out confidentiality assurances:

• All employees at ED’s contractor working on this project will sign a confidentiality pledge emphasizing its importance and describing their obligations under it (please see Appendix H for the confidentiality pledge).

• All research projects that have access to identifiable private or proprietary data need to have a Data Safeguarding Plan reviewed and approved by RAND’s Human Subjects Protection Committee. The Data Safeguarding Plan includes information on who is responsible for data safeguarding, the types of sensitive information to be transferred and stored, the mode of data transfer, client and respondent agreements, disclosure risks, audit and monitoring plans, and the procedures to be employed for data safeguarding.

• Any electronic transmission and sharing of individually identifiable data will be encrypted. This procedure will prevent anyone without permission to access and enter the data system

• Access to the data shall be limited to the minimum number of individuals necessary to achieve the approved purpose and to those individuals on a need-to-know basis only.

• Identifiable data will be stored in a locked container when not in use. We will store original and derivative data files only on disks (e.g. servers, local hard disks) that are not routinely backed up. We will keep all hardcopy materials containing sensitive data in a locked file cabinet when not in use.

• When no longer needed, we will discard sensitive output in a shredder or sensitive-waste container. We will destroy all individual linkages to data after a respondent ceases participation in the project.

Also, the REL study team has submitted to the NCEE security officer a list of the names of all people who will have access to respondents and data. The contractor, on behalf of ED, will track new staff and staff who have left the study and ensure that signatures will be obtained or clearances revoked, as necessary.

ED’s contractor will make every effort to maintain confidentiality of the data, and that in no instance will responses or data be made available except in in aggregate statistical form. The following statement will appear on all letters to respondents on data collection:

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district, school, or individual. The contractor will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.

11. Justification for Sensitive Questions

Teachers completing the survey will be asked about the frequency, structure and content of the post-observation feedback conference, the type of professional development courses they were recommended to take and completed, the quality of the professional development courses offered by the school district, their opinions about the effectiveness of the school leadership and the teacher evaluation system. Teachers will also be asked more sensitive questions, such as to rate their own performance and to rate the quality of the school leadership. These questions are sensitive because teachers would risk embarrassment or damage to their reputation or employability were their answers to be revealed. As noted above, all responses will be kept confidential and the data safeguarded. The questions regarding teachers’ own ratings of performance are important to anchor the principal as a strict or lenient rater, and the questions regarding the quality of school leadership are an important control variables to establish the school environment. Principals completing the survey will be asked about the structure and content of the post-observation feedback conference, the type and quality of professional development offered by the school district, the training they received on the observation rubric, and their opinions about the teacher evaluation system. Principals will also be asked to answer similar sensitive questions such as to rate the quality of his/her teachers. While all responses will be kept confidential, these questions are important to establish the environment of the school.

In addition, participants will be informed that their responses are voluntary, and they may decline to answer any question.

12. Estimates of Hour Burden

There are three components for which ED’s contractor has calculated hours of burden for this clearance package: recruitment activities, extant data provided by the state to ED’s contractor, and survey data collected from study participants by ED’s contractor. Table 3 shows the hourly burden overall and for all three components. The total burden associated with this study, across three study years, is 5,352.75 hours, with an annualized burden of 1,784.25 hours over three years. The recruitment burden is 1,107.75 hours, the extant data collection burden is 105 hours, and the survey data collection burden is 4,140 hours. The annualized number of responses is 10,788 (for a total of 32,364 across all three years).

Table 3. Total Estimated Hourly Burden

Instrument	Person Incurring Burden	Number of respondents	Responses per Respondent	Total Responses	Hours per Response	Total Burden Hours
Recruitment
Principal and Teacher Contact Information (name, email)	State Data Manager	1	3	3	10	30
First Principal contact (e‑mail)	Principal	855	1	855	0.05	42.75
Round 1 Follow-up for nonresponding Principal	Principal	500	1	500	0.05	25
Round 2 Follow-up for nonresponding Principal	Principal	400	1	400	0.05	20
Round 3Follow-up for nonresponding Principal	Principal	300	1	300	0.05	15
First Teacher contact (e‑mail)	Teacher	6,000	1	6,000	0.05	300
Round 1 Follow-up for nonresponding Teacher	Teacher	5,000	1	5,000	0.05	250
Round 2 Follow-up for nonresponding Teacher	Teacher	4,500	1	4,500	0.05	225
Round 3Follow-up for nonresponding Teacher	Teacher	4,000	1	4,000	0.05	200
Subtotal	---	21,556	---	21,558	10.4 (across responses)	1,107.75
Extant Data Collection
Teacher and School Administrative Data	State Data Manager	1	3	3	15	45
Teacher Observation Score Data	State Data Manager	1	3	3	20	60
Subtotal	---	1	---	6	35 (across responses)	105
Survey Data Collection
Principal Consent	Principal	600	1	600	0.15	90
Teacher Consent	Teacher	3,000	1	3,000	0.15	450
Wave 1 Principal Survey	Principal	600	1	600	0.5	300
Wave 2 Principal Survey	Principal	600	1	600	0.5	300
Wave 1 Teacher Survey	Teacher	3,000	1	3,000	0.5	1,500
Wave 2 Teacher Survey	Teacher	3,000	1	3,000	0.5	1,500
Subtotal	---	10,800	---	10,800	2.3	4,140
Totals	---	32,357	---	32,364	47.7 (across responses)	5,352.75

Burden for Study Recruitment

Our recruitment strategy will focus on both New Mexico public school principals and New Mexico public school teachers. ED’s contractor will conduct recruitment in a top-down approach, with principal recruitment preceding teacher recruitment. Our burden table is a conservative estimate of the highest potential burden with this recruitment approach, but ED’s contractor anticipates that fewer districts and schools will actually be involved in recruitment activities.

First, ED’s contractor will collect contact information for principals and teachers from the state data manager at NM PED, with a second collection 2 months later to ensure that the information is up to date, at a total burden of 20 hours for NM PED. Principals will be contacted by email, with an estimated burden of 3 minutes to read and respond to the email (0.05 hours). We will re-contact non-responding principals up to three times via email and then twice by letter mailed to their school. The target sample size for initial contact with principals is 855 principals with an estimated response rate of 70 percent (600 principals). Once the principals have been enrolled in the study, we will then contact up to 10 teachers in the school by email, with an estimated burden of three minutes per teacher per email to read and respond to the email. We will re-contact non-responding teachers up to three times via email and then twice by letter mailed to their school. The target sample size for initial contact with teachers is at most 6,000 teachers, with an estimated sample of at most 3,000 teachers consenting to participate in the study.

Burden for Extant Data Collection

The total estimated burden for NM PED to compile and transmit secondary data to ED’s contractors is 105 hours. This calculation assumes one data manager works a collective total of 2.625 weeks on compiling the data request.

Burden for Survey Data Collection

The total estimated burden for survey data collection for the impact and implementation study is 4,140 hours. Principals and teachers will each be consented into the study, and will be surveyed two times, once at baseline, and a second time one year later.

Total Cost to Respondents

The total cost to respondents for the two components of this study—recruitment activities and survey data collection—is provided in Table 4. The total respondent cost associated with this study is approximately $184,590. The annualized cost for each year of the three-year study is $61,530. The recruitment cost is $37,152; the respondent cost for the data collection is $147,438.

Table 4. Total Cost to Respondents

Instrument	Person Incurring Burden	Total Burden Hours	Hourly Wage Rate	Total Respondent Cost
Recruitment
Principal and Teacher Contact Information (name, email)	State Data Manager	20	$45	$900
First Principal contact (e‑mail)	Principal	42.75	$44.23	$1,891
Round 1 Follow-up for nonresponding Principal	Principal	25	$44.23	$1,106
Round 2 Follow-up for nonresponding Principal	Principal	20	$44.23	$885
Round 3Follow-up for nonresponding Principal	Principal	15	$44.23	$663
First Teacher contact (e‑mail)	Teacher	300	$32.52	$9,756
Round 1 Follow-up for nonresponding Teacher	Teacher	250	$32.52	$8,130
Round 2 Follow-up for nonresponding Teacher	Teacher	225	$32.52	$7,317
Round 3Follow-up for nonresponding Teacher	Teacher	200	$32.52	$6,504
Subtotal	---	1,097.75	---	$37,152
Survey Data Collection
Principal Consent	Principal	90	$44.23	$3,981
Teacher Consent	Teacher	450	$32.52	$14,634
Wave 1 Principal Survey	Principal	300	$44.23	$13,269
Wave 2 Principal Survey	Principal	300	$44.23	$13,269
Wave 1 Teacher Survey	Teacher	1,500	$32.52	$48,780
Wave 2 Teacher Survey	Teacher	1,500	$32.52	$48,780
Teacher and School Administrative Data	State Data Manager	45	$45	$2,025
Teacher Observation Score Data	State Data Manager	60	$45	$2,700
Subtotal	---	4,245		$147,438
Totals	---	5,342.75		$184,590

Note: The hourly wage rates used to calculate total respondent cost for principals are based on the salary schedule of Albuquerque High school principals with more than 10 years of experience (http://www.aps.edu/human-resources/salary-schedules/salaries/spe-salary-schedule). For teachers, the salary is based on the salary schedule of Albuquerque Level III teachers with a doctorate (http://www.aps.edu/human-resources/salary-schedules/salaries/at3-salary-schedule). For the data manager hourly wage rates are based on ED’s contractor’s knowledge of New Mexico wage rates.

13. Estimated Total Annual Cost Burden to Respondents or Recordkeepers

There are no start-up costs associated with this collection.

14. Estimated Annualized Cost to the Federal Government

The total cost to the federal government for work conducted over all three years is $1,133,000 and the estimated annualized cost to the federal government for each year of the study is $377,667.

15. Program Changes or Adjustments

This is a new study.

16. Plans for Tabulation, Publication, and Schedule for Project

a. Tabulation Plans

All results for REL rigorous studies will be made available to the public through peer-reviewed evaluation reports that are published by IES. The datasets from these rigorous studies will be turned over to the REL’s IES project officer.

After the study report is finalized, ED’s contractor will prepare restricted-use data files in accordance with NCES standards. These files will contain all the primary survey data collected for the study with all personal identifiers removed. Thorough documentation will be provided for each data file, including a detailed codebook and explanations of the unit of observation, weights, and methods for handling missing data. These data will become IES restricted-use data sets requiring a user’s license that is applied for through the same process as NCES restricted-use data sets. Even the ED contractor would be required to obtain a restricted-use license to conduct any work with the data beyond the original evaluation.

b. Publication Plans

All results for REL studies are made available to the public through peer-reviewed reports that are published by IES. The data sets from these studies will be turned over to the REL’s IES project officer. These data may become IES restricted-use data sets requiring a user’s license that is applied for through the same process as National Center for Education Statistics restricted-use data sets (see http://nces.ed.gov/pubs96/96860rev.pdf for procedures related to obtaining and using restricted-use data sets). The REL contractor also would be required to obtain a restricted-use license to conduct any work with the data beyond the original report.

The answer to RQ1 – RQ4 will indicate whether the treatment guide relative to the control guide improved or otherwise changed principals’ and teachers’ perceptions and execution of post-observation conferences. To analyze the impact of the feedback protocol on teacher and principal outcomes, we will examine differences in outcomes between teachers or principals who were in schools that were randomly assigned to receive the treatment guide with their counterparts in the control group who were assigned to receive the control guide. The data for the evaluation is hierarchical, with students and teachers nested within school (or principals), nested within districts. Because units within a group are not statistically independent, ED’s contractors will use hierarchical linear modeling to account for the statistical dependence of the error terms. Randomization will occur at the principal/school level within district, and ED’s contractors will stratify the random assignment by school level. ED’s contractors will estimate the intent-to-treat effect in a two-level model for principals and a three-level model for teachers.

Using similar modeling techniques, the answer to RQ5 will indicate whether the treatment guide, relative to the control guide, improved principals’ subsequent ratings of teachers’ classroom instructional techniques. For each outcome in the impact analysis (RQ1-5), ED’s contractor will first construct the outcome measure by using exploratory factor analysis, taking the measure directly from the survey or using the measure from Observation Rubric administrative data. ED’s contractor will then estimate six regressions, one for each of the following outcomes:

• Good Practices Index (measure of post-observation feedback conference content and structure)

• Feedback Conversation Perceptions Index (measure of the perceived quality of feedback)

• Average number of minutes it takes to complete the post-observation conference

• PD is tailored to needs identified in formal classroom observation

• Fraction of targeted Professional Development that was recommended by the principal and taken by the teacher

• Domain level Observation Rubric score

• Observation proficiency level

Analysis for RQ6 and RQ7 will compare all principals and all teachers in the treatment group to their counterparts in the control group. These questions will assess the fidelity of implementation, the enacted contrast in service between the control and treatment group, and will help NM PED understand the likely proportion of users who would take up a guide where NM PED to issue one to school staff.

Analysis for RQ8 through RQ9 will go beyond the hypothesized first-order impacts of guidance on post-observations conferences and classroom instruction to understand what factors, if any, moderate the use of the guidance (RQ 9) and whether the feedback guidance impacts student achievement, which is the ultimate outcome of interest. However, RQ 8 and 9 are framed as exploratory because ED’s contractors anticipate that sample size limitations and a single school year of implementation may be insufficient to detect statistically significant effects at the principal- and teacher-subgroup level and on student achievement. These research questions will employ the same statistical model specifications for the impact research questions, with the introduction of interaction terms to assess differences in responses of principals and teachers according to their membership in the subgroup of interest.

Finally, descriptive analyses for RQ10 and 11 will provide NM PED feedback about how the two types of guidance might be improved in future use and barriers to its use. This will allow NM PED, and potentially other capacity-constrained State Education Agencies, to adopt a revised version of feedback guidance and professional development for users of the guidance that will further enhance feedback and subsequent classroom instruction.

No responses or data will be reported for individual staff members, students, or schools. Reported data will contain no fewer than four cases per reported table cell to protect confidentiality and mask individually identifiable data.

c. Project Time Schedule

The timeline for the activities in this project, including data collection, analyses and reporting are in Table 5.

Table 5 – Project Time Schedule

Documentation of Institutional Review Board approval	May 1, 2014
Draft Office of Management and Budget (OMB) package	August 15, 2014
Submit 60 day FRN	TBD
Submit 30 day FRN	TBD
Draft proposal approved by ED	March 12, 2014
Final proposal approved by ED	August 15, 2014
Expected OMB Clearance Date	April 1, 2015
Start Teacher Recruitment	April 15, 2015
Start Principal Recruitment	April 15, 2015
Complete Teacher Recruitment and Collect Wave 1 Survey	August 7, 2015
Complete Principal Recruitment and Collect Wave 1 Survey	August 7, 2015
Complete random Assignment	August 15, 2015
Distribute Conversation Protocol to Principals at Treatment Schools and Principal Control Guide to Principals at Control Schools	August 15, 2015
Distribute Conversation Protocol to Teachers at Treatment Schools and Teacher Control Guide to Teachers at Control Schools	1st day of School, Fall 2015
Collect First round of NM TEACH Observation rubric scores	December 1, 2015
Collect Second round of NM TEACH Observation rubric scores	February 1, 2016
Collect Wave 2 Teacher Survey	May 1, 2016
Collect Wave 2 Principal Survey	May 1, 2016
Collect spring 2016 student achievement data from NM PED	June 15, 2016
Complete Data Analyses	December 31, 2016
Draft Making Impact Report	March 1,2017
Revised Making Impact Report	May 1,2017
Draft Stated Briefly Report	July 1, 2017
Final Making Impact Report approved by ED	September 1,2017
Revised Stated Briefly Report	September 1,2017
Restricted Use Files	October 15, 2017
Final Stated Briefly Report approved by ED	November 30, 2017

17. Display of Expiration Date

Approval not to display the expiration date for OMB approval is not requested.

18. Exceptions to the Certification Requirement

No exceptions to the certificate statement are being sought.

REFERENCES

Ball, D.L. and Cohen, D.K. (1999). Developing practice, developing practitioners: Toward a practice-based theory of professional education. In L.D. Hammond and G. Sykes (Eds.), Teaching as the learning profession: Handbook of policy and practice, pp. 3-32. San Francisco: Jossey-Bass.

Bill and Melinda Gates Foundation. (2012). Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle, WA: Author. Available: http://www.metproject.org/downloads/MET_Gathering_Feedback_Practioner_Brief.pdf

Correnti, R. (2007). An empirical investigation of professional development effects on literacy instruction using daily logs. Educational Evaluation and Policy Analysis, 29(4): p. 262-295.

Correnti, R. and Rowan, B. (2007). Opening up the black box: Literacy instruction in schools participating in three comprehensive school reform programs. American Educational Research Journal, 44(2), p. 298-338.

Garet, M.S., Porter, A.C., Desimone, L., Birman, B.F. and Yoon, K.S. (2001). What makes professional development effective? Results from a national sample of teachers. American Journal of Educational Research, 38(4): p. 915-945.

Haynes, A. B., Weiser, T. G., Berry, W. R., Lipsitz, S. R., Breizat, A. H. S., Dellinger, E. P., ... & Gawande, A. A. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine, 360(5), 491-499.

Heckman-Stone, C. (2003). Trainee Preferences for Feedback and Evaluation in Clinical Supervision. Clinical Supervisor, 22(1), 21-34.

Hill, H.C. (2009). Fixing teacher professional development. The Phi Delta Kappan, 90(7): 470-476.

Hill, H.C., L. Kapitula, and K. Umland. (2011). A validity argument approach to evaluating teacher value-added scores. American Educational Research Journal, 48(3): p. 794-831.

Little, J.W. (1993). Teachers’ professional development in a climate of educational reform. Educational Evaluation and Policy Analysis, 15(2): 129-151

McCaffrey, D.F., Lockwood, J.R., Koretz, D., Louis, T.A. and Hamilton, L. (2004). Models for value-added modeling of teacher effects. Journal of Educational and Behavioral Statistics, 29(1): 67-101.

Mihaly, K., McCaffrey, D. F., Staiger, D. O., and Lockwood, J. R. (2013). A composite estimator of effective teaching. Seattle, WA: Bill and Melinda Gates Foundation.

Myung, J. and Martinez, K. (2013). Strategies for enhancing the impact of post-observation feedback for teachers. Stanford, CA: Carnegie Foundation for the Advancement of Teaching.

Pianta, R.C. and Hamre, B.K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38(2): pp. 109-119.

Penuel, W.R., Fishman, B.J., Yamaguchi, R. and Gallagher, L.P. (2007). What makes professional development effective? Strategies that foster curriculum implementation. American Journal of Educational Research, 44(4): p. 921-958.

Sartain, L., Stoelinga, S.R., and Brown, E.R. (2011). Rethinking teacher evaluation in Chicago: Lessons learned from classroom observations, principal-teacher conferences, and district implementation. Chicago: Consortium on Chicago School Research.

Sporte, S.E., Stevens, W.D., Healey, K., Jiang, J. and H. Hart. 2013. Teacher evaluation in practice: Implementing Chicago’s REACH Students. Chicago: Consortium on Chicago School Research.

Supovitz, J.A. and Turner, H.M. (2000). The effects of professional development on science teaching practices and classroom culture. Journal of Research in Science Teaching, 37(2), p. 963-980.

Wilson, S.M., and Berne, J. (1999). Teacher learning and the acquisition of professional knowledge: An examination of research on contemporary professional development. Review of Research in Education, 24: 173-209.

Yeager, D. S., & Walton, G. M. (2011). Social-psychological interventions in education They’re not magic. Review of Educational Research, 81(2), 267-301.

APPENDIX A – Teacher and Principal Survey Instruments

Spring 2015 School Leader Survey

See attachment

Spring 2016 School Leader Survey

See attachment

Spring 2015 Teacher Survey

See attachment

Spring 2016 Teacher Survey

See attachment

APPENDIX B – Principal and Teacher Recruitment Materials

Wave 1 Principal Recruitment Script– SPRING 2015

See attachment

Wave 1 Teacher Recruitment Script– – SPRING 2015

See attachment

Wave 2 Principal Recruitment Script– – SPRING 2016

See attachment

Wave 2 Teacher Recruitment Script– – SPRING 2016

See attachment

All Follow Up Scripts to Principals and Teachers

See attachment

APPENDIX C – Secondary Data Elements

Teacher and School Administrative Data Variables List

See attachment

NM TEACH Reflect Data Variables List

See attachment

Principal and Teacher Contact Information

See attachment

APPENDIX D – Technical Working Group Suggestions

In January 2014, Southwest REL researchers obtained comments from two TWG reviewers: Dan Goldhaber and Geoff Borman. This Appendix summarizes in bullet form the conceptual suggestions that Southwest REL researchers received. Line edits are omitted from this summary. The researchers response are noted under each bullet prefaced by “RESPONSE.”

• The power of the intervention to produce changes in practice may be compromised if there are limited quality PD opportunities for principals to recommend in order for teachers to improve their instruction. This could lead the investigators to incorrectly conclude that the guide was not useful for changing practice and, eventually, achievement.
  
  

  • RESPONSE: While ED’s contractor agrees that the quality of the PD modules could be an important moderator of the impact of the Guide, the decentralized acquisition or creation of PD by districts or by schools within districts makes it impractical to know ahead of time what the PD will be on offer during the school year 2015-2016. However, ED’s contractor believes that this lack of a standardized menu of pre-approved/pre-screened PD sessions is representative of the norm and thus a good test of the efficacy of a feedback guide in other states besides New Mexico.

• Unless it is vital for treatment group teachers to receive the treatment guide, it seems inadvisable because the threat of cross-over seems quite strong.

  • RESPONSE: ED’s contractor decided to distribute the guide to teachers as well as to principals for the following two reasons: (a) To minimize Hawthorne effects, ED’s contractor intends to trump principals’ survey responses regarding usage of the guide with teachers’ responses about having seen or ever used the feedback checklist. (b) ED’s contractor intends to disseminate the checklist to teachers because the point of the treatment guide is to structure a two-way, mutual conversation instead of principals talking at teachers (as is currently the common practice). ED’s contractor agrees this increases the risk that the treatment guide will be shared with control group school staff. ED’s contractor think the best idea to discourage contamination and reduce John Henry effects is to disseminate a “Guide” to all principals, blinding them to their status in treatment or control group, which is addressed in the next point.

• The provision of [only] the treatment guide and the reliance on self-reported survey data (and principal-reported classroom observation scores) could cause Hawthorne effects due to both principals and teachers expecting improvement solely because they have been offered a new treatment. Providing a "guide" to all participating principals could, potentially, minimize the risk of cross-over (because everyone gets something), and would certainly address the issue of a Hawthorne effect.

  • RESPONSE: ED’s contractor agrees and has adopted this suggestion.

• None of the outcome measures are well-validated outcomes, and it is not clear whether the authors intend to construct composite measures from the surveys or whether outcomes will be based on single items.

  • RESPONSE: ED’s contractor agreed and has included student achievement as an outcome (a well-validated outcome) and developed composite indexes derived from survey questions.

• The estimated recruitment rates of 65% for principals and 60% of teachers seem relatively high.

  • RESPONSE: ED’s contractor has powered the study for a range of participation rates, between 50 and 65 percent for principals and between 50 and 60 percent for teachers, and demonstrated that for the research design the drop-off is very small from the upper to the lower ends of these ranges (approximately 0.02 to 0.03 in MDES). ED’s contractor based the upper end of our anticipated participation rates on prior experience with a similar efforts in New York City Public Schools, where there was also a direct connection to regular school responsibilities (although somewhat weaker than in the case of the current study) and participation was well over 60% (McCombs, Kirby, and Mariano, 2009).

• The teacher inclusion criteria for the study could bias the results of the study. 

  • RESPONSE: ED’s contractor agreed, and has dropped eligibility screeners for either principals or teachers. The only remaining eligibility criteria is that the study participant work in NM public schools at the time of the survey.

• The study would be considerably better if it had value added components.

  • RESPONSE: ED’s contractor agreed that value-added would be a highly useful, validated outcome. Therefore, ED’s contractor added student achievement as an outcome measure.

• It would be beneficial to include charter schools.

  • RESPONSE: This suggestion has been enacted.

APPENDIX E – Educational Sciences Reform Act

See attachment

APPENDIX F – IRB Approval

NOTIFICATION OF INITIAL APPROVAL

From:

Carolyn Tschopik

To:

Kata Mihaly  - PI

Cc:

Heather Schwartz - Primary Contact
Lou Mariano - Study Team
Heather Schwartz - Study Team

ID:

2014-0239

Title:

New Mexico teacher feedback

Description:

The above study has been approved and all requirements have been met. 

Approval of the study is for the period of 4/22/2014 to 4/21/2015.

Action Required:

No action is required of you at this time. 

CURRENT STUDY REVIEW STATUS

Project Number:

There are no items to display

Funding Source:

Department of Education

Prime Recipient:

Other, Specify:
SEDL

RAND Unit(s):

Education

Human Subjects:

Yes

Type of Review:

Expedited

Component Approvals:

NM students

Procedure	Approved	Motion	Category
secondary data	Tue Apr 22 00:00:00 PDT 2014	Does not involve human subjects

principals & teachers who work in any NM public school

Procedure	Approved	Motion	Category
secondary data	Tue Apr 22 00:00:00 PDT 2014	Expedited - Approved	Expedited Category: 7 - Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)

principals & teachers whom we will recruit to pilot our surveys

Procedure	Approved	Motion	Category
Pilot test surveys	Tue Apr 22 00:00:00 PDT 2014	Expedited - Approved	Expedited Category: 7 - Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)

principals whom we will recruit to participate in study

Procedure	Approved	Motion	Category
Randomized Controlled Trial intervention	Tue Apr 22 00:00:00 PDT 2014	Expedited - Approved	Expedited Category: 7 - Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)
Survey	Tue Apr 22 00:00:00 PDT 2014	Expedited - Approved	Expedited Category: 7 - Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)

teachers whom we will recruit to participate in the study

Procedure	Approved	Motion	Category
Randomized Controlled Trial intervention	Tue Apr 22 00:00:00 PDT 2014	Expedited - Approved	Expedited Category: 7 - Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)
Survey	Tue Apr 22 00:00:00 PDT 2014	Expedited - Approved	Expedited Category: 7 - Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)

Contingencies:

Description	Date Due	Date Completed
There are no items to display

Assurance number:

FWA00003425

IRB number:

IRB00000051

Administrator:

Carolyn Tschopik

 
 

The HSPC is RAND's Institutional Review Board to review research involving human subjects, as required by federal regulations. RAND's "Federalwide Assurance for the Protection of Human Subjects" (FWA00003425, effective through July 1, 2018, at http://intranet.rand.org/groups/hspc/fwa.pdf ) serves as our assurance of compliance with the regulations of 16 federal departments and agencies. According to this assurance, the Committee is responsible for review regardless of source of funding.

 

APPENDIX G –Study Participants’ Consent

Principal Wave 1 and Wave 2 Consent

See attachment

Teacher Wave 1 and Wave 2 Consent

See attachment

APPENDIX H - Confidentiality Form and Affidavits

CONFIDENTIALITY AGREEMENT

Impacts of a Detailed Checklist on Feedback to Teachers

(RAND under Contract No. ED-IES-12-C-0012)

Safeguards for Individuals Against Invasion of Privacy: In accordance with the Privacy Act of 1974 (5 United States Code 552a), the Education Sciences Reform Act of 2002 (Public Law 107-279), the Federal Statistical Confidentiality Order of 1997, the E-Government Act of 2002 (Public Law 107-347), and the Computer Security Act of 1987, American Institutes for Research (RAND) and all its subcontractors are required to comply with the applicable provisions of the legislation, regulations, and guidelines and to undertake all necessary safeguards for individuals against invasions of privacy.

To provide this assurance and these safeguards in performance of work on this project, all staff, consultants, and agents of RAND, and its subcontractors who have any access to study data, shall be bound by the following assurance.

Assurance of Confidentiality

1. In accordance with all applicable legislation, regulations, and guidelines, RAND assures all respondents that their responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S. Code, § 9573].

2. The following safeguards will be implemented to assure that confidentiality is protected as allowable by law (20 U.S.C. § 9573) by all employees, consultants, agents, and representatives of RAND and all subcontractors and that physical security of the records is provided:

1. All staff with access to data will take an oath of nondisclosure and sign an affidavit to that effect.

2. At each site where these items are processed or maintained, all confidential records that will permit identification of individuals shall be kept in a safe, locked room when not in use or personally attended by project staff.

3. When confidential records are not locked, admittance to the room or area in which they reside shall be restricted to staff sworn to confidentiality on this project.

4. All electronic data shall be maintained in secure and protected data files, and personally identifying information shall be maintained on separate files from statistical data collected under this contract.

5. All data files on network or multi-user systems shall be under strict control of a database manager with access restricted to project staff sworn to confidentiality, and then only on a need-to-know basis.

6. All data files on single-user computers shall be password protected and all such machines will be locked and maintained in a locked room when not attended by project staff sworn to confidentiality.

7. External electronically stored data files (e.g., tapes on diskettes) shall be maintained in a locked storage device in a locked room when not attended by project staff sworn to confidentiality.

8. Any data released to the general public shall be appropriately masked such that linkages to individually identifying information are protected to avoid individual identification in disclosed data.

9. Data or copies of data may not leave the authorized site for any reason.

3. Staff, consultants, agents, or RAND and all its subcontractors will take all necessary steps to ensure that the letter and intent of all applicable legislation, regulations, and guidelines are enforced at all times through appropriate qualifications standards for all personnel working on this project and through adequate training and periodic follow-up procedures.

By my signature affixed below, I hereby swear and affirm that I have carefully read this statement and fully understand the statement as well as legislative and regulatory assurances that pertain to the confidential nature of all records to be handled in regard to this project, and will adhere to all safeguards that have been developed to provide such confidentiality. As an employee, consultant, agent, or representative of RAND or one of its subcontractors, consultants, agents, or representatives, I understand that I am prohibited by law from disclosing any such confidential information to anyone other than staff, consultant, agents, or representatives of RAND, its subcontractors, or agents, and Institutes of Education Science. I understand that any willful and knowing individual disclosure or allowance of disclosure in violation of the applicable legislation, regulations, and guidelines is punishable by law and would subject the violator to possible fine or imprisonment.

(Signature) (Date)

AFFIDAVIT OF NONDISCLOSURE

Impacts of a Detailed Checklist on Feedback to Teachers

(RAND under Contract No. ED-IES-12-C-0012)

[insert name]

[insert position]

Date of Assignment to Impacts of a Detailed Checklist on Feedback to Teachers: November 2013

RAND

1776 Main Street
Santa Monica, CA 90401-3208

I, [insert name], do solemnly swear (or affirm) that when given access to any Impacts of a Detailed Checklist on Feedback to Teachers databases or files containing individually identifiable information, I will not:

1. use or reveal any individually identifiable information furnished, acquired, retrieved or assembled by me or others, under the provisions of Section 183 of the Education Sciences Reform Act of 2002 (PL 107-279) and Title V, subtitle A of the E-Government Act of 2002 (PL 107-347) for any purpose other than statistical purposes specified in the NCES survey, project or contract;

2. make any disclosure or publication whereby a sample unit or survey respondent could be identified or the data furnished by or related to any particular person under this section could be identified; or

3. permit anyone other than the individuals authorized by the Commissioner of the National Center for Education Statistics to examine the individual reports.

(Signature)

(The penalty for unlawful disclosure is a fine of not more than $250,000 [under 18 U.S.C. 3571] or imprisonment for not more than 5 years [under 18 U.S.C. 3559], or both. The word "swear" should be stricken out wherever it appears when a person elects to affirm the affidavit rather than to swear to it.)

State of _____________________________

County of _______________________________

Subscribed and sworn/affirmed before me, ______________________, a Notary Public in and for

________________County, State of ________________________, on this date, ______________________.

___________________________________________

Notary Public

My commission expires: _____________________________.

1 The recruitment does not include all types of school leaders such as assistant principals or deans because it is a school-based decision which school leaders, if any, besides principals formally observe and rate teachers using the NM TEACH Observation Rubric. Therefore, among school leaders, ED contractors will recruit only principals into the study, and then ask participating principals to disseminate the guide they receive (whether the treatment or control guide) to all relevant school leaders in the building who also conduct formal classroom observations.

2 See, for example, Mihaly et al., 2013; Bill and Melinda Gates Foundation, 2012; Pianta and Hamre, 2009; and Hill, Kapitula and Umland, 2010.

3 For more information, see http://www.rand.org/labor/mmic.html