TSL_OMB_Part_A.revised.1.9.20

^{A nchor}

Design of an Impact Evaluation to Inform the Teacher and School Leader Incentive Program

Part A: Supporting Statement for Paperwork Reduction Act Submission

August 26, 2019

Submitted to:

U.S. Department of Education

Institute of Education Sciences

550 12th Street, S.W.

Washington, DC 20202

Project Officer: Elizabeth Warner
Contract Number: 91990018C004

Submitted by:

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543-2393

Telephone: (609) 799-3535

Facsimile: (609) 799-0005

Project Director: Phil Gleason
Reference Number: 50714

CONTENTS

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION 1

Justification 1

A1. Circumstances necessitating the collection of information 1

A2. Purpose and use of data 7

A3. Use of technology to reduce burden 13

A4. Efforts to avoid duplication of effort 14

A5. Methods of minimizing burden on small entities 14

A6. Consequences of not collecting data 14

A7. Special circumstances 15

A8. Federal register announcement and consultation 15

A9. Payments or gifts 16

A10. Assurances of confidentiality 17

A11. Justification for sensitive questions 18

A12. Estimates of hours burden 18

A13. Estimate of cost burden to respondents 20

A14. Annualized cost to the federal government 20

A15. Reasons for program changes or adjustments 20

A16. Plans for tabulation and publication of results 20

A17. Approval not to display the expiration date for OMB approval 22

A18. Exception to the certification statement 22

REFERENCES 23

APPENDIX A: DISTRICT INTERVIEW PROTOCOL

APPENDIX B: TEACHER LEADER APPLICANT BACKGROUND FORM

APPENDIX C: SCHOOL INFORMATION QUESTIONNAIRE

APPENDIX D: CONFIDENTIALITY PLEDGE

TABLES

A.1 Research questions 5

A.2 Major project activities 6

A.3 Data collection for the study 8

A.4 List of Technical Working Group members, their affiliation and relevant expertise. 16

A.5 Estimate response time for data collection. 19

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

This package requests clearance for the initial data collection activities to support an evaluation of the Teacher and School Leader Incentive Program (TSL). The Institute of Education Sciences (IES), U.S. Department of Education (ED) has contracted with Mathematica and its partners Public Impact, Applied Engineering Management Corporation, Decision Information Resources, and Dr. Jason Margolis to conduct this evaluation.

This study will include two evaluation components:

1. Descriptive study of TSL grantees’ programs. Data collection includes interviewing all districts included in FY17 TSL grants to obtain information on TSL grantees’ programs and experiences.

2. Implementation, impact, and cost-effectiveness study of designating one or more “teacher leaders” as coaches in schools. A random assignment study of this common TSL strategy will be conducted in non-TSL schools. Data will also be collected from TSL grantee schools on their implementation of the teacher leader role, in order to connect the impact findings from non-TSL schools to the TSL implementation experience.

TSL grants fund multiple strategies, and the strategies in the 2017 grants include support for improving human capital management systems (HCMS) and a range of strategies using the HCMS -- such as performance-based compensation, incentives to recruit or retain teachers, pipelines to hire and prepare effective new teachers, and career pathways opportunities like teacher leaders.  This variation makes it necessary to focus on one activity in order to do a useful, rigorous evaluation.

Supporting teacher leaders is the most common activity for recent grantees – and, in fact, it’s currently an emerging policy approach.  For this reason, the teacher leader strategy is the focus of this evaluation. IES worked closely with the program office to determine the study focus.  In addition in 2018, IES briefed the Hill on the planned study focus before moving forward with a procurement. An additional reason for focusing on teacher leaders in this evaluation is that other strategies supported by TSL have been addressed previously in other IES studies, including an evaluation focusing on performance-based compensation; an evaluation focusing on the formative assessment information provided by evaluation systems such as those supported by TSL; an evaluation focused on incentives for recruiting high-performing teachers to teach in high-need schools; and a new IES evaluation of residency programs, the feasibility of which is currently being assessed.

This package provides a detailed discussion of both evaluation components. However, the package only requests clearance for data collection associated with the first component and initial data collection for the second component. A separate package will be submitted at a later date for all remaining instruments and data collection activities, which will take place in SY2020-21.

Justification

A1. Circumstances necessitating the collection of information

a. Policy context and statement of need

Congress mandated that IES conduct an independent evaluation of the TSL program.¹ Specifically, the legislation called for the evaluation to “measure the effectiveness of the program in improving student academic achievement, the satisfaction of the participating teachers, principals, or other school leaders, and the extent to which the program assisted the eligible entities in recruiting and retaining high-quality teachers, principals, or other school leaders, especially in high-need subject areas.”

The Teacher and School Leader Incentive Program (TSL), the successor to the earlier Teacher Incentive Fund (TIF) program, supports a variety of strategies aimed at improving the quality of teaching and attracting and retaining effective educators. The first TSL grants awarded in 2017 included 14 grant awards and funded a number of strategies, with the mix of strategies supported varying across grantees.

In response to the legislative mandate to evaluate the TSL program, the first evaluation component addresses the need to understand the characteristics of districts that received TSL grants and the key strategies they are using to improve educator effectiveness and student achievement. Although a prior study (Chiang et al. 2017) provided information on the 2010 TIF grantees’ programs, the program has evolved since then. This component of the study will update our knowledge of the current grantees’ activities.

The focus of the second evaluation component arises from a need to assess effectiveness. It will focus on a single, common strategy because TSL grantees’ use of multiple and different strategies makes it difficult to estimate the combined impacts of all TSL-supported strategies in a way that can be interpreted and used for policy and program improvement. This strategy of focusing on a single, common strategy of grantees is part of an evidence-building strategy for the program that complements evidence on other aspects of the grant that have been previously evaluated.²

The second evaluation component will focus on the effectiveness of a commonly-used strategy—designating teacher leaders to provide coaching to other teachers. This teacher leader role strategy was selected as the study’s focus because 12 of the 14 TSL grantees fund teacher leader positions in their schools through their TSL grant.

The impact study will be conducted in schools not receiving TSL funding. The rationale for including non-TSL schools is two-fold. First, since most TSL grantees are currently funding teacher leader positions and have already started implementing their TSL programs (including teacher leader roles), rigorously assessing the impact of the strategy in the context of the grants is not feasible. In recent years, the teacher leader role also has grown as a strategy funded by Title II, Part A in ESSA.³ Thus, learning about the effectiveness of the teacher leader role strategy in a more general context is useful for TSL and other policy.

Although existing research offers little guidance on whether this teacher leader strategy improves teacher effectiveness and student achievement (Wenner and Campbell 2017), reasons to believe the strategy is promising include:

• Teacher leader roles provide a mechanism for high-performing teachers to share their knowledge and expertise with colleagues, enhancing the quality of their colleagues’ instruction (Margolis 2012; Mangin and Stoelinga 2008).

• Teachers may be more receptive to coaching from their peers than from non-teaching staff or outsiders. They may feel that coaching from a teacher leader may be more accessible, well-grounded in course content, and appreciate that they are all “on the same team” and that the teacher leaders understand the school and local context (Mangin and Stoelinga 2008).

• By providing opportunities for high-performing teachers to grow in their responsibilities and earn higher pay for doing so, teacher leader roles may improve the teacher workforce by attracting and retaining more effective educators (Workman and Wixom 2016; Sutcher et al. 2016).

• Using stipends to pay teacher leaders to take on coaching activities may be a more cost-efficient approach to providing individualized coaching than other approaches, such as creating a new coach position.

The second component of the evaluation will use random assignment to study the impacts (and implementation and cost-effectiveness) of the teacher leader role in non-TSL districts. We will identify a set of non-TSL districts that are not currently using teacher leaders and we will randomly assign half of the study schools in these districts to either implement the teacher leader role or not. The design of the teacher leader role implemented in the treatment schools will be based, in part, on teacher leader roles used by TSL grantees to facilitate the connection between the second evaluation component and the Congressional mandate to inform TSL. In addition, the evaluation will collect information on the implementation of the teacher leader role from both TSL schools (those schools implementing a teacher leader role funded through their TSL grant) and non-TSL treatment schools in the evaluation (those schools selected to implement a teacher leader role that will be supported as part of the impact study). This will enable the evaluation to further provide ED with contextual information to help interpret the extent to which findings from the impact evaluation will shed light on the effectiveness of the teacher leader role implemented by TSL grantees.

b. Treatment for the Impact Component

The second component, an impact evaluation, is designed to examine a model of a teacher leader role that is consistent with emerging research and reflects common features of TSL grantees’ programs based on our review of grantees’ TSL grant applications and interviews conducted with 9 TSL grantees in late 2018 and early 2019 whose programs included teacher leader roles.

The key aspects of the strategy are:

• Principals of treatment schools will identify two teams that will consist of all teachers in a given grade or grade-subject (for example, fourth grade math teachers) that the teacher leader will lead. The teams will be chosen from among teachers of math and ELA in grades 3 through 8. Each team will have no more than 5 to 6 teachers. Each teacher leader leads one team.

• Each treatment school will have two teacher leaders who will perform three key activities: (1) coaching of individual teachers, (2) general support for their teams, and (3) teaching their own classes.

• The principal will recommend two teachers on each team to be “high-priority” teachers who receive more coaching from the teacher leader than others.

• Treatment schools will implement the teacher leader role for two years – the 2020-2021 and 2021-2022 school years.

• The study team will provide support and monitor treatment schools’ implementation of their teacher leaders’ roles to ensure high-quality implementation.

The teacher leader role strategy could improve student achievement directly by improving educator effectiveness or indirectly by improving their satisfaction and retention (Figure A.1). Teacher leader roles may improve educator effectiveness by improving teachers’ practices and giving principals a tool with which to recruit high-performing teachers who seek career advancement and other teachers who value the support they provide. Teacher leader roles might improve the satisfaction of high-performing teachers because it provides an opportunity for career advancement without requiring the teacher to leave the classroom or seek an administrative position. It may also improve other teachers’ satisfaction, particularly novice and low-performing teachers who may benefit the most from the support provided by teacher leaders.

F
igure A.1. Logic model for potential impacts of teacher leader role for random assignment evaluation

c. Study design and research questions

This first evaluation component will address a variety of questions about TSL implementation and, in particular, will describe TSL grantees and their key program strategies (Table A.1, Research Questions (RQ) 1-3). We will obtain information on TSL grantees’ programs and experiences through interviews with all TSL grantee districts. We will conduct interviews at the district level (rather than with the grantee, if the grant covers multiple districts) because some districts within a grant may implement different programs.

The second evaluation component will examine questions about the implementation, impacts, and cost-effectiveness of a teacher leader role (Table A.1, RQs 4-7). To address these questions, we will recruit 100 schools in 10 non-TSL districts and randomly assign participating schools within each district so that one set of schools (the treatment schools) receives the treatment, while the remaining schools (the control schools) continue with business-as-usual. The random assignment evaluation will estimate the impact of having well-supported teacher leaders on educator and student outcomes, including whether it affects the retention and recruitment of other teachers, teachers’ and principals’ satisfaction, and students’ math and ELA achievement.

We will measure the impacts of teacher leader roles on three key groups of teachers and their students.

• Teacher teams and their students. To measure the impacts on all teachers and students who could be directly affected by a teacher leader, we will compare outcomes of the teacher leader teams in treatment schools with outcomes of teachers in the same grades and subjects in control schools.

• High-priority teachers and their students. High-priority teachers are those who principals believe can most benefit from additional support. To conduct this analysis, we must identify comparable teachers in control schools. For this reason, before random assignment, we will ask principals of all study schools (treatment and control) to assess their teachers’ potential to benefit from coaching and encourage principals to prioritize novice and low-performing teachers―two groups that TSL grantees target for support. We will compare the outcomes of the teachers identified as high-priority (and their students) in treatment and control schools.

• Teacher leaders and their students. Both teacher leaders and their students could be impacted by this role. To identify comparable teachers in control schools, before random assignment we will ask principals to identify teachers they believe are most likely to be selected as a teacher leader if their school is assigned to the treatment group. We will compare the outcomes of the teachers identified to be selected as a teacher leader (and their students) in treatment and control schools.

Table A.1. Research questions

Component 1: TSL grantee implementation: programs and experiences

1. What were the characteristics of 2017 TSL grantees?

2. What key strategies did FY 2017 TSL grantee districts use their TSL funds to support? To what extent did they use funds to:

   1. improve recruiting practices to attract a high-quality and diverse workforce?

   2. support the development and improvement of the district’s teachers or school leaders?

   3. improve practices designed to retain and reward high-performing educators?

   4. improve measures of educator effectiveness used to inform performance-based compensation decisions?

   5. improve other measures to inform human capital management decisions?

   6. improve districts’ data systems? 

1. For these key strategies supported by districts’ TSL funds, which activities did they consider were their primary way to achieve those goals? How did districts implement these activities?

   1. Was this a new activity or an extension/revision of an existing activity?

   2. Why did they make this a key way to accomplish the goals of their program activities?

   3. To what extent did grantee districts use measures of educator effectiveness or other data to implement these key strategies?

Component 2: Implementation of teacher leader roles

4. What are the experiences of principals and teachers at schools that implement a teacher leader role? Specifically,

   1. What characteristics did principals emphasize when selecting teacher leaders?

   2. How did teacher leaders carry out their responsibilities and target supports to specific teachers?

   3. To what extent did teachers in treatment and control schools differ in the support they received?

   4. How does the teacher leader role implemented in the non-TSL schools in the impact study compare with the teacher leader role implemented in TSL grantee schools?

Component 2: Impacts and cost-effectiveness of teacher leader roles

5. What is the impact on student achievement of having teacher leaders?

6. What is the impact of teacher leaders on educator satisfaction, recruitment, and retention?

7. What is the cost-effectiveness of having teacher leaders compared with other strategies to improve student achievement?

Two important goals of the implementation analyses are to i) highlight overall descriptive information for the TSL program, including the teacher leader role, and ii) to provide context for the impact results. To meet these objectives, we will compare TSL grantee and districts participating in component 2, including district characteristics and aspects of their teacher leader roles. Documenting differences and similarities between TSL grantee districts and the non-TSL districts participating in component 2 and their implementation of the teacher leader role will help with interpretation of the second evaluation component impact results. It also has the potential to provide information for TSL program improvement.

A2. Purpose and use of data

Our proposed data collection activities will provide comprehensive information to describe the 2017 TSL grantees’ programs and experiences and the implementation and impact of teacher leader roles in non-TSL districts participating in the second evaluation component. Below we describe the planned data collection activities for the study and summarize them in Table A.3. At this time, we are only requesting clearance for the district interview protocol, teacher leader applicant background form, and the school information questionnaire. All other data collection activities will be part of a separate request.

a. Data collection under current clearance request

District interviews. To describe the key strategies used by TSL grantees to improve educator effectiveness as well as understand the specific activities they implemented, we will conduct interviews in spring 2020 and spring 2021 with all 25 districts covered by a 2017 TSL grant (Appendix A). The spring 2020 interview will be conducted in three stages. Using a single-stage interview would be challenging given the multiple and different TSL program strategies that grantee districts may use, since different follow-up questions may be needed for different strategies. Obtaining information in three stages will allow us to tailor the interviews so that they efficiently include the most appropriate questions. The stages are:

1. Initial email. The email will inform TSL grantee districts about the interview and ask one question about how they use the TSL grant funds to improve educator effectiveness. We are asking for this information in an email in order to give the district respondents time to organize the information to estimate the amount of funding from their TSL grant devoted to each of six general strategies (along with a seventh, “other” category; see Appendix A). We will also ask them to rank each strategy by its strategic importance, or the extent to which it contributes to the district’s overall goals for their TSL program. We will use this information to identify the top three strategies that the district is using to improve educator effectiveness.

2. Introductory interview. We will conduct a 15-minute phone call to learn more about the activities conducted by the grantee within the top three strategies the district identified in the email. For each of these top three strategies, we will ask them about the specific activities they are doing. For example, districts reporting retaining and rewarding effective teachers as one of their top three TSL strategies, we would ask about activities they might use to accomplish the strategy such provide additional pay to teachers based on performance (see Appendix Afor other possible activities to accomplish this strategy). We will also ask districts to identify their three highest-priority activities, or those that contribute most to their TSL goals, across all of the strategies discussed in this interview. The interviewers will use this information to tailor the questions to be covered in the in-depth interviews.

3. In-depth interview. Finally, we will hold a 45-minute follow-up call to get additional information about the three highest-priority activities identified in the introductory interview. This call will ask them to describe the highest-priority activities, including how they are implemented. The interview will also collect information on the extent to which districts use measures of educator effectiveness or related data to inform these high-priority activities, and whether TSL funds were used to create or refine these measures.

The spring 2021 interview will feature many of the same topics as the first, but will focus on experiences since the first interview, the challenges faced by districts, how they may have addressed those challenges, and the sustainability of their program.

Teacher leader applicant background form. Ideally, teachers selected for the teacher leader role will be the applicants with strong teaching and coaching skills. To examine the extent that this occurred, we will have principals complete a teacher leader applicant background form (Appendix B). Principals will rate applicants on multiple criteria, such as (1) a written application, (2) responses to in-person interview questions, (3) a demonstration modeling lesson in which applicants demonstrate their skills in modeling instructional practices, and (4) a demonstration feedback session in which applicants demonstrate their skills providing feedback to teachers. We will examine if principals rated teachers selected for the teacher leader role higher, on average, than unsuccessful applicants. The application form will also request information on teachers’ background characteristics, such as years teaching, that we will use to describe the characteristics of the teacher leaders, and the applicants’ scores on the district evaluation system. In spring and summer 2020, we will collect teacher leader applicant background forms (paper or electronic records) from principals of treatment schools for all applicants for teacher leader positions.

Table A.3. Data collection for the study

Data source	Purpose of data collection (RQ)a	Key information/outcomes	Sample(s)	Mode and timing	Clearance requested now or in future submission?
District interviews	Describe grantee programs and experiences (RQ2,RQ3)	Components of the grantee’s human capital management system; teacher leaders’ responsibilities, compensation, selection, training, and support; challenges encountered in administering the program and obtaining buy-in	25 districts (across 14 grantees) that participate in 2017 TSL grants	1-hour phone interviews in spring 2020 and spring 2021	Clearance requested now
Teacher leader applicant background form	Describe implementation in treatment schools (RQ4)	Candidates’ background characteristics; principals’ ratings of candidates’ written applications, in-person responses to interview questions, and demonstration of coaching and feedback skills; candidates’ scores on district’s evaluation system	Applicants to teacher leader positions in treatment schools	Paper or electronic forms requested from treatment principals in spring/ summer 2020	Clearance requested now
School information questionnaire	Describe implementation in treatment schools; estimate impacts of teacher leaders in treatment and control schools (RQ4,RQ5)	List of high-priority teachers for coaching; list of potential teacher leaders; grade (or grade-subject) for teacher leader team; schools’ staffing structure (departmentalized or self-contained), number of teachers in each grade-subject, and number of expected vacancies	All treatment and control schools	Paper form submitted by principals before random assignment in spring 2020	Clearance requested now
Teacher leader activity logs	Describe implementation of teacher leaders in treatment schools (RQ4)	Types of leadership activities that teacher leaders conduct and the teachers they support during their release time; practices on which each coaching cycle is focused and amount of time	Teacher leaders in treatment schools	Weekly web-based checklist in 2020–2021 and 2021-2022 school years	Clearance requested in future submission
Principal surveys	Describe grantee programs and experiences; describe implementation and estimate impacts of teacher leaders in treatment and control schools (RQ4,RQ6)	Types and frequency of coaching, mentoring, common lesson planning, and professional development in principal’s school; strategies for recruiting and retaining teachers; activities of teacher leaders; principal satisfaction	Principals of 100 randomly selected schools that receive TSL funds to support teacher leader roles (2021 survey only) Principals of treatment and control schools	30-minute web-based survey in spring 2021 and spring 2022	Clearance requested in future submission
Teacher surveys	Describe characteristics and implementation of teacher leader role in grantee schools; describe implementation and estimate impacts of teacher leaders in treatment and control schools (RQ4, RQ6)	Frequency of providing or receiving coaching, mentoring, common lesson planning, and professional development; satisfaction with job and colleagues, and feedback	200 teachers from 100 randomly selected schools that receive TSL funds to support teacher leader roles (2021 survey only) All study teachersb in treatment and control schools	30-minute web-based survey in spring 2021 and spring 2022	Clearance requested in future submission
District administrative records on teachers	Estimate impacts of teacher leaders in treatment and control schools (RQ5)	Student rosters of study teachers	Study teachers in treatment and control schools fall 2020 and fall 2021	Electronic records requested from districts in fall 2020 and fall 2021	Clearance requested in future submission
District administrative records on teachers	Describe implementation of teacher leaders in treatment schools; estimate impacts of teacher leaders in treatment and control schools (RQ4,RQ5, RQ6)	Teachers’ scores on their district’s evaluation system in 2019–2020; background characteristics; teaching assignments	Teacher leader applicants (if necessary)c; teachers in treatment and control schools in spring 2020 (before random assignment); study teachers in treatment and control schools fall 2020, fall 2021,and fall 2022	Electronic records requested from districts in fall 2022	Clearance requested in future submission
District administrative records on students	Estimate impacts of teacher leaders in treatment and control schools (RQ5)	Students’ math and reading test scores in 2019-2020; students’ math and reading test scores in 2020-2021 and 2021–2022 linked to their teachers; student background characteristics and attendance	Students assigned to study teachers in treatment and control schools in fall 2020; students enrolled in study schools in fall 2020 who are projected to be in the grades covered by study teams in fall 2021, along with new students assigned to study teachers in fall 2021	Electronic records requested from districts in fall 2022	Clearance requested in future submission

^a RQ references the research question(s) from Table A.1 that the data collection activity contributes to answering.

^b Within each treatment and control school paired together at the time of random assignment, study teachers will consist of (1) teacher leaders and the teams they support in the treatment school, and (2) teachers in the control schools who teach the same grades and subjects as those on the teams in the treatment school.

^c We will only request teacher leader applicants’ evaluation scores and background characteristics if the principal did not provide the information on the teacher leader applicant background form. We will not request teacher leader applicants’ teaching assignments.

School information questionnaire. We will use the school information questionnaire for several purposes (Appendix C).

• To help create matched pairs of schools for random assignment with similar context. We will ask principals (1) to identify the grade (or grade-subject) they would assign the teacher leaders to work with if their school is assigned to the treatment group; and (2) whether their teachers teach self-contained classes or if their school has departmentalized instruction.

• To identify high-priority teachers that might benefit from coaching. Collecting this information in both treatment and control schools will allow the study to assess whether teacher leader roles improve the effectiveness of high-priority teachers; those teachers principals believe would benefit the most from coaching by teacher leaders.

• To estimate the impact on teacher leaders and their students. We will ask principals to identify potential teacher leaders prior to conducting random assignment.

b. Data collection under future clearance request

Teacher leader activity logs. To monitor and describe teacher leader activities, teacher leaders will indicate what they did, whom they supported (specific teachers, the full team, a subset of the team), and the focus of the activity for each 30 to 60 minute block of time per week that teacher leaders were released from their classroom. We will also use these data to examine whether impacts may have been associated with the amount and content of the coaching they provided.

Principal surveys. We will administer a survey in spring 2021 to all principals of treatment and control schools and 100 randomly selected schools that receive TSL funds to support teacher leader roles for three main purposes:

1. To describe the implementation of the teacher leader role in treatment schools

2. To estimate the impact of teacher leaders on principals’ satisfaction and recruitment strategies

3. To compare the teacher leader role implemented by TSL schools and non-TSL treatment schools

We will also survey all principals of treatment and control schools again in spring 2022 to contribute to the analysis of implementation and impacts after two intervention years.

Teacher surveys. To learn about teachers’ perspectives about the type and amount of support they receive and to estimate the impact of teacher leader roles on teacher satisfaction, we will administer a survey in spring 2021 to teachers in TSL-supported schools and to teachers in treatment and control schools. We will administer a second round of the teacher survey in spring 2022 to teachers in the treatment and control schools to estimate impacts after nearly two years of implementing the intervention. For consistency with the survey sample in the impact evaluation component, all teachers in the TSL sample will teach English language arts or math in grades 3 through 8. As with the principal survey, we will use the information from teachers to describe implementation of the teacher leader role in treatment schools, estimate impacts on teacher-reported outcomes, and compare the teacher leader role implemented by TSL schools and non-TSL treatment schools.

District administrative records on teachers. We will collect the following district administrative data on teachers in treatment and control schools in fall 2020, fall 2021, or fall 2022:

• Student rosters of study teachers. In order to estimate the impact of teacher leaders on the student achievement of two key subgroups (students of high-priority teachers and those of the teacher leaders) we will collect student rosters in the fall of each implementation year (fall 2020 and fall 2021).

• Teachers’ district evaluation scores from the year before the intervention (the 2019–2020 school year). We will use these data to compare the effectiveness of teacher leaders with other teachers in their schools and compare the evaluation ratings of the teacher leader with the ratings of the teachers they support. We will also use this information to examine the correlation between the effectiveness of a school’s teacher leaders and their impacts on student achievement.

• Teachers’ school, grade and subject assignments. In order to examine the impact on the retention and recruitment of effective teachers, in fall 2022 we will collect information on teachers’ school, grade and subject assignments for teachers in treatment and control schools in spring 2020 (before random assignment), fall 2020, fall 2021, and fall 2022.

• Teachers’ demographic characteristics. To describe the study sample, we will collect information on teachers’ demographic characteristics (for example, age, gender and race), educational background (e.g. certifications, degrees, and scores on licensure or certification exams), and years of teaching experience. We will collect these data at the same time we collect data on teachers’ school, grade and subject assignments.

District administrative records on students. To estimate the impact of teacher leaders on student achievement, we will collect administrative data on students from each district in the random assignment evaluation. We will collect state test score data in reading and math from the baseline year (2019-2020) and each implementation year (2020-2021 and 2021-2022). We will use demographic, socioeconomic, and baseline test score data to describe the students in the study and compare the characteristics of students in treatment and control schools. In addition, we will control for students’ baseline characteristics and test scores in our impact estimation models to increase the precision of the estimates.

A3. Use of technology to reduce burden

The data collection plan is designed to obtain information in an efficient way that minimizes respondent burden, including the use of technology when appropriate. For example, the principal and teacher surveys will be web-based, which will enable respondents to complete the data collection instrument at a location and time of their choice. Sample member outreach materials provide instructions for accessing the web surveys. Built-in editing checks and programmed skips will also reduce the level of response errors and data retrieval callbacks. However, respondents will be able to respond to the survey by mail or phone if they prefer. As another example, we will ask districts to provide electronic copies of student and teacher records. While we will specify the required data elements, we will accept any format the district wishes to use, to reduce burden for them. To help ensure study participants’ confidentiality, districts will upload data files directly to a secure data site.

A4. Efforts to avoid duplication of effort

No similar evaluations are being conducted, and there is no equivalent source for the information to be collected. Moreover, the data collection plan reflects careful attention to the potential sources of information for this study, particularly to the reliability of the information and the efficiency in gathering it. The data collection plan avoids unnecessary collection of information from multiple sources. For example, student achievement will be measured using scores from state-administered student assessments, instead of administering an assessment as part of this study.

Information obtained from the surveys, teacher leader applicant background forms, school information questionnaires, and teacher leader activity logs is not available elsewhere.

A5. Methods of minimizing burden on small entities

No small businesses or entities will be involved as respondents.

A6. Consequences of not collecting data

The data collection plan described in this submission is necessary for ED to describe TSL grantees’ strategies for improving educator effectiveness and to conduct an impact evaluation on the effect of teacher leaders (a key component of many 2017 TSL grantees’ programs) on student achievement, teachers’ and principals’ job satisfaction, and principals’ ability to attract and retain effective teachers. The consequences of not collecting specific data are outlined below:

• Without the district interviews, we would not have the data to describe TSL grantee districts’ primary strategies for improving educator effectiveness, along with the specific activities they used to carry out these strategies. We would also not have the data to describe grantee districts and to understand the challenges of implementing the TSL program and how to improve the program.

• Without the teacher leader applicant background forms, we would not have the data to describe the teaching and coaching skills of applicants to confirm that they are among the strongest available at the study schools. We would also not have teacher background characteristics needed to describe the characteristics of teacher leader applicants.

• Without the school information questionnaire, we would not have the data needed to match pairs of schools for random assignment based on the grade (or grade-subject) the principal would select for the teacher leader to work with. We would also not be able to identify potential teacher leaders and high-priority teachers in control schools to estimate the impact of teacher leaders on these subsamples of teachers and their students.

• Without the teacher leader activity logs, we would not have data on teacher leadership activities, which will be used to describe the implementation of the teacher leader role in treatment schools.

• Without the principal surveys, we would not have the data to describe how schools implemented teacher leader roles and the challenges they encountered implementing these roles. We would not be able to describe principals’ characteristics and the impact of teacher leader roles on principal satisfaction and principals’ use of the roles to recruit and retain teachers. We would also not be able to compare the teacher leader roles implemented in treatment schools (which are funded by the impact study) and those in TSL schools (which are funded through a TSL grant).

• Without the teacher surveys, we would not have the data to describe how schools implemented teacher leader roles. We would not be able to describe the impact of teacher leader roles on teachers’ satisfaction with their job and colleagues, and the type and amount of support and feedback they receive on their teaching. We would also not be able to compare the teacher leader roles implemented in treatment schools and those in TSL schools.

• Without the district administrative records on teachers, we would not be able to assess whether principals chose teacher leaders with stronger evidence of effectiveness relative to other applicants and the teachers they support, whether teacher leader roles enabled schools to attract and retain more effective teachers, or estimate the impact of teacher leaders among subgroups of teachers and their students.

• Without the district administrative records on students, we would have to administer student assessments instead of using their state math and reading test scores. Without information on student characteristics, we would not be able to fully describe the study sample or verify the effectiveness of school random assignment. Teacher-student links are necessary to estimate the impact of teacher leaders on subgroups of teachers’ students, such as students of high-priority teachers and the teacher leaders.

A7. Special circumstances

There are no special circumstances.

A8. Federal register announcement and consultation

a. Federal register announcement

A 60-day notice to solicit public comments was published in the Federal Register, Volume 84, No. 169, page 45744-45745 on August 30, 2019.

The 30-day notice will be published to solicit additional public comments.

b. Consultations outside the agency

In formulating the intervention and evaluation design for this evaluation, the study team sought input from several individuals with expertise in teacher support and professional development, teacher quality, teacher policy, and evaluation methods. This input will help ensure the study is of the highest quality and that findings are relevant to policymakers, school districts, teachers, and principals. Table A.4 lists the individuals who have agreed to serve on the technical working group (TWG), their affiliation, and their relevant expertise.

Table A.4. List of Technical Working Group members, their affiliation and relevant expertise

Name	Affiliation	Area of expertise
		Teacher support and professional development	Teacher quality	Teacher policy	Evaluation methods
David Blazar	University of Maryland	X	X	X	X
Chelsea Collins	New Jersey Department of Education	X	X	X
Jim Kemple	New York University		X	X	X
Melinda Mangin	Rutgers University	X	X	X
Jeff Smith	University of Wisconsin			X	X
Eric Scott Taylor	Harvard Graduate School of Education	X	X	X	X
Suzanne Wilson	University of Connecticut	X	X	X
Tonya Wolford	School District of Philadelphia	X	X	X	X

A9. Payments or gifts

Incentives have been proposed for teachers participating in the study. The proposed amounts are within the incentive guidelines outlined in the March 22, 2005 memo, “Guidelines for Incentives for NCEE Evaluation Studies,” prepared for OMB. To maximize the success of our data collection effort we will provide incentives to teachers to offset their time and effort with completing the data collection activities. Incentives are also proposed because high response rates are needed to make the study findings reliable. Principals and teachers are the targets of numerous requests for data on a wide variety of topics from state and district offices, independent researchers, and ED. Although some districts will have solicited buy-in from school staff to participate in the evaluation, our recent experience with numerous school data collection efforts supports our view that obtaining district buy-in does not guarantee all staff will devote the time it takes to complete data collection activities, and monetary incentives increase the likelihood of their cooperation.

Principal and teacher respondent payment. To acknowledge the 30 minutes required to complete the principal and teacher surveys, we propose to offer a $30 incentive to principals and teachers who complete the survey.

A10. Assurances of confidentiality

Mathematica and its research partners will conduct all data collection activities for this study in accordance with relevant regulations and requirements, which are:

• The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a)

• The “Buckley Amendment,” Family Educational Rights and Privacy Act (FERPA) of 1974 (20 U.S.C. 1232g; 34 CFR Part 99)

• The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98)

• The Education Sciences Reform Act of 2002, Title I, Part E, Section 183

The research team will protect the confidentiality of all data collected for the study and will use it for research purposes only. The Mathematica project director will ensure that all individually identifiable information about respondents remains confidential. All data will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required. All members of the study team having access to the data will be trained and certified on the importance of confidentiality and data security. When reporting the results, data will be presented only in aggregate form, such that individuals, schools, and districts are not identified. Included in all voluntary requests for data will be the following or similar statement:

“Responses to this data collection will be used only for research purposes. The report prepared for this study will summarize findings across the sample and will not associate responses with a specific district, school, or individual. We will not provide information that identifies you, your school, or your district to anyone outside the study team, except as required by law. Additionally, no one at your school or in your district will see your responses.”

The following safeguards are routinely used by Mathematica to maintain data confidentiality, and they will be consistently applied to this study:

• All Mathematica employees are required to sign a confidentiality pledge (Appendix D) that emphasizes the importance of confidentiality and describes employees’ obligations to maintain it.

• Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by random, study-specific identification numbers.

• Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.

• Access to computer data files is protected by secure usernames and passwords, which are only available to specific users who have a need to access the data and who have the appropriate security clearances.

• Sensitive data is encrypted and stored on removable storage devices that are kept physically secure when not in use.

Mathematica’s standard for maintaining confidentiality includes training staff regarding the meaning of confidentiality, particularly as it relates to handling requests for information, and providing assurance to respondents about the protection of their responses. It also includes built-in safeguards concerning status monitoring and receipt control systems. In addition, all study staff who have access to confidential data must obtain security clearance from ED which requires completing personnel security forms, providing fingerprints, and undergoing a background check.

The program is currently preparing a system of records notice (SORN) and a privacy impact assessment (PIA). The data are to be stored both electronically and in paper copy. The data will be retrievable by ID, and will be maintained and disposed of in accordance with the Department’s Records Disposition requirements. The electronic files will be kept on a password protected server. The paper copy will be kept in a locked file cabinet, and all access to data in both electronic and paper form will be restricted to study staff on a need to know basis. The security protections for the content will be identified in the SORN.

A11. Justification for sensitive questions

No questions of a sensitive nature will be included in this study.

A12. Estimates of hours burden

Table A.5 provides an estimate of burden for the data collections included in the current request, broken down by instrument and respondent. These estimates are based on our prior experience collecting data from districts.

The number of targeted respondents and responses are each 220. The total burden is estimated at 135 hours or an average of 45 annual burden hours calculated across 3 years of data collection.

Table A.5. Estimate response time for data collection

Respondent/Data request	Number of targeted respondents	Expected response rate (%)	Expected number of respondents	Unit response time (hours)	Annual total response time over 3-year data collection (hours/year)	Total burden (Hours)
Current clearance request
District Interview (Spring 2020)	25	100	25	1	8.3	25
District Interview (Spring 2021)	25	100	25	1	8.3	25
Teacher leader applicant background forms (Spring/Summer  2020)	50	100	50	0.5	8.3	25
School information questionnaire (Spring 2020)	120	100	120	0.5	20	60
Total (rounded)	220a		220		45	135

^a The subtotal 1 number of targeted respondents (220) is the sum of targeted responses across data requests from a total of 145 unique respondents including 25 TSL district respondents and 120 principals. We assume 120 principals will complete the school information questionnaire, although only 100 schools will participate in the random assignment evaluation.

The total of 135 hours includes the time for:

• 25 districts supported by 2017 TSL funds to complete 2 rounds of interviews (50 hours);

• 50 principals to complete teacher leader applicant background forms (25 hours); 120 principals to complete a school information questionnaire (60 hours)

A13. Estimate of cost burden to respondents

There are no direct or start-up costs to respondents associated with this data collection.

A14. Annualized cost to the federal government

The total cost to the federal government for this study is $9,233,300. The estimated average annual cost – including recruiting districts, coaching and supporting teachers, designing and administrating all collection instruments, processing and analyzing the data, and preparing reports is $1,538,883 (the total cost divided by 6 years of the study).

A15. Reasons for program changes or adjustments

This is a new collection.

A16. Plans for tabulation and publication of results

a. Analysis plan

This study will produce an implementation brief and a final report. Below, we describe the main analyses for each.

Implementation brief. The implementation brief will describe TSL grantee districts and their programs in the following ways:

• Using data from the Common Core of Data and EDFacts, we will describe grantee districts, such as the number and percentage of grantees by region and locality (urban, suburban, or rural) and provide summary statistics describing total enrollment and student characteristics.

• We will describe the primary strategies used by grantee districts to improve educator effectiveness, as well as the activities districts used to carry out these strategies. For activities districts indicate were key to accomplish their main goals, we will use detailed information about those activities to illustrate why and how grantees implemented specific activities. We will indicate what proportion of these were new activities and what proportion were extensions of revisions of existing activities. We will also indicate whether districts used measures of educator effectiveness or other data to help them design the activity or inform the way they carry it out.

• We will describe districts’ challenges in implementing and sustaining TSL activities. For example, we will report the number and percentage of grantees that report union resistance to the teacher leader role, difficulty recruiting or retaining minority teachers, and limited funding to sustain incentives to retain and reward high-performing educators.

Final report. The final report will focus on implementation and impacts among the evaluation districts; it will also include information on TSL grantees who implemented teacher leader roles. The study includes several subgroup analyses to understand the impacts of teacher leaders on key groups of teachers and their students.

• Implementation analyses. We will describe implementation of teacher leader roles implemented in the treatment schools, such as teacher leader qualifications and characteristics, the composition of teacher teams, and how teacher leaders spend their time. A detailed understanding of the teacher leader roles implemented will provide important information for districts considering implementing similar teacher leader roles, support replication of these roles in other districts, and provide context for impact findings. We will also describe and compare the coaching and support received by teachers in treatment and control schools. A clear description of what the control group schools are doing compared to the treatment schools can help identify possible reasons for the presence or absence of impacts. Finally we will compare the implementation of the teacher leader role in TSL grantee schools and non-TSL treatment schools. This comparison will help policymakers understand the extent to which findings based on the schools in the random assignment evaluation may be relevant for TSL grantee districts.

• Impact analyses. We will estimate the impact of teacher leader roles on principals’ strategies for recruiting teachers, the retention and recruitment of effective teachers, principals’ and teachers’ satisfaction, and students’ math and ELA achievement. To do this, we will use regression models to compare these outcomes among those in the treatment and control groups. With a random assignment design, comparing outcomes in the treatment and control groups should yield unbiased estimates of the treatment’s impacts. To increase the precision of our estimates, we will also control for student, teacher, and school characteristics. We will examine several subgroups of teachers and their students to determine how impacts vary for high-priority teachers, teacher leaders, and all teachers on the teacher leader team. If we find that having teacher leaders improves student achievement, we will also examine whether this strategy is cost-effective relative to other approaches to improving teacher effectiveness.

b. Publication plan

We will produce an implementation brief with an anticipated release in 2021 and a final report with an anticipated release in 2024. The implementation report will use information from grantee interviews and the 2020 district interviews to describe TSL grantee districts and their programs, including teacher leader roles. The final report will include both implementation and impact analyses, as described in the prior section.

A17. Approval not to display the expiration date for OMB approval

The Institute of Education Sciences is not requesting a waiver for the display of the OMB approval number and expiration date. The study will display the OMB expiration date.

A18. Exception to the certification statement

No exceptions to the certification statement are requested or required.

REFERENCES

Aragon, Stephanie. “Teacher Development and Advancement.” Policy Snapshot. Denver, CO: Education Commission of the States, 2018.

Chiang, Hanley, Cecilia Speroni, Mariesa Herrmann, Kristin Hallgren, Paul Burkander, and Alison Wellington. “Evaluation of the Teacher Incentive Fund: Final Report on Implementation and Impacts of Pay-for-Performance Across Four Years.” NCEE 2017-4004. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2017.

Education Commission of the States. “Teacher Leadership and Licensure Advancement Data Visualization.” 2018a. Available at https://www.ecs.org/teacher-leadership-and-licensure-advancement-data-visualization/. Accessed November 30, 2018.

Education Commission of the States. “Teacher Leadership and Licensure Advancement: State Profiles.” 2018b. Available at https://www.ecs.org/teacher-leadership-and-licensure-advancement-state-profiles. Accessed November 30, 2018.

Garet, Michael S., Andrew J. Wayne, Seth Brown, Jordan Rickles, Mengli Song, and David Manzeske. “The Impact of Providing Performance Feedback to Teachers and Principals.” NCEE 2018-4001. Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, 2017.

Mangin, M.M., and S.R. Stoelinga. “Teacher Leadership: What It Is and Why It Matters.” In Effective Teacher Leadership: Using Research to Inform and Reform, edited by M.M. Mangin and S.R. Stoelinga. New York, NY: Teachers College Press, 2008.

Margolis, Jason. “Hybrid Teacher Leaders and the New Professional Development Ecology.” Professional Development in Education, vol. 38, 2012, pp. 291–315.

Sutcher, Leib, Linda Darling-Hammond, and Desiree Carver-Thomas. “A Coming Crisis in Teaching? Teacher Supply, Demand, and Shortages in the U.S.” Palo Alto: Learning Policy Institute, 2016.

Wenner, Julianne A., and Todd Campbell. “The Theoretical and Empirical Basis of Teacher Leadership: A Review of the Literature.” Review of Educational Research, vol. 87, no. 1, 2017, pp. 134–171.

Workman, Emily, and Micah Ann Wixom. “Mitigating Teacher Shortages: Evaluation and Feedback.” Denver: Education Commission of the States, 2016.

1 The TSL program was established and the evaluation mandated by the Elementary and Secondary Education Act of 1965 (ESEA), as reauthorized on December 10, 2015 through P.L. 115-64, the Every Student Succeeds Act (ESSA).

2 Two recent evaluations from the Institute of Education Sciences (IES) examined a subset of common TIF/TSL strategies. One examined the impact of measuring educator effectiveness and providing educators with feedback on their performance (Garet et al. 2017). The other focused on offering educators pay-for-performance bonuses (Chiang et al. 2017). Neither evaluated the impact of teacher leaders, the focus of this evaluation.

3 In 2017, 9 states enacted legislation to provide teachers with career advancement opportunities, such as teacher leader roles (Aragon 2018). As of September 2018, 22 states offered a teacher leader license or endorsement, 17 states had adopted teacher leader standards, 13 states prescribed the role of a teacher leader in a statute or regulation, and 24 states provided formal supports or incentives to teacher leaders (Education Commission of the States 2018a, 2018b).