Data Collection Part A_FINAL

A Study of Teacher Residency Programs

Part A

April 28, 2011

Contract Number: ED-IES-10-C (0001) Mathematica Reference Number: 06748-510 Submitted to: Institute of Education Sciences U.S. Department of Education 555 New Jersey Avenue, NW Washington, DC 20208 Project Officer: Melanie Ali Submitted by: Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Philip Gleason

An Study of Teacher Residency Programs Part A April 28, 2011

CONTENTS

SUPPORTING STATEMENT For Paperwork REDUCTION ACT 1

PART A: JUSTIFICATION 1

1. Circumstances Necessitating the Collection of Information 1

a. Statement of Need for a Study of TRPs 1

b. Research Questions 2

c. Study Design 2

d. Data Collection Needs 3

e. Study Activities and Data Collection Timeline 5

2. Purposes and Uses of the Data 6

3. Use of Technology to Reduce Burden 7

4. Efforts to Avoid Duplication 8

5. Methods to Minimize Burden on Small Entities 8

6. Consequences of Not Collecting Data 8

7. Special Circumstances 9

8. Federal Register Announcement and Consultation 9

a. Federal Register Announcement 9

b. Consultations Outside the Agency 9

c. Unresolved Issues 10

9. Payments or Gifts 10

10. Assurances of Confidentiality 11

11. Justification for Sensitive Questions 12

12. Estimates of Hours Burden 12

13. Estimates of Cost Burden to Respondents 13

14. Annualized Cost to the Federal Government 13

15. Reasons for Program Changes or Adjustments 14

16. Plan for Tabulation and Publication of Results 14

a. Tabulation Plans 14

b. Publication Plans 15

17. Approval Not to Display the Expiration Date for OMB Approval 16

18. Exception to the Certification Statement 16

REFERENCES 17

TABLES

Table 1 Overview of TRP Involvement in the Study 3

Table 2 Schedule of Major Study Activities 5

Table 3 Research Questions and Data Sources 7

Table 4 Incentives by Respondent Type Proposed for the Study 10

Table 5 Estimated Response Time for Data Collection 13

SUPPORTING STATEMENT For Paperwork REDUCTION ACT

This OMB package requests clearance for data collection activities to support a study of teacher residency programs (TRPs). This study will provide important implementation information on TRPs funded by the U.S. Department of Education (ED), as well as information on the achievement outcomes of the students whose teachers participate in TRPs (including some funded by ED). The study will focus primarily on TRPs that received Teacher Quality Partnership (TQP) grants from ED in late 2009 and early 2010. ED’s Institute of Education Sciences (IES) has contracted with Mathematica Policy Research and its partner, Decision Information Resources (DIR), to conduct the study.

The main objective of the study is to describe the characteristics of TRPs and their participants. We will also summarize the academic outcomes of students taught by novice TRP teachers and examine the retention rate of novice TRP teachers. This is the second of two requests for the study, the first of which requested approval for recruitment of TRPs, school districts, and schools. We are submitting the package in two stages because the study schedule required that participant recruitment begin before all the data collection instruments are developed and tested. In this package, we are requesting approval for data collection activities that will support the full-scale study.

PART A: JUSTIFICATION

1. Circumstances Necessitating the Collection of Information

a. Statement of Need for a Study of TRPs

The TRP study is authorized in Title II, Part A of the Higher Education Act, as amended on August 14, 2008, by the Higher Education Opportunity Act (Publ. L. 110-315, Sections 201–204) (HEA). The law stipulates that federal funds are to be used to evaluate activities that are authorized under this act. The TQP grant program was funded in fiscal year 2009 with $43 million, and received an additional $100 million from the American Recovery and Investment Act (ARRA) of 2009.

The study is also authorized under Title II, Part A of the Office of Elementary and Secondary Education Act (ESEA) Title II, Part A, 20 U.S.C. 6601-6641. This legislation includes the Improving Teacher Quality State Grants program, aimed to increase the academic achievement of all students by helping schools and districts improve teacher and principal quality and ensuring that all teachers are highly qualified. Providing support for TRPs is an allowable use of program funds.

Research suggests that for many teachers the early years represent a difficult transition period—first-year teachers tend to be less effective than their experienced counterparts (Clotfelter et al. 2007; Hanushek et al. 2005) and newer teachers are more likely to leave the profession (Ingersoll and Smith 2003). The situation may be more pronounced at high-need schools where teachers leave at higher rates and positions are more difficult to fill compared with more advantaged schools (Hanushek et al. 2004; Lankford et al. 2002; Berry 2008). While reasons for these patterns are complex, some teachers attribute staffing difficulties at high-need schools to a lack of support and training on how to effectively teach students with social and behavioral challenges (Berry 2008).

Some districts have responded to staffing challenges by creating or adopting TRPs. The TRP model combines elements from different models of teacher preparation. As with alternative routes to teaching, TRPs give candidates a “fast track” to the classroom without having to complete an undergraduate major in education, enabling them to start teaching prior to receiving initial certification. TRPs involve a year-long “clinical” experience (the “residency”) shadowing and co-teaching with an experienced mentor, similar to but longer than the student teaching component of traditional routes. TRPs also provide continued support and mentoring after participants become teachers of record, similar to what is provided in teacher induction programs. Before and during their residencies, participants in TRPs take coursework usually resulting in a master’s degree.

TRPs represent an innovative training strategy, but there is little rigorous evidence connecting specific aspects of teacher training programs to teacher effectiveness (Constantine et al. 2009; Wilson et al. 2002). Given this lack of evidence and the recent infusion of federal grant money to create or expand TRPs, ED has requested a study of TRPs as a model for preparing new teachers. The study will summarize the outcomes of students with novice TRP teachers, examine the retention rates of TRP novice teachers, and describe the programs and their participants. The results of this study will provide critical information on the implementation of this model of teacher preparation to educators, policymakers, and researchers.

b. Research Questions

The study’s primary research questions are:

1. What are the characteristics of teacher residency programs?

2. What are the characteristics of participants in teacher residency programs?

3. What is the average performance of novice¹ TRP teachers as measured by value-added benchmarked against novice and all teachers in the district?

4. What are the retention rates of novice TRP teachers and their novice colleagues who did not go through TRPs?

c. Study Design

To the extent possible, we will put the information about TRPs in context by also describing the traditional and other alternative-route programs. We will also examine differences in attributes across the TRPs included in the study (for example, differences in selection criteria or training strategies), and if we notice important variation in any attributes, we will attempt to explore how those differences are correlated with student outcomes.

As part of the TRP study, and to address the research questions, Mathematica will:

• Collect student administrative records data to summarize the achievement of students in TRP and non-TRP classrooms.

• Track teacher mobility through district records and teacher mobility surveys in order to examine retention among novice TRP and novice non-TRP teachers.

• Survey and interview TRP administrators to describe the programs and their implementation.

• Survey teachers as well as TRP residents and their mentors to describe these sample members and their teaching experiences.

Different sets of TRPs will be needed for the four major analytical components of the study, as shown in Table 1. For example, all TQP grantees operating TRPs will be surveyed about basic program characteristics, but only a subset of about 15 of their administrators will be interviewed for more program details. Finally, 8 of the TRPs that have been in operation for at least one year will be recruited for the outcomes study. We will seek to include only TQP grantees in the study. However, if we are unable to identify 8 TQP grantees with student-teacher linked data or are unable to secure the participation of a sufficient number of grantees, we will supplement the sample with non-grantees.

Table 1. Overview of TRP Involvement in the Study

	Number	Student Achievement Outcomes Study	Teacher Retention	Descriptive Analysis of TRPs	Descriptive Analysis of TRP Participants
All TRPs that Received TQP Grants in 2009-2010	28			√
Subset of Above Group For In-Depth Study	15a			√	√
Experiencedb Grantees Specifically Targeted For Outcomes Study	6	√	√	√	√
Other Experienced TRPs (Non-Grantees) Specifically Targeted For Outcomes Study	2	√	√	√	√

^a Estimate – at this time not enough is known about TRPs—especially the number and type of teachers they will have placed in residency in fall 2011, and how those placements will be distributed across partner districts and schools—to cite specific numbers with certainty. These determinations will be made during the selection and recruitment process

^b Experienced grantees are those that began operation in 2009 or earlier.

The samples for various components of the study will be selected through a series of recruitment activities, which were described in more detail in the first clearance package.

d. Data Collection Needs

To address the four research questions, this study includes several data collection efforts, each of which is described below.

Student records data. Following the 2011-2012 and 2012-2013 school years, we will ask districts to provide data for all students in the district who are in tested grades and subjects. We will collect both demographic data (that is, age, race/ethnicity, English language proficiency, disability status, and eligibility for school lunch program) and data on students’ performance on state or district tests in the current year and two previous school years. We will send the district a letter that will specify the data items requested and a non-technical brochure providing additional study information (Appendix J).

TRP survey. We will mail a self-administered 35-minute hardcopy survey to a director at each of the 28 TRPs that received a TQP grant in fall 2009 or spring 2010 and to any non-grantees included in the outcomes study. This survey, to be administered in spring 2011, will address TRP characteristics, admission requirements, and key program features, providing the foundation for answering the first research question. The mailing will contain a grantee or non-grantee cover letter (Appendix A), questionnaire (Appendix D) and a non-technical brochure providing additional study information (Appendix J). The letter, which will be on ED’s stationery, will describe the study and its objectives and the need for TRP participation, address confidentiality, and provide a telephone number and email address for questions or concerns.

TRP director interview. In spring 2011, we will conduct semi-structured interviews, by telephone or in person, with the directors of the 15 TRPs identified for the in-depth implementation study. The 45-minute interview will collect detailed information on the amount of instruction in different substantive and pedagogical subject areas provided to candidates at various points in the program (before, during and after the residency year; during and after the first year as a teacher of record). The open-ended questions in the TRP interviews will allow us to collect more in-depth information than that collected from the survey, and to probe for clarification if necessary (Appendix H). We will contact potential respondents in advance and provide them with a list of topics to be covered and any general information about the study as needed. (They will be familiar with the study from our previous contacts with them during the recruitment stage.)

Resident teacher survey. A 25-minute mail survey of 300 TRP participants who serve their residency during the 2010-2011 school year will be conducted in spring 2011. It will collect descriptive information on resident teachers’ backgrounds as well as experiences during their residency year—for example, interactions with resident mentors, classroom responsibilities, and views on the program. This survey will be administered to all residents from the same set of TRPs included in the program director interviews. The mailing will contain a cover letter (Appendix A), questionnaire (Appendix F) and a non-technical brochure providing additional study information (Appendix J). In the cover letter, we will describe the study and its objectives and the need for resident teacher participation, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns.

Mentor teacher survey. In spring 2011, we will mail a 20-minute self-administered hardcopy survey to the 300 mentors associated with each of the teachers targeted for the resident teacher survey. Respondents will be asked to provide descriptive information on their backgrounds, qualifications, and training for the mentor role, their residents’ responsibilities, and their interactions with residents. Parallel questions across the resident and mentor surveys will allow for corroboration during analysis. The mailing will contain a mentor teacher cover letter (Appendix A), questionnaire (Appendix E) and a non-technical brochure providing additional study information (Appendix J). In the cover letter, we will describe the study and its objectives and the need for mentor teacher participation, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns.

Teacher of record survey. In fall 2011, we will contact districts to request a list of teachers hired within the last two years, their current school assignment, and date of hire (Appendix A). The lists will provide the sample for the teacher of record survey. In spring 2012, all novice teachers in the 8 districts in the outcomes study will be asked to complete a 25-minute self-administered hardcopy mail survey on their background characteristics, experiences during the 2011-2012 school year, and views on teaching. Teachers of record who will have completed the resident survey the previous year will complete a shorter version of the teacher of record survey—one that excludes questions about background characteristics. The mailing will contain a teacher of record cover letter (Appendix A) questionnaire (Appendix G) and a non-technical brochure providing additional study information (Appendix J). In the cover letter, we will describe the study and its objectives and the need for participation from teachers of record, address confidentiality, and provide a toll-free telephone number and email address for questions or concerns. We expect to survey 800 teachers.

Teacher employment data. In fall 2012 and fall 2013, districts will be asked to verify whether the novice teachers in the outcomes study are still employed by the district. We will send each district a data request form that will specify the data requested (Appendix C), the list of novice teachers, and a non-technical brochure providing additional study information (Appendix J).

Teacher mobility survey. In fall 2012 and fall 2013, all teachers in the outcomes study will be asked to complete a 15-minute self-administered hardcopy mail survey on their current employment status and reasons for job changes, if applicable (Appendix I). The mailing will contain a cover letter (Appendix A) that will describe the purpose of the survey and the need for participation from all teachers regardless of current employment status, address confidentiality, and provide a toll-free number and email address for questions or concerns. We expect to survey 800 teachers in each wave of the survey.

e. Study Activities and Data Collection Timeline

The study will be completed in five years. Table 2 shows the schedule of data collection activities and the overall study timeline.

Table 2. Schedule of Major Study Activities

Activity	Spring 2011	Fall 2011	Spring 2012	Fall 2012	Fall 2013	Fall 2014
Recruit TRPs, Districts, and Schools	√
Conduct TRP Survey	√
Conduct TRP Director Interview	√
Conduct Resident Teacher Survey	√
Conduct Mentor Teacher Survey	√
Obtain List of Novice Teachers		√
Conduct Teacher of Record Survey			√
Collect Student Administrative Records Data				√	√
Collect Teacher Employment Data				√	√
Conduct Teacher Mobility Survey				√	√
Submit First Report					√
Submit Second Report and Restricted-Use Data File						√

2. Purposes and Uses of the Data

Information on the Study of Teacher Residency Programs will be collected and analyzed by Mathematica and its research partner, DIR. This work will be conducted under Contract Number ED-IES-10-C (0001). To address the research questions, data will be collected from student administrative records, teacher employment data, TRP interviews, public records, and surveys of teachers and TRPs.

• Student administrative records. ED will use data on student achievement and demographic and other background information collected through student administrative records to get a refined measure of the study outcomes. These data will inform research question three.

• TRP survey. Information collected by the TRP survey will provide ED with a more comprehensive view of TRPs than any previous study and will ensure we have information on TRPs that will be in the outcomes study sample. ED will use this survey data as a point of comparison and a broader context for the programs represented in the study. For the subset of TRPs whose teachers end up in the outcomes study, ED may also use this survey’s information in subgroup or non-experimental analyses that will be conducted in relation to research question three.

• TRP director interview. In spring 2011, we will conduct semi-structured interviews by telephone or in person with the directors of the 15 TRPs included in the in-depth implementation study. The 45-minute interviews will collect detailed information on the amount of instruction in different substantive and pedagogical subject areas provided to candidates at various points in the program (before, during and after the residency year; during and after the first year as a teacher of record). The open-ended questions in the TRP interviews will allow us to collect more in-depth information than that collected from the survey, and to probe for clarification if necessary (Appendix H).

• Resident and mentor teacher surveys. In addressing the first two research questions, ED will use the resident teacher survey to describe residents’ backgrounds and their experiences in the TRP (for example, instruction received, and interactions with their mentors). Similarly, ED will use data from the mentor teacher surveys to describe mentors’ backgrounds, qualifications and training for the mentor role as well as their experiences in the TRP (for example, interactions with the residents they mentor). Questions that are parallel between the resident and mentor teacher surveys will allow for corroboration during analysis.

• Teacher of record survey. Information on novice teachers provided by districts will be used to identify the sample for the teacher of record survey. ED will use data from the teacher of record survey in describing the characteristics of TRP and non-TRP teachers in the study, and to examine how the outcomes of the students of TRP teachers correlate with teacher characteristics such as educational background and work experience.

• Teacher employment data. Another outcome of interest for this study is novice teacher retention. District employment records will help ED determine whether novice TRP and non-TRP teachers return in the 2011-2012 and 2012-2013 school years. The data will be used to answer research question four.

• Teacher mobility survey. District records will only indicate whether the teacher is still employed by the district. The teacher mobility survey will provide more detailed information about whether departed teachers are still teaching (in another district or in a private school) or have moved to another occupation. The survey will also provide information about the factors that contributed to a teacher’s decision to leave a school. The data will be used to address question four.

Teacher quality is a critical component of student achievement, one in which the federal government has, through the TQP grants, invested substantial funds. While TRPs represent an innovative training strategy, very little is known about these programs’ participants, approaches to teacher training, or the effectiveness of the teachers they prepare. This study will provide policymakers and practitioners with important insight into practices for teacher recruitment, preparation, support, and retention in high-need schools. Table 3 lists the study’s research questions and the data to be collected to address each question. Study findings will be presented in two reports. In addition, the data collected by the study will be submitted to ED as restricted-use data files that will serve as a valuable resource for other researchers.

Table 3. Research Questions and Data Sources

Research Question	Data Sources
1. What are the characteristics of TRPs?	TRP surveys TRP interviews Resident teacher surveys Mentor teacher surveys
2. What are the characteristics of participants in TRPs?	TRP surveys TRP interviews Resident teacher surveys Teacher of record surveys
3. What is the average performance of novice TRP teachers as measured by value-added benchmarked against novice and all other teachers in the district?	District administrative records on student achievement and background characteristics
4. What are the retention rates of novice TRP teachers and their novice colleagues who did not go through TRPs?	Teacher employment data Teacher mobility survey

3. Use of Technology to Reduce Burden

The data collection plan is designed to obtain reliable information in an efficient way while minimizing respondent burden. We will set up a toll-free telephone number and email address specific to the study so that participants can easily contact researchers with questions they may have.

Where feasible, we will gather information from existing data sources, using the most efficient methods available. Existing data sources will include student achievement test scores, student demographic information, and teacher administrative records. This information will be obtained in the form of computer files provided by school districts. We will provide clear instructions on the data requested. Some data, however, can only be obtained directly from teachers and TRPs.

The TRP survey and all teacher surveys will be mailed to respondents to complete and return, with telephone and email follow-up for nonresponse or consistency checks. The study team considered other modes of survey administration for these instruments, such as a computer automated telephone interview (CATI) or a Web-based survey. Because the sample size for each respondent category is relatively small (15 – 800), however, the predicted cost of developing computer-based surveys outweighed the expected benefits. Some respondents may also find a mail questionnaire to be less burdensome because computer-based interviews typically need to be conducted when the respondent has access to a telephone or computer, and self-administered surveys are generally more convenient because respondents can complete the paper instruments at a time and place of their choosing.

The semi-structured TRP director interviews will be conducted by telephone or in person, thereby providing flexibility in scheduling interviews at each participant’s convenience. Collecting the data through interviews will also allow a conversational exchange to answer the open-ended questions needed to obtain detailed information on the TRPs and participants—more details than can easily be obtained through the previously described TRP survey.

4. Efforts to Avoid Duplication

No other national study has been conducted or is underway to address the same research questions as this study. This submission represents the sole, official study of TRPs being sponsored by ED. Prior to issuing the performance work statement and request for proposals, ED determined that a national study examining TRPs is needed, and that this study would not be duplicative.

5. Methods to Minimize Burden on Small Entities

The primary entities for the study are TRPs (most of which are operated by colleges and universities), school districts, and teachers. We will minimize burden for all respondents by requesting only the minimum data required to meet study objectives. Burden on districts and teachers will be further minimized through the careful specification of information needs. We have also kept data collection instruments short and focused on the data of most interest and will speak with relatively few respondents in person. The sample size and data requirements of each respondent group were determined by careful consideration of the information needed to meet study objectives and were reviewed by the study’s technical working group (TWG).

6. Consequences of Not Collecting Data

The data collection activities described in this submission are necessary for ED to conduct a detailed study of TRPs and to understand more about this approach to teacher preparation. The study represents an important step in developing a systematic study agenda in the area of teacher training and retention.

Data for interviews and surveys will be collected only once from each respondent. School records and teacher employment data will be collected twice to allow for a longitudinal summary of teacher retention. The consequences of not collecting specific data items are discussed below.

• Without information from TRP interviews and surveys, the study cannot provide a comprehensive view of these programs and their strategies for recruiting and training teachers.

• Without resident teacher survey data, we will not have the necessary data to describe experiences of teachers in their first year of apprenticeship as they undergo this new method of teacher preparation.

• Without the mentor teacher survey data, we will not be able to obtain critical information about resident teachers from the perspectives of experienced teachers who spend the most classroom time with resident teachers. Mentors are a critical component of TRPs, and this survey will allow us to describe characteristics of the mentor teachers, such as their background, experience, and training.

• Without the teacher of record survey, it will be challenging to report on TRP teachers’ experiences during their first years of teaching and to contrast these experiences with those of non-TRP teachers in the sample. Without the list of novice teachers provided by districts, the study will be unable to identify the sample for the teacher of record survey.

• Without the students’ administrative records data, it will be impossible to summarize the outcomes of the students of TRP and benchmark those against the outcomes of non-TRP teachers.

• Without teacher employment data, we will not have complete data to examine the retention rate of TRP teachers and their novice colleagues who did not go through the residency program (research question four). These data will complement the teacher mobility survey, especially for cases in which the teacher does not respond to the survey.

• Without the teacher mobility survey, we will not be able to determine whether departed teachers are still in the teaching profession. We also would not be able to describe the factors that contributed to a teacher’s decision to leave.

7. Special Circumstances

There are no special circumstances associated with this data collection.

8. Federal Register Announcement and Consultation

a. Federal Register Announcement

The 60-day notice to solicit public comments was published in Volume 75, Number 173, page 61709 of the Federal Register on October 6, 2010. One comment was received from the public and the comment and response are attached. The 30-day notice will be published to solicit additional public comments.

b. Consultations Outside the Agency

In formulating the study design, the study team sought input from the TWG, which includes some of the nation’s experts in teacher preparation, study methodology, and education policy. We will continue to consult with the TWG throughout the study on other issues that would benefit from their input. The TWG members are:

• Jeffrey Smith, professor of economics, University of Michigan

• James Wyckoff, professor of education, University of Virginia

• Sandra Odell, professor of education, University of Nevada-Las Vegas

• Pamela Grossman, professor of education, Stanford University

• Diana Montgomery, independent research consultant

• Jon Fullerton, executive director, Project for Policy Innovation, Harvard University

• Jason Snipes, director of research, Council of Great City Schools

• Elizabeth Stuart, assistant professor of mental health and biostatistics, Johns Hopkins University

c. Unresolved Issues

There are no unresolved issues.

9. Payments or Gifts

Table 4 shows the incentive structure, by respondent type, requested for the study. A description and rationale for each incentive is provided below.

Table 4. Incentives by Respondent Type Proposed for the Study

Respondent	Incentive
Resident Teacher	$25 gift card
Mentor Teacher	$20 gift card
Teacher of Record	$25 gift card (teacher of record survey) $20 gift card (teacher mobility survey 1) $20 gift card (teacher mobility survey 2)

Incentives for teachers. Experience has demonstrated the need for a teacher-level incentive to achieve high response rates. Thus, we propose offering a $25 gift card to resident teachers and a $20 gift card to mentor teachers who complete the survey. We propose offering teachers of record a $25 gift card for the teacher of record survey and a $20 gift card for the mobility survey. The size of the incentive payments is based on guidelines we have used before – $1 per minute of expected burden. Since we are asking teachers of record to provide information multiple times, we feel it is necessary to offer an incentive that is slightly higher than the $1 per minute estimate to achieve the desired response rate on the mobility survey. These amounts are consistent with incentives approved on similar teacher surveys for ED impact evaluations. For example, for the OMB-approved Evaluation of the Impact of Teacher Induction Programs study (OMB Control Number 1850-0802), teachers received $30 for a baseline survey that averaged about 30 minutes to complete. To achieve the high response rates, we believe the gift cards are an efficient way to obtain response rates of at least 85 percent from resident teachers and teachers of record, and 90 percent from mentor teachers.

The study will not give incentives to TRPs for completing the interview and survey, or to districts for providing student administrative records and teacher employment data.

10. Assurances of Confidentiality

The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant regulations and requirements, including

• The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a);

• The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99);

• The Protection of Pupil Rights Amendment (PPRA) (20 U.S.C. 1232h; 34 CFR Part 98);

• The Education Sciences Reform Act of 2002 (ESRA), (20 U.S.C. 9573; Title I, Part A, Section 183).

Mathematica and its subcontractor DIR will protect the confidentiality of all information for the study and will use it for research purposes only. The project director will ensure that all individually identifiable information about respondents will remain confidential. All data will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required. All members of the study team having access to the data will be trained and certified on the importance of confidentiality and data security. When reporting the results, data will be presented only in aggregate form, such that individuals and institutions will not be identified. Included in all requests for data and communications about the study will be the following statement:

“Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.”

The following safeguards are routinely employed by Mathematica to carry out confidentiality assurances; these safeguards will be consistently applied during this study:

1. All employees sign a confidentiality pledge (Appendix K) that emphasizes its importance and describes employees’ obligation.

2. Personally identifiable information (PII) is maintained on separate forms and files, which are linked only by sample identification number.

3. Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets and discarded materials are shredded.

4. Access to computer data files is protected by secure usernames and passwords, which are only available to specific users.

5. Especially sensitive data is encrypted and stored on removable storage devices that are kept physically secure when not in use.

The plan for maintaining confidentiality includes signing confidentiality agreements and notarized nondisclosure affidavits obtained from all personnel who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information, and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single data base manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility.

Letters and other materials will be sent to school administrators describing the voluntary nature of this survey. The material sent will include a brochure to describe the study and to convey the extent to which respondents and their responses will be kept confidential (Appendix J).

11. Justification for Sensitive Questions

No questions of a highly sensitive nature are included in teacher questionnaires. Teachers will be asked to provide demographic information (ethnicity, race, age) and information about their educational and professional background and current employment. Data on these topics are important to collect because they will help us understand the experiences of TRP and non-TRP teachers. They will also help us to describe how the outcomes of TRP teachers vary with their characteristics. To address concerns about disclosing such personal information, the surveys will clearly state that all responses will be treated as confidential, that participation is voluntary, and that failure to provide some or all requested information will not affect the teachers’ professional status in any way. The questions are also worded in a sensitive, nonjudgmental manner and have all been successfully pretested; most have also been used extensively in previous studies with no evidence of harm.

Test scores and some demographic information about the students may be sensitive. Test score data is essential for this study as it is the outcome of interest in the outcomes analysis. Demographic information is important to control for differences in the characteristics of students.

There are no questions of a sensitive nature in the TRP survey or interview.

12. Estimates of Hours Burden

All data collection activities will be voluntary. Table 5 reports the estimated burden hours for TRP staff, district staff, and teachers who will participate in the data collection. The 1613.05 burden hours includes 25.25 hours for TRP staff, 936 hours for district staff, and 651.8 hours for teachers. These estimates are based on our experience collecting data from grantees as well as district and school staff for similar studies.

Table 5. Estimated Response Time for Data Collection

Respondent/Data Request	Number of Targeted Respondents	Expected Response Rate (%)	Number of Respondents	Unit Response Time (Hours)	Total Response Time (Hours)
TRP Staff
Phone Interview with TRP Directors	15	100	15	.75	11.25
TRP Survey	35	80	28	.50	14
District Staff
List of Novice Teachers (1 hr/District)	8	100	8	1	8
1st Student Administrative Records Data (.5 hrs/Classroom; 100 Classrooms/District)	8	100	8	50	400
2nd Student Administrative Records Data (.5 hrs/Classroom; 100 Classrooms/District)	8	100	8	50	400
1st Teacher Employment Data (10 Minutes/Teacher)	8	100	8	8	64
2nd Teacher Employment Data (10 Minutes/Teacher)	8	100	8	8	64
Teacher Surveys
Resident Teacher Survey	300	85	255	0.42	107.1
Mentor Teacher Survey	300	90	270	0.33	89.1
Teacher of Record Survey	800	85	680	0.42	285.6
Teacher Mobility Survey	800	85	680	0.25	170
Total					1613.05

13. Estimates of Cost Burden to Respondents

There are no start-up costs for respondents.

14. Annualized Cost to the Federal Government

The estimated annual cost of data collection activities is $699,035. The estimated total cost of the data collection over three years is $2,097,104. This cost includes:

1. Training staff and developing data management systems, $219,747

2. Collecting data, $567,760

The estimated average annual total cost of the study to the federal government over the five years of the contract is $1,064,538. The estimated total cost of the contract is $5,322,690, which includes recruiting TRPs, district and schools; designing and administering data collection instruments; administering student assessment; processing and analyzing data; and preparing reports.

15. Reasons for Program Changes or Adjustments

This is a new data collection.

16. Plan for Tabulation and Publication of Results

a. Tabulation Plans

Our tabulation plans include four sets of analyses aligned to the research questions.

Describing the achievement outcomes of students with TRP teachers.

The study will not seek to identify the causal effect of having TRP teachers on students’ achievement levels. Instead, we will describe the average growth in achievement in the students of novice TRP teachers benchmarked against the average growth of students of all other teachers as well as the subset of novice non-TRP teachers in the district. Rather than using the simple change in test scores, we will attempt to get the most precise measure of growth possible using a value-added model:

(1)

where Y_ijk is the test score of student i in a class taught by teacher j in year t, Yi_(-t) is a vector of the previous two years of test scores for student i, X_ijk is a vector of student baseline characteristics, the T_i’s are indicator variables for each teacher j, µ_j is a classroom-specific random error term, ε_ij is a student-level random error term, and β, , and γ represent parameters to be estimated. The model will be estimated with ordinary least squares (OLS), using standard errors that account for classroom-level clustering.

The estimates of γ represent the change in student achievement that each teacher produces in excess to what would have been expected based on the characteristics and prior achievement level of their students. We will take an average of the s for TRP teachers and provide the average value of for all teachers² and the average for novice non-TRP teachers as benchmarks.

TRP and non-TRP teacher retention rates. We will also summarize the retention rates of TRP and non-TRP teachers in the district. Measures of teacher retention may include whether the teacher remained at the same school, moved to another school in the district, moved to another district, or left the teaching profession.

The timeline for retention data collection is as follows. In fall 2012 and fall 2013, we will contact districts to request data on employment status among all novice teachers. In fall 2012, we will determine which of the novice teachers from the previous year are still teaching in the district. In fall 2013, we will again determine which of the teachers in our analysis sample are still teaching in the district. In addition to the district employment data, we will administer a teacher mobility survey in fall 2012 and fall 2013. We will use the responses from these surveys to supplement the information collected through the district employment data.

Describing the characteristics of TRPs. Drawing on data from the TRP survey and interviews with the directors of TRPs considered for the study, we will use descriptive statistics to paint a profile of these programs. Our analyses will provide information on the types of teachers that TRPs train; the amount, content, and timing of the coursework they provide; and their support and mentoring activities. To the extent possible, we will put the information about TRPs in context by comparing them with traditional and other alternative-route programs. We will also examine differences in attributes across the TRPs included in the study (for example, differences in selection criteria or training strategies), and if we notice important variation in any of these attributes, we will explore how it is correlated with student outcomes. All TRP surveys and director interviews will take place in spring 2011.

Describing the characteristics of participants in TRPs. We will conduct three sets of teacher-level analyses. First, drawing on the more detailed data from the resident teacher survey, we will use descriptive statistics to provide additional information on characteristics of program participants and their experiences and opinions regarding the program. Second, we will use descriptive statistics to profile characteristics of TRP teachers’ mentors during the residency year. We will compare the responses of residents and mentors to questions about their joint activities, which will provide a more balanced and comprehensive view of the residency experiences than would be possible from examining the responses of only one of these groups. Third, to provide context for the study, we will describe TRP and non-TRP teachers’ backgrounds and experiences. Potential characteristics to examine include prestige of undergraduate institution, college major, and prior work experience.

Conducting exploratory analyses assessing variation in outcomes. Our outcomes analysis may be enhanced through exploratory analyses that provide information on key program features and teacher characteristics associated with the outcomes of TRP teachers as well as the mechanisms through which teachers trained in TRPs may influence student achievement. These findings can inform policymakers and program directors seeking to improve the effectiveness of existing and future TRPs. Our exploratory analyses will include subgroup analyses as well as an analysis of potential mediators of TRP outcomes (that is, the characteristics or experiences of TRP teachers that are associated with outcomes that are more positive). In each case, we will note that observed relationships may not be causal, and may reflect the influence of additional, unobserved factors.

b. Publication Plans

We will prepare two reports presenting the results of the study. The first report, with a projected release date in fall 2013, will address all four research questions based on data from the 2010-2011 and 2011-2012 school years. The second report, with a projected release date of fall 2014, will update our findings on student achievement and teacher retention using data from the 2012-2013 school year, and it will include exploratory analyses of factors related to outcomes.

17. Approval Not to Display the Expiration Date for OMB Approval

The study will display the OMB expiration date.

18. Exception to the Certification Statement

No exceptions are being sought.

REFERENCES

Agodini, Roberto, and Dynarski, Mark. “Are Experiments the Only Option?” The Review of Economics and Statistics, vol. 86, no. 1, February 2004, pp. 180–194.

Anderman, C., A, Cheadle, S. Curry, P. Diehr, L. Shultz, and E. Wagner. “Selection bias related to parental consent in school-based survey research.” Evaluation Review, vol. 19, no. 6, 1995.

Angrist, J., G. Imbens, and D. Rubin. “Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association, vol. 91, no. 434, 1996, pp. 444–445.

Berry, Barnett. “Staffing High-Needs Schools: Insights from the Nation’s Best Teachers.” Phi Delta Kappan, vol. 89, no. 10, 2008, pp. 766–771.

Clotfelter, Charles, Helen Ladd, and Jacob Vigdor. “How and Why Do Teacher Credentials Matter for Student Achievement?” CALDER Working Paper No. 2. 2007.

Constantine, Jill, Daniel Player, Tim Silva, Kristin Hallgren, Mary Grider, and John Deke. “An Evaluation of Teachers Trained Through Different Routes to Certification. Final Report.” Princeton, NJ: Mathematica Policy Research, 2009.

Eaton, Danice K., Richard Lowry, Nancy D. Brener, Jo Anne Grunbaum, and Laura Kann. “Passive versus active parental permission in school-based survey research: Does the type of permission affect prevalence estimates of risk behaviors.” Evaluation Review, vol. 28, 2004.

Ebsensen, Finn-Aage, Michelle Hughes Miller, Terrance Taylor, Ni He, and Adrienne Freng. “Differential attrition rates and active parental consent.” Evaluation Review, vol. 23, no. 316, 1999.

Hanushek, Eric, John Kain, and Steven Rivkin. “The Market for Teacher Quality.” NBER Working Paper 11154. Cambridge, MA: National Bureau of Economic Research, 2005.

Hanushek, Eric, John Kain, and Steven Rivkin. “Why Public Schools Lose Teachers.” Journal of Human Resources, vol. 39, no. 2, 2004, pp. 326–354.

Ingersoll, Richard, and Thomas Smith. “The Wrong Solution to the Teacher Shortage.” Educational Leadership, vol. 60, no. 8, 2003, pp. 30–33.

Jackson, Russell, Ann McCoy, Carol Pistorino, Anna Wilkinson, John Burghardt, Melissa Clark, Christine Ross, Peter Schochet, and Paul Swank. “National Evaluation of Early Reading First: Final Report.” Washington, DC: U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance, May 2007.

Kearney, Kathleen, Ronald Hopkins, Armand Mauss, and Ralph Weisheit. “Sample bias resulting from a requirement for written parental consent.” Public Opinion Quarterly, vol. 47, 1983.

Lankford, Hamilton, Susanna Loeb, and James Wyckoff. “Teacher Sorting and the Plight of Urban Schools: A Descriptive Analysis.” Educational Evaluation and Policy Analysis, vol. 24, no. 1, 2002, pp. 37–62.

Lueptow, L., S. Mueller, R. Hammes, and L. Master. “The impact of informed consent regulations on response rate and response bias.” Social Methods and Research, vol. 6, 1977.

Peikes, Deborah N., Lorenzo Moreno, and Sean Michael Orzol. “Propensity Score Matching: A Note of Caution for Evaluators of Social Programs.” The American Statistician, vol. 62, no. 3, August 2008, pp. 222–231.

Pokorny, Steveb B., Leonard Jason, Michael Schoeny, Stephanie Townsend, and Carrie Curie. “Do participation rates change when active consent procedures replace passive consent?” Evaluation Review, vol. 25, no. 567, 2001.

Rock, Donald A., and Judith Pollack. "Early Childhood Longitudinal Study - Kindergarten Class of 1998-99 (ECLS-K): Psychometric Report for Kindergarten Through First Grade." Washington, DC: National Center for Education Statistics, August 2002.

Smith, Jeffrey A., and Petra E. Todd. “Reconciling Conflicting Evidence on the Performance of Propensity-Score Matching Methods.” American Economic Review, vol. 91, no. 2, May 2001, pp. 112–118.

Thompson, Teresa, “A comparison of methods of increasing parental consent rates in social research.” The Public Opinion Quarterly, vol. 48, no. 4, 1984.

Wilson, Suzanne, Robert E. Floden, and Joan Ferrini-Mundy. “Teacher Preparation Research: An Insider’s View from the Outside.” Journal of Teacher Education, vol. 53, no. 3, 2002, pp. 190–204.

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research

1 Throughout this document, novice teachers are defined as those in their first or second year of teaching.

2 In order to make the test scores comparable, we intend to normalize test scores to have a mean of zero and a standard deviation of one. Therefore, the average value of for all teachers will be equal to zero.