Supporting Statement Part B Teacher Prep

This Information Collection Request seeks clearance to select teacher preparation programs, and recruit districts and schools, collect student rosters, and administer a baseline student achievement test for a rigorous study of the effect on student learning of teachers who have experienced intensive clinical practice within the university-based preparation programs they chose to attend.

This study is being conducted by the Institute of Education Sciences, U.S. Department of Education (ED); it is being implemented by Abt Associates Inc. and its wholly owned subsidiary Abt SRBI, along with its partners, Chesapeake Research Associates, The Bench Group, Dillon-Goodson Research Associates, and Drs. Sharon Vaughn and Karen Wixson (together, the “Abt study team”).

The objective of this study is to use rigorous methods to examine certain university-based clinical practice features for novice teachers. Teachers who have experienced intensive clinical practice features as part of their preservice teacher preparation program, that they chose to attend, are hypothesized to produce higher average student test scores than teachers who have not experienced intensive clinical practice in the program that they chose to attend. Using a randomized controlled trial (RCT), students will be randomly assigned to a pair of teachers in the same school and grade level, one of whom will have experienced intensive clinical practice as part of their chosen preservice teacher preparation program (“treatment”) while the other will not have had that same experience as part of their chosen preservice teacher preparation program (“control”). The study will then examine the impact on student achievement of teachers who choose to enter teaching through a traditional university-based teacher preparation program that includes promising preparation features versus those teachers who choose to enter teaching through university-based programs that have more typical features.

The information collection request (ICR) will be submitted in two phases, because the study schedule requires that the process of identifying eligible teachers – including 1) the preliminary steps of defining the specific features of intensive clinical practice upon which the study will focus, 2) identifying and selecting the elementary school teacher preparation programs that provide such clinical practice features, 3) district and school recruitment, and 4) the identification of matched teacher pairs – will begin before all of the data collection instruments are developed and tested. Additionally, this ICR includes the collection of student rosters and the fall administration of student achievement tests which will be used to examine the statistical equivalence of students in the randomly assigned classrooms. The student rosters are needed in order to prepare for random assignment prior to the beginning of the school year.

Phase I – Recruitment and Random Assignment ICR. The study will use a multi-step process to identify feasible states for the study, select specific features related to intensive clinical practice, identify university-based teacher preparation programs that require such clinical practice, identify feasible districts and schools for the study, confirm eligibility of potential teachers for the study, and implement random assignment of students to treatment and control classrooms. The Phase I - Recruitment and Random Assignment ICR requests approval to 1) collect information from preparation programs about their requirements, focusing on features of clinical practice specifically, 2) collect information from teachers about their preparation to determine their eligibility for the study, and 3) to randomly assign students to treatment and control classrooms. This package also provides an overview of the study, including its design and data collection procedures.

A second package, Phase II - Data Collection ICR, to be submitted in June 2012 will request clearance for winter and spring data collection activities for the study. The second package will provide a detailed discussion of the data collection activities and copies or descriptions of the instruments (i.e., teacher survey and observation protocol).

In addition to the impact study described above, a second component of the study will use state level administrative data and value-added analyses to examine the relationship between achievement levels of students who are English Language learners (EL) and (as a contract option that may be exercised) students with disabilities (SWD), and the preparation requirements teachers experienced with regard to teaching such student populations. This component will gather information about preparation programs’ requirements with a focus on requirements that prepare teachers for working with diverse populations. Using extant test score data and information about teachers’ preparation programs in several states, it will examine the relationship between student test scores and different novice teachers’ preparation features. This component is based on extant data and is not included in the information collection requests for this study.

Respondent Universe and Sampling Methods

The respondent universe of the study will consist of recently graduated elementary-grade teachers who experienced intensive clinical practice as part of their preservice teacher preparation program that they chose to attend (“treatment”) and recently-graduated elementary-grade teacher who did not have that same experience as part of their preservice teacher preparation program that they chose to attend (“control”). As described in Part A of this Supporting Statement, the sample will be selected as follows:

The goal of Phase I – Recruitment and Random Assignment is to identify teachers for the study (and their schools and districts) by first identifying which teacher preparation programs deliver intensive clinical practice features of interest, and then finding teachers who (likely) experienced those features by locating the districts and schools that hired them. The final step of recruitment is to collect data to verify that teachers participated in intensive clinical practice while attending the program. The process for Phase I – Recruitment and Random Assignment is illustrated in Exhibit A.1 and detailed in the remainder of this section. Phase II – Data Collection is the subject of a subsequent ICR.

Selection of study states. State selection will be based on identifying states in which the study is most feasible, that is, where the study team is most likely, and most efficiently, able to recruit eligible teacher pairs for the study, as described above. Using the initial data on potential program candidates and the feedback from additional consultants to the study, the study team has selected the eight states listed in Part A as the primary focus of study recruitment efforts.

Exhibit B. 1: Teacher Preparation Program and District/School Recruiting and Student Random Assignment Process

Selecting Teacher Preparation Programs

The study hypothesis is that teachers who receive intensive clinical practice will be more effective teachers, and their students will therefore perform better on standardized assessments. An essential step is to identify preparation programs that implement those features. Research has indicated that there can be a gap between what programs describe and what they actually deliver (Walsh, 2011). Confirming that the teacher preparation programs have these features is therefore critical to ensuring a meaningful experimental contrast.

The identification of teacher preparation programs will involve compiling extant data to obtain systematic information on the eligible teacher preparation programs in the selected eight states. The extant data will consist of publically available program descriptions and requirements obtained from university and program websites, student handbooks, and course catalogues. The next step is a screen based on two criteria: (1) it must be an undergraduate program that trains teachers for elementary grades, and (2) it must send at least some graduates to public schools (rather than non-public schools).

Based on consultation with experts and IES, the study team will begin to collect data from teacher preparation programs about the targeted clinical experience feature(s) to expedite the identification of programs for the district and school recruiters. As described above, this information will be collected via teacher preparation program websites and by conducting interviews with program administrators using the ‘Teacher Preparation Program Interview Guide’ (Appendix B). The second activity involves prioritizing preparation programs based on at least two criteria: 1) information from large districts about the programs from which they often hire teachers; and 2) programs for which the study team has prior knowledge that they may implement intensive clinical experience feature(s).

Based on the information listed above, the study team will then conduct an in-depth review of data on preparation programs. This information will then be used for the recruitment of districts and schools, and the identification of potential matched teacher pairs.

The next step involves verification of the information gleaned from extant data sources and collection of information through interviews with program administrators. The study team will send administrators of eligible teacher preparation programs in the study states two documents: 1) the Teacher Preparation Program Support Letter from IES (Appendix B), which encourages programs to participate in the study and lets the administrator know that the study team will contact him or her, and 2) the Study Fact Sheet (Appendix B), which describes the study. Interviews with program administrators then will be conducted to confirm that identified features are available to all students and have been in place for at least three academic years, as well as to learn about relationships with hiring districts. To facilitate these calls, the study team will use the topics outlined in the Teacher Preparation Program Interview Guide, to develop and complete a questionnaire which will be programmed into an Access data entry form so it can be administered in a manner similar to Computer Aided Telephone Interviewing (CATI); Appendix B contains the topics which will be used to program the Access data entry form.

The data collected through the program review and interview process will be used to determine which programs are most likely to produce treatment or control teachers. These designations will be utilized by the recruitment team when recruiting districts, schools and teachers. Although the data collection on preparation programs will provide preliminary information about treatment and control status of program graduates, and will provide guidance to the recruitment team on likely treatment and control teachers, the final determination of a teacher’s status into treatment or control condition will come from the Teacher Background Form, as described below.

Selecting Targeted School Districts

In parallel with investigating and selecting teacher preparation programs that will be the focus of this study, the study team will also identify school districts that are targets for recruitment activities. Within the selected study states, the study team will first use extant data to identify districts that provide a greater likelihood of finding graduates of the selected teacher preparation programs along with a matched “control” group teacher in the same school and grade. This will be based on two criteria: (1) “large” districts, i.e., those with 10 or more elementary schools with >75 students in at least one K-6 grade and (2) districts located within a 100-mile proximity to one or more of the possible study teacher preparation programs. Geographic proximity of programs to hiring districts has been shown to be particularly important for teachers; recent research indicates that substantial proportions of teachers attend (teacher preparation programs in) college within 50 miles of their homes, and further, that substantial proportions of new teachers obtain employment within 50 miles of their homes (Boyd, Lankford, Loeb, & Wyckoff, 2005). Because the study will be conducted in states that cover large geographic areas, the study has expanded the proximity to a 100-mile radius. The study team will also obtain information from district web sites on the typical sources of new teacher hires for districts meeting these two criteria, and will augment this with informational calls to district human resource directors.

This information, combined with decisions about including particular programs which produce teachers trained with the features of interest, will be used to develop a short list of potential study districts that will be targeted for recruitment.

Recruiting Study Districts

The study team will contact targeted school districts (1) to confirm the feasibility of conducting the study in their elementary schools; (2) if implementation appears feasible, to obtain permission to begin contacting administrators about their possible study participation; and (3) to move forward in districts that use their own approval processes for research studies.

This process will begin by sending letters to the district superintendents that include, as attachments, a District Letter of Support from IES, expressing support and encouragement for participation in the study, and a District Study Fact Sheet both of which are provided in Appendix B. Following confirmation of delivery of the district recruitment package, study staff will call targeted districts. This contact will begin with a determination of the appropriate point of contact in the district. In some cases it may be the superintendent, and in other situations he/she may suggest working with another district-level administrator (e.g., the assistant administrator for elementary education, the head of human resources/teacher hiring, etc.).

Once the contact has been identified, recruiters will talk with designated staff about the study’s feasibility and acceptability for implementation in the 2012-13 school year. Recruiters will cover the following points:

An overview of the study and the purpose of the call – the sponsorship of the study by IES/ED, the goal and importance of the study, the general design of the study, and what we’re trying to accomplish with this call;
District’s eligibility for the study – describe district and school eligibility criteria, and determine if there are potentially eligible schools in the district;
Interest in study participation – describe in broad terms what the study will entail and get an indication of possible district participation; and,
Necessary procedures to secure district permission to begin contacting schools for recruitment and for permission to conduct the study, in particular district requirements regarding parental consent, the requirements for submission of Institutional Review Board (IRB) research applications, and the main contact point for the application, if required.

Especially with large districts (i.e., those with potential for multiple eligible schools) – the study team also is planning to meet with district staff in person. These meetings will cover the same points as the calls but will allow for participation of a wider set of decision-makers, and for a more in-depth exploration of study feasibility.

District authorization of the study will be formalized in a memorandum of understanding (MOU) to be signed by a district representative and the study director. In addition, the study will submit formal Institutional Review Board (IRB) research applications where necessary; in some instances, the district’s written application approval may substitute for the MOU to limit redundancy.

School Identification and Recruitment

In districts that give the study permission to contact schools, schools will be identified based on teacher placement information provided by the selected teacher preparation programs or school districts. As with districts, priority will be given to schools likely to have the highest potential for teacher matches. The school recruitment process will begin with a mailing to school principals that will include a School Letter of Support from IES, expressing support and encouragement for participation in the study, and the same District Study Fact Sheet (Appendix B). Recruiters will call principals to confirm the feasibility of finding matched pairs of eligible teachers, and to determine acceptability of conducting the study during the 2012-13 school year. Recruiters will focus conversations on these discussion points:

Introduce recruiter/study and the purpose of the call (refer to the mailed study materials). Indicate the sponsorship of the study by IES/ED, and the district’s approval to contact them directly.
Provide a brief overview of the study
Describe the need to find teacher pairs as a basic requirement of eligibility, and what the study team means by a teacher pair.
Review the key data collection requirements for a participating school.
Note the study team’s efforts to minimize the burden to be placed on schools and teachers.
Assess questions or concerns, arrange for a visit if possible

As noted, once interest and feasibility has been determined, the study team will schedule face-to-face meetings with the principal and other decision-makers at the school. This meeting will be an opportunity to discuss the study’s eligibility requirements of matched treatment and control teacher pairs, random assignment of students, data collection activities, and the study timeline. The desired outcome of these school meetings is evidence that there is a strong likelihood of having at least one matched teacher pair at the school for fall 2012, and school administration agreement to participate in the study.

The next step is to collect information on potential teacher pairs for the study. The Teacher Background Form (see Appendix B) is a short paper questionnaire to assess whether teacher pairs satisfy the eligibility criteria, including whether: 1) the candidate teacher is the lead teachers in a self-contained elementary school classroom in the same grade and teaches both reading and math; 2) students in the teacher pair’s classrooms come from the same pool (e.g., all regular students); 3) one teacher (i.e., the “treatment” teacher) in the match has experienced intensive clinical practice as part of the preservice teacher preparation program that he/she chose to attend and graduated from within the past three years, while the other teacher (i.e., the “control” teacher”) has not had that same experience as part of the preservice teacher preparation program that he/she chose to attend and graduated from within the past three years. To ascertain teacher eligibility, the Teacher Background Form will include short questions that pertain to characteristics of teachers (e.g., experience) and their preparation programs, in particular, to verify that teachers report having experienced the features of interest (for treatment teachers) or having not (for control teachers) in the preservice teacher preparation programs they chose to attend. Once the teacher background forms are completed and reviewed, determinations can be made about treatment/control teacher pairs within schools.

Random Assignment

The final step in the Phase I Recruitment and Random Assignment ICR is to collect student rosters to prepare for random assignment. Working closely with schools having eligible teacher pairs, the study team will obtain rosters of students who will be taught by those teachers. Using the student lists, a well-developed randomization procedure will be used to assign students to the two teachers’ classrooms. Students will have an equal chance of being assigned to either of the teachers in the pair. In some schools, students are divided into groups (e.g., “houses”, “pods”, “families”, “academies”), and random assignment may be coordinated with the creation of such groups. As necessary, the study will accommodate a small number of exceptions to random assignment, such as students who must be paired with a particular teacher or separated from other specific students, or students who must be purposefully assigned to achieve gender balance across classrooms (often a consideration in elementary classrooms). Because student enrollment will not be static, student rosters will be collected twice during the first two weeks of school and random assignment will continue from spring 2012 through the second week of school in fall 2012.

The study plans to administer student achievement tests in fall 2012 and spring 2013 to measure changes in students’ achievement over the course of the year; this activity will be discussed in more detail in the Phase II – Data Collection ICR. To verify the equivalence of student proficiency in the two randomly assigned classrooms, the study will use the fall (pre) test scores to ensure there are no significant baseline differences in student ability in the participating classrooms. Therefore, consent forms will be distributed to parents of students in classrooms of the teacher pairs in fall 2012 prior to the initial student achievement testing (Appendix B). While final decisions about the student achievement measure will be made over the course of the study’s first year and will take into consideration those assessments that can most effectively assess the skills related to clinical practice program features, a potential option is the SAT 10.¹

The ability of a study to detect differences between the treatment and control groups depends, in large part, on sample sizes. The study will consist of 100 “teacher pairs” (a total of 200 teachers) in approximately 100 schools. Each teacher pair consists of two teachers who meet the following criteria in the 2012-13 school year:

Both teachers are the lead teachers in their respective classes and teach both reading and math to students in those classes. This is a typical self-contained elementary school classroom.²
The students whom they are assigned come from the same “pool” (i.e., general education students, or both teachers serve predominantly ELL students or students with disabilities).
One teacher (which the study calls the “treatment” teacher) will have experienced intensive clinical practice as part of their teacher preparation program from which they graduated within the past three years. The other teacher (the “control” teacher) will not have had that same experience as part of their preservice teacher preparation program from which they graduated within the past three years.

Furthermore, it must be possible for researchers to randomly assign students between the teacher pair with no disruption to school scheduling procedures. Assuming each of the 200 teachers in the sample teaches a single class of students, and assuming 25 students per class, the study will include approximately 5,000 students. Based on experiences in two similar studies, IES believes that identifying and recruiting these teachers may require the study team to contact and screen as many as 100 teacher preparation programs and 1,000 schools in 10 states. ³

Statistical Methods for Sample Selection and Degree of Accuracy Needed
1. Sample Selection

Random sampling is not feasible because 1) no sampling frame exists, and 2) the study will necessarily and exclusively be limited to schools with eligible teacher matches. Thus, the study team proposes to create a purposive sample designed to meet the specified statistical standard of precision. As described above, the results will be valid estimates of the effect of the teachers who met the eligibility criteria. The results will not be formally generalizable to the universe of elementary grade teachers who experienced intensive clinical practice as part of the preservice teacher preparation program that they choose. This section describes in detail how the sample will be drawn and provides a summary of Phase I – Recruitment and Random Assignment that is further detailed in Part A of this Supporting Statement.

Selection of States. States will be identified purposively based on where the study team is most likely and most efficiently able to recruit eligible teacher pairs for the study. States will not be randomly chosen. Instead, recruitment will focus on large states with large districts, which in turn will have large schools, in order to maximize the chance of finding teacher matches and efficiently utilize the study’s resources.
Selection of Teacher Preparation Programs. The study team will collect extant data for all university-based programs in study states. The study team then will identify programs with the specified clinical experience features, and call program administrators to verify the universality and maturity of those features. Additionally, the calls may yield information about the relationships that programs with the targeted features have with hiring districts.
Selection of Districts. The goal of district recruitment is to secure permission to conduct the study and to obtain authorization to contact schools about the study. Districts will not be randomly selected. The study team will prioritize recruitment efforts by identifying from extant data the relationships teacher preparation program have with hiring districts (confirmed by calls with district administrators) and by cross-referencing information about teacher preparation programs that have graduated a large enough number of teachers and the names of districts that have hired and are likely to hire 20 or more elementary grade teachers per year. School districts will be identified based on where the study is most likely and most efficiently able to recruit eligible teacher pairs for the study.

Once study districts are identified, superintendents in the targeted districts will receive a letter from ED encouraging them to work with the study team and providing information about contacting the study team along with a study fact sheet. Following the mailings, senior study staff will conduct the initial telephone outreach and negotiations with school district staff. If districts have processes for study approval, the study team may need to submit formal research applications or conduct in-person meetings and presentations with the district.⁴

Selection of Schools. Once district approval is obtained, the study team will contact schools to determine if they anticipate having at least one eligible pair of “teacher matches” in the 2012-13 school year. Schools will be selected purposively based on their likelihood of having an eligible teacher pair and the feasibility of the study procedures. Once study schools are identified and districts have given permission, principals in those schools will receive a letter from ED including facts about the study and how to contact the study team. The team then will call the school principals to provide additional information about the study, answer questions and determine interest in and feasibility of participation. Once interest and feasibility have been determined during the introductory phone call, the study team will schedule an in-person meeting with each potential school principal and other appropriate school staff to discuss the study’s eligibility requirements of matched treatment and control teachers, random assignment of students, data collection activities, and timeline, as well as to verify that schools have hired teachers from the selected preparation programs within the past two years and/or plan to do so for the 2012-13 school year.
Selection of Teachers. Once a school has granted permission for the study, the study team will initiate a process to identify eligible teacher pairs. Eligibility criteria for inclusion in a teacher pair are delineated above in Part B.1 and in Part A of this Supporting Statement. Teacher (pair) eligibility will be determined using a short background form. In each school, the study team will include as many eligible teacher pairs as possible to reduce costs and maximize the efficiency of the study.
Selection of Students. All students who would be assigned to the classroom of one of the teacher pair and who are not exceptions to the random assignment (e.g., parents may want their child placed with a particular teacher; the study team will limit such exceptions to no more than 10%) will be included in the study sample. The eligible students will be randomly assigned to either classroom within the matched teacher pairs described above.

Estimation Procedures

This section presents the study’s analytic approach for addressing the study’s research questions (See Appendix A):

What is the impact on student achievement of teachers who choose to enter teaching through a traditional university-based teacher preparation program that includes promising preparation features versus those teachers who choose to enter teaching through university-based programs that have more typical features?
Among the teachers studied, what are the core features of their teacher preparation? In particular, to what extent does preparation vary on dimensions of clinical preparation?

3. What is the impact on the classroom practices of novice elementary school teachers who experienced intensive clinical practice as part of their preservice teacher preparation program that they chose to attend compared to novice elementary school teachers who did not have the same experience as part of their preservice teacher preparation program that they chose to attend?

4. What teacher preparation features (such as opportunities to teach throughout the preparation program, extent or nature of the clinical practice, and structured feedback during clinical practice) are associated with teacher effectiveness?

5. What teacher preparation features are associated with teacher effectiveness for special populations (i.e. Special Education Students and English Language Learners)?

Question 1 will be addressed within the experimental framework by impact analyses while the remaining questions will be addressed using descriptive and non-experimental analyses.

Impact Analyses

Random assignment of students to treatment/control (T/C) teachers is a central feature of this study; it ensures equivalence on average of students in both classrooms of the pair. Observed student outcome differences can then be attributed to teachers. However, differences may not be directly attributed to the difference in the intensity of clinical experience. The difference includes both “program” and “teacher” effects, where the latter reflects systematic differences between teachers due to their self-selection into their preparation program (for example, teachers with greater potential may choose to attend programs that have more intensive clinical experience).

As its wording implies, research question 1 above acknowledges this potential confound between the teacher and program effect. The most direct way of addressing this question is producing an estimate of the combination of the two effects, which is relevant as it is analogous to the hiring decision of a district administrator or a school principal: whether to hire a teacher who graduated from a program that featured intensive clinical practice.

To estimate the combined teacher and program effect, the study will measure effects using the following prototypical model:^⁵

(1)

where:

: post-test score (administered in spring of 2013) of student i in school j;

: indicator for the j^th school, which equals 1 if student i is in school j and 0 otherwise;

: indicator set to 1 if student i is assigned to a treatment teacher and 0 otherwise;

: vector of student i’s characteristics such as his/her pre-test score (from fall of 2012) and other demographic attributes such as gender, race/ethnicity, reduced price lunch eligibility, etc.;

: residual for student i assumed to be normally distributed with mean 0 and variance of .

Following Schochet (2008a) and Constantine et al. (2009), the model does not include a teacher level, since each school is likely to have only one treatment-control pair and the teacher level will be confounded with the school level.

In the model in Equation 1, is the effect (i.e., adjusted treatment vs. control difference) for school j and captures the “fixed” effect for school j.⁶ The effects for each school will be averaged with equal weights to yield an overall effect estimate, . The resulting estimates can be expressed in effect size units, calculated by dividing by the standard deviation of the outcome in the control group.

The study will also assess heterogeneity in estimated effects with respect to a limited number of student and teacher characteristics via subgroup analyses. Potential student characteristics for subgroup analyses include EL status, pre-test score, and grade-level. Subgroup analyses by teacher characteristics will be based on attributes of the treatment teacher, which may include experience and amount of course work required by the preparation program.

To supplement the analyses described above and to best isolate the program effect from the teacher effect, the study team will conduct two additional analyses. First, it will investigate which teacher characteristics explain self-selection of treatment and control teachers’ into preparation programs with and without intensive clinical practice by fitting a regression model that uses the treatment indicator as the outcome variable and teacher characteristics as the predictors. Second, the team will estimate the impact model in Equation 1, adding teacher characteristics believed to be correlated with selection of teachers into different preparation programs. Since this analysis attempts to account for teacher effects, it will address a slightly different question than question 1. It is important to note that these analyses may not account for all differences between the treatment and control teachers and the resulting impact estimate may still include a residual teacher effect; therefore, these analyses will be framed as exploratory with appropriate caveats indicating that the results are outside the experimental framework.

Descriptive and Non-Experimental Analyses

To answer the second research question, the study will provide a qualitative summary of the core features of preparation programs obtained from the extant data review (confirmed by the Teacher Preparation Program Interview) of the preparation programs of both treatment and control teachers and the surveys teachers were administered. In addition, for each dimension of clinical preparation, the study team will present descriptive tables that display the percentage of teachers who attended a preparation program implementing that dimension.

To address the third research question, the study team will estimate the average treatment-control differences in the teacher classroom practice measures that will be created based on classroom observations. These analyses will help identify potential pathways through which intensive clinical practice influences teachers’ effectiveness. The model below will be used:

(2) C

where:

: teacher classroom practice measure for teacher k;

: treatment indicator set to 1 for treatment teachers and to 0 for control teachers; and

: residual for teacher k, normally distributed with mean zero and variance .

In Equation 2, captures the average treatment-control difference across all treatment-control pairs and the estimate of reflects both the effect of preparation programs (program effect) and the effect of differences between treatment and control teachers (teacher effect) on the classroom practice measures since the model does not include any teacher controls.

The fourth research question aims to identify specific preparation features (such as opportunities to teach throughout the preparation program, extent and nature of clinical practice) that are related to teacher effectiveness (as measured by higher student test scores), essentially considering each preparation feature as a potential channel through which preparation programs affect teacher effectiveness. To address this question, the study will estimate the relationship between the school-specific estimates yielded by the impact model in Equation 1 ( ) and the treatment-control differences in the indicator variables that capture whether the preparation program of a teacher implemented a particular feature or not:

(3)

where:

: impact estimate for school j;

equals , where equals one if the treatment teacher in school j experienced the m^th targeted feature and zero otherwise and is defined in a similar fashion for the control teacher in school j; and

: residual for school j assumed to be normally distributed with mean zero and variance

In this model, captures the association between teachers’ effectiveness and the m^th feature while controlling for the other features. It is important to note that indicators of the treatment-control differences in the targeted features may be correlated, which may reduce the ability to disentangle one feature from others. Also note that the estimated associations between teachers’ effectiveness and program features may not be causal because of the self-selection of teachers into preparation programs.

Finally, the analyses conducted to address the fifth research question will compare test scores of EL students taught by teachers who graduated from preparation programs implementing all or some of the key targeted program features (group A) to those taught by teachers who have not experienced the targeted features (group B) using extant data collected from states in which it is possible to link student records to teachers. If there is sufficient extant data, the study team will perform a similar analysis for students with disabilities (SWD). These analyses will also control for student scores from the previous year and grade to account for the fact that particular types (e.g., higher or lower performing) of students may be assigned to particular teachers, essentially comparing value-added (for EL students) of teachers from group A to that from group B.

Degree of Accuracy Needed

This section presents the power analyses conducted to estimate the minimum detectable effect sizes (MDES) for the treatment/control differences on student and teacher outcomes to address the first and second research questions, respectively. The power analysis conducted for the student outcomes utilizes the MDES formula provided by Schochet (2005, 2008a) for an RCT design that entails the random assignment of students within schools and the corresponding impact model in Equation 1. A technical consideration is that this model treats school effects as fixed instead of random since the study will utilize a purposively selected sample of schools and will not generalize results beyond this sample. And as noted above it does not include a teacher level because teachers and schools would be confounded.

Power analyses were based on the following MDES formula:

(4)

where:

: significance level for a two-tailed test;

: statistical power;

is the inverse of the student’s t distribution function with degrees of freedom (which equals );

: standard deviation of the outcome measure (assumed to be 1 in this case);

: total number of schools;

t: number of teachers per school (set to 2);

: average number of students per classroom with a valid outcome measure after account for attrition and non-response; and

proportion of the student-level variance explained by the regressors in the model

The MDES estimates are based on the following parameters, which are consistent with those reported by Schochet (2005), Bloom et al. (2007), Hedges and Hedberg (2007), and Kane and Staiger (2008):

Two-sided hypothesis test with the significance level ( ) = 0.05;
Statistical power ( ) = 0.80;
Average number of students per classroom (n): 15 which reflects an average class size of 25 students and a combined total attrition and non-response rate of 40%;
Proportion of the outcome variance explained by covariates (e.g., pre-test, school fixed-effects, other covariates) – : As mentioned above, previously reported values of these parameters tend to vary between 0.25 and 0.75 and considering this range, the study team has conducted two separate analyses with these values to represent a lenient case (R²=0.75) and a more conservative case (R²=0.25).

The first two rows in Exhibit B-1 present the corresponding MDES estimates for the impact analyses that use student test scores and the full analytic sample of 100 teacher-pairs and those that are conducted with subsamples containing 25, 50, and 75 teacher-pairs. The latter estimates are intended to inform the statistical power for analyses that investigate (i) the impact of a particular feature implemented by a subgroup of the preparation programs included in the study and (ii) the impact for a particular subgroup of teachers defined by a characteristics of interest (e.g., experience level, highest degree obtained). As seen in Exhibit B-1, the estimated MDES for the full sample is 0.05 for the lenient value and 0.09 for the conservative value. The study team anticipates the “realized” MDES for the study to be between these two estimates, which are deemed as policy-relevant and attainable as the year-to-year growth in reading and math test scores of the targeted student population is estimated to be around 0.4 standard deviations (Bloom, Hill, Black, and Lipsey, 2008). Exhibit B-1 also shows that for the subgroup analysis conducted with 25 teacher pairs, the MDES estimate is 0.10 for the lenient case and 0.18 for the conservative case.

As noted above, the team has also calculated MDES estimates for the comparisons of classroom practices across the treatment and control teachers using the regression model in Equation 2. These analyses utilized the following formula:

(5)

where denotes the standard deviation of the teacher practice measure (assumed to be 1 in this case) and the remaining parameters are as described above except for the fact that the degrees of freedom for these analyses is set to . The third row in Exhibit B-1 reports the corresponding MDES estimate, which is 0.4 standard deviations for the full-sample and 0.81 standard deviations for the quarter of the full-sample.

Exhibit B-1. MDES Estimates for Student and Teacher Measures

Outcome Measure	Percentage of the Full Analytic Sample Used in the Analysis
Outcome Measure	100%	75%	50%	25%
Student, Lenient	0.05	0.06	0.07	0.10
Student, Conservative	0.09	0.10	0.13	0.18
Teacher	0.40	0.46	0.57	0.81

Unusual Problems Requiring Specialized Sampling Procedures

Unusual problems that require specialized sampling procedures are not anticipated.

Use of Periodic Data Collection Cycles to Reduce Burden

This is a one-time data collection effort.

Methods to Maximize Response Rates and Deal with Nonresponse

This section describes the strategies and methods that will be used to maximize response rates and deal with non-response for Phase I – Recruitment and Random Assignment, i.e., collecting information on the characteristics of teacher preparation programs using the Teacher Preparation Program Interview and determining the eligibility of interested teachers through completion and review of the Teacher Background Form. For both activities, all sample members must have completed the data collection process in order to be considered eligible for the study. The study team will therefore have data on 100 percent of the study sample in this data collection.

In order to obtain responses from the study sample, the study team has developed the following strategies for facilitating communication with district and school respondents during recruitment activities and maximizing response rates for telephone calls, on-site meetings, and the completion of study forms. These strategies have proven successful in the study team’s extensive experience conducting large-scale evaluation studies (e.g., The Reading First Impact Study, Evaluation of the U.S. Department of Education’s Student Mentoring Program).

Trained senior-level recruiters. Recruiters will be senior staff trained to present information, address concerns, and respond to questions clearly, quickly, and efficiently. Recruiters will also be trained to demonstrate knowledge and understanding of the issues facing district and school administrators and elementary school teachers trying to complete their day-to-day activities.
Credible study. The study team will use letters from ED, on ED letterhead, for notification letters and recruiters will indicate that they are calling on behalf of a study funded by ED when they speak with representatives of programs, districts, and schools. Where appropriate, recruiters and study materials will also note that the study has been reviewed by both the Office of Management and Budget (OMB) and an independent Institutional Review Board (IRB). Finally, the team may be able to leverage the support of teacher preparation programs to recruit their alumni.
Flexible and responsive recruitment. Recruitment is scheduled over a period of several months to ensure that the busy schedules of program, district and school administrators can be accommodated. Study team members will maintain regular contact to monitor progress and resolve problems quickly. The team will be led by a recruitment task leader who will monitor recruitment and issues daily so as to quickly resolve any obstacles to participation.
Designated point of contact. To build connections with study participants, a single point of contact will be assigned from the recruiting team to each program, district and school. Administrators will be asked to designate a point of contact for their organization as well. The recruiter will use follow-up and reminder calls and emails to stay in contact with study participants, and to encourage those who have not responded to outreach efforts or returned study forms to do so.
Fully-informed decision-making. Recruiters will provide sufficient information about the study design, objectives, and methodology so that potential participants have an informed basis for their decision to participate. Additionally, they will provide potential participants with a realistic appraisal of the contributions in time, information, space, and human resources they will be expected to invest in the study effort and a statement of anticipated benefits (including honoraria and incentives).

These strategies have been proven to foster honest and collaborative relationships between the research team and study participants, which in turn, lead to high participation rates in telephone conversations and onsite meetings and high response rates on study surveys and forms.

Test of Procedures and Methods to be Undertaken

The Teacher Preparation Program Interview was created for this survey. To obtain a true estimate of the burden associated with the Teacher Preparation Program Interview and to assess the efficacy of elements selected for inclusion, the Teacher Preparation Program Interview Guide was pilot tested with 9 teacher preparation program administrators in 9 programs. Pilot testing simulated the planned administrator interview. The respondents were interviewed by phone by a study member using the protocol. Administrators responded to all of the questions on the protocol. Interview completion time was recorded for each administrator, along with an indication of any question where clarification was sought, inappropriate information was provided, or the respondent appeared to hesitate or stumble when answering the question.

Modifications to the length, content, and structure of the Teacher Preparation Program Interview have been made based on the results of the feedback provided by administrators involved in the pilot test. Several questions have been reworded to ensure clarity and reliability across program administrators in all states. Other questions were deemed unnecessary to determine characteristics of the teacher preparation program and were deleted from the guide.

The Teacher Background Form was modeled on forms used in previous education studies by Abt Associates on teacher educational background and on two ED studies on alternative certification.⁷

To obtain a true estimate of the burden associated with the Teacher Background Form and to assess the efficacy of elements selected for inclusion, it was pilot tested with 9 teachers. The respondents were asked to complete the forms and return them to the study team. A member of the study team reviewed the completed forms and conducted a short telephone follow-up with each teacher to ask respondents to provide feedback on the time necessary to complete the form and the extent to which the questions on the form were clear and able to be understood.

Modifications to the length, content, and structure of the Teacher Background Form have been made based on the results of the feedback provided by teachers involved in the pilot test. Several questions have been reworded to ensure clarity and reliability across teachers in all states. Other questions were deemed unnecessary to determine characteristics of teachers’ preparation and were deleted from the form. The structure of the form was also examined during the pilot test and questions on the form were rearranged in order to determine initial eligibility within the first few questions. This reorganization will reduce the burden of having ineligible teachers complete the full array of questions.

Individuals Consulted on Statistical Aspects of the Design

The following individuals were consulted on the statistical aspects of the study:

Name	Title/Affiliation	Telephone
Dr. Fatih Unlu	Associate/Scientist, Abt Associates	617-520-2528
Dr. Mark Dynarski	Director/Pemberton Research	609-443-1981
Mr. Cristofer Price	Principal Scientist, Abt Associates	301-634-1852

References

Bloom, H.S. (1984). Accounting for no-shows in experimental evaluation designs. Evaluation Review, 8(2), 225-246.

Bloom, H.S., Richburg‐Hayes, L., & Black, A.R. (2007). Using covariates to improve precision for studies that randomize schools to evaluate educational interventions. Educational Evaluation and Policy Analysis, 29(1), 30–59.

Bloom, H.S., Hill, C.J., Black, A.R. & Lipsey, M.W. (2008). Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions. MDRC Working Papers on Research Methodology. New York: MDRC.

Clotfelter, C.T., Ladd, H.F., & Vigdor, J.L. (2007). How and why do teacher credentials matter for student achievement? (NBER Working Paper 12828). Cambridge, MA: National Bureau of Economic Research.

Constantine, J., Player D., Silva, T., Hallgren, K., Grider, M., & Deke, J. (2009). An Evaluation of teachers trained through different routes to certification, final report (NCEE 2009- 4043). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Decker, P.T., Mayer, D.P., & Glazerman, S. (2004). The effects of Teach For America on students: Findings from a national evaluation. Princeton, NJ: Mathematica Policy Research Inc.

Gennetian, L., Morris, J.B. & Bloom, H. (2005). Coupling the non-experimental technique of instrumental variables with experimental data to learn how programs create impacts. In H. Bloom (Ed.), Moving to the next level: Combining experimental and non-experimental methods to advance employment policy research. New York: Russell Sage Foundation.

Hedges, L.V., & Hedberg, E. (2007). Interclass correlation values for planning group-randomized trials in education. Educational Evaluation and Policy Analysis, 29(1), 60–87.

Kane, Thomas J., & Staiger, Douglas. (2008). Estimating teacher impacts on student achievement: An experimental evaluation (NBER Working Paper Series, Vol. 14607). Cambridge, MA: National Bureau of Economic Research.

Puma, M.J., Olsen, R.B., Bell, S.H. & Price, C. (2009). What to do when data are missing in group randomized controlled trials (NCEE 2009-0049). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Rivkin, S.G., Hanushek, E. A., & Kain, J.F. (2005). Teachers, schools, and academic achievement. Econometrica, 73(2), 417-58.

Rockoff, J.E. (2004). The impact of individual teachers on student achievement: Evidence from panel data. American Economic Review, 94(2), 247-52.

Schochet, P.Z. (2005). Statistical power for random assignment evaluations of education programs. (Document No. PR05-36). Princeton, NJ: Mathematica Policy Research

Schochet, P.Z. (2008). Statistical power for random assignment evaluations of education programs. Journal of Educational and Behavioral Statistics, 33 (1), 62-87.

Staiger, D. O., & Rockoff, J. E. (2010). Searching for Effective Teachers with Imperfect Information. Journal of Economic Perspectives, 24(3), 97-118.

1 http://education.pearsonassessments.com/haiweb/cultures/en-us/productdetail.htm?pid=SAT10C&Community=EA_PreK-12_API_Achievement

2 There may be some schools where upper elementary teachers are responsible for all the science or math instruction, for example, and students rotate through different teachers; random assignment in such schools is likely to be infeasible.

3 This assumption is based on experiences from two recent evaluations with methodological and contextual similarities to the proposed study: the national evaluation of the Teach for America program (Decker, Mayer, and Glazerman, 2004) and the recent IES-funded study of the alternative certification programs (Constantine et al., 2009).

4 All recruiters will participate in a training to ensure uniform understanding and consistency across recruiters in the use of the eligibility verification and site visit guides.

5 This approach is consistent with two recent evaluations with methodological and contextual similarities to the proposed study: the national evaluation of the Teach for America program (Decker, Mayer, and Glazerman, 2004) and the recent IES-funded study of the alternative certification programs (Constantine et al., 2009).

6 As mentioned above, the assumption is that each study school will have a T/C teacher pair. If there are any schools with more than one pair, fixed effects will be included and separate impact estimates will be calculated for each pair; essentially deeming each T/C teacher pair as a mini-experiment.

7 See Constantine, Player, Silva, Hallgren, Grider, and Deke, 2009; the same survey has also been adapted for NCEE’s current Impact on Secondary Math Achievement of Highly Selective Routes to Alternative Certification.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Abt Single-Sided Body Template
Author	Katie Gan
File Modified	0000-00-00
File Created	2021-01-31

Supporting Statement Part B Teacher Prep

Supporting Statement Part B Teacher Prep.docx

Study of Promising Features of Teacher Preparation Programs; Phase I - Recruitment

OMB: 1850-0891

Collection of Information Employing Statistical Methods

Introduction

Respondent Universe and Sampling Methods

Selecting Teacher Preparation Programs

Selecting Targeted School Districts

Recruiting Study Districts

School Identification and Recruitment

Statistical Methods for Sample Selection and Degree of Accuracy Needed

Sample Selection

Estimation Procedures

Descriptive and Non-Experimental Analyses

Degree of Accuracy Needed

Unusual Problems Requiring Specialized Sampling Procedures

Use of Periodic Data Collection Cycles to Reduce Burden

Methods to Maximize Response Rates and Deal with Nonresponse

Test of Procedures and Methods to be Undertaken

Individuals Consulted on Statistical Aspects of the Design

References