Att_OMB-PartA HSAC-Package.v9

Contract No.:
ED-04-CO-0112 (09)
MPR Reference No.: 6522-530

An Evaluation of
Secondary Math Teachers
From Two Highly
Selective Routes to
Alternative Certification Addendum
Part A: Supporting
Statement for Paperwork
Reduction Act Submission
April 23, 2009
Revised July 10, 2009
Revised July 31, 2009

Submitted to:
Institute of Education Sciences
IES/NCEE
U.S. Department of Education
555 New Jersey Avenue, NW
Washington, DC 20208
Project Officer:
Stefanie Schmidt

Submitted by:
Mathematica Policy Research, Inc.
P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005
Project Director:
Sheena McConnell

CONTENTS

Page
PART A: SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT
SUBMISSION ................................................................................................................1
A. JUSTIFICATION ....................................................................................................2
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

Circumstances Necessitating Collection of Information .................................2
Purposes and Uses of Data .............................................................................11
Use of Technology to Reduce Burden ...........................................................12
Efforts to Avoid Duplication of Effort ...........................................................14
Methods of Minimizing Burden on Small Entities ........................................14
Consequences of Not Collecting Data ...........................................................15
Special Circumstances ...................................................................................16
Federal Register Announcement and Consultation .......................................16
Payment or Gift to Respondents ....................................................................17
Confidentiality of the Data.............................................................................19
Additional Justification for Sensitive Questions............................................21
Estimates of Hour Burden ..............................................................................21
Estimate of Total Annual Cost Burden to Respondents or
Recordkeepers ................................................................................................23
Estimates of Annualized Cost to the Federal Government ............................24
Reasons for Program Changes or Adjustments .............................................24
Tabulation, Publication Plans, and Time Schedules ......................................25
Approval Not to Display the Expiration Date for OMB Approval ................27
Exception to the Certification Statement .......................................................27

REFERENCES..............................................................................................................29

ii

APPENDICES

APPENDIX A: TEACHER CONTACT FORM
APPENDIX B: TEACHER MATH ASSESSMENT
1. Teacher Letter
2. Teacher Consent Form
APPENDIX C: PARENT CONSENT
1. Passive Consent Letter for Middle School Students
2. Passive Consent Letter for High School Students
3. Active Consent Letter for Middle and High School Students
4. Active Consent Form for Middle School Students
5. Active Consent Form for High School Students
APPENDIX D: STUDENT RECORDS DATA REQUEST
1. District Letter
2. District Student Records Data Request Form
APPENDIX E: TEACHER SURVEY
1. Teacher Letter
2. Teacher Survey
APPENDIX F: STUDENT ASSENT FOR MATH ASSESSMENT
APPENDIX G: HSAC PROGRAM ADMINISTRATOR INTERVIEW
PROTOCOL

iii

EXHIBITS

Exhibit

Page

1

DATA COLLECTION PLAN ........................................................................................6

2

DECISION RULES FOR ADMINISTERING THE PRAXIS II SERIES
MATH SUBJECT TESTS ..............................................................................................9

3

BURDEN IN HOURS TO RESPONDENTS ...............................................................22

4

COST BURDEN TO RESPONDENTS .......................................................................24

5

TIMELINE FOR THE STUDY ....................................................................................28

iv

PART A: SUPPORTING STATEMENT FOR PAPERWORK
REDUCTION ACT SUBMISSION

This package requests clearance for data collection activities to support a rigorous
evaluation of secondary math teachers who have entered teaching through highly selective routes
to alternative certification (HSAC). This evaluation is being conducted by the Institute of
Education Sciences (IES), National Center for Education Evaluation, U.S. Department of
Education (ED); it is being implemented by Mathematica Policy Research, Inc. (MPR) and its
partners—Chesapeake Research Associates LLC and Branch Associates.
The objective of the evaluation is to estimate the impact on secondary student math
achievement of teachers who obtain certification via HSAC routes compared with teachers who
receive certification through traditional or less selective alternative certification routes. The
evaluation design is an experiment in which the researchers will randomly assign secondary
school students to a treatment or control group. The treatment group will be taught by an HSAC
teacher and the control group will be taught by a non-HSAC teacher. Both teachers must teach
the same math class at the same level under the same general conditions at the same school. We
will compare student math achievement between the treatment and control groups to estimate the
impact of HSAC teachers.
This is the second submission of a two-stage clearance request. The package was submitted
in two stages because the study schedule required that district and school recruitment begin
before all the data collection instruments were developed and tested. The first package requested
approval for recruitment of schools, a teacher background form, a pilot of the student
assessment, and the random assignment of students. In this package, we are requesting approval
for:
A teacher survey and collection of teacher contact information
A teacher math content knowledge assessment—the Praxis II series math subject
test—to be administered to teachers who were not required to take this test for
certification
A form for all teachers—whether they took the Praxis II series math subject test to
obtain certification or just for this study—to provide consent for the Educational
Testing Service (ETS) to release their scores on this assessment to the study team
Parent/guardian consent forms for the administration of a math assessment to high
school students and the collection of school records on middle and high school
students
Collection of school records data on student characteristics and scores on state or
district math assessments
A student math assessment and students’ assent for taking the assessment
1

A protocol for semi-structured interviews of HSAC program administrators
This package provides a detailed discussion of the procedures for these data collection
activities and copies of the forms and instruments.
A. JUSTIFICATION
1.

Circumstances Necessitating Collection of Information

a.

Statement of Need for a Rigorous Evaluation of HSAC Teachers

The specific legislation authorizing this data collection is Section 9601 of the Elementary
and Secondary Education Act of 1965 (ESEA), which permits ESEA program funds to be used
to evaluate activities that are authorized under this act. The No Child Left Behind Act of 2001
(NCLB), which reauthorized ESEA, emphasizes the importance of teacher quality in improving
student achievement. Title II, Part A of ESEA—the Improving Teacher Quality State Grants
program—provides nearly $3 billion a year to states to prepare, train, and recruit high-quality
teachers, especially in the areas of mathematics and science. The purpose of Title II, Part A is to
help states and local school districts improve student achievement through strategies for
improving teacher quality and increasing the number of highly qualified teachers. One allowable
use of Title II, Part A funds is ―carrying out programs that establish, expand, or improve
alternative routes for state certification of teachers and principals, especially in the areas of
mathematics and science.‖ Teachers who have not yet obtained full state certification can meet
the highly qualified teacher requirements of NCLB if they are participating in an alternative
route to certification program and demonstrate satisfactory progress toward full state certification
(Title I: Improving the Academic Achievement of the Disadvantaged, Final Regulations, 34 CFR
Part 200.56, December 2, 2002).
In response to the increasing teacher shortages faced by many school districts, 47 states have
established alternative routes to certification that allow teachers to begin teaching before
completing all of the training required for certification. While most teachers still follow
traditional certification routes, an increasing number of teachers are entering the profession
through alternative certification routes. By some estimates, about one-third of any given year’s
teachers have entered the profession via alternative certification routes (Feistritzer and Chester
2002).
Many alternative certification programs are not very selective and accept most of their
applicants. However, some are highly selective, requiring applicants to undergo challenging
interviews and a rigorous screening process and rejecting many or even most of their applicants.
Teachers from these highly selective alternative certification programs have often been seen as a
way to fill teacher shortages, especially in the areas of math and science (Ingersoll 2003; Boyd et
al. 2006). The number of new teachers entering teaching through HSAC programs has grown
rapidly since the founding of Teach For America, the first HSAC program, in 1990.
Despite the rising number of HSAC teachers, policymakers lack rigorous research evidence
about the effectiveness of HSAC teachers in improving student achievement, particularly at the
2

secondary level. To date, there has been one experimental study of HSAC teachers at the
elementary level (Decker et al. 2004). Several well implemented nonexperimental studies have
findings that suggest students of HSAC teachers at the secondary level perform at least as well
and sometimes slightly better on mathematics achievement tests than students of traditionally
certified teachers (Boyd et al. 2006; Kane et al. 2006; Xu et al. 2008). However, the
nonexperimental methods used by these studies leave open the possibility that any observed
differences in student achievement may be due to factors other than the HSAC teachers.
This evaluation is thus essential to determining whether efforts to place high-quality
alternatively certified teachers in classrooms are, in fact, having a measurable impact on student
achievement. This study aims to fill the knowledge gap by focusing on secondary teachers from
the two largest and most well-known HSAC programs: (1) Teach For America (TFA), and (2)
the Teaching Fellows programs and similar programs by other names, fostered by The New
Teacher Project (TNTP). TFA recruits recent graduates of some of the nation’s most prestigious
colleges. TNTP-affiliated programs focus on highly accomplished people who began their
careers in other fields, but want to become teachers. For the feasibility study for this evaluation,
we conducted an extensive search for HSAC programs and identified five smaller HSAC
programs. We have excluded those smaller programs from our sample because they provide only
about 100 teachers per year (across all subjects and grade levels), compared with over 16,000
teachers provided by TFA and TNTP.
The study focuses on math at the secondary level for four reasons. First, secondary math
teacher shortages are widespread, so there is a high demand for HSAC teachers in this area.
Second, the United States lags behind many other industrialized countries in secondary math
achievement, suggesting a need for evidence on ways to enhance math achievement at this level.
Third, some have argued that HSAC teachers are most effective at teaching the more advanced
technical courses, such as secondary math. Fourth, a previous rigorous study found that TFA
elementary teachers produced greater achievement gains in math than other teachers in the same
grades and schools, but there were no differences in reading (Decker et al. 2004).
b. Research Questions
The primary research question of the evaluation is:
What is the impact on student math achievement of secondary school HSAC math
teachers compared with non-HSAC teachers?
The evaluation also will address the following secondary research questions:
What is the impact of secondary math TFA teachers compared with non-HSAC
teachers? What is the impact of secondary math TNTP-affiliated teachers compared
with non-HSAC teachers?

3

What is the impact of middle school HSAC math teachers? What is the impact of
high school HSAC math teachers?
To what extent do HSAC teachers differ in their educational backgrounds,
experience, and math content knowledge from other math teachers in the same
schools?
How does the impact of HSAC teachers vary with their educational backgrounds,
experience, and math content knowledge?
How do HSAC programs recruit, train, and support secondary school math teachers?
c.

Study Design

To answer the primary and secondary research questions, this study will use an experimental
design in which students in the same school are randomly assigned to either a class that is taught
by an HSAC teacher or a class that is taught by a non-HSAC teacher. The teachers in this
―classroom match‖ must teach the same subject at the same level under the same general
circumstances (for example, the same number of teachers in the classroom) in the same school.
Students randomly assigned to an HSAC teacher comprise the treatment group; those randomly
assigned to a non-HSAC teacher comprise the control group.
Random assignment is considered the ―gold standard‖ for social policy evaluations because
it, more than any other approach, minimizes the chance that any observed differences in
outcomes between the study groups are due to unmeasured, pre-existing differences between
members of the groups being studied. To determine whether an experimental evaluation of
HSAC teachers would be feasible, MPR staff visited 28 purposively selected schools and
concluded that under certain circumstances random assignment of students to HSAC and nonHSAC teachers was possible and that it was feasible to conduct an experimental evaluation of
HSAC teachers (Clark et al. 2008).
The ability of the study to detect policy-relevant differences between the treatment and
control groups depends, in large part, on the sample sizes. The study aims to include 450
classrooms matches or about 900 classes. Assuming an average of 20 students per class, the
study will include approximately 18,000 students. We expect that these matching classrooms will
include about 150 pairs of teachers (300 teachers), 112 schools, and 20 districts.
To examine the separate impacts of TFA and TNTP-affiliated programs, we will aim to
include roughly equal numbers of teachers from both types of programs. To examine the separate
impact of middle and high school HSAC teachers, we will aim to include roughly equal numbers
of middle and high school teachers.
Districts, schools, and classes/teachers will be selected purposively based on the feasibility
of their participation in an experimental evaluation and their willingness to participate. All
districts that expect to employ secondary math teachers from TFA or TNTP programs in the
study school year (2009-2010) are eligible to participate in the study. We will prioritize districts
with HSAC programs that have been in operation for three years or more, and districts with a

4

larger number of HSAC secondary math teachers. These districts are likely to include Baltimore,
Miami, New York City, Philadelphia/Camden, the San Francisco Bay Area, and
Washington, D.C.
A school is eligible to participate in the study if it: (1) is a public secondary school (and so
contains at least one of the grades 6-12) and (2) will have at least one set of two matching
classrooms—one taught by an HSAC teacher and one taught by a non-HSAC teacher—in the
2009-2010 school year and it is possible to randomly assign students to the classes. To
participate in the study, teachers of both classes must teach the same math class at the same level
under the same general circumstances. We expect that it will be feasible to randomly assign the
students to the classes most frequently when the classes are taught during the same class period,
as this is the least disruptive to schools’ schedules. For example, a match could be formed if
there was a first period Algebra I class taught by an HSAC teacher and a first period Algebra I
class taught by a non-HSAC teacher. Schools will be prioritized, like districts, to maximize
recruiting success, targeting the largest schools and those identified with the most potentially
eligible HSAC teachers.
d. Recruitment of Districts and Schools
To identify districts with HSAC teachers, we will request from TFA and TNTP programs
the names of current HSAC program participants and alumni who are teaching secondary math,
by region, district, and school. After prioritization of the districts, we will begin to contact and
recruit the districts. School recruitment will begin when districts grant us permission to begin
contacting schools directly. We will contact those schools that we expect will employ HSAC
math teachers during the study school year. As part of the school recruitment process, we will
initially screen schools for eligibility and willingness to participate, and we will confirm the
eligibility of teachers during in-person visits.
e.

Data Collection Needs

To address the study’s research questions, data will be required on students, teachers,
schools, and HSAC programs.
Students. The key outcome of interest for this evaluation is the students’ math achievement
at the end of the 2009-2010 school year. For middle school students, we will collect the spring
2010 test scores from state- or district-administered math assessments rather than administering
an assessment. As not all high school students take state- or district-administered math
assessments, and the tests that are administered to high school students are often not well aligned
to the course material they are taught, high school students will be administered an Northwest
Evaluation Association (NWEA) in-class adaptive, computerized math assessment in spring
2010. The assessment will measure each student’s level of knowledge in the math course he or
she is taking—Algebra I, Algebra II, Geometry, or a general math course.
Information on students’ demographic and socioeconomic characteristics and on their math
test scores prior to the study school year will be used both to describe the students in the study

5

and to use as covariates in regression models to develop more precise impact estimates. These
data will be obtained from students’ school records.
Teachers. To examine how HSAC and non-HSAC teachers differ, teachers will be
administered a survey in spring 2010 to collect information about their educational and
professional background and the training and support they receive over the 2009-2010 school
year. To ensure that we are able to administer the teacher survey to teachers who have left the
school during the school year, we will administer a teacher contact form in fall 2009. The contact
form will collect mailing address, telephone number, and e-mail address information from the
teachers.
A key difference between HSAC and non-HSAC teachers may be their knowledge of the
subject matter they teach. To explore the differences in math knowledge of the treatment and
control teachers in our sample, we will obtain study teachers’ scores on a teacher math content
knowledge assessment.
HSAC Programs. To understand how HSAC programs prepare people for teaching, the
team will conduct semi-structured telephone interviews with the administrators of all the
programs attended by teachers in the study.
f.

Data Collection Activities

Exhibit 1 summarizes the data collection activity (in chronological order). A brief
description of each activity is provided after Exhibit 1.
The first stage of this package requested clearance for the pilot study of the student
assessment, teacher background form, and the collection of classroom rosters for random
assignment, including the initial rosters, lists of late enrolled students, and updated classroom
rosters. The remaining data collection activities—collecting the teacher contact form, requesting
the teachers take a math assessment and release the scores directly to MPR, requesting student
records, administering a teacher survey, obtaining parent/guardian consent for the student’s
participation in the study, administering a math assessment to high school students, and
interviewing HSAC program administrators—are part of this clearance request.
EXHIBIT 1
DATA COLLECTION PLAN
Schedule
Spring 2009
(completed)

Activity

Respondent

Pilot Study of Student Students will participate in pilot
Math Assessmenta
administration of NWEA
General Math, Algebra I,
Algebra II, and Geometry
assessments

6

Mode

High school
Computer
students in General adaptive
Math, Algebra I, assessment
Algebra II, and
Geometry classes

Exhibit 1 (continued)

Schedule

Activity

Respondent

Mode

Spring - Summer
2009; Fall 2009 Spring 2010

Teacher Background During school recruitment visit, Teachers (or
Forma
request teachers (or principals on principals)
behalf of teachers) complete
form to confirm their route to
certification. Also request
information from any
replacement teacher during the
2009-2010 school year.

Hard copy

Spring - Summer
2009

Classroom rostersa

Electronic or
hard copy

Obtain classroom rosters of
School staff
students in 2009/10 school year
to randomly assign students to
either HSAC or non-HSAC
classrooms

Fall 2009 (first two List of late enrolling Obtain names of students who
weeks of fall
studentsa
enroll in school after initial
semester)
random assignment has been
conducted

School staff

Fall 2009

Teacher contact form Obtain personal contact
Teachers
information from study teachers
to enable contact if teacher
leaves the study school prior to
spring data collection

Fall 2009

Teacher math
assessment and
consent form to
release scores to
MPR

Request teacher take the ETS
Praxis math subject test and
obtain consent to obtain score
from ETS

Consent forms for
school records data
collection and for
testing (high schools
only)

School records: consent not
Parents and legal
required by Federal law; obtain guardians of
passive consent if district
students
requires consent

Fall 2009

Electronic or
hard copy

Hard copy

Teachers in states Praxis hard copy
that don’t require it assessment
for certification
Consent form
requested from all
teachers

High school math assessment:
passive consent, unless district
requires active consent

7

Hard copy

Exhibit 1 (continued)
Schedule

Activity

Respondent

Mode

Fall 2009
Early Spring 2010
Spring 2010

Classroom roster
checksa

Verify the students enrolled in
study classes

School staff

Electronic or
hard copy

Spring 2010

Teacher survey

Collect data on training and
support received by teachers
during school year

Teachers

Web, hard copy,
telephone
reminder

Spring 2010

High school student Measure high school student
Students
math assessment and math achievement in current
student assent
math course and request student
assent prior to assessment

NWEA computer
adaptive
assessment

Student records data Collect the following data
collection
Spring 2006-2010 math
standardized test score data

Electronic or
hard copy

Spring 2010 –
Initial request
Summer 2010 Collect data

District or school
staff

Student characteristics data for
school year 2009-2010
Spring 2010

a

HSAC program
administrators

Collect information on HSAC
HSAC program
programs’ recruiting, selection, administrators
placement, training, and support
strategies for secondary school
math teachers

Telephone semistructured
interviews

Clearance requested in prior OMB package submission; package was cleared on 6/7/09.

Teacher Contact Form. We will administer a teacher contact form at the beginning of the
2009-2010 school year and whenever a new teacher joins a study classroom (Appendix A). This
form will request detailed contact information for each teacher in the study. The information
collected by the form will be used to contact teachers who leave the school during the study
school year so we can ask them to complete the teacher survey in the spring.
Teacher Math Assessment. In fall 2009, we will administer the ETS Praxis Middle School
Mathematics (0069) test to teachers in grades 6-8 and the Praxis Content Knowledge in
Mathematics (0061) test to teachers in grades 9-12 based on whether the state required teachers
to pass these tests for certification (see Exhibit 2). In the states where we will not administer the
Praxis math subject test, we will collect the scores that study teachers obtained when they took
the test. These two hour long tests include both the math content we expect study teachers to be
teaching as well as some more advanced content. Both tests cover numbers and operations,
algebra and functions, geometry and measurement, data analysis, statistics, and probability. The
Content Knowledge in Mathematics test also covers trigonometry, calculus, and matrix algebra
Teachers will be administered the paper-and-pencil test at a school or site in their district
(Appendix B). All teachers will be asked to sign a teacher consent form to allow the study team
to obtain the teacher’s test score directly from ETS (Appendix B).

8

EXHIBIT 2
DECISION RULES FOR ADMINISTERING THE
PRAXIS II SERIES MATH SUBJECT TESTS
STATE CERTIFICATION REQUIREMENTS

ADMINISTER OR ACCEPT

High School Teachers
State requires Praxis 0061 test
State does not require Praxis 0061 test

Accept extant test score for Praxis 0061 test
Administer Praxis 0061 test

Middle School Teachers
State requires Praxis 0061 test only
State requires Praxis 0069 test only
State requires Praxis 0061 or 0069 test

Accept extant test score for Praxis 0061 test
Accept extant test score for Praxis 0069 test
Accept extant test score for Praxis 0061 or
Praxis 0069 test
State requires neither Praxis 0061 nor Praxis Administer Praxis 0069 test
0069 test

Parent Consent for High School Student Math Assessment and Middle and High
School Student Records Data Collection. Federal rules permit the U.S. Department of
Education and its designated agents to collect student demographic and existing achievement
data from schools and districts without prior parental or student consent. Since this study will not
involve administering a math assessment to middle school students, parental consent for these
students is not required by federal law. To maximize response rates and minimize burden on
schools and parents, we prefer to follow federal rules and forgo consent for middle school
students and use passive consent for the high school student math assessment. However, we will
comply with whatever procedures the school districts deem appropriate for middle and high
school students, including obtaining active consent for all students if necessary.
In fall 2009, all students for whom passive or active consent is required will be asked to take
home a consent form and/or notification letter to their parents or guardians (Appendix C). These
documents will be translated into other languages as needed. The documents will inform parents
and guardians that their child’s classroom has been selected for a national study of HSAC
teachers, that participation in the study is voluntary, and that it will involve schools/districts
providing characteristics and test score data for their child. For the high school students, the
consent documents will indicate that students will be asked to complete a standardized math
assessment in class at the end of the school year. The consent documents will also specify that
the information collected will be kept confidential and will only be reported in aggregate. The
consent documents are modeled after documents we have used in other evaluations.
If passive consent is acceptable to school districts for some or all students, the letters will
ask parents to call MPR if they do not want their child to participate in the study. Some districts,
however, may require active consent. If this is the case, the consent form will ask parents and
guardians to permit their child to participate in the study by returning a signed consent form to

9

their children’s school. If parents or guardians do not return a signed consent form, their child
will be excluded from the study.
To comply with the requirements of the Chicago public school district, we will use active
parental consent. We will also modify the consent form to inform parents that they can call MPR
to obtain their child’s test scores and to request permission to provide their child’s test score to
the teacher (Appendix C).
Student Records Data Collection. We will request standardized math test scores for all
students for spring 2006 through spring 2010. We will also request data on student
characteristics for school year 2009-2010, including sex, race/ethnicity, date of birth, grade,
whether they are repeating a grade, whether they are eligible to receive free or reduced-price
school lunch, whether they are an English language learner, and whether they have an individual
education plan (IEP) or 504 plan. We will request these data first from the district. If the data are
not available from the district, we will request the data from the schools. To collect these data,
we will send the district or school a letter that will specify the data requested, the students in the
study, and alternative ways to submit the data (Appendix D).
Teacher Survey. Teachers will be asked to complete a 30-minute survey in spring 2010, at
the end of the study school year (Appendix E). This survey will ask about the college they
attended, their college major and minor, any math-related coursework, and previous math-related
work experience; the content and timing of certification-related coursework; training, mentoring,
and coaching experiences during the school year; and student teaching experience. Teachers will
have the option of completing the teacher survey online, using a self-administered paper
questionnaire, or via telephone.
Student Math Assessment. At the end of the 2009-2010 school year, high school students
will be asked to take the Northwest Evaluation Association (NWEA) end-of-course math test for
the course they are taking—General Math, Algebra I, Algebra II, or Geometry. The NWEA
assessment is adaptive—the difficulty of the questions presented to the student adapts to the
student’s performance on prior questions—and so can precisely measure student achievement.
The test is taken on a computer.
Some students will be absent on testing day, move to another class during the school year, or
leave the school. We will attempt to test all high school students except those who move out of
the district. We will ask school officials to allow students who have switched classes to attend
the regularly scheduled test session. We will ask students who cannot attend the regularly
scheduled test session, have switched schools but stayed in the district, or have dropped out of
school to take the test on a Saturday at a central community location, such as a library.
Student Assent for High School Student Math Assessment. Prior to beginning the test,
each student will be asked to read a question that asks for his or her assent to take the test
(Appendix F). The student will agree to participate in the assessment by checking the appropriate
box. Any students who choose not to participate in the assessment will be asked to sit or read
quietly at their desk while their classmates continue with the test, or will engage in another
activity outside of the classroom as deemed appropriate by the teacher or school.

10

HSAC Program Administrator Interviews. Semi structured interviews will be conducted
with HSAC regional program administrators in spring 2010. The interview protocol will collect
information on the strategies used to recruit, select, place, train, and support secondary math
teachers (Appendix G).
g.

Analysis

The study will estimate overall impacts and impacts for subgroups, including TFA teachers,
TNTP teachers, middle school teachers, and high school teachers. We will investigate the extent
to which differences in effectiveness are correlated with differences in educational background,
experience, and math content knowledge between the HSAC and non-HSAC teachers. To
understand how the impact of HSAC teachers varies with their characteristics, we will estimate
the impact of teachers defined by certain characteristics, including years of experience.
h. Study Timeline
The study is expected to be completed in four years. The experimental evaluation will be
implemented in the 2009-2010 school year. A report on the study findings will be available in
spring 2012.
2.

Purposes and Uses of Data

Information will be collected by Mathematica Policy Research, Inc. and its partners,
Chesapeake Research Associates LLC and Branch Associates, under contract with ED [contract
number ED-04-CO-0112 (09)].
The information collected by the teacher contact form will be used to contact study teachers
who leave the school during the school year so that we can ask them to complete the teacher
survey in the spring. This information will increase the response to the teacher survey.
The teacher math assessment will measure differences in math knowledge between the
HSAC and non-HSAC teachers. These data will be used to describe the differences in math
content knowledge between HSAC and non-HSAC teachers and to analyze the relationship
between teacher math content knowledge and student achievement at the secondary level. The
teacher consent form will be used to allow the study team to obtain the teachers’ test scores
directly from ETS, thus reducing the burden on the teachers and increasing the proportion of
teachers for whom we have scores.
The parent consent forms for the student math assessment and student records data
collection will be used to identify students who are permitted by their parents to participate in the
study.
Data collected by the teacher survey will be used to describe the teachers and to analyze
how the impact of HSAC teachers varies with their educational backgrounds and work
experiences.

11

The scores obtained from the high school student math assessment and the 2010 state or
district math assessment scores collected from administrative records will be used to measure
the key outcome of interest—student math achievement. Administering the math assessment to
high school students is necessary because districts are not required by NCLB to test students in
grades 9 through 12 annually. Moreover, the tests they do administer are often not specific to a
particular course and so may not detect differences in student achievement. However, when
available, we will collect 2010 scores on district or state assessments for high school students so
we can compare the findings from the district/state assessment and the assessment we administer.
The school records data on student characteristics and scores on prior years’ state or district
math assessments will be used to describe the student sample and to improve the precision of the
impact estimates by including these characteristics as covariates in the impact estimation models.
They can also be used to confirm that random assignment has been well implemented, creating
treatment and control groups with similar characteristics. We will request data on state or district
assessments as far back as spring 2006 to ensure that we have prior test scores for all high school
students, including those in 12th grade who may have taken their last district/state assessment in
8th grade.
The information collected from the interviews with the HSAC program administrators will
be used to understand the teacher preparation programs in the study and provide detailed
contextual information to interpret findings on the impacts of HSAC teachers.
The study findings as a whole will be used to inform the efforts of national, state, and local
policymakers, districts, schools, and parents to improve student outcomes. This information will
help guide school districts in their teacher hiring decisions. The study results may also provide
policymakers with information on how to improve secondary math achievement in the United
States. Math achievement can have a meaningful impact on the future economic well-being of
students, with research confirming a correlation between student achievement on standardized
tests at the secondary level with post high school earnings (Murnane et al. 1995; Murnane et al.
2001; Lazear 2003; Deke and Haimson 2006). Knowledge of the HSAC teachers’ effectiveness
will help teacher preparation and certification program developers to design programs that have
the best chance of improving student outcomes.
Findings will be presented in a report in spring 2012. In addition, the data collected by the
evaluation will be submitted to ED as restricted-use data files that will serve as a valuable
resource for other researchers to further examine this issue.
3.

Use of Technology to Reduce Burden

For each data collection task, we have selected the form of technology that enables us to
obtain reliable information in an efficient way that minimizes respondent burden.
The teacher math assessments— the Praxis Middle School Mathematics (0069) test and the
Praxis Content Knowledge in Mathematics (0061) test—are administered only as pencil and
paper instruments and will be administered by MPR at a school or a site in the teacher’s district.
The Praxis (0069) assessment will be administered to only middle school teachers in states that
do not require this test for certification. High school teachers in states that do not require the
12

Praxis (0061) test for certification will be administered this test. ETS will have the option of
providing us the score data in electronic or in hard copy format.
At the beginning of the school year, we will request that teachers complete a hard-copy
teacher contact form and consent to release assessment scores forms. To minimize burden, we
will deliver the forms in person to the teachers if we are at the school at that time and teachers
will then have the opportunity to complete it quickly and hand it to the researchers in person.
Otherwise, we will mail the forms to the teachers at their schools and they will have the option of
returning the forms to MPR by mail or fax.
To maximize access, as not all parents will have access to email, the parent consent forms
will be available in hard copy format. We will request that teachers ask their students to give the
consent documents to their parents and in the case of active consent forms, to return signed
forms to their teachers. The consent document will list a telephone number which parents can
call to ask questions about the study.
Districts and schools providing student records and test score data can provide the records
in whatever format is least burdensome. They will have the option of delivering the data
electronically, as well as submitting hard-copy documents that already exist. We will provide
clear instructions on the data requested.
The teacher survey will be administered as a web-based survey; teachers will also have the
option of completing a self-administered paper questionnaire or completing the survey by
telephone. The online survey will enable teachers to complete the survey at a location and time
of their choice, and its automatic editing system will reduce the level of response errors. We will
set up a toll-free telephone number and electronic mail address so that teachers can easily contact
researchers with questions they may have.
The high school student math assessment is a computerized adaptive test developed by
NWEA. We have selected this assessment to minimize the burden on students. Adaptive tests
have been found to be more efficient (taking less time to complete) and decrease the possibility
of floor or ceiling effects (Rock and Pollack 2002). The assessment adapts the test questions to a
student’s ability level until the student’s achievement level is precisely estimated. The test begins
with a question (presented on the computer screen) that is at the middle range of difficulty for the
course material. If the student answers the question correctly, the next question presented will be
more difficult. If the question is answered incorrectly, the next question presented will be easier.
This process continues until a final assessment score is determined. The math assessment has
been thoroughly tested (NWEA 2003).
We will administer the math assessment in a group setting to all students whose parents have
not explicitly refused to allow their children to participate. Students will benefit from the
guidance of a test administrator and proctor who will be present to administer the student math
assessment, explain directions, and answer questions. MPR will provide laptop computers to
administer the assessment. The laptops are battery powered, and we will have a computer
technician available during the assessment to address any technology problems. Teachers will be
asked to stay during the testing but they will not be required to administer the assessment.

13

For the middle school students, we have sought to avoid imposing additional burden by
using test scores from administrative school records in place of administering a student math
assessment.
Interviews with the HSAC program administrators will be conducted by telephone, which
will allow us flexibility to schedule the interviews at their convenience.
4.

Efforts to Avoid Duplication of Effort

No other experimental evaluation of HSAC math teachers at the secondary school level has
been conducted. To date, there has been only one experimental evaluation of HSAC teachers that
studied teachers at the elementary level (Decker et al. 2004). Although there have been
nonexperimental studies (such as Boyd et al. 2006; Kane et al. 2006; Xu et al. 2008), the findings
are mixed and the nonexperimental methods used by these studies leave open the possibility that
observed differences in student achievement might be due to underlying differences between the
students taught by the HSAC and non-HSAC teachers rather than to true causal effects of the
HSAC teachers themselves.
To the extent possible, we will use existing data for the study rather than duplicate data
collection efforts. We are using test scores from the state- or district-administered student
assessments, instead of administering an assessment, to measure math achievement for the
middle school students. Because state assessments are often unavailable for high school students,
or are poorly aligned with their courses, we will administer a math assessment to high school
students. However, we will seek to collect the test scores for high school students on the 2010
state- or district-administered student assessments so that we can compare the findings for the
state/district assessments and the assessments we administer for some students.
Only middle school teachers in states that do not require the Praxis 0061 or 0069 test for
certification will be administered this test. Only high school teachers in states that do not require
the Praxis 0061 test for certification will be asked to take this assessment. Teachers in states that
do require the test for certification will not be asked to take the test again. Instead, we will obtain
from ETS the teachers’ scores on the test they took to obtain math certification.
The information collected from the teacher contact form, teacher consent form, student math
assessments, teacher survey, parent consent forms, and HSAC program administrator interviews
is not available elsewhere. No information will be collected from more than one source.
5.

Methods of Minimizing Burden on Small Entities

The primary small entities for this study are the districts and schools in which the study
teachers teach. For the data collection, burden is reduced for all study participants by requesting
only the minimum information required to meet the study objectives. The burden on schools and
districts has been minimized through the careful specification of information needs and
restricting items for the administrative data collection. By administering the student assessment
in class during one class session, we will not ask schools, which often have space limitations, to
locate an additional room for the testing. We will provide a laptop computer for each student to
14

take the test as well as a secure server and a trained computer technician will be present in each
school during testing.
The teacher assessment will be scheduled after school hours to minimize disruption to
classes and teaching schedules. We will work closely with the schools to select the most
convenient location to administer the teacher assessment. To minimize traveling time for the
teachers, we will first explore the possibility of administering the teacher assessment at the
school. If school policy or space constraints preclude this, we will identify a place to administer
the test that is close to the school.
6.

Consequences of Not Collecting Data

This evaluation is consistent with the goals of NCLB to raise student achievement by
requiring that all teachers in core academic subjects be highly qualified. Despite the increasing
use of HSAC teachers, there have been very few experimental studies on the effectiveness of
HSAC teachers. Thousands of new teachers are hired every year from HSAC programs with
little or no scientifically based evidence on whether these programs produce teachers who are
likely to be effective in the classroom. In the absence of this evaluation, ED will not be able to
gauge HSAC teachers’ effects on student achievement. This study will thus be an important
contribution to the policy debate. Its rigorous methodological design incorporating random
assignment of students will ensure that highly credible evidence about the impact of HSAC
teachers on student achievement is obtained.
The consequences of not collecting specific data items are discussed below.
Without the information from the teacher contact forms, it will be more challenging
to achieve a high response rate for the teacher survey.
Without a teacher math assessment, we will not have the data necessary to describe
the differences in math content knowledge between HSAC and non-HSAC teachers
and to analyze the relationship between teacher math knowledge and student math
achievement. While a National Mathematics Advisory Panel task group assigned to
examine teacher education concluded that teacher math content knowledge is likely
an important determinant of student achievement, it also emphasized the need for
further research on the role of content knowledge in instructional effectiveness (U.S.
Department of Education 2008a, 2008b). Without the signature on the teacher consent
form to release the score from ETS, we would need to request the score directly from
the teacher, reducing the response rate and increasing the burden on the teacher.
Without the student records data, we will have to administer a math assessment to
middle school students, in place of using their 2010 district/state math test scores, to
measure student math achievement. Without the data on student characteristics, we
will not be able to fully describe the study sample and verify the effectiveness of the
random assignment. Prior years’ math test scores together with data on student
characteristics will also be used as covariates in regression models and so increase the
precision of the impact estimates.

15

Without the teacher survey, we will not have the data necessary to describe how
HSAC and non-HSAC teachers differ and to analyze how student math achievement
may vary with differences in teachers’ characteristics, educational background, and
work experience.
Without the high school math assessment, we will not be able to obtain a valid
measure of math achievement for high school students because, unlike middle
schools, high schools are not required to administer a math assessment at every grade
level. Moreover, existing high school math assessments tend to be poorly aligned
with specific high school math courses (focusing instead on more general math
knowledge), which would limit their ability to detect impacts on students’ math
achievement of a particular teacher in a particular course.
Without the interviews with the HSAC program administrators, we will not be able
to describe the HSAC programs through which the HSAC teachers are recruited,
selected, trained, and supported.
7.

Special Circumstances
There are no special circumstances involved with the data collection.

8.

Federal Register Announcement and Consultation

a.

Federal Register Announcement

A 60-day notice to solicit public comments was published in the Federal Register on
February 24, 2009.
b. Consultations Outside the Agency
Professional counsel was sought from a number of experts during the feasibility study for
this evaluation. In January 2008, MPR convened a meeting of a Technical Working Group,
consisting of a broad range of researchers, to provide input on study design issues and the data
collection plan. These individuals were:
Brian Jacob, Walter H. Annenberg Professor of Education, Gerald R. Ford School of
Public Policy, University of Michigan
John Pane, Senior Scientist, RAND Corporation
Michael Petrilli, Vice President for National Programs and Policy, Thomas B.
Fordham Institute
Jeffrey Smith, Professor of Economics, University of Michigan
James Wyckoff, Professor of Economics, Rockefeller College, University of Albany

16

Paul Decker, President, Mathematica Policy Research, Inc.
John DeFlaminis, Practice Professor of Education, University of Pennsylvania
Graduate School of Education
c.

Unresolved Issues
There are no unresolved issues.

9.

Payment or Gift to Respondents

We propose offering teachers $5 total for completed teacher contact and consent forms.
This small payment will compensate the teachers for the time they spend completing the forms
and returning them to MPR. Obtaining these completed forms is key to obtaining a high response
rate to the teacher survey and to obtaining Praxis math scores for a high proportion of the study
teachers.
Payments will be given for participating in the teacher math assessment. Teachers will be
offered $120 payment to take the two-hour Praxis II series math subject test. A payment of $60
per hour is necessary given the burden of the test. Taking the math assessment will require
sustained concentration and may elicit anxiety about a low performance score. Furthermore,
teachers will have to spend two hours to take the test out of their own personal time and may
have to pay for child or other dependent care during that time. While we will try to schedule the
assessment on a weekday after school hours, scheduling conflicts may demand that the
assessment be scheduled for the weekend, thereby requiring teachers to make a special trip to the
testing site. Teachers taking the teacher math assessment will lack the types of compensation for
participation present in other studies conducted by the National Center for Education Evaluation
(NCEE)—they will receive no professional development or curriculum intervention. The amount
we propose is consistent with the guidelines for NCEE evaluation studies, prepared March 22,
2005, for a two-hour teacher assessment with high respondent burden.
The payment of $60 for each hour spent on the teacher assessment is consistent with
payments in previous ED and non-ED studies. In ED’s Mathematics Professional Development
study, the incentive for a 60-minute test on mathematics teacher content knowledge was $75 if
they went to a central location to take the test and $60 if they took the assessment at their own
school. ED’s Evaluation of Professional Development in Early Reading Instruction administered
a 30-minute teacher knowledge assessment that was created for the study, the Reading Content
and Practices Survey, and compensated the teachers $30 for taking the survey. Response rates for
the survey ranged from 91 to 97 percent across the three administrations. In the Longitudinal
Study of Certification Programs conducted by the American Board for Certification of Teacher
Excellence (ABCTE), teachers were asked to take a Praxis or ABCTE subject examination in
secondary math, elementary education, or pedagogy (which ranged from 2 hours to 4 hours
long). Teachers were paid $100 for taking each test and up to an additional $100 for performing
well on the test. Of the convenience sample of 117 teachers who agreed to take one of these tests,
97 teachers completed a test, for a completion rate of 83 percent.

17

To express our appreciation for participation in the high school student math assessment,
we propose offering a $5 gift to participating students. The $5 gift to students participating in the
assessment is necessary because NCEE has found in other studies, such as the DC Choice study,
that getting secondary students to take assessments seriously is a challenge, and we hope the
payment will ameliorate that problem. Students who take the test at a Saturday make-up session
will be offered $25 for their time and travel expenses in addition to the $5 gift offered to students
who take the test in class.
We propose to pay teachers $30 for a completed teacher survey as compensation for their
time and effort. The teacher survey will require 30 minutes to complete and is therefore
considered high burden. The $30 incentive is consistent with the incentive amount approved by
OMB for the Evaluation of the Impact of Teacher Induction Programs. In this study, teachers
received $30 to complete a baseline survey and permission form that averaged about 30 minutes
to complete. For the 20 minute Induction Activities survey, which was administered three times
within the first school year, teachers were offered $20 for each survey. Response rates for these
surveys averaged 90 percent.
The teacher and student payments are all within the incentive amounts suggested in the
memo, ―Guidelines for Incentives for National Center for Education Evaluation Impact
Evaluations,‖ prepared for OMB on March 22, 2005.
To explore whether incentives are effective in increasing the rate at which students return
consent forms in active consent districts, we propose to conduct an incentive experiment. The
experiment will investigate the effectiveness of two types of incentive. The first incentive is $25
offered to classrooms that collect signed consent forms for at least 95 percent of their students.
The second incentive is a $5 gift card offered to each student who returns the signed consent
form. Both types of incentives will be paid on the basis of returned forms, regardless of whether
the parent provided consent. We based our decision to provide a $5 gift card to students on two
studies. In the Impact Evaluation of Mandatory Random Study Drug Testing (OMB Approval
#1850-0818), students received a movie ticket ($7 value) for the return of a completed consent
form, regardless of the consent status. In the Evaluation of the Youth Transition Demonstration
Projects (OMB Approval #0960-0687), youth received a $10 Target gift card or Metrocard if
they returned the consent form, regardless of consent status.
Schools in districts that require active consent will be randomly assigned to one of three
groups:
1. Treatment 1: Classroom receives a $25 incentive if 95 percent or more of the consent
forms are returned and individual students are offered a $5 gift card if they return the
consent form.
2. Treatment 2: There is no classroom incentive; individual students are offered a $5 gift
card if they return the consent form.
3. Control. There is neither a class incentive nor a student financial incentive for
returning the form.

18

In each group of schools, we will document the percentage of forms returned each week,
starting with the week the forms are sent home with the students. Other procedures used to
encourage the return of the forms not involving financial incentives will be similar in each group.
Comparisons across groups of the number of forms returned each week will provide estimates of
whether student or classroom incentives are effective and whether offering both student and
classroom incentives is more effective than either student or classroom incentives alone. A
power analysis concluded that we will be able to detect a difference by group of 14 percentage
points or more in the rates at which the forms are returned, a difference much lower than found
in previous experiments (Thompson 1984). The results of the experiment will be documented
and presented to OMB.
Districts and schools will be offered nonfinancial compensation for participation in the data
collection activities, including the distribution and collection of parental consent forms,
facilitation of the administration of the student assessment, and collection of student records
data. Many school administrators hunger for information about their students and teachers as
well as evidence-based policy recommendations. When providing schools information from the
study, it is paramount that we preserve the confidentiality of the study participants. Hence, we
cannot offer schools student-level data on the results of the student math assessment that we will
administer in the high schools as this would violate the students’ confidentiality. Providing
student scores aggregated by teacher may discourage teachers, and hence schools, from agreeing
to participate in the study.
We will, however, offer high schools information on how their students performed on
aggregate on each math test, in comparison to students in other schools within the district and to
students nationally. If the school wishes, we could provide the mean and quartiles of scores for
students who take each test (Algebra I, Algebra II, Geometry, and Basic Math) in their school
along with the mean and quartiles of scores for all students in the district taking this test for our
study (as long as there are more than two schools in the district). This would provide school
administrators with information on how well their students are performing in his/her school
compared to students in other similar schools. We could also provide the national norms for
scores on each test by grade level that are constructed by NWEA. These data could be provided
with the school and district data so that the school administrators can compare how well their
students are performing compared with students in the same grade level nationwide.
We also will offer to notify schools when the results of the study are made public. By
sending them a short summary of the results and a link to the study report, the participating
schools can be among the first to know about the study findings.
10. Confidentiality of the Data
All data collection activities will be conducted in full compliance with ED regulations to
maintain the confidentiality of data obtained on private persons and to protect the rights and
welfare of human research subjects as contained in ED regulations.
The contractor will follow the policies and procedures required by the Education Sciences
Reform Act of 2002, Title I, Part E, Section 183, which requires ―[a]ll collection, maintenance,
use, and wide dissemination of data by the Institute‖ to ―conform with the requirements of
19

Section 552 of Title 5, United States Code, the confidentiality standards of Subsection (c) of this
section, and Sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g,
1232h).‖ These citations refer to the Privacy Act, the Family Educational Rights and Privacy
Act, and the Protection of Pupil Rights Amendment.
In addition, the contractor will ensure that all individually identifiable information about
students, their academic achievements, their families, and information with respect to individual
schools shall remain confidential in accordance with Section 552a of Title 5, United States Code;
the confidentiality standards of Subsection (c) of this section; and Sections 444 and 445 of the
General Education Provision Act. Subsection (c) of Section 183 referenced above requires the
Director of the Institute of Education Sciences to ―develop and enforce standards designed to
protect the confidentiality of persons in the collection, reporting, and publication of data.‖
Subsection (d) of Section 183 referenced above prohibits disclosure of individually identifiable
information as well as making any publishing or communicating of individually identifiable
information by employees or staff a felony.
Because this collection includes personally identifiable information, we are in the process of
obtaining approval for a System of Records Notice (SORN), which will ensure the secure
collection and storage of such information.
MPR and its subcontractors will protect the confidentiality of all information collected for
the study and will use it for research purposes only. No information that identifies any study
participant will be released. Further, personally identifiable data will not be entered into the
analysis file and data records will contain a numeric identifier only. When reporting the results,
data will be presented only in aggregate form so that individuals and institutions will not be
identified. A statement to this effect will be included with all requests for data. The teacher
contact form, teacher consent form, teacher survey, parent consent forms, and student assent
statement will include a reminder about confidentiality protection in compliance with the
legislation. When data are collected through telephone interviews, respondents will be reminded
about the confidentiality protections, the voluntary nature of the survey, and their right to refuse
to answer individual questions. Further, no individually identifiable information will be
maintained by the study team. All members of the study team having access to the data will be
trained on the importance of confidentiality and data security. All data will be kept in secured
locations, and identifiers will be destroyed as soon as they are no longer required.
The following safeguards will be employed by MPR to carry out confidentiality assurances
during the study:
All employees at MPR sign a confidentiality pledge emphasizing its importance and
describing their obligation.
Access to identifying information on sample members is limited to those who have
direct responsibility for providing and maintaining sample locating information. At
the conclusion of the research, these data are destroyed.
Identifying information is maintained on separate forms and files, which are linked
only by sample identification number.
20

Access to the file linking sample identification numbers with the respondents’ ID and
contact information is limited to a small number of individuals who have a need to
know this information.
Access to the hard-copy documents is strictly limited. Documents are stored in locked
files and cabinets. Discarded materials are shredded.
Computer data files are protected with passwords, and access is limited to specific
users. Especially sensitive data are maintained on removable storage devices that are
kept physically secure when not in use.
The Privacy Act of 1974 applies to this collection. MPR will make certain that all surveys
are held strictly confidential, as described above, and that in no instance will responses be made
available except in tabular form. Under no condition will information be made available to
school personnel. District and school staff responsible for assisting MPR in the data collection
will be fully informed of MPR’s policies and procedures regarding confidentiality of the data.
In addition, the following verbatim language will appear on all letters, brochures, and other
study materials:
Per the policies and procedures required by the Education Sciences Reform Act of
2002, Title I, Part E, Section 183, responses to this data collection will be used only for
statistical purposes. The reports prepared for this study will summarize findings across
the sample and will not associate responses with a specific district or individual. We
will not provide information that identifies you or your district to anyone outside the
study team, except as required by law. Any willful disclosure of such information for
nonstatistical purposes, without the informed consent of the respondent, is a class E
felony.
11. Additional Justification for Sensitive Questions
Some teachers may consider their contact information personal, but this information is
necessary to ensure that we are able to administer the teacher survey to teachers who leave their
school during the school year.
The test score data collected by the teacher assessment may be considered sensitive because
of the potential for school administrators to use these data improperly to evaluate the quality of
the teachers. To protect the confidentiality of these data, we will not link the scores to personal
identifiers and these data will be accessible only to the study team.
The teacher survey will contain background questions on teachers’ education level,
institutions at which they received their degree(s), race, ethnicity, and age. Some teachers may
consider this information sensitive. However, data on these topics are important to collect
because they will help us understand the differences between HSAC and non-HSAC teachers
and how the impact of HSAC teachers varies with their characteristics. Questions used to obtain

21

personal background information have been asked frequently in other surveys and the teachers
who participated in the pretest of the survey did not consider these questions to be sensitive.
Test scores and some other demographic information about the students may be sensitive.
Test score data is essential for this evaluation because student math achievement is the primary
outcome of interest. Demographic information is important to control for differences in the
characteristics of students in the classes that may have arisen by chance. The consent form will
address the collection of test scores and school records. In all districts except for Chicago Public
School District, only the study team will have access to these data. Individual test scores will not
be given to students, parents, teachers, schools, or districts. However, at the request of the
Chicago Public School District, we will provide teachers in Chicago access to the test scores of
students in their classes (with proper parental consent) and parents will have the opportunity to
obtain their child’s scores. In the Chicago Public School District, parents will have the
opportunity to request a copy of their child’s test score and to opt-out of providing their child’s
test score to their child’s teacher.
All teacher and student data collected will be kept confidential and reported in aggregate
form only.
12. Estimates of Hour Burden
Exhibit 3 provides an estimate of time burden. The total reporting burden for the data
collection effort covered by this clearance request is 10,397 hours. The total estimated annual
direct hour burden estimate for the data collection activities discussed under the first OMB
submission for this study is 1,185 hours. Adding the 10,397 burden hours from this clearance
request will increase the total estimated direct annual burden for the study to 11,582 hours.
The district and/or the school may take up to 8 hours completing our data request. We
assume that the study will include 20 districts.
The teacher contact and consent forms together will take 5 minutes to complete.
Based on past experience, a 95 percent response rate is expected for these forms.
Hence, we expect that 285 (95 percent of 300) teachers will complete the forms.
We expect that we will be able to collect existing Praxis math subject test scores for
15 percent of the teachers, based on the proportion of study teachers that is employed
in states that require the test for certification. Of the remaining 85 percent of the
teachers, we expect that 90 percent of these teachers will take the teacher math
assessment. In total, we expect that we will administer the test to 230 (90 percent of
85 percent of 300) study teachers. Teachers will have up to 2 hours to complete the
assessment.
The teacher survey is 30 minutes long and a 90 percent response rate is anticipated.
The majority of the burden hours are for administering the high school student test in
spring 2010. We assume that 8,100 (90 percent of 9,000) high school students in the
study will participate in the assessment. A total of 60 minutes has been allotted per
test administration, which includes the time taken for the provision of assent.
22

The interviews with the HSAC program directors are expected to last 45 minutes. We
expect interviews will occur with 24 program directors.
EXHIBIT 3
BURDEN IN HOURS TO RESPONDENTS

Activities

Number of
Respondents

Number of
Responses
Per
Respondent

Total
Number of
Responses

Average
Burden
Hours Per
Response

Total Burden
Hours

Burden for Initial Data Collection
Activitiesa

1,185

Burden for Added Data Collection
Activities

10,397

Districts
Student records data
Teachers
Teacher contact and consent forms
Teacher math assessment
Teacher survey
Students/Parents
Parental consent forms
Student math assessment
(including assent)
HSAC Program Directors
Interviews

20

1

20

8.0

160

285
230
270

1
1
1

285
230
270

0.08
2.00
0.50

24
460
135

18,000
8,100

1
1

18,000
8,100

0.08
1.00

1,500
8,100

24

1

24

0.75

18

Total Direct Burden

11,582

a

Includes a teacher background form, a pilot of the student assessment, and the random assignment of students. These data
collection activities are described in the previous OMB package approved on 6/7/09.

13. Estimate of Total Annual Cost Burden to Respondents or Recordkeepers
There are no direct costs to individual participants. Exhibit 4 presents estimates of indirect
costs to the respondents. The only indirect cost to respondents is the cost of their time, which
totals $96,778.

23

EXHIBIT 4
COST BURDEN TO RESPONDENTS

Activities
Districts
Student records data
Teachers
Teacher contact and
consent forms
Teacher math assessment
Teacher survey
Students/Parents
Parental consent forms
Student math assessment
(including assent)
HSAC Program Directors
Interviews

Number of
Respondents

Number of
Responses
Per
Respondent

Average
Burden
Hours Per
Response

20

1

8.0

$50

$8,000

285

1

0.08

$34

$808

230
270

1
1

2.00
0.50

$34
$34

$15,640
$4,590

18,000
8,100

1
1

0.08
1.00

$7
$7

$10,500
$56,700

24

1

0.75

$30

Total Cost Burden

Average
Hourly
Cost per
Response

Total Cost
Burden

$540
$96,778

14. Estimates of Annualized Cost to the Federal Government
The estimated cost to the federal government for the study—including recruiting districts
and schools, designing and administering all data collection instruments, processing and
analyzing the data, and preparing reports—is $8,087,800. Recruiting, data collection, and
reporting activities will be carried out over approximately four years (fall 2008 to summer 2012).
Thus, the average annual cost of data collection and analysis is $2,021,950.
15. Reasons for Program Changes or Adjustments
Due to the added data collection activities, there will be a program change of 10,397 hours,
increasing the total burden for the study to 11,582 hours. These data collection activities include
a teacher contact form, a high school student math assessment, a test of teacher math content
knowledge, a teacher consent form, a teacher survey, collection of student records data, parent
consent forms for the student math assessment and student records data collection, student assent
form, and interviews with HSAC program directors.

24

16. Tabulation, Publication Plans, and Time Schedules
We discuss plans for analysis and publication, and the timeline for the study in the following
subsections.
a.

Tabulation Plans

To estimate the impact of HSAC teachers on secondary student math achievement for the
full evaluation, we will treat each classroom match as a separate ―mini-experiment.‖ For each
classroom match, we will compare the average end-of-year math assessment score of students
randomly assigned to the class taught by the HSAC teacher to the average score of those
assigned to the non-HSAC teacher—the difference in average scores will provide an estimate of
the HSAC teacher’s impact in that particular classroom match. We will then average the impact
estimates across all classroom matches in the study to come up with an overall estimate of the
HSAC teachers’ impact on secondary student math achievement.
Primary Impact Analysis. Due to random assignment, the differences in mean outcomes in
each classroom match will provide an unbiased estimate of the impact of HSAC teachers.
However, the precision of the estimates can be improved by controlling for student-level baseline
characteristics that may explain some of the differences in achievement, such as sex, race,
free/reduced price lunch eligibility, special education status, whether the student is an English
language learner, and prior math achievement. We will therefore estimate the following model of
student math achievement for student i in classroom match j:
(1) Yij

Pj

X ij

Tij

i

where Yij is the outcome math test score of student i in classroom match j, Pj is a vector of
classroom match indicators, Xij is a vector of student-level baseline characteristics, Tij is an
indicator for whether the student was in the HSAC teacher’s class in classroom match j, εi is a
random-error term that represents the influence of unobserved factors on the outcome, and β and
δ are vectors of parameters to be estimated. Because the randomization is done within classroom
matches within schools, and schools may differ from each other in student compositions, the
model includes a vector of classroom match indicators, Pj, to control for differences in the
average student characteristics between classroom matches and schools. If a sufficient number of
classroom matches contain three teachers instead of two, the estimated standard errors will
account for clustering of students within classroom.
The vector δ represents the experiment-level impacts of the HSAC teachers in each
classroom match that can then be aggregated to estimate the overall HSAC impact. The simplest
and perhaps most intuitively appealing way to aggregate these impacts is to calculate an equally
weighted average of the classroom match-level impacts. In this way, each classroom match will
have an equal influence on the overall impact estimate. As a specification check, we will also
explore alternative weighting schemes that have the potential to provide greater statistical
efficiency and test the robustness of the findings, including giving greater weight to more
precisely estimated classroom match-level impacts and weighting proportionally to the size of
the sample in each classroom match.

25

Subgroup Analyses. In addition to estimating the overall impact of HSAC teachers on
secondary student math achievement, we will conduct a limited number of subgroup analyses.
Specifically, we will separately estimate the impact of TFA and TNTP teachers, middle and high
school HSAC teachers, and novice and experienced HSAC teachers. To calculate subgroup
impacts, the classroom match-level impact estimates will be aggregated for each relevant
subgroup. For example, to calculate the subgroup impacts for high school and middle school
teachers, the impact estimates from experiments in high schools will be aggregated separately
from those from the experiments in middle schools. While we will test the statistical significance
of the impact for each subgroup, we will not test the significance of differences between
subgroups (for instance, between TFA and TNTP teachers), as the sample will not provide
adequate statistical power for these comparisons.
Non-Experimental Analysis. If we find that HSAC teachers are more effective than nonHSAC teachers, policymakers will want to understand the reasons they are more effective. To
shed light on this, we will investigate whether there are particular observable teacher
characteristics that are correlated with the impacts. Because the effects of the teacher
characteristics cannot be separated from the HSAC recruiting model experimentally, we will rely
on non-experimental methods for this exploratory analysis.
For the non-experimental analysis, we will estimate variations of Equation 1 that introduce
within-experiment differences in teacher characteristics:
(2) Yij

Pj

X ij

Tij

Cij

i

where Cij represents a vector of observable characteristics of student i's teacher, γ is a vector of
parameters to be estimated, and all other variables are defined as above. Since these models
include classroom match-level fixed effects, the coefficients in vector γ represent the correlations
between the within-match differences in teacher characteristics and the within-match differences
in student outcomes. These exploratory analyses will be guided in large part by differences
between HSAC and non-HSAC teachers that are observed through the teacher survey and that
have been hypothesized to influence student achievement. For example, HSAC teachers are often
perceived to be different from non-HSAC teachers in their subject knowledge, the selectivity of
their undergraduate colleges, and their experience, all of which have been connected to student
achievement in prior research (Clotfelter et al. 2007). Therefore, using data from the teacher
survey and teacher math knowledge assessments (if the option is exercised), we will examine
how the differences between the HSAC teachers and the non-HSAC teachers along these
dimensions are correlated with student outcomes.
Non-Response and Crossovers. Although, we will take steps to minimize the amount of
missing data, some student non-response for this evaluation is inevitable. This non-response may
lead to biased impact estimates if the non-response is correlated with math achievement and
whether the student was assigned to an HSAC teacher. To address this, we will use propensity
score matching and create non-response weights that appropriately weight those for whom we
have outcome math test scores, so that the weighted sample of students with nonmissing data is
representative of the full sample. In addition, some students who are assigned to an HSAC
teacher may crossover into a class with a non-HSAC teacher or vice versa. Including crossover
26

students might bias the impact estimates by attributing the performance of the HSAC teacher to a
non-HSAC teacher and vice versa. We can adjust the estimates for these crossovers using the
students’ assignment status as an instrumental variable for having an HSAC teacher (Angrist et
al. 1996).
b. Publication Plans
The draft of the final report will be prepared during the third year of the study—by June
2011. The report will be written in a style and format accessible to policymakers and researchsavvy practitioners. A revised draft will be prepared by August 2011. The final report, which will
address peer-review comments, will be prepared by April 2012.
c.

Schedule
The timeline for the evaluation is shown in Exhibit 5.

17. Approval Not to Display the Expiration Date for OMB Approval
Approval not to display the expiration date for OMB approval is not requested.
18. Exception to the Certification Statement
No exceptions to the certification statement are requested or required.

27

EXHIBIT 5
TIMELINE FOR THE STUDY
Activity

Time Period

Recruit districts and schools
Administer teacher background form
Pilot of student assessment
Conduct random assignment
Collect teacher contact form and teacher consent form
Conduct teacher math assessment
Collect consent forms for testing
Administer teacher survey
Administer high school math assessment with student assent
Conduct HSAC program administrator interviews
Collect school records data
Draft report
Final report

Fall 2008-spring 2009
Spring 2009-Spring 2010
Spring 2009
Spring 2009 – fall 2009
Fall 2009
Fall 2009
Fall 2009
Spring 2010
Spring 2010
Spring 2010
Summer 2010
June 2011
April 2012

ED = U.S. Department of Education
HSAC = highly selective routes to alternative certification.

28

REFERENCES

Angrist, Joshua D., Guido W. Imbens, and Donald R. Rubin. ―Identification of Causal Effects
Using Instrumental Variables.‖ Journal of the American Statistical Association, vol. 91,
1996, pp. 444-472.
Boyd, Donald, Pamela Grossman, Hamilton Lankford, Susanna Loeb, and James Wyckoff.
―How Changes in Entry Requirements Alter the Teacher Workforce and Affect Student
Achievement.‖ Education Finance and Policy, vol. 1, no. 2, Spring 2006, pp. 178-216.
Clark, Melissa, Sheena McConnell, Kristen Hallgren, Daniel Player, and Alison Wellington.
―Evaluating Highly Selective Programs That Provide Alternative Routes to Teacher
Certification: Feasibility and Design Issues.‖ Princeton, NJ: Mathematica Policy Research,
Inc., March 28, 2008.
Clotfelter, Charles T., Helen F. Ladd, and Jacob Vigdor. ―Teacher Credentials and Student
Achievement in High School: A Cross-Subject Analysis with Student Fixed Effects.‖
National Bureau of Economic Research Working Paper no. 13617, November 2007.
Decker, Paul T., Daniel P. Mayer, and Steven Glazerman. ―The Effect of Teach for America on
Students: Findings from a National Evaluation.‖ Princeton, NJ: Mathematica Policy
Research, Inc., 2004.
Deke, John, and Joshua Haimson. ―Valuing Student Competencies: Which Ones Predict
Postsecondary Educational Attainment and Earnings and for Whom?‖ Princeton, NJ:
Mathematica Policy Research, Inc., 2006.
Feistritzer, Emily, and David Chester. ―Alternative Teacher Certification: A State-by-State
Analysis 2002.‖ Washington, DC: National Center for Education Information, 2002.
Ingersoll, Richard M. ―Is There Really a Teacher Shortage?‖ Philadelphia: Consortium for
Policy Research in Education, University of Pennsylvania, 2003.
Kane, Thomas J., Jonah E. Rockoff, and Douglas O. Staiger. ―What Does Certification Tell Us
About Teacher Effectiveness? Evidence from New York City.‖ National Bureau of
Economic Research Working Paper No. 12155, Washington, DC: National Bureau of
Economic Research, April 2006.
Lazear, Edward P. ―Teacher Incentives.‖ Swedish Economic Policy Review, vol. 10, no. 3, 2003,
pp. 179–214.
Murnane, Richard J., John B. Willett, and Frank Levy. ―The Growing Importance of Cognitive
Skills in Wage Determination.‖ Review of Economics and Statistics, vol. 77, no. 2, May
1995, pp. 251–266.

29

Murnane, Richard J., John B. Willett, M. Jay Braatz, and Yves Duhaldeborde. ―Do Different
Dimensions of Male High School Students’ Skills Predict Labor Market Success a Decade
Later? Evidence from the NLSY.‖ Economics of Education Review, vol. 20, no. 4, August
2001, pp. 311–320.
Northwest Evaluation Association. ―Technical Manual for the NWEA Measures of Academic
Progress and Achievement Level Tests.‖ Portland, Oregon: Northwest Evaluation
Association, 2003.
Rock, Donald A., and Judith Pollack. "Early Childhood Longitudinal Study - Kindergarten Class
of 1998-99 (ECLS-K): Psychometric Report for Kindergarten Through First Grade."
Washington, DC: National Center for Education Statistics, August 2002.
Thompson, Teresa. ―A Comparison of Methods of Increasing Parental Consent Rates in Social
Research.‖ The Public Opinion Quarterly, vol. 48, no. 4, 1984.
U.S. Department of Education. ―Foundations for Success: Report of the Task Group on Teachers
and Teacher Education.‖ Washington, DC: U.S. Department of Education, 2008a.
U.S. Department of Education. ―Foundations for Success: The Final Report of the National
Mathematics Advisory Panel.‖ Washington, DC: U.S. Department of Education, 2008a
Xu, Zeyu, Jane Hannaway, and Colin Taylor. ―Making a Difference? The Effects of Teach For
America in High School.‖ Washington, DC: Urban Institute, March 2008.

30