OMB section A_response_July 30 2010

National Title I Study of
Implementation and
Outcomes: Early Childhood
Language Development
Section A
July 30, 2010

Contract Number:
ED-04-CO-0112/0011

National Title I Study of

Mathematica Reference Number:
06692.602

Outcomes: Early Childhood

Submitted to:
National Center for Education Evaluation
and Regional Assistance
555 New Jersey Ave., Capital Place, NW
Washington, DC 20208
Project Officer: Tracy Rimdzius
Contract Officer: Brenda Jefferson
Submitted by:
Mathematica Policy Research
600 Maryland Avenue, S.W.
Suite 550
Washington, DC 20024-2512
Telephone: (202) 484-9220
Facsimile: (202) 863-1763
Project Director: Christine Ross

Implementation and
Language Development
Section A
July 30, 2010

CONTENTS
TITLE I RECRUITMENT OMB PACKAGE: SECTION A................................................... 1
A. Justification ............................................................................................. 1
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.

Circumstances Necessitating the Collection of the Information ........ 1
Purposes and Uses of the Data......................................................... 8
Use of Technology to Reduce Burden ............................................... 9
Efforts to Avoid Duplication ........................................................... 10
Methods to Minimize Burden on Small Entities ............................... 11
Consequences of Not Collecting Data ............................................ 11
Special Circumstances ................................................................... 12
Federal Register Announcement and Consultation ......................... 12
Payments of Gifts........................................................................... 13
Assurances of Confidentiality......................................................... 13
Additional Justification for Sensitive Questions .............................. 14
Estimates of Hours Burden ............................................................. 15
Estimates of Cost Burden to Respondents ...................................... 15
Estimates for Annual Costs to the Federal Government .................. 16
Reasons for Program Changes or Adjustments ............................... 16
Plans for Tabulation and Publication of Results .............................. 16
Approval to Not Display the OMB Expiration Date .......................... 22
Explanation of Exceptions .............................................................. 22

REFERENCES......................................................................................................... 23
APPENDICES
APPENDIX A: Legislation Authorizing the Study
APPENDIX B: Letter to Accompany School Data Form
APPENDIX C: School Data Form
APPENDIX D: Confidentiality Pledge
APPENDIX E: Initial Letter to Districts and Notification Packet Materials
APPENDIX F: School Sample Notification Letter to Districts
APPENDIX G: Letter to Schools and Notification Packet Materials

iii

TABLES

1

Timing of Data Collection for Title I ECLD ............................................... 7

2

Research Questions and Data Collection Methods ................................... 8

3

Expert Panel Members .......................................................................... 12

4

Estimate of Hours Burden ..................................................................... 15

5

Estimated Cost of Study Components ................................................... 16

6

Association between Factor Scores and Student’s Language
Development, Grade X .......................................................................... 19

iv

Mathematica Policy Research

ED-04-CO-0112/0011

TITLE I RECRUITMENT OMB PACKAGE: SECTION A
This Office of Management and Budget (OMB) package contains two main sections: Section A,
“Justification,” and Section B, “Collection of Information Employing Statistical Methods.” OMB
clearance for this project will be sought in two stages. This initial package seeks clearance for the
School Data Form (discussed in Section A and attached to this clearance package in Appendix C),
our approach for collecting the information necessary to identify the sample, as well as the study
sampling plan (described in Section B). A second OMB clearance package will be prepared and
submitted in December 2010 to obtain clearance for the full set of data collection instruments for
the study (summarized in Section A). The study is submitting the package in two stages because
sample identification and recruitment activities must begin before the data collection instruments are
developed.
A. Justification
1.

Circumstances Necessitating the Collection of the Information
Learning to read is a fundamental task of the early grades in school. Research has documented

the variety of skills and knowledge children need to acquire as they build toward reading
comprehension achievement. Language development, background knowledge (basic knowledge of
the social, cultural, scientific, and mathematical world), letter recognition and phonological
awareness all provide a foundation for skilled reading (National Early Literacy Panel 2008; Snow,
Burns, and Griffin 1998; Whitehurst and Lonigan 1998). Language and background knowledge
develop from birth as the child interacts with others, though children are exposed to different
amounts of language and background knowledge in the home and early care environments (Hart and
Risley 1995). Through instruction, children learn to decode, then to read with fluency and growing
comprehension. By the end of third grade, most children are making the transition to skilled, fluent

1

Mathematica Policy Research

ED-04-CO-0112/0011

reading that enables them to learn; by fourth grade, students are expected to be able to read to
abstract meaning from expository texts in core subjects such as social studies and science.
The 2007 National Assessment of Educational Progress (NAEP) found that 33 percent of
fourth-grade students had not achieved a basic level of proficiency in reading (U.S. Department of
Education 2009). Children who fail to learn to read by third or fourth grade are at high risk for
school dropout, with its negative implications for employment, income, and civic participation over
their lifetimes (U.S. Census Bureau 2005). Furthermore, research has identified subgroups of
children at high risk of having difficulty learning to read, including children from low-income
families and dual-language learners (DLLs) (Rueda and Windmueller 2006; National Research
Council 1998; National Research Council and Institute of Medicine 2000).
Since the mid-1990s, efforts to improve reading/language instruction in schools and preschools
serving high proportions of children at risk for reading difficulties have centered on the use of
scientifically-based reading/language instruction. The Classroom Literacy Interventions and
Outcomes in Even Start (CLIO) evaluation of literacy-focused curricula for disadvantaged children
in Even Start found no significant impacts on children’s language and comprehension outcomes
relative to the prevailing curricula used by Even Start programs (Judkins, et al. 2008). Additional
studies of similar efforts show some positive effects on letter recognition and decoding skills, fewer
effects on language development, and no effects on reading comprehension (Gamse et al. 2008a and
b; Jackson et al. 2007; Preschool Curriculum Evaluation Research [PCER] Consortium 2008). While
the impacts on letter recognition and decoding may seem encouraging, decoding skill does not itself
lead to reading comprehension (Snow et al. 1998; National Institute of Child Health and Human
Development 2000; National Early Literacy Panel 2008). For that, language development is critical,
and few of the curricula and teaching strategies tested over the past decade have had a positive effect
on language development.

2

Mathematica Policy Research

ED-04-CO-0112/0011

Reading experts also note that background knowledge improves the ability to comprehend text
(Snow et al. 1998). General knowledge measured in the fall of kindergarten was more strongly
associated with reading achievement in grade three than in grade one, which suggests a relationship
between reading comprehension and background knowledge (Duncan et al. 2007).
The lack of instructional strategies known to boost language development is important because
many children from low-income or dual-language homes arrive at preschool and kindergarten with
language and literacy scores well below the average four- or five-year-old (Tarullo et al. 2008;
Jackson et al. 2007; Chernoff et al. 2007). Even as students are entering kindergarten, differences in
background knowledge exist. These differences are related to various family characteristics, including
mother’s education, language spoken at home, and home activities such as reading (West et al. 2000).
Therefore, closing this gap in language development and background knowledge is critical if
disadvantaged children are to attain adequate comprehension skills and succeed in upper elementary
school and beyond.
To identify programs and practices associated with better language development, background
knowledge, and comprehension outcomes for children in prekindergarten through third grade, the
U.S. Department of Education (ED) has requested a national study of Title I schools that focuses
on the prekindergarten through third-grade levels and examines the development of language,
background knowledge, and comprehension in schools with high and low reading achievement
scores. The study is being conducted as a component of the National Assessment of Title I,
mandated by Title I, Part E, Section 1501 of the Elementary and Secondary Education Act (see
Appendix A).
Overview of the Study. In October 2009, ED began working with Mathematica Policy
Research, in partnership with Decision Information Resources, Inc. (DIR) and Dr. Timothy
Shanahan of the University of Illinois-Chicago, on a national study of 100 Title I schools to identify

3

Mathematica Policy Research

ED-04-CO-0112/0011

school programs and teacher instructional practices associated with improved language
development, background knowledge, and comprehension outcomes for children in prekindergarten
through third grade. Schools in the study will be divided between those with high and those with
low average reading achievement scores. The study will include up to five grade cohorts
(prekindergarten, kindergarten, and first through third grades) and classrooms and student samples
will be selected for each. ED will make a final decision on the grades to be included based on the
availability of funds and considering the input of the study’s expert panel (first meeting to be held in
early April, 2010).
Data for the full study will be collected on each grade cohort using a battery of student
assessments, classroom observations, parent interviews, teacher and administrator questionnaires,
and student record reviews. Analyses will estimate which school programs and teacher instructional
practices are associated with growth in student outcomes to inform future rigorous evaluations of
strategies to improve language and reading comprehension outcomes for at-risk children.
The study will seek to answer the following questions about the growth of children’s
achievement from preschool through grade three and its association with school programs and
instructional practices:
1. How do language development, background knowledge, and comprehension develop
across preschool through grade three?
2. What school programs are being used in the sample of schools, and what teacher
instructional practices are observed to support children’s language development,
background knowledge, and comprehension?
3. What school programs are associated with greater student progress in language
development, background knowledge, and comprehension?
4. What instructional practices are associated with greater student progress in language
development, background knowledge, and comprehension?
5. How does the quality of the home literacy environment interact with teacher
instructional practices in predicting children’s progress over the school year in language
development, background knowledge, and comprehension?

4

Mathematica Policy Research

ED-04-CO-0112/0011

In addition, the study will address the following questions about the methodology of identifying
high- and low-performing schools and measuring teaching practices:
6. Can we accurately identify high- and low-performing schools using readily-available
school-level performance data and demographic information? Do schools tend to have
consistently high- or low-performance across grades and across classrooms? Are third
grade assessment measures (typically the first year states collect standardized results)
indicative of cumulative school effects in earlier grades?
7. How can researchers measure instructional practices more reliably?
Study Timeline. The study began in October 2009 and is a five-year project. The first year
(October 2009 to September 2010) involves several activities that set up the study, including
selecting and convening an expert panel, identifying student assessments and classroom observation
instruments to be used, drafting additional data collection forms, and finalizing the study’s design.
Preparation for identifying and recruiting the school districts that will be asked to participate will
also begin toward the end of the first year. The second year (October 2010 to September 2011)
focuses on finalizing data collection forms, sampling and recruiting schools to participate in the
study, and developing and finalizing data collection training plans. Data collection begins toward the
end of year two and continues into the third year (October 2011 to September 2012). Year four
(October 2012 to September 2013) focuses on cleaning the data files and preparing them for
analysis, developing analytic variables, refining the analysis plan, analyzing the data, and preparing
for the report. The final year of the study (October 2013 to September 2014) will be centered on
finalizing the report and preparing the data file for use by other members of the research
community.
Study Sample. The study sample will be comprised of 100 Title I schools: 50 with consistently
high reading achievement scores and 50 with consistently low reading achievement scores. Within
each school, we will randomly sample 3 classrooms per grade (prekindergarten through grade three),
for a total of 1,500 classrooms. Within each classroom, we will randomly sample 7 students for a

5

Mathematica Policy Research

ED-04-CO-0112/0011

total of 10,500 students (due to student mobility, we expect the spring student sample will decrease
to 7,500).
Data Collection Plan. As part of recruitment, we will use the School Data Form (see
Appendix C) to collect data from districts on selected schools. Items include whether the school is a
Title I school, number of students enrolled in prekindergarten through third grade, percent of
students eligible for free or reduced-price lunch, percent of third graders classified as reading
proficient based on state assessments, and recent or expected substantial changes in school
administration or student demographics. These data will confirm that the information collected
from previously existing data sources on the schools is accurate, and that the selected schools meet
the study’s eligibility criteria (see Section B for a description of eligibility criteria and the school
selection process). At the time schools are selected, the most recent data on school characteristics
will reflect the 2007-2008 school year. Therefore, we will ask a small set of questions (e.g.,
enrollments by grade, percent of students eligible for free or reduced-price lunch) in order to update
the information that is available through the Common Core of Data. We will also ask district staff to
indicate whether each of the selected schools is a Title I school and that it has students enrolled in
each of prekindergarten through third grade in order to confirm that the school meets the study’s
eligibility criteria. It is important that we know if a school has changed substantially (e.g., new
principal, undergoing restructuring, substantial increase in enrollment) from what was known about
the school when the samples of low performing and high performing schools were selected, because
changes of this type could have an impact on school programs, teaching practices, and student
achievement. Additional items on the School Data Form include whether the prekindergarten and
kindergarten classes are half-day, a full day, or both, and the percent of students classified as English
language learners. The study should include schools that vary along these dimensions, so this
information will be taken into account in sampling schools.

6

Mathematica Policy Research

ED-04-CO-0112/0011

The full-scale study includes several complementary data collection efforts that will support
answers to the study’s research questions. Table 1 lists the timing of the different data collections,
and a brief description of each is provided below:
Table 1. Timing of Data Collection for Title I ECLD
Data Collection Activity
Principal survey

Fall 2011

Spring 2012

X

Teacher survey

X

Parent interview

X

Teacher-student report

X

School records

X

Student assessments

X

X

Classroom observations

X

X

Principal survey. Hard copy surveys will be administered to 100 principals in fall 2011.
Questions will address instructional reading programs and practices used from
prekindergarten through grade three; curricula used in prekindergarten and grade three,
particularly those to support language development and early reading; the extent to
which curriculum and instructional programs are coordinated in prekindergarten
program/classes and in the kindergarten through grade three classes in the school;
supports for struggling readers; and professional development (related to reading and
general teaching practices) available to teachers.
Teacher survey. Web-based surveys will be completed by 1,500 teachers in spring 2012.
Items will focus on teacher background, credentials, professional development, reading
programs used, books/readers used in the classroom, reading instructional activities and
teaching strategies, support for struggling readers and dual language learners (DLLs), and
general behavior of children in the classroom, such as disruption and cooperation.
Parent interview. Telephone surveys will be conducted with 7,500 parents in spring
2012. Items will address family resources and risk factors, including parent education,
employment status, income level, marital status, race/ethnicity, and language spoken in
the home; home literacy environment, including reading to the child and availability of
literacy materials; and parental and family involvement with students’ education,
including help with homework and providing children with out-of-home enrichment
activities.
Teacher-student report. The study will use a web-based report to collect student-level
data from teachers on individual child engagement/attention, instructional grouping for
reading, special placement and receipt of services, support for reading, and disruptive
behavior. These data will be collected in spring 2012. A total of 7,500 teacher-student
reports will be completed.

7

Mathematica Policy Research

ED-04-CO-0112/0011

School records. The study will collect school records data for all children in the study in
spring 2012. The data will be collected electronically and will include the date each child
began at the school, receipt of special education services, grade repetition, standardized
test scores, and attendance.
Student assessments. The study will assess the language development, background
knowledge, and comprehension of 7,500 students in the 1,500 classrooms participating
in the study. A computer-assisted, one-on-one assessment will be administered to all
prekindergarten through third grade students in fall 2011 and again in spring 2012 to
measure these outcomes. An additional group-level assessment will be used to assess
reading comprehension for second and third grades in fall 2011 and spring 2012.
Classroom observations. Each of the 1,500 classrooms in the study will be observed
twice during the 2011-2012 school year: once in fall 2011 and again in spring 2012.
These measures will assess the emotional supportiveness or positive climate of the
classroom, teacher language modeling and support for learning, and approaches to
supporting children’s language development, comprehension of oral and written
information (i.e., listening and reading comprehension, respectively), and expansion of
background knowledge.
2.

Purposes and Uses of the Data
Table 2 lists the study’s research questions and the data collection that will support answers to

each question.
Table 2. Research Questions and Data Collection Methods
Research Question

Data Collection Method

1. How do language development, background
knowledge, and comprehension develop across
preschool through grade three?

Student assessments

2. What school programs are being used in the sample
of schools, and what teacher instructional practices
are observed to support children’s language
development, background knowledge, and
comprehension?

Principal survey
Classroom observations
Teacher survey

3. What school programs are associated with greater
student progress in language development,
background knowledge, and comprehension?

Principal survey
Student assessments

4. What instructional practices are associated with
greater student progress in language development,
background knowledge, and comprehension?

Classroom observations
Teacher survey
Student assessments

8

Mathematica Policy Research

ED-04-CO-0112/0011

Research Question

Data Collection Method

5. How does the quality of the home literacy
environment interact with teacher instructional
practices in predicting children’s progress over the
school year in language development, background
knowledge, and comprehension?

Parent interview
Teacher survey
Classroom observations
Student assessments

6. Can we accurately identify high- and low-performing
schools using readily-available school-level
performance data and demographic information? Do
schools tend to have consistently high- or lowperformance across grades and across classrooms?
Are third grade assessment measures (typically the
first year states collect standardized results)
indicative of cumulative school effects in earlier
grades?

Student assessments
Classroom observations

7. How can researchers measure teaching practices
more reliably?

Classroom observations

Note:

Not included in the table are the Teacher-Student Report and School Records. These
instruments will collect data on items that will be used as covariates in the analysis.

The data collected will identify promising programs and practices for student reading outcomes.
Future studies could focus on evaluating the impacts of these practices on language and
comprehension outcomes for at-risk children in the early years of school. In addition, the study will
provide important information about how to (1) accurately identify high- and low-performing
schools and (2) measure instructional practices reliably.
3.

Use of Technology to Reduce Burden
The data collection plan was designed to obtain reliable information efficiently while

minimizing the burden on respondents. Consistent with that goal, information will be gathered from
existing data sources, where feasible. To reduce the burden on school districts and school
administrators, the Common Core of Data (CCD) will be a primary source of data on the school
characteristics used when designing and selecting the school sample, and that will be used to
describe the characteristics of the sample or as covariates in the analyses. The School Data Form
(Appendix C) is limited to a small set of questions that are required to update the information on

9

Mathematica Policy Research

ED-04-CO-0112/0011

schools before the final sample is finalized. Data on these characteristics is either not available
through the CCD or other sources or will not be available for the 2010-2011 school year at the time
the sample is finalized.
Additional existing data sources will include students’ school records and scores for schooladministered tests. This information will be obtained in the form of computer files, if a school
prefers this method. If it is too burdensome or not possible for a school to provide this information
as a computer file, schools will be asked to provide copies of the relevant information, which will be
coded by the study team.
The teacher survey and teacher-student report are both web-based data collections, and the
school records will be collected electronically to reduce burden on teachers and school staff. The
parent interview is a computer-assisted telephone interview (CATI). The use of web-based and
CATI data collection instruments reduces respondent burden by facilitating routing and skip
patterns. The principal survey is hard copy; however, with only 100 principals participating in the
study, the cost of developing a CATI or web-based survey outweighs the benefits.
All of the individual student assessments will be conducted using the computer-assisted
personal interview (CAPI). This approach has many advantages, including marginally reducing the
length of the assessment since the assessor does not have to interrupt the flow of the assessment to
calculate stopping points. Assessors can move more quickly through the assessment because
complicated rules about which item or set of items comes next are controlled by the instrument
software. Both of these features reduce burden and errors, and improve the quality of the data and
the accuracy of the child’s scores.
4.

Efforts to Avoid Duplication
No equivalent sources of data exist for the study. Several studies collect data on classroom

practices and children’s language development, background knowledge, and comprehension (for

10

Mathematica Policy Research

ED-04-CO-0112/0011

example, the Head Start Family and Child Experiences Survey (FACES), the Early Childhood
Longitudinal Study—Kindergarten Class of 1998-99 (ECLS-K), and the Early Childhood
Longitudinal Study—Birth Cohort (ECLS-B), but they are limited in several ways. The FACES
study collects information through observations of general classroom practices, but does not focus
specifically on practices targeted at reading. ECLS-K also gathers information on teacher practices,
but the data are collected through a teacher self-report and not at the level required for this study.
ECLS-B does not fully assess background knowledge that would support later reading
comprehension.
5.

Methods to Minimize Burden on Small Entities
The primary entities for the study are school districts and schools. Burden is minimized for all

respondents by requesting only the minimum data required to meet the study’s objectives. The
burden on districts and schools will also be minimized through careful specification of information
needs, restriction of questions to generally available information, and the design of the data
collection strategy. The study’s expert panel will review all data requirements before the full OMB
package is submitted in December 2010.
6.

Consequences of Not Collecting Data
The data collection plan described in this submission is necessary for conducting ED’s National

Title I Study of Implementation and Outcomes: Early Childhood Language Development and,
consistent with the goal of Title I legislation, may help identify programs and practices to improve
language development and comprehension outcomes for at-risk children. The study represents an
important next step in developing a systematic and rigorous evaluation agenda in the areas of early
childhood and early reading.

11

Mathematica Policy Research

ED-04-CO-0112/0011

7.

Special Circumstances
There are no special circumstances associated with this data collection.

8.

Federal Register Announcement and Consultation

a.

Federal Register Announcement
A 60-day notice was published in the Federal Register with an end date of April 14, 2010. No

public comments were received.
b. Consultations Outside the Agency
The study team has contacted members of its Expert Panel for advice on various aspects of the
study design and data collection instruments. Their feedback was obtained through in-person
meetings and telephone conversations. Members of the Expert Panel for this study include:
Table 3. Expert Panel Members
Expert Panel Member

Organizational Affiliation

Thomas Cook

Professor of Sociology, Psychology, Education and Social Policy,
Northwestern University

David Dickinson

Professor of Education, Vanderbilt University

Barbara Foorman

Francis Eppes Professor of Education, Florida State University

Christopher Lonigan

Professor, Florida State University

Charles Perfetti

Distinguished University Professor of Psychology, University of
Pittsburgh

Ray Reutzel

Emma Eccles Jones Endowed Chair and Distinguished Professor of Early
Childhood Education, Utah State University

Don Rock

Senior Research Scientist, Educational Testing Service

Christopher Schatschneider

Associate Professor, Florida State University

Catherine Snow

Henry Lee Shattuck Professor of Education, Harvard University

Throughout the study, the team will consult with the panel on additional issues that would
benefit from their input, such as selection of assessments, evaluation methodology, and study design.

12

Mathematica Policy Research

ED-04-CO-0112/0011

9.

Payments of Gifts
We do not intend to provide a payment or gift to respondents to the School Data Form.

We realize, however, that participation in the full study will place demands on each of the
participants. Payments will be similar to those offered to respondents to comparable instruments in
other studies. A complete description of incentives will be included in the OMB package submitted
in December 2010.
10. Assurances of Confidentiality
None of the information collected will be reported or published in a manner that would identify
individual respondents. Responses to this data collection will be used only for statistical purposes.
The reports prepared for this study will summarize findings across the sample and will not associate
responses with a specific district, school, or individual.
Mathematica and its subcontractors follow the confidentiality and data protection requirements
of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183), which requires
“All collection, maintenance, use, and wise dissemination of data by the Institute” to “conform with
the requirements of section 552 of title 5, United States Code, the confidentiality standards of
subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20
U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and
Privacy Act, and the Protection of Pupil Rights Amendment. In addition for student information,
“The Director shall ensure that all individually identifiable information about students, their
academic achievements, their families, and information with respect to individual schools, shall
remain confidential in accordance with section 552a of title 5, United States Code, the confidentiality
standards of subsection (c) of this section, and sections 444 and 445 of the General Education
Provision Act.” Subsection (c) of section 183 referenced above requires the Director of IES to
“develop and enforce standards designed to protect the confidentiality of persons in the collection,

13

Mathematica Policy Research

ED-04-CO-0112/0011

reporting, and publication of data”. Subsection (d) of section 183 prohibits disclosure of individually
identifiable information, as well as making any publishing or communicating of individually
identifiable information by employees or staff a felony.
Every data collector will be required to sign a pledge to protect the privacy of respondent data
and to ensure the security of the assessment materials (see Appendix D for a copy of Mathematica’s
Confidentiality Pledge). The pledge indicates that any violation or unauthorized disclosure may result
in legal action or other sanctions by Mathematica, including the termination of employment. A
discussion of human subject protection will be a part of the interviewer/assessor training. A copy of
the signed pledges will be kept on file and will, upon request, be submitted to ED.
When reporting the results, data will be presented only in aggregate form, so that individuals
and institutions will not be identified. A statement to this effect will be included on the parental
consent forms. The teacher survey, principal survey, and parent survey will include a reminder about
privacy in compliance with the legislation (P.L. 93-579). When data are collected through in-person
or telephone interviews, respondents will be reminded about the privacy protections and their right
to refuse to answer questions.
11. Additional Justification for Sensitive Questions
The parent interview will include questions about household income, home language, family
composition and parent education, which some may view as sensitive items. We will use these data
and data from other parent interview questions as covariates in the analyses to adjust for factors
related to students’ language development, background knowledge, and reading comprehension or
to their self-selection into a preschool program or school. Parents will be reminded about the
privacy of the information and the voluntary nature of the interview prior to asking any potentially
sensitive questions.

14

Mathematica Policy Research

ED-04-CO-0112/0011

12. Estimates of Hours Burden
As part of the recruitment process, districts will be asked to complete a School Data Form (see
Appendix C). We are asking districts to verify the schools’ Title I status, provide the number of
students in prekindergarten through third grade and indicate whether there is more than one class
per grade, report the percent of students eligible for free or reduced price lunch, report the percent
of students classified as English language learners (ELL), report the percent of third graders
classified as reading proficient on state assessments in the most recent school year, and provide
information on any recent or expected substantial changes in school administration or student
demographics. We estimate it will take approximately 2.25 hours on average for district staff to fill
out the information for the schools in the district. The total estimated burden for all sixteen districts
is 36 hours (see Table 4).
Table 4. Estimate of Hours Burden
Instrument

Respondent

Number of
Respondents

Estimated Time
to Complete

Total Estimated
Burden

Districts

16

2.25 hours per
respondent

36 hours

School Data Form

Estimates of burden for participation in the full study will be provided in the OMB package
submitted in December 2010.
13. Estimates of Cost Burden to Respondents
There are no additional respondent costs associated with this data collection, other than the
time estimated as burden for district staff to provide the study team with information collected on
the School Data Form.

15

Mathematica Policy Research

ED-04-CO-0112/0011

14. Estimates for Annual Costs to the Federal Government
The estimated average annual cost of the study over the five years of the base contract is
$1,008,397, or a total estimated cost of $5,041,983 for the base contract of data collection for
prekindergarten students. The study also includes five options for additional data collection efforts.
These optional tasks include data collection for kindergarten, first, second, and third graders, as well
as an audiotaping option. The estimated cost of the optional tasks across the five years is $8,058,995.
The total estimated cost of the study (base contract plus options) is $13,100,978, an annual cost of
$2,620,196 across five years.
The estimated cost of the base plus contract options for components of the study is (Table 5):
Table 5. Estimated Cost of Study Components
Study Component

Estimated Cost

Study Design

$725,079

Expert Panel, Management

$455,827

Instrument Development/OMB Package

$718,987

Site Selection and Recruitment

$374,557

Data Collection

$10,249,103

Analysis and Reporting

$1,033,248

The estimated cost of the observational measures, including instrument development, observer
training, and data collection for this component, is $ 3,769,853.
15. Reasons for Program Changes or Adjustments
This is a new study.
16. Plans for Tabulation and Publication of Results
The analytic strategies will be aligned with the study’s research questions (Section A.1,
Overview of the Study). Specifically, the analyses are designed to: (1) describe how language

16

Mathematica Policy Research

ED-04-CO-0112/0011

development, background knowledge, and comprehension develop during the school year from
preschool through grade three (research question 1); (2) describe the school programs and classroom
practices used to support children’s language development, background knowledge, and
comprehension outcomes (research question 2); (3) analyze the relationships between school
programs and teacher practices and children’s progress in language development, background
knowledge, and comprehension (research questions 3 and 4); and (4) explore how the quality of the
home literacy environment interacts with teaching practices to predict children’s progress in
language development, background knowledge, and reading achievement (research question 5). In
addition, we will address methodological questions about (1) identifying high- and low-performing
schools based on readily-available data on school-level performance and student demographics
(research question 6); and (2) how to measure instructional practices more reliably (research
question 7).
Direct child assessments will provide data on children’s language development, background
knowledge, and comprehension at the beginning and end of the school year. Information on school
programs and teacher practices will draw on principal and teacher interviews and structured
observations of the classrooms. Parent interviews will provide information about the home literacy
environment and other family background information.
Analyses will employ a variety of methods, including cross-sectional and longitudinal
approaches, descriptive statistics (means, percentages), simple tests of differences across subgroups
and over time (t-tests, chi-square tests), and multivariate analysis (regression analysis, hierarchical
linear modeling). The first two research questions can be answered by calculating averages and
percentages of children, classrooms, or programs falling into various categories; comparisons of
these averages across subgroups; and changes in children’s outcomes over time. More complex
analyses of the relationships among school programs and teaching practices and children’s

17

Mathematica Policy Research

ED-04-CO-0112/0011

development that address questions three through five can be done through hierarchical linear
modeling (HLM).
For questions about the characteristics of teachers and children, the development of children’s
language, background knowledge, and comprehension over the school year, and the types of school
programs and teacher practices found in the sample of schools, we will calculate averages and
percentages. For example, we will calculate the average scores on the language development
assessment for preschool-age children in the fall and spring and the average gain score between fall
and spring. Similarly, we will calculate the percentage of schools using particular reading curricula in
preschool (for example, Opening the World of Learning). For all descriptive analyses, we will
calculate standard errors, taking into account multilevel sampling and clustering at the appropriate
level (school, classroom, and child). We will use analysis weights taking into account complex
multilevel sampling and nonresponse at each level.
Analyses of the relationships between school programs, teacher practices, and children’s
development will use a value-added HLM approach that links student achievement with practices
and programs in the study schools and classrooms, while properly accounting for the nested
structure of the data, prior student achievement, and other confounding factors. The study’s main
analytic models will be estimated by grade, using the scores obtained from fall and spring studyadministered assessments of language development, background knowledge, and listening or reading
comprehension. The models will include controls for the students’ pretest performance, teacher
demographics, and other potential confounds.
The main independent variables of interest are the observed teaching practices, such as the
quality and complexity of the teacher’s language use, minutes of instruction in components of
reading instruction, and mode of instruction. Because we will include a number of practice
measures, we will use factor analysis to reduce the number of variables to a smaller number of

18

Mathematica Policy Research

ED-04-CO-0112/0011

factors. The factor analysis will generate factor scores—that is, estimates of scores that would have
been received on each of the identified factors had they been estimated directly—for classrooms and
schools. These estimated factor scores are typically more reliable than the score of the individual
observed variables.
In addition, by reducing the number of variables to be estimated, we should be able to estimate
the relationship between teacher practices and student learning more precisely. We will assess the
predictive validity of the factor scores by examining the extent to which they are significantly
associated with differences in student learning across study schools and classrooms (having
controlled for confounding factors). Significant factors will signal which programs and practices best
help predict which schools and classes generate higher student achievement, and thus may deserve
further study.
Table 6 illustrates how the association between practices and students’ outcomes may be
displayed. The estimates and statistical significance of the factor scores, which represent the
associations of classroom and school programs and practices with student outcomes, are the focus
of this study.
Table 6. Association between Factor Scores and Student’s Language Development, Grade X
Student Language Development
Outcome 1
Coefficient (s.e.)

p-value

Student Language Development
Outcome 2
Coefficient (s.e.)

p-value

Classroom Practices
Factor Score 1
Factor Score 2
School Practices
Factor Score 3
Factor Score 4
Notes:

† p < 0.10, * p < 0.05, ** p , 0.01
In addition to the results included in this table, the models controlled for the
relevant pretest, student, classroom, and school covariates.

19

Mathematica Policy Research

ED-04-CO-0112/0011

The data from the study will also be used to answer methodological questions about how to
accurately identify high- and low-performing schools and how to improve the reliability of
instructional practice measures.
Education policymakers and practitioners seek to identify schools on the high and low ends of
performance for distinct purposes. High-performing schools have been identified to offer bestpractice models. Low-performing schools have been identified for significant intervention.
This study will identify consistently high- and low-performing schools for the sample using
publicly-available information on school proficiency levels on state-administered 3rd grade reading
assessments and the percentage of students eligible for free or reduced price lunch. School
performance will be measured by overall reading proficiency levels measured across a three-year
span and by the extent to which a school exceeds or fails to meet expectations of proficiency levels,
conditional on the students' level of economic disadvantage (that is, a residual measure from a
regression model).
Information we collect in the study will help examine how consistently low- or high-performing
these schools are. Using the student-level measures that are comparable across schools and more
sensitive than the state assessment measures used to identify high- and low-performing schools for
the study, we will measure how consistent each school's performance is, both across prekindergarten through grade 3 on average and within these grades. We will measure the distribution
of student growth associated with teachers within a school and with each grade level and analyze the
extent to which student growth in each grade can predict whether the school was identified as high
or low performing. We will analyze whether the teaching practices that are associated in the study's
analyses with greater student growth can predict whether the school is classified as high- or lowperforming.

20

Mathematica Policy Research

ED-04-CO-0112/0011

Past research has often shown a weak association between measures of teaching practices
collected through intensive observation and student outcomes. Reliability of classroom observation
measures, or the ability of the measure to provide stable and consistent judgments about the quality
of instruction, has been cited as a key issue that may explain the weak measured relationships
between teaching practices and student learning in previous studies.
The study team is developing an observation instrument (protocol for conducting observations
and rubrics for assessing the observations), training materials, and training procedures designed to
measure teaching practices more reliably than has been done in past studies. The rubric will have
more "anchors" explaining clearly how to score each dimension; will include more dimensions of
practice to distinguish different aspects of teaching; offer participatory training designed to increase
understanding and confidence in using the measure and making scoring judgments; and require high
levels of exact agreement for certification in coding videotaped classroom situations and actual
classrooms. Thus, the observation measure and training materials represent an important
contribution to the study of teacher quality and student learning. The study will discuss the
observational protocol and measure, and its reliability compared with other classroom measures. It
will analyze inter-rater reliability as well as the variation across observations for a single teacher
during the year. It will discuss how the measure, training materials, and training procedures were all
designed to improve reliability, highlighting differences from measures used in previous studies and
lessons learned in this study. Recommendations for improving the reliability of teaching practice
measures in future studies will be included.
Publication Plans. The study is currently scheduled to prepare one report summarizing the
analyses and findings. The report will present the descriptive findings on the extent of children’s
development during the school year and the school programs and teacher practices found across the
schools in the sample. The report will also present the multivariate analyses and findings on the

21

Mathematica Policy Research

ED-04-CO-0112/0011

association between teacher practices, school programs and the growth of children’s language
development, background knowledge, and comprehension outcomes, as well as the findings from
the methodological studies The draft of the final report is due to ED in September 2013.
17. Approval to Not Display the OMB Expiration Date
The study will display the OMB expiration date on all respondent materials and study
instruments.
18. Explanation of Exceptions
No exceptions of the certification statement are being sought.

22

Mathematica Policy Research

ED-04-CO-0112/0011

REFERENCES
Chernoff, Jacobson, K.D. Flanagan, C. McPhee, and J. Park. Preschool: First Findings From the Preschool
Follow-up of the Early Childhood Longitudinal Study, Birth Cohort (ECLS-B). (NCES 2008-025).
Washington, DC: U.S. Department of Education, Institute of Education Sciences, National
Center for Education Statistics, 2007.
Duncan, Greg, Chantelle J. Dowsett, Amy Claessens, Katherine Magnuson, Aletha C. Huston,
Pamela Klebanov, Linda Pagani, Leon Feinstein, Mimi Engel, Jeanne Brooks-Gunn, Holly
Sexton, Kathryn Duckworth, and Crista Japel. “School Readiness and Later Achievement.”
Developmental Psychology, vol. 43, no. 6, 2007, pp. 1428-1446.
Gamse, B.C., H.S. Bloom, J.J. Kemple, and R.T. Jacob. Reading First Impact Study: Interim Report.
(NCEE 2008-4016). Washington, DC: U.S. Department of Education, Institute of Education
Sciences, National Center for Education Evaluation and Regional Assistance, 2008a.
[http://ies.ed.gov/ncee/pubs/20084016/].
Gamse, B.C., R.T. Jacob, M. Horst, B. Boulay, and F. Unlu. Reading First Impact Study Final Report.
(NCEE 2009-4038). Washington, DC: U.S. Department of Education, Institute of Education
Sciences, National Center for Education Evaluation and Regional Assistance,
2008b.[http://ies.ed.gov/ncee/pubs/20094038].
Garet, Michael S., Stephanie Cronen, Marian Eaton, Anja Kurki, Meredith Ludwig, Wehmah Jones,
Kazuaki Uekawa, Audrey Falk, Howard S. Bloom, Fred Doolittle, Pei Zhu, and Laura
Sztejnberg. The Impact of Two Professional Development Interventions on Early Reading Instruction and
Achievement. (NCEE 2008-4030). Washington, DC: U.S. Department of Education, Institute of
Education Sciences, National Center for Education Evaluation and Regional Assistance, 2008.
Hart, B., and T. Risley. Meaningful Differences in the Everyday Experiences of Young American Children.
Baltimore, MD: Brookes Publishing, 1995.
Jackson, Russell, Ann McCoy, Carol Pistorino, Anna Wilkinson, John Burghardt, Melissa Clark,
Christine Ross, Peter Schochet, and Paul Swank. National Evaluation of Early Reading First: Final
Report. U.S. Department of Education, Institute of Education Sciences. Washington, DC: U.S.
Government Printing Office, 2007.
Judkins, David, Robert St. Pierre, Babette Gutmann, Barbara Goodson, Adrienne von Glatz,
Jennifer Hamilton, Ann Webber, Patricia Troppe, and Tracy Rimdzius. A Study of Classroom
Literacy Interventions and Outcomes in Even Start. (NCEE 2008-4028). Washington, DC: National
Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S.
Department of Education [http://ies.ed.gov/ncee/pubs/20084028].
National Early Literacy Panel. Developing Early Literacy: Report of the National Early Literacy Panel.
Washington, DC: National Institute for Literacy, 2008.
National Institute of Child Health and Human Development. Teaching Children to Read: An
Evidence-Based Assessment of the Scientific Research Literature on Reading and Its
Implications for Reading Instruction. 2000.

23

Mathematica Policy Research

ED-04-CO-0112/0011

National Research Council and Institute of Medicine. From Neurons to Neighborhoods: The
Science of Early Childhood Development. Committee on Integrating the Science of Early
Childhood Development. Jack Pl Shonkoff and Deborah A. Phillips (Eds.). Board on Children,
Youth and Families, Commission on Behavioral and Social Sciences and Education.
Washington, DC: National Academy Press. 2000.
National Research Council Committee on the Prevention of Reading Difficulties in Young Children.
Preventing Reading Difficulties in Young Children Committee on the Prevention of Reading
Difficulties in Young Children. Catherine E. Snow, M. Susan Burns, and Peg Griffin (Eds.)
Board on Children, Youth and Families, Commission on Behavioral and Social Sciences and
Education. Washington, DC: National Academy Press.1998.
Preschool Curriculum Evaluation Research (PCER) Consortium. Effects of Preschool Curriculum
Programs on School Readiness. Washington, DC: U.S. Department of Education, Institute of
Education Sciences, National Center for Education Research, 2008.
Rueda, R., and M. P. Windmueller. "English Language Learners, LD, and Overrepresentation: A
Multiple Level Analysis. Journal of Learning Disabilities, 39(2), 99-107.
Snow, Catherine E., M. Susan Burns, and Peg Griffin. Preventing Reading Difficulties in Young Children.
Washington, DC: The National Academies, 1998.
Tarullo, Louisa, Jerry West, Nikki Aikens, and Lara Hulsey “Beginning Head Start: Children,
Families, and Programs in Fall 2006.” Washington, DC: Mathematica Policy Research, 2008.
U.S. Department of Education. The Nation’s Report Card: Reading 2007. (NCES 2007-496).
[http://nces.ed.gov/nationsreportcard/pdf/main2007/2007496.pdf].
West, Jerry, Kristin Denton, and Elvira Germino-Hausken. America’s Kindergartners. Washington, DC:
U.S. Department of Education, 2000.
Whitehurst, G. J., and C. L. Lonigan. “Child Development and Emergent Literacy.” Child
Development, vol. 69, no. 3, 1998, pp. 848-872.

24

www.mathematica-mpr.com

Improving public well-being by conducting high-quality, objective research and surveys
Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC
Mathematica® is a registered trademark of Mathematica Policy Research