PART A
On April 19, 2006, the Office of Management and Budget (OMB) cleared the information collection request for the Evaluation of Mathematics Curricula—an evaluation sponsored by the U.S. Department of Education (ED). The evaluation is authorized under the No Child Left Behind Act of 2001, Section 1501 (PL No 107-110). The study is examining the relative effects of four curricula that represent the diverse approaches used to teach elementary school math in the United States. Experimental methods are being used to determine the relative effects of the curricula.
The existing OMB clearance (Number 1850-0813) expires on September 30, 2008 and this document requests a revision for an additional year of information collection through September 30, 2009. We begin with a summary of progress made on the study to date, and briefly describe research evidence that can be supported by the additional data collection. A more detailed justification for the information collection (Part A of the submission) follows. A separate document (Part B of the submission) provides more details about the analyses that can be conducted with the additional data collection.
Progress to Date. A competitive process was used to select the curricula included in the study. On December 6, 2005, the study issued a Request for Proposals that invited developers and publishers of early elementary school math curricula to submit a proposal to include their curricula in the evaluation. A technical working group (TWG) convened by the study team reviewed the submissions and recommended to ED curricula suitable for the study. Four curricula were selected at the beginning of March 2006. The names and publishers of the curricula appear in Table 1.
Table 1
Math Curricula Selected for the Study
Curriculum |
Publisher |
|
|
Investigations in Number, Data, and Space |
Pearson Scott Foresman |
Math Expressions |
Houghton Mifflin |
Saxon Math |
Harcourt Achieve |
Scott Foresman-Addison Wesley |
Pearson Scott Foresman |
The study’s goal was to recruit districts that meet the following criteria:
Have Title I Schools. Including districts that have Title I schools is consistent with the policy interest that underlies Title I for studying effective approaches to help low-income children meet state standards for academic achievement.
Are Geographically Dispersed. Geographic diversity helps establish “face validity” for the findings, though districts and schools were purposively selected and the findings are not externally valid, as described below.
Contain at Least Four Schools Interested in Study Participation. Requiring that each district contain at least four elementary schools supports implementation of all four curricula in each district, and makes it possible to examine whether curriculum effects vary across sites.
Because it would be impossible to select a representative sample of districts that both meet the criteria above and are interested in participation, the study did not statistically sample sites, but instead identified and recruited suitable sites. National district data sets do not contain information that can be used to identify districts that would be interested in a study of this kind and, therefore, information that could be used to select a representative sample. It would be extremely costly to collect the information directly.
Among districts that meet the criteria above, those that are actually interested in study participation may be unique in other ways. For example, interested districts must be willing to implement four very different curricula and each participating school must be willing to use the curriculum randomly assigned by the study. Sites that are comfortable with these participation requirements may value research evidence and would like to use direct evidence for their district to inform a future curriculum adoption decision. These participation requirements also may be acceptable to districts with tight budgets, because the free curriculum training and materials provided by the study may free up funds that districts can use in other ways.
Recruiting districts and schools typically involved three main activities. The first activity included identifying sites that meet the criteria above followed by initial outreach to assess district interest. Various sources were used to identify sites that meet the criteria above, including national district data sets (such as the Common Core of Data), the hundreds of districts MPR has worked with on previous studies, publisher nominations of districts that have expressed interest in using their curricula, and announcements about the study in national publications. Letters were then sent to each potential district. The study team followed up the letters with phone calls to assess each district’s interest.
The second recruiting activity involved site visits to interested districts that did not object to three critical elements of the study: (1) implementing all four of the study’s curricula, (2) random assignment of curricula to participating schools, and (3) the study’s data collection plan. Sometimes, several follow-up visits were required, so recruiters could describe the study to all individuals that would be involved if the district participated.
The third and final recruitment activity was to enroll schools, teachers, and any other relevant school or district staff that were interested in study participation. Enrollment began by confirming that schools interested in participation clearly understood the study’s parameters. Most importantly, recruiters confirmed that schools were willing to use any of the study’s four curricula and would support the study’s data collection. Recruiters provided consent forms to distribute to teachers in interested schools. A school was considered a participant when the study team received consent forms for all teachers at the target grade levels in the school.
Random assignment of curricula to schools was conducted separately for each participating district.1 The study used a “blocked” random assignment procedure that provides similar numbers and types of schools, teachers, and students to each curriculum. The procedure divides schools in each district into blocks, where each block contains from four to seven schools with similar baseline characteristics. Random assignment of curricula to schools is then conducted within each block. This procedure helps to minimize chance differences in school characteristics and sample sizes across curriculum groups, which help to increase the face validity and statistical power of the evaluation design.
The study recruited 12 districts and 110 schools (which is consistent with the study’s initial target of 12 districts and 108 schools). The four curricula were first implemented in the first grade during the 2006-2007 school year; and implementation was moved into the second grade during the 2007-2008 school year. This forms clearance package is for data collection during the 2008-09 school year when the four curricula will be implemented for the first time in the third grade in 20 of the original sample of schools.
Research Evidence Supported by Additional Data Collection. With the information collected through the OMB existing clearance, the study is able to examine the first-year effects of the curricula in the first grade and in the second grade. By extending the clearance for an additional year, the study would be able to examine the sustained (second-year) effects of the curricula through the third grade. Specifically, the study would be able to examine the:
Effects of the curricula for students with two years of curriculum experience
Effects of the curricula for first- and second-grade teachers with two years of curriculum experience
Below we provide a more detailed justification for collecting the additional information that supports these analyses.
Many U.S. children start school with weak math skills and there are differences between students from different socioeconomic backgrounds—those from economically disadvantaged families lag behind those from more affluent ones. These differences also grow over time, resulting in substantial differences in math achievement by the time students reach the fourth grade (National Assessment of Educational Progress, 2007). At the same time, under the No Child Left Behind Act (NCLB), Title I schools must make adequate yearly progress (AYP) in bringing their students to state-specific targets for proficiency in math and reading. The goal of this provision is to ensure that all students are proficient in math and reading by 2014.
The purpose of this large-scale evaluation study is to determine whether some early elementary school math curricula are more effective than others at improving student math achievement, thereby providing educators with information that may be useful for making AYP. A small number of curricula dominate elementary math instruction (seven textbooks make up 91 percent of the books used by K-2 educators), and the curricula are based on different theories for developing student math skills. NCLB emphasizes the importance of adopting scientifically proven educational practices; however, there is little research evidence to support one theory or curriculum over another.
Through the information collection thus far, the study will help to fill the knowledge gap about the effects of early elementary school math curricula by answering the following main questions:
What are the relative effects of different early elementary math curricula on student math achievement in disadvantaged schools during one year of curriculum usage in the first and second grades?
Under what conditions is each math curriculum most effective?
What is the relationship between teacher knowledge of math content/pedagogy and the effectiveness of the curricula?
By repeating information collection in the first and second grades, and extending it to the third grade, the study can answer an additional question:
Which math curricula result in a sustained impact on student achievement?
This sustained effects question would be answered by examining curriculum effects on student achievement for (1) first grade teachers with two years of curriculum experience that taught students who were new to the curricula, (2) second grade teachers and students with two years of curriculum experience, and (3) third grade teachers that were new to the curricula and taught students with two years of experience.
The following information collection would support the analyses of sustained effects:
Assessment of Teacher Knowledge of Math Content and Pedagogy. Math content/pedagogical knowledge of teachers that are new to the study would be assessed at the initial teacher training sessions before the curricula are introduced, using an assessment developed by researchers at the University of Michigan (Hill 2004).
Teacher Surveys. Two surveys would be administered to teachers. The first survey would be administered in the fall to teachers that are new to the study and would focus on teacher background information, classroom characteristics, math instructional approaches used before joining the study, and curriculum training provided by the publishers up to that point. The second survey would be administered in the spring to all teachers in the study and would gather information on follow-up training provided by the publishers, usage of the assigned curriculum and any other math curricula, and math instructional practice used during the year.
Classroom observations. Each of the third grade classrooms included in the study would be observed once during the school year.
Student Math Assessment. The study would assess student math achievement using the math assessment developed for the Early Childhood Longitudinal Survey (ECLS-K). This assessment meets the study’s requirements regarding validity, reliability, ability to measure achievement gains among low and high achievers, appropriateness for students from a wide range of backgrounds, length of the test, and ease of administration. The assessment also is adaptive, which limits the amount of time children are away from their classrooms and reduces the risk of ceiling or floor effects in the test score distribution, which can have adverse effects on measuring achievement gains.
Class Rosters. The study would collect rosters for each classroom in the study to build the frame for the student sample. Student demographic information would be requested as part of the roster collection, including student gender, date of birth, race/ethnicity, free/reduced-price meals eligibility, limited English proficient or an English language learner, and an Individualized Education Plan or receipt of special services for students with a disability.
Appendix A contains the letter to principals requesting class lists and student demographics. Appendix B contains a letter to teachers requesting that they complete the fall teacher survey, which also is included in the appendix. Appendix C contains the same information as in Appendix B, but for the spring teacher survey. The teacher survey forms are the same as those in the original OMB clearance package, with the exception of updating specific items to reflect the appropriate school year and asking about third grade students. The general classroom observation protocol is in Appendix D, and an example of the curriculum-specific classroom observation protocol is in Appendix E. The teacher knowledge assessment and the student assessment are not included because those instruments are copyrighted; however, a copy of the assessments can be provided to OMB upon request.
Table 2 lists the data collection efforts, and the respondents and timing for each collection effort. As the table shows, first grade students would be tested in both the fall and spring to obtain both a baseline and follow-up measure of math achievement. However, second and third grade students would only be spring tested because the study already assessed baseline achievement of these students.
TABLE 2
DATA COLLECTION EFFORTS
|
|
Timeframe |
|
Data Collection Efforts |
Respondent |
Fall 2008 |
Spring 2009 |
Assessment of Teacher Knowledge |
Teacher |
|
|
Teacher Surveys |
Teacher |
|
|
Observations of Third Grade Classrooms |
Classrooms |
|
|
Student Math Assessment |
Student |
|
|
Grade 1 |
|
|
|
Grade 2 |
|
|
|
Grade 3 |
|
|
|
Class Rosters |
School staff |
|
|
Information from the various data collection efforts would be used in the following ways:
Assessment of Teacher Knowledge of Math Content and Pedagogy. Scores on the test would be included in the analysis of student achievement to examine the relationship between teacher math content/pedagogical knowledge and the effects of the curricula.
Teacher Surveys. Demographic information collected through the fall survey data would be used to set a context for the study’s results. Some of the demographic information also would be included in the analysis of student achievement to increase the precision of the results. Information on the spring survey would be used to assess teacher-reported adherence to the study’s curricula.
Classroom observations. Information collected during the visits would be used to assess teacher adherence to the study’s curricula. Comparable adherence information collected through the teacher surveys would be used to assess the consistency between the two forms of data collection—that is, adherence information reported by teachers or directly observed by the study team.
Student Math Assessment. Student test scores would be used to determine the relative effects of the curricula.
Class Rosters. Student demographic information requested as part of the roster collection would be included in the analysis of student achievement to help increase the study’s statistical power.
The data collected for the study will serve two other purposes. First, each district participating in the study will receive aggregate results about the effects of the curricula for their schools. Second, the data collected by the evaluation will serve as a valuable resource for other researchers to study early elementary math curricula.
The data collection plan was designed to obtain reliable information in an efficient way that minimizes respondent burden. Consistent with that goal, information will be gathered from existing data sources, where feasible. Existing data sources will include existing class rosters to which the school can add readily available requested demographic information. This information will be obtained in the form of computer files, if a school prefers this method. If it is too burdensome or not possible for a school to provide this information as a computer file, schools will be asked to provide paper copies of the relevant information that will be coded by the study team using a student demographics form (see Appendix A).
Teacher surveys will be mailed to teachers. We considered other ways of administering the teacher surveys, such as a computer assisted telephone interview (CATI) or a web-based survey. However, because the study will survey about 400 teachers during the 2008-09 school year, the cost of developing a computer-assisted survey outweighs the benefits. Teachers also may find a mail questionnaire to be less burdensome because a computer-assisted interview would typically need to be conducted when teachers are at home since access to telephones in schools is uneven.
No equivalent sources of data exist for the study.
The primary entities for the study are districts, schools, teachers, and students. Burden is minimized for all respondents by requesting only the minimum data required to meet the study’s objectives. The data requirements were determined by careful consideration of the information needed to meet the study’s objectives and was reviewed by the study’s technical working group (TWG) listed in Section A.8.b.
The data collection plan described in this submission is necessary for conducting ED’s Evaluation of Mathematics Curricula and, consistent with the goal of Title I legislation, will provide information on the effectiveness of strategies for improving math achievement of students in disadvantaged schools.
There are no special circumstances associated with this data collection.
a. Federal Register Announcement
A request for comment on the proposed data collection activities and instruments was published in the Federal Register on May 22, 2008 in Vol. 73, No 100, page 29742. No public comments were received.
b. Consultations Outside the Agency
During the preparation of the data collection plan for this evaluation, input was sought from the study’s TWG. The TWG includes a number of the nation’s leading researchers in areas that are relevant to the study including evaluation, math education, and testing:
Richard Askey, University of Wisconsin-Madison
Doug Clements, State University of New York at Buffalo
Thomas Cook, Northwestern University
Lynn Fuchs, Vanderbilt University
Tom Loveless, The Brookings Institution
Kevin Miller, University of Michigan
Don Rock, Educational Testing Service
Hung-Hsi Wu, University of California at Berkeley
The study team will continue to consult with the TWG throughout the study on other issues that would benefit from their input.
c. Unresolved Issues
None.
The study is not planning to pay or give gifts to schools. To support implementation of the study’s curricula, all schools will receive free use of the math curricula materials for the school year, supported by free training for the teachers using the interventions.
We propose continuing with the $30 response incentive for teachers to complete the 30 minute fall survey, and the $10 response incentive for teachers to complete the 20 minute spring survey. We also propose continuing with the $75 response incentive for teachers to complete the 40 minute assessment of teacher knowledge of math content and pedagogy. These response incentives were approved as part of the original OMB clearance request.
The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant federal regulations and requirements. These include the Education Sciences Reform Act of 2002, Title I, Part E, Section 183 that requires “All collection, maintenance, use, and wise dissemination of data by the Institute: to “conform with the requirements of section 552 of Title 5, United States Code, the confidentiality standards of subsections (c) of this section, and sections 444 and 445 of the General Education Provisions Act (20 U.S.C. 1232 g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. In addition, for student information, the data collection efforts will ensure that all individually identifiable information about students, their academic achievements, their families and information with respect to individual schools, shall remain confidential in accordance with section 552a of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act. The study will also adhere to requirements of subsection (d) of section 183 prohibiting disclosure of individually identifiable information, as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a felony.
Data to be collected will not be released with individual student, teacher, or school identifiers. Data will be presented in aggregate statistical form only. A statement to this effect is included in a letter accompanying each questionnaire and will be read to students by a field examiner before administering tests. All MPR field examiners will be knowledgeable about confidentiality procedures and will be prepared to describe them in full detail, if needed, or to answer related questions raised by respondents. Respondents will be assured that all information identifying them or their school or program will be kept confidential.
The following safeguards are routinely employed by MPR to carry out confidentiality assurances:
All employees at MPR sign a confidentiality pledge that emphasizes the importance of confidentiality and describes their obligations (see Appendix F).
Access to sample selection data is limited to those who have direct responsibility for providing the sample and maintaining sample locating information. At the conclusion of the research these data are destroyed.
Identifying information is maintained on separate forms and files, which are linked only by sample identification number.
Access to the file linking sample identification numbers with the respondents’ identification and contact information is limited to a small number of individuals who have a need to know this information.
Access to the hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded material is shredded.
Computer data files are protected with passwords and access is limited to specific users. With especially sensitive data, the data are maintained on removable storage devices that are kept physically secure when not in use.
MPR will make certain that all surveys are held strictly confidential, as described above, and that in no instance will responses be made available except in tabular form. Under no condition will information be made available to school or program personnel. Project and school staff responsible for assisting MPR in the data collection will be fully informed of MPR’s policies and procedures regarding confidentiality of the data.
The Privacy Act of 1974 applies to this collection. A notice entitled System of Records: Evaluation of Math Curricula has been prepared for publication in the Federal Register.
We do not anticipate that any of the data collection forms will contain items considered to be of a sensitive nature.
Table 3 below reports estimates of burden hours for respondents. The study will assess teacher knowledge, survey teachers, and assess student math achievement. Administrative staff will be asked to provide class rosters, taking approximately one hour per school in the fall and an additional hour per school in the spring. Part B presents the rationale for respondent sample sizes.
TABLE 3
ESTIMATES OF BURDEN HOURS FOR RESPONDENTS
Instrument |
Respondents/Responses |
Response Time |
Total Time |
Teacher assessment fall 2008 |
60 third grade teachers (20 schools x 3 teachers/school) |
40 minutes |
40 hours |
Teacher survey |
|
|
|
Fall 2008 |
60 third grade teachers (20 schools x 3 teachers/school) |
30 minutes |
30 hours |
Spring 2009 |
405 first, second, and third grade teachers (45 schools x 9 teachers/school) |
20 minutes |
135 hours |
Class rosters Fall 2008 Spring 2009 |
45 schools 45 schools |
1 hour 1 hour |
45 hours 45 hours |
TOTAL |
615 responses |
|
295 hours |
A total of 295 burden hours are estimated for the 2008-09 school year. This estimate includes the teacher assessment, teacher surveys, and classroom rosters to be completed by school staff. Classroom observations and student assessments are not included in the burden estimate because the study team will carry out these activities.
Assuming an average teacher hourly wage of $30.75 (U.S. Department of Labor, 2005), this represents $6,304 in teacher burden. The additional time for school staff to provide class rosters (estimated at $25 per hour) represents an additional $2,250 in administrative staff time.
There are no additional respondent costs associated with this data collection.
The total estimated cost of the study is $18,382,563. This includes data collection, analysis, and report writing during the study’s remaining time period. The estimated average annual cost of the study over five years is $3,676,513.
This is a one-year extension of data collection with an increase of 295 hours and 615 responses. The increase is for an information collection that will occur during the 2008-09 school year to support analyses of the sustained effects of the study’s curricula. The burden hours being requested in this collection are only for one additional year. This results in a program change reduction of 1,013 burden hours, since the data collection burden hours for the currently approved collection have been completed.
The study will examine both curriculum implementation and the impacts of the curricula on student achievement.
Implementation Analysis. Teacher survey and classroom observation data will be used to assess curriculum implementation. The goal will be to assess teacher adherence to the features of their assigned curricula. Implementation data also will be used in the student achievement analysis described below to examine the relationship between implementation and impacts.
Curricula Impacts. The study will conduct two analyses to examine second-year effects of the curricula. The first analysis will be based on a new cohort of students who have teachers that already participated in the study for a year. The second analysis will be based on students that already participated in the study for a year. Hierarchical linear modeling (HLM) techniques will be used to estimate these effects. Statistical tests also can be conducted to determine if effects change as teachers acquire more experience with their assigned curriculum, and if effects change as students are exposed to the curricula for more than one year. We also can examine results for important subgroups of students, such as those in schools with lower baseline achievement. Results for the subgroup analyses could be useful for understanding how using a particular curriculum could help reduce (or even eliminate) achievement gaps that exist between groups of students.
The report is scheduled for public release in September 2010. The report will be based on data collected during the 2008-09 school year, and will include results from both descriptive analyses and impact analyses of data from teacher surveys, teacher assessments classroom observations, student assessments administered by the study, and school records.
The study will display the OMB expiration date.
No exceptions to the certification statement are being sought.
1 Although a school-level random assignment design requires a larger sample to detect effects than a student- or classroom-level design, a school-level design is more natural in this setting because curricula are typically implemented among all classes and students at a target grade level.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Roberto Agodini |
| Last Modified By | katrina.ingalls |
| File Modified | 2008-07-30 |
| File Created | 2008-07-30 |