Att_OMB 1850-0813 revision request-Part B 7-22-08

REQUEST FOR REVISION OF OMB INFORMATION COLLECTION FOR AN EVALUATION OF MATHEMATICS CURRICULA

PART B

The Evaluation of Mathematics Curricula is examining the relative effects of four curricula that represent the diverse approaches used to teach elementary school math in the United States. As noted in Part A of this submission, the study did not statistically sample sites, but instead identified and recruited sites that meet several criteria. Participating districts have Title I schools, are geographically dispersed, have at least four schools that were interested in participating in the study, and are willing to implement four different early math curricula in the district.

The study used a school-level random assignment design, which involved randomly assigning participating elementary schools in each district to the curricula included in the study. Random assignment of curricula to schools was conducted separately for each participating district, and only after all teacher consent forms for all participating schools in a district were received.

The study recruited 12 districts and 110 schools (which is consistent with the study’s initial target of 12 districts and 108 schools). The four curricula were first implemented in the first grade during the 2006-2007 school year; and implementation was moved into the second grade during the 2007-2008 school year. This forms clearance package is for data collection during the 2008-09 school year when the four curricula will be implemented for the first time in the third grade in 20 of the original sample of schools.

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

During the next (2008-09) school year for which an OMB extension is being requested, 5 districts and 45 schools will continue to participate in the study. Curriculum implementation will occur in both the first and second grades in all 45 schools. In 20 schools, curriculum implementation also will be expanded to the third grade. As described below, these data will be used to significantly improve the detectable effect size for the study’s research question about sustained effects.

2. Statistical Methods for Sample Selection and Degree of Accuracy Needed

The additional 2008-09 school year data will enable the study to conduct the following analyses of sustained effects with their associated minimum detectable effect sizes (MDE):¹

1st grade sustained effects

• Teachers with two years of curriculum experience and students with one year of experience (MDE=0.19). Without the additional data collection, the MDE=0.34.

2nd grade sustained effects

• Teachers and students with two years of curriculum experience (MDE=0.24). This analysis would not be feasible without the additional data collection.²

3rd grade sustained effects

• Students with two to three years of curriculum experience and teachers with one year of experience (MDE=0.38). This analysis would not be feasible without the additional data collection.

These minimum detectable effect sizes represent 13 to 26 percent of the one-year math achievement gain made by the average first grader from a low socioeconomic family—the type of students that largely are part of this evaluation. This statistic is based on data from the national Early Childhood Longitudinal Study (ECLS) (Pollack et al. 2004).

3. Methods to Maximize Response Rates

MPR has achieved high response rates on the data collection planned for the next (2008-09) school year. For example, during the 2006-2007 school year data collection, 96 percent of teachers completed the teacher knowledge assessment, 97 percent completed the fall teacher survey, and 88 percent completed the spring teacher survey. Student records were collected for 97 percent of students. Fall student testing was completed with 96 percent of eligible students, and spring student testing was completed with 94 percent of students who were enrolled and eligible for testing in both fall and spring.

4. Pilot Testing

All data collection instruments that require OMB clearance were pre-tested and cleared under OMB number 1850-0813.

5. Individuals Consulted on the Statistical Aspects of the Design

Don Jang, Senior Statistician, Mathematica Policy Research

The TWG members listed in section A.8.b. were also consulted on the statistical aspects of the evaluation design for this study.

1 An effect size is defined as a fraction of the standard deviation of the test score, and equals the difference in average student math scores of any two curriculum groups divided by the pooled standard deviation of the score for the two curricula being compared. A minimum detectable effect size is the smallest effect size that would be statistically significant (under conventional levels of confidence) given the study’s sample size, how the sample is distributed across the curriculum groups, and the extent to which students are clustered in schools and classrooms according to achievement, after adjusting for student, teacher, and school characteristics included in the analysis.

2 Using data currently being collected, the study also can examine sustained effects for 2nd grade based on students with two years of curriculum experience and teachers with one year of experience (MDE=0.34). None of the additional 2008-09 school year data that would be collected add to this analysis.

3