Impact Evaluation on Student Achievement of Teacher Professional Development In Mathematics (KA)
Impact Evaluation on Student Achievement of Teacher Professional Development In Mathematics
Att_PD MATH OMB Screener and Pilot TKT Draft 073106
Impact Evaluation on Student Achievement of Teacher Professional Development In Mathematics (KA)
OMB: 1850-0816
STUDY OF THE IMPACT ON STUDENT ACHIEVEMENT OF TEACHER PROFESSIONAL DEVELOPMENT DESIGNED TO ENHANCE TEACHER CONTENT KNOWLEDGE AND PEDAGOGICAL CONTENT KNOWLEDGE IN MATHEMATICS
OMB Clearance Request
May 2006 (Revised July 2006)
Prepared for:
Institute of Education Sciences
United States Department of Education
Contract No. ED-01-CO-0026/0020
Prepared By:
American Institutes for Research®
Table of Contents
Page
Description of the Mathematics PD Impact Study 5
Treatment Selection and Characteristics 6
Instruments for Which Clearance is Being Sought 16
Supporting Statement for Paperwork Reduction Act Submission 18
1. Circumstances Making Collection of Information Necessary 18
2. Purposes and Uses of the Data 19
3. Use of Technology to Reduce Burden 20
4. Efforts to Identify Duplication 21
5. Methods to Minimize Burden on Small Entities 21
6. Consequences of Not Collecting the Data 21
8. Federal Register Comments and Persons Consulted Outside the Agency 22
10. Assurances of Confidentiality 25
11. Justification of Sensitive Questions 26
12. Estimates of Hour Burden 26
13. Estimate of Cost Burden to Respondents 27
14. Estimate of Annual Cost to the Federal Government 28
15. Program Changes or Adjustments 28
16. Plans for Tabulation and Publication of Results 28
17. Approval to Not Display OMB Expiration Date 30
18. Explanation of Exceptions 30
B. Description of Statistical Methods 31
District Screener
1. Respondent Universe and Sampling Methods 32
2. Procedures for Data Collection 37
3. Procedures to Maximize Response Rates 37
5. Names of Statistical and Methodological Consultants and Data Collectors 38
Teacher Knowledge Test
1. Respondent Universe and Sampling Methods 39
2. Procedures for Data Collection 40
3. Procedures to Maximize Response Rates 40
5. Names of Statistical and Methodological Consultants and Data Collectors 42
list of exhibits
Page
Exhibit 1. Summary of Treatment 9
Exhibit 3. Sample Size by Treatment Group 12
Exhibit 4. Minimum Detectable Effect Sizes 14
Exhibit 5. Summary of Data Collection Instruments and Schedule 17
Exhibit 6. Project Advisor, Project Consultant, and Technical Working Group Members 23
Exhibit 7. Hour Burden for Respondents 27
Exhibit 8. Schedule for Dissemination of Study Results 29
Exhibit 9. Mapping of District Screener Items to Constructs 37
Introduction
Research has shown that teachers often lack an in-depth understanding of mathematical concepts, and strongly suggests that this lack of understanding compromises student learning. Professional development is one key to improving mathematics knowledge in the current teacher corps. Yet little is known about what kind of professional development works to enhance teacher knowledge, improve instruction, and increase student acievement in mathematics. This study will address this critical gap in the research.
According to Milgram (2004), “school mathematics is ultimately about rational numbers,” and students with a “fluent and adaptable” knowledge of fractions, decimals, and percents are well positioned for the second half of the K-12 mathematics curriculum (p. 222). However, student achievement data from the past three decades makes it clear that a great majority of U.S. students in grades 4-7 struggle to solve even routine problems involving rational numbers. The famous estimation problem from the 1978 NAEP mathematics assessment illustrates this point. When 13-year-old students were asked to estimate the answer to 12/13 +7/8 :
• 7 percent chose “1”
• 28 percent added the numerators and chose “19”
• 27 percent added the denominators and chose “21”
• 14 percent chose “I don’t know”
• a mere 24 percent correctly chose “2”
Results were not substantially better for 17-year-olds (Carpenter, et al., 1989). Almost twenty years later, on the 1996 NAEP mathematics assessment, only 35 percent of 8th graders identified the correct ordering of three fractions, all in reduced form, and only 12 percent of 8th graders correctly solved a problem involving the comparison of two rates (Silver & Kenny, 2000; National Research Council [NRC], 2001).
Students are not the only ones who face these conceptual hurdles. While we lack nationally representative data on the content knowledge of middle school teachers in the area of rational numbers, several small-scale studies suggest that many pre-service and practicing teachers lack strong algorithmic, formal, and intuitive understanding of rational numbers. For example, in a recent study of 147 pre-service teachers (26 of whom were undergraduate mathematics majors), Tirosh, Fischbein, Graeber, and Wilson (1999) found that, while most pre-service teachers were successful in adding, subtracting, and multiplying fractions, many had difficulties in division of fractions. Fewer than half of the 121 non-mathematics majors in their sample, for example, correctly computed the answer to 0.2/3. These prospective teachers also demonstrated weak conceptual understanding and had trouble constructing representations of key concepts. For example, 43 percent of the sample claimed that there is no number between 1/5 and 1/4, and very few were able to go beyond area model representations of fractions to construct set models, number lines, or ratio models. This indicates that too few teachers know the underlying mathematics and understand how to convey mathematical concepts well enough to help students build strong foundations for later mathematics growth.
The No Child Left Behind (NCLB) legislation sent a message that the federal government seeks to improve the quality of schooling in the United States for all students. Good teachers, of course, are critical to the improvement of schooling and student achievement (ECS, 2003; Rowan, 2002; Sanders & Rivers, 1996; Whitehurst, 2002). NCLB recognizes this point, as evidenced by the Title I requirement that every child have “highly qualified” teachers and the requirement that states report the percentage of their teachers participating in high-quality professional development. Title II places heavy emphasis on achieving this goal by seeking to improve both pre-service and in-service professional development. Many states and districts have also recognized the importance of professional development and have launched ambitious initiatives to upgrade the knowledge and skills of their teachers.
Preliminary research suggests that professional development that focuses on content—i.e., learning the subject to be taught, and how to teach the subject—is the key to changing teacher practice and student outcomes (Cohen & Hill, 2001; Kennedy, 1998). Other features of professional development, for example the extent to which teachers have opportunities for practice and reflection, also appear likely to be essential (Garet et al., 2001). Initial research that supports these beliefs includes a small set of small-scale randomized trials (Carpenter et al., 1989; McCutchen et al., 2002); some quasi-experimental studies (Bos et al., 1999; O’Connor, 1999; Gearhart et al., 1999); and some large-scale natural variation studies (Garet et al., 2001; Cohen & Hill, 2001). While there is experimental evidence demonstrating the positive impact of well-designed professional development programs, these studies have generally employed relatively small, local samples of volunteer teachers. Before widespread adoption of such programs can occur, they need to be tested in a wider range of districts and schools, under a greater variety of conditions, using a powerful research design. This is of key importance, because effective professional development that focuses on content and other features that reflect the consensus of researchers is likely to be more costly than the professional development that teachers typically experience (Garet et al., 1999).
States, districts, and schools have a critical need for information on what methods of providing professional development are most effective in improving teacher practice and student achievement since they are required to use Title II, Part A funds only for educational practices based on rigorous scientific research. Little such research currently exists. Thus, the Department of Education has initiated a rigorous study of the effectiveness of professional development. To address the focus of NCLB on improving the academic achievement of students in high-poverty schools, the study is focusing on schools with more than one-third of their students eligible for free or reduced-price lunch.
The purpose of the Study of the Impact on Student Achievement of Teacher Professional Development to Enhance Teacher Content Knowledge and Pedagogical Content Knowledge in Mathematics (“Mathematics PD Impact Study”) is to test a model of professional development that holds promise for improving middle school mathematics instruction and student achievement. Not only is this research needed to fill a gap in the middle school mathematics and professional development literature, but the success of several federal programs (e.g., Title I, Title II) relies on the selection of effective professional development strategies. Many educators and researchers agree that the key to improving instructional practice and student achievement in mathematics is to provide teachers with a core understanding of mathematics and how students learn to do mathematics. There is also agreement that the most effective way to change instructional practice is to provide teachers with ample opportunity to practice and reflect on what they learn in professional development. Therefore, the Institute of Education Sciences (IES) of the U.S. Department of Education has commissioned the Mathematics PD Impact Study to evaluate whether comprehensive content-based in-service professional development programs that have these characteristics can be designed to substantially improve middle school mathematics instruction and thereby improve students’ mathematics achievement. The study will focus on seventh grade teachers who are already using popular mathematics curricula, and will focus on the critical domain of rational numbers (i.e., fractions, percents, decimals, and proportional reasoning). Mastery of rational numbers is an essential foundation for algebra, and rational numbers account for a significant percentage of grade 7 mathematics content in all states.
A small-scale pilot of the professional development intervention will be conducted in the summer of 2006 and during the 2006-07 school year, and schools will be recruited for the full study from late fall of 2006 through February of 2007. The full-scale implementation of the professional development interventions will take place in the summer of 2007 and during the 2007-2008 school year. Data for the full study will be collected in the summer of 2007 and during the 2007-08 and 2008-09 school years.
The Institute of Education Sciences (IES) of the U.S. Department of Education requests clearance to screen and recruit districts and schools, to pilot a teacher knowledge test, and to carry out data collection activities for the Mathematics PD Impact Study. OMB clearance is sought in a two-stage process. This initial package requests clearance to: 1) contact a sample of districts and schools to establish their eligibility for the study and recruit them to participate in the full study, and 2) conduct a pilot test of a teacher knowledge test (TKT) in the area of rational numbers in order to ensure that this critical outcome will be well measured in the full study. Screening districts to establish eligibility will involve the administration of the district screener included as Appendix A and described under the Data Collection section of this package. A second OMB clearance package will be submitted in the fall of 2006 to request clearance for the full study’s data collection instruments. This two-stage process will enable IES to obtain approval for the two activities involving 10 or more respondents that must occur in the year prior to the full study—recruiting for the full study and ensuring the quality of the newly developed test to measure the critical teacher knowledge outcome.
This document contains three major sections. The first section is a full description of the Mathematics PD Impact Study, which provides context for the district screening and recruitment procedures and TKT pilot procedures for which we are seeking clearance. The second section contains Parts A and B of the Supporting Statement for the Paperwork Reduction Submission. The final section (a set of appendices) contains the district screener and sample informational materials to be used in recruitment.
Description of the Mathematics PD Impact Study
IES and its contractors have designed a randomized field trial, hereafter referred to as the Mathematics PD Impact Study, to examine the impact of professional development on teacher knowledge, instruction and student achievement in mathematics. This study, focusing on the topic of rational numbers in seventh grade, will examine two experimental conditions:
Treatment: A professional development intervention consisting of a three-day summer institute (18 hours per teacher), five one-day seminars held during the school year (30 hours per teacher), and 10 days of intensive in-school coaching (20 hours per teacher)
Control: A “business as usual” condition within the same districts where teachers receive the mathematics professional development that is typically covered, absent our intervention
The Mathematics PD Impact Study is actually two parallel sub-studies of the same design conducted in in two different , but widely used curricular contexts. The two curricula cover essentially the same mathematics content, but represent contrasting instructional approaches and consequently make somewhat different demands on teachers’ skills.
A pilot test of the professional development intervention will be conducted during the 2006-07 school year (draft data collection instruments will also be piloted with less than 10 respondents). Purposive samples of districts using each of the two predetermined curricula will be recruited for the full study during the 2006-2007 school year. The professional development intervention will be implemented during the summer of 2007 and the 2007-08 school year, and data will be collected on the implementation and impact of the intervention from the summer of 2007 until the spring of 2009. To ensure objectivity, the study will be carried out by two teams. One team will select the professional development intervention and implement it at the sites recruited for the study, and a separate team will lead the data collection and analysis of the effects of the selected approaches to professional development. The design of the study, the professional development treatment, and the process of selecting participating districts and schools are described in more detail in the Treatment Selection and Characteristics section below.
The proposed study focuses on one central research question:
What are the impacts on teacher knowledge, teacher practice, and student mathematics achievement of providing teachers with intensive professional development in the area of rational numbers?
In addition, attention will be given to the following supplementary questions:
Are the mathematics professional development activities implemented with fidelity (treatment only)?
To what extent do teachers (both treatment and control) participate in mathematics professional development activities?
To what extent is the amount of teacher participation in the professional development activities related to changes in teacher practice and student learning?
What is the relationship between teacher experience or prior knowledge and the impacts of the professional development intervention?
What is the relationship between student characteristics and the impacts of the professional development intervention?
Do the impacts of the professional development interventions change over time?
Of practical importance to administrators, what are the per teacher costs of participating in this type of professional development?
Treatment Selection and Characteristics
The professional development model at the core of this study has been developed through a national study of the implementation of the Eisenhower Professional Development Grant Program and extensive reviews of the literature, where there is concurrence regarding three core features and three structural features of professional development that show strong associations with changes in teacher practice.
The following is a brief definition of each of the features that have been found to be important in designing professional development.
Core Features
The three key core features include (1) a focus on the content teachers teach, (2) opportunities for teachers to learn and connect their learning to practice, and (3) coherence among professional development goals, teachers’ own goals, and the standards and assessments that should guide teachers’ practice (Garet et al., 2001).
Focus on Content. Professional development that focuses on what students are expected to learn and how students learn the subject matter appears to support teacher knowledge and practice in ways that improve student achievement (Cohen & Hill, 2001; Garet et al., 2001; Kennedy, 1998; Carpenter, Fennema, et al., 1989).
Opportunities for active learning. Active learning refers to the engagement of teachers in the learning process through observation, meaningful discussion, practice, and reflection. Teachers appear to benefit through opportunities to observe and be observed by expert teachers; opportunities to integrate learning into classroom practice; opportunities to review student work with others; and opportunities to reflect, discuss, and write about their learning (Garet et al., 2001; Lieberman, 1996; Loucks-Horsley et al., 1998).
Coherence of professional development activities with other important aspects of teachers’ professional work. Professional development appears to be more effective when the activities and goals involved are aligned with other initiatives designed to change instruction, including standards and assessments and curriculum adoptions; when they are consistent with teachers’ personal goals for their development; and when they afford opportunities for teachers to communicate with others involved in similar professional development activities (Cohen & Hill, 1998; Garet et al., 2001; Grant, Peterson & Shojgreen-Downer, 1996; Lieberman & McLaughlin, 1992).
Structural Features
The structural features are:
Form of the activity, how professional development activities are organized. Research suggests that professional development activities that are incorporated in teachers’ daily school work, such as coaching, mentoring, and in-school discussion groups, provide more opportunities for active learning and encourage greater coherence of activities with teachers’ and schools’ larger goals and teachers’ communications with others than professional development not incorporated in their school work. Furthermore, it helps sustain professional development over time (Garet et al., 2001; Hargreaves & Fullan, 1992; Little, 1993; and Stiles, Loucks-Horsley, & Hewson, 1996.)
Duration of the activity. Duration refers both to the time span of the effort and the number of hours committed to the effort. Duration appears to be supported by the form of the activity. In turn, both span and number of hours of professional development are associated with opportunities for active learning (Garet et al., 2001; Cohen & Hill, 2001; and O’Connor, 1999).
Collective participation of groups of teachers. Including teachers from the same school, same department within the school or, ideally, the same grade level in the school is thought to foster opportunities for collegial development that improves professional development in the short-term and helps sustain it over the long-term (Ball, 1996; Knapp, 1997; Talbert & McLauglin, 1993; Elmore, 2002).
These core and structural features became the specifications for the selection of models and materials for the Mathematics PD Impact Study’s treatments. After reviewing potential professional development models for which there existed well-specified “off the shelf” interventions for use in the study, project staff, in consultation with external advisors with expertise in mathematics professional development, focused on interventions that follow a general model—intensive, content-based summer institutes with follow-up seminars, and ongoing coaching during the school year. Based on this model we selected two professional development providers through a competitive process to implement the experimental condition of the Mathematics PD Impact Study. In April 2006, America’s Choice and Pearson Achievement Solutions were selected to provide the study’s professional development treatment.
This treatment encompasses many features of high-quality professional development; it is of reasonable duration relative to other professional development treatments that have shown promising evidence of effectiveness (e.g., Carpenter et al, 1989; Gerhardt, Saxe, et al, 1999), it focuses on important mathematics content, and it provides opportunities for follow-up and supervised practice and feedback. “Business as usual,” representing what is typically provided, serves as the control condition. See Exhibit 1 for the content and duration of the professional development interventions that each teacher in the treatment group will experience.
Exhibit 1. Summary of Treatment
Professional Development Activities |
Treatment |
Summer Institute for teachers |
18 hours |
Seminars during the school year |
30 hours (5 days) |
On-site coaching during the school year |
20 hours per teacher (10 days per school) |
Total per teacher, 2007-08 |
68 hours |
The Mathematics PD Impact Study will employ an experimental design with randomization of schools—not individual teachers—to treatment and control conditions for reasons that are theoretical, methodological, and programmatic. Theory suggests that professional development can work best when it takes place in an environment where there is mutual reinforcement of what is learned among co-workers. In addition, many forms of professional development are geared toward changing organizational norms, changing expectations and behaviors, and changing specific professional practices. Targeting professional development at schools (or at all of the seventh-grade teachers in schools) is consistent with this theory.
Randomizing schools is methodologically important because it maintains a spatial separation between teachers who receive the different forms of professional development being tested. This separation limits the natural opportunities for teachers to share what they learn with each other and thereby “contaminate” the experimental treatment contrast. Programmatically, it is important in that randomizing schools is consistent with how professional development would be provided under real-world operating conditions—using the same approach for all teachers in the same grade at a given school. Thus, both in terms of promoting verisimilitude (face validity) and reflecting a cost basis that is relevant for assessing the different forms of professional development, randomizing schools is more appropriate than randomizing teachers.
The principal drawback of randomizing schools is the need for a larger sample of schools and teachers. Based on findings in Bloom (2003), the evaluation team has determined that, within each of our two parallel sub-studies, a sample of 64 schools is necessary in order to provide the required statistical power to detect policy-relevant effects.
One sub-study will be conducted in a set of districts using typical commercial texts, either Prentice Hall Mathematics or Glencoe McGraw-Hill Mathematics: Applications and Concepts; the parallel study will take place in a set of districts using a contrasting text, Connected Mathematics, which uses different instructional strategies. For each sub-study, a sample of 64 schools will be built by recruiting 8 qualifying districts located in several different states, and then selecting 8 schools per district to participate in the study.
Within each district, four of the eight middle schools will be randomly assigned to each professional development condition. 1 This will yield 32 schools per condition within each parallel sub-study (see Exhibit 2 for the complete structure of the design and Exhibit 3 for a summary of the sample sizes). Project staff anticipate approximately three seventh-grade teachers per school, each teaching an average of three relevant seventh grade mathematics class sections with roughly 25 students per class section, i.e., 225 students per school, in a given academic year. Thus, the total sample of teachers across the two sub-studies will be about 384; the total sample of students will be about 28,800.
With the resulting design and conditions, the key comparison, treatment vs. “business as usual” (the control condition), can be examined in order to address the primary research question regarding the effects of professional development in mathematics on teacher instruction and student achievement separately within the context of each type of mathematics curriculum.
District |
Mathematics Curriculum |
Treatment Group |
Number of Schools (unit of random-ization) |
Number of Teachers (based on estimate of 3 teachers per school) |
Number of Students (based on estimate of 25 students in each of three relevant class sections) |
District 1 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 2 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 3 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 4 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 5 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 6 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 7 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 8 |
Prentice Hall or Glencoe |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 9 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 10 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 11 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 12 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 13 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 14 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 15 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
District 16 |
Connected Mathematics |
Treatment |
4 |
12 |
900 |
Control |
4 |
12 |
900 |
||
Total |
|
|
128 |
384 |
28800 |
Exhibit 3. Sample Size by Treatment Group
-
Treatment Group
Number of Schools
Number of Teachers
Number of Students
Sub-study conducted within Prentice Hall or Glencoe Mathematics curricular context
Treatment
32
96
7200
Control
32
96
7200
Sub-study conducted within Connected Mathematics curricular context
Treatment
32
96
7200
Control
32
96
7200
Total
128
384
28800
In order to assess the statistical power of the current design, we used data from several large urban school districts from across the country to calculate the variance components and estimate Minimum Detectable Effect Sizes (MDES) directly for seventh grade mathematics achievement.2 Based on this analysis, Exhibit 4 presents estimates of how MDES for estimated program effect on seventh grade achievement outcomes would vary with different configurations of school and student sample sizes. These estimates confirm that, with the current design parameters, average estimated MDES fall within the target effect size range for the study. In particular, the fifth column of Exhibit 4 suggests that MDES for an experiment involving 64 schools and 225 students per school (that is, three seventh grade teachers per school, each teaching three classes averaging 25 students per class) are expected to be .16 standard deviations. Exhibit 4 also shows that, if the samples from the two studies are combined for a total sample of 128 schools, estimated MDES fall to .11 standard deviations. The statement of work in which ED specified the desired precision identified a target range for MDES of .15 to .2 standard deviations. The estimated MDES suggest that precision for this study fits within this range. No absolute standard exists as to what represents a large versus a small effect size. Nevertheless, many researchers have relied on a rule of thumb that suggests that effect sizes of approximately .20 standard deviations or less be considered small, effect sizes of .50 be considered moderate, and effect sizes of .80 be considered large (Cohen, 1988). Further, a meta-analysis of treatment effectiveness studies by Lipsey (1990) found that, out of 102 studies, most of which were from education research, the vast majority found effects larger than the MDES implied by the design parameters of this study. In particular, the bottom third of the distribution of impacts ranged from about 0 to .32, the middle third of impacts ranged from .33 to .50, and the top third of impacts ranged from .56 to 1.26. Relevant studies focusing on teachers’ content knowledge reviewed by Kennedy (1998) obtained an effect size of .4 or larger for some outcomes – substantially greater than the .2 minimum detectable effect size target for our design, but with interventions of greater intensity and volunteer teachers. In short, prior research and our analysis of data across several large urban school districts suggest that the design parameters specified in the RFP are sufficient to detect policy-relevant effects should they exist. It is also worth noting that the MDES do not change dramatically as we reduce the total number of students per school.
Exhibit 4. Minimum Detectable Effect Sizes
Number of Schools |
Students Per Grade
|
Multiplier for J-k |
|||||
60 |
75 |
100 |
200 |
225 |
400 |
||
50 |
0.35 |
0.34 |
0.34 |
0.33 |
0.33 |
0.33 |
5.3633 |
64 |
0.17 |
0.17 |
0.17 |
0.16 |
0.16 |
0.16 |
2.9846 |
70 |
0.16 |
0.16 |
0.16 |
0.15 |
0.15 |
0.15 |
2.9321 |
80 |
0.15 |
0.15 |
0.14 |
0.14 |
0.14 |
0.14 |
2.8899 |
90 |
0.14 |
0.14 |
0.13 |
0.13 |
0.13 |
0.13 |
2.8683 |
100 |
0.13 |
0.13 |
0.13 |
0.12 |
0.12 |
0.12 |
2.8552 |
110 |
0.12 |
0.12 |
0.12 |
0.12 |
0.12 |
0.12 |
2.8464 |
128 |
0.11 |
0.11 |
0.11 |
0.11 |
0.11 |
0.11 |
2.8362 |
During this stage of the clearance process, clearance is being sought only for screening and recruiting districts and for piloting the teacher knowledge test; however, it is important to view these activities within the context of the overall data collection plan.
The data collections for the PD Mathematics Impact Study serve four broad purposes:
Screening districts for eligibility to participate in the full study.
Documenting the implementation of the two interventions, both to verify the fidelity with which the models were implemented and to produce a description of the interventions that will allow others to replicate them.
Assembling contextual data to help understand the results:
data to describe the sample of schools and teachers
data to compare treatment and control schools and teachers prior to the implementation (i.e., to assess how well randomization has balanced the samples)
covariates (control variables) that can be included in analyses to reduce unexplained variance
variables that may interact with the treatment
Measuring the outcomes, including teacher knowledge, teacher practice, and student achievement
An overview of the study’s instruments, their primary purposes, and the schedule for their use is provided in Exhibit 5. The activities for which clearance is being sought in this package—district screening and recruitment and the pilot of the teacher knowledge test—are indicated in the first two rows of the exhibit and are highlighted with bold text.
Rows 3 and 4 describe the various forms that will be used to document the delivery of of the intervention and the participation of each teacher in the intervention. Among these, only the teachers’ PD evaluation forms and logs of coaching activities impose a burden on participants, but the other instruments are listed in order to give a broader overview of the data that the study will have available for analysis. For example, study staff will observe the institutes and seminars and will record their observations on a standardized PD observation form. Also, the coaches, who are, of course, employed by the study, will be asked to keep their own logs of coaching contacts with teachers, and this information will be triangulated with the information from the teachers’ coaching logs to gain a more reliable picture of the amount and type of coaching in which each individual teacher participated.
Row 5 indicates the teacher survey, a written questionnaire that will be used to collect information on mathematics teaching experience and training. Teachers will complete the questionnaire at baseline and will update the information on their training at each annual follow-up.
Rows 6 and 7 describe the instruments that will be used to collect outcome information on teachers’ knowledge and practice. The knowledge test is, of course the final version of the teacher knowledge test for which we are requesting pilot clearance. The classroom observation forms will be used by study staff to collect standardized information about each teacher’s classroom practice. We expect to observe each teacher twice during the intervention year and twice during the follow-up year.
Rows 8 and 9 describe the instruments that will be used to collect information on the final critical outcome—changes in student knowledge. Row 8 indicates the student achievement test that will be administered to all students in grade 7 classes taught by participating teachers. By administering the test in the fall and spring, we will be able to compute growth scores for individual students. Furthermore, because the test will be given to two cohorts of 7th graders: those who are in participating teachers’ classes during the intervention year, and those who are in these teachers’ classes during the follow-up year, we will be able to determine whether the professional development has an increased impact on student outcomes once teachers have had time to assimilate their training. The Xs marked as optional in row 8 indicate the possibility of collecting a second year of test data on the students who were 7th graders during the intervention year.
Finally, row 9 refers to data collection protocols that will be used to collect state test data for students of participating teachers. State test scores have the greatest policy relevance but will not provide a standardized measure across all students, given that we expect the study to be spread over districts in several states.
Instruments for Which Clearance is Being Sought
During the first stage of the clearance process, clearance is being sought only for the procedures for contacting a sample of districts and schools to establish their eligibility for the study and recruit them to participate in the full study, and for the procedures for piloting the teacher knowledge test (TKT). Both of these data collections occur in the year prior to the full study and are necessary to ensure that we can effectively conduct the full study. Establishing district eligibility involves a screener that will be administered via telephone interview and will capture critical information about the districts’ mathematics curricula and existing professional development efforts. Piloting the TKT will allow the study team to refine both the test items and test administration procedures to ensure that this key outcome of the study will be well measured in the full study. The requested two-stage clearance process will enable IES to begin recruiting sites for the full study in fall 2006. It also will allow for a fall 2006 pilot testing of a newly-developed TKT. The remaining study instruments will be based on previously developed instruments and piloted using samples of nine or fewer respondents. All final instruments for the full study will be included in the second OMB package.
Exhibit 5. Summary of Data Collection Instruments and Schedule
Primary Purpose |
Data Collection Instrument |
Data Collection Schedule |
||||||||||||||||||||||||
Document Treatment |
Provide Screen/ Context/ Covariates |
Measure Outcomes |
Fall 2006 and Winter 2007 |
Spring 2007 |
Summer 2007 |
Fall 2007 |
Winter 2008 |
Spring 2008 |
Summer 2008 |
Fall 2008 |
Winter 2009 |
Spring 2009 |
Summer 2009 |
|||||||||||||
|
X |
|
1. District Screener |
X |
|
|
|
|
|
|
|
|
|
|
||||||||||||
|
X |
X |
2. Pilot of Teacher Knowledge Test |
X |
|
|
|
|
|
|
|
|
|
|
||||||||||||
X |
|
|
3. Institute/Seminar Documentation Protocols: --PD Observation Form --Training Sign-in Sheet --PD Evaluation Form: Teacher Reflections |
|
|
X |
X |
X |
X |
|
|
|
|
|
||||||||||||
X |
|
|
4. Coaching Documentation Protocols: --Logs of Coaching Activities: Coach Form --Logs of Coaching Activities: Teacher Form |
|
|
|
X |
X |
X |
|
|
|
|
|
||||||||||||
X |
X |
|
5. Teacher Survey |
|
|
|
X |
|
X |
|
|
|
X |
|
||||||||||||
|
X |
X |
6. Teacher Knowledge Test |
|
|
X |
|
|
X |
|
|
|
X |
|
||||||||||||
|
|
X |
7. Classroom Observation Forms |
|
|
|
X |
|
X |
|
X |
|
X |
|
||||||||||||
|
X |
X |
8. Student Achievement Test 7th grade (Student Achievement Test 8th grade) |
|
|
|
X
|
|
X
|
|
X X(option) |
|
X X(option) |
|
||||||||||||
|
X |
X |
9. Extant Student Achievement Data Collection Protocol |
|
|
|
|
|
|
X |
|
|
|
X |
||||||||||||
Supporting Statement for Paperwork Reduction Act Submission
The No Child Left Behind (NCLB) legislation sent a message that the federal government seeks to improve the quality of schooling in the United States for all students. Good teachers, of course, are critical to the improvement of schooling and student achievement (ECS, 2003; Rowan, 2002; Sanders & Rivers, 1996; Whitehurst, 2002). NCLB recognizes this point, as evidenced by the Title I requirement that every child have “highly qualified” teachers and the requirement that states report the percentage of their teachers participating in high-quality professional development. Title II places heavy emphasis on achieving this goal by seeking to improve both pre-service and in-service professional development. School districts and states receive nearly $3 billion in federal Title II, Part A funds and may use them for a wide range of activities, including providing professional development to improve the knowledge of teachers. The law stipulates that these funds be used for professional development activities that advance teacher understanding of effective instructional strategies that are based on scientifically based research and that improve student academic achievement or substantially increase the knowledge and teaching skills of teachers [Title IX, Part A, Section 9101(34)].
Many states and districts have also recognized the importance of professional development and have launched ambitious initiatives to upgrade the knowledge and skills of their teachers. However, without studies such as this one, states and districts have little scientific research on which to base their decisions about where their professional development dollars should be spent.
The Mathematics PD Impact Study is aligned with the larger goals of NCLB in two ways. First, it is one of the first rigorous, large-scale studies of the impact of professional development. Second, since it is the intention of the Department of Education to conduct this work in high-poverty schools and focus on the effectiveness of professional development for improving the mathematics achievement of students in high-poverty schools, it aligns with the interest of NCLB in improving the academic achievement of students in such schools.
To identify the districts that this study should include, it is important to implement a thorough screening process. This will ensure that the resulting study sample consists of districts with middle schools that have at least one-third of their students in poverty, that have adequate enrollments and grade organization to meet the power requirements of the study, that are implementing the mathematics curricula of interest, and that are not participating in other intensive professional development initiatives which could confound the results.
The study’s primary research question is: What are the impacts on teacher knowledge, teacher practice, and student mathematics achievement of providing teachers with intensive professional development? To answer the first part of this question, a teacher knowledge test in the area of rational numbers at the seventh grade level must be developed, as such a test does not currently exist. In order to ensure that this critical outcome will be well measured in the full study, a pilot test of the TKT is essential.
As described previously, the district screener will allow the project staff to identify districts and schools that are eligible for the Mathematics PD Impact Study. The Mathematics PD Impact study will be conducted as two parallel sub-studies: one in high poverty schools implementing typical commerical texts, either Prentice Hall Mathematics or Glencoe McGraw- Hill Mathematics, and another in similar schools using a contrasting approach, Connected Mathematics.
The Mathematics PD Impact Study will test the impact of a professional development approach for mathematics instruction in the middle grades that reflects high-quality features suggested by existing research against “business as usual” (control condition). The outcomes of interest include teacher knowledge, teacher instructional practice, and student achievement. To measure teacher practice and student achievement outcomes, the team proposes to use established instruments with some adaptations and refinements based on trial observations or cognitive interviews with samples of fewer than 10 respondents per instrument. However, to measure seventh grade teachers’ content and pedagogical knowledge specifically in the area of rational numbers, such an established instrument does not exist.
The team therefore proposes to develop a customized instrument drawing appropriately from high-quality instruments that have been proven in other studies but that do not themselves adequately map the rational numbers strand at the seventh grade level. Fewer than 10 respondents will not provide an adequate pretest of this newly developed instrument. Rather, piloting the TKT with a larger sample of the types of seventh grade mathematics teachers with which the TKT is intended to be used is necessary to ensure that both the instrument and the administration procedures work well and that one of the primary outcomes of the study can be reliably and validly measured.
The Mathematics PD Impact Study will be of immediate interest and import for policymakers, researchers, and practitioners, demonstrating the degree to which professional development can affect teacher knowledge, classroom practices, and student achievement. Important contextual information also will be collected and taken into account in the final analyses (e.g., checking that random assignment resulted in balanced groups; describing the study sample; and documenting the content and dosage of the treatments). The variables and scales to be created from the data collected for this purpose will be detailed in a future request.
Information collected via the screener will be collected through telephone interviews. Telephone data collections have many advantages over mail surveys. A telephone survey is less burdensome because the respondents can provide verbal answers without the time and effort that is needed to complete a written questionnaire. Consequently, a phone interview is more likely than a paper survey to yield a better response rate.
Moreover, data collected by phone are more efficient and error free. Telephone interviews can generate responses within minutes once the interviewer reaches the respondent. Further, the interviewer can probe for further information to clarify ambiguous or conditional responses.
Information about teachers’ mathematics content knowledge will be collected with a pencil and paper test. In order to ensure high response rates and quality data, the administration of the TKT will be proctored. Administering and proctoring a pencil and paper test, rather than administering a computer-based test, will allow us to verify that the teachers have completed the TKT without consulting reference materials or peers, and will ensure that teachers receive, complete, and return the TKT in a timely manner. During each of the administrations, AIR will have a project representative (e.g., a classroom observer or district liaison) administer and collect the tests.
Before administering the screener, every effort will be made to collect the needed information via the CCD, district Web sites, mathematics curriculum publishers and other knowledgeable consultants. However, much of the information required to identify eligible districts and schools is either not publicly available or is not kept up to date. The screening interviews will therefore allow study staff to collect information not available elsewhere and verify information gathered from public sources.
The assessment of teacher content knowledge in mathematics does not duplicate other efforts. For purposes of evaluating the impact of the professional development provided to teachers, teacher content knowledge must be assessed both before and after the delivery of the professional development, and the instruments used to measure the change in knowledge must be standard across the teacher populations.
No small businesses or entities will be involved as respondents. Screening data will be collected only from districts. Information about teachers’ mathematics content knowledge will be collected only from teachers in the TKT pilot.
The Mathematics PD Impact Study represents the first effort by the Department of Education to conduct a rigorous study of the effects of professional development in mathematics. As required by NCLB, states must adopt professional development that is grounded in scientifically based research. Without this study, states and districts will have a limited basis on which to comply. In order to conduct the Mathematics PD Impact Study and provide this much-needed information, districts must be screened for eligibility to ensure that they represent the interests of the study, and the TKT that provides one of the main outcomes for the full study must be piloted to improve the test’s reliability and validity.
No special circumstances apply to this study.
The 60-day notice for this collection was published in the Federal Register on 3/12/06 and ended 5/16/06. The team has addressed all public comments received.
To assist with the development of the screening criteria and the study as a whole, project staff have drawn on the experience and expertise of a network of outside experts. The consultants and their affiliations are listed in Exhibit 6.
Exhibit 6. Project Advisor, Project Consultant, and Technical Working Group Members
Project Advisor
Expert |
Affiliation |
Andy Porter |
Patricia and Rodes Hart Professor of Educational Leadership and Policy, and Director of the Learning Sciences Institute at Vanderbilt University |
Project Consultant
Expert |
Affiliation |
Mark Hoover Thames |
Research Scientist, Learning Mathematics for Teaching Project, University of Michigan |
Technical Working Group Members
Expert |
Affiliation |
Julian Betts |
Professor of Economics, University of California-San Diego |
Doug Carnine |
Director, National Center to Improve the Tools of Educators, University of Oregon |
Mark Dynarski |
Senior Research Fellow, Mathematica Policy Research |
Lynn Fuchs |
Professor, Department of Special Education, Vanderbilt University |
Russell Gersten |
Professor Emeritus in the College of Education, University of Oregon |
Roger Howe |
Professor, Yale University |
Kenneth Koedinger |
Associate Professor, Carnegie Mellon University |
Brian Rowan |
Professor, University of Michigan |
John Woodward |
Professor of Education, University of Puget Sound |
Hung-Hsi Wu |
Professor of Mathematics, UC Berkeley |
To date the project advisor and TWG members have convened once, in October 2005, and provided comments on the study design, the treatments, and the data collection instruments. Meetings are scheduled to occur twice in the second and third years, and once in the fourth year. Project staff also will use outside experts individually for consultation on an as-needed basis.
No payments will be given for participating in the district screening interviews. Payments will be given for participating in the pilot test of the TKT. Since the purpose of the pilot TKT is to assess the difficulty of the TKT items, these payments must be sufficient to ensure that the pilot sample is not skewed towards teachers who volunteer frequently and are confident in their mathematical knowledge. Teachers will be offered $60 if arrangements can be made to convene teachers at a location that does not entail additional travel e.g., immediately before or after a district-wide professional development event or at their school work site. Teachers will be offered $75 if teachers must come to a central location in the district for the specific purpose of taking the pilot test. The added incentive will offset the inconvenience of making a special trip to the testing site and will compensate for teachers for their travel time and expenses. In either case, teachers will complete the pilot TKT in a proctored setting, during a time window for which they will not already be being paid.
The minimum incentive amount of $60 is appropriate given that the time required for teachers to complete the TKT is 60 minutes. We also expect approximately 15 minutes of administrative time (e.g., assembling teachers for the group administration, handing out test documents, and reading instructions). In the Study of Professional Development Impact in Reading, the incentive for the 30-45 minute Reading Content and Practices Survey was $30 per administration. The ESSI teacher knowledge survey paid an incentive of $145 dollars per K-2 general elementary teacher (i.e., $30 per hour across 4 hours plus $25 for travel and meal). The guidelines for National Center for Education Evaluation (NCEE) evaluation studies, prepared March 22, 2005, stipulate $100 incentives for 1 hour teacher assessments in which there is a high respondent burden. Note that the teachers taking the pilot TKT will lack two other potential incentives for participation because they will receive no professional development through the Mathematics PD Impact Study and will not receive reports of their scores or other feedback.
Turning to the study in general, the following are incentives for participation in the full study that will be explained to schools and districts during the recruitment process:
Districts’ treatment group schools will receive the professional development treatment at no cost;
Districts will be paid stipends of up to $200 per day, depending on existing district contract rates, for any days that treatment group teachers attend the professional development which are not already covered in their contracts; and
The study will pay districts for substitute teachers whenever teachers in the treatment group are removed from their classes for professional development seminars or coaching (up to $200 per day, depending on existing district substitute teacher pay schedules).
Because control schools will not receive the professional development or the other incentives listed above but will be subjected to student testing and other data collection activities, each control school will receive $1,000 in compensation . Note that the limit of $200 per day is based on a review of rates that districts pay to teachers for professional development that occurs at times not covered by their employment contract, and these rates have been used successfully in the Study of Professional Development Impact in Reading.
No information collected during screening interviews or piloting of the TKT will be reported or published in a manner that would identify individual respondents. The pilot version of the TKT will not include any personal identifying information or subject identifier. An explicit statement regarding anonymity will be communicated to all teacher respondents. Results from the pilot TKT will only be reported in aggregate, and all aggregate data categories will be constructed in a manner that does not compromise confidentiality of districts and schools.
During the course of the screening process, information collected by the screening protocol will not be of a personal nature nor will it rely on the individual respondent’s opinion. All necessary screening information and documents will be kept in a file accessible only by project staff under the supervision of the Project Director. After the project is completed, the contractors will destroy all information identifying screened districts and schools.
For the full study, to ensure that the data collected are not available to anyone other than authorized project staff, staff will adhere to the following procedures:
All project staff will agree to an assurance of confidentiality;
All project staff will keep completely confidential the names of all respondents, all information or opinions collected during the course of the study, and any information about respondents learned incidentally;
Reasonable caution will be exercised in limiting access to data collected only to persons working on the project who have been instructed in the applicable confidentiality requirements for the project; and
The Project Director will be responsible for ensuring that all contractor personnel involved in handling data on the project are instructed in these procedures and will comply with these procedures throughout the study.
No questions of a sensitive nature will be included in the screener or collected during the piloting of the TKT.
The total estimated hour burden for the screening districts for eligibility for the PD Impact Study is 51 hours. For recruitment the estimated burden is 960 hours. Based on average hourly wages for participants, this amounts to $45,495 for screening and recruitment combined. Exhibit 7 summarizes the estimates of respondent burden for these two study activities. Burden estimates for other data collection activities will be included in the second OMB package. The burden estimate for eligibility screening includes time for 85 percent of the 120 district officials in the 120 candidate districts to respond to a 30-minute district screening protocol. The burden estimate for recruitment will vary greatly by the district’s persistence in the pool of potential candidates. The averaged burden estimate for recruitment of 24 hours/district includes time for all 40 viable districts to read recruitment materials and participate in a follow-up phone call and time for progressively smaller numbers of districts to participate in further follow-up calls, to host site visits, to attend a (voluntary) recruitment conference, and to negotiate final agreements.
Exhibit 7. Hour Burden for Respondents
Task |
Total Sample Size |
Estimated Response Rate |
Number of Respondents |
Time Estimate (in hours) |
Number of Admini-strations |
Total Hours |
Hourly Rate |
Estimated Monetary Cost of Burden |
District Screening |
120 |
85% |
102 |
0.5 |
1 |
51 |
$45 |
$2,295 |
Recruitment Activities |
40 |
100% |
40 |
24 |
1 |
960 |
$45 |
$43,200 |
There are no additional respondent costs associated with this data collection other than the hour burden accounted for in item 12.
The estimated cost for all aspects of the study is $10,552,543 over five years, making the annual cost to the federal government $2,110,509.
This collection is new and therefore has a program change of 1011 hours.
There will be no formal tabulations or reports based on the district screening or the pilot of the TKT. This information will be used for internal purposes only.
Findings from the Mathematics PD Impact Study will be reported to IES by AIR and MDRC in two substantive reports. The first report will document the results of the data analyses conducted on the data collected during the 2007-08 school year. This report will include a description of the study design (i.e., treatments, sample size, study sites) and employed methodology. More specificially, the descriptive analysis will include the following:
Descriptive statistics characterizing the randomly selected school districts and schools in the treatment and control groups;
Description of the treatment and control groups to address group equivalence after randomization; strategies taken to correct for remaining treatment/control group differences;
Description of dependent variables (i.e., teacher knowledge, teacher practice, and student achievement) for available time points, including baseline;
Descriptive information on the fidelity of implementation of the treatments and dosage of professional development delivered to the treatment group; and
Description of the rate of student and teacher mobility over the period following random assignment, and the characteristics of students and teachers entering and leaving the study schools.
In addition, the report will provide preliminary results regarding the effects of the treatments on the three outcome measures during the treatment year: teacher knowledge as measured in summer 2007 and in spring 2008, teacher practice over the 2007-08 school year, and student achievement over the 2007-08 school year. These analyses will be carried out using hierarchical linear modeling, to take into account the nesting of students and teachers within schools, and they will incorporate covariates measured at baseline to maximize precision. To avoid potential selection bias, the study will employ an “intent to treat” approach, in which all teachers in all randomly assigned schools during the 2007-08 school year are included in the analysis, whether or not the teachers actually participated in the intended professional development or participated to the full extent expected.
The final report will be a capstone report summarizing the entire project and its results. The main focus of the report will be the results pertaining to the effects of the treatments on the three outcomes during the year after the treatment (e.g., persistence or late appearance of effects, depending on the first-year results): teacher knowledge as measured in spring 2009, teacher practice in the 2008-09 school year, and student achievement over the 2008-09 school year. As in the first report, these analyses will be conducted using hierarchical linear modeling, incorporating covariates to maximize precision. In addition, the report will examine the possible relationships between teacher experience or prior knowledge and the impact of the interventions; the relationship between student characterstics and impact; the relationship between dosage of professional development received and impact; and the cost of the treatments. (See the Description of the Mathematics PD Impact Study for a more complete discussion of the research questions that will be addressed.) The report also will include a comprehensive analysis of teacher and student mobility and its potential effects on study results. The timeline for the dissemination of the study results is summarized in Exhibit 8.
Exhibit 8. Schedule for Dissemination of Study Results
-
Activity/Deliverable
Due Date
First Report
Summer 2009
Final Report
Summer 2010
Approval is not being requested; all data collection instruments will include the OMB expiration date.
No exceptions are requested.
As mentioned earlier, the proposed study focuses on the central research question:
What are the impacts on teacher knowledge, teacher practice, and student mathematics achievement of providing teachers with intensive professional development in the area of rational numbers?
The analysis will compare business as usual at the seventh grade in a set of school districts with a professional development treatment implemented within the same districts. Within each school district, the study will randomly assign schools to two groups: one that receives a professional development intervention consisting of a three-day summer institute, five one-day seminars held during the school year, and 10 days of intensive in-school coaching per school; and a second group that receives whatever professional development the district usually provides (business as usual).
The basic logic of our analytic strategy is to compare schools that are randomly assigned to receive an intervention with those that are randomly assigned to the group that does not. Because treatment groups are determined at the school level, the primary unit of analysis is the school. The average outcome levels in the group of schools not receiving the intervention in question represents a reliable estimate of the outcome levels that would have been observed in the absence of the program. Therefore, the difference between the average outcomes in the schools that receive the intervention and those randomly assigned to the status quo in the district represents a reliable estimate of the program’s impact.
This approach is known as an “intent to treat” analysis, in which members of the
experimental groups are compared with one another regardless of their actual participation in the intervention in question. In order to determine the effect of the intervention on mathematics achievement, we will compare average seventh grade mathematics achievement among schools randomly assigned to the treatment group with average seventh grade mathematics achievement at schools randomly assigned to business as usual, regardless of the extent to which teachers at each set of schools actually participated in the professional development activities associated with the treatment. As a result, the interpretation of these intent-to-treat estimates is that they are the effects of assigning schools to particular professional development regimes rather than the effect of teachers’ actual participation in them.
In some respects, these estimates mirror those likely to be observed in real-world settings, where PD policies are established, but where teacher participation in these activities is less than 100 percent.
In order to produce findings in two curricular contexts that are both in wide use but represent contrasting instructional approaches, the overall study consists of two parallel sub-studies, implemented in two sets of districts with contrasting curricular contexts. Within these contexts, each sub-study will include 64 schools located in 8 districts. Schools will be randomly assigned to treatment and control groups within districts. This is the most appropriate design because, even within a given curricular context, districts are likely to differ in policies and practices that might influence the ways the treatments are implemented, and in the nature and intensity of business as usual. Within each sub-study, the effects of the treatment will be estimated separately for each district and then pooled across districts to create an average effect of the treatments (meta-analysis).
In the remainder of this section, we address the screener and then separately discuss the TKT. For each of these two data collections we address the following issues: respondent universe and sampling, data collection procedures, methods to maximize response rates, pretesting procedures, and consultants.
District Screener
The Mathematics PD Impact Study will test the effectiveness of an approach to professional development for middle school mathematics instruction on rational numbers. It is not a program evaluation and will not employ random sampling of districts or schools for the sake of generalizability. Instead, districts will be screened and recruited based on characteristics required by the study design, such as the number of middle schools in the district and existing professional development initiatives. To achieve a final sample of 16 districts, AIR and MDRC will administer screening interviews across a pool of 120 districts, attempt to recruit 24-30 eligible districts, and negotiate final agreements with 16 districts. Each recruited district will allow the study team to randomly assign its middle schools to the two conditions: treatment and business as usual (control).
Identifying the Pool of Districts to be Screened
Sixteen districts will be recruited for the full study from among the pool qualifying based on demographic criteria. A review of recent CCD data (2003-2004) revealed that 120 districts have the following demographic characterististics:
The district operates at least 6 middle schools that have at least 200 seventh grade students in each school; and
The overall percentage of students eligible for free or reduced price lunch in those schools is 33 percent or more.
Screening Eligible Districts
Prepare Recruitment Materials
Materials will be drafted by the Mathematics PD Impact Study staff that indicate the goals of the study, its methods, the roles and responsibilities of the organizations involved, and the reasons to participate. This task is anticipated to be completed by August 2006, in order to have materials available before the commencement of screening interviews with potential districts and schools. Draft versions of these materials are included in Appendix B.
Identify Eligibility Through Screening Interviews
In September of 2006, an informational e-mail will be sent to each of the 120 districts identified using recent CCD data. Following the e-mail, an evaluation team member will call each district to inform them about the study and ask them to participate in a telephone interview. The screening protocol that will be used to guide this interview is presented in Appendix A, and is described below under the section entitled, Procedures for Data Collection.
The district will be considered eligible for the study if it meets all three of the following criteria:
Curriculum. One of the mathematics curricula (textbooks) of interest: either Prentice Hall Mathematics, Glencoe McGraw- Hill Mathematics: Applications and Concepts or Connected Mathematics is the primary mathematics curriculum for seventh grade in at least six middle schools.
Change in curriculum. The district has not recently changed its mathematics curriculum and does not anticipate a significant change in the next two years.
Duplicate treatment. The district is not already planning to provide professional development similar to that planned for the Mathematics PD Impact Study for seventh grade teachers.
The screening protocol is designed to allow early termination of the interview if none of the mathematics curricula of interest are in use by the district.
We anticipate that approximately 40 of the 120 districts will meet the curriculum and change in curriculum criteria. Based on preliminary conversations with publishers, we anticipate that approximately 20 of these 40 districts will use Prentice Hall Mathematics or Glencoe McGraw- Hill Mathematics: Applications and Concepts, and 20 will use Connected Mathematics. These expectations are based on the assumption, supported by our preliminary conversations with publishers, that each of our two curricula encompasses no less than one sixth of the market share among the 120 large districts.
Prioritize Districts for Recruitment
Among the 40 districts, some districts will be more appropriate candidates for the study than others. AIR and MDRC staff will use the additional information gathered in the screening interviews to prioritize eligible districts for recruitment efforts. The following criteria will be used:
Interest. Districts that signal greater interest will be given higher priority. Interviewers usually receive some signals about the district’s level of interest in the study even though the screener contains no questions about interest.
Feasibility of implementation. Districts that have fewer competing initiatives will be given higher priority. For example, districts that offer extensive coaching or professional development on another topic would receive lower priority, since the initiative would make scheduling more difficult.
Number of schools. Districts that have more schools will receive higher priority. Selecting such districts reduces the number of districts needed and therefore saves the government resources.
Geographic diversity. Some districts may receive higher priority in order to ensure geographic diversity. While not essential, geographic diversity among the districts would add to the policy relevance of the findings.
Consideration of these criteria among the 40 eligible districts will allow AIR and MDRC to prioritize each list of 20 districts before beginning recruitment efforts in October, 2006.
Recruiting the Eligible Districts
Past experience indicates that site visits to districts are necessary to ensure eligibility and reach final agreement on participation. To initiate recruitment efforts, the evaluation team will send brochures and question and answer guides (see Appendix B for sample recruitment materials) to officials in eligible districts, beginning with those determined to be of the highest priority. Senior evaluation team members will follow up with these districts by telephone in order to do the following:
Determine which district officials must be involved in a decision to participate;
Communicate the specific benefits of participating in the study;
Describe the ways in which the evaluation team will minimize the burden of participation; and
Determine whether the district is sufficiently interested and, if so, offer to visit the district to further discuss participation.
We expect over half of the 20 districts in each group to express sufficient interest given the benefits of participation. Senior staff from AIR and MDRC will visit these 12-15 interested districts in each group, for a total of 24 to 30 district site visits. The site visits will allow us to provide an in-person presentation, discuss the benefits and responsibilities of participation with additional district officials, and respond to any questions and concerns that they might have.
During the recruitment visit, study staff will work with the district to identify schools that meet demographic criteria (have at least 33 percent of students eligible for free or reduced price lunch), are believed to be implementing one of the mathematics curricula of interest, and have at least three seventh grade mathematics teachers. To maximize the number of schools that agree to participate, AIR and MDRC will ask district leadership to express support for the study and to set the expectation that the qualifying schools participate in the study. If requested, AIR and MDRC will also conduct presentations to educate principals about the study and its requirements.
Study staff will identify several incentives to participation as part of the recruitment process. While districts will not receive cash incentives to participate in the study, schools in the treatment group will receive the professional development at no cost, and payments will be made to cover non-contract professional development days and substitute pay for participating teachers in the treatment group. Schools in the control group will each receive a $1,000 incentive for participating.
Finally, in January 2007, AIR and MDRC will host a conference for the top 20 prospects—10 from each group—to sample the professional development and coaching, to discuss the content of the project in detail, and to review random assignment and research related responsibilities. Two staff per district will be invited to this conference. The study will pay for their travel expenses but will not provide any additional incentive. Attendance will not be mandatory, and districts can still participate in the study if they decline to participate in the conference. However, similar conferences have been used quite effectively in other random assignment studies to deepen district understanding about the study, answer questions, and solidify support for the evaluation. Such conferences have also proven to be efficient since they reduce the amount of staff time needed to make visits to districts and reduce overall recruitment costs.
Negotiating Final Agreements
Shortly after the conference, district administrators will be asked to reach a final agreement to participate, and a memorandum of understanding with interested districts will be prepared. As a condition of participation, districts must also ask middle school principals to submit signed statements reflecting an intention to participate. If necessary, project staff will make additional visits to build consensus and obtain commitment from principals or other affected parties. The school agreements are expected to be completed shortly after the district memorandum of understanding in order to maintain momentum and allow random assignment prior to the end of the 2006-07 school year.
The district screener instrument is included in this package in Appendix A. The items on the screener are mapped to the constructs that they measure in the exhibit below. The screener will be administered by project staff via telephone interviews with district administrators, as described above.
Exhibit 9. Mapping of District Screener Items to Constructs
Instrument |
Grade 7 Mathematics Course Offerings |
Use of Curriculum/ Textbook |
Likelihood of Change in Curriculum/ Textbook |
Intensity of Existing Professional Development Institutes, Seminars, Workshops |
Intensity of Existing Coaching |
District Screener |
1 |
2,3,4,5 |
6,7 |
8,9,10,11,12,13,14,15,16,17 |
18,19,20,21,22, 23,24,25 |
The anticipated response rate is approximately 85 percent for the district screeners. These estimates are based on the previous experience of study staff in conducting eligibility screening. The following procedures will be used to ensure high response rates:
Obtaining high response rates depends in part on the quality of the instruments. The screener will be pretested to ensure that the questions are clear and as simple as possible for respondents to complete.
The study will offer a social incentive to respondents by stressing the importance of the data collections as part of a high-profile study that will provide much-needed information to districts and schools.
The district screener is being pretested with a small sample of respondents (fewer than 10) for two purposes—to ensure that the instrument and procedures work effectively, and to reinforce estimates of the respondent burden required by the full study screening and recruitment.
Estimating Burden
Similar research conducted by study staff was used for an initial estimate of burden. Minor changes will made as a result of a small pretesting of the screener (see below), and the burden estimates will be updated accordingly.
Pretesting Screener and Procedures
To test the effectiveness and clarity of the screeners, they are being pretested as part of the screening and recruitment of districts for the pilot test of the intervention. Four districts are being recruited for this pilot.
Determining Methods for Data Collection
The PD Impact Study staff gave consideration to two modes of data collection for screening: telephone interviews and mail surveys. Mail surveys would have been considered more seriously if there were larger numbers of districts to screen. However, telephone surveys are considered less burdensome to respondents, and produce data more rapidly. Given the timeline of the study and the need for a high response rate, the decision was made to collect screening data via telephone interviews.
This project is being conducted under contract to the Department of Education by AIR and MDRC. The district screener was developed by Michael Garet, Andrew Wayne, Fran Stancavage, James Taylor and Helen Duffy of AIR, and Fred Doolittle and Robert Ivry of MDRC. Data collection will be carried out by project staff at MDRC and AIR.
Teacher Knowledge Test (TKT)
The pilot of the TKT will take place during the summer and early fall of 2006, the year prior to the full study. In order to avoid eliminating districts from eligibility for the full study by revealing the TKT items to teachers in those districts, we will seek out districts that just missed the cutoffs for eligibility into the full study. For instance, we will focus our efforts in similar large, urban districts that have fewer large-enrollment, high-poverty middle schools than required to be eligible for the full study or that base their mathematics programs on similar, but not identical textbooks to those targeted for the study . We will attempt to pilot in 10 or more districts that are geographically diverse.
In these districts, we will work with district and external professional development providers to arrange for teachers to take the pilot version of the TKT at the beginning or end of a professional development activity. Our focus will be on large professional development activities for middle school mathematics teachers. Although priority will be given to seventh grade professional development that involves rational numbers, the content of the activities need not be restricted to rational numbers and the middle school mathematics teachers need not be only seventh grade teachers.
The three forms of the TKT for the full study require a total of 90 items. We will pilot 120 items with the expectation that due to the complexity of writing such items, at least 25 percent will not meet our quality standards. Given the pool of 120 candidate items, we will construct several 60-item pilot forms that contain different combinations of the 120 candidate items, and we will administer these forms to a total of 500 teachers using a design that insures 250 potential respondents for each item. The 500 teachers will comprise approximately 50 mathematics teachers from each of the 10 or more large, urban districts recruited for the TKT pilot. Assuming an 80 percent response rate, we will administer each TKT pilot test item to 250 teachers to achieve the needed 200 teacher responses per item. The results will allow us to calculate preliminary item characteristics such as item difficulties and biserial correlations. The item characteristics will be used to identify items that are too hard or too easy, identify other poor performing items, and help balance the three parallel final forms of the TKT.
The TKT will cover the major content categories in rational numbers (i.e., number sense, computation and estimation, and problem solving involving fractions, decimals, percents, and proportional reasoning) that current policy documents identify as important for middle school mathematics teachers (Milgram, 2004; National Research Council [NRC], 2001), and which are the major topics to be covered in the professional development provided by the Mathematics PD Impact study. The TKT also will cover what research defines as the major domains of foundational and pedagogical content knowledge that teachers need to teach mathematics (Ball et al. 2003; Hill et al. 2004). Items will not require teachers to recall esoteric details that could be known only by participants in the study’s professional development intervention (e.g., the specific jargon or organizing frameworks used in the training curriculum).
Each form of the TKT for the full study will consist of 30 items and will be administered in 30 minutes or less. Each of the pilot versions of the TKT will contain 60 items and will be expected to take 60 minutes to administer. The use of these double-length forms allows us to reduce the total number of teachers in the pilot by half. Test administration will be managed and proctored by our subcontractor, REDA International.
To promote participation and encourage completion of the pilot of the TKT, we will stress the social value of assisting with the study to develop a better understanding of what teachers know. We will also ask the districts and providers of the professional development sessions serving as the setting for the test administration to stress the importance of participation. We also will provide an incentive of up to $75, as discussed above under A.9.
Where appropriate, items for the test will be drawn from existing tests of teacher knowledge. The advantages of these items are that they have been developed by respected experts in the field of mathematics and the teaching of mathematics, they have been previously piloted with large numbers of teachers, and most have existing psychometric information. Unfortunately, these existing tests contain only a limited number of items that deal specifically with rational numbers. With the help of one of the developers of the Test of Content Knowledge for Teaching Mathematics, Mark Thames, who serves as a consultant to this study, and in-house mathematics test development experts, AIR will write additional items to augment the item pool, but will need to pretest and pilot these items to ensure that they are of high quality.
The test development process will involve several steps. First, item writers will produce draft items based on information from published sources on scientifically based mathematics research and practice. The draft items will go through multiple rounds of review and revision, and then will be tested in cognitive interviews with up to nine seventh grade teachers. Cognitive interviews provide estimates of item difficulty, and allow identification of problems still in need of amelioration (e.g., ambiguities in wording, the use of unnecessarily technical language, or potentially offensive language or situations depicted in items). Information from the cognitive interviews will result in additional revisions and culling of items. Finally, the items will be reviewed by members of the study’s Technical Work Group and organized into test forms to be used in the pilot.
This pilot will provide technical information on the quality of the items administered. This information includes whether many teachers did not complete the item, the proportion of teachers answering correct, and the identification of items that were otherwise problematic (e.g., a distractor response that is too attractive). This information will be used to delete problematic items from the pool, adjust the anticipated difficulty level of the full study test to the tested population (i.e., ensure it is not too easy or too hard), and roughly balance the difficulty of the parallel forms for the full study administration. Because it is not feasible to obtain a true random sample of the population of interest, these item parameters will only be approximate. However, they are expected to be sufficiently close to the operational parameters to provide good guidance for form construction. The pilot also will help gauge the willingness of teachers to participate, uncover weaknesses in test procedures, and ensure that the test can be expected to reliably measure and distinguish between levels of knowledge.
This project is being conducted under contract to the Department of Education by AIR and MDRC. The pilot version of the Teacher Knowledge Test will be developed by Michael Garet, Andrew Wayne, Fran Stancavage, James Taylor, and others at AIR. Mark Thames of the Learning Mathematics for Teaching Project at the University of Michigan will serve as a consultant to the test development effort. Data collection for the pilot of the TKT will be carried out by REDA International. AIR will have overall responsibility for managing data collection and ensuring quality, coordination, and timeliness.
Ball, D. L. (1996). Teacher learning and the mathematics reforms: What we think we know and what we need to learn. Phi Delta Kappan, 77, 500-08.
Ball, D., Rowan, B and Hill, H. (2003) Effects of teachers' mathematical knowledge for teaching on student achievement. Paper presentation at the annual meeting of the American Educational Research Association.
Bloom, H.S. (2003) Sample Design for An Evaluation of the Reading First Program. New York: MDRC.
Bos, C. S., Mather, N., Narr, R. E., & Babur, N. (1999). Interactive, collaborative professional development in early literacy instruction: Supporting the balancing act. Learning Disabilities Research and Practice, 14, 227-238.
Carpenter, T.P., Fennema, E., Peterson, P.L., Chiang, C., & Loef, M. (1989). Using knowledge of children’s mathematics thinking in classroom teaching: An experimental study. American Educational Research Journal. 26, 499-531.
Cohen, D.K. & Hill, H.C. (2001). Learning policy: When state education reform works. New Haven: Yale University Press.
Cohen, D. K., & Hill, H. C. (1998). Instructional policy and classroom performance: The mathematics reform in California (RR-39). Philadelphia: Consortium for Policy Research in Education.
Cohen, Jacob. 1988. Statistical Power Analysis for the Behavioral Sciences. Second Edition. Hillsdale, NJ: Lawrence Erlbaum.
Education Commission of the States (2003). Eight questions on teacher preparation: What does the research say? Denver, CO: Author.
Elmore, R. (2002). Bridging the gap between standards and achievement: The imperative for professional development in education. [Online.] Available: http://www.ashankerinst.org/Downloads/Bridging_Gap.pdf.
Garet, M, Porter, A, Desimone, L, Birman, B., & Yoon, K. (2001). What makes professional development effective? Results from a national sample of teachers. American Educational Research Journal, 38, 915–945.
Garet, M., Birman, B. F., Porter, A. C., Desimone, L., Herman, R., & Yoon, K. S. (1999). Designing effective professional development: Lessons from the Eisenhower program. Washington, DC: American Institutes for Research.
Gearhart, M., Saxe, G.B., Seltzer, M., Schlackman, J., Ching,C.C., Nasir, N., Fall, R., Bennett, T., Rhine, S., & Sloan, T.F. (1999). Opportunities to learn fractions in elementary mathematics classrooms. Journal for research in mathematics education. 30, 286-315.
Grant, S. G., Peterson, P. L., & Shojgreen-Downer, A. (1996). Learning to teach mathematics in the context of systemic reform. American Educational Research Journal, 33, 502-541.
Hargreaves, A. and Fullan, M. G. (1992). Understanding teacher development. London: Cassell.
Hill, H., Schilling, S., & Ball, D. (2004). Developing measures of teachers' mathematics knowledge for teaching. Elementary School Journal, 105, 11-30.
Kennedy, M. (1998). Form and substance in inservice teacher education. Research Monograph No. 13. National Institute for Science Education. Madison, WI: University of Wisconsin.
Knapp, M. S. (1997). Between systemic reforms and the mathematics and science classroom: The dynamics of innovation, implementation, and professional learning. Review of Educational Research, 67, 227-266.
Lieberman, A. (1996). Practices that support teacher development: Transforming conceptions of professional learning. In M. W. McLaughlin & I. Oberman (Eds.), Teacher learning: New policies, new practices (pp. 185-201). New York: Teachers College Press.
Lieberman, A., & McLaughlin, M. W. (1992). Networks for educational change: Powerful and problematic. Phi Delta Kappan, 73, 673-677.
Lipsey, Mark, 1990. Design Sensitivity: Statistical Power for Experimental Research. Newbury Park, CA: Sage Publications, Inc.
Little, J. W. (1993). Teachers’ professional development in a climate of educational reform. Educational Revaluation and Policy Analysis, 15, 129-151.
Loucks-Horsley, S., Hewson, P. W., Love, N., & Stiles, K. E. (1998). Designing professional development for teachers of science and mathematics. Thousand Oaks, CA: Corwin Press, Inc.
McCutchen, D., Abbott, R. D., Green, L. B., Beretvas, S. N., Cox, S., Potter, N. S., Quiroga, T., & Gray, A. (2002). Beginning literacy: Links among teacher knowledge, teacher practice, and student learning. Journal of Learning Disabilities, 35, 69-86.
Milgram, J.R. (2004).The mathematics that pre-service teachers need to know. Department of Mathematics. Stanford, CA: Stanford University.
National Research Council. (2001). Adding it up: Helping children learn mathematics. J. Kilpatrick, J. Swafford, and B. Findell (Eds). Mathematics Learning Study Committee, Center for Education, Division of Behavioral and Social Sciences and Education. Washington, DC: National Academy Press.
O’Connor, R. E. (1999). Teachers learning Ladders to Literacy. Learning Disabilities Research and Practice, 14, 203-214.
Rowan, B., Correnti, R., & Miller, J. (2002). What large-scale survey research tells us about teacher effects on student achievement: Insights for the Prospects study of elementary schools (CPRE Research Report Series RR-051). Philadelphia: Consortium for Policy Research in Education.
Sanders, W. and Rivers, J. (1996). Cumulative and residual effects of teachers on future academic achievement. Knoxville, Tenn.: University of Tennessee Value-Added Research and Assessment Center.
Silver, E.A. & Kenny, P.A. (2000). Results from the seventh mathematics assessment of the National Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics.
Stiles, K., Loucks-Horsley, S., & Hewson, P. (1996). Principles of effective professional development for mathematics and science education: A synthesis of standards (NISE Brief, Vol. 1). Madison, WI: National Institutes for Science Education.
Talbert, J. E., & McLaughlin, M. W. (1993). Understanding teaching in context. In D. K. Cohen, M. W. McLaughlin, & J. E. Talbert (Eds.), Teaching for understanding: Challenges for policy and practice (pp. 167-206). San Francisco: Jossey-Bass, Inc.
Tirosh, D., Fischbein, E., Graeber, A.O., & Wilson, J.W. (1999) Prospective elementary teachers’ conceptions of rational numbers. Available: http://jwilson.coe.uga.edu/Texts.Folder/tirosh/Pros.El.Tchrs.html.
Whitehurst, G. J. (2002, Summer). Improving teacher quality. Spectrum: The Journal of State Government, 12-15.
Appendix
A
District Level Screening Protocol
Mathematics Professional Development Impact Study
District Level Screening Protocol
Paperwork Burden Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is XXXX–XXXX. The time required to complete this information collection is estimated to average 30 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202–4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Elizabeth Warner, U.S. Department of Education, Institute of Education Sciences, Room 308D, 555 New Jersey Avenue, NW, Washington, DC, Washington, D.C. 20202–0001.
Advance Letter
Dear [District Official],
I am writing to introduce you to a new Department of Education funded professional development project aimed at improving math instruction in the 7th grade, called the Mathematics Professional Development Impact Study. AIR and MDRC are undertaking this project, which will provide intensive professional development and coaching to 7th grade math teachers focused on the teaching and learning of rational number skills, including: fractions, decimals, percents, ratios, and proportions.
I would like to schedule a brief phone call with you to tell you more about the study and to learn more about the 7th grade math program in your district: what math courses are taught in the 7th grade, what math text books are used, how math instruction is structured in the 7th grade (for example, do middle school math teachers typically get assigned a single grade), and what professional development 7th grade math teachers currently receive. Please let me know if there is a particular day or time that would be best for you.
Thank you for your time, and I look forward to speaking with you.
Regards,
District Level Screening Protocol
Background for interviewers
To qualify for the Mathematics Professional Development Impact Study, districts must meet the following criteria:
At least 6 schools that include grade 7 (middle schools, junior high schools, or K-8 schools)
At least 200 students in grade 7
At least 33 percent students in poverty (based on reported eligibility for school lunch program)
Widespread use one of the following textbooks
Prentice Hall Mathematics
Glencoe McGraw- Hill Mathematics: Applications and Concepts
Connected Mathematics
Textbook stability (Textbook will not be in its first year of use as of the year of the PD intervention--2007-2008 school year for main study--and will continue to be used for at least one year following
Sufficient contrast between normal PD for grade 7 teachers and the planned study intervention
Through an initial screening process using recent (2003-2004) CCD statistics, we have identified districts that meet criteria 1 through 3. The protocol described here will be used to screen for criteria 4 through 6.
Protocol
The call should be made to the district person responsible for curriculum and instruction in middle grades mathematics or the person responsible for coordinating professional development activities involving middle grade mathematics teachers. These individuals may be identified on the district’s website under instruction/curriculum, mathematics, professional development etc. You may need to call the district’s general number, however, and ask for the best person to talk to about mathematics instruction or professional development. If the person you reach does not agree that he or she is the right person to provide the necessary information, ask them to recommend someone.
Good morning {or afternoon}, I am _____, with [MDRC/the American Institutes for Research], a research firm in [New York/Oakland/Washington, DC]. We are doing preliminary work for a U. S. Department of Education research project on effective professional development. The project is focused on professional development for seventh grade mathematics teachers. As a first step we are contacting a number of large districts to learn more about their middle school mathematics programs, including the textbooks that are in use and the professional development that is being offered. Do you have time to talk now? My questions will take about a half hour.
1. Seventh Grade Mathematics Texts Used by District
First, I would like to ask questions about the mathematics courses that are offered in your district at grade 7 and the textbooks that are used for these courses.
What specific mathematics courses are offered at grade 7? (Mark all that apply)
General math/grade 7 math
Pre-algebra
Remedial math
Other, specify______________________________
Other, specify_______________________________
Which of these statements best describes how seventh grade mathematics textbooks are chosen by teachers in your distict? (Read choices)
There is a single mandated textbook for each mathematics course
Schools or teachers can select from a short list of authorized textbooks for each course
Schools or teachers can select from a wide range of textbooks
[if mandated or short-listed] Does the district require that teachers use these textbooks more or less as published? Or may teachers make modifications to the textbooks’ curriculum in order to fit district needs?
As published
Teachers make modifications, but the modifications are uniform across the district [Probe for whether the modifications are minor or major]
Teachers make modifications according to preferences of school/teacher [Probe for whether the modifications are minor or major]
Other, specify___________________________________
I am going to read you the names of three textbooks. For each of these, please tell me whether it is used in your district. If it is used, I will ask you a few more questions about how it is used.
[If none of the three study textbooks is used to a significant extent3] What are the primary textbooks used in your district for seventh grade math?
For each textbook identified in Qs 4 and 5, probe as necessary to answer the following questions:
What specific edition is used? (if more than one edition in use, fill in matrix for each edition)
Is this a mandated/short-listed textbook?
If yes, when was it adopted?
How many schools in your district use this textbook?
For which mathematics courses is this textbook used?
When it is used, is this textbook used in a relatively uniform manner across the district?
Text |
Edition |
Used? |
Man-dated? |
Adoption Date |
No of schools |
Which Courses? |
Uniform use? |
Prentice Hall Mathematics
|
Xxx |
|
|
|
|
|
|
Prentice Hall Mathematics
|
Xxx |
|
|
|
|
|
|
Glencoe McGraw- Hill Mathematics: Applications and Concepts
|
Xxx |
|
|
|
|
|
|
Glencoe McGraw- Hill Mathematics: Applications and Concepts
|
Xxx |
|
|
|
|
|
|
Connected Mathematics
|
Xxx |
|
|
|
|
|
|
Connected Mathematics
|
Xxx |
|
|
|
|
|
|
Other text 1, specify
|
|
|
|
|
|
|
|
Other text 2, specify
|
|
|
|
|
|
|
|
Other text 3, specify
|
|
|
|
|
|
|
|
If the respondent has not indicated that at least six schools use one of the four mathematics texts of interest to the study and has not identified a closely related text as their primary text, thank the respondent and end the interview.
When is the next time that you expect to begin using different grade 7 mathematics texts for your district?
2006-2007 school year
2007-2008 school year
2008-2009 school year
2009-2010 school year or later
Don’t know
What kind of change do you anticipate?
Major new adoption
Other change, describe_______________________
2. Professional Development Targeting Seventh Grade Mathematics Teachers
Next I would like to ask questions about professional development related to mathematics that is offered to seventh grade teachers in your district during the current school year 2006-07 including this past summer of 2006.
During the current school year, are there mandated professional development hours that are being used for mathematics professional development? (Include hours over the summer, before the start of classes.)
Yes → How many hours are mandated? _______________
No
Whether or not the hours are mandated, is there any specific mathematics-related professional development this year in which at least three-quarters of the district’s seventh grade mathematics teachers participate? (Include PD offered over the summer, before the start of classes.)
Yes → Skip to Q.11
No
If it isn’t a case of everyone doing the same thing, can you tell me what is available this year with regard to mathematics-related professional development and how the teachers are distributed across these choices? (Include PD offered over the summer, before the start of classes.)
[Record response. If answer indicates that there really is some form of PD that applies to at least ¾ of the seventh grade mathematics teachers (for example, everyone, takes a 3-day summer institute but chooses between 2 or 3 possible institutes with different mathematics topics), continue with Qs 11-15. Otherwise, skip to Q 17.]
Who is delivering that professional development (the professional development that at least ¾ of seventh grade mathematics teachers attend)? (Mark all that apply)
District personnel
Textbook publisher
Other company that offers PD, specify_____________________
Outside mathematics consultants other than above
Did it involve a workshop or seminar offered over the summer?
Yes → How many hours? _______________
No
Does it involve workshops or seminars offered during the school year?
Yes → How many hours? _______________
How are these hours distributed?__________________
No
Are there other components to the professional development besides workshops or seminars?
Yes → What components?_____________________________________
How many hours? _______________
No
What is the topical focus of the professional development?
In terms of format and numbers of hours, is this typical of the professional development you would expect seventh grade mathematics teachers in your district to attend over the next few years?
Yes
No → How does it differ? What do you expect over the next few years?
Are there specific professional development requirements for seventh grade mathematics teachers at schools that fail to meet AYP goals or are otherwise identified as failing?
Yes → What requirements?
Roughly how many seventh grade mathematics teachers were affected this year? _______________
No
In this district, are there mathematics specialists or coaches who spend at least half of their time working with middle school teachers to help improve mathematics instruction? For example, these could be. (Mark all that apply)
Mathematics specialists or coaches assigned half time or full time to specific middle schools → Answer Qs 19-21
District level mathematics specialists or coaches who circulate to all or most middle schools → Answer Qs 22-25
No dedicated mathematics specialists or coaches meeting criteria → skip to end of interview
[If the respondent says something like, “We don’t have a specialists or coaches, but many schools have teacher leaders, mentors and department chairs” note this response and treat this as a “No” response by skipping to the end of the interview.]
Staff assigned to specific schools:
During the current school year, how many middle schools have a full time mathematics specialist or coach? ______________
How many have a half time mathematics specialist or coach? _________
Do you expect these designated school positions to be continued at this level over the next few years?
Yes
No → Explain what will change.
Circulating district level staff:
How many district staff work full time with middle school mathematics teachers in this coaching/support capacity?____________
How many (other) district staff work at least half time with middle school mathematics teachers in this coaching/support capacity?________
Do they divide their time equally across all middle school mathematics teachers in the district? If not, describe where they place their emphasis.
Do you expect these district-wide positions to be continued at this level over the next few years?
Yes
No → Explain what will change.
Thank you, this is all the information we need now. If we have more questions in the future, will it be ok for me to call again? In the meantime, if you have any questions for me, you can reach me at… Thank you for your time.
Appendix
B
SAMPLE RECRUITMENT MATERIALS
one-page STUDY overview
Up to 16 school districts throughout the country will have an opportunity to participate at no cost in a state
of the art professional development and coaching project aimed at helping seventh grade students master
rational numbers topics: fractions, decimals, ratios, proportions and percents. The project, known as
the Mathematics Professional Development Impact Study (or Math PD Impact Study for short),
focuses on improving the content knowledge and instructional practice of seventh grade mathematics
teachers. Participating districts will also be involved in an evaluation to measure the effectiveness of the
professional development and coaching on both teachers and student achievement. The Math PD Impact
Study is funded by the U.S. Department of Education.
Benefits of Participation
Qualifying districts will receive:
• High-quality Professional Development (PD) for participating seventh grade mathematics teachers
provided at no cost, including:
o One, three-day summer institute
o Five, day-long seminars delivered during the school year
o Five, 2-day coaching sessions delivered during the school year
• Compensation for teachers during their professional development activities (e.g. stipends and/or
reimbursement for the cost of substitutes) paid for by the study.
• Recognition for involvement and the opportunity to contribute to the broader knowledge and policy
development about PD and coaching for middle-school mathematics teachers.
About The Professional Development Activities
The PD activities, which will take place during the summer of 2007 and during the 2007-2008 school year,
will focus on increasing seventh grade mathematics teachers’ content knowledge and pedagogical
knowledge of rational number topics. They will also support teachers’ use of this knowledge in instruction
through exercises and activities that involve practice, role-play and discussion. The PD and coaching will
be delivered by one of two well-respected staff development providers for mathematics teachers that
were selected through a highly competitive application process – either America’s Choice or Pearson
Achievement Solutions (formerly LessonLab & Co-nect). The PD activities will be designed to
complement your district’s seventh grade mathematics textbook and pacing guide.
Criteria for Participation
Based on our research, it is likely that several schools in your district are eligible to participate. To be
eligible, schools should:
• Have at least one third of students enrolled in the National School Lunch Program.
• Use Prentice Hall’s Connected Mathematics Project, Prentice Hall’s Mathematics or Glencoe’s
Applications and Concepts as the primary text for seventh grade math and have used it for at least
one year.
• Have at least two teachers teaching regular sections of seventh grade math.
For Additional Information
The Math PD Impact Study is being conducted by the American Institutes for Research (AIR), MDRC, and
REDA International. For additional information, please contact:
• Robert Ivry, MDRC (www.mdrc.org), (212) 340-8672, robert.ivry@mdrc.org
• Andrew Wayne, AIR (www.air.org), (202) 403-5483, awayne@air.org
Questions and Answers Document
What is the Math PD Study?
The Mathematics Professional Development Impact Study is a national, high-profile project designed to test the effectiveness of professional development and coaching on teacher knowledge, instruction and student achievement. This pioneering project focuses on seventh grade mathematics and delivers professional development and coaching designed to enhance teaching of rational numbers topics—fractions, decimals, ratios, proportions and percents—which are critical for student success in the second half of the K-12 mathematics curriculum. This project is sponsored by the U.S. Department of Education and is being conducted by the American Institutes for Research (AIR), MDRC, and REDA International.
The research team is seeking all eligible schools in up to 16 districts nationwide to take part in the study during the 2007-2008 school year.
What are the benefits of the study?
Qualifying Districts will receive:
High-quality Professional Development (PD) for participating seventh-grade mathematics teachers provided at no cost, including:
A Summer Institute: A three-day, locally-delivered, summer institute designed to strengthen teachers’ content knowledge and pedagogical content knowledge of rational number skills and concepts in seventh grade.
Teacher Seminars: Five, day-long, locally-delivered, teacher seminars delivered throughout the school year to sustain and extend teachers’ knowledge of rational numbers and the teaching of rational numbers.
Intensive Coaching: A two-day, site-based coaching session delivered shortly after each of the five seminars (10 days total) designed to help teachers use the knowledge they have gained from the institutes and seminars to improve their instruction of rational numbers topics.
Recognition for involvement and the opportunity to contribute to the broader knowledge and policy development about PD and coaching for middle-school mathematics teachers.
What are the goals of the Professional Development?
The institutes, seminars and coaching are designed to meet two goals, which contribute to the primary goal of raising student achievement.
Goal 1: To strengthen teachers’ content knowledge in the domain of rational numbers. The professional development and coaching will focus on activities such as: defining the rationale for key concepts; interpreting fractions, decimals and percents; converting between decimals, percents and fractions; conducting arithmetic of rational numbers; comparing and ordering rational numbers; and solving applied problems involving rational numbers, especially problems involving ratios, proportion and rates.
Goal 2: To strengthen teachers’ pedagogical knowledge—that is their knowledge of ways to represent and explain rational numbers content and to gauge student understanding. The professional development and coaching will build teachers’ knowledge of students’ mathematical thinking (common conceptions, misconceptions and developmental issues), as well as improve their ability to determine if students’ proposed solutions are valid. It will also extend teachers’ knowledge of the strengths and limitations of representations of key concepts (e.g., using pies, number lines or sets/subsets to represent fractions), as well as their knowledge of ways of explaining and connecting rational numbers concepts.
How is the Professional Development designed and who will deliver it?
The PD activities will support teachers’ use of content and pedagogical content knowledge in instruction through exercises and activities that involve practice, role-play, and discussion. They will be applied to the seventh grade mathematics textbook and curriculum schools are already using – either Connected Math, Prentice Hall Mathematics or Glencoe’s Applications and Concepts - and will be designed to compliment your districts’ seventh grade mathematics textbook and pacing guide. All PD activities will be delivered by one of two well-respected staff development providers for mathematics teachers that were selected through a highly competitive application process - America’s Choice or Pearson Achievement Solutions (formerly LessonLab & Co-nect).
Are the schools in my district eligible to participate?
Based on our research, it is likely that several schools in your district are eligible to participate. To be eligible, schools should:
Have at least one third of students enrolled in the National School Lunch Program.
Use Prentice Hall’s Connected Mathematics Project, Prentice Hall’s Mathematics or Glencoe’s Applications and Concepts as the primary text for seventh grade math and have used it for at least one year.
Have at least two teachers teaching regular sections of seventh grade math.
What are the responsibilities of the participants?
The research team, in consultation with the U.S. Department of Education and experts in the field of education policy research, has designed this study to examine the impact of professional development and coaching on seventh grade mathematics instruction and student achievement. The study combines scientific rigor with an understanding of how schools operate and what schools are currently doing to improve mathematics instruction and performance. The following are some of the responsibilities we will ask districts and schools to assume.
Participating districts and schools would agree to make a “good faith effort” to support the professional development and coaching activities and to strongly encourage or require seventh-grade mathematics teachers to participate.
Principals and mathematics department chairs at participating schools would be willing to consider attending the summer institute and follow-up seminars.
Participating districts and schools would agree to pre-arranged observations of professional development and coaching activities, coaching sessions and classroom instruction.
Participating districts and schools would agree to participate in data collection activities, including surveys and student assessments.
What are the responsibilities of the research team?
The research team will also assume responsibilities designed to protect the participants and to ensure that the PD activities are of the highest quality possible. These responsibilities include:
Providing ongoing monitoring and support to ensure that the PD is implemented properly.
Assigning districts with a site coordinator that will be the point of contact for study related activities.
Undergoing thorough IRB review related to data collection activities.
Assuring the confidentiality of participants involved in data collection activities.
How will the study be designed?
The Institute of Education Sciences at the U.S. Department of Education convened an expert panel of researchers to seek advice about how best to measure the impacts of the professional development. The advisory panel concurred that the most rigorous method is to use a lottery, in which either teachers or schools are randomly chosen to receive or not receive the professional development. With this method, teachers’ classrooms or schools start out as truly comparable, on average, because teachers or schools are designated as “research schools” at random. Thus, subsequent differences between those participating in the study’s professional development and those who participate in their usual professional development activities can be attributed confidently to the professional development model.
Lotteries have been used widely in educational and other public service settings, including mandatory and voluntary job education, job training, welfare, and health-related programs. Also, both MDRC and AIR are currently conducting several education studies that involve lotteries. Lotteries, however, are only feasible and appropriate under certain conditions:
There must be an important question about policy or program impacts.
The demand or eligibility for resources, services, or program slots must exceed their availability.
Participants must be informed about the study, including the random assignment process.
The potential benefits of the study must outweigh the potential burden to participants.
Under these conditions, a lottery is a fair method of allocating scarce resources and a valid method for assessing the impact of those resources on the participants.
What do participants in other studies say?
Members of the research team have a considerable history of working with districts and schools in similar research studies, including another ongoing U.S. Department of Education funded professional development study aimed at improving student achievement in reading. The feedback provided by participants in the Reading PD study has been positive.
“I have never found any workshops dealing with literacy to be as useful as this one. I am very excited to continue working with the team.”
“I loved the role-playing activity at the end and I will definitely use this to introduce spelling rules to my class!”
“The vowel chart and consonant phoneme chart helped me better understand student errors.”
“I can now see the connection between analyzing spelling and how it relates to reading and fluency.”
“I feel all primary teachers should get this training!”
Who is conducting the study?
The study is sponsored by the U.S. Department of Education’s Institute of Education Sciences and is being conducted by:
The American Institutes for Research (AIR) (www.air.org), a not-for-profit organization in Washington, DC with a 50-year history of conducting applied research and providing technical assistance in the areas of education, health, workforce and international development.
MDRC (www.mdrc.org), a non-profit, non-partisan research organization in New York City experienced in conducting rigorous studies aimed at enhancing the effectiveness of education and other social policies and programs.
REDA International, Inc. (www.redainternational.com), a professional services firm specializing in social science research, program evaluation, and developmental services that is located in the Washington, DC area.
For more information, please contact:
Robert Ivry, MDRC, (212) 340-8672, robert.ivry@mdrc.org
Andrew Wayne, AIR, (202) 403-5483, awayne@air.org
credentials of the research team document (sent only upon request)
THE RESEARCH ORGANIZATIONS AND TEAM MEMBERS
Purpose:
The Mathematics Professional Development Impact Study gives seventh grade mathematics teachers
professional development and coaching designed to enhance their teaching of rational numbers topics –
fractions, decimals, ratios, proportions and percents – that are critical for student success in the second
half of the K-12 mathematics curriculum. The purpose of the study is to test the effect of this professional
development and coaching on teacher knowledge, instruction and student achievement.
Key Organizations:
The study is sponsored by the U.S. Department of Education’s Institute of Education Sciences and is
conducted by:
• The American Institutes for Research (AIR) (www.air.org), a non-profit organization in
Washington, DC with a 50-year history of conducting applied research and providing technical
assistance in the areas of education, health, workforce and international development.
• MDRC (www.mdrc.org), a non-profit, non-partisan research organization in New York City
experienced in conducting rigorous studies aimed at enhancing the effectiveness of education and
other social policies and programs.
• REDA International, Inc. (www.redainternational.com), a professional services firm specializing in
social science research, program evaluation, and developmental services that is located in the
Washington, DC area.
Senior Team Members:
• Dr. Fred Doolittle, co-project director, is a Vice-President at MDRC and has 20 years experience
leading large, multi-site randomized field trials. He is currently the co-project director of the PD
Impact-Reading study; project director of the Enhanced Instruction in After-School Programs
project, which involves professional development in mathematics instruction; and a research
director of Project GRAD, which involves training of teachers in mathematics instruction. Dr.
Doolittle holds a law degree and a Ph.D. in economics from the University of California at Berkeley.
• Dr. Michael Garet, co-project director, is a Chief Research Scientist at AIR. He has led several
large-scale research projects in schools focusing especially on professional development. He
directed an NSF-supported longitudinal study of the effects of Math Science Partnership (MSP)
professional development on improving mathematics and science instruction in middle schools and
is currently the co-project director of the PD Impact-Reading study. Dr. Garet also currently serves
as the co-PI for the National Longitudinal Study of No Child Left Behind. He holds a Ph.D. in
Management from M.I.T. and a B.A. in Mathematics from the California Institute of Technology.
• Robert Ivry, Task Leader for Recruitment, is Senior Vice President for Development and External
Affairs at MDRC where he has worked for 25 years. Mr. Ivry has played a lead role identifying
states, localities, and school districts for MDRC’s demonstrations and evaluations; negotiating the
terms of the evaluation and building a consensus of support for the study with stakeholders; and
providing technical assistance on program design and implementation issues. In particular, Mr. Ivry
has led the recruitment of hundreds of sites for participation in random assignment studies focusing
on K–12 education, postsecondary education, welfare, and workforce development. Mr. Ivry
received both a B.A. and a Master’s from The Johns Hopkins University.
• Steven Leinwand, Task Leader for Professional Development Interventions, is Director of
Mathematics Item Development at AIR, overseeing mathematics item development for major
projects in several states. He has over 20 years of experience in leadership positions in
mathematics education. His most recent work at AIR resulted in What the United States Can Learn
from Singapore's World-Class Mathematics System. Before joining AIR in 2003, he spent 22 years
as Mathematics Consultant with the Connecticut Dept. of Education, where he was responsible for
the development and oversight of a broad statewide program of activities in K–12 mathematics. Mr.
Leinwand holds a Master’s in Educational Administration and Supervision from Central Connecticut
State University and a B.A. in Economics from Wesleyan University.
• Fran Stancavage, Task Leader for Instrumentation, is Managing Research Scientist with over 30
years of experience at AIR. Ms. Stancavage directs the Validity Studies project for the National
Assessment of Educational Progress (NAEP) and co-directs a study of the Sensitivity of NAEP to
the Effects of Reform-Based Teaching and Learning in Middle School Mathematics. She also
oversaw the monitoring and analysis of program implementation for a recently completed
randomized field trial, Closing the Reading Gap, which evaluated four reading interventions for
striving readers. Ms. Stancavage holds a Master’s and a B.A. in Sociology from California State
University, San Jose and Vassar College, respectively.
• Dr. Andrew Wayne, Deputy Project Director, is a Senior Research Analyst at AIR. His research projects and publications focus on policies affecting teachers and the quality of teaching. He has served in leadership roles on projects for the U.S. Department of Education, the National Board for Professional Teaching Standards, and the Carnegie Corporation of New York. He recently coauthored a report for ED on teachers’ use of technology, which addressed the influence of technology-related professional development. He has taught middle and high school computer science. Dr. Wayne holds a Ph.D. in Policy studies from the University of Maryland, a Master’s in Curriculum and Instruction from the University of Texas at Austin and a B.A. from Dickinson College in Physics.
Other Team Members:
• Terry Anstrom, Senior Research Analyst, AIR
• Helen Duffy, Research Analyst, AIR
• Christian Geckeler, Research Analyst, MDRC
• Kristin Porter, Research Associate, MDRC
• Shelley Rappaport, Research/Operations Associate, MDRC
• Stephanie Safran, Research Associate, AIR
• James Taylor, Research Analyst, AIR
• Kirk Walters, Research Analyst, AIR
For more information, please contact:
• Robert Ivry, MDRC, (212) 340-8672, robert.ivry@mdrc.org
• Andrew Wayne, AIR, (202) 403-5483, awayne@air.org
PowerPoint presentation for Recruitment Site Visits
(sample version will be revised as appropriate)
1 Although it may not be feasible to include exactly the same number of schools per district in the evaluation sample, the study will approximate this objective as closely as possible.
2 In particular, we used district-wide individual student databases from four recent school years in Houston, TX, Columbus, OH, Atlanta, GA and Newark, NJ in order to calculate the individual and school level parameters required to estimate minimum detectable effect sizes. These effects were calculated assuming “fixed effects” and the availability of both individual and school level prior achievement data. They were calculated using the equation:
MDES
=
,
where
M
= 
,
and
is the multiplier that translates the standard error into a minimum
detectable effect estimate. It is equal to the t
critical value for
,
the
significance level of the intended statistical test, plus the t
critical value for
,
the
likelihood of detecting significant effects given a true effect of a
particular, size, i.e., the power of the test.
= the
school level variance component;
= the
school level variance, after controlling for whatever student or
school level characteristics are to be added to the impact
regression;
= student
level variance of the outcome in question;
= student
level variance after controlling for student or school level
characteristics added to the regression;
P= the proportion of treatment schools;
J = the total number of schools in the analysis;
n = the number of students within each school.
3 Significant use means that the text is used by at least half of the schools in the district for their main (mid-track) mathematics class.
| File Type | application/msword |
| File Title | PROFESSIONAL DEVELOPMENT IMPACT |
| Author | SCronen |
| Last Modified By | DoED |
| File Modified | 2007-05-09 |
| File Created | 2007-05-09 |
© 2024 OMB.report | Privacy Policy