NWREL 61 Supporting Statement A

AN INVESTIGATION OF THE IMPACT

OF A TRAITS-BASED WRITING MODEL

ON STUDENT ACHIEVEMENT

PAPERWORK REDUCTION ACT

CLEARANCE REQUEST

SUPPORTING STATEMENT PART A

Prepared For:

Institute of Education Sciences

United States Department of Education

Contract No. ED-06-CO-0016

Prepared By:

Northwest Regional Educational Laboratory

Center for Research, Evaluation, and Assessment

March 2007

Table of Contents

Page

List of Exhibits ii

SUPPorting Statement for Paperwork Reduction Act Submission

A. JUSTIFICATION

1. Circumstances Making Collection of Information Necessary 1

2. Purposes and Uses of the Data 3

3. Use of Technology to Reduce Burden 4

4. Efforts to Identify Duplication 4

5. Methods to Minimize Burden on Small Entities 4

6. Consequences of Not Collecting the Data 4

7. Special Circumstances 5

8. Federal Register Comments and Persons Consulted Outside the

Agency 5

3. Payments or Gifts 5

4. Assurances of Confidentiality 6

5. Justification on Sensitive Questions 6

6. Estimates of Hour Burden 6

7. Estimate of Cost Burden to Respondents 7

8. Estimate of Annual Cost to the Federal Government 7

9. Program Changes or Adjustments 7

10. Plans for Tabulation and Publication of Results 7

11. Approval to Not Display OMB Expiration Date 9

12. Explanation of Exceptions 9

REFERENCES 9

B. DESCRIPTION OF STATISTICAL METHODS Separate file

APPENDICES

Appendix A: Teacher Survey on Writing Instruction Separate file

Appendix B: Student Essay Instructions and Sample Student Essay Booklet Separate file

Appendix C: Coded Student Data Form Separate file

List of Exhibits

Page

1. Methodology Consultants and Technical Working Group Members 5

2. Annual Hour Burden for Respondents 6

3. Study Timeline 8

INTRODUCTION

Improving student literacy is identified as a high priority in regional needs assessments conducted over the past few years by NWREL; teachers and principals expressed a need for professional development for teaching writing, and for research-based practices to support the various writing modes. Improving student literacy is also one of the main goals of NCLB, which includes goals to narrow the achievement gap for a number of student subgroups. The sample will include students in many of the AYP reporting categories for NCLB and will enable an exploration of the impact of the treatment with these student populations. The model is focused on formative classroom assessment, which is also identified as a high priority in regional needs assessments.

While writing is not a focus of the AYP provisions of NCLB, it is a critical component of language and literacy broadly, and includes an essential set of skills for continued academic development as well as for success in many occupations. Writing is obviously crucial for higher educational achievement and the development of higher order thinking skills, and is also one of the Foundation Skills identified by the U.S. Department of Labor’s Commission on Achieving Necessary Skills for a broad range of jobs.

National NAEP results in 2002 showed that just over a quarter of fourth grade students (27 percent) were proficient in writing, with similar proficiency levels at 8th grade (31 percent) and 12th grade (24 percent). In the Pacific Northwest, overall fourth grade proficiency rates on the 2002 NAEP writing tests were even lower than the national average in three states. Idaho, Montana and Oregon, all had proficiency rates of 22 percent, while the national average was 27 percent. The proficiency rate in Washington was 30 percent. Alaska did not participate in state-level NAEP.

Proficiency rates on the NAEP writing test were much lower among at-risk groups. For students eligible for free or reduced price lunch, the proficiency rates ranged from 13 to 17 percent in Idaho, Montana, Oregon and Washington, which is near the national average of 16 percent. Proficiency rates for Black students were 14 percent in Oregon and 20 percent in Washington, compared to the national average of 17 percent; for Hispanic students, the proficiency rates were 9 percent in Oregon, 11 percent in Idaho, and 12 percent in Washington, all below the national average of 17 percent. American Indian and Alaska Native students, a subgroup encompassing 5 percent of all students in the Northwest region, had a proficiency rate of only 8 percent in Montana, well below the national average for this group of 15 percent. (States not listed in this section did not have data reported because the sample size was insufficient to permit a reliable estimate.)

In statewide achievement test results, students in the Northwest region demonstrate lower rates of proficiency in writing than in other aspects of language arts. For example, in the 2005 Oregon Statewide Assessment Testing (OSAT) results, 32 percent of Oregon fourth graders met state writing standards. This is less than half the rate of students meeting state reading and literature standards.

Similarly, in the 2005 Idaho Direct Writing Assessment (DWA) results, 34 percent of fifth graders were proficient or advanced. This is less than half the rate of fifth graders meeting state reading standards. Only 17 percent of Idaho Hispanic students and 21 percent of American Indian students scored at the proficient or advanced levels in writing.

In Alaska, a substantial achievement gap exists between Alaska Native and white students in writing. In fifth grade, 54 percent of Alaska Native students met standards in 2005, compared to 84 percent of white students. In sixth grade, 51 percent of Alaska Native students met standards in 2005, compared to 87 percent of white students.

These assessment results raise concerns because writing is a critical life skill and writing also supports the development of reading and thinking skills. There is a strong connection between writing and learning to think systematically (Sommers, 1982; Zinsser, 1988). While reading and mathematics are typically the core subjects of greatest concern to schools, writing also received a substantial emphasis in regional needs assessments (Barnett & Greenough, 2004; 2005). Many schools implement “writing across the curriculum” or use writing projects as integrative, interdisciplinary projects to raise student engagement and achievement in reading, mathematics, science and social studies, and to prepare students for advanced studies or careers. Better understanding of what works in the teaching of writing will benefit schools and students in the region and in the nation.Our regional needs assessments consistently identify improving student literacy as a high priority area (Barnett & Greenough, 2004; 2005). Teachers and principals express a need for professional development for teaching writing, and for research-based practices to support the various writing modes. Improving student literacy is also one of the main goals of NCLB, which includes goals to narrow the achievement gap for various student populations, including English language learners.

While writing is not a focus of the AYP provisions of NCLB, it is a critical component of language and literacy, and includes an essential set of skills for continued academic development as well as for success in many occupations. Writing is crucial for higher educational achievement and the development of higher order thinking skills, and is also one of the Foundation Skills identified by the U.S. Department of Labor’s Commission on Achieving Necessary Skills for a broad range of jobs.

National NAEP results in 2002 showed that just over a quarter of fourth grade students (27 percent) were proficient in writing, with similar proficiency levels at 8th grade (31 percent) and 12th grade (24 percent). In the Pacific Northwest, overall fourth grade proficiency rates on the 2002 NAEP writing tests were even lower than the national average in three states. Idaho, Montana and Oregon, all had proficiency rates of 22 percent. The proficiency rate in Washington was 30 percent. Alaska did not participate.

Proficiency rates on the NAEP writing test were much lower among at-risk groups. For students eligible for free or reduced priced lunch, the proficiency rates ranged from 13 to 17 percent in Idaho, Montana, Oregon and Washington near the national average of 16 percent. Proficiency rates for Black students were 14 percent in Oregon and 20 percent in Washington, compared to the national average of 17 percent. For Hispanic students, the proficiency rates were 9 percent in Oregon, 11 percent in Idaho, and 12 percent in Washington, all below the national average of 17 percent. American Indian and Alaska Native students, an important subgroup in the Northwest region, had a proficiency rate of 8 percent in Montana, well below the national average for this group of 15 percent.

On statewide achievement tests, students in the Northwest region demonstrate lower rates of proficiency in writing compared to the other aspects of language arts. In the 2005 Oregon Statewide Assessment Testing (OS) results, 32 percent of Oregon fourth graders met state writing standards. This is less than half the rate of students meeting state reading and literature standards. In the 2005 Idaho Direct Writing Assessment (DWA) results, 34 percent of fifth graders were found proficient or advanced. This was less than half the rate of fifth graders meeting state reading standards. Only 17 percent of Idaho Hispanic students and 21 percent of American Indian students scored at the proficient or advanced levels in writing.

In Alaska, a substantial achievement gap exists between Alaska Native and white students in writing. In fifth grade, 54 percent of Alaska Native students met standards in 2005, compared to 84 percent of white students. In sixth grade, 51 percent of Alaska Native students met standards in 2005, compared to 87 percent of white students.

These assessment results raise concerns because writing is a critical life skill, not to mention that writing supports the development of reading and thinking skills. There is a strong connection between writing and learning to think systematically (Sommers, 1982; Zinsser, 1988). Many schools implement “writing across the curriculum” or use writing projects as integrative, interdisciplinary projects to raise student engagement and achievement in reading, mathematics, science and social studies, and to prepare students for advanced studies or careers. Better understanding of what works in the teaching of writing will benefit schools and students in the region and in the nation.

The trait-based approach to writing instruction is a popular model in widespread use in the Northwest region and across the nation, despite a relative paucity of well controlled studies demonstrating its effectiveness. Many school staff members, administrators, policy makers and parents view this as a valuable approach to the teaching of writing, and it is the subject of numerous publications and training programs, including but not limited to those provided by NWREL. The model has been incorporated into major published language arts curriculum materials, guides to writing instruction provided by several major educational publishers, and a host of training workshops, web sites and other resources for schools.

However, this popularity is based largely on rational arguments for the underlying model, perceived face validity of the assessment instruments and training materials, and practitioner perceptions that the model has utility and improves student performance. Given this widespread interest, it is important that the education community has access to high quality scientific evidence on the effectiveness of the approach. Schools and policy makers will benefit from improved understanding of the extent to which the approach works, for whom, and under what conditions. The proposed research will contribute to that knowledge base, so that decisions about whether to expand, contract, or modify the adoption of this approach can be based on reliable data.

Only two experimental studies have been completed examining the efficacy of the model, and these are insufficient to provide the broad research base that should underlie such a widely implemented approach. Therefore, the aim of this project is to provide a rigorous, relatively large scale scientific test of the approach, in conditions that are commensurate with the way the intervention is typically implemented in many school settings. The present study will improve on previous studies by involving a larger number of schools, strengthening implementation fidelity, and improving the research methodology, including properly modeling the nested data structure.The trait-based approach to writing instruction is a popular model in widespread use across the nation, despite a relative paucity of well controlled studies demonstrating its effectiveness. Many school staff members, administrators, policy makers and parents view this as a valuable approach to the teaching of writing, and it is the subject of numerous publications and training programs, including but not limited to those provided by NWREL.

In particular, the 6+1 Trait® Writing classroom professional development is currently in wide use across the United States and in other countries. Since 1990, more than 500 training sessions have been conducted by NWREL trainers, and these sessions have involved more than 20,000 participants from all 50 states and 17 countries. NWREL also conducts a training-of-trainers institute; educators who attend this institute receive a full training package and are given permission to conduct training in their local school jurisdictions. Since 1998, a total of 1,365 trainers have been trained by NWREL. NWREL has produced three key publications and a number of classroom support materials for 6+1 Trait® Writing that are used extensively by classroom teachers. Over 350,000 copies of the three key publications have been sold. Given this widespread use, it is important to expand the available research base on the effectiveness of the approach under various conditions and for specific subgroups of students and teachers.

The 6+1 Trait® Writing intervention is an approach to teaching and assessing student writing that consists of a set of strategies to facilitate the integration of assessment and instruction with a specific focus on seven traits of effective writing. The intervention is supported by strategies for direct instruction on the traits and a writing process that emphasizes the effective use of teacher and peer feedback to support students in the continual review and revision of their writing using rubrics containing specific criteria. It is supported by professional development and materials designed to build teacher understanding of their instructional use, and enhance teacher skills in supporting students to improve writing.

The proposed study will extend previous studies on the 6+1 Trait® Writing model by involving a larger number of schools and providing more training and systematic support for teachers throughout implementation. The study will explore relationships among district, school, teacher, and student variables to provide more reliable information for decision-makers to use in determining whether and how to implement the model. The study will also contribute to an area with a shortage of current research, since much of the research on the impact of professional development on student achievement has been conducted in science and mathematics education (Guskey, 2003).

DESCRIPTION OF THE NWREL TRAITS-BASED WRITING MODEL STUDY

PURPOSE OF THE STUDY

The goal of the study is to provide high quality evidence on the effectiveness of the analytical trait-based model for teaching and assessing student writing, by examining the impact of the model on student achievement in writing. The goal of this study is to determine the effectiveness of professional development for a trait-based writing model. Toward this goal, the impact of the model on the achievement of 5^th graders will be examined. The model, 6+1 Trait^® Writing, will be implemented by the NWREL Traits Writing Assessment Training Unit. The model is designed to improve student writing through an integrated approach to teaching and assessing writing skills, and it incorporates ten instructional strategies to develop the specific traits of writing. Key components of the model include effective feedback to students and the engagement of students in self-assessment.

EXISTING EVIDENCE SUPPORTING THE MODEL

In the first major study of the writing process, Emig (1971) drew on her own knowledge and experience as a writer to develop a process model that emphasized writing processes that are recursive rather than linear, including planning, organization, drafting, editing, and writing processes that vary according to task and instructional context. By 1986, in What Works, the Department of Education identified process as the most effective way to teach writing (in Newell, 1998). Meanwhile, Diederich (1974; Diederich, French & Carlton, 1961) and Purves (1988) were instrumental in moving writing research away from holistic assessments to classroom-based analytical assessments of student writing, to meet the needs of teachers for diagnostic assessment data on which to base decisions about instruction.

In 1983, the writing process and Diederich’s scales were integrated by teachers in Beaverton, Oregon (Grundy, 1986) to build the foundation for what became the 6+1 Trait^® Writing model. This model is based on recursively planning, analyzing and revising writing using a framework of analytic traits that characterize quality writing. The model was also informed by a review of 20 years of research on student writing conducted by Hillocks (1987). His review of 2,000 studies identified the use of scales as having a positive effect on improving student writing, second only in effect size to inquiry. This method, of which the 6+1 Trait Writing model is an example, employs sets of criteria to evaluate pieces of work, which helps students develop an understanding of discourse knowledge, improving the ability to present ideas and information in a coherent manner.

The 6+1 Trait Writing model is intended to help teachers provide effective feedback to students, and to develop student self-assessment skills and metacognition related to writing. Reviews of the research on formative assessment have consistently demonstrated that assessment that provides students with feedback about their performance increases subsequent achievements (Natriello, 1987; Crooks, 1988; Black & Wiliam, 1998). Hattie (1992) found that “the most powerful single moderator that enhances achievement is feedback.” Formative assessment is grounded in the feedback models of Crooks (1988) and Sadler (1989), in which students have access to three types of information about a particular performance: (a) the intended instructional outcome, (b) how current performance matches or does not match that expectation, and (c) a mechanism to move students from current performance toward that vision. Marzano (2003) identified a number of features of feedback that make it successful, such as being timely and ongoing throughout the learning process, specific to the content being learned, aligned with assessment, and corrective. Fontana & Fernandes (1994) reported that self-assessment based on an understanding of learning goals and evaluation criteria improves student achievement.

As noted above, there is strong evidence for the effectiveness of formative assessment and feedback as general strategies to support learning. Specific prior research on the 6+1 Trait Writing model includes two experimental studies that examined the impact of the model on student achievement and teacher practices, a correlational study examining the relationship of trait-based assessments of student writing to statewide assessment scores, and several studies of teacher opinions of the analytical trait model.

Two experimental research studies, each involving a small number of schools, have been conducted to examine the impact of the 6+1 Trait Writing model on student learning (Arter, Spandel, Culham and Pollard, 1994; Kozlow & Bellamy, 2004). The first study was conducted during the 1992-1993 school year in six grade 5 classrooms that were randomly assigned to either an experimental or control condition. Teachers in the treatment group received training and specific lesson plans and strategies to enhance student writing performance, plus follow-up assistance during the year. The study found that students in the treatment group had a significantly higher increase in test scores for the ideas trait when compared to students in the control group. Gains in all other traits also favored the treatment group, but were marginally significant or non-significant. The original analysis did not properly account for the nested data structure, and the raw data no longer exist, preventing a proper re-analysis that could yield valid estimates of effect size. However, the overall pattern of mean differences favoring the treatment group on all six outcome measures is unlikely to have occurred by chance.

The second study, conducted during the 2003-2004 school year, involved 72 classes in grades 3 to 6 in one school district. Within each grade, half of the classes were randomly assigned to the treatment group and half to the control group. Teachers in the treatment group received two days of training in November, and were asked to implement the trait approach to teaching and assessing writing for the remainder of the school year. Teachers in the control group were asked to continue teaching and assessing writing following their normal practices. Classroom visits showed considerable variation in the extent of implementation by treatment teachers, as well as substantial implementation of similar practices in the control group. The outcome measure was administered only a few months after the training, and survey comments indicated a need for more training and more time to implement the model. The data were properly analyzed using a mixed model framework to account for the nested data structure. Differences between the treatment and control group were not found to be statistically significant. The findings were deemed inconclusive due to the failed implementation and contaminated control group.

In a study of the correspondence of six-trait writing scores and performance on the Washington Assessment Student Learning writing test (scored holistically), Coe (2000) found that students who scored low in the six-traits tended to also score low in the WASL. For example, 28.6 percent of students who had at least one trait score less than three succeeded in passing the WASL (using a 5-point scale with half-point scores possible due to averaging the scores given by two raters). Conversely, 83.1 percent of students with scores of 3 or above on all six traits passed the WASL. Among students with all trait scores above 3.5, 93.8 percent were successful on the WASL writing test. These findings lend support for the idea that formative assessment based on the six trait model may provide students with useful feedback as they work toward meeting state standards.

Teacher opinions of the approach are very positive as evidenced by the ongoing popularity and widespread use of models of this genre. In the Kozlow & Bellamy (2004) study, over 80 percent of teachers reported the perception that the approach helped them to improve their writing instruction, their understanding of the qualities of good writing, and their ability to provide effective feedback to students. Van Hoet-Hill and Wright (2000), in a study of teacher practices and attitudes to writing instruction, found that given the choice, two thirds of the teachers responding chose to use the six-trait model for assessing student writing, citing the strength of the elements (criteria), rigor, and alignment with state assessments as reasons. Of the teachers using the trait model, 100 percent reported the perception that the writing performance of their students was improved as a result of using the model.Support for the intervention for the proposed study is provided below through a synthesis of the research literature on student writing and formative assessment that supports the model, a synthesis of the research literature on professional development that supports the approach taken in the professional development on 6+1 Trait® Writing, and a compilation of information on the potential efficacy of the intervention.

Research literature supporting a traits model. Much of the developmental work for the 6+1 Trait® Writing model was based on the work of Diederich (1974) and Purves (1988) who were instrumental in moving the educational research field away from holistic assessments to classroom-based analytical assessments of student writing to meet the needs of teachers for diagnostic assessment data on which to base their instruction. Another area of research that influenced the 6+1 Trait® Writing model is writing process. In the first major study of the writing process, Emig (1971) drew on her own knowledge and experience as a writer to develop a process model that emphasized writing processes that are recursive rather than linear, including planning, organization, drafting, editing, and writing processes that vary according to task and instructional context. By 1986, in What Works, the Department of Education identified process as the most effective way to teach writing (in Newell, 1998).

In 1983, the writing process and Diederich’s scales were brought together by teachers in Beaverton, Oregon (Grundy, 1986), thus providing the foundation for the development of the 6+1 Trait® Writing model. This model incorporates writing process as an integral component of a recursive set of activities that develops the seven traits that characterize quality writing.

Traits-based writing instruction is also supported by a meta-analysis of 20 years of research on student writing conducted by Hillocks (1987). His review of 2,000 studies identified the use of scales as having a positive effect on improving student writing, second only in effect size to inquiry. This method, of which the 6+1 Trait® Writing model is an example, employs sets of criteria to evaluate pieces of work, which helps students develop an understanding of discourse knowledge, improving the ability to present ideas and information in a coherent manner.

A key focus of the 6+1 Trait® Writing model is to help teachers to provide effective feedback to students and to develop self-assessment skills in students. Reviews of the research on formative assessment have consistently demonstrated that assessment that provides students with feedback about their performance increases subsequent achievements (Natriello, 1987; Crooks, 1988; Black & Wiliam, 1998). Hattie (1992) found that “. . . the most powerful single moderator that enhances achievement is feedback.” Formative assessment is grounded in the feedback models of Crooks (1988) and Sadler (1989) whereby students have access to three types of information about a particular performance: (a) the intended instructional outcome, (b) how current performance matches or does not match that expectation, and (c) a mechanism to move students from current performance toward that vision. Marzano (2003) identified a number of features of feedback that make it successful, such as being timely and ongoing throughout the learning process, specific to the content being learned, aligned with assessment, and corrective. Fontana & Fernandes (1994) reported that self-assessment based on an understanding of learning goals and evaluation criteria improves student achievement.

Research literature on professional development. The professional development for the intervention draws on the research on effective strategies for professional development. Although there are inconsistent research findings on the effectiveness of various approaches to professional development, six general features have been linked, in varying degrees, to changes in teacher performance and/or student learning in four of the more rigorous studies identified in the literature (Desimone, et al., 2002; Garet, et al., 2001; Cohen & Hill, 1998; Kennedy, 1998): form of activity, collective participation and collaboration, content and pedagogical focus, active learning, sustained time and duration, and coherence.

The professional development for the proposed study incorporates these six most likely predictive features of effective professional development. The form of activity is a workshop supported by coaches, teacher team meetings, and online support that will extend over a two-year period. The degree of collaboration is supported by the coaches who will facilitate teacher meetings, and by the online support that will enable teachers to communicate with each other. Active learning is built into the workshop and the coaches provide support for teachers to engage in active learning at the school site. Time and duration is systematically supported through direct coaching and feedback to teachers, teacher team meetings, and online support. Content and pedagogical focus is supported as the coach provides direct motivation to maintain focus on the targeted improvement efforts and reinforcement of learned concepts through discussion. >>> add a sentence describing how “active learning” is supported. >>> add a sentence describing how “sustained time and duration” is supported. Coaches support alignment with school/district goals, assist with mapping to district/state standards, and provide consistency across grades and classrooms, thereby addressing coherence.

Another general finding from the research on professional development is that a focus on specific teaching practices increases the probability of the use of those practices (Desimone et al., 2002). The content of the professional development for 6+1 Trait® Writing places a high emphasis on specific practices and materials that teachers can use immediately following the training. The follow-up processes provide opportunities for teachers to use those strategies and to interact with a trainer, a coach, and other teachers to share their experiences.

Potential efficacy of the intervention. Factors critical to demonstrating the potential benefits of the intervention are (1) the likelihood that the professional development will result in the intended changes in teacher practices, and (2) the likelihood that these changes in teacher practices will result in improved student achievement. There are two sources of evidence concerning the potential efficacy of the professional development for 6+1 Trait® Writing: teacher opinions about past training, and two experimental studies that examined the impact of this training on student achievement and teacher practices.

Teacher opinions in two areas are relevant to potential efficacy: their perceptions of the effectiveness of the professional development in providing knowledge, skills, and strategies that will assist them in changing their practices; and their perceptions of the effectiveness of these strategies for improving student achievement in writing. Results from evaluation forms completed by 720 teachers who attended workshops on 6+1 Trait® Writing in recent years are very positive. When asked if the materials provided would be immediately useful, 94 percent of teachers said “yes” or “mostly,” and 87 percent of teachers gave these responses when asked if the training met their expectations. Survey responses from 34 teachers in grades 3 to 6 who received training for an experimental study on 6+1 Trait® Writing indicate that teachers found the training to be effective, based on implementing the model for six months (Kozlow & Bellamy, 2004). Concerning the effects of training on their teaching, 80 to 90 percent of teachers agreed that the training helped them to improve their writing instruction, their understanding of the qualities of good writing, and their ability to provide effective feedback to students; and agreed that their students developed a good understanding of the traits.

Van Hoet-Hill and Wright (2000) in a study of teacher practices and attitudes to writing instruction found that given the choice, two thirds of the teachers responding chose to use the six-trait model for assessing student writing, citing the strength of the elements (criteria), rigor, and alignment with state assessments as reasons. Of the teachers using the trait model, 100 percent reported improved student writing performance as a result.

Two experimental research studies, involving a small number of schools, have been conducted by NWREL that specifically examined the impact of the 6+1 Trait® Writing model on student learning (Arter, et al., 1994; Kozlow & Bellamy, 2004). The first study was conducted during the 1992-1993 school year in six grade 5 classrooms that were randomly assigned to either an experimental or control condition. Teachers in the treatment group received training and specific lesson plans and strategies to enhance student writing performance, plus follow-up assistance during the year. The study found that students in the treatment group had a significantly higher increase in test scores for the ideas trait when compared to students in the control group.

The second study, conducted during the 2003-2004 school year, involved 72 classes in grades 3 to 6 in one school district. Within each grade, half of the classes were randomly assigned to the treatment group and half to the control group. Teachers in the treatment group received two days of training in November, and were asked to implement the trait approach to teaching and assessing writing for the remainder of the school year. Teachers in the control group were asked to continue teaching and assessing writing following their normal practices. A small number of classroom visits showed considerable variation in the extent of implementation by treatment teachers. Survey comments indicated a need for more training and more time to implement the model. Although significant differences were not identified when a linear mixed model was applied to test for differences between treatment and control groups and for a treatment by test-time interactions, some moderate effect sizes were obtained that would have potential for significant differences with larger numbers of teachers at each grade level.

In a research study of the correspondence of six-trait writing scores and performance on the Washington Assessment Student Learning writing test (scored holistically), Coe (2000) found that students who scored low in the six-traits tended to also score low in the WASL. For example, 28.6 percent of students who had at least one trait score less than three on a 5-point scale succeeded in passing the WASL. Conversely, 83.1 percent of students with scores of 3 or above on all six traits passed the WASL. Among students with all trait scores above 3.5, 93.8 percent were successful on the WASL writing test. These findings lend support for the idea that formative assessment based on the six trait model may provide students with useful feedback as they work toward meeting state standards.

Teacher interviews from the experimental study conducted in 2004 indicated that teachers needed more time to implement 6+1 Trait® Writing model. This was supported by classroom observations that indicated a wide range of levels of implementation. As a result, the training and support for 6+1 Trait® Writing model has been subsequently strengthened and the amount of time to effectively absorb the training and implement effective change has been lengthened. The intervention timeline now allows for the thorough planning, teaching, and practicing of each trait to ensure student understanding and use in their writing.

RESEARCH DESIGN

The study participants will be 5th grade students and teachers from 64 elementary schools in Oregon. The study will employ a cluster-randomized design with random assignment at the school level; students will be nested within classrooms (teachers) which will be nested within schools.

There are several reasons for random assignment at the school level rather than at the teacher or student level. Chief among these are 1) it would be difficult to eliminate the potential of contamination across conditions if both the treatment and control teachers were present in the same school building, and 2) the intervention is intended to be implemented in a school wide context whenever possible, so it would be an artificial change in the intervention to prevent teacher collaboration and interaction within a school.

The analysis of student outcomes will estimate the effect of the intervention on student achievement. Appropriate statistical tests will be used to take into account the nested, multilevel structure of the data. Additional analyses of student outcomes will examine the heterogeneity of impacts using baseline measures of school, teacher and student characteristics.

Teacher-level data will be examined for an analysis of implementation fidelity and as part of fully describing the treatment and counterfactual conditions. Since the 6+1 Trait Writing model has achieved popularity in the region and nationally, it is likely that many teachers at control group schools will have awareness and perhaps some experience with this model. In order to address this problem of prior exposure, it will be necessary to exclude schools from the study before randomization if they are already thoroughly trained in a trait-based approach and employing this approach with students. We will also collect information from teachers in treatment and control group schools concerning their prior experience with trait-based writing models in general and the 6+1 Trait Writing model in particular.

Teacher knowledge, attitudes and behaviors will be analyzed to determine the level of implementation of classroom practices designed to impact students and to describe the characteristics of the classrooms in both control and experimental schools. All data collection, data management, and data analysis will be conducted by the research team or research contractors, not by the intervention delivery team.The study participants will be 5th grade students and teachers from 44 elementary schools in the Anchorage School District, a large district in Alaska. The study will employ a cluster-randomized design with random assignment at the school level; students will be nested within classrooms (teachers) which will be nested within schools. There are two reasons for random assignment at the school level rather than at the teacher or student level. First, it would be difficult to eliminate the potential of contamination across conditions if both the treatment and control teachers were present in the same school building. Second, the intervention is intended to be implemented in a school wide context whenever possible.

Since the intervention is provided as professional development for teachers with the understanding that changes in teaching practice will cascade down to students, both teacher and student data will be collected. The analysis of student outcomes will estimate the effect of the intervention on student achievement. Appropriate statistical tests will be used to take into account the nested, multilevel structure of the data. Additional analysis of student outcomes will examine the role of pre-existing teacher and student characteristics as moderators of program effects.

Teacher-level data will be examined for an analysis of implementation fidelity and as part of fully describing the treatment and counterfactual conditions. Since the 6+1 Trait® Writing Model has achieved popularity in the region and nationally, it is likely that many teachers at control group schools will have some experience with this model. In order to address this problem of prior exposure, it will be necessary to also collect information from teachers in control group schools concerning their prior experience with trait-based writing models in general and the 6+1 Trait® Writing Model in particular.

Teacher knowledge, attitudes and behaviors will be analyzed to determine the level of implementation of classroom practices designed to impact students and to describe the characteristics of the classrooms in both control and experimental schools. All data collection, data management, and data analysis will be conducted by the research team or research contractors, not by the intervention delivery team.

Research Questions. This study will answer two experimental questions to determine the impact of the intervention on student achievement in writing:

1. What is the impact of 6+1 Trait Writing on student achievement in writing?

2. How do student impacts vary by pre-existing characteristics of schools, teachers and students?

In addition, descriptive studies of both treatment and control classrooms will be conducted to help interpret and understand the results of the experimental research questions, including the fidelity of treatment implementation and the differences and similarities between treatment and control classrooms. The questions with respect to implementation are the following:

3. To what extent did teachers in the treatment group implement the intervention with fidelity to the intended model?

What differences and similarities exist with respect to implementation of writing instruction between treatment and control classrooms?This study will answer two experimental questions to determine the impact of the intervention on student achievement in writing:

1 Are there differences in student writing achievement in the treatment and control schools?

2 How do student demographic variables and other context variables moderate the impact of the treatment on student achievement?

In addition, descriptive studies of both treatment and control classrooms will be conducted to help interpret and understand the results of the experimental research questions, including the fidelity of treatment implementation and the differences and similarities between treatment and control classrooms. The questions with respect to implementation are the following:

To what degree was the intervention implemented as intended by teachers in the treatment group?

What differences and similarities exist with respect to implementation of writing instruction between treatment and control classrooms?

Sample. At present, discussions are in progress with Oregon school districts and buildings to build a pool of study sites. The results of this experimental study will be potentially relevant to all elementary schools, particularly those in the U.S. northwest states, and specifically to 5^th grade writing teachers and students. The universe of cases will be those 5^th grade students in schools which volunteered to participate in the study, and other students/schools like them. Since the schools are not randomly sampled from a known population, generalization to other schools cannot be made strictly on statistical grounds based on this single study. In other words, it cannot be known with certainty whether any other specific school or student is part of the population from which this sample was drawn. Rather, as in most group randomized trials, generalization to other sites must be based on considered judgments about the similarity of those sites to schools in the study, and/or schools from replication studies in additional sites and under additional circumstances.

The study will be conducted in approximately 64 elementary schools (32 experimental schools and 32 control schools) in Oregon. Oregon schools are similar to those in other Pacific Northwest states in proportions of English Language Learners, students from low-income families, and students from language/racial/ethnic minority groups.

The study will be conducted in two waves of data collection. Training, implementation and data collection will occur during a one year period for each of the two waves. Control group schools will receive the training and online support resources during the year following the experimental year for each wave. See the timeline on page six above for a visual representation of the schedule for data collection.

The initial sample of schools will consist of a set of elementary schools that are not already thoroughly implementing a 6-trait based writing approach, and that are willing to participate in the research protocol. Oregon uses a trait-based framework for statewide writing assessments in grades 4, 7 and 10, but the state provides little support for integration of this framework in classroom teaching and learning. Many schools provide teachers with professional development in trait-based writing instruction to fill this gap. A screening process will exclude those schools in which elementary level teachers are already thoroughly trained in this model and are already integrating this approach in their classroom instruction and student assignments. It is expected that principals and teachers at schools which have not made strong efforts in the past to use this approach in their classrooms will be interested in participating in the study, in order to receive the professional development that is provided. Furthermore, these schools will provide a reasonable counterfactual, with teachers who have some awareness of the popular trait-based writing models but who have not been thoroughly trained and have not used this model with students.

All participating elementary schools will be randomly assigned to either the control or experimental condition. Randomization of schools will occur within strata defined by districts. Within districts, the two schools with the highest percentage of students eligible for free or reduced price lunch will be randomly assigned to conditions, then the next highest pair of schools, and so on, so that the experimental and control groups will be reasonably well balanced in terms of socio-economic status and variables that correlate highly with SES. In districts with an odd number of participating school, the unpaired school will be randomly assigned to condition. Within a participating school, all 5th grade writing teachers will participate regardless of previous writing content skills. Typical Oregon schools have two 5^th grade teachers. That will yield approximately 64 teachers and 1536 students in each condition at baseline.

The planned baseline samples of teachers and students are large enough to withstand naturally occurring attrition during the study. The student sample will include on average approximately 48 students from each school at baseline. The average 5^th grade class size in Oregon schools is 24 students per classroom. Assuming that as many as one sixth of students present at baseline may not complete the post-test, yielding a conservative estimate of 20 students per classroom, the final student sample for this analysis is expected to be at least 1,280 in the experimental group and 1,280 in the control group.

Student attrition, as well as crossovers and students who enter study classrooms after baseline, will be monitored. Attrition rates will be reported in detail; if attrition is large, or different across conditions, the final sample may be weighted during the analysis to preserve the representative nature of the sample. Students who cross over from treatment schools to baseline schools will be included in the treatment sample for the main analysis, which will be an intent-to-treat (ITT) analysis. If particular teachers within the treatment group fail to implement the model in their classrooms (“no-shows”), those teachers and students will still be included in the treatment group for the ITT analysis. Students entering study schools during the year will not be included in the analysis.At present, discussions are in progress with the Anchorage School District to conduct the study within the 61 regular elementary schools in the district. The results of this experimental study will be potentially relevant to all elementary schools in the five northwest states, and specifically to all 5^th grade writing teachers and students.

The study will be conducted in approximately 44 schools (22 experimental schools and 22 control schools) in the Anchorage School District in Alaska. Anchorage schools are similar to northwest schools in proportions of English Language Learners, students from low-income families, and students from language/racial/ethnic minority groups. In particular, Anchorage has significant student populations of Native American and Alaska Native students, Hispanics, and recent immigrants from many Asian and world cultures.

The initial sample of schools will consist of all Anchorage SD elementary schools not already thoroughly implementing a 6-trait based writing approach. All participating elementary schools will be randomly assigned to either the control or experimental condition. Within a participating school, all 5th grade writing teachers will participate regardless of previous writing content skills. Most schools have three 5^th grade teachers. That will yield slightly over 60 teachers in each of the treatment and control groups at baseline.

The student sample will include approximately 60 students from each school. The baseline student sample for this analysis is expected to be approximately 1,320 in the experimental group and 1,320 in the control group. These are conservative estimates of 20 students per classroom.

The planned baseline samples of teachers and students are large enough to withstand naturally occurring attrition during the study. Teacher attrition will be monitored through personnel records and the project database. Student attrition, as well as crossovers and students who enter study classrooms after baseline, will be monitored. The primary concern in this regard will be to detect any systematic differential attrition or systematic pattern of crossovers between the conditions. However, since the design of the study includes assignment of entire schools to conditions, student crossovers within a school are not possible. Crossovers could only occur if students transfer to a different school within the district during the student treatment year.

Treatment. The intervention under study is the 6+1 Trait^® Writing model, an analytic approach to teaching and assessing student writing developed and disseminated by the NWREL Traits Writing Assessment Training unit. In order to decrease any bias or appearance of bias in the study, all key research activities such as data collection, management and analysis will be completed by NWREL research staff or independent subcontractors, not by intervention delivery staff. Independent contractors will manage the raw data and the master data set to insure transparency and replicability of methods, as described later in this document. The program unit responsible for the development and dissemination of the intervention will only be involved in training and implementation activities, and will not have access that would allow conscious or unconscious contamination of the data or study procedures.

The 6+1 Trait^® Writing model consists of a set of strategies to facilitate the integration of assessment with instruction, targeting seven “traits” of effective writing: Ideas, Organization, Voice, Word Choice, Sentence Fluency, Conventions, and Presentation. The model employs specific instructional strategies that provide teachers with a range of activities to support direct instruction on the traits, and to engage students in learning about and practicing the use of the traits in planning, assessing and revising their writing. The approach is focused on formative assessments providing effective and timely feedback on performance. These strategies include teaching students the language of rubrics for the traits, having students score and justify their scores on writing samples, and using a writing process that emphasizes feedback, revision, and editing. The intervention is not an alternative writing curriculum designed to replace existing writing programs in schools, but rather an additional, complementary set of tools to aid in conceptualizing, assessing and describing the qualities of writing. It is used in conjunction with existing writing curricula to provide a framework for dialog, designed to improve the ability of teachers and students to plan, evaluate, discuss and revise their writing.

Professional development and support materials will be provided to build teacher understanding of the traits and the instructional strategies that are aimed at supporting students in the development of their writing. Teachers will receive training and resource materials during a summer institute and additional meetings and site visits during the school year. In addition, an online support system will provide additional resource materials and additional access to interaction with trainers and other teachers.

The implementation of the classroom writing intervention will consist of the following activities:

Teachers will attend a four-day summer institute that will provide comprehensive training and resource materials on the traits model.

The 6+1 Trait Writing trainers will support teachers on an ongoing basis at their schools by observing and critiquing teachers’ lessons and meeting with teams of teachers to facilitate discussions of implementation issues and the sharing of experiences among teachers. Teachers will attend 3 days of meetings during the school year to further their understanding of how to implement the model, including practice using the scoring rubric and other resources. Trainers will also visit each treatment group teacher in the classroom four times during the year to answer questions, provide feedback, and suggest ways to improve implementation of the model.

The online support system will provide the capability for teachers to interact with NWREL trainers and other teachers. The facility will include sample papers that teachers can access for illustrations of the traits, opportunities to practice and refine their scoring skills, and a forum in which they may share sample lessons and activities they have found to be particularly effective.

Teachers will integrate the traits model into their existing classroom writing instruction and activities with students. The strategies cover issues of curriculum and instruction, formative assessment, and student engagement in their learning. Table 1 shows the ten strategies by these three purposes.

Table 1 – Instructional Strategies by Purpose

		Curriculum & Instruction Issue	Formative Assessment Issue	Student Engagement Issue
1	Teaching the language of rubrics for writing assessment	X
2	Reading and scoring papers and justifying the scores	X	X
3	Teaching focused revision strategies	X
4	Teachers modeling participation in the writing process		X	X
5	Reading a lot of materials that demonstrate varying writing quality	X		X
6	Writing to effective prompts		X	X
7	Weave writing lessons into other subjects	X
8	Students setting goals and monitoring their progress		X	X
9	Integrating learning goals for writing into curriculum planning	X
10	Teaching ways to structure nonfiction writing	X

The comparison condition will be the regularly planned writing instruction at each control group school.The basis of the intervention, the 6+1Trait^® Writing model, is an analytic approach to teaching and assessing student writing that consists of a set of strategies to facilitate the integration of assessment with instruction; it has a specific focus on the following seven traits of effective writing: Ideas, Organization, Voice, Word Choice, Sentence Fluency, Conventions, and Presentation. The model employs specific instructional strategies that provide teachers with a range of activities to support direct instruction on the traits, and to engage students in learning about and practicing the use of the traits in planning, assessing and revising their writing. The approach is focused on formative assessments providing effective and timely feedback on performance. These strategies include teaching students the language of rubrics for the traits, having students score and justify their scores on writing samples, and using a writing process that emphasizes feedback, revision, and editing. The intervention is not an alternative writing curriculum, but rather an approach to conceptualizing, assessing and describing the qualities of writing. It is used in conjunction with existing writing curricula to provide a framework for dialog, designed to improve the ability of teachers and students to plan, evaluate, discuss and revise their writing.

The professional development and support materials under study were designed to build teacher understanding of the traits and the instructional strategies, and to improve teacher skills in supporting students in the development of their writing. The extent and nature of the content makes a 2-year training and implementation timeline necessary. The intervention for this study consists of professional development, local support, and online resources for trait-based writing. Teachers will receive training and will begin the process of integrating the system into their classrooms during the first implementation year. In addition, local writing coaches will be trained and begin assisting teachers during the first year. During both years of implementation, teachers will receive support from interaction with the local coaches, visits from trainers, and through an online support system. The second year of teacher implementation will be the “treatment year” for students in the study.

The professional development will consist of the following:

An initial four-day training session prior to the first implementation year will provide comprehensive training on the traits model.

Teachers will implement the traits model during this first year and will be supported by a local coach and the 6+1Trait^® Writing trainer, who will make two consulting trips to each treatment school. In addition, support will be provided by 6+1Trait^® Writing training staff through the Internet.

A second four-day training session during the summer after the first implementation year will provide a review and a more detailed analysis of the traits approach. Teachers will bring their first-year experience to bear on this analysis, will share experiences, and will raise questions for the trainer.

During the second implementation year, which is the treatment year for the class of 5^th grade students to be studied, teachers will continue to implement the traits model, integrated with their classroom writing curriculum. Each school will again receive two consulting visits from the NWREL writing expert, support from coaches at the district level, and from NWREL staff through the Internet.

The 6+1Trait^® Writing trainer and district coaches will support teachers on an ongoing basis at their schools by observing and critiquing teachers’ lessons and meeting with teams of teachers to facilitate discussions of implementation issues and the sharing of experiences among teachers. They will conduct demonstration lessons, conduct monthly leveling and scoring meetings, and assist teachers in alignment between classroom lessons, state standards and district curriculum. This will build capacity in the school district to provide ongoing support for sustained implementation after the end of the study. One or more new district coaches will be hired and will receive extensive training in 6+1 Trait^® Writing, with ongoing support from 6+1Trait^® Writing trainers. They will also receive the training-of-trainers workshops, training on effective coaching, and methods for aligning instruction to standards.

The online support will provide the capability for teachers to interact with NWREL trainers, coaches, and other teachers. It will include sample papers that teachers can access for illustrations of the traits and to refine their scoring skills, and will allow teachers to post sample lessons and activities that they have found to be particularly effective so that other teachers may use them.

The on-line support will provide teachers with resources and a discourse community after training has been completed and while implementation is in progress. On-line support for teachers addresses at least three of the six features of effective professional development. These are collective participation and collaboration, content/pedagogical focus, and sustained time and duration. On-line support provides a forum for teachers to discuss issues that arise during implementation with project staff and other teachers. It is also a way of maintaining content and pedagogical focus by giving teachers access to additional materials and resources, as well as creating a forum. Finally, on-line support provides a sustained focus over time on key implementation issues.

6+1Trait^® Writing staff will train all coaches and teachers, and will interact with coaches through the Internet, by phone, and through periodic meetings. A close working relationship will be established with Anchorage School District staff to coordinate the field work. Since some schools will implement the intervention and some will not, district wide professional development for literacy that is not related to the study will be carefully planned and coordinated so that it will not confound the study.

The comparison condition will be the regularly planned writing instruction at each control group school.

DATA COLLECTION

Student Outcome Measure

Student achievement in writing will be assessed using raters trained to score essays using the 6+1 trait model. NWREL maintains a trained pool of raters who are regularly monitored and calibrated to insure inter-rater reliability and fidelity to the assessment model. Each student essay will be scored by two teams of raters. One team will produce a holistic score reflecting the overall quality of the student writing. The other team will produce a set of scores based on the traits of writing that are included in the intervention. The primary hypothesis tests will be conducted using the holistic ratings, which are not overly aligned with the intervention, as the dependent variable. The trait-based ratings will be available for supplementary, parallel analyses of the impact of the intervention on specific aspects of student writing. The raters will score the essays without any knowledge of a student’s experimental condition or whether a given essay was a pre- or post-test. The spring essay scores (at the end of the treatment year) will be used in the outcome analyses as dependent variables, while essay scores from the beginning of the year will be entered as covariate measures of baseline level of writing performance.

In addition, an independent scoring of a random subsample of the end-of-year essays will be made using one of the currently available automated essay scoring systems to provide comparison scores for essays at the end of the year. This will provide additional evidence about the reliability and validity of the human ratings, as well as add to the growing literature on the validity of writing scores produced by artificial intelligence engines.

School-level performance on the Oregon statewide writing assessment will also be used as a school-level covariate, to reduce the between-school variance and improve the efficiency of the design. This assessment is only administered in 4^th, 7^th and 10^th grades, so the 4^th grade aggregate school-level data from the experimental and control group students will be used as a school-level covariate. As noted above, each student’s essay score from the beginning of the year will also be entered as a student-level covariate.

Students typically produce their best writing with prompts that give them some options for personalizing a topic. Prompts will be selected that provide students a clearly defined audience, role, and mode (purpose) for their writing. All prompts for the writing assessment will be in the expository mode because it is generally introduced in the 4^th grade classroom and should be familiar to 5^th grade students. The essay writing sessions will be proctored by participating teachers from both control and treatment groups. Each essay will be scored by raters blind to condition, using the holistic and six analytic rubrics from the 6+1 Trait Writing model.

Teacher Implementation Measure

Teacher classroom practices will be measured in both treatment and control classrooms using teacher surveys to collect data related to teacher implementation of the ten instructional strategies that form the basis of training. This will provide data on both fidelity of implementation in the treatment group schools and contamination in the control group schools. The instruments will be pilot tested during the planning period for this study in separate schools from those which will later make up the treatment and control groups. Analysis of inter-rater reliability and validity will occur prior to the treatment year and will be used to make necessary instrument refinements. Results will be included in project reports. Section 5.2 provides more specific information on the data collection procedures for the following three instruments.

The teacher survey will be administered to all teachers in both treatment and control teachers. The survey contains Likert-type questions about teacher practices and opinions, descriptive information about use of instructional time, and open-ended questions about instructional and learning issues. The results will be used by the research team to describe the treatment and control teachers and their classrooms. Student Outcome Data. Student achievement in writing will be measured using two measures: (1) Alaska State Writing Assessment (ASWA), and (2) NWREL Writing Essay Assessment.

The 5^th grade ASWA score for participating students in the treatment year will be the primary dependent variable and the previous year’s 4^th grade ASWA score will serve as a covariate for those students. The ASWA is administered in May of each year from grades 3 through 10. In addition, an in-class student writing essay assessment will be administered in the fall and spring of the treatment year. These essays will be collected and scored using analytic rubrics for the six traits of the 6+1 Trait® Writing Model by trained NWREL raters. The raters will be blind to the experimental condition of each student. The spring essay scores will be used in the outcome analyses as dependent variables and the fall essay scores will be entered as covariate measures of beginning level of writing performance.

Teacher Implementation Data. Information on teacher classroom practices will be collected through: (1) teacher surveys, (2) teacher interviews, and (3) classroom observations. An instrument will be developed for each of these data collection activities. These instruments are aimed at collecting data related to teacher implementation of the ten instructional strategies that form the basis of training. The instruments will be pilot-tested during the planning period for this study in separate schools from those who will later make up the treatment and control groups. Analysis of inter-rater reliability and validity will occur prior to the treatment year and will be used to make necessary instrument refinements. Results will be included in project reports.

MEASUREMENT INSTRUMENTS FOR WHICH CLEARANCE IS SOUGHT

Student Outcome Data. Student achievement in writing will be measured using two measures: (1) Alaska State Writing Assessment (ASWA), and (2) NWREL the Writing Essay Assessment modeled very closely after the Oregon State Writing Assessment.

Alaska State Writing Assessment

The Alaska State Writing Assessment was developed in 2004-2005 by a testing contractor, Data Recognition Corporation (DRC) as one part of the new and comprehensive Alaska Standards Based Assessment. DRC has test development experience with 18 states. The steps in DRC’s item and test development process were designed in accordance with the Standards for Educational and Psychological Testing, (AERA, NCME, APA, 1999). The development process began with an analysis of the Alaska content standards and grade-level expectations, leading to development of a preliminary test blueprint and item specifications. This defined the standards to be assessed per grade arranged by the relative importance of the content domains. DRC developed item specifications and then conducted a series of meetings over a 20 month period with Alaska teachers and subject-area supervisors to develop the ASWA.

Educators from diverse Alaska geographic and demographic populations participated in all stages of item and test development process. They were involved in a) reviewing of grade-level expectations, b) deciding what the test should measure, c) writing and reviewing of test questions, and d) making decisions as to what test questions actually appeared in the test. The review process included a thorough attention to bias, fairness, and sensitivity issues. In addition, all test questions were reviewed for content validity by an independent team of writing experts from Alaska and across the entire country.

Results from a reliability analysis for the state writing assessment are reported in the “Spring 2005 Alaska Standards Based Assessments (SBAs) Operational and Field Test Technical Report.” A series of tables presents reliability of the writing assessment broken down by student special population groups, including by ethnicity, socioeconomic status, English language proficiency status, migrant status, special education status, and gender. The reliability coefficients for the 5^th grade writing assessment ranged from r =.89 to r =.92, indicating a high level of consistency of scores across the writing items for all special populations. These results are based upon 9,700 5^th grade students taking the test in Spring 2005.

Question types for the ASWA include multiple-choice, short answer, and longer written narratives in the form of narrative, informative, and descriptive writing. Some multiple-choice questions are based on short fiction or nonfiction passages. The content strands of the ASWA measure student ability to: a) write using paragraphs that maintain focus, b) write different types of compositions, such as stories and personal letters, and c) edit sentences using the punctuation and capitalization conventions of Standard English.

NWREL The Writing Essay Assessment

1. NWREL Student achievement in writing will be analyzed by having student essays scored by teams of raters, using a process parallel to that used in the 4^th and 7^th grade Oregon statewide writing assessment. Students will receive the essay prompt and instructions to guide them through a three-day process of planning, drafting, and finalizing an essay. This process is very common in writing classrooms whether or not a trait-based approach to instruction is used and, therefore, should provide students a good opportunity to show their best writing. Specific directions for assessment administration will be provided to teachers, explaining the purpose and procedures for each of the three days. Also, teachers will be provided copies of the current state of Oregon Accommodations Table and Modifications Table for writing test administration. Student accommodations do not change the content and/or performance standards of what is being measured by the assessment. Student essays written with assessment accommodations will be included in all data analyses.

Baseline writing samples will be collected in September or early October of each data collection year; follow-up samples will be collected the following May. All prompts for the writing assessment will be in the expository mode for consistency of performance comparison across all subgroups of students. Students will work on their essays for 45 minutes on each of 3 successive days, to provide the opportunity for a natural writing process including planning, drafting, and revision. The intervention is largely aimed at improving the willingness and ability of students to substantively revise their writing after reflection upon first drafts, therefore it is not possible to estimate the impact of the intervention based on an assessment of first drafts produced during a single writing session. The essay writing sessions will be proctored by the participating teachers in both control and treatment groups.

Each essay will be scored using the holistic and six analytic rubrics from the 6+1 Trait Writing model; presentation will not be scored. NWREL maintains a pool of writing assessment raters who will score the essays without any knowledge of a student’s experimental condition. These raters have scored papers for school district clients in the U.S. and foreign countries for over 20 years, and have recently begun scoring writing samples provided by job applicants for major corporations. The team is experienced in using the 6+1 Trait rubrics as well as client-specific rubrics on papers written in persuasive, expository, and narrative modes. For this study, one scoring team will apply the holistic rubric while a second team will apply the six analytic rubrics.

The rating team is re-calibrated at the beginning of each set of papers to insure proper application of rating scales to each sample and to insure inter-rater reliability, which is regularly monitored in real time during each scoring job. Each essay is scored by two raters; if their scoring of any trait differ by more than one point, a third rater resolves the discrepancy. Ratings are entered directly into an online database which is prepared for each job using identifying information provided by the district or client. This application includes a variety of reporting features, including facilities for ongoing monitoring of rater reliability using the AC1 statistic (Gwet 2002a, 2002b) and a variety of detailed and summary reports.

We plan to have each student essay scored by two teams of raters. One team will produce a holistic score reflecting the overall quality of the student writing. The other team will produce a set of scores based on the traits of writing that are included in the intervention. The first set of hypothesis tests will be conducted using the holistic ratings, which are not overly aligned with the intervention, as the dependent variable. The trait-based ratings will then be used for supplementary, parallel analyses of the impact of the intervention on specific aspects of student writing.

We are making arrangements with Chesapeake Research Associates to receive the original raw student essays from classrooms and to repackage the student papers for processing by the rating teams. All participating teachers will use a pre-addressed and stamped package to send all completed student essays to CRA, who will code each essay to maintain anonymity and then forward them to NWREL to be scored. The raters will be blind to whether each essay is a pre-test or post test, or whether it is from an experimental or control group school. The essay ratings will then be returned to Chesapeake Research Associates and assembled into a data file for analysis, along with relevant school level data collected from NCES and state department of education sources.

By using this method, all research team members at NWREL will be blind to the source of the essays during the rating process. Chesapeake Research Associates will maintain the master copy of the raw essays and the complete raw data file; CRA (or other IES consultants) will thus be able to verify that all reported statistical analyses are completed as appropriate, using the entire original data set.Writing Essay Assessment will be administered in the fall and spring of the treatment year, and it will provide a more specific look at student essay writing performance. This will complement the ASWA results that include one short writing item with a larger number of multiple-choice items. All prompts for the writing assessment will be in the expository mode for consistency of performance comparison across all students. These essays will be administered by participating teachers from both control and treatment groups. Each essay will be scored using the holistic and seven analytic rubrics from the 6+1 Trait® Writing Model. NWREL maintains a trained pool of writing assessment raters who will score the essays without any knowledge of a student’s experimental condition.

Teacher Implementation Data. Information on teacher classroom practices will be collected through: (1) teacher surveys, (2) teacher interviews, and (3) classroom observations administered three times during the treatment year.

Teacher Survey

A teacher survey will be administered to all teachers in both treatment and control schools. The survey contains Likert-type questions about teacher practices and opinions, descriptive information about use of instructional time, and open-ended questions about instructional and learning issues. The results will be used by the research team to describe the treatment and control teachers and their classrooms.

Teacher Interview Protocol

A teacher interview protocol will be used for collecting teacher input on their instructional and assessment practices, curriculum design, and specific learning objectives for students. It will be used in connection with a classroom observation protocol described next.

1. Classroom Observation Protocol

A classroom observation protocol will document how teachers implement a specific lesson in their classrooms. The protocol will ask observers to make note of specific aspects of classroom environment, teacher instructional and assessment activities, and student engagement. The data will correspond to the ten 6+1 Trait® Writing instructional strategies. The information gathered through classroom observations will be compared to the interview data, and used to build a more complete picture of classroom instruction and learning.

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION

A. JUSTIFICATION

1. Circumstances Making Collection of Information Necessary

The Regional Educational Laboratories were authorized by Congress in the Education Sciences Reform Act of 2002, Part D, Section 174, (20 U.S.C. 9564). The central mission for the regional laboratories is to “support applied research, development, wide dissemination, and technical assistance activities by … developing and widely disseminating, including through Internet-based means, scientifically valid research, information, reports, and publications that are usable for improving academic achievement, closing achievement gaps, and encouraging and sustaining school improvement ….” This section goes on to include as part of this mission: “in the event such quality applied research does not exist as determined by the regional educational laboratory or the Department, carrying out applied research projects that are designed to serve the particular educational needs (in pre-kindergarten through grade 16) of the region ….” The act further specifies that one of the key activities undertaken by regional educational laboratories should be to “Identify successful educational programs that have either been developed by such laboratory in carrying out such laboratory’s functions or that have been developed or used by others within the region served by the laboratory….” The current research study addresses this authorization by conducting an experimental evaluation of a widely used and promising approach to the teaching of writing.

To guide the work of the Regional Educational Labs, annual needs assessments are administered by each laboratory, pursuant to the legislative directive to conduct “a continuing survey of the educational needs, strengths, and weaknesses within the region ....” Recent needs assessments conducted by the Northwest Regional Educational Laboratory (NWREL) have indicated that improving student literacy is a high priority, with both teachers and principals expressing a need for professional development for the teaching of writing, and for research-based practices to support the various writing modes.

Improving student literacy is also one of the main goals of the No Child Left Behind Act (NCLB), which includes goals to narrow the achievement gap for a number of student subgroups. While writing is not a specific focus of the Annual Yearly Progress (AYP) provisions of NCLB, it is a critical component of language and literacy broadly, and includes an essential set of skills for continued academic development as well as for success in many occupations. Writing is obviously crucial for higher educational achievement and the development of higher order thinking skills, and is also one of the Foundation Skills identified by the U.S. Department of Labor’s Commission on Achieving Necessary Skills for a broad range of jobs (U.S. Department of Labor, 1991).

Despite the importance of writing, the results of the 2002 National Assessment of Educational Progress (NAEP) – the nation’s education “report card”— are disappointing, with only about a quarter of fourth grade students (27 percent) exhibiting proficiency in writing, with similar proficiency levels at 8th grade (31 percent) and 12th grade (24 percent). In the Pacific Northwest, overall fourth grade proficiency rates on the 2002 NAEP writing tests were even lower than the national average in three states: Idaho, Montana, and Oregon; all had proficiency rates of 22 percent (Washington had a proficiency rate of 30 percent; Alaska did not participate in state-level NAEP).

Proficiency rates on the NAEP writing test were much lower among at-risk groups. For students eligible for free or reduced price lunch, the proficiency rates ranged from 13 to 17 percent in Idaho, Montana, Oregon, and Washington, which is near the national average of 16 percent. Proficiency rates for black students were 14 percent in Oregon and 20 percent in Washington, compared to the national average for this group of 17 percent; for Hispanic students, the proficiency rates were 9 percent in Oregon, 11 percent in Idaho, and 12 percent in Washington, all below the national average of 17 percent for this group. American Indian and Alaska Native students, a subgroup encompassing 5 percent of all students in the Northwest region, had a proficiency rate of only 8 percent in Montana, well below the national average for this group of 15 percent.

In statewide achievement test results, students in the Northwest region demonstrate lower rates of proficiency in writing than in other aspects of language arts. For example, in the 2005 Oregon Statewide Assessment Testing (OSAT) results, 32 percent of Oregon fourth graders met state writing standards. This is less than half the rate of students meeting state reading and literature standards. Similarly, in the 2005 Idaho Direct Writing Assessment (DWA) results, 34 percent of fifth graders were proficient or advanced. This is less than half the rate of fifth graders meeting state reading standards. Only 17 percent of Idaho Hispanic students and 21 percent of American Indian students scored at the proficient or advanced levels in writing. In Alaska, a substantial achievement gap exists between Alaska Native and white students in writing. In fifth grade, 54 percent of Alaska Native students met standards in 2005, compared to 84 percent of white students. In sixth grade, 51 percent of Alaska Native students met standards in 2005, compared to 87 percent of white students.

These assessment results raise concerns because writing is a critical life skill and writing also supports the development of problem solving and thinking skills. There is a strong connection between writing and learning to think systematically (Sommers, 1982; Zinsser, 1988). While reading and mathematics are typically the core subjects of greatest concern to schools, writing also received a substantial emphasis in regional needs assessments (Barnett & Greenough, 2004; 2005).

This study is designed to test the effectiveness of an analytical trait-based model for teaching and assessing student writing, called 6+1 Trait® Writing, by examining its impact on the writing achievement of 5^th graders. The model is designed to improve student writing through an integrated approach to teaching and assessing writing skills, and it incorporates ten instructional strategies to develop the specific traits of writing. Key components of the model include effective feedback to students and the engagement of students in self-assessment.

The trait-based approach to writing instruction is a popular model in widespread use in the Northwest region and across the nation. Many school staff members, administrators, policy makers and parents view this as a valuable approach to the teaching of writing, and it is the subject of numerous publications and training programs. The model has been incorporated into major published language arts curriculum materials, guides to writing instruction provided by several major educational publishers, and a host of training workshops, Web sites, and other resources for schools.

However, this popularity is based largely on rational arguments for the underlying model, perceived face validity of the assessment instruments and training materials, and practitioner perceptions that the model has utility and improves student performance. Only two experimental studies have been completed examining the efficacy of the model (Arter, Spandel, Culham and Pollard, 1994; Kozlow & Bellamy, 2004), and these are insufficient to provide the broad research base that should underlie such a widely implemented approach. Therefore, the aim of this project is to provide a rigorous, relatively large scale scientific test of the approach, in conditions that are commensurate with the way the intervention is typically implemented in many school settings. The present study will improve on previous studies by involving a larger number of schools, strengthening implementation fidelity, and improving the research methodology, including properly modeling the nested data structure.

Given the widespread interest in this approach to writing instruction, it is important that the education community has access to high quality scientific evidence on the effectiveness of the model. Schools and policy makers will benefit from improved understanding of the extent to which the approach works, for whom, and under what conditions. The proposed research will contribute to that knowledge base, so that decisions about whether to expand or modify the adoption of this approach can be based on reliable data.

Improving student literacy is one of the goals of NCLB, including the goal of narrowing the achievement gap. The training for the 6+1 Trait® Writing aims at improving literacy of not only students in general but also student sub-populations such as English language learners. The training includes attention to stages of second-language acquisition and issues surrounding multicultural and bilingual education. Reflecting this, the study sample will include students in many of the reporting categories for NCLB to enable an exploration of the impact of the treatment with these student sub-populations.

Though writing is not a focus of the AYP provisions, it is a critical component of language and literacy. Many schools implement “writing across the curriculum” as integrative, interdisciplinary projects to raise student engagement and achievement in subjects beyond language and literacy – including mathematics, science and social studies – in their attempt to prepare students for advanced studies or careers. Writing is crucial for the development of higher order thinking skills and higher educational achievement. According to the U.S. Department of Labor’s Commission, writing is identified as one of the Foundation Skills on achieving necessary skills for a broad range of jobs. Better understanding of what works in the teaching of writing will benefit schools and students in their effort to meet the requirements of NCLB and beyond.

Although the 6+1 Trait® Writing is used widely in the Northwest and across the nation, it suffers from a relative paucity of well controlled studies demonstrating its effectiveness. There is a need for conducting a rigorous evaluation of this intervention, in conditions that are commensurate with the way it is typically implemented.

The five two measures for which clearance is sought are used for the proposed experimental study which fills the paucity of well controlled studies.

The Writing Essay isMeasures of student writing achievement are used to assess the impact of the intervention. Alaska State Writing Assessment (ASWA) was selected for its relevance to the participating schools in the Anchorage School District. NWREL Writing Essay Assessment was selected for itsin a manner that is in close alignment to the nature of the intervention. These two measures of student writing achievement will be complementary to each other.

Teacher surveys, teacher interviews, and classroom observations are used to collect information on the fidelity of implementation as well as the context in which the intervention was applied. Such information is vital for the accurate interpretation of the impact assessment.

Purposes and Uses of the Data

This information to be gathered in this study will be used to complete and report on an experimental study of the effectiveness of the 6+1 Trait® Writing model with 5^th graders in a sample of 64 schools in Oregon. Approximately half of the schools will begin during the 2007-08 school year and half the following year. From a pool of 64 schools, 32 will be randomly selected into the treatment group and the remaining 32 schools will serve as the control group. The teachers in the treatment group will receive training in the writing program, while control group teachers will receive no training until the data collection phase of the study is complete. This counterfactual condition will, therefore, represent the regularly planned writing curriculum. The data collection will provide educators and policy makers with better information on which to base decisions about the teaching of writing.

The overall goal of this study is to determine if the 6+1 Trait® Writing model can improve the writing achievement of 5^th graders. Specifically, the study is designed to answer the following research questions:

What is the impact of 6+1 Trait Writing on student achievement in writing?

How do student impacts vary by pre-existing characteristics of schools, teachers, and students?

In addition, descriptive studies of both treatment and control classrooms will be conducted to help interpret and understand the results of the experimental research questions, including the fidelity of treatment implementation and the differences and similarities between treatment and control classrooms. The questions with respect to implementation are the following:

To what extent did teachers in the treatment group implement the intervention with fidelity to the intended model?

What differences and similarities exist with respect to implementation of writing instruction between treatment and control classrooms?

Data to answer these questions will come from three sources – student essays, a teacher survey, and a brief listing of coded, non-personally identifiable student indicators related to completion of the essay assessment – the instruments for which clearance is requested in this submission.

Student Outcome Measure. Student achievement in writing will be measured using the Writing Essay Assessment modeled very closely after the Oregon State Writing Assessment. Baseline writing samples will be collected in September or early October of each data collection year; follow-up samples will be collected the following May. All prompts for the writing assessment will be in the expository mode for consistency of performance comparison across all subgroups of students. Students will work on their essays for 45 minutes on each of 3 successive days, to provide the opportunity for a natural writing process including planning, drafting, and revision. The intervention is largely aimed at improving the willingness and ability of students to substantively revise their writing after reflection upon first drafts; therefore, it is not possible to estimate the impact of the intervention based on an assessment of first drafts produced during a single writing session. The essay writing sessions will be proctored by the participating teachers in both control and treatment groups. The student essays will be scored by teams of raters, using a process parallel to that used in the 4^th and 7^th grade Oregon statewide writing assessment. Active parental consent is not required for this study; no individually identifiable data will be released to researchers, there is no more than minimal risk to children, and the procedures involve normal and widely used educational activities that would not require informed consent outside of the research context.

Teacher Implementation Measure. The teacher survey will be administered three times during the treatment year to all teachers in both treatment and control schools. The survey contains Likert-type questions about teacher practices, descriptive information about use of instructional time, and open-ended questions about instructional and learning issues. This will provide data on both fidelity of implementation in the treatment group schools and contamination in the control group schools. As described later in this document, these data will be used in a rigorous analysis to estimate the impact of the 6+1 Trait® Writing model on student achievement.

Coded Student Data Form. This spreadsheet will be used to insure that student essays in the fall and spring are coded with a consistent ID number (created just for the study) before being sent to the research team. In addition, this form will be used to code whether students complete the essay according to the regular protocol, and to note student gender and race/ethnicity.

Student outcome data collected through the Alaska State Writing Assessment (ASWA) and NWREL Writing Essay Assessment will be used to estimate the effect of the intervention on student achievement. Appropriate statistical tests will be used to take into account the nested, multilevel structure of the data. Additional analysis of student outcomes will examine the role of pre-existing teacher and student characteristics as moderators of program effects.

Teacher-level data collected through teacher surveys, teacher interviews, and classroom observations will be examined for the analysis of implementation fidelity. These data will be used also for fully understanding the treatment and its counterfactual states. Because of its popularity in the region, it is likely that many teachers at control group schools will have some experience with the 6+1 Trait® Writing. In order to address the issue of prior exposure, it will be necessary to collect information from teachers in the control group schools concerning their prior experience with trait-based writing models in general and the 6+1 Trait® Writing in particular.

Teacher surveys will contain Likert-type questions about teacher practices and opinions, descriptive information about use of instructional time, and open-ended questions about instructional and learning issues. These data will be used to describe the treatment and control teachers and their classrooms.

Through teacher interviews, data will be collected on teachers’ instructional and assessment practices, curriculum design, and specific learning objectives for students. These data will be compared to the classroom observation data, and used to build a complete picture of classroom learning.

Through classroom observations, data will be collected on how teachers implement a specific lesson in their classrooms. Observers will be instructed to make note of specific aspects of classroom environment, teacher instructional/assessment activities, and student engagement. These data will correspond to the ten 6+1 Trait® Writing instructional strategies.

2. Use of Technology to Reduce Burden

Information on relevant characteristics of the selected study schools will be obtained from existing electronic data sources and therefore will not require a burden on school personnel; research staff will be responsible for this task. The primary school-level covariate will be the schoolwide aggregate student performance on the 4^th grade writing component of the Oregon statewide assessment system. Subsequent analyses will test the effect of additional demographic covariates at the school and teacher levels. These data are readily available from electronically available public records.

The Alaska State Writing Assessment is a state standard-based test; therefore, there will be no additional respondent burden associated with the use of its data as measures of student outcome.

NWRELThe Writing Essay Assessment will be administered as a paper-and-pencil test. Computer-based assessment at this grade level is not ideal, as it may introduce a confounding due to variations among students in their experience in writing with the computer, including their ability to use a keyboard.

The teacher survey will also be administered as a paper form. This short survey, which will be used three times during the year, takes the same amount of time in either format, but the paper version gives teachers greater flexibility in when and where they complete the survey.

An electronic spreadsheet will be provided to sites for the management and collection of student individual data, which will be in a coded form so that researchers will never have identifiable information for each student. The forms will be emailed to the research team. >>Verify this last statement.>>Verify this statement.

Teacher interviews and classroom observations will be conducted in person at the school. We believe that important contextual information will be lost, unless observers are physically present in the classroom to conduct interviews and observations. As such, no tele- or video-conferencing is planned for teacher interviews. Likewise, no classroom observation will be based on a video-recorded data.

3. Efforts to Identify Duplication

The Alaska State Writing Assessment (ASWA) was selected as one of the student outcome measure, so as not to duplicate the assessment that was already in place at school. Student scores on the ASWA will be sent directly to us from Alaska Department of Education and Early Development (AKEED).

NWRELThe Writing Essay Assessment, though alsoemploys a similar administration process as the Oregon State Writing Assessment (OSWA). However, it is not a duplication of effort since the OSWA ais not administered to 5^th grade students in Oregon. The Teacher Surveys are specific to teacher background and practices relevant to this study and do not duplicate existing data measures. measure of student writing, is quite different from ASWA in several ways. For instance, whereas ASWA relies heavily on multiple-choice items, NWREL Writing Essay Assessment involves more actual writing. These two are also different in that NWREL Writing Essay Assessment produces assessment data that are aligned to the trait-based writing model. As such, NWREL Writing Essay Assessment should not be considered as a duplication of ASWA. These two measures are complementary to each other.

>> Describe how (1) Teacher Survey, (2) Teacher Interview, and (3) Classroom Observation, differ from one another in terms of the type of information collected. MH had difficulty figuring that out. Since this section is about “efforts to identify duplication,” the usual argument based on triangulation will not fly.

4. Methods to Minimize Burden on Small Entities

A number of procedures have been designed to minimize the data collection burden on schools and teachers participating in this study. District personnel involved in the study will be trained on the procedures and provided with explicit documentation of the necessary steps to facilitate the collection and processing of data. Documentation will include check-off lists of study activities and forms to keep records. Project staff will be available by e-mail and phone to answer questions and will initiate contact monthly to identify and address potential problems.

No data will be collected from small businesses or other small entities. The Alaska State Writing Assessment (ASWA) will be administered to the students as a part of the state standard-based test. We will obtain ASWA data directly from Alaska Department of Education and Early Development (AKEED). NWRELThe Writing Essay Assessment will be administered to the students. Teacher surveys, teacher interviews, and classroom observations will be used to collect data from teachers.

Consequences of Not Collecting the Data

As noted above, the 6+1 Trait® Writing model and similar instructional methods are currently being used in a relatively large number of schools, yet the effectiveness of this approach has not been established using a scientifically rigorous method. The current study will enable a scientific test of the model that meets the current scientific standards and priorities of the US Department of Education’s Institute of Education Sciences (IES). The study will be a cluster-randomized trial (CRT), which enables a test of causality. Without the study, schools and districts will continue to make decisions about whether to use this model based only on preliminary, non-experimental evidence of its effectiveness. That would hinder the current effort of NCLB and IES – promoting the use of interventions that have been tested for effectiveness in experimental studies, such as randomized-controlled trials (RCTs) and cluster-randomized trials (CRTs).

Assessment of student writing is the primary outcome measure in this study. Without this measure, the effectiveness of the 6+1 Trait® Writing could not be tested. Information on teacher classroom practices is necessary for understanding the difference between the treatment and its counterfacturals which will be represented by the classrooms in the control condition. Information on teacher practice is also important for gaining insight into the context in which the treatment is applied. Without these pieces of information, an accurate interpretation of the outcome data will not be possible.

Although a vary popular model of writing instruction, the effectiveness of the 6+1 Trait® Writing has not been scientifically established in the past studies. The current study will enable a scientific test of the model that is at the same time fair and rigorous. The study will involve a sufficient number of schools and better training/support for teachers. The study will be a cluster-randomized trial (CRT), which enables a test of causality. Without the study, schools and districts will continue to use the instruction model with no scientific evidence of its effectiveness. That would hinder the current effort of IES – promoting the use of intervention that has been tested for its effectiveness in experimental studies, such as randomized-controlled trials (RCTs) and cluster-randomized trials (CRTs).

5. Special Circumstances

No special circumstances will exist in connection to the data collection for this study.

6. Federal Register Comments and Persons Consulted Outside the Agency

The notice for the data collection was announced on page 17526 in the April 9, 2007 issue of the Federal Register. No public comments were received in response to this listing.

The study team has drawn on the methodological expertise of two outside consultants – Michael Puma and David Connell from Chesapeake Research Associates, LLC – in planning the current study, including the design, instrumentation, and plans for analysis. Additional review of the study design has been obtained from members of the external technical working group (TWG). The names and institutional affiliations of the TWG members appear in Exhibit 1.

In addition, the study design has gone through extensive review by staff of the Analytical and Technical Support (ATS) group contracted by IES to review all of the research being conducted by the 10 Regional Educational Laboratories.

The study team has drawn on the methodological expertise of the Chesapeake Research Associates in planning the current study, including the design, the instrumentation, and the plan for the analysis. Our technical working group (TWG) has also reviewed, and made extensive suggestions to, the planning of the current study. The names and institutional affiliations of the TWG members appear in Exhibit 1.

Exhibit 1. Methodology Consultants and Technical Working Group Members

Methodology Consultants: Chesapeake Research Associates
Mike Puma
Dave Connell
Technical Working Group
Ray Barnhardt (University of Alaska)
Hans Bos (Berkeley Policy Associates)
Audrey Champagne (SUNY Albany)
Bill Demmert (Western Washington University)
Allen Glenn (University of Washington)
Dan Goldhaber (University of Washington)
Brian Gong (National Center for the Improvement of Educational Assessment)
Joan Herman (University of California, Los Angeles)
Michael Kamil (Stanford University)
LeAnne Robinson (Western Washington University)
Lynn Santelmann (Portland State University)
Sam Stringfield (University of Louisville)

7. Payment or Gifts

No gifts or incentives will be provided to respondents for their completion of specific data collection activities, except that providing the intervention itself to both treatment and control groups could potentially be viewed as an incentive. The intervention cost is dependent on the number of teachers a school wishes to have trained. For example, if a school wished to train their 5th grade teachers in a design similar to that of this study, the cost would be $600 per trainee for the three day summer institute, plus $300 per trainee for the three days of follow-up training and planning during the school year, plus any salary, benefits, travel and per diem costs associated with the specific district, contract year, and locale.

8. Assurances of Confidentiality

NWREL follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). NWREL will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution but not to any individually identifiable information. No individually identifiable information will be maintained by the study team. All institution-level identifiable information will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required. NWREL obtains signed NCEE Affidavits of Nondisclosure from all employees, subcontractors, and consultants that may have access to this data and submits them to our NCEE COR. All members of the study team having access to the institution-level data have been certified by [Name's] Institutional Review Board as having received training in the importance of confidentiality and data security.

Study participants will be assured that any information obtained for the purpose of the study will be held confidential to the extent allowed by law, and that any information obtained for the study will be used by NWREL and the U.S. Department of Education solely for evaluating the effectiveness of the 6+1 Trait® Writing intervention. All data collection activities will comply with the Privacy Act of 1974, the Family Educational and Privacy Act of 1974, and related regulations, including regulations on protection of human research subjects provided in 34 CFR Part 97. All NWREL staff members working with the data have been trained in the application of these laws and guidelines.

NWREL will ensure that only the staff members directly involved with the study will have access to the information. NWREL will receive student information from districts that is coded by ID numbers created for the study; the research team will not receive any student names or personally identifiable data. Student essay booklets and teacher survey forms will be stored in a locked file cabinet during the study and will be destroyed at the end of the study. Electronic files will be maintained in a secure, password-protected network environment.

The study team will communicate the following assurances of confidentiality to the participating schools.

1 NWREL will acknowledge that information obtained through Alaska State Writing Assessment (ASWA), NWRELthe Writing Essay Assessment and , Teacher Surveys, Teacher Interviews, and Classroom Observations, is considered confidential. As such, any information that NWREL obtains for the purpose of the study will be used solely for evaluating the effectiveness of the 6+1 Trait® Writing. NWREL will ensure that only the staff members directly involved with the study will have access to the information, and the database containing the information will be destroyed upon completion of the study. NWREL will further ensure that the results from the study will be reported only in aggregate for quantitative data or in anonymity for qualitative data.

Justification on Sensitive Questions

No sensitive information will be collected through the instruments for which clearance is sought.

9. Estimates of Hour Burden

Alaska State Writing Assessment

Students take the Alaska State Writing Assessment once a year in May as a part of their required state standard-based assessment. No hour burden is expected beyond what is considered as this customary and usual practice at schooFifth grade students at participating schools (i.e., both the treatment and the control condition schools) will take NWREL the Writing Essay Assessment twice during the year of the study. Each administration requires three 45-minute class periods. Treatment condition teachers and control condition teachers will take the brief teacher survey twicethree times: near the beginning of the study year, again in the middle of the year, and again near the end of the year. Each teacher will also provide a limited set of information on each student, which will be coded so that student identities are not revealed. Hour burden is summarized in Exhibit 2 for each year data collection period.

1. Teacher Interview Protocol

A representative sample of teachers, about 20 from each of the conditions, will be interviewed three times during the study year. Those teacher interviews will take place at the time when the NWREL observer visits their classrooms. Hour burden for this measurement is summarized in Exhibit 2.

2. Classroom Observation Protocol

Classroom observations and teacher interviews are conducted as a set, three times during the study year. As such, classroom observations will be conducted for the same sample of teachers who will be interviewed. Hour burden for this measurement is summarized in Exhibit 2.

Exhibit 2. Annual Hour Burden for Respondents

Measure	Expected Number of Respondents Per Year			Average Hour Burden Per Administration			Frequency of Administration Per Year			Total Hour Burden Per Year			Average Respondent Wage Rate			Total Cost Per Year
Alaska State Writing Assessment			1,320			0			1			0			N/A			0
NWREL Student Essay		TBD1,536			TBD2.25			2			TBD6912			N/A			0
Teacher Survey		64			.5			3			96			$28			$2,688
Coded Student Data Form		64			. 5			2			64			$28			$1,792
Total		1,600			N/A			N/A			7072			N/A			$4,480
Teacher Interview			40			TBD			3			TBD			TBD			TBD
Classroom Observation			40			TBD			3			TBD			TBD			TBD

10. Estimate of Cost Burden to Respondents

Aside from the hour burden detailed in Item #12, no additional cost burden to respondents is expected.

11. Estimate of Annual Cost to the Federal Government

The total budget for the study is $2,982,260 over the five years, making the average annual budget of the study $596,452.

12. Program Changes or Adjustments

The information collection activities described in this request are all new and therefore do not represent a change or modification to a previously submitted Clearance request.

The information collection activities described in this request are all new. Collectively, these will result in a program change of X hours. >>fill-in the numbers.

13. Plans for Tabulation and Publication of Results

Data analyses will take place during the Year 4 of the study, in 2009/10. Dissemination of the results will take place in the Year 5, in 2010/11.

Student outcome data will be entirely quantitative, and they will be analyzed using the hierarchical linear modeling (HLM) to reflect the nested structure in the data. For the analysis of the difference between the treatment and control groups in student achievement, HLM will be reduced to a mixed model ANCOVA in which the effect of the experimental manipulation is estimated as a fixed effect, while the effects of school level variables and the individual differences among teachers and students will be estimated as random effects. Details are provided in section B.2.

Alternatively, student outcome data could be analyzed using more traditional fixed-effects ANCOVA for nested designs. Currently, NWREL and the IES are weighing the advantages and disadvantages of these two analytical options.

Teacher implementation data will have both quantitative and qualitative components. The first stage of the analysis will assess the degree to which the quantitative items from the teacher survey, classroom observations, and interview data converge. The pattern of correlations among data from these sources, collected from a sub-sample of teachers, will provide evidence on the validity of the survey data that will be available from the full sample of treatment and control group teachers.

Data from the teacher surveys will be analyzed The second stage of the analysis will compare practices in the treatment and control classrooms, again using the quantitative items from the three measurement instruments. Based on the degree of convergence between the teacher survey, classroom observations, and interview data, these may be combined into a single index of overall implementation fidelity and a set of indices of implementation fidelity related to key components of the intervention. Scores on these indices will be derived from combined information from all three data sources from some teachers, but will be derived from the survey data only for the remaining teachers. If it is not feasible to combine the data sources in this way, they will be analyzed and reported separately.

Qualitative data from open-ended questions and observations will be analyzed thematically, reported and interpreted in relation to the quantitative items, in order to further characterize the fidelity of trait-based writing implementation in the treatment schools, and to describe classroom conditions and practices in both treatment and the control schools. These analyses will help contextualize the student achievement results.

The time schedule for the entire project is shown in Exhibit 3.Exhibit 3. Time Schedule for the Study of Traits-Based Writing Model

Activity

Exhibit 3. Study Timeline

Contract Year 1

Contract Year 2

Contract Year 3

Contract Year 4

Contract Year 5

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

Q1

Q2

Q3

Q4

2006

2007

2008

2009

2010

Feb thru April

May thru July

Aug thru Oct

Nov thru Jan

Feb thru April

May thru July

Aug thru Oct

Nov thru Jan

Feb thru April

May thru July

Aug thru Oct

Nov thru Jan

Feb thru April

May thru July

Aug thru Oct

Nov thru Jan

Feb thru April

May thru July

Aug thru Oct

Nov thru Jan

2006-07 school year

2007-08 school year

2008-09 school year

2009-10 school year

Refine design; approval by TWG & IES





x

Refine Web support procedures









x

Refine and pilot data collection instruments





x

Obtain IRB and OMB clearance





x

Wave 1

Recruit sites and assign teachers





x

Collect baseline data on teacher practices



x

Train & support treatment group teachers









x

Student pre-assessments

x

Post-assessments of teacher practices



x

Student post-assessments

x

Train & support control group teachers









x

Wave 2

Recruit sites and assign teachers













x

Collect baseline data on teacher practices



x

Train & support treatment group teachers









x

Student pre-assessments

x

Post-assessments of teacher practices



x

Student post-assessments

x

Train & support control group teachers









x

Analyze data and prepare reports















x

Disseminate reports







Activity

Year/Quarter

Year 1

Year 2

Year 3

Year 4

Year 5

1

2

3

4

1

2

3

4

1

2

3

4

1

2

3

4

1

2

3

4

Refine research design; approval by TWG & IES





x

Recruit and assign sites







x

Prepare coaches









x

Refine Web support procedures





















Refine and pilot data collection instruments







Obtain NWREL IRB clearance



x

Obtain OMB clearance





x

Collect baseline data on teacher practices



x

Train teachers

x

x

Administer student pre-assessments

x

Implement traits model including ongoing coaching and collect data on level of implementation

















Administer post-assessments of teacher practices





x

Administer student post-assessments

x

Train control teachers

x

Analyze data and prepare report for impact of 2-year implementation, including appropriate IES review







x

Dissemination of reports







14. Approval to Not Display OMB Expiration Date

Exemption from the required display of the OMB expiration date is not requested. All the data collection instruments will include the OMB expiration date.

15. Explanation of Exceptions

No exceptions are requested.

REFERENCES

Arter, J.A., Spandel, V., Culham, R., & Pollard, J. (1994, April). The impact of training students to be self-assessors of writing. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. (ERIC Document Reproduction Service No. ED370975).

Barnett, L. & Greenough, R. (2004). Regional needs assessment 2004: Northwest educators’ priorities for improving low-performing schools. Portland, OR: Northwest Regional Educational Laboratory.

Barnett, L. & Greenough, R. (2005). Regional needs assessment 2005: Northwest educators’ priorities for improving low-performing schools. Portland, OR: Northwest Regional Educational Laboratory.

Kozlow, M., & Bellamy, P. (2004). Experimental study on the impact of the 6+1 Trait^® Writing model on student achievement in writing. Unpublished manuscript, Northwest Regional Educational Laboratory, Portland, OR.

Sommers, N. (1982). Responding to student writing. College Composition and Communication, 33(2), 148-156.

U.S. Department of Labor (1991). What Work Requires of Schools: A SCANS Report for America 2000. The Secretary's Commission on Achieving Necessary Skills.

Zinsser, W. (1988). Writing to learn. New York, NY: Harper