Collaborative Strategic
Reading Study
OMB Clearance Request—Part A
Supporting materials
March 2007
Prepared For:
Institute of Education Sciences
United States Department of Education
Contract No. ED‑06‑CO‑0017
Prepared By:
Regional Educational Laboratory—Southwest
Edvance Research, Inc.
9901 IH‑10 West, Suite 700
San Antonio, Texas 78230
(210) 558‑1902
(210) 558‑1075 (fax)
Contents
Supporting Statement for Paperwork Reduction Act Submission: Section A 2
A. Justification 2
1. Circumstances Making Collection of Information Necessary 2
Rationale 2
Overview of the Study Design 2
Overview of Data Collection Plan 2
Recruitment 2
District Screening Process 2
School Screening Protocol 2
Student Background Data Request 2
Baseline Data 2
Students 2
Group Reading Assessment and Diagnostic Evaluation (GRADE). 2
Teachers 2
Teacher Surveys. 2
Implementation 2
Study Staff 2
Collaborative Strategic Reading Implementation Validity Checklist (CSRIVC). 2
Expository Reading Comprehension Classroom Observation (ERCCO). 2
Outcomes 2
Students 2
Group Reading Assessment and Diagnostic Evaluation. 2
Timeline of Data Collection Plan 2
2. Purposes and Uses of the Data 2
3. Use of Technology to Reduce Burden 2
4. Efforts to Identify Duplication 2
Recruitment 2
Implementation 2
Teacher Background/Classroom Context 2
Student Outcomes 2
Contents (Continued)
5. Methods to Minimize Burden on Small Entities 2
6. Consequences of Not Collecting the Data 2
7. Special Circumstances 2
8. Federal Register Comments and Persons Consulted Outside the Agency 2
9. Payment or Gifts 2
10. Assurances of Confidentiality 2
Consent Forms 2
11. Justification of Sensitive Questions 2
12. Estimates of Hour Burden 2
13. Estimate of Cost Burden to Respondents 2
14. Estimate of Annual Cost to the Federal Government 2
15. Program Changes or Adjustments 2
16. Plans for Tabulation and Publication of Results 2
17. Approval to Not Display OMB Expiration Date 2
18. Explanation of Exceptions 2
List of Exhibits
Exhibit 1. Study Design 2
Exhibit 2. School Information Sheet Item Level Justification 2
Exhibit 3. Student Data Request Item Level Justification 2
Exhibit 4. Fall Teacher Survey Item Level Justification 2
Exhibit 5. Spring Teacher Survey Item Level Justification 2
Exhibit 6. Data Collection Instruments and Schedule 2
Exhibit 7. Technical Working Group Members of the CSR Intervention Study 2
Exhibit 8a. District and School Hour Burden,
Combined Burden Estimates for
Phases 1 and 2 2
Exhibit 8b. Teacher and Student Hour Burden,
Combined Burden Estimates for
Phases 1 and 2 2
Exhibit 9. Estimated Annualized Costs 2
Exhibit 10. Schedule for Dissemination of Study Results 2
Elementary school English Language Learners (ELLs) are at‑risk for academic failure and there is evidence that the number of these students is on the rise in the Southwestern region of the United States. Meanwhile, reviews of the literature suggest there are few empirically supported interventions that meet their academic needs. One promising intervention is Collaborative Strategic Reading (CSR), which can be delivered at the classroom level via teacher training. As the name implies, CSR focuses on reading comprehension and has students collaborate with each other when learning new material. CSR combines essential reading comprehension strategies that have been demonstrated as effective in improving students’ understanding of text (Palincsar & Brown, 1984; Rosenshine & Meister, 1994) with the use of cooperative learning groups (Johnson & Johnson, 1989; Kagan, 1990). During a CSR session, students of varying reading abilities work in small groups to help each other apply four critical reading comprehension strategies (preview, click and clunk, get the gist, wrap up strategies) to comprehend expository text. The vast majority of the studies on CSR have investigated the program impacts on students’ comprehension of social studies text. The proposed study of the effectiveness of the CSR will focus on three questions: (1) Do CSR classrooms outperform control classrooms on a measure of reading achievement? (2) What are the impacts for ELL students? (3) What are the impacts for native English speakers?
CSR incorporates reading comprehension strategies and cooperative learning techniques. Students are placed in small learning groups and are presented with a reading passage. They then engage in a four part approach to the material. Prior to reading the passage, students preview the text to determine what they know and what they think they are going to learn. Secondly, they are also trained to recognize when they are comprehending material and when they are not. In CSR parlance, students are told that understanding material means the concepts are “clicking.” If students are experiencing difficulties comprehending material, they are “clunking.” The Click and Clunk strategy in CSR is designed to help students monitor their comprehension by identifying difficult vocabulary and conceptual information. Students use the “fix‑up” strategies, which are based on context and structural analysis to determine meaning. Thirdly, after reading a portion of the text, students use the “Get the Gist” strategy to ensure they grasp main idea. Wrap up consists of two parts. First students ask and answer recall questions, as well as questions that require them to use background knowledge to move “beyond the text.” Students then write a summary of the content.
The Assessing the Impact of Collaborative Strategic Reading on Fifth Graders’ Comprehension and Vocabulary Skills study (hereafter CSR Study) is the first large-scale randomized control trial testing the effectiveness of CSR. This study targets 5th grade teachers who teach both English/language arts (ELA) and social studies. The goal of the study is to establish whether CSR intervention increases reading comprehension and vocabulary skills of 5th graders, for both native English speakers and English Language Learners. The 4th and 5th grades are particularly troublesome for ELL students because the curriculum becomes more subject‑matter specific. This study will assess impacts within social studies curricula because of the greater focus of specific subject‑matter in the later elementary grades, and because these curricula tend to be taught primarily in English. Moreover, given the lack of true experimental data on CSR, it is important that the content area in this study be consistent with prior research. Assessing impacts on the comprehension of social studies text also provides the chance to evaluate whether or not the intervention has an impact on general reading abilities as well as within a specific academic area.
All research activities refrred to in this document, including data collection by Regional Educational Laboratories, are authorized by H.R. 3801 The Education Sciences Reform Act of 2002 (Part D, Section 174). (See Appendix C).
There is some evidence suggesting that CSR can be an effective intervention for ELL 5th graders (e.g., Klingner & Vaughn, 2000; Klingner, Vaughn, & Schumm, 1998). Many school districts in the Southwestern Region of the United States serve a large number of ELLs. Indeed, Texas serves the second highest enrollment of ELLs among all states (California is first) and its neighboring states have shown marked increases in their ELL populations (National Clearinghouse for English Language Acquisition, 2002; Ruiz‑de‑Velasco & Fix, 2000). ELL students represent an academically at‑risk group (Baca & Cervantes, 1989; Moss & Puma, 1995), and therefore, research into better ways of serving ELL students is especially important. Unfortunately, there are relatively few replicable interventions designed to meet the needs of these students, and even fewer that have been rigorously evaluated: Prior studies have only used quasi‑experimental designs to assess program impacts, or qualitative methodologies to describe how the intervention works (e.g., Klingner & Vaughn, 2000; Klingner, Vaughn, Hughes, & Arguelles, 1999; Klingner, Vaughn, & Schumm, 1998).
The No Child Left Behind (NCLB) legislation emphasizes raising achievement in reading for all students, including those with special learning needs like ELLs. Since very few studies in the research literature address ELLs, states and districts need evidence‑based curricula and strategies that facilitate reading achievement for these students. Additionally, 5th grade can be a difficult time in the academic development of ELL students because many are moved from bilingual to English immersion classrooms at that time. The CSR study’s focus on improving reading achievement particularly for ELLs in this age group will help inform educators about ways to provide successful transitions into English‑only instruction while avoiding compromising students’ literacy development. Consistent with the larger goals of NCLB, this study is a rigorous, large‑scale evaluation of the impact of a curricular intervention; and the study focuses on the reading achievement of students who are academecially at risk. Results of this study will inform teachers, administrators, and researchers about the effectiveness of one promising approach to reading instruction for ELL students.
IES and its subcontractors have designed a randomized field trial to examine the impact of CSR on students’ reading abilities. This study, focusing on 5th grade teachers and students in school districts/schools where many students are ELLs, will examine two experimental conditions:
Treatment: Teachers will be trained to use CSR prior to the
beginning of the school year and will receive subsequent coaching
throughout the school year. Before students are placed in CSR groups,
it is imperative that they are able to implement the strategies with
a minimum of teacher assistance and to function as a cohesive group.
To that end, the teachers in the treatment condition will teach their
students the four reading comprehension strategies and the roles and
responsibilities of CSR group members during the first three weeks of
the intervention using whole group instruction. For the remainder of
the intervention, heterogeneous groups of 4–6 students will
implement the CSR strategies approximately 3 times per week in their
social studies class, for
30–40 minutes using the
adopted basal social studies textbook. Note that the CSR intervention
can be used with any existing textbook series we expect to see in the
sampling pool.
Control: Using the same adopted social studies textbook as the students in the treatment condition, the control condition will consist of the instructional techniques that the teacher normally uses. The pedagogy in elementary social studies classrooms typically consists of Round Robin reading, where students take turns reading a portion of the text aloud, followed by answering questions at the end of the section or chapter and/or completing an assignment from a workbook or supplementary activity sheet provided by the publisher.
The underlying theory of action suggests that the CSR intervention and the professional development related to it will increase teachers’ knowledge about reading comprehension instruction and consequently affect teacher practice in the classroom during social studies. The changes in teacher practice (high level of implementation of CSR) as well as the direct impact of students using CSR, are in turn assumed to affect students’ reading comprehension and vocabulary development.
The underlying theory of the intervention suggests that the strongest program impacts will be realized only if there is a proper application of both explicit teaching strategies that focus on comprehension and cooperative learning techniques. Delivery therefore requires professional development and subsequent coaching for proper implementation. It is likely that simple knowledge transfer via training will not lead to the changes in pedagogy needed for CSR to work. CSR is a classroom-level intervention and cross-teacher spillover within schools is of limited concern. The nature of the CSR delivery requires that proper group instruction take place at the beginning of the intervention. Furthermore, CSR entails a specific mix of cooperative learning and explicit reading comprehension instruction in order for groups to function with only minimal teacher monitoring. It would be unlikely that the control teachers can learn to implement the intervention even after extensive conversation with experimental teachers.
The CSR intervention will be implemented during the 2007–2008 school year for phase 1, and during the 2008–2009 school year for phase 2. Teachers and students participating in each phase of the study will be recruited from different schools. Data will be collected on the implementation and impact of the interventions during each school year. Details about the treatment procedures, the design of the study, and data collection are described in more detail below.
This study will use cluster randomized control trial (RCT) with one treatment and one control group. Students will be randomly assigned to classrooms, and classrooms/teachers will be assigned to treatment/control conditions within schools. Random assignment of students significantly decreases classroom‑level clustering, creating a statistically powerful design. Treatment and control classrooms will be distributed evenly in schools such that each school will have the same number of treatment and control classrooms. The study will analyze posttest data with regression adjustment for baseline values. The selected design treats sites (schools) as random effects.
Given the desired Minimum Detectable Effect Size (MDES; Bloom, 2005) between 0.15 and 0.20, it has been determined that a sample of 80 classrooms from 20 schools will yield adequate statistical power. Each phase of the study will consist of 40 classrooms: 20 treatment and 20 control. The sample will be drawn from the Southwest region, specifically urban, high‑poverty school districts with high proportions of ELL students.
To be included in the study’s sample, classrooms must be serving 5th grade English language learners at the time of the study. Participating teachers must be willing implement the intervention in the treatment group, and be willing to participate in the data collection activities. Individual classrooms should offer both English Language/Arts and social studies curricula. Each school must have at least two classrooms participating in the study, so that each school has at minimum, one treatment and one control classroom. Schools will be excluded if they do not agree to this condition, and if they do not serve 5th grade ELLs. Compartmentalization of the ELA and social studies subjects into separate classrooms will likely bar including the school in the study.
This study has two phases, each phase including a different set of schools, teachers and students (see Exhibit 1). Schools, teachers and students for phase 1 will be recruited in spring of 2007, and the study will commence during the 2007–2008 school year. Phase 1 teachers will receive CSR training in the late summer of 2007. Phase 1 students will be pre‑tested in the fall of 2007 and post‑tested in spring 2008. Classroom observations of phase 1 implementation will occur in fall 2007 and in spring 2008. Phase 2 will be recruited in spring of 2008 for the 2008–2009 school year. Phase 2 teachers will receive CSR training in the late summer of 2008, and phase 2 students will be pre‑tested in the fall of 2008 and post‑tested in spring 2009. Classroom observations of phase 2 implementation will occur in fall 2008 and in spring 2009.
Phase of the Study |
Number of Schools |
Experimental Conditions |
Number of Classrooms |
Number of Students
|
1 |
10 |
Treatment |
20 |
500 |
Control |
20 |
500 |
||
2 |
10 |
Treatment |
20 |
500 |
Control |
20 |
500 |
||
Total |
20 |
|
80 |
2,000 |
The intervention consists of four aspects: (1) teachers receiving training to use the CSR intervention effectively, (2) teachers teaching students to use CSR comprehension strategies and to work cooperatively and successfully in CSR groups, (3) coaches supporting teacher learning and implementation of CSR, and (4) students successfully using CSR learning strategies in cooperative small groups. In order for students to implement the strategies in the CSR groups, their teachers need to be able to teach these strategies well. Consequently, the training that teachers will receive as a part of the intervention is extremely important.
Intervention teachers will be trained by project staff with CSR experience in how to implement the intervention. The CSR training underscores the importance of using explicit reading comprehension instruction as the vehicle to teach the metacognitive and self‑regulation knowledge students need to comprehend text. Explicit comprehension instruction is based on Vygotsky’s (1978) research using scaffolded instruction (i.e., modeling, teacher‑assisted and independent practice) and thinking aloud. The teachers learn and practice the instructional technique. They will also be given a CSR manual containing explicit lessons in the four comprehension strategies (Preview, Click and Clunk, Get the Gist, Wrap Up) whose design is based on the tenets of scaffolded instruction and thinking aloud (Klingner, Vaughn, Dimino, Schumm, & Bryant, 2001). Each student has a role in the CSR group (Leader, Timekeeper, Clunk Expert, Gist Expert, Scorekeeper, Encourager). The second aspect of the training involves teaching students the CSR cooperative learning roles and how to function as a group. It is imperative for students to understand the purpose and function of the roles to ensure an effective use of the comprehension strategies and an efficient use of time.
To promote a high level of implementation of the CSR intervention, each treatment teacher will be provided additional school‑based coaching. It is crucial that a coach visit each classroom during the first three weeks of the intervention to ensure that explicit reading comprehension instruction is being implemented to teach the four CSR strategies. After the students begin working in groups, the coach will visit each classroom during the second, third and fourth month of the intervention.
This section describes the full set of data collection instruments. The data collections for the CSR Study serve four broad purposes:
Recruitment, to identify schools that will participate in the study;
Documenting the intervention, to verify the fidelity with which CSR was implemented;
Assembling contextual data, to help understand the results:
data to describe the sample so that differential outcomes for groups might be elucidated;
data to compare treatment and control schools, teachers, and students prior to the implementation (i.e. to assess how well randomization has balanced the samples);
covariates (control variables) that can be included in analyses to reduce unexplained variance;
variables that may interact with the treatment.
Measuring the outcome, student reading achievement.
To address these objectives the study will collect data from the following types of respondents: principals, teachers, and students. In addition, study staff, including the CSR coaches and evaluation staff, will use data collection instruments designed to document CSR implementation and gather information from teachers.
District screening process does not include systematic data collection from school district personnel.
Initial district identification and recruitment efforts will utilize existing data sources. Districts with characteristics that meet study criteria will be identified using data from Market Data Retrieval (MDR), which utilizes publicly available datasets to describe characteristics of regions and communities. MDR will be provided with criteria, such as a high proportion of ELL students and four or more 5th grade classrooms in schools, that will allow them to produce a list of districts in the Southwest region that match study eligibility requirements. Study staff will then utilize their state contacts to learn which of the identified districts are good fits for the study. Study staff will identify key personnel in each district and attempt to make contact by telephone and mail (see Appendix A: Letters). The purpose of the contact with district personnel is to seek permission to contact and recruit schools in the district. The approval will be formalized with a Partnering Expectations Document (see Appendix B: Supporting Materials). Additionally, study staff will learn whether the district has an IRB or other research board that needs to approve the study activities and initiate that process, if necessary.
Once a school district has shown interest in participating in the study, we will use existing data sources, such as Common Core Data (CCD) and publicly available school report cards to enhance the identification of a potential pool of participating schools developed with the MDR data. We will approach schools after gaining documented approval from district‑level staff. School principals will be contacted first by mail, and then by telephone. Though initial recruitment efforts will confirm the schools’ eligibility based on existing data, each school principal will be asked to update/correct our information by filling out a School Information Sheet with our assistance if the school is willing to participate in the study. See Exhibit 2 for item‑level justification of this instrument. The form may be filled out over the telephone, faxed or completed electronically by either the principal or study staff. The items in the School Information Sheet will yield contact information for the school contact person, details about student‑body characteristics, and information about 5th grade instructional practices. The Sheet should take the principal about 20 minutes to complete.
Exhibit 2. School Information Sheet Item Level Justification
Item |
Purpose |
School Name…e-mail address |
To provide us with current information with which to contact the school. |
Grade levels |
To confirm that the school has 5th grade classrooms, for eligibility reasons. |
…both ELA and social studies |
To confirm that reading and social studies are generally taught in the same classroom by the same teacher for each class, for study eligibility purposes. |
Number of students per class |
To confirm that there are about 25 students per class, for study design reasons. |
ELA and social studies textbooks, Other instructional materials… |
To provide context information about the school’s curriculum for implementation analysis purposes. |
ELL and reading comprehension initiatives in the school |
To provide context about other activities at the school that could conflict with study activities for implementation and data collection purposes. |
Percentage of ELL 5th graders |
To confirm that the school serves ELL students, preferably a high proportion, for study design purposes. |
Annual student attrition |
To inform the study about student attrition levels to be expected at the school for study design purposes. |
Availability of Internet/e-mail access |
To inform study staff whether or not electronic communication and data collection are accessible to the school staff. |
Other information |
To provide us with information the staff deem relevant to study activities. |
Once district and school recruitment is complete, districts will be asked to provide background data for the 5th grade students at participating schools. The Student Background Data Request includes the following variables:
Gender,
Birth date,
Race/Ethnicity,
ELL status (level, competency score, etc.),
Free/reduced lunch eligibility status,
Special education status,
State assessment percentile/standard scores in reading.
See Exhibit 3 for item‑level justification of this instrument. If necessary, study staff will work with district personnel to retrieve the requested data and assemble it into an electronic data file that can then be exchanged electronically in a secure format. It will take a district analyst approximately 15 hours to assemble the requested data.
Exhibit 3. Student Data Request Item Level Justification
Item |
Topic |
Purpose |
1 |
Gender |
Variable for testing treatment/control group equivalence at baseline and a potential control for final analysis to eliminate the effects of gender in analysis of CSR effectiveness. |
2 |
Birth date |
To examine the sample for subjects who are outside the customary age range for 5th grade. |
3 |
Origin/race/ethnicity |
Variable for testing treatment/control group equivalence at baseline and a potential control for final analysis to eliminate the effects of origin/race/ethnicity in analysis of CSR effectiveness. |
4 |
Free or reduced‑priced lunch status |
Poverty indicator variable for testing treatment/control group equivalence at baseline and a grouping/control variable in final analysis to determine whether CSR is effective particularly for low‑income students. |
5 |
ELL status, ELL test scores |
Variables for testing treatment/control group equivalence at baseline and grouping/control variables for final analysis to determine whether CSR is effective particularly for ELL students. |
6 |
Special education status |
Variable for testing treatment/control group equivalence at baseline and a potential control variable for final analysis to eliminate the effects of special education status in analysis of CSR effectiveness. |
7 |
Spring 2006 achievement test scores |
Variable for testing treatment/control group equivalence at baseline. |
Group Reading Assessment and Diagnostic Evaluation (GRADE). The GRADE will be administered in fall 2007 and spring 2008 to all students in cohort 1 and in fall 2008 and spring 2009 to all students in cohort 2. The GRADE for 5th grade (Level 5) assesses the following domains:
Vocabulary (35 items): Students read a phrase or short sentence in which the target word is highlighted, then select the target word’s meaning from a list of four or five choices.
Sentence Comprehension (19 items): Students are presented with a single sentence with a missing word and four or five single‑word choices.
Passage Comprehension (6 passages, 30 items): Students read one‑ to two‑ paragraph passages and answer five multiple choice questions about each passage.
Listening Comprehension (17 items): The test administrator reads a sentence aloud and students mark one of four pictures that best matches the sentence.
It takes 5th graders one hour, on average, to complete the GRADE. Struggling readers may need more time, so two administration sessions will be scheduled to allow all students to finish the test.
GRADE results include total scores, as well as scores for each subsection, and these scores are provided in the forms of standard scores, percentile ranks, and grade equivalents. The GRADE has the advantage of providing detailed information about students’ performance using a group‑administered format. Unlike state assessment tests, the GRADE provides profiles of students’ strengths and weaknesses in several domains. Fall scores will be used in final analyses as baseline predictors.
Teacher Surveys. The teacher survey will be administered in two phases and the purpose of the surveys is to collect descriptive data. The Fall Teacher Survey will focus on the teacher’s background information and recent professional development experiences in reading instruction. It will take no more than 40 minutes to complete. See Exhibit 4 for item‑level justification of this instrument. The consent form will also be distributed and collected with the Fall Teacher Survey. The Spring Teacher Survey will focus on the classroom instructional context such as the presence of aides and specialists and the teachers’ perceptions of the classroom management challenges presented by their classes. It will take no more than 20 minutes to complete. See Exhibit 5 for item‑level justification of the Spring Teacher survey.
Exhibit 4. Fall Teacher Survey Item Level Justification
Item |
Topic |
Purpose |
1 |
Education |
To determine whether the teachers’ levels of education differ between the treatment and control groups. To investigate whether teachers with different levels of education or types of degrees implement CSR differently. |
2 |
Certification |
To determine whether teachers’ levels or types of certification differ between the treatment and control groups. To investigate whether teachers with different levels or types of certification implement CSR differently. |
3 |
Full time/part time |
To ascertain that all teachers in the study are regular full time teachers. |
4–6 |
Teaching experience |
To determine whether teachers’ prior teaching experiences differ between treatment and control groups. To investigate whether teachers with different amounts or types of teaching experience implement CSR differently. |
7 |
Preparation time |
To determine whether the amount of teachers’ prep time differs between treatment and control groups. To investigate whether the amount of teacher prep time has an impact on CSR implementation. |
8 |
Gender |
To determine whether treatment and control groups are equivalent at baseline. |
9–10 |
Ethnicity and race (separate questions) |
To determine whether treatment and control groups are equivalent at baseline. |
11 |
Spanish language fluency |
To determine whether teachers’ Spanish language fluency differs between treatment and control groups: In classes with a high proportion of Spanish ELL students, Spanish language fluency could impact many aspects of teaching and learning. To investigate whether teachers’ Spanish language fluency has an impact on CSR implementation. |
12–16 |
Professional development relating to reading instruction |
To determine whether the type, duration, and content of professional development differs between treatment and control groups. To investigate whether the type, duration, and content of professional development has an impact on CSR implementation. |
Exhibit 5. Spring Teacher Survey Item Level Justification
Item |
Topic |
Purpose |
1 |
Reading and social studies instruction |
To determine the extent to which the study teachers have responsibility for teaching reading and social studies to their students. If, for some reason, another teacher teaches these subjects some or all of the time, there are implications for CSR implementation (treatment group) and analysis (both groups). |
2 |
Student teacher |
To determine whether the teachers in either group had a student teacher, and, if so, the duration for which the student teacher took over the teaching of the class. A student teacher taking over the class will have implications for CSR implementation and analysis (both groups). |
3 |
Reading class stability |
To determine whether the teacher will teach reading to the same students all year. A change in students’ reading teachers will have implications for data analysis. |
4 |
Description of students’ reading abilities |
To understand the context, from the teacher’s perspective, for reading instruction in the class. Teachers’ perceptions of students’ reading abilities could have implications for reading instruction, including implementation of CSR. |
5 |
Class behavior |
To understand the extent to which teachers find student behavior in the class difficult to manage. Teachers’ perceptions of behavior management difficulties could have implications for implementation of CSR. |
6 |
Instructional resources |
To understand the extent to which the teacher has specialist and paraprofessional assistance in the classroom. This could have implications for CSR implementation and instruction. Differences between treatment and control classrooms will be examined. |
7–10 |
Reading program/Social studies curriculum |
To understand the nature of comprehension and social studies instruction teachers may already use in the classroom. This could have implications for CSR implementation, and differences between control and treatment classrooms will be examined. |
11-13 |
Use of CSR approach (for treatment group teachers only) |
To understand the extent of implementation and CSR related support that takes place in CSR classrooms. |
14 |
Effect of CSR (for treatment group teachers only) |
Teachers’ self-report regarding the effectiveness of the CSR approach in their classrooms. |
The purpose of administering the Fall survey is to collect data that will enable comparisons between the treatment and controls prior to intervention, and provide covariate and interaction variables for the impact analyses. The purpose of administering the Spring survey is to to identify other factors that are likely to affect the level of CSR implementation or student outcomes.
Collaborative Strategic Reading Implementation Validity Checklist (CSRIVC). The CSRIVC was designed exclusively to assess the fidelity of a teacher’s implementation of CSR. The instrument is a checklist on which the observer can indicate that each listed aspect of CSR occurred fully, occurred with modifications, or was not observed. There are 21 items on the CSRIVC, 11 that pertain to child behaviors, 7 to teacher behaviors, and 3 to the classroom setting. Fidelity observations using the CSRIVC will be conducted two times per year, once prior to the Thanksgiving break and once before the end of the school year. All teachers implementing the CSR intervention (treatment group teachers) will be observed both times. These observations will be conducted by trained observers. The observations will each last approximately 30–45 minutes. Implementation observations do not constitute burden to teachers participating in the study.
Expository Reading Comprehension Classroom Observation (ERCCO). The ERCCO consists of 5 to 6 10‑minute observation intervals. During each interval, the observer records the nature of comprehension and vocabulary instruction, grouping arrangements, type of text, and the occurrence of features of generally effective instruction. There are 56 items on the ERCCO protocol. A brief list of items will be added to the ERCCO regarding the nature of any group instruction that took place during the observation. An additional 30 minutes after the observation period will be needed for the observer to count tallies, to complete the coversheet and observation summary items. The ERCCO will be used during social studies instruction (on days when CSR is not being implemented, for treatment classrooms). The ERCCO requires the use of expository text, which is most likely during social studies.
All teachers participating in the study (treatment and control teachers) will be observed once using the ERCCO in the spring, between February and April. The purpose of this observation is to document if CSR training impacts teachers’ use of comprehension teaching strategies in contexts external to the intervention, to assess the extent to which collaborative learning and explicit comprehension instruction occur in control classrooms and to describe ‘business as usual’ control classrooms. It should be noted that fidelity observations and ERCCO observations cannot be conducted simultaneously. Observations will be conducted by observers who have previous experience using the ERCCO protocol.
Group Reading Assessment and Diagnostic Evaluation. The GRADE (described above) will be administered again in the spring to all students participating in the study to capture potential growth in reading achievement. Fall scores will be used in final analyses as baseline predictors, allowing for the isolation of variance in spring outcome scores attributable to CSR treatment as opposed to pre‑existing reading abilities.
This request is for the administration of student, teacher, and classroom data collection instruments that are included in the appendices and described under the Overview to Data Collection section of this package. The data collections and their timing in the study are displayed in Exhibit 6.
Exhibit 6. Data Collection Instruments and Schedule
Data Collection Instrument |
Summer/ Fall 2007 |
Fall 2007 |
Winter/ |
Fall 2008 |
Winter/ |
District and School Information for Recruitment Purposes |
|||||
Student Background Data Request |
P1* |
|
P2* |
|
|
School Screening Protocol |
P1 |
|
P2 |
|
|
Students |
|||||
Group Reading Assessment and Diagnostic Evaluation (GRADE) |
|
P1 |
P1 |
P2 |
P2 |
Teachers |
|||||
Fall Teacher Survey |
|
P1 |
|
P2 |
|
Spring Teacher Survey |
|
|
P1 |
|
P2 |
Classroom Observations |
|||||
Classroom Observation: Collaborative Strategic Reading Implementation Validity Checklist |
|
P1 |
P1 |
P2 |
P2 |
Expository Reading Comprehension Classroom Observation (ERCCO) |
|
|
P1 |
|
P2 |
* P1 = Phase 1, P2 = Phase 2
The CSR study is designed to evaluate the effectiveness of CSR intervention in 5th grade classrooms with a large percentage of ELL students. Thus, the primary research question of interest is whether CSR classrooms outperform control classrooms on the proposed outcome measure: the Group Reading Assessment and Diagnostic Evaluation (GRADE). As discussed previously, the primary research question is based on impact analyses using the full sample, which includes native English speakers as well as English Language Learners.
There are two research questions of secondary interest in this study. The first is whether the CSR intervention has a positive effect for ELL students. For the purpose of this study we classify students to three subgroups (students who have been previously classified as ELL, currently assigned an ELL status, and native English speakers) because most ELL students are placed in English emersion classes by the 5th grade. However, depending on their English language fluency they may still be in a position to benefit more from explicit comprehension instruction than native English speakers.
The other secondary research question is based on the presumption that the level of implementation of the CSR intervention is positively related to the student outcomes. Therefore, it is of interest to ask whether classrooms with higher level of implementation benefit more from the intervention. Fidelity of the intervention will be measured by conducting observations using the Collaborative Strategic Reading Intervention Validity Checklist (CSRIVC) developed by Vaughn, Hughes, Schumm, and Klinger (1998). These observations will be conducted two times per year. The purpose of this observation measure is to determine the extent to which the components of CSR are being implemented. The degree of correlation between level of implementation and each of the outcome measures will be reviewed, and should there be a significant relationship, the relationship will be further studied by analyzing the data from the treatment classrooms only.
The collected data will be used by IES and its subcontractors to study the effects of CSR intervention, to answer the research questions stated above. Student achievement data will be collected evaluate the effect of CSR intervention on students’ reading comprehension and vocabulary development. Classroom observation will be conducted to evaluate the level of CSR implementation and potential contamination of control classrooms. In addition, both student‑ and teacher-level demographic data will be collected to enhance the estimation of the potential effect of CSR and the interpretation of the study results. The study results will assist states and districts as they choose strategies to raise achievement for a variety of students. This evidence will be of immediate interest and import for policymakers, researchers, and practitioners because of the paucity of strategies that have been empirically tested with ELL students.
Whenever possible, we will use information technologies to maximize the efficiency and completeness of the information gathered for this evaluation and to minimize the burden on respondents. Initial recruitment at the district level will begin with electronic searches of existing data sources to identify eligible districts. Study staff will then use telephone and other electronic communication as necessary, in addition to mail, to gain district approval for school recruitment. Additionally, the Student Data Background Request will be managed by telephone and e-mail, to the maximum extent possible, so that school or district staff can provide us with student data in electronic format if their systems permit. Study staff will use telephone, e-mail, and other electronic resources to complete the School Information Sheet so that school staff does not need to track and mail any forms.
The Teacher Survey will not involve information technology, as developing web‑based methods would not be cost‑effective given the relatively modest sample size of the study and the lack of access to the Internet for elementary school teachers in high poverty schools. Additionally study staff will be on‑site with teachers for training and classroom observations and can administer surveys in person at no extra cost and with a higher response rate than via telephone or electronic means. The Spring Teacher Survey may be administered electronically only if it can be determined that all teachers have access to the Internet and that personal administration will not yield a higher response rate than electronic administration.
We will utilize existing data sources whenever feasible. However, there are no equivalent sources of data available currently for measures/information required for successful completion of the study. As is discussed above in the Overview of Data Collection Plan, data are collected for four purposes: recruitment, documentation of the intervention, documentation of the study context, and measurement of student outcomes.
We have developed a customized School Information Sheet to summarize publicly available data and to collect information that is not readily available in public sources. The School Information Sheet will be administered to those principals only whose school fits the initial recruitment criteria and who are willing to make a commitment to participate in the study.
The level of implementation and potential contamination across classrooms will be captured by an implementation checklist (Collaborative Strategic Reading Implementation Validity Checklist) and observation protocol (Expository Reading Comprehension Classroom Observation) tailored to the CSR intervention.
We will be collecting demographic information from teachers by administering Fall and Spring Teacher Surveys, because no reliable and up‑to‑date source of data on teachers’ background characteristics, including their education and most recent professional development experiences exists. Moreover, the instructional context in classrooms changes each year, so it will be necessary to ask teachers about the circumstances of their current class. Thus, it will be necessary to create and administer teacher background surveys for the study.
Reading achievement data cannot be obtained from state assessments because (1) the assessments are administered only once during the year, and we require assessment that occurs specifically before the intervention begins and at the end of the school year, and (2) all students in the study must take the same reading assessment for the study, and the existing assessment data will be from different instruments used in different states.
Existing data about districts, schools, and students will be utilized whenever possible. MDR and study staff will utilize publicly available datasets to identify districts and schools that meet study eligibility criteria. Study staff will use personal contacts, websites, and other publicly available information sources to learn about schools and districts prior to contacting them. This will make recruitment efforts more straightforward, and it will reduce the burden on school principals to the extent that school‑specific information can be pre‑loaded into the School Information Sheet so that they will only need to confirm that the information is current.
The CSR Study is one of only a few experimental studies that focus on a strategy to improve reading achievement of ELL students. As required by NCLB, states must improve achievement, particularly in reading, for all students. Since ELLs are a large and expanding group of academically at‑risk students, evidence of strategies that are effective for this group is needed. Without this study, states and districts will have almost no basis on which to choose appropriate reading interventions for late‑elementary ELL students.
The data collections for which we are requesting clearance—School Information Sheet, Student Background Data Request, and Teacher Surveys—are vital to this study because they will yield important contextual variables that will help us understand the results. For example, student background data will be used to determine if CSR is more or less effective for certain groups of students, and both student background snd teacher survey data will be used compare control and treatment groups to see if we can assume that the groups are similar enough to yield causal inferences. Meanwhile, information about 5th grade instructional practices from the School Information Sheet will be important for understanding the context of CSR implementation and control classroom instruction. All of these data collections will result in information that will broaden understanding of results and strengthen the conclusions of the study.
No special circumstances apply to this study.
The notice for the full study data collection was announced in the Federal Register on XXX. No public comments have been received. PLACE HOLDER FOR FINAL TEXT
To assist with the development the CSR Evaluation Study, project staff have drawn on the experience and expertise of a network of outside experts. The consultants and their affiliations are listed in Exhibit 7.
Exhibit 7. Technical
Working Group Members of the
CSR Intervention Study
Technical Working Group Members |
|
Expert |
Affiliation |
Dr. Roger Bybee |
Biological Sciences Curriculum Study (BSCS) |
Dr. David Chard |
University of Oregon |
Dr. David Francis |
University of Houston |
Dr. Jeremy Kilpatrick |
University of Georgia |
Dr. David Myers |
American Institutes for Research |
To date the project advisors and TWG members have convened twice—in June and September 2006—and provided comments on the study design, the treatment, and the data collection instruments. Additional meetings will be scheduled throughout the life of the project as deemed appropriate. Project staff also will use outside experts individually for consultation on an as‑needed basis.
Teachers will be compensated $20 for completion of the Fall Teacher Survey and $10 for the Spring Teacher Survey. This is consistent with teachers’ estimated average hourly rate of $29 given that the Fall Survey will take a maximum of 40 minutes and the Spring Survey 20 minutes to complete. Teachers will also be paid the district hourly rate for any study activities they complete outside the hours of the school day, such as coaching that takes place in the evening or training that takes place on a weekend. Teachers whose school districts will not pay them their regular salaries for training that takes place during a weekday will also be compensated at the district daily rate.
REL Southwest
follows the confidentiality and data protection requirements of IES
(The Education Sciences Reform Act of 2002, Title I, Part E, Section
183). REL Southwest will protect the confidentiality of all
information collected for the study and will use it for research
purposes only. No information that
identifies any study
participant will be released. Information from participating
institutions and respondents will be presented at aggregate levels in
reports. Information on respondents will be linked
to
their institution but not to any individually identifiable
information. No individually identifiable information will be
maintained by the study team. All institution-level
identifiable information will be kept in secured locations and
identifiers will be destroyed as soon as they are no longer
required.
REL Southwest obtains signed NCEE Affidavits of
Nondisclosure from all employees, subcontractors, and consultants
that may have access to this data and submits them to our NCEE COR.
Teachers who participate in the study will be requested to sign a consent form that provides information about the study (roles and responsibilities of study participants) and study related data collection (see Appendix A: Letters). The CSR intervention and randomization will be conducted as a part of the regular curriculum offered by a school district, hence we will not be requesting parental consent for the randomization of students to intervention or control classrooms. However, we will be collecting parental consent forms regarding testing students and using the collected test data and demographic data collected by the school districts (see Appendix A: Letters). This study will be conducted in close partnerships with the school districts, which will aid the process of collecting parental consent forms. We will also ask help from the school staff, both school administrators and teachers. Specifically, we will ask the teachers to distribute and collect the consent forms. Teachers in the treatment group will be informed about the parental consent forms during their CSR training, while the control school teachers will receive this information during an informational meeting about the study. If possible, the request for parental consent will come from the school district on the behalf of REL-Southwest. Students can decline to participate in any data collection activity during the study.
No questions of a sensitive nature will be included in the CSR Evaluation Study.
Exhibits 8a and 8b below provide annual hour burden estimates for district, school, and teacher data collections3. The total estimated hour burden for the full CSR Study is 138 hours; or 69 hours per each year of the study. Based on average hourly wages for participants the burden (138 hours) amounts to a total monetary burden of $4,788 ($2,394 per year). The total burden estimate includes:
Time for 100 percent of district personnel to respond to the Student Background Data Request;
Time for 90 percent of principals to complete, with study staff assistance, the School Information Sheet;
Time for 90 percent of 80 teachers in the study to respond to:
A 40‑minute Fall Teacher Survey;
A 20‑minute Spring Teacher Survey;
Teachers will be compensated for the burden of completing the surveys, which should facilitate a high response rate. District and school staff will not be compensated for the hour burden of study participation because the tasks we ask of them will not be outside the realm of the duties for which they are already compensated through their employment
Exhibit 8a. District and School Hour Burden, Combined Burden Estimates for Phases 1 and 2 (2 years)
Task |
Number of Respondents |
Estimated Response Rate |
Number of Respondents |
Time Estimate |
Total Hours |
Hourly Rate |
Estimated Monetary Cost of Burden |
Suggested Respondent Compensation |
Districts |
||||||||
Student Background Data Request |
4 |
100% |
4 |
15 |
60 |
$40 |
$2,400 |
$0 |
Schools |
||||||||
School Information Sheet |
20 |
90% |
18 |
.33 |
6 |
$50 |
$300 |
$0 |
Exhibit 8b. Teacher and Student Hour Burden, Combined Burden Estimates for Phases 1 and 2 (2 Years)4
Task |
Number of Respondents |
Estimated Response Rate |
Number of Administrations |
Number of Total Respondents |
Time Estimate |
Total Hours |
Hourly Rate |
Estimated Monetary Cost of Burden |
Suggested Respondent Compensation |
Teachers |
|||||||||
Fall Teacher Survey |
80 |
90% |
1 |
72 |
.67 |
48 |
$29 |
$1,392 |
$20 |
Spring Teacher Survey (recall there is version A for treatment teachers and B for control, estimate 36 per group for a total of 72) |
80 |
90% |
1 |
72 |
.33 |
24 |
$29 |
$696 |
$10 |
There are no additional respondent costs associated with this data collection other than the hour burden accounted for in item 12.
Exhibit 9 displays the estimated total annual cost to the Federal Government. Estimated year one expenditures include study planning (study design and study material development) and recruitment preparation (recruitment planning and development of recruitment material development). Project costs for year two and three include site recruitment and management activities, CSR intervention related activities (training and coaching as well as classroom observations), and data related activities (pretest/posttest data collection, data entry). Project costs in years four and five include completion of data collection, data clean up, data analysis and final product development (including dissemination efforts).
Exhibit 9. Estimated Annualized Costs
Study Year (dates) |
Total Study Costs per Year |
Year 1 (3-15- 2006---03-14-2007) |
$707,462 |
Year 2 (3-15- 2007---03-14-2008) |
$1,215,907 |
Year 3 (3-15- 2008---03-14-2009) |
$1,147,798 |
Year 4 (3-15- 2009---03-14-2010) |
$808,581 |
Year 5 (3-15- 2010---03-14-2011) |
$67,604 |
Total |
$3,947,352 |
No changes or adjustments are requested at this time.
This study will result in a final report, which will be submitted to IES after internal and TWG review in March, 2009 (see Exhibit 10). The final report will include results of analyses outlined under the Estimation Procedures/Analysis Methods heading of Section B.
A dissemination plan will be created in coordination with the REL Southwest Marketing department. Multiple products will be created and various dissemination channels will be utilized dependent on the relevant target audience identified for each product. A product announcement will be developed prior to the start of the study and periodic interim research bulletins will be created and distributed throughout the life of the study.
Members of the study team may, in coordination with REL Southwest Marketing, and as per any relevant IES guidelines, submit articles to peer‑reviewed journals, though specific results of the current study will first have to be disseminated to IES.
Members of the study team may submit proposals for conference presentations subject to appropriate IES guidelines and contract restrictions. Presentations of preliminary data may be made at national scholarly conferences pending approval by REL Southwest, and following REL Southwest contract guidelines.
Exhibit 10. Schedule for Dissemination of Study Results
Activity/Deliverable |
Due Date |
Intermediate Data Report |
August 2008 |
Final Report |
March 2010 |
All data collection instruments will include the OMB expiration date.
No exceptions are requested.
1 Based on 5 CFR 1320.3(d) we provide a discussion of student achievement measures used in the study, but do not seek clearance for them. The student assessments described here do not pose any burden to teachers because study personnel will administer them.
2 The two instruments described in this section (CSRIVC and ERCCO) do not pose any burden to teachers, thus we are not seeking clearance for them; we are only seeking clearance for the Fall and Spring Teacher Surveys.
3 Based on 5 CFR 1320.3(d) we provided a discussion of student achievement measures used in the study in section 1 but do not seek clearance or declare burden for them. The student assessments described do not pose any burden to teachers because study personnel will administer them. In addition, we do not state burden for observational measures, because they do not pose burden to the teachers.
4 Note that we list 47 for the number of respondents in item 16a of the IC Part 1 Information Request. This is based on 72 teachers, 4 Student Background Data Requests, and 20 School Information Sheets. This yields 94 respondents, or an average of 47 per year. This same approach should be used for understanding our response to item 16b (72+72+18+4=166. 166/2=83).
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSION |
| Author | Rebecca Holland Coviello |
| Last Modified By | Sheila.Carey |
| File Modified | 2007-07-12 |
| File Created | 2007-07-12 |