Att_Supporting Statement Part A

A STUDY OF THE DIFFERENTIAL EFFECTS
OF ELL TRAINING AND MATERIALS
(Study 2.1c)

OMB Clearance Package Supporting Statement

Part A: Justification

Regional Educational Laboratory

for the

Central Region

Contract #ED-06-CO-0023

Submitted to:		Submitted by:
	Institute of Education Sciences U.S. Department of Education 555 New Jersey Ave., N.W. Washington, DC 20208		REL Central at Mid-continent Research for Education and Learning 4601 DTC Blvd., #500 Denver, CO 80237 Phone: 303-337-0990 Fax: 303-337-3005
Project Officer		Project Director:
	Sandra Garcia, Ph.D.		Louis F. Cicchinelli, Ph.D.

Deliverable 2007-2.3/2.4

March 26, 2007

©2007

This report was prepared for the Institute of Education Sciences under Contract #ED-06-CO-0023 by Regional Educational Laboratory Central Region, administered by Mid-continent Research for Education and Learning. The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

TABLE OF CONTENTS

A. Justification 1

Introduction 1

1. Circumstances Necessitating the Data Collection 1

Data Collection Sources. 2

Student Data. 4

Teacher Data. 4

Classroom Observation. 6

Teacher Background /Classroom Level Data. 7

2. Purposes and Uses of the Data 7

3. Use of Improved Information Technology to Reduce Burden 8

4. Efforts to Identify and Avoid Duplication 8

5. Impacts on Small Businesses and Other Small Entities 9

6. Consequences to Federal Programs or Policies if Data Collection Is Not Conducted 9

7. Special Circumstances 9

8. Federal Register Comments and Persons Consulted Outside the Agency 10

9. Payments to Respondents 10

10. Assurances of Confidentiality Provided to Respondents 10

11. Justification for Questions of a Sensitive Nature 12

12. Estimates of Hour Burden of Data Collection 12

13. Estimates of Total Cost Burden to Respondents 14

14. Estimates of Annualized Cost to the Federal Government 14

15. Reasons for Changes or Adjustments in Burden 15

16. Tabulation, Analysis, and Publication Plans and Schedule 15

Analysis. 17

Pre-intervention Analyses. 17

Descriptive Statistics. 17

Assumptions/Outliers/Data Treatment. 18

Attrition. 18

Main Effects. 18

Reporting. 21

Study Report Preparation. 21

Public- or Restricted-Use Data Files. 21

Dissemination. 22

17. OMB Expiration Date 22

18. Exceptions to Certification Statement 22

References 22

a. justification

Introduction

This submission requests approval for a data collection plan for a study of the differential effects of training for teachers of English Language Learners and materials for their English Language Learner students. The project is sponsored by the Institute of Education Sciences within the U.S. Department of Education, and will be conducted by the Central Region Educational Laboratory (contract #ED-06-CO-0023) at Mid-continent Research for Education and Learning (McREL).

The Regional Educational Laboratory Programs (REL) are authorized under the Education Sciences Reform Act of 2001 (Pub. L. 107-279) Part D, Section 174, (20 U.S.C. 9564), and are administered by the Institute of Education Sciences’ National Center for Education Evaluation and Regional Assistance. The primary mission of the regional educational laboratories is to serve the educational needs of each region, using applied research, dissemination, and technical assistance to bring the latest and best scientifically valid research and proven practices into school improvement efforts.

1. Circumstances Necessitating the Data Collection

The No Child Left Behind Act (PL 107-110) requires schools to place English Language Learners (ELLs) in “high quality language instruction educational programs that are based on scientifically based research demonstrating the effectiveness of the programs in increasing (a) English proficiency; and (b) student academic achievement in the core academic subjects” (Title III, Sec. 3115(c)(1)). ELL student inclusion in standards-based testing programs is also required by the law. Schools are held accountable for ensuring that ELL students make “Adequate Yearly Progress” (AYP), or else they face serious sanctions.

The population of school-age children whose first language is not English is increasing exponentially (Santos, 2004), and Spanish-speaking ELL students’ performance on the National Assessment of Educational Progress (NAEP) is far below that of their Anglo counterparts (Slavin & Cheung, 2003). Given the increased numbers and the lagging achievement of ELLs, it is not surprising that there is a parallel (and acute) demand for teachers to effectively address their needs (Hill & Flynn, 2004) and to ensure that ELLs have the same opportunity to learn as their native-English-speaking peers (Herman & Abedi, 2004).

This study responds to regional and national needs by examining the impact on student achievement of using ELL-specific materials in the classroom in combination with a professional development program for teachers that is aligned with those materials. The curricular program being examined is On Our Way to English (OWE) and the professional development program being examined is Responsive Instruction for Success in English (RISE).

OWE is a comprehensive, three-component program for elementary classrooms (grades Kindergarten through 5) designed by Harcourt to provide ELLs simultaneous access to (a) English oral language development, (b) comprehensive literacy instruction, and (c) standards-based content area information (Freeman, Freeman, Stack, Colon Garcia, McCloskey, Silva & Gottlieb, 2004). The classroom program, OWE, provides teachers a scope and sequence, structure, and material resources to begin and continue language and literacy instruction for ELLs.

RISE is a Harcourt professional development program designed to meet the growing needs of teachers of ELLs for training and support. The content of RISE aligns with Fillmore and Snow’s (2000) proposal for what teachers need to know about language to help ELLs meet the high levels of language proficiency required for success in school. The professional development program, RISE, complements the classroom program by providing teachers sustained adult learning opportunities to become familiar with and understand the content of the classroom program, the rationale for its structure, and practical strategies for its use in real time and real situations.

It is hypothesized that classrooms where teachers have been provided with training specific to the needs of ELLs or where teachers use materials specifically designed for use with ELLs will have (a) improved teacher knowledge and skills for instructing ELLs; and (b) higher student achievement (specifically in vocabulary, fluency, reading comprehension, grammar use, writing, and listening skills) compared to classrooms in which neither of these elements is evident. The focus of this study is on Spanish-speaking students in grades 1 – 5.

Little empirical evidence has been collected regarding the effectiveness of RISE, and available research evidence on the OWE materials suggests differential findings. Although not specifically produced to support each other, Harcourt developers are encouraging the use of RISE in conjunction with OWE. This is illustrated through the use of videotaped lessons of teachers and students using OWE materials as part of the RISE training. To this date, no research has been conducted that examines the impact of using the OWE program materials in combination with the RISE professional development course. That said, both OWE and RISE have been widely implemented: implementation of one or both of the interventions has occurred in all 50 states.

Data Collection Sources. Exhibits A and B provide detailed descriptions of data collection instruments as well as the instruments themselves. Exhibit A provides a Measurement Concept Map for the proposed student instruments as well as a Measurement Concept Map for the proposed school / teacher instruments. Exhibit B provides copies of instruments to be used in the study. Additional detail regarding the proposed data collection efforts are described in the following paragraphs. Table 1 provides a summary of the data collection plan associated with this study.

Table 1. Summary of Study: Data Collection Plan

Group		Fall 2007	Spring 2008	Fall 2008	Spring 2009
RISE + OWE	Student Data	Idea Proficiency Test (IPT) Form A1	IPT Form B		IPT Form B
			State achievement data		State achievement data
	Teacher Data	Teacher log (quarterly)	Teacher log (quarterly)	Teacher log (quarterly)	Teacher log (quarterly)
		RISE trained trainer artifacts (monthly)	RISE trained trainer artifacts (monthly)	RISE trained trainer artifacts (monthly)	RISE trained trainer artifacts (monthly)
		Classroom Observation and teacher interview (sample)			Classroom Observation and teacher interview (sample)
Group		Fall 2007	Spring 2008	Fall 2008	Spring 2009
Control	Student Data	IPT Form A	IPT Form B		IPT Form B
			State achievement data		State achievement data
	Teacher Data	Teacher log (quarterly)	Teacher log (quarterly)	Teacher log (quarterly)	Teacher log (quarterly)
		Classroom Observation and teacher interview (sample)			Classroom Observation and teacher interview (sample)

Student Data. A number of English language assessment instruments for ELLs are available, but many of these are intended to be used for placement decisions rather than to assess progress. In this study, student progress will be assessed using the revised IDEA Proficiency Tests, or IPT (IPT Testing System, 2005), which were designed to respond to No Child Left Behind requirements for English language proficiency². Previous versions of the IPT have been used by 30 states with a combined LEP enrollment of over 2 million students (IPT: The New Title III Testing System, 2004). The IPT yields separate scores for listening, speaking, reading, writing, and comprehension, and offers a standardized scale for reporting scores across all levels.

Students in grades 1-4 will be pre-tested in Fall 2007 with the appropriate level of IPT Form A, post-tested in spring 2008 with Form B, and post-tested again in Spring 2009, when the original sample is in grades 2-5. Students entering treatment and control schools after the start of the study will also be tested. Because the IPT is vertically equated, use of this instrument to assess school-level outcomes is appropriate. Student time to take the test will be approximately 90 minutes. Across all testing periods, a random sample of students’ oral language will be assessed (in each research condition). Collecting student data three times will provide researchers with baseline data and data to examine growth over time. In addition to the IPT, state assessment data will be collected.

Site coordinators will be selected at each school to facilitate study activities and to serve as the RISE trainers for treatment condition schools; these individuals will also be tasked with collected student-level information such as ethnicity, language, eligibility for free and reduced-price lunch, primary language spoken at home, and, when available, the student’s level of literacy. Site coordinators will be provided a data collection summary sheet for documenting these data at the beginning of each school year. These data summary sheets will then be mailed to McREL and information entered into a database. Site coordinators will also be asked to collect and mail summary sheets containing these data for any students entering during the middle of the year.

Teacher Data. Teacher online logs will be used to assess pedagogical practices for teaching ELLs as well as implementation fidelity. Online logs provide numerous advantages over traditional paper and pencil methods. These include allowing for designs that contain complex skip patterns; range and consistency checks that enhance data quality; availability of previous information, which reduces respondent burden; quick availability of data; and a decrease in the number of clerical errors that can occur during data-entry. The logs have been structured to include checklists and rating scales that were derived from research findings on best practices for teaching ELLs. Teachers are asked to detail classroom practices, activities, and any ELL accommodations or ELL-specific strategies used during their classes. The creation of an instrument was necessary because existing instruments failed to assess teachers of ELLs on all of the constructs considered crucial for teaching language learners. Items address the following constructs identified through the literature on teaching ELLs:

• Organization of Curriculum and Pedagogy: whether the curriculum is organized around “big” questions (or themes) relevant to the students’ own lives (Freeman & Freeman, 2000; NCTE, 2006); whether students work collaboratively or in small groups for instruction, and how such work is accomplished (Henderson & Landesman, 1992; Klingler & Vaughn, 2000); and the frequency with which teachers expose students to oral language and the types of oral language being developed.

• Use of Authentic & Meaningful Experiences: whether students are involved in reading and writing experiences that are considered authentic, using comprehensible fiction and nonfiction (NCTE, 2006); the types of pedagogical strategies employed for reading instruction, e.g. reading with / reading for students; whether student background knowledge and interests are considered (Darder, 1993; Godina, 1998; Krashen, 1996; Smith, 1983); and whether the content being taught serves a purpose for learners, with instruction and activities that serve to advance academic proficiency as well as to develop English language skills (Freeman & Freeman, 2000; NCTE, 2006).

• Assessment: whether teachers use student assessment data to inform or guide their teaching.

• Teacher Beliefs: whether teachers have high expectations for their students and faith in their abilities to be successful (Collier, 1995).

Completion of the logs will take approximately 25 minutes per session (based on a pilot test using McREL staff who are former teachers), with teachers completing these in August, November, February, and May of each study year³. All teachers of grades 1-5 during the study period will be asked to complete these logs electronically, with teacher responses to items directly entered into a database. Teachers will be provided contact information for a McREL staff member assigned to this research study who will be able to answer any questions they have about the study or the completion of the instruments.

The online teacher log will also be used to assess implementation fidelity, i.e., the degree to which they implemented the intervention as it was intended by the developer and researchers. These data will not be used to test statistical significance but rather are for the purposes of examining whether critical intervention components have been implemented. Thus, in addition to questions regarding practices, logs contain several critical intervention components identified through a careful review of RISE and OWE materials and in discussions with trainers and authors of the programs. Treatment teachers will be asked to report on the specific OWE lessons used to gauge progress in OWE; student engagement in activities; the duration of ELL-specific activities or strategies, including discussions, writing, and use of graphic organizers; and outside-of-classroom activities. This information will provide researchers with information on teacher use of materials and student exposure to materials. Less frequent data collection would likely be incomplete, as this would require teachers to reflect on practices and activities that may have occurred five months prior to data collection.

RISE- trained trainers, who are also serving as site coordinators, will be asked to provide the artifacts of their training sessions with participating teachers. These artifacts will include attendance lists, training sessions completed, and meeting minutes. These will be submitted to McREL once per month.

Classroom Observation. A classroom observation (a modified version of the English Language Learner Observation Instrument discussed below) and a brief interview protocol to be used have been developed to assess changes in teachers’ behaviors and skills related to ELLs. Observations will complement the online teacher logs, which are self-reported. A sample of treatment and control classrooms will be observed using the observation protocol⁴. Trained observers will observe classroom activities such as grouping arrangements, teacher pedagogy, and student activities during a language arts period two times in the course of the study. Depending on the classroom grade (1^st – 5^th), it is anticipated that class periods will last between 30 and 50 minutes; therefore, observations will require 30-50 minutes. Brief interviews with classroom teachers, conducted after observations (preferably on the same day as the observation), will focus on teachers’ lesson plans, materials, and instruction during the observation. Questions related to the RISE and OWE interventions will also be asked of treatment group members. Teachers will be observed in the fall of 2007, and again in the spring of 2009. Initial observations (in 2007) will be used as a baseline; observations in 2009 will provide an index of growth over time. A modified version of the English Language Learner Classroom Observation Instrument (ELCOI) (Gersten, et. al., 2005; Haager, et. al., 2003), a 30-item moderate inference instrument using a Likert scale, will be used in this study. ELCOI consists of six empirically derived subscales adapted from observational, cognitive learning, and sheltered instruction research on effective reading instruction (Englert, 1984; Brophy & Good, 1986; Tikunoff et al., 1991). As noted, additional items will be incorporated into the treatment and control group ELCOI protocols relative to the instructional setting (e.g., availability of ELL reading and display materials, room and space arrangement). The treatment group protocol will also include practices specific to the OWE intervention that may be observed (e.g., use of the thematic unit teacher’s guide, the chant poster).

Teacher Background /Classroom Level Data. Information on teacher characteristics will be collected during required Study Orientation Sessions. Teachers will be asked to complete an information sheet requesting general demographic information such as gender and ethnicity as well as teaching experience (years teaching, teaching certification status) and experience teaching ELLs (years teaching ELLs, knowledge of other languages, prior exposure to professional development for teaching ELLs). Definitions for these characteristics will be consistent with national databases; completion of this instrument will require approximately five minutes. In addition, we will collect information from site coordinators to ensure that their characteristics correspond to those reflective of the expectations for this role⁵. These profiles of characteristics will help to ensure that the site coordinators, who are also serving as the trainers in treatment schools, are qualified to serve as site coordinators, and, for treatment schools, that these individuals are qualified to serve as trainers for the RISE program. Site coordinators will also be asked to complete questions about their respective schools.

2. Purposes and Uses of the Data

This research will address the following primary question:

Does participation in RISE in conjunction with the use of OWE result in positive effects on student language skills?

A secondary research question addresses issues related to the effects of professional development or materials on teacher pedagogy and practice:

Does the combination of RISE and OWE result in changed teacher behavior, knowledge, and skills related to ELLs?

Forty-eight schools will be randomly assigned to one of the two conditions: RISE in combination with OWE, or control. All classrooms with eligible ELL students in each school will participate in the condition assigned to the school, e.g., if School A is assigned to the RISE and OWE treatment condition, all grade 1–5 teachers with eligible ELL students in that school will get the RISE professional development and will use the OWE materials in their classrooms.

This study has been evaluated in light of OMB’s information quality guidelines for utility, integrity, and objectivity as well as those of IES. Based on that evaluation, this proposed collection of information will result in information that will be collected, maintained, and used in a way consistent with the information quality guidelines of OMB and IES. Further, as shown in Exhibit A, each data source detailed above serves an identified and important purpose, germane to addressing the research questions. The column titled “Data Use” provides details regarding the practical utility of each data collection instance.

3. Use of Improved Information Technology to Reduce Burden

Three general strategies will be used to minimize the reporting burden for participants. First, data collection activities will be accomplished online. The online nature of the data collection will allow participants to complete the data collection more quickly because they will not have to manage paper documents or mailing activities. Teachers will receive an e-mail message providing them with a link to the online teacher log and a requested timeline for completion. Acknowledgements of receipt, reminders, and other communication can be received without addition to the current paperwork burden for teachers, and implementation fidelity can be monitored. Researchers will be able to tailor distribution of reminders so that only non-responders will be contacted. Second, the full schedule of data collection windows will be communicated at the onset of the study, with reminders of each upcoming event sent two weeks in advance via e-mail and a two-week response window provided for the actual collection of data. This advance schedule, reminder, and response window structure will allow participants to plan and to incorporate the data collections into their schedules. Third, instruments have been designed to reduce response burden by focusing only on the information necessary to carry out the study successfully. Constant information (e.g., name, grade level, etc.) provided previously by a respondent will be available in follow-up data collections, reducing participant burden. Requests for data will be limited to only those data to be included in pre-specified analyses, with log length kept to a minimum.

4. Efforts to Identify and Avoid Duplication

The purpose of this experimental study is to evaluate the effectiveness of the OWE program for ELL students in conjunction with the RISE program for ELL teachers for (1) helping students attain skills critical to developing English language skills in vocabulary, fluency, reading comprehension, grammar use, writing, and listening skills and (2) increasing teacher skills in teaching ELLs. Although not specifically produced to support OWE, RISE can be used in conjunction with OWE. Developers are encouraging this combination, as illustrated through the use of videotaped lessons of teachers and students using OWE materials as part of RISE training. According to the developers, no research has been conducted that examines the impact of using the OWE program materials in conjunction with the RISE professional development course. Thus, this study represents a unique contribution to an examination of teaching and learning for language learners.

5. Impacts on Small Businesses and Other Small Entities

No small businesses will be included in our sample. The primary respondents in our study will be teachers and students. In addition, participating school districts will be asked to provide state assessment data on the students of the participating teachers. The burden placed on district staff in

retrieving these data will be minimized by requesting existing assessment results and carefully specifying the study information needs.

6. Consequences to Federal Programs or Policies if Data Collection Is Not Conducted

In response to regional and national needs to comply with the NCLB requirement that all subgroups of students achieve to the same high level, this study will examine the impact of ELL-specific instructional materials and teacher professional development on the achievement of non-English speaking students in primary grades. Increasing numbers of ELL students and the consequent acute demand for teachers trained in comprehensive, multi-year English acquisition curriculum aligned with standards, require highest-quality evidence of effective approaches to raising student achievement among ELLs. Under the Education Sciences Reform Act of 2001, the regional educational laboratories are charged with providing exactly such evidence by conducting rigorous research. Without the data and findings from this study, REL Central and the National Laboratory Network will be unable to disseminate scientifically valid research on the use of ELL-specific materials in the classroom in combination with a professional development program aligned with those materials. Further, without these data and findings, the laboratory network will be limited in its ability to promote the use and application of scientifically valid research to improve classroom practice.

7. Special Circumstances

None of the special circumstances listed apply to this data collection.

8. Federal Register Comments and Persons Consulted Outside the Agency

A 60-day notice to solicit public comments was published in the Federal Register on April 11, 2007. The Department will publish a 30-day Federal Register Notice following the 60-day notice to allow public comment. Comments were received in regards to ______, ______, and______. Revisions made to the study to address these comments include______, ______, and______. See Appendix A for copies of the Federal Register notices pertaining to this study, numbers ______ and______.

In addition, throughout the course of this study, we have and will continue to draw on the experience and expertise of a technical working group (TWG) that will provide a diverse range of experience and perspectives as well as expertise in relevant methodological and content areas. The first meeting of the TWG was held from May 31 through June 2, 2006. The second meeting of the TWG was held from September 5 through September 7, 2006. The members of this group are

Dr. Geoffrey Borman, Associate Professor, University of Wisconsin-Madison

Dr. Robert Boruch, Professor, University of Pennsylvania, Wharton School,
Graduate School of Education

Dr. Robert D. (Robin) Morris, Vice President of Research, Georgia State University

Dr. Andrew Porter, Director, Learning Sciences Institute, Vanderbilt University

Dr. Robert St. Pierre, President, STP Associates

Dr. Kathy Escamilla, Associate Professor of Education, University of Colorado

Dr. John Golden, President/CEO, Cross Cultural Consulting, Inc.

9. Payments to Respondents

Not Applicable.

10. Assurances of Confidentiality Provided to Respondents

During recruitment, information on each district’s IRB policies and procedures will be collected. The study’s Principal Investigator, Co-PI, and study team members will adhere to all district and state IRB policies and procedures. The PI and authorized personnel from each participating district and school will sign Memoranda of Understanding (MOUs) to help ensure clarity of expectations and roles and responsibilities. A copy of the MOU appears in Exhibit C.

Informed consent will be sought and obtained from each participating teacher and their students’ parents or legal guardians. Informed consent will be sought after school principals have approved the conduct of the study in their school and signed an MOU. Teacher and parent informed consent letters are written to clearly communicate the research purposes, procedures, and risks and benefits. The consent letters will assure participants that reports prepared for this study will summarize findings across the sample and will not associate responses with a specific school or individual, and that the study team will not provide information that identifies specific schools or individuals to anyone outside the study team, except as required by law. Also included are statements offering teachers and parents the opportunity to ask questions and withdraw at any time. We will provide informed consent letters written in other languages as necessary and requested by districts or schools. It should be noted that both teacher and parent consent letters have been written to reflect a 10^th and 8^th grade reading level, respectively.

The likelihood of breaching anonymity of teacher data is low to none. The procedures used to collect and analyze the data are designed to prevent any supervisor or other district personnel from knowing teachers’ responses, including self-addressed stamped envelopes into which teachers seal completed logs and work samples and mail directly to McREL researchers, assigning subject ID numbers, and stripping all data of identifying information. For each data collection instrument, the following notice will be included:

Responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law.

McREL and other organizations that are part of the research team will follow procedures for ensuring and maintaining confidentiality that are consistent with the provisions of the Privacy Act of 1974, 5 USC 552a; The Freedom of Information Act, 5 USC 522; and Title 34, Code of Federal Regulations, Part 97, Protection of Human Subjects, which includes Subpart A, Basic Policy, and Subpart D, Additional Protections for Children.

McREL follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183).  McREL will protect the confidentiality of all information collected for the study and will use it for research purposes only.  No information that identifies any study participant will be released.  Information from participating institutions and respondents will be presented at aggregate levels in reports.  Information on respondents will be linked to their institution but not to any individually identifiable information.  No individually identifiable information will be maintained by the study team.  All institution-level identifiable information will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required.  McREL obtains signed NCEE Affidavits of Nondisclosure from all employees, subcontractors, and consultants that may have access to this data and submits them to our NCEE COR.

11. Justification for Questions of a Sensitive Nature⁶

No questions of a sensitive nature will be included in the study.

12. Estimates of Hour Burden of Data Collection

Table 2 provides estimates of hour burden for data collection. As shown in the table:

• The annual cost to the federal government (across the three years of study) for data collection is estimated at $9,375;

• The total number of respondents (annually) will be 224;

• The total number of annual responses will be 896; and

• Approximately 80% of responses will be submitted electronically.

Table 2. Estimated Respondent Burden

Data Collection Activity & Responsible Party	Number of Respondents per Data Collection	Number of Instances	Number of responses	Time per Response (in minutes)	Total Hour Burden	Hourly Rate*	Total Monetary Burden
Teachers
Teacher Background /Classroom Level Data (Treatment and Control Teachers)	192	1	192	10	32	$30.10	$963
Participating Teacher Online Log (Treatment and Control Teachers)	192	8	1536	25	640	$30.10	$19,264
Classroom Observation Protocol & Interview (observations during regular classtime; no additional time needed) and Interview (Treatment and Control Teachers)	192	2	384	15	96	$30.10	$2,890
Site coordinators
IPT & Student Data (administration of IPT at both Treatment and Control School Classrooms)**	192	3	576	14	240	$30.10	$4,045
Retrieval of state achievement test results (Treatment and Control Schools --Site Coordinators & Liaisons)	48	2	96	30	48	$30.10	$1,445
Site Coordinator Background / School Level Data (Treatment and Control Teachers)	48	1	48	15	12	$30.10	$2,709
Site Coordinator Artifact Submission (format varies)	24	8	192	5	16	$30.10	$482
Scheduling logistics (format varies)	48	2	96	10	16	$30.10	$482
Total	936		3,120	135	1,100		$35,458
Average Annual burden	312		1,040		367		$11,819

*The U.S. average elementary public school teacher salary for 2003-04 was $44,300
(see http://nces.ed.gov/pubs2006/2006313.pdf). This corresponds to $30.10 per hour (assuming teachers are contracted for 184 days per school year and that the average length of a school year is 180 days: http://nces.ed.gov/surveys/pss/tables/table_15.asp)

** Average test time is 14 minutes per classroom. Student demographic data will be collected at this time.

13. Estimates of Total Cost Burden to Respondents

There are no respondent costs associated with this data collection other than the hour and cost burden estimated in item 12. There are no start-up costs associated with the data collection for this project.

14. Estimates of Annualized Cost to the Federal Government

The estimated cost to the federal government of conducting the Study of the Differential Effects of ELL Training and Materials is approximately $2 million total across the entire course of the study.
The annual cost is approximately $500,000 per year with larger budgets for years three, four, and five when data collection and analysis occur. The cost of the five-year study total includes the following expenses:

Semi-annual meetings with REL Directors and Department of Education, planning, development, document review and revision, and consultations with Mathematica and IES	$164,000
Consultation with Technical Working Group	$150,000
Recruitment of sites	$25,000
Design, IRB, and OMB approval processes	$40,000
Baseline data collection and random assignment	$20,000
Data collection in Years 2, 3, and 4	$703,000
Data analysis	$550,000
Report preparation	$368,000

15. Reasons for Changes or Adjustments in Burden

This is a new collection.  Therefore, the entire burden is new.

16. Tabulation, Analysis, and Publication Plans and Schedule

This study begins in 2007. OWE and RISE will be implemented in treatment schools during the 2007-2008 and 2008-2009 school years. Data on classroom practices, student activities, and student language and literacy will be collected each of these years. Intermediate and cumulative effects of the interventions will be analyzed using year-end data and data collected over the course of the study. Other analyses may explore education mechanisms that contribute to variation in the impact on achievement. Table 3 presents the schedule for the major activities that will occur over the course of the study.

Table 3. Schedule of Activities

Activity	Schedule
Design
TWG review of research design; revisions & finalization. Submit to IES & obtain approval	2006-07 school year
Design
Instrumentation
Develop & pilot test instruments	2006-07 school year
Human Participants
Submit OMB package & obtain clearance; Obtain IRB reviews; secure permissions	2006-07 school year
Sampling
Identify & recruit pool; Select schools & randomly assign to treatment or control groups	2006-07 school year
Establish MOUs w/ participating schools & conduct orientation meeting	2006-07 school year
Implementation & Fidelity
Document aspects of interventions	Ongoing
Implement RISE, OWE (Harcourt)	2007-08 school year; 2008-09 school year
Monitor & document implementation fidelity	Ongoing
Data Collection
Administer online teacher logs	2007-08 school year; 2008-09 school year
Administer & collect student achievement data instrument (IPT)	2007-08 school year; 2008-09 school year
Collect state student achievement data	2007-08 school year; 2008-09 school year
Data Analysis
Finalize plan; obtain TWG review	2008-09 school year
Analyze data (all sources)	2008-09 school year; 2009-10 school year

Activity	Schedule
Reporting
Write technical & non-technical reports; review by TWG & internal QA; revise & finalize reports (submit to IES)	2009-10 school year
Disseminate reports; write journal article; present findings at research conferences	2009-10 school year
IES to post report on the IES Regional Educational Laboratory program website	2010
Create study data file & documentation; submit to IES	2009-10 school year

Analysis. This section provides detail on how a number of key issues related to data analysis will be addressed.

Pre-intervention Analyses. Pre-intervention analysis will be conducted prior to the implementation of the intervention to assess the comparability of the treatment and control groups to ensure the two groups are statistically equivalent. Although random assignment to treatment and control conditions ostensibly renders all groups equivalent, it is theoretically possible that treatment and control groups will differ on important variables prior to exposure to the intervention. Because of this possibility, researchers will compare the treatment and control groups on variables such as teachers’ service years, teacher background/content area, extent of teacher preparation for teaching ELLs, teacher and student gender, teacher and student ethnicity, student free or reduced-price lunch eligibility, student’s primary language spoken at home, and student’s level of literacy in their primary language, where available. In addition, it is expected that student scores on the initial administration of the IPT will be equivalent across all groups. In the event that large differences are discovered between the two groups, a review of the random assignment procedure will ensue along with the use of statistical methods to adjust for pre-intervention differences.

Descriptive Statistics. Descriptive statistics will be collected to describe the characteristics of participating teachers and students. Descriptive data for all participating teachers and their students will be collected. These data will include teacher characteristics such as number of years teaching, background/content area, and the extent of preparation for teaching students learning English. Student characteristics will include demographic data such as gender, ethnicity, free or reduced-price lunch eligibility, and primary language spoken at home. Moreover, descriptive statistics will be produced for both groups on all variables included in the study, and for sample sizes for each experimental group and for sub-groups within the experimental groups. Descriptive statistics (e.g., means, standard deviations, frequency distribution, item-total correlations, and internal consistency) will also be used to examine the psychometric adequacy of all instruments used in the study.

Assumptions/Outliers/Data Treatment. Data will be examined to determine whether relevant statistical assumptions were met. Once data have been collected, researchers will check for issues of range restriction for assumptions concerning normality. Data will also be examined for outliers. All adjustments to data to address violations of assumptions or outliers will be documented and reported. Additional assumption checks specific to the HLM analyses will also be undertaken; these will include, but not be limited to, the examination of the normality of residuals for the multi-level models as well as the variance homogeneity of the residuals.

Attrition. To explore the internal validity threat posed by attrition, a number of analyses will be conducted in order to examine the rate of attrition and differential attrition. Differential attrition will be addressed by demonstrating post-attrition equivalence of the two groups. Following the advice of Shadish, Cook, and Campbell (2002), researchers will provide descriptive information about attrition, describing the overall rate of attrition, the differential rate for treatment and control conditions, and whether study completers differ from non-completers. In other words, analyses will be conducted and reported to examine the possibility of differential attrition; these analyses will be conducted for both the treatment and control conditions separately and then attrition rates will be compared across the two conditions. Reasons for exiting the study, pretest scores, and variables considered to be correlated with pretest data such as that concerning socio-economic status will be analyzed. To the extent possible (and with appropriate consent), participating students who exit the study will be followed through the duration of the study.

If it is determined that attrition is a factor in groups differing, methods for accounting for attrition will be undertaken. The goal of these analyses will be to estimate findings as if no attrition had occurred. Assuming that sound predictors of missing data are available, researchers will impute missing covariate data. Because of the known concerns with imputing sample means on variables of interest, researchers will impute values using the maximum likelihood algorithm method for multiple imputation based on multiple regression analyses in concert with estimations of random error.

Main Effects. Consistent with the random assignment of schools to either the intervention or control group, main effects will be analyzed at the school level and outcome data will be analyzed at the level of the student and at the level of the teacher. As previously specified, this design includes the assignment of schools to treatment and control conditions. Obviously this means that data are being collected on intact groups of students, as opposed to randomly assigning each student within a school to be — or not be — exposed to the intervention. Because students are nested within contexts (e.g., schools), it is imperative to use alternative methods of data analysis. Multilevel modeling approaches provide an alternative to analytic approaches based on ordinary least squares regression (OLS); because students are nested within contexts such as teachers, classrooms, and school districts, the assumption of independence of observations that underlies OLS is violated. Moreover, assessment of cross-level effects, or the assessment of how variables measured at one level affect variables at another, is problematic for analytic approaches based on OLS. In OLS, all data would need to be restructured to eliminate one level of the hierarchy. In other words, school-level characteristics — and any associated error — would be uniformly attached to each student’s case or student level data would be aggregated so that they can be analyzed at the school level. The proposed cluster randomized trial involves randomization at the level of the school and collection of outcome data at the level of the student and at the level of the teacher. Thus, estimation of treatment effects using models that take into account the structure of the data is considered the most appropriate methodology.

In the research at hand, the independent variable, assignment to treatment or control condition, is at the group level; however, the dependent variables, student achievement and teacher pedagogical practices, are measured at the individual level. This nested design necessitates analysis via multilevel models. Two separate models will be estimated — one to address the primary research question regarding student achievement, and one to address the secondary research question regarding changes in teacher pedagogy.

Estimation of Model 1: Student Achievement. In order to assess changes over time with regard to student achievement, the multilevel model estimated for students will be a growth curve model. The following reflects the revised unconditional model and the proposed plan for the conditional model.

Unconditional Model.

Level 1: Time within Students

Y

tijk

Level 2: Students within Classroom

π_0ijk = β_00jk + r_0ijk,

π_1ijk = β_10jk + r_1ijk

Level 3: Classroom within School

β_00jk = γ_000k + µ_00jk

β_01jk = γ_000k + µ_01jk

...

Level 4: School

٧ _000k = η₀₀₀ + ٧_000k

...

Conditional Model. The conditional model will be the same at Level 1 as the unconditional model. As appropriate, researchers will put in student-level and class-level covariates at Levels 2 and 3 (respectively) and school-level covariates and the dummy variable for treatment condition at the school level (Level 4; treatment versus control). Further, researchers will treat variables in models as random or fixed as appropriate.

Estimation of Model 2: Teacher Pedagogy. To examine the secondary research question addressing teachers’ classroom practices — Does the use of OWE in combination with RISE participation result in changed teacher pedagogical practices reflected in teacher behaviors and skills related to ELLs? — a two-level HLM, with teachers nested within schools, will be developed. Initial teacher practices (as assessed via teacher logs and observation) will be used as a covariate in the conditional model. The proportion of post-intervention variance anticipated to be explained by pre-intervention survey data was set at R2 = .50, which assumes a strong relationship between teacher practices on pre- and post-measures.

Unconditional Model. An unconditional, two-level hierarchical linear model will be estimated to address the secondary question.

Level 1: At Level 1 (within subjects, individual growth model), teacher practices will be modeled⁷:

(within-teachers model),

where is the outcome at time t for teacher i in school j, is the mean for cluster j and is the error associated with person i in cluster j.

Level 2: The Level 2 (cluster-level/school-level unconditional model) will be estimated as follows:

β_0j = Y₀₀ + Y₀₁W_j + Y₀₂X_ij + u_0j,

where Y₀₀ is the grand mean, Y₀₁ is the main effect for treatment, W_j indicates the treatment contrast (1/2 for treatment, -1/2 for control), Y₀₂ is the regression coefficient for the cluster-level covariate (pretest for teacher practices), X_ij is the cluster-level covariate centered around the group mean, and u0j is the random effect associated with each cluster.

Conditional Model. As was the case with the student model, unconditional models will be fit prior to consideration of the conditional (explanatory) models. The reliabilities of effects at each level, correlations among growth parameters, and the variability partitioned into individual growth parameters in Level 2 components will be examined before estimating conditional models. Such information will be used to determine whether the conditional (explanatory) model must be estimated. It is anticipated that Level 1 (teacher level) will predict teacher practices as a function of a combination of Level 1 variables that may include general teaching experience (years teaching, teaching certification status) and experience teaching ELLs (years teaching ELLs, knowledge of other languages, prior exposure to professional

development for teaching ELLs). Level 2 will take into account a regression coefficient for the cluster-level covariate (pre-measures for teacher practices).

Other multilevel models including additional school-level covariates may also be explored; these more complex models will be examined with respect to how well they explain between-school variance and improve the precision of the treatment effect estimates. The most parsimonious model will be selected.

Effect sizes will be calculated for all the outcome variables, regardless of the direction of effect. Means and standard deviations for each group and any appropriate subgroups will be provided in final reports to allow for the calculation of effect sizes and effect directions by the reader.

Reporting. The following sections provide an overview of the reporting plan for this study.

Study Report Preparation. During 2009-2010, the study team will prepare a technical report consistent with IES technical standards. The report will fully explicate the rationale, study questions, research design, method, and results. Findings for each of the research questions will be presented, threats to validity will be considered and ruled out as appropriate, and conclusions about the research questions will be drawn based on these considerations. The report will be prepared such that it is appropriate for a peer-reviewed scholarly journal. A non-technical report will also be prepared that discusses the study rationale and summarizes the findings. This non-technical report will highlight selected conclusions given in the technical report and discuss implications of the study for education policy and practice. The non-technical report will be made available to the public on the IES Regional Educational Laboratory program website.

Public- or Restricted-Use Data Files. Prior to data collection, the study team will begin a data codebook. Datasets will be created, updated, and managed during the duration of the study’s data collection and analysis efforts. The codebook will be updated throughout the study. The study team will consult with the Technical Working Group regarding the development of the dataset(s) and codebook(s). Upon completion of the study, the data file and an accurate codebook will be finalized to document the public- or restricted-use data files. Any identifiers will be removed from the final files.

Dissemination. Pending approval of findings, the study team will submit a version of the technical report for publication in a scholarly journal and for presentation at one or more research conferences, as appropriate. A non-technical report also will be prepared that discusses the study rationale, presents the research questions, and summarizes the findings. The report will summarize the study results and highlight selected conclusions. There will be no risk of deductive disclosure in the reports as all findings will be presented in aggregate form. Implications of the study for education policy and practice will be discussed. Dissemination will be aligned with the dissemination activities described in McREL’s dissemination plan under the current Laboratory contract.

17. OMB Expiration Date

Not applicable. We are not seeking this and plan to display the OMB control number and expiration of OMB approval on data collection forms.

18. Exceptions to Certification Statement

No exceptions to the certification statement are requested or required.

References

Bliss, G. (2005). Filling the void: Lessons from core knowledge schools. Washington, D.C.: The Core Knowledge Foundation.

Brophy, J., & Good, T.L. (1986). Teacher behavior and student achievement. In M. Witrock (Ed.), The third handbook of research on teaching (pp. 328-375). New York: Macmillan.

Collier, V. P. (1995). Acquiring a second language for school. Directions in Language and Education, 1(4). Available online: http://www.ncela.gwu.edu/pubs/directions/04.htm

Darder, A. (1993). How does the culture of the teacher shape the classroom experience of Latino students? The unexamined question in critical pedagogy. In Rothstein, S. (Ed.), Handbook of Schooling in Urban America. Westport, CT: Greenwood Press.

Englert, C. (1984). Effective direct instruction practices in special education settings. Remedial and Special Education, 5(2), 38–47.

Freeman, D. & Freeman, Y. (2000). Meeting the needs of English Language Learners. Talking Points, 12(1), 2-7.

Gersten, R., Baker, S. K., Haager, D., & Graves, A. W. (2005). Exploring the role of teacher quality in predicting reading outcomes for first-grade English learners. Remedial and Special Education, 26(4), 197-206.

Godina, H. (1998). Mexican-American high school students and the role of literacy across home-school-community settings. (Doctoral dissertation, University of Illinois at Urbana-Champaign, 1990). Dissertation Abstracts International, 59(8), 2825.

Haager, D., Gersten, R., Baker, S., & Graves, A. W. (2003). The English-language learner classroom observation instrument for beginning readers. In Vaughn, S. & Briggs, K. L. (Eds.) Reading in the classroom: Systems for the observation of teaching and learning. Baltimore, MD: Brookes.

Henderson, R.W., & Landesman, E.M. (1992). Mathematics and middle school students of Mexican descent: The effects of thematically integrated instructions (Research Rep. No. 5). Santa Cruz, CA: University of California, National Center for Cultural Diversity and Second Language Learning.

Herman, J.L., & Abedi, J. (2004). Issues in assessing English language learners opportunity to learn mathematics. Paper presented at the meeting of the American Educational Research Association, San Diego, CA.

Hill, J.D., & Flynn, K. (2004). English language learner resource guide: A guide for rural districts with a low incidence of ELLs. Aurora, CO: The Mid-continent Research for Education and Learning.

Klingler, J. K. & Vaughn, S. (2000). The helping behaviors of fifth graders while using collaborative strategic reading during ESL content classes. TESOL Quarterly, 34(1), 69-98.

Krashen, S. (1996). Under attack: The case against bilingual education. Culver City, CA: Language Education Associates.

Ma, J. (2002). What works for the children? What we know and don’t know about bilingual education. Cambridge, MA: Harvard University, Civil Rights Project.

Menken, K. (2000). What are the Critical Issues in Wide-Scale Assessment of English Language Learners? Washington, D.C.: National Clearinghouse for Bilingual Education.

Minucucci, C. (1996). Learning science and English: How school reform advances scientific learning for limited English proficient middle school students (Educational Practice Report No. 17). Santa Cruz: University of California, National Center for Research on Cultural Diversity and Second Language Learning.

National Council of Teachers of English. (2006). NCTE position paper on the role of English teachers in educating English language Learners (ELLs). Available online: http://www.ncte.org/about/over/positions/category/div/124545.htm

Santos, M. G. (2004). Raising the achievement of English-language learners: How principals are working to make a difference. Harvard Education Letter, 20(2), 6–8.

Slavin, R. E. & Cheung, A. (2003). Effective reading programs for English language learners. Report No. 66. Center for Research on the Education of Students Placed at Risk: Author.

Smith, F. (1983). Essays into literacy: Selected papers and some afterthoughts. Portsmouth, NH: Heinemann.

Tikunoff, W. J., Ward, B. A., van Broekhuizen, L. D., Romero, M., Castaneda, L. V., Lucas, T., et al. (1991). Final Report: A descriptive study of significant features of exemplary special alternative instructional programs. Los Alamitos, CA: Southwest Regional Educational Laboratory.

1 The IPT Form A will be administered to all ELL students, regardless of when they enter the participating school. The IPT Form A will be administered through April 6, 2008. Administration of the IPT Form B will begin April 7, 2008, with all students entering after that date completing the IPT Form B. Form A and B are parallel test forms; however, alternating these will diminish the internal validity threat, testing., which refers to effects due to the number of times participants take the test rather than the effect being due to the intervention.

2 Depending on the state from which we draw our sample, the requirements for ELLs will differ. We will have a controlled data collection effort, with all students being tested with the same assessment instrument, regardless of state testing requirements (and state-allowed exceptions for ELLs). This necessitates the use of an assessment instrument other than the state assessments used to meet NCLB requirements. To the extent possible, state testing data will be collected for participating students to gain a better understanding of how/whether the IPT correlates with the state test and to gauge student gains in reading and mathematics.

3 Toward the beginning of each identified month, an email (containing a link to the study instrument) will be sent to participating teachers to remind them to complete their online instrument. This instrument will be piloted with a group of nine teachers prior to use with the study sample.

4 To provide sufficient confidence in findings, approximately 130 classrooms will be observed.

5 Site Coordinator characteristics will be developed in conjunction with Harcourt; REL Central anticipates that characteristics will include those of the Site Coordinator Characteristics and Information Sheet, including professional education experience: administrative experience and teaching experience.

6 This study was reviewed and approved by McREL’s Institutional Review Board and has undergone internal peer review.

7 To ensure that the observed data could be assumed to have reasonably come from a population in which the model is working, empirical growth plots will be examined.