SIOP OMB Supporting Statement A Revised 3.10.08 with TRK CHaccepted

This document presents the Supporting Statement for a study of the Sheltered Instruction Observation Protocol (SIOP) model, an approach to enhancing instruction of English Language Learners (ELLs) in mainstream classrooms. We are seeking a 3-year clearance to collect and analyze data as part of a study investigating The Effectiveness of Sheltered Instruction on English Language Learners in Georgia 4^th and 5^th Grade Classrooms (SIOP). This study will consist of a group-randomized control trial with quantitative and qualitative data collection to be conducted by the Regional Educational Laboratory—Southeast (REL-SE) at the SERVE Center at the University of North Carolina at Greensboro. The study has two main objectives: (a) to enable the southeastern region’s policymakers and educators to make decisions about approaches to supporting mainstream teachers’ instruction of ELLs; and (b) to expand the research base on the Sheltered Instruction approach to teaching English Language Learners (ELLs). The initial training of the treatment group teachers in the SIOP model is scheduled to take place during Summer 2008, and direct collection of data would take place in Spring 2009. Training in the SIOP model would be offered to control group teachers in Summer 2009.

The REL-SE is one of ten regional laboratories funded by the U.S. Department of Education's Institute of Education Sciences for the purpose of providing research-based information and services to all 50 states and territories. These Laboratories form a nationwide education knowledge network, building a bank of information and resources shared and disseminated nationally and regionally to improve student achievement.

The REL-SE has been charged with state and district responses to No Child Left Behind (NCLB), specifically by conducting studies that reflect NCLB's emphasis on evidence-based education and NCLB requirements to improve student outcomes. One key area of need in the REL-SE’s area is improvement of educational outcomes of ELLs--the Southeast has become a region of hypergrowth for immigrant populations. The state of Georgia, one of six states served by the REL-SE, has expressed an interest in hosting this study as it confronts record growth in its student ELL population. Georgia’s official educational approach to providing service to ELLs is to provide “pull-out” instruction for ELLs where the students are pulled from their mainstream classrooms for some period in the day and provided with basic instruction in the English language. The state is exploring ways in which schools can expand the instructional options utilized with English language learners in order to improve the achievement of this student subgroup. Therefore, there is interest at the state and local levels in identifying approaches that will solve this problem.

Sheltered Instruction (SI) is an approach to teaching ELLs in mainstream classrooms using a variety of techniques and materials that enhance student learning. One model of SI, the Sheltered Instruction Observation Protocol (SIOP) model, is readily available and widely used but has limited evidence of efficacy. Together, the fact that the state of Georgia is interested in exploring alternative approaches to supporting ELLs’ instruction and the need for rigorous study of the effectiveness of the SIOP model combine to form a strong rationale for this study.

The study will employ a random-assignment design whereby 88 schools will be recruited and randomly assigned to either receive the full treatment during the study year or receive a partial treatment during the summer following the study year. One-half of the schools will be assigned to the treatment condition, the other half to the control condition.

Specifically, the study seeks to answer the following research questions:

Do SIOP-trained teachers score significantly higher on the Standards Performance Continuum (SPC; CREDE, 2002) observation protocol than do control teachers? This analysis will investigate whether the SIOP-trained teachers demonstrate greater use of instruction practices believed to be effective for diverse learners—practices associated with sheltered instruction and that are known to benefit students at risk of educational failure because of cultural, linguistic, or economic risk factors” (Doherty, Hilburg, Epaloose, & Tharp, 2002, p.78)—compared with the teachers who have not received SIOP training.
Do ELL students in schools with SIOP-trained teachers score significantly higher than ELL students in schools with non-SIOP-trained teachers on the ACCESS assessment (Georgia's state-adopted English-language proficiency assessment)? The aim of this investigation is to examine whether SIOP procedures benefit the development of ELL students’ academic English language proficiency.
Do ELL students in schools with SIOP-trained teachers score significantly higher than ELL students in schools with non-SIOP-trained teachers on the CRCT five content areas (Georgia’s end-of-year achievement tests)? The aim of this investigation is to examine whether SIOP methods benefit the development of ELL students’ achievement in the measured subject areas; that is, whether the SIOP-trained teachers facilitate ELLs’ access to academic content in five subject areas.

This study will also address the following additional research question to understand the nuances of the impact findings.

Do non-ELL students (i.e., native English-speaking students) in schools with SIOP-trained teachers score significantly higher than non-ELL students in schools with non-SIOP-trained teachers on the CRCT five content areas (Georgia’s end-of-year achievement tests)? The aim of this investigation is to examine whether SIOP methods benefit the development of non-ELL students’ achievement in the measured subject areas; that is, whether the SIOP-trained teachers facilitate non-ELLs’ access to academic content in five subject areas.

A table of units of study, analysis variables, and measures that will be used to answer each research question is included in Appendix A.

The Teacher Survey, which is the focus of this Information Collection request, will yield constructs that will be used as covariates in our model for estimating impacts on teachers’ instructional behaviors. In order to attribute effects to the SIOP training and support, we must take into account the other professional development activities in which teachers participate and support received by teachers. Some items on the Teacher Survey will serve primarily to provide context for the study in order to help readers interpret our findings.

This submission provides an overview of the planned data collection and provides details on forms and procedures used for recruitment of participants, participant consent forms, and data collection materials. In addition, the submission includes estimates of respondent burden that are associated with data collection efforts described herein.

Overview

Georgia is a state with fairly recent exponential growth of English Language Learners in the public school population. Between 1993 and 2004, Georgia experienced a 378% increase in its school-age ELL population (NCELA, 2004). Of the 75,000 ELLs currently enrolled in Georgia’s public schools, at least 70% are Spanish-speaking, while the remaining 30% represent over 100 different languages. Statewide, ELLs comprise 4.6% of all public school students, and two-thirds of ELLs are elementary school children. The percentage of ELLs in individual Georgia public schools ranges from 0 to 83%. Though ELLs are concentrated in the Atlanta metro area districts (in the northern portion of the state), there are ELLs in schools in all regions of the state. Therefore, there exists a strong need for proven interventions that support ELLs and their academic achievement.

Sheltered Instruction (SI) refers to philosophies and techniques of classroom instruction for school-age children that address the challenges that English Language Learner (ELL) students confront in the school environment. SI is intended to assist learners in developing communicative, grade-level language proficiency alongside their attainment of content knowledge and skills across the range of subject areas in mainstream educational settings. This approach is not new and rests on the premise that language learning is most effective and meaningful when connected to content instruction (Snow, Met, & Genesee, 1989). Academic subjects are what students spend considerable time communicating about in school, and content-based language instruction motivates students while providing them with the opportunity to engage in meaningful communication. SI focuses on “making content comprehensible…because it explicitly emphasizes language and content objectives” (Herrera & Murry, 2005, p. 251).

The Sheltered Instruction Observation Protocol (SIOP) model operationalizes the SI approach by combining instructional techniques with support for lesson planning and implementation. The model incorporates eight components, which comprise the observation protocol:

Lesson preparation
Building Background
Comprehensible input
Strategies
Interaction
Practice and application
Lesson delivery
Indicators of review and assessment

The SIOP is an observation protocol, which contains a rubric that can be used for self-evaluation, peer evaluation, and as a basis for coaching observations. Its eight components (containing a total of 30 individual indicators) are bundled and serve as the foci of “component enrichment” training sessions and coaching visits, both of which occur at intervals in the year following the initial training. The model to be tested in the present study is one that is promoted by the publisher as the “typical” model for use by districts and schools, a package that is believed to be sufficient by itself to produce changes in instructional behaviors that will lead to improved outcomes for students. The package includes a 3-day intensive training institute, four follow-up day-long component enrichment training sessions, and two coaching visits, with all training and coaching provided by the publisher’s faculty. This model is in wide use across the country in spite of a lack of rigorous research evidence to support its effectiveness.

A search of the literature shows that there has never been an experimental research study to test the efficacy of the SIOP model. In fact, the single published research study (Guarino, Echevarría, Short, Schick, Forbes, & Rueda, 2001) employed a quasi-experimental design. Nevertheless, the popularity of the SIOP model rests in part on its rigorous development with involvement from teacher practitioners, its dissemination efforts, and anecdotal evidence of effectiveness in the schools. As part of a multiyear research project, SIOP developers worked with middle school teachers in four large metropolitan school districts to identify key practices for sheltered instruction and to develop a professional development model and observation protocol (Echevarría, Vogt, & Short, 2000; Short & Echevarría, 1999). A study of SIOP was later conducted to establish the validity and reliability of the SIOP as an observation instrument. Reported findings show that the instrument had high inter-rater reliability (0.99) when the instrument was used to differentiate between high- and low-implementers of the SIOP model. In addition, middle school students in classrooms with SIOP-trained teachers were reported to achieve significantly higher scores on expository and narrative writing assessments than comparison group students (Echevarría, Vogt, & Short, 2004; Echevarría & Vogt, 2006). No information is available on the internal consistency of the SIOP instrument with respect to the 30 indicators. Information is also lacking on the content-area achievement as well as language proficiency in other modalities (e.g., listening, speaking, and reading) of ELL students in classrooms of SIOP-trained teachers.

In summary, the limited available evidence to support SIOP is a single published quasi-experimental study that focused on middle school students and is not, in any case, rigorous enough to warrant confidence in its results. The current study will focus on 4^th and 5^th grade students in Georgia and will utilize a rigorous experimental design.

The rationale for designing an experiment to test the effectiveness of SIOP in 4^th and 5^th grades in Georgia is driven by an identified need for information about these grade levels but also rests on the following logistical advantage: When teaching and instructional resources for ELLs are limited (as in Georgia where school/program funding formulas are such that not every school with ELLs receives funding for such resources and where there is a shortage of teachers trained in teaching English to Speakers of Other Languages (ESOL) and sheltered instruction), often the limited support is focused only on the lower grades, K-3, with 4th and 5th graders receiving little to no support. Hence, this allows us to provide needed support to ELLs at grade levels that are often overlooked. Further, that these grade levels are still located within the elementary school context is advantageous, as one trained teacher can provide instruction to all students in his/her class for all subject areas, as opposed to the situation in middle school or high school where students receive instruction from several teachers. Hence, 4th and 5th grades were strategically chosen as the focus for the study.

Study Approach

To address the research questions of the SIOP Study, the REL-SE will: a) select and recruit 88 elementary schools (from Georgia public schools) for the study; b) randomly assign the school to either a treatment group that will receive the full treatment or a control group that will receive partial treatment after the conclusion of the study year; c) collect a variety of forms of data on teacher characteristics and activities and student achievement; and d) analyze the data and report the results of our analysis.

Selecting sample. The evaluation will focus on the effects of the SIOP model in upper elementary school grades (grades 4 and 5). Upper elementary grades were chosen primarily because of the lack of information about the effectiveness of educational interventions, in particular the SIOP model, at this grade level. We will select 88 schools for the study based on the percentage of ELL students in their school population (for example, they must have at least 5% ELLs in their school and be in districts with at least one school that has at least 15% ELLs). The schools must also have at least one teacher at each grade level willing to participate in the study in order for the school to be eligible to participate. Additional details on the selection of schools for the study are provided in Section B.1.

Randomly assigning schools. Of the 88 schools selected for the study, 44 of the schools will be randomly assigned to the treatment group, and offered the full treatment, and 44 of the schools will be randomly assigned to the control group and offered only the initial training portion of the treatment in the summer following the study year. These schools will yield samples of approximately 616 teachers (308 treatment, 308 control), 1848 ELL students, and 15,400 non-ELL students (with numbers based on assumptions about number of participating 4^th and 5^th grade teachers in a school, 2006-2007 data from the Georgia DOE, and estimates of class size). The statistical power of this design is discussed in Section B.2.

Collecting data. For the evaluation, we will collect data from various sources. We will collect baseline data from administrative records held in the Georgia Department of Education (GA DOE): databases on staff characteristics (CPI data base) and student assessments (Testing database). We plan to collect these data in the summer at the start of the study year. In the spring of the study year, we will conduct site visits to observe all study classrooms, and we will conduct web-based surveys of teachers in study classrooms. We will also collect follow-up student assessment data from the GA DOE database in the summer at the end of the study year. An overview of the data collection plan for the study, along with key pieces of information to be obtained from each data source, is shown in Table 1.

Ensuring that the comparison group provides a valid counterfactual. Although SIOP is very popular and Pearson Achievement Solutions has made training readily available, the cost of providing training to teachers (see question #5 below) has prevented most districts and schools from providing high quality training in the SIOP approach to date. In recruiting schools, we propose to include only schools that could provide a suitable comparison with the proposed treatment. That is, one of two types of schools: First, schools where teachers have not yet received training in SIOP; second, schools in which teachers have received a only very small “dose” of the treatment. Schools in which teachers have received a large “dose” of the treatment would be unlikely to be interested in participating in the study as their teachers would not be motivated to repeat training they have already received. Our Teacher Survey includes questions aimed at identifying teachers who have received any SIOP training prior to the study or receive it during the study year (“crossover” of comparison group members). Our Policy Analyst at the GaDOE, Dr. Kimberly Anderson (who is a SERVE employee), is aware of several levels of SIOP exposure currently in the field in Georgia:

Districts identified enhancing preparation of mainstream teachers’ ability to work with English Language Learner (ELL) students as a key area for professional development, have sufficient financial resources to address this need, and have provided training for:

ESOL (Title III) staff at the district level: Districts that have adopted this strategy may have arranged for Title III staff to receive a 3-day training and some follow-up workshops. The Title III staff then provide periodic support to mainstream teachers in the district (in the form of in-service workshops, for example). This—train-the-trainer—model of SIOP training is considered less desirable (but more affordable) than the simplest treatment model of teachers receiving the 3-day training or than the model offered in our proposed study. Teachers in schools who had received this sort of train-the-trainer treatment would still be suitable for either the comparison or treatment group.
ESOL (full or partially Title III-funded) staff at the school level: Districts and schools that have adopted this strategy are still following a train-the-trainer model for training their mainstream teachers in SIOP, and it is still considered less desirable. In these schools, the ESOL staff provide limited amounts of training on SIOP components to mainstream teachers, or alternatively, may not even be expected to train their fellow teachers at all but instead may be expected to shoulder the load of enhancing educational services independently of the mainstream teachers (as per usual and customary implementation of the Georgia ESOL pull-out model of instructional support for English Language Learners). We feel that in most schools that have used either of these approaches, the administrators and teachers who would be interested in participating in the study would make that decision based on their judgment of the effectiveness of having mainstream teachers trained directly. Schools that express interest in participating would, thus, also be suitable candidates for inclusion in the study.

Districts identified low performance of ELLs as a reason for not meeting Adequate Yearly Progress (AYP) goals and have begun to dedicate financial resources to solving the problem. Efforts have taken several forms:

Funding has been made available for SIOP training for ESOL staff only (see a.2 above).
Funding has been made available for SIOP training for all interested staff: In schools like this, typically administrators and teachers will not be interested in participating in the study, and we will not include them.

Districts have identified enhancing services to ELLs as an area of need but have not been able to dedicate financial resources to providing training for mainstream teachers and/or have such low percentages of ELLs in their schools that they do not qualify for Title III funding (or qualify for less than a FTE-worth of funding), and so have not funded any SIOP training. In such districts and schools, there are two types of schools:

Schools in which no staff have received any SIOP training: These schools are typically very eager to participate in the study. If assigned to comparison group, they would provide an ideal “uncontaminated” comparison.
Schools in which some staff have received SIOP training: These schools are typically eager to participate in the study, but the staff who have already received training may not be interested in participating in the study, leaving only staff who resemble those in group c.1 above.

In summary, we believe that staff who have already received high-quality SIOP training will be less likely to participate in the study, leaving only staff who have not received treatment like what we are proposing in this study, and the comparison group would thus be relatively uncontaminated. In addition, we propose to administer the SIOP protocol as part of the classroom observation in all classrooms (treatment and comparison), which will help us detect SIOP-like instructional behaviors, and our Teacher Survey includes questions asking teachers about their exposure to SIOP training.

We will exclude schools from participation if SIOP training has already been provided to teachers in the targeted grade levels (grades 4 and 5). We will make this decision at the time of verbal communication with the school as a part of determining whether the school is eligible. Schools are eligible for the study if they (a) have students in grade 4 and 5, (b) meet the percentage ELL criterion (5% or more), and (c) have not provided all or most of their teachers in grades 4 and 5 with SIOP training. Eligibility decisions will be made as study team members contact potentially eligible schools to provide information about the study and what study participation would entail.

It is possible that a teacher previously trained in SIOP will be trained again (though, as with the schools that have previously had SIOP training, these teachers are extremely unlikely to choose to participate), and it is also possible that a teacher previously trained in SIOP will be in a school assigned to the control group. It is not a problem that a small number of individual teachers have prior SIOP experience, either in the control group as part of the counterfactual or in the treatment group as the soon-to-be-double-trained, if one is trying to learn how much difference school-wide or grade-level-wide launching of SIOP will make in today’s Georgia environment. Having some SIOP exposure in these settings already is desirable, not problematic, when doing a study that will guide policy decisions about expanding or eschewing SIOP in just such environments.

Schools that made funding available to provide SIOP training to most or all of their teachers in grades 4 and 5 are not eligible for the study. Therefore, they will be excluded from the study and not given the opportunity to choose to participate or opt out.

As a note, the approximate cost to a school of providing the intervention model employed in this study would be $2,031.25 per participant [teacher or administrator] during one school year. This estimate is based on the 2007-2008 school year contract price negotiated with Pearson Achievement Solutions.

Table 1

Data Collection Plan

Subject	Mode	Timeline	Key Data
Teachers	GA CPI database Web-based Survey Classroom observation	Late Spring 2008 Late Spring 2009 Spring 2009	Demographic information Professional Development activities & support Instructional Activities
ELL Students	GaDOE student assessment database: ACCESS & CRCT	Summer 2008 (baseline) Summer 2009	English language proficiency Subject area mastery
Non-ELL Students	GaDOE student assessment database: CRCT	Summer 2008 (baseline) Summer 2009	Subject area mastery

Analyzing data. The analysis will focus on estimating the impact of teachers’ SIOP training on English Language Learner (ELL) student achievement in both English language proficiency and in the core subject content areas. Two reports will summarize the findings of the study: a final technical report summarizing the findings of the experiment, in December 2009, and a web-based policy brief, in January/February 2010. Articles will be submitted for consideration for publication in education and policy journals by 2010.

Part A. Justification

Explain the circumstances that make data collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

Since its development and publication less than a decade ago, the SIOP model has been steadily gaining popularity and is currently being used in all 50 states. The publisher estimates that there are at least 650 public school districts in the United States, serving students with limited English proficiency, that are currently using the SIOP model.

The model is promising, in that it is aimed at making subject area content accessible to English Language Learners, improving the academic performance of this growing student subgroup. The federal government supported the development of the SIOP model through funding of the research underlying the model, the development of the protocol, and efficacy trials. However, to date, no rigorous studies of the SIOP model have been completed, and none are underway that examine the efficacy of the model when used with students in the upper elementary grades. One published quasi-experimental study focused on effects on the writing performance of students in grades 6-8. Two additional studies of the SIOP model are currently underway, but no other experimental design study of the SIOP model has been completed. In addition, there is little research on the effectiveness of this model in teaching upper-elementary grades (4^th and 5^th grade), the grades targeted in the study that is the object of this Information Collection request.

The two current studies involving the SIOP Model focus on two different populations. The subjects in one study are 7^th grade students (Impact of the SIOP Model on Middle School Science Language Learning) while the other study spans K-3^rd grade (SAILL). Neither study focuses on the grade levels of the proposed study that is the subject of this Information Collection Request (grades 4 and 5).

The Impact of the SIOP Model on Middle School Science Language Learning study uses an experimental design (three-group random assignment of 25 schools, two treatment groups and a control group) and focuses on middle school science instruction. In the SAILL study, 1192 students were randomly assigned to treatment or control groups within language of instruction groups (ESL or Bilingual Education)—a two-by-two design.

The Impact of the SIOP Model on Middle School Science Language Learning study may indeed meet the WWC criteria of rigor. It would not have met the criterion for inclusion in the current WWC ELL Topic Report because it did not focus on the grades targeted in that report (elementary grades, K-6).

The SAILL study may meet the criteria for inclusion in WWC in the current ELL Topic Report and future ones.

We are attaching documents describing these two studies for further elaboration (See Appendix B).

According to Cynthia Ryan, Director of Discretionary Grants at OELA (personal communication, March 2008), IES has not evaluated interventions for OELA. However, below we detail studies that have been conducted, including those funded by or conducted by IES, that might be of interest to OELA.

The What Works Clearinghouse (WWC) Topic Report on English Language Learners reviewed twelve interventions (WWC, 2007), none of which specifically looked at Sheltered Instruction. Many of the interventions themselves focus on elements included in SIOP, such as peer/group work (Instructional Conversations and Literature Logs, Peer Tutoring and Response Groups, and Peer-Assisted Learning Strategies) and vocabulary development (Vocabulary Improvement Program), but there has been no evaluation of a “sheltered instruction” intervention.

Of the articles comprising the What Works for English Language Learners (WWELL) bibliographic database (National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs, 2008), the majority of studies were not related to sheltered instruction and ELLs. Of the six on approaches to instructing ELLs, the focus was on other approaches, such as bilingual education, English only, two-way immersion, English immersion, and transitional bilingual education, and not on any specifically operationalized model of sheltered instruction. The WWELL also included a Slavin and Cheung (2005) synthesis of 17 studies comparing bilingual and English only reading programs for ELLs that did not include SIOP specifically.

A search of the Research of the Effective Education of English Language Learners (REEELL) database (National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs, 2008) yielded 35 studies funded by IES. Again, the majority had no substantive relation to SI and ELLs. Five pertained to reading development and ELLs, two to assessment accommodations, two to teacher quality, four to overall research in the field, one to adult ESOL, three on Success for All, and one on 21^st Century Community Learning Centers. Four were potentially relevant to the SIOP Study and required further examination.

The first two studies (Optimizing Education Outcomes for ELLs—David Francis (PI), and Project English Language Literacy Acquisition—Raphael Lara-Alecio (PI)) appear to be connected to Success through Academic Intervention in Language and Literacy (SAILL) (Francis, D., Carlson, C.D., Rivera, H., Short, D., Vaughn, S.R., and Linan-Thompson, S.), which we discuss in Section A.1 of Statement A, due to its inclusion of SIOP for part of the intervention. The third (Oracy/Literacy Development in Spanish-speaking Children—Vaughn, et al.) examines Spanish- and English-language reading interventions, while the final study (The Big-Picture—Kellie Rolstad) is a meta-analysis of bilingual and all-English programs. While some of the studies may contain elements of SIOP, only the SAILL study (described in Section A.1) examines sheltered instruction or SIOP specifically.

The present study aims to address this gap in the research on the efficacy of the SIOP model through a rigorous experimental design study involving random assignment of schools and a focus on 4^th and 5^th grade classrooms. To estimate the impact of SIOP training, schools will be randomly assigned into two groups: a) a treatment group whose 4^th and 5^th grade teachers will attend a 3-day training institute in the first summer, will participate in four day-long enrichment trainings during the school year, and will be visited by SIOP coaches twice during the school year; and b) a control group whose 4^th and 5^th grade teachers will continue with “business-as-usual”—participating in whatever professional development opportunities their school or district would ordinarily have offered. The control group will be offered the three-day institute in the following summer.

The schools included in the study will not be representative of typical Georgia elementary schools in that only schools with at least 5% ELLs in their school that are in districts in which at least one school has at least 15% ELLs will be eligible to participate. This requirement is strictly for convenience—to enable us to recruit, train, and collect data on a sufficient number of schools to meet our power requirements and to remain within the resource constraints of the study. The power analyses are discussed in greater detail in Supporting Statement B, question #1.

Data to be collected include two record abstractions, one survey, and one round of classroom observations. We will collect extant data on administrative records for staff and accountability (student assessment) data on students. We will collect information about staff professional development activities through a brief survey and will collect information about instructional practices through direct observation. These data collections are described in greater detail below.

Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

Information on the effectiveness of SIOP will be collected by the Regional Educational Laboratory—Southeast (REL-SE) housed at the SERVE Center at the University of North Carolina at Greensboro (SERVE) and by its subcontractor, Abt Associates Inc. (Abt), under contract number ED-06-CO-0028 with ED. The data will also be analyzed by SERVE and Abt. The specific data to be collected (as summarized in Table 1) will be obtained from the GaDOE databases (the Classified/Certified Personnel Information, CPI, for staff data; the GaDOE student assessment database for student data), a web-based teacher survey, and classroom observation. (Please see the Teacher Survey in Appendix C and the Standards Performance Continuum and SIOP Protocol (observation instruments) in Appendix D).

These data will be maintained in a secure manner (described below, in response to Item 10), and respondents will not be publicly identified. In reports and publications, data will be aggregated to a group level in statistical models (e.g., student data will be aggregated by classroom, classrooms will be aggregated to the school level, schools will be aggregated to treatment group or control group). The purpose for sharing information through publications will be to inform the public about the effectiveness of the approach being tested in this study. Information will not be used to identify districts, schools, or individuals for any purpose.

Agreement Forms and Consent Forms

Our recruitment protocol begins with gaining support from the top level of educational administration in the State (Georgia’s State Superintendent of Schools) and then the Local Educational Agencies (where we obtain a non-binding agreement from the District Superintendent for district support of the study), proceeding to the school (where we obtain a non-binding agreement from the principal for the school’s participation in the study), and finally to the teachers (from whom we obtain signed consent forms). See Appendix E for the LEA Agreement and Appendix F the School Agreement.

Once school principals have agreed for their schools to participate, we will solicit signed consent forms from their 4^th and 5^th grade teachers. Appendix G shows the proposed Teacher Consent form.

Privacy of individual respondents is also protected through use of non-binding participation and consent forms. Participants who consent to be a part of the study do so voluntarily and are made aware that they may freely opt to discontinue participation in the study at any time without penalty. At no time will participants be coerced to provide information; any participant who wishes to guard his/her privacy may decline to permit classroom observation and will not be obligated to complete the Teacher Survey.

A parental consent form will not be required for access to student assessment data because the student assessment data that we propose to obtain will have been stripped of all identifiers that could lead to the identification of individual subjects. The Georgia Department of Education Data Collections Divisions Operations Analyst/Technology Management has assured our project manager that this process will be utilized.

Based on the published decision-making documents of the Institutional Review Boards (IRBs) at both The University of North Carolina at Greensboro (UNCG) and Abt Associates Inc., the following criteria [from 45 CFR part 46] were used in assessing whether or not parental consent would be required to access student assessment data from the state agency:

If the activity is a systematic investigation, designed to develop or contribute to generalizable knowledge [45 CFR 46.102(d)] [Yes], and

b. If the activity is research, involving obtaining information about living individuals

[45 CFR 46.102(f)] [Yes], and

If the research involves intervention or interaction with the individuals [45 CFR 46.102(f)(1),(2)] [No, because the students will just be participating in normal school activities led by their teachers], and

If the information will be individually identifiable (i.e., the identity of the subject is or may readily be ascertained by the investigator or associated with the information [45 CFR 46.102(f)(2)] [No], then

The research is not research involving human subjects, and 45 CFR part 46 does not apply; thus, parental consent will not be required for either students’ participation in the study or for the use of their de-identified assessment data.

Students in schools that are participating in the study will not be asked to do anything other than what is a normal part of the school’s adopted instructional approaches and curricula. Other IES studies may have required direct assessment of students for which parental consent would be necessary, but this proposed study (SIOP) involves no direct assessment of students beyond that conducted by schools in accordance with their state accountability mandates.

As noted above, the UNCG and Abt Associates IRBs agree that students in the SIOP study would not meet the criteria for Human Subjects such that they would be covered by 45 CFR part 46.

Administrative Records and Accountability Data

State administrative records will be used to measure key baseline and outcome variables in the analysis. Most importantly, we will measure student achievement using standardized state test scores from school records (collected from the state database). We will also measure baseline characteristics (such as teacher and school demographics) using state administrative records. School data to be collected cover the 2007-2008 baseline year (collected beginning in Summer 2008) as well as one follow-up year (2008-2009; collected in Summer 2009). Selected baseline characteristics may be used as covariates in our models in estimating impacts at follow-up if using them will increase the precision of our estimates.

Classroom Observations

Classroom observations will be conducted to obtain detailed information about classroom instruction to support causal inferences. The key proposed classroom observation instrument is the Standards Performance Continuum (SPC), a protocol with five standards, each rated according to a five-point rubric, designed to measure pedagogical effectiveness, specifically for teaching children “at risk of academic failure due to cultural, linguistic, or economic factors” (Doherty, Hilberg, Pinal, and Tharp, 2003, p.1). The five standards measured by this instrument are: a) Joint Productive Activity (“teacher and student producing together”); b) Language Development (“developing language and literacy across the curriculum”); c) Contextualization (“making meaning: connecting school to students’ lives”); d) Challenging Activities (“teaching complex thinking”); and e) Instructional Conversation (“teaching through conversation”) [retrieved January 22, 2007 from http://crede.berkeley.edu/tools/research/standards/spac.shtml)]

This instrument was selected because “consistent findings from correlational, quasi-experimental, and true experimental designs have documented a systematic relationship between use of the Five Standards [measured by this instrument] and a broad range of affective, behavioral, and cognitive indicators of improved student performance” (Doherty et al., 2003, p.3). The SPC has been used at grade levels targeted in the present study, in various classroom settings, and has well documented inter-rater reliability.

The SPC was developed to measure the five standards of effective pedagogy for diverse learners set out by Tharp and his colleagues (Tharp, Estrada, Dalton, and Yamauchi, 2000) within a socio-cultural constructivist theoretical framework. These are the same standards that formed the basis of the SIOP model. While the SIOP protocol delineates particular elements of the sheltered instruction model, the SPC focuses on the five underlying standards. Thus the SPC includes elements that we would expect to see in a highly implemented SIOP classroom as well as other aspects of instruction believed to be effective for diverse learners, including English Language Learners, that do not follow directly from SIOP. Appendix J shows a crosswalk of this overlap between the two instruments.

A validation study of the SPC (Doherty, Hilberg, Epaloose, & Tharp, 2002) provided evidence of inter-rater reliability, concurrent validity, and criterion-related validity supporting the validity of interpretations of data gathered with the SPC. This study found that in a school serving predominantly poor Latino English Language Learners, higher SPC total scores were associated reliably with greater student achievement on end-of-year standardized tests (in this case, the SAT-9). We do not have strong evidence that the instrument will be sensitive to the kinds of changes that the professional development intervention might produce, but at this time, there is not an instrument with demonstrated sensitivity that is well matched to our research question..

In addition, the Sheltered Instruction Observation Protocol (SIOP) tool will be used to document fidelity of implementation in the treatment classrooms and to document the presence or absence of comparable instructional practices in the control classrooms. While the SIOP protocol appears to be a checklist in some respects, it is a 5-point rubric. It measures the 30 features of the eight components of instruction targeted by the SIOP professional development intervention. It is the only existing measure of fidelity of implementation and is the measure that is used in the field by the developers, trainers, peer coaches, and supervisors. However, it does not have established psychometric properties. For this reason, we do not plan to include it in our statistical models; rather we intend to use it as a tool for making qualitative characterizations of the classrooms and the extent to which they resemble classrooms in which the treatment model is highly implemented.

To reiterate: We are intending to use the SIOP protocol for observation in treatment classrooms to document the extent of fidelity of implementation and in control classrooms to document circumstantial contamination. We were not intending to use these measurements in our statistical models.

Please note that there is no respondent burden associated with collection of information through the classroom observations (using the Standards Performance Continuum), which contain no questions to be asked of teachers. All information will be collected through direct observation rather than interviews, and all work will be done by SERVE staff.

Teacher Survey

In Spring 2009, we will conduct a web-based survey of all of the participating 4^th and 5^th grade teachers. This survey will yield information on the professional development activities in which teachers have engaged prior to the study and during the study year as well as other types of professional support available during the year. The emphasis of these questions is on activities that support high-quality instruction of English Language Learners (ELLs). This information will be used to characterize professional development activities and other types of support for instruction of ELLs in control schools relative to treatment schools in order to take into account the professional support available to teachers in absence of (or in addition to) the treatment. The treatment involves one 3-day training, four days of follow-up component enrichment training, and two classroom observation and coaching visits from an instructional coach. Since the treatment involves a substantial level of professional support, it will be important to document the level of support available to teachers in the absence of treatment. We plan to pilot the paper version of the web survey with nine teachers in the fall of 2008 and to pilot the web version with a different nine teachers in order to ensure that the software permits sufficient ease of use and functions as intended (e.g., access to the file from the web link with secure identification code, download of survey, completion of the survey, submission of survey with secure identification information, research staff retrieval of data submitted in appropriate form). Web-based surveys have been successfully used by the Georgia Department of Education with teachers throughout Georgia.

Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. Wherever possible, information will be gathered from existing data sources rather than imposing additional burden by collecting primary data. A key consideration in the collection of student achievement data via existing administrative records as opposed to administering standardized tests to participants was to minimize evaluation costs and reduce respondent burden. Similarly, we restricted teacher survey content to areas not covered by the existing state administrative records in order to minimize respondent burden and are planning to utilize a web-based format in order to reduce paperwork.

The Teacher Survey will be administered on-line. We anticipate a response rate of at least 85% and plan to follow up with mail surveys to non-responders. In this case, 85% of the survey would be collected electronically.

Describe efforts to identify duplication. Show specifically why any similar information available cannot be used or modified for use for the purposes described in item 2 above.

The purpose of the data collection is to estimate the impacts of SIOP teacher training on student achievement in both English language proficiency and core subject content areas. While one previous study has examined this effect with middle school students, that study was not sufficiently rigorous and was focused on older students. This study will employ a rigorous, experimental design and will focus on teachers and students in the 4^th and 5^th grades.

Conducting a randomized trial to evaluate the impact of SIOP training requires randomly assigning eligible schools to treatment and control groups. To the extent possible, we will use existing data for the study rather than duplicating data collection efforts. For example, in collecting demographic data on schools and teachers, we will obtain any required data that is available in the state database from that source rather than from a questionnaire to be completed by school staff. In addition, a major part of our data collection effort will be the collection of student assessment records data, which will be used in place of administering the standardized tests ourselves.

The information to be collected as part of the Teacher Survey, however, will not be available elsewhere. The information collected will represent the individual experiences of teachers—information they have that is not currently available elsewhere.

If the collection of information impacts small businesses or other small entities (item 5 of OMB Form 83-1), describe any methods used to minimize burden.

The primary entities for this study are schools and the districts to which they belong, along with the children who attend them. Burden is reduced for all respondents by requesting only the minimum information required to meet the study objectives. The burden on schools and districts has been minimized through the careful specification of information needs, restricting questions to generally available information (such as the professional development activities in which one participated during the current year), and designing the data collection strategy to minimize burden on respondents. All data collection will be coordinated by SERVE and its subcontractor, Abt Associates, so as to minimize burden on school and district staff and children. The only data being collected at the school level is classroom observation and the short teacher survey.

Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

The No Child Left Behind Act insists that education providers base their decisions on scientifically based research. However, there is very little research that carefully measures the effectiveness of different pedagogical strategies for teaching ELLs. Furthermore, there is a gap between the theories on which the published materials are based and the realities in the classroom. Even with the most rigorously designed academic research, a persistent gap remains between what educational scientists know about what works in general and what works in their particular school district. In the case of SIOP, the state of Georgia is seeking scientific evidence that is not otherwise available in order to make critical policy decisions regarding this instructional model.

Abstraction of extant data for sampling purposes is proposed to begin in Summer 2007 Data collection (classroom observation and teacher surveys) will be conducted in Spring 2009. Surveys will be available online between 30 and 45 days prior to classroom observation visits, and teachers will be notified of the site by e-mail and again by mail at the time of scheduling the classroom visits. Classroom observers will also be able to distribute and retrieve paper versions of the surveys from participants, at the request of the participants, at the conclusion of the observation visit. By utilizing extant data for the majority of data collection and employing a random assignment design, we have minimized the amount of new data collection required. New data will be collected once, in Spring 2009.

As stated, there is very little scientifically-based information related to the effectiveness of the SIOP model on students’ academic outcomes. Results of the current study are needed to assist policymakers in Georgia and across the nation to make informed decisions on its implementation within their schools. While findings will not be generalizable to the entire state (due to the fact that most schools in the state have a much lower density ELL population, where our study eligibility criterion is that a school’s population be composed of at least 5% ELLs and be within a district in which at least one other school has at least 15% ELLs), they should be valid for schools with similar demographics in Georgia. Similarly, findings should be generalizable to schools in other parts of the country with comparable demographics (comparable urbanicity, comparable density of ELL student population). Given that the population of English Language Learners in schools is growing nationwide, findings from this study should provide useful information to a broad audience.

Explain any special circumstances that would cause an information collection to be conducted in a manner: *requiring respondents to report information to the agency more often that quarterly; *requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it; *requiring respondents to submit more than an original and two copies of any document; *requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years; *in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study; *requiring the use of a statistical data classification that has not been reviewed and approved by OMB; *that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; *or requiring respondent to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

There are no special circumstances that would cause collection of information to be conducted in any manner listed under section seven of the OMB guidelines.

If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken buy the agency in response to these comments. Specifically address comments received on cost and hour burden. Consultation with representatives of those from who compile records should occur at least once every 3 years – even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

Federal Register Announcement

In accordance with the Paperwork Reduction Act of 1995, a 60-day notice to solicit public comments was published in the Federal Register on July 30, 2007 (p. 36436). No comments have been made to date.

A copy of the Federal Register Notice is attached in Appendix H.

Consultations Outside the Agency

Consultations on the research design, sample design, data sources and needs, and study reports have occurred during the study’s design phase and will continue to take place throughout the study. The purpose of such consultations is to ensure the technical soundness of the study and the relevance of its findings, and to verify the importance, relevance, and accessibility of the information sought in the study.

SERVE Center at the University of North Carolina at Greensboro (UNCG) and its subcontractor Abt Associates Inc. have provided substantial input to ED for the study. Senior technical staff from these organizations who are conducting the study are listed below:

SERVE Center at UNCG Ludwig D. van Broekhuizen 336-315-7402

Micheline Chalhoub-Deville 336-334-3472

SERVE Center at GaDOE Kimberly Anderson 404-657-6174

Abt Associates Inc. Stephen Bell 301-635-1721

Carolyn Layzer 617-520-3597

In addition to the above, an advisory panel has provided substantial input on the study design and data collection plan. The advisory panel members represent a number of the nation’s leading researchers on educational practice and policy and evaluation design. The panel includes:

Coyne, Michael D.	University of Connecticut
Dodge, Kenneth A.	Duke University
Dadisman, Kim	University of North Carolina, Chapel Hill
Vernon-Feagans, Lynne	University of North Carolina, Chapel Hill
Floden, Robert	Michigan State University
Gersten, Russell M.	Instructional Research Group
Kamil, Michael L.	Stanford University
Snow, Catherine	Harvard University

Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

No payments or gifts will be provided to teachers for responding to the survey. It will be possible for teachers to earn Georgia Professional Learning Units through participation in this training, which provides some incentive in itself. In addition, the challenge of making content comprehensible to English Language Learners in order to improve the academic achievement of this subgroup has made access to these strategies attractive. The training is normally expensive for schools that are already experiencing budget cuts, so we expect that the possibility of attending training, free of charge, will serve as an incentive. These assertions have been confirmed by the staff within the Georgia Department of Education, who work with School Improvement directors and coaches.

Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

SERVE and Abt will follow procedures for ensuring and maintaining confidentiality, consistent with the Privacy Act Section 552 of Title 5 of the United States Code, the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, which requires all collection to conform to the requirements of the Privacy Act, as well as the confidentiality standards of the Protection of Pupil Rights Amendment, subsection (c) of this section and sections 444 and 445 of the General Education Provision Act (20 USC 1232h). We will also follow procedures consistent with the Federal common rule or Department final regulations on protection of human research subjects^¹. As detailed below, the system of records that will be established will be in accordance with Privacy Act stipulations, such that no individual will be identifiable by his/her data, as described in greater detail below.

Data to be collected will not be released with individual student, teacher, or school identifiers. Data will be presented in aggregate statistical form only. A statement to this effect is included in a letter accompanying each teacher survey (sent to participants to provide information about completing the survey) and is part of the proposed consent document. All SERVE data collectors (conducting classroom observations) will be knowledgeable about confidentiality procedures and will be prepared to describe them in full detail, if needed, or to answer questions raised by participants. Participants will be assured that all information identifying them or their school or program will be kept confidential in compliance with the legislation (P.L. 103-382)^². Prior to consenting to participation, subjects will be informed that information about the study will ultimately be disseminated in reports and publications but that such information will not contain individual identifiers (such as county/district, school, or individuals’ names) and that information will be presented in a way that will not make individual identification possible.

The following safeguards are routinely employed by SERVE and Abt to carry out confidentiality assurances:

Access to sample selection data is limited to those who have direct responsibility for providing the sample and maintaining sample locating information. At the conclusion of the research, these data are destroyed.
Unique study identification numbers are assigned to subjects (in this case, teachers) that enable the study team to link a subject’s data to his/her identification number rather than his/her name and school name. Identifying information is maintained on separate forms and files, which are linked only by sample identification number.
Access to the file linking sample identification numbers with participants’ identification and contact information is limited to a small number of individuals who have a need to know this information.
Access to the hard copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded material is shredded.
Individual teachers’, schools’, and students’ names do not appear in reports produced for the study, and no reports describe teachers in a way that would allow them to be identified.
No information reported to the Federal Government or submitted for publication will contain information that could be used to identify individual students, teachers, or schools.
Computer data files are protected with passwords, and access is limited to specific users. With especially sensitive data, the data are maintained on removable storage devices that are kept physically secure when not in use.

Teachers within study schools will be required to sign a Consent to Act as a Human Participant form, as approved by the University of North Carolina at Greensboro’s Institutional Review Board^³, which ensures that research involving human subjects follows federal regulations (see Teacher Consent, Appendix G). Consent forms provide information about the scope of the study and particular individual responsibilities of both parties to the agreement, and inform participants of the following:

Risks and benefits of participation in the study;
Voluntary nature of participation in the study, including non-penalty for withdrawal from the study at any time;
Assurances of confidentiality and measures to ensure anonymity; and
Information about parties conducting the study and individuals who can be contacted for more information about the study and participation in the study.

A copy of the Certification of Confidentiality is provided as Appendix I.

Thus, consenting subjects will be affirming that they are aware that their participation in the study is voluntary, that they may discontinue participation at any time, and that they may decline to submit information about themselves for inclusion in the study.

Provide additional justification for any questions of a sensitive nature such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

No questions of a sensitive nature will be asked. Please see Teacher Survey (Appendix C).

Provide estimates of the hour burden of the collection of information. The statement should: *Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices. *If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-1. *Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriated wage rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead this cost should be included in Item 13.

We anticipate that there will be a modest burden on Georgia Department of Education staff, school administrative staff, and teachers, which we outline below.

The REL-SE Senior Policy Research Analyst for Georgia, Dr. Kimberly Anderson, also serves as SIOP Study Manager. She has worked closely with Georgia Department of Education Associate Superintendent for Assessment and Accountability, Dr. Chris Domaleski, and Mrs. Levette Williams, Director of Data Collections and Reporting, to establish a procedure for the request and transfer of data. To confirm and finalize this procedure, an informational meeting was held on site at the Department in March 2007, where staff members from these divisions met with Dr. Anderson; Dr. Carolyn Layzer, Abt Associates, study Co-PI; and Dr. Pamela Finney, REL-SE Task 2 Management Leader. The proposed procedure is as follows: the study team will work through Dr. Anderson to make data requests of the Georgia Department of Education. Dr. Domaleski and Mrs. Williams will then assign particular junior staff to retrieve the data requested, as they determine appropriate. Data will be provided to Dr. Anderson, who will then pass it on to the study team. Dr. Anderson will be the liaison for arranging any additional correspondence between study team members and the Georgia Department of Education staff.

The estimated burden on Georgia Department of Education staff is as follows:

Hours that will be spent by Georgia Department of Education Data Collections Division Operations Analyst/Technology/Management personnel to provide data that the team requests. Annual salaries for these staff range from $34,533 to $60,487 (average = $47,510); this translates to roughly $22.84/hour. Staff will be asked to:

Provide percentages and numbers of ELLs per school as well as number of 4th and 5th grade teachers per school in Georgia for sampling and recruitment purposes. This information would be needed during the recruitment phase of the study. Estimated time required to fulfill requests: 14 hours.

Provide information about the nature of the database containing staff information that we would use to supplement information requested in the Teacher Survey (e.g., teacher job codes). This information would be needed during the instrument design phase of the study. Estimated time required to fulfill requests: 0.25 hours.

Estimated total time required: 14.25 hours

Cost: 14.25 hours x $22.84/hour = $325.47

Hours that will be spent by the Georgia Department of Education Assessment Specialist/Research and Data Analysis personnel to provide the data that we request. Salaries for these staff range from $30,792 to $80,760 (average = $55,776); this translates to roughly $26.82/hour. Staff will be asked to:

Answer questions about the structure of data reported (e.g., vertical scaling of scores, interpretation of scores). This information would be needed once only. Estimated time required to fulfill request: 0.25 hours.

Provide de-identified student assessment data for the period requested. This information would be needed once only. Estimated time required to fulfill request: 6 hours.

Estimated total time required: 6.25 hours

Total Cost: 6.25 hours x $26.82/hour = $167.63

We also estimate that facilitating classroom observations could entail a modest amount of a school administrative staff person’s time, but we have used the average teacher salary ($30.18) as it could also be done by a teacher assigned as the study liaison. Estimated time required to fulfill request is approximately 10 minutes per teacher and 15 minutes per school:

[(10 minutes/teacher x 616 teachers) + (15 minutes/school x 88 schools)] x $30.18

We have added this information to the burden calculation [Table 2, page 16].

[7480 minutes = 124.67 hours x $30.18 = $3762.44]

[7480 minutes∕88 schools = 85 minutes/school = 1.417 hours/school respondent]

Estimated total time required: 124.67 hours

Total Cost: 122.67 hours x $30.18 = $3762.44 (at $42.75 per respondent)

Finally, we estimate that obtaining school contact information will entail some time on the part of the principal or principal’s designee. Most likely, the principal will take approximately 10 minutes at a regular staff meeting to present information about the study (the research team provides informational materials for this purpose), distribute informational materials to teachers who express interest, and invite teachers to sign up to participate in the study. Using average principals’ salary ($93,573/year, or $44.99/hour) as the basis for an estimate of burden, we estimate that the additional burden associated with completing the school contact information sheet will be $5518.56:

Assuming approximately ((15 minutes/school x 88)) x $44.99/hour = $989.78

[22 hours x $44.99/hour = $989.78]

Estimated total time required: 11.25 hours

Total Cost: 22 hours x $44.99/hour = $989.78 (at $11.25 per principal)

Table 2 shows the estimated burden on Georgia Department of Education staff, school administrative staff, and study participants (teachers) for the administration of the professional development survey.

Table 2

Estimated Total Burden

Type of Respondent	Number of Respondents	Data Collection Instrument	Number of Responses	Hours per Respondent	Total Time Burden Hours	Estimated Cost to Each Respondent	Estimated Total Cost
Georgia Department of Education staff	2	Requests for information (see above)	(see above)	7.12	14.25	$162.74	$325.47
Georgia Department of Education staff	2	Requests for student assessment data	(see above)	3.13	6.25	$83.81	$167.63
School Principals	88	School Contact Sheet	1/principal	0.25	22	$11.25	$989.78
School administrative staff	88	School Contact sheet and calendar (not provided)	1/school	1.417	124.67	$42.75	$3762.44
Teachers	616	Teacher Survey	1/teacher	0.25	154	$7.55*	$4,647.72
Total	796				321.17		$9,893.04

*Based on $30.18 average hourly teacher salary. Source: Georgia State Department of Education teacher compensation chart.

All other SIOP research study outcomes are based on observation and student test data and thus do not impose a burden on subjects.

Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14.) The cost estimate should be split into two components: (a) a total capital and start-up cost component (annualized over its expected useful life) and (b) a total operation and maintenance and purchase of services components. The estimates should take into account costs associated with generating, maintaining, and disclosing or providing the information. Include descriptions of methods used to estimate major cost factors including system and technology acquisition, expected useful life or capital equipment, the discount rate(s), and the time period over which costs will be incurred. Capital and start-up costs include, among other items, preparations for collecting information such as purchasing computers and software; monitoring, sampling, drilling, and testing equipment; and record storage facilities. *If cost estimates are expected to vary widely, agencies should present ranges of cost burdens and explain the reasons for the variance. The cost of purchasing or contracting out information collections services should be a part of this cost burden estimate. In developing cost burden estimates, agencies may consult with a sample of respondents (fewer than 10). Utilize the 60-day pre-OMB submission public comment process and use existing economic and regulatory impact analysis associated with the rulemaking containing the information collection, as appropriate. Generally, estimates should not include purchases of equipment or services, or portions thereof, made: (1) prior to October 1, 1995, (2) to achieve regulatory compliance with requirements not associated with the information collection, (3) for reasons other than to provide information or keep records for the government, or (4) as part of customary and usual business or private practices.

There are no direct costs to respondents beyond the time reported above.

Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operation expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

The estimated cost to the federal government for conducting the randomized control trial (RCT) of the Effectiveness of Sheltered Instruction on English Language Learners in Georgia 4^th and 5^th Grade Classrooms (SIOP)—including designing the study, recruiting schools, implementing random assignment, collecting school/teacher/student data, processing and analyzing the data, and preparing the reports and summarizing the results—is estimated to be $3.3 million over 5 years which includes an estimated $900,000 for SIOP teacher training, coaching, and follow up. The SIOP training is to be $650,000 in Year 3 of our contract (begins March 15, 2008, and estimated to be $250,000 in Year 4 of our contract. The remaining balance after training is the cost of REL-SE and Abt work spread over the 5 years

The cost of the data collection activities associated with this project is projected to be $464,684. The study period is from March 2006 to March 2011, with data collection taking place from Spring 2008 to Summer 2009.

Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1.

This is a new study.

For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

Our discussion of analysis, tabulation, and publication of results focuses on the four main research questions outlined in the introduction.

Equivalence of Groups at Baseline

Our analysis rests on the assumption that random assignment will be successful, resulting in two groups roughly comparable on characteristics of interest prior to the treatment. These characteristics are determined through a combination of existing records from the Georgia state databases and our Teacher Survey. For any baseline characteristics that are statistically significantly different, we will include them as covariates in our impact models to control for these baseline differences. We will also include selected covariates in our models if we find that they increase the precision of our impact estimates (see below).

Covariates from the teacher survey will be constructed from responses to multiple questions. We will run factor analysis to ensure that the responses composite well. The constructs that we intend to include are: Breadth and Depth of ESOL Professional Development (from Teacher Survey questions 5, 7, and 10), Level of Support Within School for Teaching ELLs (from Teacher Survey questions 6 and 9), and Materials to Support Teaching ELLs (from Teacher Survey question 8).

Table 3

Descriptive Statistics at Baseline

	SIOP		Control		Overall
	Mean	(s.d.)	Mean	(s.d.)	Mean	(s.d.)
School Characteristics at Baseline
Total enrollment
Total enrollment of 4^th and 5^th grade students
% English Language Learners in school
Number of 4^th and 5^th grade classrooms/school
% English Language Learners in 4^th and 5^th grades, combined
Classroom Characteristics at Baseline
Class size (enrolled)
Class size (observed, at follow-up)
Student Characteristics at Baseline
4^th and 5^th Grade ACCESS scores, Composite
4^th and 5^th Grade CRCT scores, Composite
Teacher Characteristics at Baseline
Highest Level of Education Attained
Percentage of teachers with Bachelor of Arts/Bachelor of Science
Percentage of teachers with Bachelor degree plus additional courses
Percentage of teachers with Master of Arts/Master of Science
Percentage of teachers with Master degree plus additional courses
Percentage of teachers with Doctorate
Percentage of teachers with Other
Certification
Percentage of teachers with Teaching Certificate
Percentage of teachers with certification in Elementary Education
Percentage of teachers with certification in Early Childhood Education
Percentage of teachers with certification in Reading
Percentage of teachers with certification in Special Education
Percentage of teachers with certification in ESOL or bilingual education
Percentage of teachers with certification in Other area
Endorsements to Certification
Percentage of teachers with endorsement in Elementary Education
Percentage of teachers with endorsement in Early Childhood Education
Percentage of teachers with endorsement in Reading
Percentage of teachers with endorsement in Special Education
Percentage of teachers with endorsement in ESOL or bilingual education
Teacher Professional Development and ESOL Support Composites
Breadth and Depth of ESOL-focused Professional Development (Composite)
Level of Support Within School for Teaching ELLs (Composite)
Materials to Support Teaching ELLs (Composite)

Sources: Georgia Certified/Classified Personnel Index (GaDOE) and Teacher Survey

*Significantly different from zero at the .10 level, two-tailed test

**Significantly different from zero at the .05 level, two-tailed test

***Significantly different from zero at the .01 level, two-tailed test

Impact of SIOP on Teachers’ Instructional Practices and Student Outcomes

The impact of participating in SIOP professional development on teachers and their students will be estimated by comparing the average observed outcomes of the SIOP group with the control group. We will calculate these average differences using hierarchical linear modeling (Raudenbush & Bryk, 2002), in order to appropriately account for the clustered nature of the data (e.g., teachers clustered within schools). This method yields a direct estimate of the average difference between the two groups on any given outcome of interest, as well as an estimate of its standard error, adjusted for clustering. If this estimate is positive and statistically significant, we can conclude that participation in SIOP professional development had a positive impact on outcomes for teachers and their students (see below for model specifications).

A central assumption of RCT designs is that randomization produces two groups (whether schools, teachers, or students) that are equivalent at the outset of a study, on average, on all measured and unmeasured characteristics potentially related to outcomes of interest, thereby allowing researchers to control for all pre-existing characteristics of teachers and students (e.g., teacher knowledge or skill) by design. Therefore, we can assume that the estimate of the impact of SIOP is unbiased with respect to pre-existing characteristics and that it is unnecessary to control for pre-existing characteristics of teachers in order to assure an accurate estimate of SIOP impact. We will, however, include covariates to improve the precision of our estimate, including baseline measures (e.g., school-level average ELL English proficiency) as well as background characteristics of schools, teachers, and students in our analyses, thereby improving our ability to detect significant effects.

Teacher-level impacts. To determine if SIOP has an impact on teachers’ instructional practices (research question 1), the analysis will be a two-level hierarchical linear model, where teachers are at Level 1, and schools are at Level 2. To the extent possible, we will also examine the relationships of SIOP effects and various teacher characteristics such as: years of teaching experience and professional qualification (certification). An example HLM equation follows.

Level-1: Teacher (for teacher i in school j)

Y_ij = β _0j + β_1j(graduate degree)_ij + β_2j(certification)_ij + β_3j(years teaching)_ij + r_ij

Where (graduate degree)_ij = 1 for teachers who have masters degrees and above; = 0 for teachers who have bachelors degrees;

(certification)_ij = 1 for teachers who have state certifications in ESL; = 0 for teachers who do not have ESL certifications;

(years teaching)_ij is a continuous variable of the number of years teachers have been teaching in Georgia;

and r_ij is the random effect at level-1 assumed to be normally distributed, with a mean of 0 and variance of ².

Level-2: School

β_0j = γ₀₀ + γ₀₁(SIOP)_j + γ₀₂(AYP)_j + u_0j,

where (SIOP)_j = 1 for schools assigned to the treatment group; = 0 for schools assigned to the control group;

(AYP)_j is a continuous variable of the school’s annual yearly progress in the prior year as a pre-test covariate to improve statistical precision;

and u_0j is the random effect at level-2 assumed to be normally distributed, with a mean of 0 and variance of τ.

β_1j = γ₁₀

β_2j = γ₂₀

β_3j = γ₃₀

In this equation, the main effect of SIOP on the teacher outcome Y_ij is represented by γ₀₁, taking into account standard demographic and performance characteristics of schools and teachers.

Student-level impacts. For analyses at the student level, outcomes will be measured across two grade levels. All analyses will be done by combining the grades, in order to have a large enough sample (and therefore enough power) to detect policy-relevant effects at the student level. For student outcomes, ACCESS scores are vertically scaled across grades 4 and 5 and so pose no problems for combined analyses. Because CRCT scores are not vertically scaled, we will transform grade-level scores to standardized scores before conducting analyses. In our HLM models for each outcome, we will have random intercepts and fixed-effects dummy variables for treatment and grade. We will fit one model with a treatment-by-grade interaction and if it is significant, we will conclude that the treatment effect differs by grade. If the interaction is not significant, we will drop the interaction term and report the combined treatment effect on 4^th and 5^th graders together.

To answer research question 4, we will run separate impact analysis on the sample of non-ELL students in treatment and control classrooms, using their CRCT scores as the outcomes and including prior school-level average CRCT scores as covariates. We will then examine and describe the patterns of estimated impacts of SIOP for each sample (ELL and non-ELL) to determine if SIOP had positive impacts on both groups.

For estimating effects on students, we will be modeling a 3-level HLM, where students are at Level-1, teachers are at Level-2, and schools are at Level-3. To increase precision, we will include student characteristics in analyses.^⁴ An example HLM equation is as follows:

Level-1: Student (for student i in classroom j in school k)

Y_ijk = π_0jk + π_1jk(Parent Education)_ijk + π_2jk (Female)_ijk + π_3jk (Black)_ijk +

π_4jk (Hispanic)_ijk + π_5jk (Other Race)_ijk + r_ijk

Level-2: Classroom

π_0jk = β_00k + β_01k (graduate degree)_jk + β_02k (certification)_jk + β_03k (years teaching)_jk ₊ u_00jk

π_1jk= β_10k

π_2jk = β_20k

π_3jk = β_30k

π_4jk = β_40k

Below are tables indicating possible tabulation of data on teacher and student outcomes (Please see Tables 4 and 5).

Table 4

Impact of SIOP Teacher Training on Classroom Behaviors

Outcome: *SPC Scores (group means)**	SIOP Schools	Control Schools	Impact	Standard Error of the Impact	Effect Size of the Impact	Statistical Significance of Impact
Joint Productive Activity
Language and Literacy Development
Contextualization
Challenging Activities
Instructional Conversation
Composite Score

Source: Classroom Observations using Standards Performance Continuum observation protocol.

Table 5

Impact of SIOP Teacher Training on Student Outcomes

Outcome: Student Test Scores (group means)	SIOP Schools	Control Schools	Impact	Standard Error of the Impact	Effect Size of the Impact	Statistical Significance of Impact
English Language Learners Only
ACCESS for ELLs
Listening
Speaking
Reading
Writing
Composite Score
CRCT
Reading
English/Language Arts
Mathematics
Science
Social Studies
Non-English Language Learners
CRCT
Reading
English/Language Arts
Mathematics
Science
Social Studies

Source: Student Achievement data from GaDOE database. Assessing Comprehension and Communication in English State to State for English Language Learners (ACCESS for ELLs) is the measure used by Georgia to assess English language proficiency in students identified as Limited English Proficient (LEP). Criterion Reference Competency Test (CRCT) is the Georgia State-mandated annual measure of student achievement of skills and knowledge based on Georgia Performance Standards (GPS) and the Quality Core Curriculum (QCC); these are assessments given to the general school population, including subgroups such as ELL.

If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The OMB approval expiration date will appear in all materials distributed to participants in the study.

Explain each exception to the certification statement identified in Item 19, “certification for Paperwork Reduction Act Submissions,” of OMB Form 83-1.

There are no exceptions to the certification statement.

References

Doherty, R.W., Hilberg, R.S., Pinal, A., & Tharp, R.G. (2003). Five standards and student achievement. NABE Journal of Research and Practice, 1, 1-24.

Doherty, R.W., Hilberg, R.S., Epaloose, G., & Tharp, R.G. (2002). Standards performance continuum: Development and validation of a measure of effective pedagogy. The Journal of Educational Research, 96, p. 78-89.

Echevarria, J. (2006). Further details on national and district training and implementation information, personal communication with research team November, 2006.

Echevarria, J. Vogt, M. E., & Short, D. (2004). Making content comprehensible for English language learners: The SIOP model. Boston: NY: Allyn & Bacon.

Echevarria, J., & Short, D. (2006). The impact of the SIOP model on middle school science and language learning. (Year 1 Annual Report: 8/1/05-7/31/06). Center for Applied Linguistics, Washington DC and California State University, Long Beach: National Research and Development Center for English Language Learners.

Echevarria, J., Short, D., & Powers, K. (2006). School reform and standards-based education: A model for English-language learners. Journal of Educational Research, 99(4), 195-210.

Echevarria, J., Vogt M.E., & Short, D. (2000). Making content comprehensible for English language learners: The SIOP model. Needham Heights, MA: Allyn & Bacon.

Herrera, S. G., & Murry K. G. (2005). Mastering ESL and bilingual methods: Differentiated instruction for culturally and linguistically diverse (CLD) students. NY: Allyn & Bacon.

National Clearinghouse for English Language Acquisition. (2004). Immigration and America's schools. Washington, DC: Author. Available at http://www.ncela.gwu.edu/resabout/ immigration/index.html

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods, 2^nd ed. Thousand Oaks, CA: Sage Publications.

Short, D., & Echevarria, J. (1999). The sheltered instruction observation protocol: A tool for teacher-researcher collaboration of professional development (Educational Practice No. 3). Santa Cruz, CA and Washington, DC: Center for Research on Education, Diversity & Excellence.

Snow, M. A., Met, M., & Genesee, F. (1989). A conceptual framework for the integration of language and content in second/foreign language instruction. TESOL Quarterly, 23, 201-217.

Web-based sources referenced:

What Works Clearinghouse. (2007). Topic Report English Language Learners. Retrieved March 6, 2008 from http://ies.ed.gov/ncee/WWC/reports/topic.aspx?tid=10.

What Works for English Language Learners Retrieved March 6, 2008 from http://www.ncela.gwu.edu/spotlight/4_whatworks.html.

National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs. (2008). What Works For English Language Learners NCELA Bibliographic Database. Retrieved March 6, 2008 from http://www.ncela.gwu.edu/rcd/search/.

National Clearinghouse for English Language Acquisition and Language Instruction Educational Programs. (2008). Research on the Effective Education of English Language Learners (REEELL) Database. Retrieved March 6, 2008 from http://www.ncela.gwu.edu/databases/reeell/.

1 REL-SE follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). REL-SE will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution but not to any individually identifiable information. No individually identifiable information will be maintained by the study team. All institution-level identifiable information will be kept in secured locations and identifiers will be destroyed as soon as they are no longer required. REL-SE obtains signed NCEE Affidavits of Nondisclosure from all employees, subcontractors, and consultants that may have access to this data and submits them to our NCEE COR.

2 The legislation provides that no person may use “individually identifiable information furnished under this title for any purpose other than a statistical purpose, make any publication whereby the data furnished by any particular person under this title can be identified, or permit anyone other than the individuals authorized by the Commissioner to examine the individual reports.”

3 IRB #067245; approved, March 22, 2007; deemed exempt as minimal risk protocol under section B1 of 45 CFR 46.101

4 For research question 4, a separate impact model will be run on the non-ELL student sample.

File Type	application/msword
File Title	A Justification
Author	Lori Sterling
Last Modified By	DoED User
File Modified	2008-03-17
File Created	2008-03-17