Att_QTEL OMB Supporting Statement Part A_4.20.07

The Quality Teaching of English Learners (QTEL) program, is a model of professional development for teachers of secondary English Language Learners (ELL), developed by WestEd’s Teacher Professional Development Program with the funding from Regional Educational Laboratory (West) (REL West). The QTEL program of teacher professional development is based on the premise that improving the education of secondary ELLs requires teachers to:

Develop deep knowledge of what it means for ELLs to participate in academic activity;
Understand, take part in, and reflect on research-based practices that support students’ development of academic literacy in English as well as deep content knowledge in academic subjects; and
Receive the necessary support to change their classroom practice.

WestEd’s QTEL staff work with middle school teachers in sustained collaborative relationships to introduce them into communities of educators with a shared vision of quality education and how to enact it.

The primary purpose of the study is to measure the impact of the QTEL professional development for teachers on student achievement. A secondary study goal is to examine the extent to which schools and teachers receive the QTEL training and coaching as intended and the extent to which the QTEL model is implemented as intended. This study will also serve to inform future program improvement and replication of the QTEL program. Moreover, it will provide documentation of the details of QTEL implementation for use by other institutions and entities. WestEd and its partner, Berkeley Policy Associates (BPA), are conducting this study for the Institute of Education Sciences (IES) of the US Department of Education.

At the request of educators in San Diego County, approximately 50 middle schools, 600 teachers, and about 16,000 sixth, seventh, and eighth grade students will be involved in a three-year longitudinal study of the impact of QTEL on student achievement patterns. The study will also assess program impacts on teacher and classroom outcomes. Eligible middle schools in participating districts, those schools with at least 10 percent of the classified as Limited English Proficient (LEP) or Redesignated as Fluent English Proficient (RFEP), will be grouped together by their location, and by their student, teacher, and school characteristics. The 50 middle schools selected and their participating teachers will then be randomly assigned within the resulting strata to treatment and control conditions. Consistent with schools’ requests for services to prepare teachers for the 2007-08 academic year, the professional development activities will begin in Summer 2007. Data collection activities to support the study will begin only upon receipt of OMB approval.

Supporting Statement for the Paperwork Reduction Act

PART A: JUSTIFICATION

Circumstances that Make Data Collection Necessary

The study addresses the challenge of providing high quality and effective instruction to English language learners (ELLs) in secondary school settings. The need for addressing this challenge is great, because the number of ELLs in schools across the U.S. is growing rapidly. Furthermore, most of these students do not perform well on academic assessments and do not have good long-term educational outcomes. Special English as a Second Language (ESL) classes have often been found to be ineffective. Teachers of English Language Arts (ELA) often fail to challenge and properly support their ELL students, resulting in very poor achievement by these students on reading and writing assessments. In turn, this lack of performance in reading and writing compromises these students’ ability to learn in other disciplines.

This research will be conducted with the rigor needed to apply for inclusion in the U.S. Department of Education’s portfolio of successful educational practices, housed at the What Works Clearinghouse (http://www.whatworks.ed.gov/).

The What Works Clearinghouse (WWC) was established in 2002 to provide educators, policymakers, and researchers with a central place they can go to for high-quality scientific research about successful programs and practices. Few resources have been available to compare the quality of competing research findings and promotional claims. The clearinghouse has developed and set rigorous standards for examining and synthesizing research on the effectiveness of educational programs, products, practices, and policies. ^¹

In order to meet the highest evidence standards of the WWC, an experimental design using random assignment is required. None of the studies reviewed and listed on the WWC targets ELLs at the middle school level. This study attempts to fill that gap.

The QTEL program addresses the needs of ESL and ELA teachers by providing them with challenging and highly intensive professional development that extends over multiple years. This professional development is an integrated package of summer institutes, in-person one-on-one coaching sessions during the school year, and collaborative lesson planning activities that are open to all teachers in the school. QTEL was created in 1999 by WestEd's Teacher Professional Development program under the direction of Aída Walqui. QTEL improves the capacity of teachers to support the linguistic, conceptual, and academic development of adolescent English learners, both immigrant and U.S.-born. QTEL is grounded in a body of research that challenges the traditional thinking that teaching and learning for ELLs must be simplified. It provides an academic framework that offers intellectual challenges and supports that strengthen teacher knowledge and their ability to apply that knowledge in classes, and in turn improve student achievement.

Purposes and Uses of the Data

The intervention’s effectiveness will be assessed in approximately 50 middle schools in San Diego County, California. Twenty-five of these schools will be randomly assigned to a treatment group, which will be eligible to participate in QTEL and twenty-five will be assigned to a control group, which will be excluded from QTEL for three years. The schools will be randomized within their school district, so that each district has at least one treatment school and one control school.^² Such stratification by district also helps to minimize random variation in background characteristics between students and teachers in the treatment and control groups. All ESL and ELA teachers in treatment schools will participate in the program and all students in grades 6, 7, and 8 will be part of the study. It is expected that the overall study sample will have approximately 600 teachers and approximately 50,000 students, of whom an estimated 12,500 will be ELLs. These teachers and students will roughly be equally distributed among grades 6, 7, and 8. Even though our impact analyses will focus only on one grade at a time, the study is adequately powered, with minimum detectable effect sizes of 0.19-0.22 for all students, 0.20-0.23 for ELL students, and 0.29-0.42 for all teachers.

Based on QTEL theoretical model as shown in Exhibit 1, the study will measure outcomes in three different domains: (1) teacher pedagogical knowledge and attitudes, (2) teacher practice, classroom environment, and student engagement and (3) student achievement. The Teacher Test of Knowledge is developed to measure teachers’ understanding of using appropriate pedagogy for ELLs. The Teacher Survey is used to measure teachers’ attitudinal changes (over time) on teaching ELLs and their experience with the QTEL program (for treatment teachers only). Some teacher background information will be also collected through this survey. The teacher practice, classroom environment, and student engagement measures will be collected with in-class observations and videotaped interaction data. The latter data will be coded outside the classroom by expert coders who are blind to the random assignment status of the schools from which the data are collected. Student achievement outcomes include the California Standards Test (we plan to focus on all content areas, but especially on ELA), the California English Language Development Test (CELDT), measures of attendance, student grades (GPA), and grade promotion. These student archived data will be collected through the participating districts/schools. In addition, there will be data collection through observations and focus groups that are related to teachers’ professional development in QTEL program. Exhibit 2 lists the instruments and the proposed dates for new data collection activities. These instruments are included in Appendix A.

The teacher data will be used as follows:

Provide an accurate account of the QTEL professional development program, as planned and as implemented
Determine if the QTEL program model was implemented as planned at program sites
Determine the degree to which individual program activities conform to the original model
Identify to whom the program was delivered and if this is the same as the intended target audience
Identify changes made to the program model during program implementation and document the reasons why these changes were made
Determine teacher satisfaction and changes in attitude that result from program participation, including key differences among participants by subgroup in these areas
Describe the treatment contrast produced by the intervention (between teachers in the treatment and control groups).

Exhibit 1: QTEL Theoretical Model

Exhibit 2: Data Collection Instruments and Proposed Implementation Dates

Data Collection Instruments	Proposed Implementation Dates
QTEL Teacher Survey^³	Fall 2007, Spring 2008-2010
QTEL Teacher Test of Knowledge	Fall 2007, Spring 2008-2010
QTEL Professional Development Observation Protocol	Fall 2007-2009
QTEL Teacher Collaboration Protocol	Fall 2007-2009
QTEL Coach Focus Group Discussion Guide	Spring 2008-2010
QTEL Teacher Focus Group Discussion Guide	Spring 2008-2010
QTEL Classroom Observation Protocol^⁴	Spring 2008-2010
QTEL Videotaped Observation Protocol	Spring 2008-2010

The data collection of teacher practice and student engagement, through classroom observation, will be used to capture program impacts on the nature of teacher-student interaction, academic discourse, and student collaboration the classrooms. This data will be used to answer the following questions:

How does participation in QTEL change teachers’ classroom practice, their interaction with students, and the structure of their lesson activities?
How does QTEL change the classroom engagement and participation in group-activities by ELLs and other students?

Student data collected will be used to study the program impact on student achievement related outcomes.

Most of these data collected will be analyzed using Hierarchical Linear Modeling (HLM) software or the PROC MIXED procedure in SAS to account for the hierarchical nature of the data (detailed analytical procedures are discussed in Section 16 of this document). Qualitative data such as in-depth observations of program implementation and the classroom environment will be analyzed using special software designed to identify and systematically describe pertinent aspects of the implementation of the QTEL program to inform subsequent replication and refinement. The study will produce an interim report in December 2008 and a final report in December 2010.

Use of Improved Technology to Reduce Burden

Wherever possible, the study team will use current information technologies to maximize the efficiency and completeness of the information needed for the study and to minimize the burden placed on respondents. During the data collection period, a toll-free telephone number and e-mail address will be available for respondents to contact the study team with any questions. This information is included on the survey forms. A Microsoft Access database will be used to monitor survey response and generate reports indicating any missing data. The research team will employ a web-based application, placed on a secure server for the teacher surveys and tests. Teachers will be provided with a login name and password to protect their data. The web-based application will simplify completing the surveys, thus reducing the burden on respondents.

Efforts to Identify and Reduce Duplication

There are no similar on-going data collection and no data will be collected that is available from other sources. There was a field study in the spring of 2005 of the QTEL professional development model which was conducted in 39 New York City middle schools involving 200 Language Arts and English as a Second Language teachers and their students. Although this previous data was similar, it was not collected using a randomized sample design and the results of that study were, therefore, not generalizable.

In addition, each instrument used in this study will be carefully reviewed (and some will be piloted using a small sample) to make sure that we only collect the most necessary information needed for this study. The secondary information such as student archived data will be accessed and collected through the electronic database at the school (or district) level.

Efforts to Minimize Burden on Small Businesses

The research team will collect data from few small entities, as most of the data sources will be from teachers and students. The few small entities are likely to be associated with the external technical assistants and consultants who may assist with data key-in and help with data management. Only minimal information will be needed from these small entities, and so no significant impact on small entities is expected.

Schools (or districts) will transmit electronic files of student achievement data to the researchers. Only existing and necessary data (e.g., standardized test scores) will be requested from these entities, thereby reducing the burden to schools/districts.

Consequences of Not Collecting the Information

This research effort is aligned with the mission of the Department of Education’s Institute of Education Sciences (IES), which is to conduct rigorous research (based on a randomized controlled trial) that supports the solution of educational problems in the United States. Thus, if this data are not collected, the U.S. Department of Education, Congress, and other stakeholders will not have detailed information about the effects of the QTEL professional development program on improving service to ELLs. Moreover, if these data were collected less frequently, then there would be no sufficient documentation of how QTEL was implemented in the schools, or the impact of the intervention on teachers’ instructional practices and student engagement.

Special Circumstances

None of the special circumstances, as listed in 5 Code of Federal Regulations Section 1320.5(d)(2), apply to this study.

Federal Register Comments and Outside Consultants

A notice about the study will be published in the Federal Register when the final OMB package is submitted.

The data collection instruments were developed at Berkeley Policy Associates by a team under the direction of Dr. Raquel Sanchez and Dr. Hans Bos. Input was obtained from WestEd staff members, Dr. Neal Finkelstein, Dr. Aida Walqui, and from the study’s technical working group (TWG):

• Professor Jamal Abedi, CRESST, University of California, Davis

• Dr. Lloyd Bond, Carnegie Foundation for the Advancement of Teaching

• Professor Geoffrey Borman, University of Wisconsin

• Professor Brian Flay, Oregon State University

• Professor Tom Good, University of Arizona

• Dr. Corinne Herlihy, Manpower Demonstration Research Corporation (MDRC)

• Dr. Joan Herman, CRESST, University of California, Los Angeles

• Professor Heather Hill, University of Michigan

• Dr. Roger Levine, American Institutes for Research (AIR)

• Dr. Jason Snipes, Council of the Great City Schools

The TWG provide consultation on the design, implementation and analysis of this study. They are expected to consult with REL West for five days per year through a combination of in-person and teleconferenced meetings. An honorarium of $1,200 will be paid to each TWG member. This amount is the required daily rate required to retain TWG members. These are senior faculty members who are distinguished in their fields and extremely knowledgeable about the methodological and statistical requirements of randomized controlled trials. The rate and the contracts have been approved by IES.

Payments to Respondents

We will compensate teacher participants for the required time to complete the tasks in order to ensure a high response rate. “Funding agencies both private and governmental, have recognized the need to subsidize basic research, researchers feel that it is conventional and a expected utility to boost response rates” (Lazear, 1997). In addition, experience has shown that, “focus groups are unique from other data-gathering processes in terms of the investment that must be made by the individual. It is therefore no surprise that a tradition has been established to provide an incentive for participation. From a practical aspect, it would be next to impossible to conduct focus groups without incentives in some situations” (Krueger 2000).

In this study, participation in the teacher surveys and tests are expected to take 0.5 and 1 hour each per person to complete. Participation in the focus group is estimated to take approximately 1.5 hours per person. Accordingly, we propose to award each participant a $20 for the teacher survey and a $30 gift card for each teacher test completed. A $45 gift card will be awarded to each teacher attending a focus group. For those teachers who decide to participate in the videotaped observations there will be an additional $50 gift card provided for assisting with the videotaping over a 3-day/5-day period. In general, these rates are based on the estimated average teacher hourly compensation across the school districts in San Diego County. The compensation is a sign of appreciation of the respondents’ time, commensurate with the value of that time. We believe it is essential to the success of this data collection effort to provide a sufficient rate for completing tests and surveys, and for participating in focus groups and/or videotaped observations. All the curriculum materials will be provided to the schools/teachers with no cost.

Assurance of Confidentiality to Respondents

Any information that is obtained in connection with this study and that can be identified with any teachers, schools, or students will remain confidential and will be disclosed only with the participants written permission or as required by law. Specifically, the Education Sciences Reform Act of 2002, Title I Part E, Section 183 requires “All collection, maintenance, use and wide disemmination of data by the Institute of Education Sciences “to conform with the requirements of section 522 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 USC 1232g, 1232h). These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protections of Pupil Rights Amendment.

Accordongly, an explicit statement regarding confidentiality will be communicated to any and all participants. The privacy of the information collected will be protected by keeping all video and paper data in locked files. All computer records will be kept in password-protected, secure storage under the direct control of the researcher team. Results will be analyzed and reported only in averages for groups of students and groups of school; no individuals or individual schools will be identified by name. Participation in the program and in the research study is completely voluntary. All original data including video recordings will be destroyed at the conclusion of the study.

Volunteers may withdraw at any time and without consequences of any kind. Consent will be obtained from teachers who will participate in the study. Separate consent will be obtained for those teachers whose classrooms will be videotaped. Similarly, separate consent will be obtained for those parents whose child’s classrooms will be videotaped. A copy of these consent forms (along with the recruitment flyers – one for general purpose and another for videotaped observation) are included in Appendix B.

Justification for Questions of a Sensitive Nature

No questions of a sensitive nature will be included in the teacher survey/test or in the focus groups.

Estimate of Information Collection Burden

Exhibit 3 summarizes the estimates of respondent burden for data collection activities administered during this research. The following instruments (also listed in Exhibit 2) will be used for various data collection activities:

Inst1a: QTEL Teacher Survey (for treatment teachers) – 30 minutes per respondent

Inst1b: QTEL Teacher Survey (for control teachers) – 20 minutes per respondent

Inst2: QTEL Teacher Test of Knowledge – 1 hour per respondent
Inst3: QTEL Professional Development Observation Protocol

Inst4: QTEL Teacher Collaboration Observation Protocol
Inst5: QTEL Coach Focus Group Discussion Guide – 1.5 hours per coach
Inst6: QTEL Teacher Focus Group Discussion Guide – 1.5 hours per teacher

Inst7: QTEL Classroom Observation Protocol (using SIOP)

Inst8: QTEL Videotaped Observation Protocol

Exhibit 3: Estimates Response Burden

Task	Number of Respondents	Frequency of Data Collection	Total Number of Responses	Time per Response (in hours)	Total Hours	Hourly Rate	Total Cost Burden
Sampling /Gaining Cooperation (from school principals)	50	1	50	1	50	$36	$1,800

Teacher Data Collection:
(1) using Inst1a (for treatment teachers)	300 (25 schools*12 teachers)	4	1,200	0.5	600	$30	$18,000
(2) using Inst1b (for control teachers)	300 (25 schools*12 teachers)	4	1,200	0.33	396	$30	$11,880
(3) using Inst2	600	4	2,400	1	2,400	$30	$72,000

Teacher Professional Development Observation:
(1) “Building the Base” training using Inst3				0	0	NA	0
(2) On-site observation of participating teacher meetings using Inst3				0	0	NA	0
(3) On-site observation of participating teacher collaboration using Inst4				0	0	NA	0
(4) On-site observation of follow-up sessions using Inst3				0	0	NA	0

Focus Group:
(1) For coaches using Inst5^⁵				0	0	NA	0
(2) For teachers using Inst6	60 (5 schools*12 teachers)	3	180	1.5	270	$30	$8,100

Teacher Classroom Observation:
(1) using Inst7				0	0	NA	0
(2) using Inst8	15 (10 treatment classrooms and 5 control classrooms)	3	45	0.33^⁶	15	$30	$450

Student Archived Data Collection (from districts)	15	3	45	3	135	$50	$6,750

TOTAL			5,120		3,866		$118,980

In order to calculate annual numbers of respondents and annual respondent burden on Form 83-I, we divided the totals in Exhibit 3 by three, assuming three years of data collection activity for the Study. This calculation results in annual responses of 1707 and annual hours burden of 1,289. The annual unduplicated number of respondent is estimated to be 632^⁷.

Estimate of Total Annual Cost Burden

There are no direct start-up costs to respondents other than their time to participate in the study, as estimated above. Estimations of the value of participation time for each task, and for the study as a whole, are presented in Exhibit 3 above.

Estimate of Annual Cost to the Federal Government

The total cost for the study is $4,849,964 over five years. The average yearly cost is $969,993.

Change in Annual Reporting Burden

The change of total 3,866 burden hours (1,289 annual burden hours) reflects new data collection.

Data Collection Schedule

The study’s data collection time-table is as follows:

Exhibit 4: Data Collection Schedule

September-December 2007: First round of teacher knowledge/attitude assessment

March-April 2008: Second round of teacher knowledge/attitude assessment

Classroom observation in both treatment and control classrooms

Videotaping in 10 randomly selected treatment group classrooms and 5 randomly selected control group classrooms

June- July 2008: Collection of district level data on student achievement

June-November 2008: Analysis of first year of classroom observation and student achievement data.

December 2008: Short interim report on implementation and early impacts

March-April 2009: Third round of teacher knowledge/attitude assessment

Classroom observation in both treatment and control group classrooms

Videotaping in 10 randomly selected treatment group classrooms and 5 randomly selected control group classrooms

June- July 2009: Collection of district level data on student achievement

June-November 2009: Analysis of second year of classroom observation and student achievement data

March-April 2010: Fourth round of teacher knowledge/attitude assessment

Classroom observation in both treatment and control group classrooms

Videotaping in 10 randomly selected treatment group classrooms and 5 randomly selected control group classrooms

June- July 2010: Collection of district level data on student achievement

May-September 2010: Analysis of third year of classroom observation and student achievement data

October 2010: Draft final impact report

December 2010: Final impact report, IES approval

January 2011: Report dissemination

Analysis of data collected will continue throughout the project, mostly during the second half of each calendar year. We plan to produce two reports, a short interim report in December 2008 and a comprehensive final report in December 2010. Most of the process and impact analyses should be completed by the spring of 2010, but the results will be updated with observation and student achievement data later that year. In preparation for the production of the final report in late 2010, we expect to convene a number of major report outline and review meetings in the early spring of 2010, which will include the TWG members and other relevant experts. All of this will serve to make the final production of the final report as streamlined as possible.

The interim and final reports will target a sophisticated policy audience and will be disseminated via the official IES channels, including the national Regional Educational Laboratory website and the What Works Clearinghouse. However, in addition to the full technical documents, we plan to prepare policy briefs and presentations that will be accessible to a wider audience, including school board members, parents, teachers, and district administrators. Some of these documents will be targeted specifically at the San Diego County education establishment, including consumers in the districts where the study is being implemented.

Analytical Procedures

The impact analysis will describe post-random assignment differences between the treatment and control groups in six distinct areas: (1) teachers’ receipt of professional development and coaching, and participation in learning communities or similar activities, (2) teachers’ understanding of how to engage ELL students and improve their ability to learn in a traditional classroom environment, (3) teachers’ attitudes about ELL students and how best to teach them, (4) classroom interaction, student engagement, and activity structure, (5) student achievement on state-referenced tests, and (6) other student outcomes. The first of these is to describe the treatment contrast, answering the question of how much the QTEL program adds to the existing professional development infrastructure available to the teachers. The second describes how the QTEL professional development and coaching changes the way teachers regard the instructional needs of ELL students and their own role in meeting these needs. This is the most immediate outcome of the QTEL program. Next, the impact analysis documents whether QTEL changes how teachers regard ELL students, how they plan to help them in the classroom, and how confident they are in their own ability to be successful with these students. We then measure impacts on how teachers interact with students and the quality of the classroom environment from the perspective of ELL students. The fifth step documents program impacts on student achievement outcomes. Finally, the sixth step describes whether and how students benefit from QTEL in ways that are not captured in test scores, for example by having lower absenteeism or being more likely to be promoted.

A major advantage of using random assignment to evaluate the QTEL program is that such a design does not rely on highly complex (and often controversial) data analytical techniques to produce valid impact estimates. The study design assures that post-random assignment differences between outcomes for teachers and students in the treatment and control groups are unbiased estimates of the program effects. However, there are significant benefits to using baseline covariates in the impact analysis, mostly in terms of increased statistical power. Thus, we propose to use multiple regression models to analyze the outcome data for this study. These models will control for student, school, and teacher background characteristics. To maximize the useful variance, these models will be hierarchical in nature (i.e., they explicitly acknowledge the nested nature of the data) and will be estimated with HLM software or with PROC MIXED in SAS.

For our covariates at the student level, we plan to use mostly school-level or grade-level covariates, because individual-level student background data is not always available and may lead to observations being dropped from the analysis due to missing data. For the purposes of increasing our statistical power, it is most important to control for student outcomes at the aggregate (school) level. School-level covariates minimize random school-to-school variation in background characteristics between the treatment and control group schools. To create these aggregate covariates we will use either academic outcomes for prior cohorts of students or we will use prior academic outcomes for the present cohort, or both, depending on data. At the teacher level, we plan to control for teacher education level and experience, as well as basic demographic characteristics, such as age, gender, and ethnicity. In a somewhat simplified format, a student-level impact regression model would look as follows:

(1)

In this model, Y_ij represents outcome Y (a CST reading comprehension score, for example), measured for student i in school j. T_j is the treatment variable, which is measured at the school level and has a value of 1 for treatment schools and 0 for control schools. ₁ is the program effect associated with this variable. Z_i and S_j vectors of control variables for students i and schools j, respectively. D_k is a series of dummy variables denoting the participating school districts. Including these dummies is necessary to account for the stratification by district. Each of these vectors is accompanied by a series of regression coefficients (_z, _x, _s, and _d). Separate error terms for schools and students are represented by _j, and _i,respectively. Although Equation 1 as written appears to represent a fixed effects regression model, we do not plan to estimate it that way, because doing so would not be appropriate given the hierarchical nature of the data. Instead, the models will be estimated as a series of two nested hierarchical models (at the school and student level), in which the unexplained error at one level becomes the outcome to be explained at the next level. To control for clustering at the school level, the student-level model will include school fixed effects. After estimating these regression models we will use the estimated coefficients to calculate regression-adjusted mean outcomes for treatment and control schools. The regression-adjusted means will be presented in tables and figures so that readers do not have to interpret regression coefficients to learn about the impacts of the program. Note that all of these analytical techniques fully preserve the experimental nature of the impact analysis.

Subgroup Analysis

We will conduct subgroup analyses to expand what can be learned from the study. For example, by dividing the sample of teachers by the amount of prior teaching experience they have, we can present independent estimates of the program effect for more experienced teachers and less experienced teachers. Such estimates are useful to assess, for example, whether an intensive intervention like QTEL is most successful with teachers whose existing teaching styles are more conventional and entrenched or with teachers who have less experience and are more easily influenced. The answer to such a question has consequences for future replication of the program and may prompt future changes in program design and operation. Similarly, program impacts could vary by student background characteristics. For example, the program may be more successful for first-generation ELL students than for those qualified as ELLs despite having been born in the U.S. The most significant subgroup breakdown will be the one that estimates separate impacts for ELLs and non-ELL students.

Analytically, conducting subgroup analyses is very straightforward and preserves the underlying experimental nature of the data. To estimate subgroup impacts, we would interact variable T_k in Equation 1 with one of the variables Z, X, or S in the same model. Equation 2, for example, shows how we would estimate separate program effects for ELL and non-ELL students:

(2)

In this model, the impact regression presented in Equation 1 is modified so that the term ₁T_j is replaced by two terms that interact program variable T_j with two 0/1 dummy variables ELL_i and NOELL_i. The effects of QTEL on ELLs and non-ELL students are captured by the coefficients _1AT_j and _1BT_j, respectively. By statistically testing the hypothesis _1AT_j=_1BT_j, we determine whether the program effect varies significantly across these two groups of students.^⁸ Although similar subgroup analyses can be conducted at the teacher and school levels, the statistical power of subgroup analyses at those levels is more limited. This is true especially at the school level. It may be possible to reliably calculate impacts for some of the larger subgroups of schools (such as those in the San Diego Unified School District), but it may not be possible to statistically confirm that differences in impacts across groups of schools are statistically significant.

A concern with subgroup analyses in experimental studies like these is that conducting a large number of them may lead one to “detect” impacts that appear statistically significant by chance. Therefore it is important to specify which subgroup breakdowns will be explored upfront. At this time, we plan to conduct the following subgroup analyses, at the student, teacher, and school levels:

Students:

ELL classification (LEP, reclassified LEP, native English speaker)
Prior academic performance
Peer academic performance (for ELL students and overall)
Race/ethnic group
Language spoken at home
Sex
Free/reduced lunch
Born in the U.S.

Teachers:

Experience
Certification/education
ELA/ESL
Sex
Age
Race/ethnic group

Schools:

District
Urban/suburban
AYP status
Achievement gap (measure to be developed)

OMB Expiration Date

All data collection instruments will include the OMB expiration date.

Exceptions to Certification Statement

No exceptions are requested.

1 Source: http://www.air.org/overview/herman.aspx

2 The exception is very small districts with only one middle school, of which there will likely be a few in the sample.

3 There are two forms – one for the treatment teachers and another for the control teachers.

4 It will be based on the Sheltered Instruction Observation Protocol (SIOP©), published by Pearson Education, Inc (2004). Since SIOP is a copyright protected instrument, only the description of the protocol is included in this package.

5 There will be no time and cost burden associated with this data collection activity since those coaches will be part of WestEd QTEL program staff.

6 Time needed to assist with videotaping.

7 632 = 50 (principals)/3 + 600 (teachers) + 15 (district staff)

8 Note that it would be possible to estimate these effects by simply adding one of the two new interaction terms to Equation 1. In that case, the significance of the interaction effect would be the significance of the added interaction coefficient. However, by replacing the original P_kwith two new interaction terms, the two distinct subgroup impacts can be read directly from the regression output, thereby avoiding the need to calculate them manually or in a separate procedure.

Evaluation of the QTEL Program OMB Clearance Package

File Type	application/msword
File Title	QTEL OMB Supporting Doc
Author	Tommy
Last Modified By	Sheila.Carey
File Modified	2007-07-12
File Created	2007-07-12

Att_QTEL OMB Supporting Statement Part A_4.20.07

Att_QTEL OMB Supporting Statement Part A_4.20.07.doc

Evaluation of the Quality Teaching for English Learners (QTEL) Program

OMB: 1850-0842

Prepared for

Prepared by

Table of Contents

List of Exhibits

Introduction

Circumstances that Make Data Collection Necessary

Purposes and Uses of the Data

Use of Improved Technology to Reduce Burden

Efforts to Identify and Reduce Duplication

Efforts to Minimize Burden on Small Businesses

Consequences of Not Collecting the Information

Special Circumstances

Federal Register Comments and Outside Consultants

Payments to Respondents

Assurance of Confidentiality to Respondents

Justification for Questions of a Sensitive Nature

Estimate of Information Collection Burden

Estimate of Total Annual Cost Burden

Estimate of Annual Cost to the Federal Government

Change in Annual Reporting Burden

Data Collection Schedule

Analytical Procedures

Subgroup Analysis

OMB Expiration Date

Exceptions to Certification Statement