Supporting Statement Part B revised 2-16-2007

U.S. Department of Education

Institute of Education Sciences

National Center for Education Evaluation

Follow Up to the Even Start

Classroom Literacy Interventions

and Outcomes Study

Office of Management and Budget

Clearance Package Supporting Statement

and Data Collection Instruments

Part B: Collection of Information

Employing Statistical Methods

February 16, 2007

TABLE OF CONTENTS

B. COLLECTION OF INFORMATION EMPLOYING

STATISTICAL METHODS 1

B.1 Sample Design 1

B.2 Data Collection 3

B.3 Methods To Maximize Response Rate 3

B.4 Pilot Testing 4

B.5 Individuals and Organizations Involved in this Project 5

FOLLOW UP TO THE EVEN START CLASSROOM LITERACY INTERVENTIONS AND OUTCOMES (CLIO) STUDY

SUPPORTING STATEMENT

FOR PAPERWORK REDUCTION ACT SUBMISSION

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Sample Design

Using the results of the Even Start screener, volunteers were recruited for the original CLIO study from projects that were interested in implementing a new family literacy intervention and willing to participate in random assignment. The procedures used to recruit and sample projects are described in the following sections.

Methodology. In the spring of 2003, CLIO surveyed all Even Start projects to identify (a) the number of children by age being served in the project, (b) where and how early childhood education services are being delivered to preschool children (e.g., services provided in the home or at a center, and, if center-based, whether the services are provided by Even Start or under another auspice such as Head Start or a school district), and (c) the distribution of Even Start preschool children across early childhood education. To be eligible for CLIO, projects had to meet the following criteria:

• Serve a minimum of either: (a) five 3- and 4-year-olds in one center-based early childhood education classroom, or (b) eight 3- and 4-year-olds in two center-based early childhood education classrooms;

• Provide at least 12 hours per week of center-based early childhood instruction;

• Serve a majority of families speaking either English or Spanish; and

• Be willing to participate in the study, including being randomly assigned to one of the five study conditions.

Due to the voluntary nature of participation and the specific eligibility criteria for the study, the CLIO projects were not intended to be nationally representative. However, the 120 recruited projects were spread over 33 states in all regions of the country, and varied a great deal on characteristics such as urbanicity, number of families served, percentage of families who were English Language Learners, and experience implementing Even Start.

Even Start projects, including the subset recruited for the original CLIO study, are extremely variable. The CLIO random assignment plan focused on ways to ensure there were no significant pre-intervention differences among the five different study groups. Before random assignment, we formed 24 strata, each containing exactly five projects. We then assigned one project from each stratum across each of the five groups. The variables used to form the strata were (1) size of project (number of families served), (2) proportion of children who were Spanish speakers, (3) year that the project is up for recompetition, and (4) region. Projects were randomly assigned to one of the five conditions in early 2004.

Preschool children and their families were eligible for participation in the original CLIO study based on the age of the child and center attendance. Each fall and each spring, we asked the CLIO projects to provide rosters for children and their families that met the age eligibility criterion. In fall 2003, spring 2004, fall 2004, spring 2005, and spring 2006, the study collected data about all Even Start enrolled 3- to 5-year-olds (and their parents) who were not yet in kindergarten.

Sample Size. The number of Even Start projects needed for the CLIO preschool sample was based on (a) the number of interventions tested, (b) the size of the difference that we wanted to be able to detect between any two groups of projects (e.g., intervention versus control, or one intervention versus a second intervention), (c) the power that we wanted the statistical tests to have, (d) the number of children to be sampled within each classroom, and (e) the intra-class correlation for child outcomes. In determining the sample size for the original CLIO study, the parameters were as follows:

• We planned to test four interventions and compare them to a control group of existing Even Start projects.

• We wanted to be able to detect a difference of .33 standard deviations between the effects of two different family literacy interventions.

• We wanted to have .80 power in the design.

• We estimated the intra-class correlation to be .10, based on data from the earlier evaluations of Even Start.

A design in which students are randomly assigned to interventions would be the most efficient, i.e., you could obtain a given level of power with the smallest sample size. Alternatively, a design in which all data are collected at the classroom level, i.e., there were no child-level data associated with classroom but only classroom means, and classroom is the unit of randomization, would be the least efficient. The original CLIO study design, in which students are nested within classrooms, is more powerful than a purely classroom-level design, since we can take advantage of the fact that we will have longitudinal data on an average of 15 students from each Even Start project. The estimated sample size for the CLIO Follow Up study is 850 children for each data collection year.

An attrition rate is built into the projected sample sizes. This attrition rate is based on ED’s experience with the Experimental Design Study of Even Start and includes refusals, children/families who moved beyond a 30-mile radius of the original Even Start project, and children and families who could not be located. Based on this experience, we expect a greater part of the attrition to be a result of children moving from the area and children who cannot be located than a result of outright refusal. These assumptions are also supported by the results of other studies of similar populations (e.g., Head Start FACES and the Comprehensive Child Development Program evaluation).

Our response rate goal for the CLIO Follow Up data collections is 80 percent.

B.2 Data Collection

The Follow Up study will collect data in the spring of each year.

Data collection in spring 2007 will include:

• In-person child assessments, parent interviews, and videotaping parents and children – training of field staff will occur in mid-March, and data collection will begin after training and conclude in June.

• Teacher surveys and TCRs – mailing to teachers will take place in late March and follow up will conclude in May.

Data collection in spring 2008 will include:

• In-person child assessments, parent interviews, and videotaping parents and children – training of field staff will occur in mid-March, and data collection will begin after training and conclude in June.

• Teacher surveys and child rating forms – mailing to teachers will take place in late March and follow up will conclude in May.

B.3 Methods to Maximize Response Rate

Westat routinely achieves high response rates by attending to the basics in conducting research. First, the project will recruit skilled and culturally sensitive field staff. In addition, data collectors will be carefully trained in the techniques for minimizing the level of nonresponse. Data collectors and supervisors will contact each family in advance of the start of the data collection. They will remind families of the purpose of the study and the activities that will be conducted. They will answer any questions the family has, and will schedule a time and place for data collection. They will schedule the data collection at the convenience of the family, always working around the family’s schedule. They will offer to come to the family’s home or to secure a more neutral, mutually agreeable location (such as a library or a community center). Furthermore, the study team in general and the supervisors and most of the data collectors, specifically, have the good will and rapport with the families that has been established from previous data collections, and these previously-established relationships will be important in minimizing non-response. (All of the supervisors and the majority of the data collectors have been with the CLIO study over several rounds of data collection.) The protocol will be to make 7 follow-up attempts, at different times of the day and on several different days, to maximize the response rate. The goal is to reach at least an 80 percent response rate in each data collection cycle.

Factors that Westat will be able to control to influence the overall response rate include the following:

• Ability to Obtain Cooperation. The use of a local site coordinator to contact families will help ensure a high response rate. Another important factor in maximizing the response rate will be the ability of the field staff to encourage the respondents to participate. All field staff will be thoroughly trained and given rapid feedback on their performance in training to eliminate behavioral patterns that can be detrimental to achieving cooperation. A key factor in the ability to obtain cooperation will be a clear understanding of the study’s purpose and importance, which will be featured prominently in training.

Training will emphasize obtaining and maintaining cooperation as well as administering the instruments. Role plays that focus on interacting with the respondent and avoiding refusals will be performed. During training, field staff will be provided with answers to many typical respondent questions and will be encouraged to practice these until they are comfortable with their ability to explain the study and encourage respondent participation.

• Flexibility in Completing Interviews. Effective contacting patterns are essential for achieving high response rates on all surveys. Our experience shows that individual respondent schedules (work, classes, recreational activities, vacations, etc.) have a negative impact on the response rate when attempts are limited to a short time span. Completion rates improve when interviewers are available on different days of the week and at varying times of the day and evening.

• Followup of Refusals. There will be a number of reasons refusals occur, and data collectors will be trained to elicit, as diplomatically as possible, the reasons for a respondent’s reluctance to participate, breaking off contact when necessary to avoid a firm refusal. Supervisors and data collectors will consult on the approaches to be taken.

• Non-English Interviewing. Westat will recruit and train bilingual (Spanish) staff for the field effort. If the respondents speak any languages other than Spanish, the languages will be identified and the interview will be conducted with the assistance of an interpreter. The interpreter will either be another person in the family or a person of the family’s choosing. An exception is that assessments will not be conducted using an interpreter.

• Incentives. As mentioned in section A9 of this clearance package, we propose that respondents be provided with modest incentives designed to facilitate their participation.

B.4 Pilot Testing

The kindergarten and first grade instruments to be used in the spring 2007 and 2008 data collections (child assessment, parent-child video, and parent interview) will be pilot tested using fewer than ten respondents each. For the teacher survey and TCR instruments, we will mail the instruments to fewer than 10 teachers to complete, and the teachers will be debriefed after the responses are received. We will make any needed revisions based on the results of the pretests.

B.5 Individuals and Organizations Involved in this Project

The contractors for this project are Westat and its subcontractor, STP Associates. Contact information for key personnel is provided below.

Company	Contact Name	Telephone Number
Westat
Project Director	Babette Gutmann	(301) 738-3626
Statistician	David Judkins	(301) 315-5970
STP Associates
Principal Investigator	Robert St.Pierre	(970) 453-7295