Q-DOT - Section B_TO OMB_FINAL_v2

Submitted by: Mathematica Policy Research 1100 1st Street, NE 12th Floor Washington, DC 20002-4221 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Louisa Tarullo

Q-DOT Pilot Study--OMB Information Collection Request 0970-0355 Supporting Statement Part B: Statistical Methods for the Study December 2013 Submitted to: Office of Planning, Research and Evaluation Administration for Children and Families U.S. Department of Health and Human Services 7th Floor, West Aerospace Building 370 L'Enfant Promenade SW Washington DC 20447 Project Officer: Ivelisse Martinez-Beck

CONTENTS

B STATISTICAL METHODS (USED FOR COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS) 1

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for Collection of Information 1

B3. Methods to Maximize Response Rates and Deal with Nonresponse 2

B4. Tests of Procedures or Methods to be Undertaken 3

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 3

B. STATISTICAL METHODS (USED FOR COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS)

B1. Respondent Universe and Sampling Methods

The sample for the Q-DOT Pilot Study will include a purposive sample of child care centers across a range of auspices, including community-based child care centers, Head Start centers, and schools with state-funded pre-kindergarten classrooms, as well as center directors and classroom teachers. We will recruit centers within a 60-mile radius of Mathematica’s Washington, DC office that participate in the Virginia Star Quality Initiative (VSQI). Using the VSQI star ratings, we will select seven Quality Rating and Improvement System (QRIS) facilities with low ratings and eight QRIS facilities with high ratings, for a total of 15 programs. We will prioritize having representation of all types of auspices within the sample.

Next, we will ask selected child care centers/programs to help us identify up to three classrooms and teachers to participate in the pilot study. We describe this process in more detail in section B2. To be included in data collection activities, teachers must use English or Spanish as the primary language of instruction in their classroom, and respondents must be able to complete questionnaire items in English.

B2. Procedures for Collection of Information

Recruitment and data collection activities for the Q-DOT Pilot Study will commence upon OMB approval and are expected to last approximately four months. Mathematica staff will schedule the data collection activities at each center/program based on its availability. We anticipate a recruiting period of two months to enroll 15 centers into the study and a data collection period of five weeks.

In addition to a number of supporting documents (elaborated on below), the Q-DOT Pilot Study includes the following data collection instruments:

1. Program Director Information Call Script and Center Director Self-Administered Questionnaire (Attachment A1)

2. Teacher Self-Administered Questionnaire (Attachment A2)

We include these data collection instruments in Attachments A1 and A2 and the additional supporting documents (i.e., Program/Center Director Invitation Letter, Q-DOT Frequently Asked Questions, Teacher Cover Letter, and Program/Center Director Cover Letter,) in Appendix A1 and B1-B3. The Program/Center Director Invitation Letter, which contains a brief overview of the pilot study goals and activities, will serve as the initial mode of contact for inviting programs to take part in the pilot study. Guided by the Program Director Information Call Script, a team of two to three Mathematica recruiters will place a follow-up call during which programs will be more formally invited to participate in the pilot study. During this call, recruiters will describe the pilot study purpose, provide an overview of the pilot study activities, and confirm the program’s interest in participating. We will work with the participating programs to identify a sample of teachers for the classroom observations. We will ask each center to identify up to three preschool teachers to participate in the classroom observations. Classrooms will be eligible if the instruction is mostly in Spanish or English and the children in the classroom range from 3-5 years old. In addition, center directors and participating teachers will each be asked to complete a brief questionnaire. The Center Director Self-Administered Questionnaire will collect information on center demographics, staffing, recruitment, training and curriculum. The Teacher Self-Administered Questionnaire will collect information on staff experiences and provide important information on the background and demographics of staff participating in the study. All data collection instruments and supporting documents to be used with Spanish-speaking staff will be translated by a certified Mathematica translator.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

1. Expected Response Rates

Mathematica has an established, successful record of gaining program cooperation and obtaining high response rates from staff and families in research studies of Head Start and early education programs. Based on our experiences with FACES 2006 and 2009, we anticipate a 90 percent response rate from the classroom observation, an 85 percent response rate from center directors and classroom teachers, and a 75 percent response rate from the center recruitment effort.

The Center Director and Teacher Self-Administered Questionnaires (SAQs) were developed using a 10th-grade reading level, as measured by the Flesch grade level readability formula, to ensure that participants can understand the questionnaire directions and items. We will provide centers and teachers with gifts of appreciation that will help ensure a high level of participation.

2. Dealing with Nonresponse

Given our approach to the recruitment of programs and participants, we do not anticipate issues associated with nonresponse. Teachers and center directors will be sent a postage-paid return envelope in which to return their SAQs. Those who have not responded by the time of the scheduled observation will be given an opportunity to do so while the observers are on site; observers will collect outstanding questionnaires and deliver them to Mathematica.

3. Maximizing Response Rates

We will offer center/program directors a $100 post-pay check for participating in the pilot study. Teachers will be offered a gift valued at $20 as a token of appreciation. These gifts of appreciation are in line with those offered in similar ACF-funded studies, such as the Head Start Family and Child Experiences Survey (FACES), the field test of the Quality of Caregiver-Child Interactions with Infants and Toddlers (Q-CCIIT), and Head Start Voices.

We will provide the child care centers with a recruitment mailing package that is easy to read and understand, and that includes a one-page letter explaining the study and the benefits of participating. The recruitment mailing also will include a Frequently Asked Questions (FAQ) document that explains the purpose of the study, what data elements are included in the study, how sample members’ information is protected, and who they can call if they have additional questions. Recruiters will use a recruitment script that introduces center directors to the study and invites them to learn more about how their participation will benefit their center and the QRIS program as a whole.

B4. Tests of Procedures or Methods to be Undertaken

The items included in the Center Director and the Teacher SAQs are either identical or similar to questions used in previous Mathematica data collections. As such, they have been thoroughly tested on large, diverse samples. The planned observation measure, the Classroom Assessment Scoring System (CLASS), has been used in several national studies such as FACES and Baby FACES, and is used in VSQI.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Ivelisse Martinez-Beck, federal project officer; Dr. Louisa Tarullo, project director; and additional staff contributed to the design of the Q-DOT Pilot Study, including Drs. Martha Zaslow and Margaret Burchinal as coprincipal investigators, and Dr. Robert Pianta as a study advisor. In addition, Drs. Burchinal and Zaslow will provide input on data analysis conducted by Dr. Elizabeth Cavadel. A team of four Mathematica survey associates will participate in data collection efforts.

Inquiries regarding the statistical aspects of the study’s planned analysis should be directed to:

Dr. Louisa Tarullo, Project Director, Mathematica Policy Research

Ivelisse Martinez-Beck, Federal Project Officer, Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services