SSB EHSREP Tracking_021313_final

Tracking of the Early Head Start Research and Evaluation Project (EHSREP) Sample

OMB Information Collection Request

0970-0388

Supporting Statement

Part B

February 5, 2013

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Project Officers:

Amy Madigan

B1. Respondent Universe and Sampling Methods

The respondent universe consists of all participants in the Early Head Start Research and Evaluation Project except those who in previous waves of data collection requested to be removed from the study. The sample size for this effort will be 2533. Response rate is expected to be 70%, which is the actual response rate achieved during the last tracking data collection, in 2011. We do not anticipate problems with item non response. In the 2011 effort we did not experience significant problems with item non response. 99% of all completing cases provided confirmation of parent and child names, updated home address, and provided a telephone number. 90% of cases were able to provide name, address, and phone information for at least one third-party contact.

B2. Procedures for Collection of Information

The data collection contractor, The RAND Survey Research Group (RAND/SRG), will employ a mixed-mode approach to include Internet, mail, and telephone surveys, distance and field tracking, and in-person interviews.

• Locating Service: Prior to the initial contact with participants, the entire EHSREP sample will be sent to a locating service for updated phone and address look-up in their database. Any new contact information provided by the locating service will be incorporated into the project record management system (RMS) before the sample is fielded.

• Email Effort: At the beginning of the field period, the Email Invitation (Attachment F) will be sent by automated system to all cases with an email address in the sample file. The email message will be the first contact and will describe the project, provide assurances of privacy, and explain the importance of continued participation in the evaluation. The email message will include a link to the Tracking survey, Internet Version (Attachment D) and a unique PIN code that will allow each respondent to access the tracking survey. The email text will also include the toll-free study number for those who would prefer to complete the interview by phone. The Email Reminder (Attachment G) will be sent one week after the initial email and every two weeks thereafter.

• Mail Survey: Coinciding with the first reminder email message, an advance mailing will be sent to all cases with a valid street address. The initial mailing will contain the Mail Cover Letter (Attachment E) that describes the project, privacy, and the importance of continued participation in the evaluation. The letter will also include the Tracking Survey, Internet Version URL and a unique PIN code. Also included in the mailing will be the Brochure (Attachment H), the Tracking Survey, Mail Version (Attachment B), and a postage-paid envelope addressed to RAND/SRG. While all materials will be printed in both English and Spanish, only those families that completed the 2011 survey in Spanish or were identified as Spanish-only prior to 2011 will be sent Spanish language materials. A second mailing will be sent approximately four weeks after the first.

• Telephone Survey: All materials will provide a study toll-free number that respondents can call either to complete the contact interview or receive more information about the study. The toll-free number will be manned by a bilingual staff person. Since the phone center will become active starting at the initial email, all cases will be loaded into the Telephone Survey Center (TSC) to receive incoming calls from respondents who wish to complete the survey by phone. Approximately three weeks after the email invitation and two weeks after the advance mailing, outgoing calls will begin to all non-responders with a telephone number. Telephone interviews will be completed using Computer-Assisted-Telephone-Interviewing (CATI) programmed in Berkeley’s CASES software. Interviewers will have access to all phone numbers provided for 2011 completers, the best previous number and up to 3 others acquired from previous efforts for 2011 non-completers, and all updated numbers from locating services for the full sample. A computerized call scheduler ensures the cases are delivered at varying times and on varying days of the week. Telephone interviewers will use the Tracking Survey, Telephone Version (Attachment C). Primary outgoing calls will last approximately six weeks; however, the TSC will remain active to support distance and field tracking throughout the data collection effort.

• Distance Tracking: In addition to the contact efforts described above, RAND/SRG will employ a multi-step tracking and locating procedure. Cases for which we have no phone number, or whose phone number is found to be bad in the TSC, or with whom we have not made contact or confirmed the number for in the TSC after seven calling attempts will be sent to distance tracking. RAND/SRG distance trackers working from the TSC use an online tracking case management system that is part of the RMS. The system updates daily with the statuses of cases in other study components, adding new cases that require tracking and finalizing cases that have completed the survey. RAND/SRG distance trackers will contact third parties (friend/family contacts) provided by participants at previous waves of data collection using the Third-Party Contact Script (Attachment K), make calls to Directory Assistance, and conduct database look-ups through licensed use of the LexisNexis database.

• Field Tracking/In-Person Interviews: When all steps in distance tracking have been exhausted without locating the respondent, the case will go to field tracking. Field tracking efforts will begin approximately four weeks after outgoing calls begin and will include cases that failed distance tracking or respondents have been non-responsive despite a confirmed valid telephone number. Field interviewers will develop a strategy for each case on the basis of what is known about the family, such as language spoken in the home, last date located, last known address, forwarding address information, third party contacts listed, distance tracking results, and any useful information obtained from field tracking efforts in 2011. Field tracking methods include, for the respondent first and then for the third party contacts, visiting last known address at different times of day and on different days, asking neighbors, current resident, apartment manager, local post office or postal carrier for information on how to contact respondent; using reverse directories; and stressing the respondent’s voluntary participation and the incentive when speaking to contacts. When a respondent is located, the field interviewer will conduct the interview on the Tracking Survey, Mail Version (Attachment B). When field interviewers are speaking with third parties, they will use the Third-Party Contact Script (Attachment K).

• Administrative Records Consent Form (Attachment J): Once child respondents turn 18, written consent will be requested to obtain administrative records.

• Record Management System: RAND/SRG will use a record management system (RMS) to manage the survey sample, document interim and final case status codes, and generate weekly progress reports.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

The expected response rate for this project is 70%. Using the same mixed mode methods described in Supporting Statement A, Part A2, the 2011 effort was able to achieve a 70% completion rate. We have no reason to anticipate greater numbers of refusals or loss under the proposed data collection. Furthermore, given the length of time since the EHSREP first began and the nature of the population targeted, we believe a 70% completion rate is a reasonable and appropriate goal that will allow the government to maximize the success of future data collection efforts. 70% completion is well above the completion rates seen with this sample at the grade five follow-up (Carlson, 2009).

Dealing with Nonresponse

The EHSREP began in 1995 with the enrollment of over 3000 families across 17 study sites. Families were randomized such that children were either enrolled the EHS program (treatment group) or not enrolled in the program (control group). Data was collected at the baseline and children were followed across multiple waves of data collection. The EHSREP sample has experienced a moderate rate of attrition since its inception in 1995. Approximately 45% of the sample was lost by the 5th grade follow-up. In an assessment of the effects of non-response bias on the EHSREP sample, Carlson (2009) found some evidence for differential attrition with the control group having slightly higher rates of nonresponse compared to the control group. She also compared the distribution of a series of variables measuring sample characteristics, believed to correlate with critical outcome measures, for the full baseline sample and for respondents to the 5th grade follow-up. For the majority of characteristics, Carlson found no differences in the distributions. There was, however, evidence for some differences with lower-educated mothers, high-risk mothers, and participants in urban programs being less likely to complete the 5th grade follow-up interview. Based on the overall pattern of similarity between the full sample and that of the grade 5 respondents in the distributions of baseline characteristics, as well as the finding that characteristics of the program and control group members were statistically similar, Carlson concluded that there was relatively limited evidence for a nonresponse bias in the EHSREP 5^th grade follow up. While this is encouraging for the data collection to date, further erosion of the study sample could ultimately compromise the ability to make estimates for key populations in the dataset.

In 2011, ACF contracted with RAND/SRG to conduct a tracking project with the EHSREP sample during which the goal of locating 70% of respondents was successfully met. The 2011 data collection was meant to help maintain and re-establish contact with families enrolled in the EHSREP sample to assist in planning for future waves of data collection. This in turn, would help to minimize any effects of non-response on non-sampling error.

For the current proposed data collection, we will examine tracking survey non-response by study condition (program vs. control groups), participation in the most recent tracking effort, and parent language. To the extent that additional sample characteristics are made available to RAND by ACF, non-response will also be examined of factors such as race/ethnicity, parental education, and family income.

Maximizing Response Rates

RAND/SRG will employ proven methods to maximize response rate. The survey methodology includes follow-up with non-respondents by email, mail, telephone, and in person, and the survey can be completed by any of these modes. Materials are designed to be easy to read and are offered in both English and Spanish. A gift of $10 in appreciation of participation will encourage participation. A $5 gift will be offered to 18 year old child respondents in order to encourage return of signed written consent form.

B4. Tests of Procedures or Methods to be Undertaken

No pre-testing of the instruments or items will be conducted as part of this project. Tracking survey items are primarily questions about contact information which are identical to those used in the 2011 tracking project and the few new child and family well-being questions are taken from existing instruments. Well-being items include parent education (2000 EHSREP parent interview), parent employment and future expectations (National Survey of Child and Adolescent Well-Being), health and mental health (SF-12), and contact with the juvenile justice system (newly drafted).

The Internet and telephone tracking surveys will be tested thoroughly to ensure that programmed fills and skip patterns are correct. There will be full-cycle testing of the record management system to ensure that status updates from the Internet and CASES (the telephone interview program) and other elements execute as designed for efficient management of the data collection.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Project design, data collection and analysis are being conducted by the contractor, The RAND Survey Research Group (RAND/SRG), under contract to the U.S. Department of Health and Human Services.

Kirsten Becker, MS

Principal Investigator and Survey Director

Survey Research Group

RAND Corporation

Email: becker@rand.org

Barbara Levitan

Survey Director

Survey Research Group

RAND Corporation

Email: blevitan@rand.org

Fred Mills

Field Specialist

Survey Research Group

RAND Corporation

Email: fmills@rand.org

Amy Madigan, PhD

Federal Project Officer

Senior Social Science Research Analyst

Office of Planning, Research and Evaluation

Administration for Children and Families

Email: amy.madigan@acf.hhs.gov

References

Carlson, B.L. (August 2009). Analysis of Nonresponse Bias in the Early Head Start Research and Evaluation Project. Paper presented at the Joint Statistical Meeting of the American Statistical Association, Washington, DC.

Dillman, D.A., Phelps, G., Tortora, R., Swift, K., Kohrell, J., Berck, J., & Messer, B.L. (2009). Response Rate and Measurement Differences in Mixed-Mode Surveys Using Mail, Telephone, Interactive Voice Response (IVR) and the Internet. Social Science Research, 38, 1–18.

Nathan, G. (1999). Review of Sample Attrition and Representativeness in Three Longitudinal Surveys. Government Statistical Service, Methodology Series, No. 13. Office for National Statistics, London.

Weiss, C. & Bailar, B. (2002). High Response Rates for Low-Income Population In-Person Surveys. In Ver Ploeg, M., Moffitt, R., & Citro, C., (Eds), Studies of Welfare Populations: Data Collection and Research Issues, pp. 86-104. The National Academic Press, Washington, DC.

List of Attachments

1. OMB 60 Day Notice

2. Tracking Survey, Mail Version

Mailed with cover letter, brochure, and reply envelope.

3. Tracking Survey, Telephone Version

Programmed for Computer-Assisted-Telephone-Interviewing (CATI).

4. Tracking Survey, Internet Version

Programmed as a web-based survey instrument. Links to the online survey are included in most materials and direct links are embedded in emails.

5. Mail Cover Letter

Included with the tracking survey, mail version.

6. Email Invitation

Text of email sent to respondents with valid email addresses inviting them to complete the tracking survey online and including a direct link to the survey.

7. Email Reminder

Text of reminder email sent to non-responders with valid email addresses reminding them to complete tracking survey online.

8. Brochure

Included with the tracking survey, mail version and used in the field tracking effort.

9. Thank you Letter

Mailed after the tracking survey is completed, all versions.

10. Administrative Records Consent Form

Requested of child participants who have reached 18 years of age.

11. Third-Party Contact Script

Used when calling third-party contacts (family and friends provided by participants in earlier data collection waves) to help locate participants.

12. New Wellbeing Items

Table of items newly added to this tracking effort.

8