SSB - VIQI GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Variations in Implementation of Quality Interventions

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

JUNE 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Ivelisse Martinez-Beck

Amy Madigan

Part B

B1. Objectives

Study Objectives

The information collection proposed as part of the Variations in Implementation of Quality Interventions: Examining the Quality-Child Outcomes Relationship in Child Care and Early Education Project (VIQI Project), consisting of up to 130 one-time interviews with center administrators and up to 20 interviews with state and local ECE stakeholders, will help contextualize the implementation and impact findings of the VIQI study. The information collection has the following objectives:

• To understand the experiences, successes and challenges faced by centers who participated in the VIQI project and how those experiences compared to prior years, and

• To understand the state and local context (e.g., economic and employment conditions, the COVID-19 pandemic, and changes to ECE standards or requirements) during the 2021-2022 school year that may have affected participating ECE centers.

Generalizability of Results

This study is intended to present descriptive information about the ECE context during the VIQI study. This information collection is not intended to promote statistical generalization within or beyond the target population.

Appropriateness of Study Design and Methods for Planned Uses

Semi-structured interviews are proposed because they enable in-depth description and understanding of the contextual information needed to help inform and contextualize the study’s findings. They allow all interviewers to have the same structured set of questions while also allowing the interviewer to follow-up and probe on individual interviewee’s responses as needed.

We will not use the data to make statistical inferences or generalize findings beyond the study sample. The information will only be used to describe the centers, localities, and states in the VIQI sample. We will include the limitation that the sample is not representative in any written products associated with the study.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The target population is center administrators at centers participating in the VIQI project and state and local ECE stakeholders from the localities or states in which the VIQI project is being conducted.

Sampling

The study team will leverage its relationships with stakeholders in each of the VIQI localities and at VIQI centers to recruit participants for these interviews. The team will use the email templates (see Appendix A) and talking points (see introductions to Instruments 1 and 2) in reaching out to their contacts.

One center administrator from each of the centers participating in the VIQI study will be asked to participate in an interview. This will be the administrator who was previously selected to participate in the VIQI project (e.g. to take the administrator survey under OMB #0970-0508).

Key state and/or local-level ECE stakeholders who have been engaged in the VIQI project (e.g., as part of our initial landscaping discussions in each locality) will be asked to participate in an interview to help us understand aspects of the local context (e.g., ECE systems-level changes, the COVID-19 pandemic context, local employment and economic conditions) that may influence implementation and business-as-usual ECE. These will include administrators who have oversight of child care and Head Start programs in that locality. We will aim to identify up to two locality or state-level informants in each of the ten VIQI localities.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The interview protocols were developed by the project team to address the research questions. We first identified key information needed for a fuller understanding of the context under which the VIQI impact evaluation unfolded, particularly related to the effects of COVID-19 on center functioning, staffing, and instruction as well as on the locality or state. The protocols have been designed so that the interviewer can tailor each interview to the intervention group the center administrator was in and to the particular role different state/local ECE stakeholders may play.

In order to avoid measurement error (e.g., leading, or unclear questions), we revised the interview protocol based on feedback from the project team, including both contractor and ACF staff.

B4. Collection of Data and Quality Control

Members of the research team will contact potential participants and facilitate the semi-structured one-on-one discussions in person, on the phone, or via video conferencing platform (e.g., Zoom). We expect that recruitment will be an iterative process. Members of the project team will contact center administrators and state and local ECE stakeholders individually by e-mail, explaining the purpose of the interview, describing the nature of the discussion, and inviting their participation (see Appendix A for an e-mail template/talking points). If the administrator agrees to participate, a member of the project team will schedule a call at a time that is convenient to the participant. If an administrator does not respond, a member of the project team may reach out by phone.

If an invited individual is unable to participate, the research team will ask whether another representative of the individual’s organization might be appropriate and available for a discussion and will invite suggested administrators as needed. For state and local ECE stakeholders, if no alternative is given, the research team will contact an alternative informant that we expect can discuss the local context in that locality or state.

A member of the research team will facilitate each interview according to the Center Administrator Protocol (Instrument 1) or the State and Local ECE Stakeholder Protocol (Instrument 2) depending on the participant’s role, tailoring the conversation according to participants’ backgrounds and expertise. Each interview will last no more than 60 minutes. The interviews will be recorded, with participants permission, and transcribed. A member of the research team will also take detailed written notes on each conversation.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. We will qualitatively assess non-responses to monitor gaps in the research conditions, ECE setting type, and localities represented in our data and to tailor our recruitment efforts to improve the data collected and reduce burden on respondents.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Electronic notes and audio recordings taken during interviews will be stored in a secure, password-protected location, according to the project’s data security plan. Audio recordings will be deleted after transcription.

Data Analysis

Interview transcripts will be analyzed by research staff to identify themes. The qualitative data analysis will follow a deductive coding process. Codes will be developed to capture information about the ECE landscape and workforce during the VIQI project. The coding team will be trained on the scheme prior to coding and, during the coding period, will meet regularly to review assumptions (ensure inter-rater reliability); address analytic challenges; discuss, compare, and cross-check themes; identify major themes that were not reflected in the deductive coding scheme; and, reach shared understandings that are valid and reliable. These analyses will be used to provide contextual information for the impact and implementation findings.

Data Use

Reports and/or briefs will be published to summarize the findings from the Impact Evaluation and Process Study. Themes and quotes from the Center Administrator and State and Local ECE Stakeholder interviews may be included in these dissemination products to contextualize implementation and impact findings. Any limitations to the data, including generalizability, will be included in publicly shared information. No personally identifying information, like names, will be shared.

B8. Contact Persons

JoAnn Hsueh, MDRC, joann.hsueh@mdrc.org

Michelle Maier, MDRC, michelle.maier@mdrc.org

Marie-Andree Somers, MDRC, marie-andree.somers@mdrc.org

Electra Small, MDRC, Electra.Small@mdrc.org

Margaret Burchinal, University of Virginia, kqu4rg@virginia.edu

Jean Lennon, RTI International, jlennon@rti.org

Jennifer Kenney, RTI International, jkeeney@rti.org

Attachments

Instrument 1: Center administrator interview protocol

Instrument 2: State and local ECE stakeholders interview protocol

Appendix A: Interview outreach

6