Ssb

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Survey of Child Care and Early Education Supply-Building and Sustainability Efforts

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

December 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Amanda Coleman and Bonnie Mackintosh

Part B

B1. Objectives

Study Objectives

A primary objective of this data collection is to collect information systematically across states and territories to inform ACF about the strategies Child Care and Development Fund (CCDF) Lead Agencies are using to build and sustain the supply of child care and early education (CCEE) for children from birth through age 12. A second objective of this survey is to inform the next stages of this project, specifically, identifying potential sites to invite for case studies of CCEE supply-building and supply-sustaining strategies and to consider for potential future evaluation design.¹

This survey is being conducted as part of an environmental scan. The findings from the survey will extend the findings from the environmental scan components completed to date. The survey will give CCDF Administrators the opportunity to clarify and provide new information on strategies that may now be more developed, relative to descriptions from publicly available information sources identified through a web scan and strategies Lead Agencies reported in their CCDF State Plans for FY 2022-24. It also gives CCDF Administrators the opportunity to share, in their words, about particular strategies that aim to meet goals of building and sustaining CCEE supply.

Generalizability of Results

This study is intended to present an internally valid description of CCEE supply-building strategies implemented in participating state and territory CCDF Lead Agencies, not to promote statistical generalization to other CCDF Lead Agencies or sites implementing CCEE supply-building strategies.

Appropriateness of Study Design and Methods for Planned Uses

This project will administer a survey to a census of state and territory CCDF Lead Agencies. This study’s descriptive design and method (i.e., administration of a web-based survey) is appropriate given that the objective of the study is to identify and describe strategies to inform future case studies and evaluation design options for a subset of efforts ready for evaluation. This study design and method allows us to gather more detailed information about the CCEE supply-building strategies being implemented across all state and territory CCDF Lead Agencies, complementing information we have collected through the environmental scan task, including expert interviews, a review of policy documents, and a web scan, specifically, a review of announcements and relevant publications on each Lead Agency’s website.

This study is descriptive, and data are not intended to be representative. The findings from the study are not intended to be generalizable to all CCDF Lead Agencies, CCDF administrators, or organizations implementing CCEE supply-building or supply-sustaining strategies. Data collected through this survey will include point-in-time information about recent and ongoing CCEE supply-building strategies, and therefore may exclude strategies that have not yet been rolled out or were implemented more than 5 years ago. Respondents will be asked to select and describe two strategies they are implementing that aim to meet goals of building and sustaining CCEE supply. As such, the information we collect may be limited by the respondents’ own perceptions, knowledge, and selection of the strategies being implemented. Findings from this study will be descriptive and cannot be used to make assessments of the effectiveness of the strategies being implemented.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

This study is designed to collect information from the full population of state and territory CCDF Lead Agencies. We will recruit CCDF administrators representing each state and territory Lead Agency to participate in the survey, and the unit of analysis for this study is the CCDF Lead Agency (N=56).

Sampling and Site Selection

The survey will be administered as a census of Lead Agencies across 50 states, DC, and 5 U.S territories. A sample survey is not appropriate for this information collection because the information we are collecting across the CCDF Lead Agencies is not intended to be generalizable. We anticipate each CCDF administrator will report on information that is unique and specific to their CCDF Lead Agency. Further, the results from this study will inform the selection of potential case study sites and future evaluation design options. To ensure we have complete information from which to invite case study sites, we must sample all state and territory CCDF Lead Agencies for this survey.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument

The research team developed the survey instrument based, in part, on the findings from earlier components of this project’s environmental scan task. A web scan and review of policy documents identified the range and types of strategies, which helped craft the wording of survey item response options. These earlier components, however, revealed little detailed information on strategies being used, which motivated the survey questions about the characteristics of the strategies, timing of implementation, funding sources, and evaluation plans—all of which are important for informing case study site selection. To narrow the survey scope, minimize length, and facilitate identification of the most relevant examples, respondents are asked to consider up to two specific strategies or initiatives from the Lead Agency that meet these criteria:

• currently underway (could have been started within the past 5 years)

• partially or fully funded by CCDF or COVID-19 Relief Funds that the CCDF Lead Agency is charged with administering

• support families of children ranging in age from birth through 12 years old (may be all children or certain age groups)

• related to any approved CCDF activity, but excluding strategies specifically intended to support increased pay or career pipelines for the workforce.

Respondents are then asked to name their first strategy and answer a set of questions about the strategy’s scope, development, implementation, and evaluation (either proposed or actualized). After answering these questions, respondents identify their second strategy and answer the same set of questions. The survey includes primarily close-ended questions, but several questions are open-ended enabling respondents to write in a response or provide web links to resources. Skip patterns direct respondents to additional or different questions depending on previous responses.

During instrument development, the survey team pretested the draft survey questionnaire with two external experts. Pre-test respondents provided written responses to the survey in an electronic version of the instrument in a word processing file, recorded the survey length, and provided written feedback on questions and response options that were unclear or could be improved. The pretest respondents confirmed the survey should take 30 minutes to complete. The survey team edited the questionnaire to reflect expert feedback.

The survey will gather information for descriptive purposes meeting the objectives of this study; therefore, we did not conduct any prior developmental or methodological testing on the survey items. Similarly, this survey will not be used to measure specific constructs, and therefore we do not have information on measurement error.

B4. Collection of Data and Quality Control

Data collection mode

CCDF administrators will self-report strategies being implemented by the CCDF Lead Agency through a web-based survey built, distributed, and managed by the Urban Institute.

Recruitment protocol

ACF maintains contact information for the CCDF Lead Administrators and will provide a list to the Urban research team at the point of OMB approval. This information is also publicly available, but ACF’s list is likely more updated and accounts for recent staff changes. We will use this information to identify and recruit participants.

About 1-2 weeks prior to survey administration, all CCDF Lead Administrators of the 56 State and Territory Lead Agencies will receive an advance email (Appendix A). The advance email will be sent by the Urban survey team to explain the purpose of the survey and to affirm that the email address is the correct one. We will work with ACF in instances where email addresses are incorrect (e.g., mail delivery failure). The email will contain a list of the survey topics to prepare the Administrator for the invitation. The advance email will also include an attached letter of support from the Office of Child Care (Appendix B).

A week after the advance email is sent, the Lead Administrators will receive an invitation email to participate in the web-based survey (Appendix C. This email will include instructions for completing the survey, as well as an Agency-specific survey URL. We will inform administrators that they can share the link with a Co-Administrator or designee who may be more knowledgeable on the topic and better suited to respond, or that they may seek input from their staff to help them complete the survey. Each agency may submit one survey response.

Given the COVID-19 context, the research team understands that the Administrators are busy in their roles and may require some time to gather necessary information and respond to the survey. The survey will be open for about 10 weeks, and we plan to send a reminder email (Appendix D) once every two weeks until the survey closes. After 3 reminder emails, administrators who have not started the survey will receive a follow-up phone call, to find out whether they received the invitation emails and to answer any questions they might have about the survey (Appendix E for the follow-up phone script). If they did not receive the previous emails, then we will try to share the link with them again via email. If they did receive the emails, we will ask if they have any questions about participation and remind them of the survey end date, should they decide to participate. We will send a final weekly reminder email one week before the survey closing date and a “last day to participate” message within 24 hours of the closing date to all respondents that either have not started the survey or have not submitted a completed survey.

Quality control

All research team members will be trained on the data recruitment protocols and materials described above. The research team will also conduct several “test” surveys to review the survey instrument and Qualtrics platform to review for any potential errors prior to survey recruitment. For example, the research team will test all survey functions (e.g., skip logic, required responses) to ensure they work as expected. The team will make any edits to the web-programmed survey questionnaire before survey recruitment. Once survey invitations are sent, the team will periodically review responses to ensure the survey functions are working as expected.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

A response rate will be calculated as a percentage of all responding Lead Agencies out of all invited Lead Agencies (RR = (N/56) x 100). Although we aim to receive responses from all invited CCDF Administrators, the research team will aim for a response rate of at least 80 percent, or about 45 of the 56 State and Territory CCDF Lead Agencies. This response rate goal is based on a 2021 survey of CCDF Lead Agencies conducted for the Child Care Research and Capacity-Building Center under formative generic clearance for research (Data collected under ACF’s umbrella generic: Formative Data Collections for ACF Research. ICR Ref. No. 202102-0970-013; Child Care Research and Evaluation Capacity Building Center Needs Assessment).

Nonresponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. However, if appropriate, we will check rates of nonresponse by the 10 ACF Regions to examine whether certain Regions are more or less represented in the data. We will document any difficulties securing responses and calculate a refusal rate.

B6. Production of Estimates and Projections

This study is descriptive. We do not intend for this information to be used as the principal basis for public policy decision. Data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

The Qualtrics survey will result in a survey data file that we will download, review for errors and completeness, and further clean and analyze. We anticipate little missing data to close-ended questions. Qualtrics allows automatic logic checks to be built into the design of surveys, including mandatory response fields, prespecified valid value ranges, and prespecified patterns of logical and illogical responses across items, which have been incorporated into this survey design. We will use these checks to reduce item missingness. Since survey responses will be completed online, there is less potential for data entry errors and items being inadvertently skipped.

During the data-cleaning stage, open-ended responses will be extracted and exported to an Excel spreadsheet where an analyst will review and clean the data, such as correcting typos and confirming links to web resources are valid. We will review the contents of these resources for relevance. For example, if the resource is an evaluation design report, we will review and consider the information as we code data to inform the case study site selection matrix. In particular, we will want to understand the status of any past, ongoing, or planned evaluations.

Data Analysis

Mixed methods analyses will be used to address the four primary research questions listed below.

1. What efforts to build and/or sustain CCEE supply are CCDF Lead Agencies administering or funding?

2. What are the key characteristics of those efforts?

3. In what ways are Lead Agencies monitoring implementation of CCEE supply-building and supply-sustaining strategies and tracking progress toward their goals?

4. To what degree do the strategies reported by CCDF Lead Agencies show promise as either potential case study sites or sites for future evaluation?

To address research questions 1 through 3, we will analyze survey responses to close-ended and open-ended survey items. Responses to close-ended survey items will be analyzed using primarily descriptive statistics with possible comparisons across groups of states, such as grouping respondents implementing similar strategies (e.g., infant-toddler care supply-building efforts) or identifying and summarizing responses across states and territories that have evaluated their strategies. We will present tabulations of survey responses using either tables or figures (e.g., a bar graph). For example, we will analyze data on the characteristics of the reported strategies and categorize these strategies in different ways, such as supply-building strategies targeting particular geographic areas (e.g., rural communities) or populations of children or families.

The survey has eight open-ended items, with some items including descriptions (e.g., of a strategy’s goals, the indicators used to track progress of a strategy, or the names of organizations or other agencies). In these cases, we will summarize responses by creating a set of categories that sufficiently captures the range of respondents’ answers. We will use tables and figures to show the distribution of responses across these categories. Other items invite a much broader range of responses, such as asking respondents about the rationale for the strategy’s scope. In these cases, we may additionally code responses for mentions of specific themes or ideas (for instance, we might code an item for references to promoting equity). We will use tables and figures to present the prevalence of key themes or ideas across survey responses.

To address research question 4, we will identify and analyze a set of variables important for informing case study site selection. These include survey questions related to timing and status of implementation, clarity of programmatic goals, availability of monitoring data, and the Lead Agency’s interest in evaluating the initiative. We will examine how respondents responded to these questions and then categorize the reported strategies (and the sites implementing them) based on whether they meet certain criteria, such as strategies meeting all identified criteria, some, or none. We will assess whether there are natural groupings of strategies based on their potential readiness for evaluation, such as strategies showing high, medium, or low levels of readiness based on the criteria they meet.

Data Use

The findings from the survey will be used to develop an internal memo that will include detailed survey findings. The memo will include various data tables and figures at the strategy-level or Lead Agency-level or in aggregate form across reported strategies to describe the ongoing efforts of CCDF Lead Agencies. The survey data will also inform, along with other data sources, an environmental scan report. There will be two versions of this report: one version will be internal for ACF’s review and use for planning purposes and the other will be public facing. The internal report will include key survey findings presented alongside findings from the web scan and policy document review from the broader environmental scan. This report will include detail on individual-level responses and activities that specific Lead Agencies are implementing. The public-facing document will include limited aggregate survey data and no individual-level responses, to provide context for the other environmental scan findings. The public-facing report will be valuable to CCDF Lead Agencies looking to learn from other states and territories.

B8. Contact Persons

Amanda Coleman, OPRE project officer, Amanda.Coleman@acf.hhs.gov

Bonnie Mackintosh, OPRE project officer, Bonnie.Mackintosh@acf.hhs.gov

Heather Sandstrom, Project Director, HSandstrom@urban.org

Gina Adams, Principal Investigator, GAdams@urban.org

Sarah Prendergast, Project Manager, SPrendergast@urban.org

Justin Doromal, Survey Lead, JDoromal@urban.org

Attachments

Instrument 1. Survey Questionnaire for CCDF Lead Agencies on Child Care and Early Education Supply-Building and Sustainability Efforts

Appendix A. Advanced Email Recruitment Message to CCDF Lead Agencies

Appendix B. Letter of Support from the Office of Child Care

Appendix C. Survey Invitation Email to CCDF Lead Agencies

Appendix D. Survey Reminder Email to CCDF Lead Agencies

Appendix E. Follow-up Phone Script for Survey Recruitment

1 As appropriate, future information collection requests will be submitted related to future information collections informed by these initial activities.

8