Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Understanding Children’s Transitions from
Head Start to Kindergarten
Pre-testing of Evaluation Data Collection Activities
0970 – 0355
Supporting Statement
Part A
February 2022
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer: Kathleen Dwyer, PhD
Part A
Executive Summary
Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (OMB #: 0970-0355). We are requesting one year of approval.
Progress to Date: The proposed information collection is part of a larger project, Understanding Children’s Transitions from Head Start to Kindergarten. The project has several components that build on the same foundational work (e.g., literature review, expert consultation, development of a theory of change). One component is a comparative multi-case study (OMB #: 0970-0581), for which recruitment is ongoing. Another is the development and initial testing of a new instrument, which is the subject of this Information Collection Request. However, the two information collections are not directly related.
Description of Request: The Office of Planning, Research & Evaluation (OPRE) within the Administration for Children and Families (ACF) at the Department of Health and Human Services (HHS) proposes to conduct cognitive interviewing of an instrument designed to assess Head Start-to-kindergarten transition practices and the transition-related policies, professional supports, and perspectives of the Head Start and K-12 systems. We intend to use the resulting data to evaluate and improve the quality of the instrument. These cognitive testing efforts are well-established methodologies and will ensure that high quality data are collected to refine the instrument. The populations to be studied include Head Start program administrators and K-12 local education agency (LEA) administrators and are not intended to generalize to broader audiences. Data will be analyzed through analysis team meetings that systematically identify issues that arose for survey items through cognitive interviews, determine the source of the issue, and identify potential revisions to survey items. We do not intend for this information to be used as the principal basis for public policy decisions.
Time Sensitivity: This request is time sensitive. Testing of instruments must be complete by June of 2022 in order to produce final versions by the end of the study contract in September of 2022.
A1. Necessity for Collection
The purpose of the Understanding Children’s Transitions from Head Start to Kindergarten (HS2K) project is to better understand how to improve children’s transitions from Head Start programs to elementary schools. Building on a review of the literature, key informant interviews, and a newly developed theory of change, the project will explore a gap in the current knowledge base through the development and refinement of four survey instruments. In particular, the field lacks measures of coordination and alignment of the kindergarten transition at the Head Start and K-12 administrative levels. To fill this gap, these new instruments will probe systems-level and cross-systems policies, practices, shared perspectives including stakeholders’ and others’ roles in supporting transition to kindergarten, and professional supports established to support children, families, and teachers during the kindergarten transition. The proposed cognitive interviews under this generic clearance will refine the newly developed instruments focused on the 4Ps (in bold).
There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose
Purpose and Use
The current information collection is intended for research purposes. The proposed cognitive interviews will support ongoing HS2K Project research activities by probing survey item and response categories’ relevance to key respondent groups, cognitive load, ability to capture key constructs, and overall burden to respondents. OPRE has contracted with NORC, along with its subcontractors (Child Trends and National P-3 Center), to carry out this work. The research team will utilize cognitive interview findings to refine the surveys for use in future ACF research.
Research Questions or Tests
Do Head Start and local education agency (LEA) administrators understand draft instrument questions related to the transition from Head Start to kindergarten in the ways the research team intended?
What edits to existing draft survey items are needed for Head Start and LEA administrators to reliably answer questions about systems-level policies, practices, professional supports, and perspectives related to the transition from Head Start to kindergarten?
Study Design
The research team aims to answer the research questions through cognitive interviews with respondents as they engage with each survey. Cognitive interviews facilitate the refinement of survey questions to ensure that questions are understood as intended by participants. Participants of cognitive interviews are asked to think aloud and describe how they understand question prompts and response options. Researchers then modify questions according to patterns of participant responses. Additional participants (in subsequent rounds of interviews) can then respond to modified questions and response options, once again “thinking aloud” to describe their understanding of questions and points of confusion, until minimal additional changes are recommended by participants. Respondents will include Head Start program administrators and K-12 LEA administrators (e.g., superintendents). The team will virtually interview (over Zoom or similar platform) up to 20 individuals per respondent group for a maximum of 40 respondents. See Supporting Statement, Part B2 for more information on respondent sampling and recruitment. Data collection will take place over approximately 3 months in the spring of 2022.
The sole purpose of the cognitive interview findings is to inform the refinement of the four surveys to maximize relevance to respondents, reduce cognitive load, improve validity, and optimize respondent burden. Cognitive interviews are an appropriate method for the intended purpose because they are designed to assess whether potential respondents interpret the survey items in intended ways and to identify potentially problematic wording or ordering to questions and response options.
Study limitations include a lack of generalizability. The data collected from 40 cognitive interviews cannot be generalized to other populations. It also is possible that other respondents may have different challenges in understanding study questions. The study recruitment is designed to collect information from respondents from different backgrounds (see Supporting Statement B, section B.1) in order to mitigate this weakness.
Exhibit 1. Data collection activities by respondent
Data collection tool |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Screener |
Respondents: Head Start program administrators and K-12 LEA administrators
Content: The screener asks questions about eligible participation in the cognitive interview focused on the type of organization and role of the respondent.
Purpose: To ensure eligible participants complete the protocols. |
Mode: Telephone
Duration: Outreach and screening activities are expected to take about .25 hours |
Head Start Program Administrator Protocols
|
Respondents: 20 Head Start program administrators
Content: Questions associated with the understanding of the instrument as described above and specific questions related to the program policies and the 4Ps. There are three separate protocols we will test with different groups of respondents to minimize the number of items (and corresponding amount of time) the respondent will review and discuss.
Purpose: To refine an instrument. |
Mode: Virtual, semi-structured cognitive interview. Electronically distributed survey.
Duration: 1.5 hours |
LEA Administrator Protocols
|
Respondents: 20 K-12 LEA administrators
Content: Questions associated with the understanding of the instrument as described above and specific questions related to the LEA composition, administrator qualifications, and 4Ps. There are three separate protocols we will test with different groups of respondents to minimize the number of items (and corresponding amount of time) the respondent will review and discuss.
Purpose: To refine an instrument. |
Mode: Virtual, semi-structured cognitive interview. Electronically distributed survey.
Duration: 1.5 hours |
A3. Use of Information Technology to Reduce Burden
The research team will employ information technology as appropriate to reduce the burden of respondents who agree to participate. The interviews will be collected virtually via Zoom or similar platform. Respondents will be given an option to call in via phone. The draft survey will be shared via an electronic format, such as PDF or online survey tool. The participants will not need to relocate to another site or print any materials to participate in the study.
With respondent approval, all interviews will be recorded and transcribed, reducing burden by not needing to confirm responses post-data collection (which would require additional respondent time) and by not needing to take notes, which also minimizes researcher time.
A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
The proposed cognitive interviews do not duplicate any other work being done by ACF and does not duplicate any other data sources. These cognitive interviews will support the refinement of novel instruments intended to improve the quality of future ACF research, evaluation, and program support regarding kindergarten transitions. These instruments may be used in future ACF data collections, as well as in studies conducted by the broader field.
The study team conducted an exhaustive review of existing measures, reviewed relevant theoretical and empirical literature, and consulted with experts prior to and during the development of these instruments. Whereas other government surveys have asked questions about the kindergarten transition (e.g., Head Start Family and Child Experiences Survey [OMB # 0970-0151], Early Childhood Longitudinal Study – Kindergarten [OMB # 1850-0750]), they have not focused specifically on cross-system collaboration and coordination. Thus, they do not allow us to understand system alignment and coordination during the transition from Head Start to kindergarten, nor the program- and system-level policies and practices that may facilitate transitions. Additionally, previous surveys have not been fielded to this set of respondents, nor constructed to provide within- and across-system alignment for the sake of comparison. The cognitive interviews are necessary to assess administrators’ ability to interpret and respond reliably to these novel surveys that are intended to fill a gap in the knowledge base.
The development of these novel instruments will enable ACF and the field to better understand systems-level collaborations that support children’s transitions from Head Start to kindergarten, and the instruments may be used in future descriptive, correlational, quasi-experimental, and/or experimental studies.
A5. Impact on Small Businesses
No small businesses will be involved with this information collection. However, the Head Start programs that choose to participate in this research may be small, not-for-profit entities. The research team will reduce burden to the extent possible for these entities by scheduling data collection at convenient times and adhering to the burden estimates provided.
A6. Consequences of Less Frequent Collection
This is a one-time data collection.
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to submit a request to OMB for review of the overarching generic clearance for pre-testing activities. This notice was published on January 5, 2021; Volume 86, Number 2, page 308, and provided a thirty-day period for public comment. During the notice and comment period, no substantive comments were received. A second notice in the Federal Register announcing the agency’s submission of the overarching generic clearance for pre-testing activities for OMB’s review. This notice was published on May 21, 2021; Volume 86, Number 97, page 27624, and provided a thirty-day period for public comment. During the notice and comment period, no comments were received.
The research team convened a panel of three experts in November of 2021 with expertise in K-12 administration, Head Start administration, kindergarten transitions, and measure development. The experts reviewed the instruments both independently and collaboratively with the research team, engaging in a facilitated discussion of study questions, response options, and terminology. This discussion informed further survey item revision and the construction of cognitive interview protocols. See Exhibit 2 for the list of experts.
Exhibit 2: Head Start to Kindergarten Transitions Project Instrument Development Expert Panel
Name |
Affiliation |
Iheoma Iruka, PhD |
Frank Porter Graham Child Development Institute, University of North Carolina |
Amanda Stein, PhD |
Start Early |
Beth Rous, EdD |
University of Kentucky |
A9. Tokens of Appreciation
No tokens of appreciation are proposed, but we do intend to provide honoraria for the professionals’ time and expertise. See section A13 for information about honoraria.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally Identifiable Information
The research team will collect the names and contact information (phone numbers and/or email addresses) for administrators in each respondent category. The research team will collect this information through existing research team relationships and recruitment efforts (see Supporting Statement B, Part 2). Respondents’ names and contact information will not be connected to their interview or survey responses. They will be saved on a secure drive to which only individuals involved in the study will have access. Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier. Only unique identifiers will be used to track and manage transcriptions or interview recordings.
Assurances of Privacy
The research team will obtain informed consent from participants to ensure that they understand the nature of the research being conducted, the voluntary nature of their participation, their rights as participants, and that their information will be kept private to the extent permitted by law (see “Attachment A—Consent Language”). The interviewer will display the survey via screensharing, and the participant will respond to the consent script and survey questions as the interviewer displays them. Accordingly, the consent script will appear as an introductory screen on all virtual data collections and will be read aloud during cognitive interviews. Participants will be asked if they agree to participate in the cognitive interview prior to beginning the interview and whether they agree to be recorded while completing the process. Interviewers will answer questions as needed. If participants have further questions about the consent statement or other aspects of the study, they may call the NORC at the University of Chicago’s (NORC) principal investigators or the administrator of NORC’s IRB.
As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.
Data Security and Monitoring
ID numbers will be assigned to each respondent. Names or other identifiers will not be attached to the interview data. Contact information and interview data will not be combined into one dataset. Identifiable information will only be collected prior to the start of the cognitive interview and will be stored separately.
As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.
As specified in the contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.
A11. Sensitive Information 1
The goal of cognitive interviews is to understand how the survey respondents interpret survey items rather than collecting information about the respondents. Therefore, the study team expects there to be no sensitive information collected.
A12. Burden
Explanation of Burden Estimates
Burden for the screener was estimated at approximately 0.25 hours due to the limited and simple required items. This estimate includes the potential time required to schedule the screener. Burden for the protocols was estimated by considering the number of instruments and protocol questions and what can reasonably be completed during one interview. The total burden per participant is estimated at 1.75 hours, including the screener and interview protocol.
Estimated Annualized Cost to Respondents
The estimated annual cost for respondents is shown in Exhibit 3. The source for the mean hourly wage information for each respondent type is Bureau of Labor Statistics, Occupational Employment and Wages, May 2020.
For Head Start program administrators, the mean hourly wage of $26.41 was used, based on the wage for education administrators in preschool and childcare centers and programs (11-9031 Education Administrators, Preschool and Childcare Center/Program) https://www.bls.gov/oes/current/oes119031.htm.
For LEA administrators, the mean annual salary of $103,010 was used, resulting in an approximate mean hourly wage of $49.52 (divided by 52 weeks/40 hours) based on the wage for administrators that plan, direct, or coordinate the academic, administrative, or auxiliary activities of public or private elementary or secondary level schools (11-9032 Education Administrators, Elementary and Secondary School) https://www.bls.gov/oes/current/oes119032.htm
The total estimated annual cost amount is $4,538.30
Exhibit 3: Respondent Cost
Respondent |
Instrument |
No. of Respondents (total over request period) |
No. Of Responses per Respondent (total over request period) |
Avg. Burden per Response (in hours) |
Total Burden (in hours) |
Average Hourly Wage Rate |
Total Respondent Cost |
Head Start Program Administrator |
Initial Outreach and Screener Questions for Programs and Schools |
30 |
1 |
.25 |
7.5 |
$26.41 |
$198.08 |
Local Education Agency Administrator |
30 |
1 |
.25 |
7.5 |
$49.52 |
$371.40 |
|
Head Start Program Administrator |
Administrator Interview Protocol |
20 |
1 |
1.5 |
30 |
$26.41 |
$792.30 |
Local Education Agency Administrator |
20 |
1 |
1.5 |
30 |
$49.52 |
$1485.60 |
|
Totals: |
75 |
|
$2,847.38 |
||||
A13. Costs
Honoraria in the amount of $40 will be provided directly to participating administrators and staff as compensation for their expertise and time participating in the proposed interviews. The honoraria will be provided in the form of a gift card. Honoraria will be available for Head Start program administrators and LEA administrators after their participation in the proposed interview. These honoraria are appropriate for these professionals as they have the specialized knowledge and perspectives sought and are being requested to respond to the information collection in addition to their regular duties as administrators and directors. Given the level of demand already on their time, the proposed honoraria are important to ensure the respondents are compensated for the additional time to participate in the study. Additionally, this supports the recent Presidential Memorandum on Restoring Trust in Government through Scientific Integrity and Evidence-Based Policy Making, as well as the ACF Evaluation Policy, which both discuss the importance of community engagement and inclusion in research.
A14. Estimated Annualized Costs to the Federal Government
The total cost for the data collection activities under this current request will be $144,807. Costs include personnel labor hours and other direct costs such as honoraria for respondents, and interview transcription. Estimated costs to the federal government over the requested one-year approval period are as follows:
Exhibit 4: Estimated Costs
Cost Category |
Estimated Costs |
Field Work |
$108,722 |
Reporting |
$36,085 |
Total costs over the request period |
$144,807 |
A15. Reasons for changes in burden
This is a new information collection request under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (OMB #: 0970-0355).
A16. Timeline
Exhibit 5: Timeline
Milestone |
2021 |
2022 |
|||||||||
11 |
12 |
1 |
2 |
3 |
4 |
5 |
6 |
7 |
8 |
9 |
|
Outreach & Recruitment |
|
|
|
Recruitment |
|
|
|
|
|
|
|
Data Collection |
|
|
|
|
Interviews |
|
|
|
|
||
OMB |
OMB Approval |
|
|
|
|
|
|
|
|
||
Analysis |
|
|
|
|
|
|
Coding |
|
|
||
Reports |
|
|
|
|
|
Interim Reports |
Summary of Measure Development |
||||
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Attachment A—Consent Language
Attachment B—Recruitment Materials
Instruments
Instrument 1—HS Program Director Protocol 1
Instrument 2—HS Program Director Protocol 2
Instrument 3—HS Program Director Protocol 3
Instrument 4—LEA Administrator Protocol 1
Instrument 5—LEA Administrator Protocol 2
Instrument 6—LEA Administrator Protocol 3
Instrument 7—Screener Questions
1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2024-07-25 |