School Readiness Goals_PartB_revised 7_24

School Readiness Goals and Head Start Program Functioning

OMB Information Collection Request

New Collection

Supporting Statement

Part B

April 2013, Revised July 2013

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

7^th Floor, West Aerospace Building

370 L’Enfant Promenade, SW

Washington, D.C. 20447

Jennifer Brooks and Mary Mueggenborg

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for Collection of Information 3

B3. Methods to Maximize Response Rates and Deal with Nonresponse 5

B4. Tests of Procedures or Methods to be Undertaken 7

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 8

B1. Respondent Universe and Sampling Methods

The universe for the Head Start School Readiness Goals study includes 1,390 grantees operating a Head Start program, an Early Head Start program, or both during the 2011/2012 school year (provided they are still in operation during the 2013/2014 school year). Data on the universe of grantees comes from the Head Start Program Information Report (PIR).

Three groups were excluded from the universe for the telephone survey and site visits:

• 30 interim grants (programs temporarily operated by a national contractor until a replacement grantee is identified) were excluded because they are unlikely to be representative of typical Head Start grantees.

• 26 Migrant and Seasonal Head Start programs were excluded because of resource constraints, and because the nature of these programs (e.g., the shorter time for service delivery) means that the process of setting and using school readiness goals differs from the process in other grantees.

• 150 grants classified in Head Start Region XI (American Indian and Alaskan Native (AIAN) tribal grantees) were excluded from the telephone survey because of resource constraints: adequately studying the differences between them and grantees in Regions I-X would require allocating more sample size to them than would be practical with a total sample size of 90 grantees and a desire to understand differences across multiple dimensions.

This latter group, the 150 AIAN grantees, is the universe for a third component of data collection, the collection of qualitative information through telephone interviews with a small number of AIAN Head Start program directors, as detailed below.

A purposive sample of 90 grantees from the proposed universe for the main study will be selected for the telephone survey. The sample will include three main strata: grantees only operating Early Head Start programs (15 EHS-only cases), grantees only operating Head Start programs (25 HS-only cases), and grantees operating both Early Head Start and Head Start programs (50 HS/EHS cases). Half of the 50 cases in the HS/EHS stratum will be asked questions about their Head Start program and half will be asked questions about their Early Head Start program (with random selection of which are asked about Head Start and which are asked about Early Head Start). As a result, the analytical sample will consist of 15 Early Head Start programs, 25 Head Start only programs, 25 Early Head Start programs in combined settings, and 25 Head Start programs in combined settings. This sampling approach reflects modest oversampling of EHS only programs (which represent 14 percent of the universe) to ensure sufficient sample size for analysis. It also reflects modest oversampling of combined HS/EHS grantees (47 percent of grantees) to allow sufficient sample size of both Head Start and Early Head Start programs in those grantees.

As described in Part A of this supporting statement, the objective of this research is to collect information from Head Start and Early Head Start programs on a topic about which relatively little is known due to the new nature of the school readiness goal requirement. The study also seeks to collect information about the full range of responses across the diverse set of grantees, for the purpose of ensuring that policy, technical assistance, and training are sensitive to the vast differences in grantee operational approaches and populations served. Thus, the study design centers on an exploratory approach that can effectively capture as much variation as possible and build our understanding to support future investigation, as opposed to an approach that measures incidence or tests hypotheses.

The purposive sampling approach reflects this exploratory emphasis and is designed to facilitate the collection of information from a maximally-diverse set of grantees. The set of characteristics that are expected to have a relationship with the research questions of interest will serve as secondary sampling strata and include:

• Program option

• Program size

• Presence of delegate agencies

• Grantee organization type

• Race/ethnicity of population served

• Primary language of population served

• Child assessment tools used

• ACF Region

• State school readiness context)

Thus, within each primary stratum (HS-only, EHS-only, and HS/EHS), grantees will be selected to ensure the sample includes a sufficient number of cases with each of the characteristics of interest. Because the total number of categories across the characteristics is greater than the sample size, multiple characteristics will be considered at the same time when selecting grantees for the sample. Distributions of sample members across strata and sub-strata will include a minimum of two sample members with a given sampling characteristic, with the remaining sample allocated roughly in proportion to the distribution of a particular characteristic in the universe, making adjustments in certain areas to increase the possibility of exploring a range of expected mediating and moderating factors between particular characteristics and grantee processes for setting and using school readiness goals.

Sampling across these characteristics is designed to ensure sample diversity. The plan is to analyze the data to assess whether there are differences between the 50 Head Start grantees and the 40 Early Head Start grantees, and, whether there are differences between the 25 Head Start only, the 15 Early Head Start only, the 25 Head Start programs in grantees that also operate Early Head Start and the 25 Early Head Start programs in grantees that also operate Head Start. The analysis will also examine some differences by grantees size (small, medium and large) and grantee organization type. With 90 grantees in the sample, there will not be sufficient size to examine many differences in the school readiness goal development process across ACF regions or other characteristics.

As the sample is selected, each sample member will have an associated “back-up case” with similar characteristics. Although a high participation rate (80 percent) is expected, this approach will help ensure that the overall sampling objectives are met even when original sample members are unable or unwilling to participate.

After conducting preliminary analyses of telephone survey data, the study team will purposively select a subset of programs to participate in a site visit. The research team will select six Head Start-only grantees and six combined Head Start-EHS grantees for a total of 12 sites. These 12 site visits will allow observations of school readiness goals in 18 program settings (six Head Start only, six Head Start programs from the combined sites, and six EHS programs from the combined sites). The distribution is shown in the table below.

	HS only	EHS only	HS in combined HS-EHS programs	EHS in combined HS-EHS programs	Total
Survey	25	15	25	25	90
Site Visit	6	0	6 combined HS-EHS programs		12

.

When selecting among the Head Start only, and among the combined HS-EHS programs, the goal will be to select programs that exhibit some range in their level of sophistication in setting school readiness goals and using school readiness data. We do not want to visit only those who are excelling in meeting the new requirements or those that are struggling. To implement this, the research team will analyze response to several survey items, such as the following:

• B6, which captures programs’ overall experiences setting school readiness goals to identify programs that had few, if any, challenges and programs that struggled with the process;

• B10, which captures the level of involvement of various stakeholders in setting the school readiness goals, to identify programs with high levels of involvement across stakeholders, concentrated involvement from particular groups, and low involvement across most stakeholders except those program leaders writing the goals;

• C2 and C14 to identify programs that have strong technological and analytic capabilities, those that are average in these areas, and those that are potentially in need of technological improvements or lack analytic capability. Similarly, we will select programs that have a designated data analyst or data manager on staff and those that do not. We see these as being important factors that will influence programs’ ability to successfully track and assess children’s school readiness data.

In addition, for Head Start-Early Head Start combination grantees, we will examine whether programs generally used a different process for setting the goals for the two age groups, and whether their goals differed substantially by child age group. If there is relative variation across grantees on these responses, we will include grantees with similar and different processes and goals by age group.

When selecting the final set of programs, it also will be important to consider a balance in terms of grantee organization type (so not all programs are of the same type) and the number of children enrolled, ensuring the inclusion of several programs identified as small, medium or large in size. The study team also will attempt to achieve relative geographic diversity and avoid concentration in any given ACF region, since regional oversight and technical assistance may influence adherence to the school readiness goals mandate.

Finally, the study team will select a very small group of four AIAN programs to participate in qualitative interviews about the process of setting and using school readiness goals. In selecting the four AIAN grantees, the study team will select grantees that operate Head Start programs; that is, the study will not examine how AIAN grantees are setting school readiness goals for Early Head Start programs.  Grantees will be selected for variation in size and geography, and to ensure that they have different tribal affiliations.

Given the diversity of experiences across the 150 AIAN programs, these interviews are not expected to capture the full variation in experiences across tribal grantees. However, they will provide valuable information to ACF regarding some of the types of experiences of AIAN grantees in meeting the school readiness goal requirements. During the study’s design phase, we found that two open-ended phone calls with grantees yielded helpful information for designing the telephone protocol, and are optimistic that four open-ended interviews with AIAN grantees will provide some information about the key empirical question of whether or not they face similar or different challenges to other grantees. While case studies with four AIAN programs will not provide information that will be generalizable to all AIAN Head Start programs, conversations with four programs will further inform the Office of Head Start about how AIAN programs are approaching this new mandate and to help the Office of Planning, Research and Evaluation think about next steps in terms of research with this important population.

B2. Procedures for Collection of Information

Data collection will occur through a telephone survey of 90 grantees, site visits to a subset of 12 surveyed grantees, and qualitative telephone interviews with 4 American Indian/Alaskan Native grantees.

Telephone Survey. Data will be collected through a telephone survey of 90 local Head Start or Early Head Start programs. One telephone survey will be conducted with each of the 90 grantees. The survey respondent may be the Head Start or Early Head Start program director, the education services manager or coordinator, an assistant director, or whoever has responsibility for setting and using school readiness goals in the local grantee.

After receiving a letter introducing the study, program directors will be contacted by phone to answer questions they have about the research and discuss their participation (Appendices A-1 and A-2). Directors who agree to their program’s participation in the telephone survey will be asked a series of questions designed to identify the most appropriate respondent in that program. These initial calls will conclude by scheduling the interviews at future time that is convenient for respondents (or establishing a procedure for scheduling with an alternative respondent) and requesting one copy of the program’s school readiness goals (if available) and one copy of the program’s organization chart (if available).

Verbal consent will be obtained from respondents at the beginning of each telephone survey (see instrument for the verbal informed consent language that will be used in the survey, in Appendix A-3). The same protocol will be used for all grantees; a skip pattern will be used to customize the protocol so that grantees that operate only one program (Head Start only or Early Head Start only) will not be asked questions that are only pertinent to grantees operating two programs (e.g., questions about possible differences in how the two programs set and use school readiness goals). Grantees that operate both Head Start and Early Head Start will be instructed to answer their questions for only one program (with random assignment so that approximately 25 programs respond to questions about school readiness goals in their Head Start program and 25 respond to questions about school readiness goals in their Early Head Start program). This procedure is followed so that grantees that operate both Head Start and Early Head Start programs will not have to answer the survey twice (once for each program). In the analysis plan, the study will whether combined grantees set and use school readiness goals separately from other grantees. That is, the study will explore how the school readiness goal setting and use vary among Early Head Start only programs, Head Start only programs, Early Head Start programs that also operate Head start, and Head Start programs that also operate Early Head Start.

Each telephone survey is expected to last approximately 45 minutes. To facilitate data collection, entry, and management, interview protocols will be programmed into a web application such as Checkbox that interviewers will access on a secure server. This application will prompt interviewers through the protocol and also serve as the mechanism by which telephone interviewers record responses to both closed-ended and open-ended survey items. All interviewers will be trained on the protocol by the Principal Investigators before the interviews. During interviewer training, the interviewers will be walked through the protocol and discuss the purpose of each item, examples of desired responses, and appropriate probes to use. In addition, before conducting each survey, researchers will review the documents provided by the program and data on key program characteristics from the Head Start Program Information Report (PIR), in order to familiarize themselves with the program in advance of the scheduled call.

Site Visits. Further qualitative data will be collected through site visits. During the month of December 2013, the contractor will examine preliminary data from the telephone survey and select 12 programs for site visits based on program characteristics and experiences setting their school readiness goals. Letters will be sent to those programs to invite them to participate (Appendix B-1), and follow-up phone calls will be made to further recruit programs and schedule the site visits for the months of January and February 2014.

A team of two researchers will visit each of the 12 selected sites. Site visits will be 1.5 days long in 6 selected sites that operate only Head Start and 2 days long in 6 selected sites that operate both Head Start and Early Head Start. The site visit team will work with the program director or designee to determine the best methods for recruiting participants in each of the targeted respondent groups, including program managers and staff, Head Start and Early Head Start parents, a member of the governing body, members of the policy council, and the liaison with the local educational agency.

During the site visits, researchers will conduct 90-minute follow-up interviews with the key staff who responded to the telephone survey (namely, Head Start and Early Head Start program directors and education services managers), and one-hour interviews with other managers and coordinators who were not surveyed (Appendices B-2 and B-3). Some managers will be interviewed in a small group interview (e.g., program coordinators/specialists for health/nutrition, disabilities/special services, family services, or other specialty area). Others are more likely to be interviewed in a one-on-one interview (e.g., data coordinator if the position exists). Education services managers/coordinators could be interviewed as well, if not already interviewed in the 90-minute interview.

In addition, researchers will conduct small group interviews with program staff who work directly with children and families (i.e., teachers and home visitors); such interviews are expected to last approximately one hour (Appendix B-4). Both Head Start and Early Head Start program staff will be interviewed in sites that have both programs. In each site, researchers will also hold a one-hour, small group interview with members of the local policy council and on-on-one interviews with a member of the Head Start program governing body and a representative from the local education agency (Appendices B-5 and B-6).

Finally, researchers will interview two Head Start parents in each site and two Early Head Start parents in sites with EHS programs (Appendix B-7). Parental engagement is an important component of the school readiness goals requirement—and of the Head Start program in general. Programs are mandated to seek parental input and involve parents in setting school readiness goals, as well as to communicate progress towards goals with parents. Obtaining information on parental perspectives on school readiness goals, their level of involvement in setting goals, and their communication with program staff is useful in helping us understand how programs are approaching this requirement. The study team plans to triangulate information obtained from the various respondents, including program leaders, staff, and parents.

Parents who serve on the policy council and parents who are staff will be excluded in the selection for interviewing parents, since staff and the policy council members will be interviewed separately concerning their role and perspectives in that given position (and not as parents). Parents will be selected with children in different classrooms, unless the program only has one classroom, and the selection will include mothers and fathers. Because the intention is not to test parental knowledge of the goals and what they know about them, but rather we want to learn from parents who may have been involved in the goal-setting process, we do not see any conflict with having the program director or other staff member help with identifying potential parent participants.

The interviews with parents are expected to last 45 minutes and will be conducted individually in either English or Spanish, depending on the preferred language of the parent, which will be determined in advance of the site visit when scheduling. All site visit interview respondents will be asked to sign an informed consent form prior to the interview. (Appendices B-8 and B-9).

Researchers also will collect readily available secondary documents from the visited programs, such as data reports illustrating how the program shares information on school readiness goals and progress with staff, parents, governing bodies and other stakeholders. Interviewers will request these materials from program directors and staff at the end of the interviews. Collection of secondary documents will not impose burden on programs.

The interviews will all use discussion guides with key topics and open-ended questions rather than close-ended questions (i.e. rigidly specified and directly quantifiable questions). This approach is the best data collection method for understanding in depth the process for setting school readiness, and the ways in which programs use school readiness goals and related data to inform their program planning. The approach will allow flexibility in adapting the discussion guide to capture key aspects of the process across diverse programs. Two site visitors will conduct the site visits as a team, with the senior researcher leading most of the interviews, and junior staff taking detailed notes as close to verbatim as possible. With the permission of the respondent, the interviews will be recorded, solely for the purpose of editing and correcting the notes and creating a targeted transcription with key responses. The researchers will be trained to mark in their notes when a key statement is made so that quotes can later be checked for accuracy with the recording.

Telephone Interviews with AIAN Grantees. Finally, the research team also will conduct qualitative, open-ended telephone phone interviews with 4 directors of AIAN Head Start programs, in order to gather exploratory information about the how American Indian and Alaskan Native grantees set and use school readiness goals. After receiving a letter introducing the study, tribal Head Start program directors will be contacted by phone to answer questions they have about the research and discuss their participation. Directors who agree to participate in a telephone interview will be interviewed by Monica Rohacek, one of the project co-Principal Investigators. Verbal consent will be requested from participants at the beginning of each telephone interview protocol. The interview will be guided by an interview protocol designed for this purpose (Appendix C).

All data and related materials will be stored in accordance with the data security plan, as approved by the contractor’s IRB.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

The contractor anticipates a relatively high response rate (80 percent) in terms of contacted grantees being willing to participate in the telephone survey. This estimate is based on the contractor’s past experience in recruiting Head Start and other early childhood programs for similar research. Similarly high participation rates are expected for the site visits. Furthermore, the scheduling of both phone survey and site visits is flexible and can, to a considerable degree, be scheduled in accordance with the needs and circumstances of potential respondents.

Dealing with Nonresponse

As the sample is selected, each primary sample member will have an associated “back-up case” with similar characteristics that will be contacted in the event the initial sample member refuses to participate.. The algorithm used to select backup respondents and prioritize program characteristics to ensure diverse coverage will be the same as the procedure for selecting primary respondents. This approach will help ensure that the overall goal of achieving diversity across important criteria through purposive sampling is met if original sample programs are unable or unwilling to participate. If a sampled program declines to participate in the research, the team will discuss the case, the concerns the site has about participating, and brainstorm options for addressing the site’s concerns. If it is ultimately determined that a selected site cannot or will not participate in the research, the contractor will reach out to the pre-identified back-up case.

Maximizing Response Rates

To encourage Head Start programs to participate in the research, the contractor has designed outreach materials that succinctly describe the research and its value and the contractor will implement a systematic plan to encourage participation in the study. The first contact with the 90 sampled sites will be a letter addressed to the Head Start director, introducing the study, inviting participation, and indicating someone from the research team will contact them soon to further discuss their participation. The letter will be followed by a phone call to each program director to discuss any questions he or she may have about the research and to identify the appropriate survey respondent. If needed, interviewers will subsequently communicate via e-mail or phone to schedule the interview. To further support a high survey completion rate, $25 will be offered to each survey respondent, as a token of appreciation.

At the end of each telephone survey with Head Start program directors or managers, the interviewer will describe the next steps in the study, including plans to conduct site visits with a small number of grantees participating in the telephone survey. The interviewer will explain that the site visits offer an opportunity for a larger number of program stakeholders to share their experiences with setting and using school readiness goals in greater depth, and ask respondents if they might be willing to participate. Experience suggests that most respondents indicate a willingness to be contacted regarding this type of follow-up data collection, particularly if researchers emphasize that agreeing to be contacted does not mean that they will have to participate in the site visit.

Site recruitment procedures for the site visits will be is similar to what is planned for the telephone interviews. Selected sites will initially be contacted by e-mail, followed by a telephone call to discuss the possibility of participation. To begin building rapport, staff who will be conducting the site visits will conduct these outreach calls. The site visit team will work with the program director or designee to determine the best methods for recruiting participants in each of the targeted respondent groups, including program managers and staff, Head Start and Early Head Start parents, a member of the governing body, members of the policy council, and the liaison with the local educational agency. Because roles and titles may vary by grantee, marketing and recruitment materials will explicitly define the roles and responsibilities of the individuals with whom the research team would like to conduct interviews rather than rely on titles. As a token of appreciation, parent interview participants will be offered $25.

Finally, once programs are selected, the researchers will take a number of steps to minimize the burden to respondents and maximize response rates. First, both the telephone surveys and the site visits will be scheduled in a manner that allows respondents to identify the most convenient time for the survey or visit within the study timeframe. Second, flexibility will be exercised in adjusting the specific order, timing, and location of the on-site interviews to meet the respondents’ needs.

B4. Tests of Procedures or Methods to be Undertaken

The telephone interview guides were formally pre-tested with three Head Start programs. The first respondent was an Early Head Start program director; the second pre-test involved two respondents (the program director and education services manager of a program that operated both Head Start and Early Head Start programs); and the third pre-test involved the education services manager of a Head Start program.

During the first pre-test, the full protocol was not completed because the respondent provided lengthy qualitative responses to many of the early questions. Between the first and second pretests, the introductory letter was revised to refer to a telephone survey, rather than telephone interview, and small adjustments were also made to the introductory questions in the protocol, to discourage long responses to close-ended questions.

The second and third pre-tests were completed successfully. Each respondent was debriefed following their interview to provide feedback on the content and order of questions. Items that were identified as confusing or repetitive were edited and streamlined.

In addition, several site visit protocols were tested through a practice site visit to a program that operates both Head Start and Early Head Start (the program that was the second respondent to the telephone pre-test). Specifically, pre-tests were conducted with 4 of the 6 site visit interview guides, including the interview guides for program directors and managers; other services managers, coordinators and specialists; front-line staff; and parents (all but the interview guides for the policy council/governing body and local education agency). Each respondent was debriefed following their interview to provide feedback on the content, clarity, and order of questions. Adjustments were made to several items to improve comprehension, flow, and timing.

The three programs that were in the pre-test will not be selected again for inclusion in the study sample of 90 programs. Because the pretests were conducted with fewer than 10 individuals, they were not included in the estimates of burden.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The study design plan and data collection protocols were developed by the following project staff at the Urban Institute:

• Julia Isaacs, Project Director

• Monica Rohacek, Co-Principal Investigator, Lead on Site Selection and Co-Lead on Study Design/Measurement

• Heather Sandstrom, Co-Principal Investigator, Co-Lead on Study Design/Measurement and Lead on Data Analysis

This leadership team brings extensive experience designing and implementing complex, mixed-methods research studies, expert knowledge of Head Start/Early Head Start and other early childhood programs, and extensive field and survey research experience and training, including overall design, instrument design, survey and qualitative interviewing, and analysis of quantitative and qualitative data. All three will engage in data collection and analysis, and will be joined by additional researchers from the Urban Institute, including Teresa Derrick-Mills, who has extensive field and survey research experience and training and also brings additional performance management experience to the team.

In developing the study design and data collection protocols, the study team consulted with:

• Rachel Chazan-Cohen, Associate Professor, George Mason University

• Katherine Magnuson, Associate Professor, University of Wisconsin-Madison

• Timothy Triplett, Senior Survey Methodologist, Urban Institute.

The Federal project officers for this project are Jennifer Brooks and Mary Mueggenborg.