Health Resources and Services Administration
Maternal and Child Health Bureau
Evaluation of the State Early Childhood Comprehensive Systems
Grant Program
1. Respondent Universe and Sampling Methods
Primary data collection for which OMB approval is being sought includes the following activities:
Web-based Survey of ECCS Coordinators. This study is surveying the universe of 52 ECCS grantees. Surveys will be completed by ECCS Coordinators, who will likely have the most comprehensive information about various aspects of ECCS-related activities.
Web-based Survey of selected ECCS State Team members. Five members of each State Team will be surveyed. The five team members will consist of two required participants for each state and three chosen by the ECCS Coordinator. The required participants are the State Head Start Collaboration Coordinator and the State Community Based Child Abuse Prevention (CBCAP) Initiative Coordinator. These participants are required because they are charged with coordinating key aspects of the early childhood system and increased collaboration between these leaders and ECCS Coordinators has been a key focus of the ECCS initiative over the past two years. The ECCS Coordinator will be asked to identify three additional State Team members to participate in the survey. These will be State Team members who are most knowledgeable about the initiative and can best report on the functioning of the State Team and the nature of the Team’s activities.
Telephone Interviews. Telephone interviews will be conducted with all 52 ECCS Coordinators.
Web-based Data Collection Form for Indicators. All 52 ECCS Coordinators will be asked to enter data into the web-based form.
2. Information Collection Procedures
These strategies are part of a mixed-method (phone interviews and data collection activities) evaluation design that addresses the complex nature of systems-building and that will ensure all of the major components of each State’s early childhood system are addressed. The design will therefore require multiple contacts with each grantee and multiple contacts with the same individuals at each site. These multiple contacts are necessary to capture different types of information and inform subsequent stages of the assessment.
In the first phase of the study, as much data as possible from grantees’ administrative documents will be abstracted to establish a baseline understanding of each State’s ECCS initiative. In the following phases, primary and secondary data will be collected across all sites by administering two web-based surveys, conducting telephone interviews, and a web-based data collection form for indicators.
Web-based Surveys
The purpose of the surveys is to gather data about the nature and extent of planning and implementation of the core components of ECCS and key elements of early childhood systems-building. The draft survey instruments may be found in Attachments A and B. The surveys consist of primarily close-ended questions with categorical response options.
To limit cost to the government and minimize burden on respondents, surveys will be conducted via a secure web-site. Recruitment for the surveys will occur through an initial notification letter from HRSA/MCHB and follow-up contacts via email and telephone from the contractor. Respondents will be given a link to their respective web-based survey and detailed instructions on how to enter and submit their responses. The contractor will also offer technical assistance to respondents to help successfully complete the surveys if needed. Once a survey has been submitted, the contractor’s survey manager will review the survey file to confirm that it is complete and, if necessary, follow-up with the grantee to fill in any missing information. Once surveys from sites have been submitted and approved, the survey manager will provide a copy of the SQL data file from the survey Website to the contractor. The analysis team will then create a fully labeled data file in SAS to analyze the descriptive quantitative and categorical responses. The SAS statistical program version 9.2 will be used to organize and conduct analysis of data from the two Web-based surveys. Descriptive analysis of the survey data will include calculation of frequencies, means, and medians. This will provide summary information that addresses the key evaluation question concerning the structure and function of the grantees’ State Teams and initiatives, how the initiatives are addressing the key elements of early childhood system building, and progress being made on interagency collaboration and integration. Differences between groups of grantees, such as large versus small states or those with higher versus lower unemployment, will also be explored using chi-square tests for categorical variables. This will help provide information on how contextual factors may have affected implementation of the ECCS initiative and progress on early childhood system building. The influence of contextual factors on the implementation of the initiative is one of the sub-questions under each of the main evaluation questions.
Telephone Interviews
The purpose of the telephone interviews is to gather more comprehensive information from all of the grantees about ECCS program planning, implementation, support from the Federal Partners Workgroup, and outcomes. The draft interview guide is located in Attachment C. The semi-structured interview guide consists of open-ended questions and a series of probes to elicit more detailed responses.
To limit cost to the government and minimize burden on respondents, the interviews will be conducted over the telephone. Recruitment for the survey will occur through an initial notification letter from HRSA/MCHB and follow-up contacts via telephone and email from the contractor. ECCS Coordinators will be asked to identify several possible dates and times to schedule the interviews; the contractor will make every effort to accommodate interviewees’ schedules. Prior to the scheduled interview date, interviewers will review the grantee profiles that were previously developed as part of the abstraction of administrative data and a summary of the grantees’ responses to the Web-based survey. This grantee-specific information will be used to familiarize interviewers with their assigned grantees’ activities and to pre-populate one section of the protocol with previously reported data.
The telephone interviews will be audio recorded pending the consent of the interviewees. During the interview, the interviewer will also take detailed typed or written notes using a standard template. Following each interview, interviewers will be required to finalize and submit their notes within 24–48 hours. Interviewers will be able to refer to audio recordings of the interviews to ensure that they have captured complete responses in their notes. Data from the interviews will be imported into the NVivo qualitative software program for content analysis. A comprehensive codebook will be created to guide the coding process. Coding of qualitative data in NVivo will entail classifying passages of interview notes by topic to determine major themes and patterns across States’ responses. The contractor will seek to determine if grantees with particular characteristics (e.g. coordinators with long tenures) are more likely to give certain types of responses. Throughout the analysis of both the survey and interview responses, the characteristics that could explain similarities and differences between grantees’ perceptions of success, weaknesses, supports, barriers, and impacts of their ECCS activities will be analyzed. Analysis will consist of grouping responses by contextual variables such as state capacity indicators (e.g., state general revenue per capita) or indicators of the economic climate (e.g., where the state ranks on unemployment) in order to see if there are any patterns in the types of responses by contextual variable. These data will provide information about how contextual factors have affected the key areas identified in the evaluation questions. The influence of contextual factors on the implementation of the initiative is one of the sub-questions under each of the main evaluation questions.
Web-based Data Collection Form for Indicators
The purpose of this form is to capture data on outcome indicators that ECCS State Teams have selected to track the effectiveness of their systems-building strategies, as well as a description of the rationale for why specific strategies will lead to measurable improvements in these outcome indicators. One of the requirements of the ECCS grant is that each initiative develops a list of population indicators that they are trying to improve. The purpose of this form is to collect information on how the grantees expect that changes will be made. The form is Attachment D. The instrument consists of a series of both closed- and open-ended responses. The burden for these indicators is approved through the Title V Information System, OMB #0915-0172.
Similar to the Web-based surveys, the indicator reporting form will also be administered through a secure web-site to limit cost to the government and minimize burden on respondents. Recruitment will also occur in similar manner as for the surveys, an initial notification letter from HRSA/MCHB and follow-up contacts via email and telephone from the contractor. Respondents will be given a link to the web-based data collection form and detailed instructions on how to enter and submit their responses. The contractor will also offer technical assistance to respondents to help successfully enter data if needed. Once all responses have been submitted, the contractor will review the outcome indicator file to confirm that it is complete and, if necessary, follow-up with the site to fill in any missing information. Once responses across all grantees have been submitted and approved, the reporting form manager will provide a copy of the Structured Query Language (SQL) data file from the website to the data analysts. The analysis team will then create a fully labeled data file in SAS to analyze the responses. The analysis will include whether the indicator has been trending in a particular direction or not and what types of indicators the grantees have chosen (e.g., child care, health, economic well-being). Explanations for how grantees expect system-building strategies to affect indicators of child and family well-being will be categorized and coded and a descriptive analysis will be conducted. The analysis will include the extent to which grantees make reference to evidence-based practices or research in positing why a strategy should affect an indicator. These analyses will address evaluation question number 4, covering the types of evidence that grantees have collected on whether and how there ECCS initiatives are improving outcomes for children and families.
Exhibit 5 below provides an estimated timeline of data collection activities:
Exhibit 5: Timeline of Data Collection Activities
Activity |
Estimated Timeline |
PHASE I DATA COLLECTION: OPTION YEAR 1 |
|
Web-based surveys |
1 to 3 months following OMB approval |
Telephone interviews |
3 to 5 months following OMB approval |
Indicator reporting form |
1 to 3 months following OMB approval |
PHASE II DATA COLLECTION: OPTION YEAR 3 |
|
Web-based surveys |
25 to 27 months following OMB approval |
Telephone interviews |
27 to 29 months following OMB approval |
Indicator reporting form |
25 to 27 months following OMB approval |
3. Methods to Maximize Response Rates
Gaining buy-in and cooperation from ECCS grantees are key requirements for this evaluation. For this data collection effort, we believe the nature of this respondent group—grantees that have received funding from HRSA/MCHB and who are interested in efforts to improve understanding of the implementation and effectiveness of the ECCS Grant Program—will increase their propensity to respond. It is important to note that the ECCS grantees are aware of this evaluation and are required to participate in the evaluation as part of their grant. However, a small proportion (less than 10 percent) of grantees may have difficulty complying with some components of the data collection process. Five of the grantees consist of jurisdictions other than states. These jurisdictions include Puerto Rico and the Republic of Palau where language barriers are sometimes a problem. In addition to Palau there are two other Pacific Islands, Guam and the Commonwealth of the Northern Marianas Islands, and time zone differences and more limited access to technology may pose a challenge for them. Every effort will be made to ensure full participation across all grantees (using methods described below). It is expected that there will be a response rate of at least 95 percent for each of the data collection methods.
Every effort will be made to meet or exceed the targeted response rate for data collection. A notification letter from the HRSA/MCHB Project Officer will be sent to sites to introduce the study, explain its importance to gaining a better understanding of ECCS, and encourage grantees to participate. Contractor staff will make a series of follow-up contacts with each site by telephone and email to reinforce the importance of participation, answer questions, and offer any support needed. In addition, the contractor will use the following proven methods to maximize participation and cooperation in the study:
Contractor staff will provide technical support if respondents encounter difficulties completing the Web-based survey or entering responses into the Web-based indicator reporting form.
Throughout the telephone interview recruitment process, staff will work with grantees to accommodate the schedules of all interviewees. Multiple staff will be available to conduct interviews increasing the likelihood that at least one contractor team member will be available at times specified by grantees. Also, interviewers will make themselves available in both the early morning and evening to accommodate sites in different time zones.
Telephone interviewees will be provided with the discussion questions in advance to allow them sufficient time to prepare.
All grantees and respondents will be given a realistic appraisal of the amount of contributions in time, information, and human resources that will be expected.
Paper copies of the survey and outcome indicator reporting form will be distributed to sites that prefer not to utilize the Web-based versions.
In previous projects involving similar data collection activities, these steps produced collegial relationships between the evaluation team and participants and contributed to high response rates.
4. Tests of Procedures
Two coordinators completed a pretest of the coordinator survey and interview and two ECCS State Team members completed the Team Member Survey. Based on the pretest revisions were made to the instruments that are detailed on page 8 in section 8. Once the web-based instruments are completed the operation of the instruments will be reviewed and tested by three contractor staff and the link will be shared with the government contract officer so HRSA/MCHB staff have an opportunity to review the forms. This process will address any programming issues related to the instruments.
5. Statistical Consultants
The following individuals contributed to the questionnaires and study design and will be involved in the interpretation and analysis of findings:
Namratha Swamy, PhD
Director of the Policy, Planning, and Evaluation Practice Area
Altarum Institute
1200 18th Street NW, Suite 700
Washington, DC 20036
(P) 202-828-5100
(F) 202-728-9469
Christopher Botsko, M.A.
Deputy Director
Policy, Planning, and Evaluation Practice Area
Co-Lead Women, Children, and Adolescents Subject Area
Altarum Institute
1200 18th Street NW, Suite 700
Washington, DC 20036
(202) 828-5100
Christopher.Botsko@altarum.org
Jodi Anthony, M.P.H.
Former Deputy Director, Strategic Analysis and Planning
Policy, Planning, and Evaluation Practice Area
Altarum Institute
1200 18th Street NW, Suite 700
Washington, DC 20036
(202) 828-5100
Amy Brown, M.P.H.
Senior Policy Associate
Policy, Planning, and Evaluation Practice Area
Altarum Institute
1200 18th Street NW, Suite 700
Washington, DC 20036
(202) 828-5100
Amy.Brown@altarum.org
Kara Rudolph, M.P.H.
Policy Associate
Policy, Planning, and Evaluation Practice Area
Altarum Institute
1200 18th Street NW, Suite 700
Washington, DC 20036
(202) 828-5100
Kara.Rudolph@altarum.org
The government project officer for this study is:
Joseph Zogby, Ph.D.
Public Health Analyst
Maternal and Child Health Bureau
Health Resources and Services Administration
5600 Fishers Lane
Parklawn Bldg., Room 18A-39
Rockville, Maryland 20857
(301) 443-4393
joseph.zogby@hrsa.hhs.gov
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Hrsa |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |