Memo - OMB - ECCS Response to Comments

memo

To:	OMB, OIRA
From:	HRSA
Date:	June 4, 2010
Re:	Response to OMB Comments on ICR: Evaluation of the State Early Childhood Comprehensive Systems Grant Program (201004-0915-001).

Revised materials in response to OMB’s comments are attached. These comments address the Evaluation of the State Early Childhood Comprehensive Systems Grant Program (201004-0915-001). HRSA is happy to make additional changes as necessary.

Part A

Comment: Please change wording to "towards addressing" (or similar) to clarify that this is an implementation evaluation assessing the implementation of activities to support these goals, rather than an impact evaluation of success in meeting specific objectives/outcomes. Please insert "implementation of" before "grantees" to clarify that this is an implementation, rather than an impact, evaluation.

Response: The suggested changes were made in order to clarify that this is an implementation evaluation assessing the implementation of activities to support these goals, rather than an impact evaluation of success in meeting specific objectives/outcomes.

Comment: For our own information, who requires these individuals to participate? Is it a statutory requirement for these individuals to take part in evaluation efforts to receive funding?

Response: We clarified that the “required participants” are not formally required to participate by changing our description of them to specified participants. ECCS grantees will be asked to request the participation of these partners in the survey, but there is no statutory or other requirement mandating their participation.

Comment: As a general note, for our review and information, it is helpful to have evaluation questions mapped onto the specific survey/interview questions (or even just to the specific instruments) used to address them. (Can do for future ICRs - Not necessary to go back and include here.)

Response: We changed the description of key evaluation question 4 to reflect that the evaluation is designed to compile whatever data grantees have collected on improvements in early childhood health and development outcomes.

Comment: How will health outcomes be measured?

Response: We revised the wording to reflect that this is an implementation evaluation and not an outcome evaluation (page 7).

Comment: How is "success" being measured? How will the influence of HRSA/MCHB support specifically be parsed out when evaluating grantees' progress?

Response: This was revised to discuss progress toward goals rather than success and to indicate that the study will measure grantee and partner perceptions of Federal support. Additional details were provided on the types of Federal support that has been provided and that are being asked about as part of the study (page 7).

Comment: Please state how participants' data will be protected, and whether or not HRSA is assuring participants of confidentiality (with statutory citation if confidentiality applies). Please include consent forms. Please include PRA/burden statement on all data collection instruments (and direct interviewer to read aloud in interview situations).

Response: We have elaborated on issues related to confidentiality and have incorporated Paperwork Reduction Act and burden statements into each of the interviews. It should be noted that the respondents are responding to a survey in the capacity of their professional positions, not as private citizens (pages 8-9).

Comment: Is this report required by statute?

Response: The report is not required by statue.

Comment: What types of progress indicators may be collected?

Response: The section was revised to indicate that the data that is analyzed will come from the survey and interview questions that focus on the seven elements of system building identified in Section 1 of Part A.

Part B

Comment: Earlier, it was stated that no personally identifiable information was collected - If there is a way that a participants' responses can be linked to their contact information, please revise "confidentiality" section as appropriate with collection and protection of respondents' information explained. Please address information protection for each mode of data collection in the confidentiality section as well.

Response: The confidentiality section has been revised to explain the procedures thoroughly. Please note that the contact information is publicly available (it is only their work contact information). No personal medical information or other private information is being collected (page 2 of Part B supporting statement).

Comment: Please briefly explain how these methods will be used to answer the "key evaluation questions" (p. 5-6).

Response: Once surveys from sites have been submitted and approved, the survey manager will provide a copy of the SQL data file from the survey Website to the contractor. The analysis team will then create a fully labeled data file in SAS to analyze the descriptive quantitative and categorical responses. The SAS statistical program version 9.2 will be used to organize and conduct analysis of data from the two Web-based surveys. Descriptive analysis of the survey data will include calculation of frequencies, means, and medians. This will provide summary information that addresses the key evaluation question concerning the structure and function of the grantees’ State Teams and initiatives, how the initiatives are addressing the key elements of early childhood system building, and progress being made on interagency collaboration and integration. Differences between groups of grantees, such as large versus small states or those with higher versus lower unemployment, will also be explored using chi-square tests for categorical variables. This will help provide information on how contextual factors may have affected implementation of the ECCS initiative and progress on early childhood system building. The influence of contextual factors on the implementation of the initiative is one of the sub-questions under each of the main evaluation questions (Page 2 of Part B supporting statement).

Comment: Please include consent form with ICR package.

Response: The consent to be recorded has been explicitly incorporated into the revised telephone interview instrument which is included as Attachment C.

Comment: Analyzed how (can be brief)? Please tie back to key evaluation questions - Which questions will these data help to answer? How will these data be used to answer them?

Response: HRSA seeks to determine if grantees with particular characteristics (e.g. coordinators with long tenures) are more likely to give certain types of responses. Throughout the analysis of both the survey and interview responses, the characteristics that could explain similarities and differences between grantees’ perceptions of success, weaknesses, supports, barriers, and impacts of their ECCS activities will be analyzed. Analysis will consist of grouping responses by contextual variables such as state capacity indicators (e.g., state general revenue per capita) or indicators of the economic climate (e.g., where the state ranks on unemployment) in order to see if there are any patterns in the types of responses by contextual variable. These data will provide information about how contextual factors have affected the key areas identified in the evaluation questions. The influence of contextual factors on the implementation of the initiative is one of the sub-questions under each of the main evaluation questions.

Comment: Please submit final version.

Response: The final form is attached. Please note, the drafts were pending any changes from the OMB process, otherwise in near-final form.

Comment: What are the outcome indicators used in this study?

Response: Additional information was added to the section. The indicators are from the Title V Information System (OMB #0915-0172) or ones that the states have decided to collect on their own to measure child and family well-being.

Comments: Please explain how responses will be analyzed. Please tie analyses back to key evaluation questions - Which questions were these data intended to help answer? How will these analyses address these key questions?

Response: Additional information has been added to address the comments. The analysis will include whether the indicator has been trending in a particular direction or not and what types of indicators the grantees have chosen (e.g., child care, health, economic well-being). Explanations for how grantees expect system-building strategies to affect indicators of child and family well-being will be categorized and coded and a descriptive analysis will be conducted. The analysis will include the extent to which grantees make reference to evidence-based practices or research in positing why a strategy should affect an indicator. These analyses will address evaluation question number 4, covering the types of evidence that grantees have collected on whether and how there ECCS initiatives are improving outcomes for children and families.