Attachment A - Pretesting Usage Report

Attachment b: use of pretesting generic CLEARANCE

(0970-0355) From 2015-2018

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

330 C Street, SW, 4^th Floor

Washington, DC 20201

In March 2015, OMB approved ACF’s request to renew the generic information collection for pre-testing of evaluation surveys (0970-0355). This report describes the use of the generic IC over the three years of approval, including the number of hours used, as well as the nature and results of the activities completed under this generic clearance.

The renewal of the generic IC was approved for three years, during which time ACF requested three generic ICs for pretesting and one change request to increase the number of respondents under a previously approved GenIC. The use of the pre-testing generic IC has been beneficial to the development of instruments for these research studies. Over the three years, ACF collected information from 1,573 respondents, for a total of 1,178 hours of burden. During the previous three years of clearance, information was collected from 3,605 respondents requiring 1,336 hours of burden. The continued use of the generic from 2015-2017 is indicative of the usefulness of pretesting our surveys and procedures.

Date	Project	Respondents	Burden Hours
3/26/2015	Permanency Innovations Initiative - California Partnership for Permanency (CAPP) - Change Request to increase number of respondents	49	22
10/2/2015	Assessing Early Childhood Teachers' Use of Child Progress Monitoring to Individualize Teaching Practices	72	36
10/14/2015	Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study, Phase 1	576	676
11/18/2015	Pre-testing of Evaluation Surveys: An Examination of the Intersection of Domestic Human Trafficking with Child Welfare and Runaway and Homeless Youth Programs	600	168
9/13/2017	Pretest of the Annual Survey of Refugees	276	276
Total over 3 Years		1573	1,178

Example uses of the Pre-testing Generic Clearance

Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study, Phase 1

The Assessing the Implementation and Cost of High Quality Early Care and Education (ECE-ICHQ) project is creating an instrument that will produce measures of the implementation and costs of the key functions that support quality in center-based Early Care and Education (ECE) serving children from birth to age 5.¹ The specific goals of this two-phase comparative multi-case study are (1) to test and refine a mixed-methods approach to identify the implementation activities and costs of key functions within ECE centers serving children from birth to age 5 and (2) to produce data for creating measures of implementation and costs.

The ECE-ICHQ study team began work by developing a conceptual framework; conducting a review of the literature (Caronongan et al., 2016); consulting a technical expert panel (TEP); and collecting and summarizing findings from Phase 1 of the study. Phase 1 was completed under the pre-testing generic clearance (0970-0355). This work included thoroughly testing data collection tools and methods, conducting cognitive interviews to obtain feedback from respondents about the tools, and refining and reducing the tools for this next phase. Data collection under Phase 1 informed the development of Phase 2, which is currently in the field (0970-0499).

The generic clearance was essential in allowing the team to refine the data collection tools before going into the field with a larger sample in Phase 2. Based on findings from Phase 1, we shortened the tools (we identified redundancies and non-essential items that could be deleted), we clarified terms, and we restructured the tools to be more user friendly (we developed questionnaire rubrics instead of interview scripts allowing for easier coding of responses following the interviews). These changes ultimately reduced burden on participants as well as burden on field staff scoring and reporting responses. Additionally, based on findings from Phase 1, recruitment of centers was modified given our difficulty initially recruiting diverse centers into the study.

Family-Teacher/Provider Relationship Quality measure development (FTPRQ) project

The Family-Teacher/Provider Relationship Quality measure development (FTPRQ) project created a set of new measures to assess the quality of the relationship between families of young children (ages 0-5 years) and the child’s teacher or child care provider. These relationships are an important component of family engagement in early childhood programs, but no appropriate measure existed.

Both the formative generic clearance (0970-0356) and the pre-testing generic clearance (0970-0355) were integral in developing these measures. Under the formative generic clearance, ACF/OPRE conducted focus groups with families and teachers/providers. These focus groups were essential for ensuring that our measures were based on a realistic conceptual model and that we were using terminology and language that families and teachers/providers would understand. After developing initial drafts of measures, which were based in part on the results of the focus groups as well as an extensive review of the literature and pre-existing related measures, ACF received permission conduct cognitive interviews under the pre-testing generic clearance. Three rounds of cognitive interviews were conducted, which further enabled us to test and refine our measures, so that they were ready for testing in pilot and field tests (conducted under OMB clearance 0970-0420). The thoroughly tested and final measures were released in November 2014 and have been very well-received by the field.

Caronongan, P., G. Kirby, K. Boller, E. Modlin, J. Lyskawa. “Assessing the Implementation and Cost of High Quality Early Care and Education: A Review of Literature.” OPRE Report 2016-31. Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation, 2016.

1 The ECE-ICHQ conceptual framework includes six key functions: (1) instruction and caregiving; (2) workforce development; (3) leadership activities, planning, and evaluation; (4) center administration; (5) child and family support; and (6) instructional planning, coordination, and child assessment.