ccb EvalSubStratOMB2 mass change
ccb EvalSubStratOMB2 mass change.doc
Evaluation of Child Care Subsidy Strategies; Massachusetts, Illinois, and Washington
OMB: 0970-0306
Evaluation of Child Care Subsidy Strategies
Request for OMB Clearance: Massachusetts, Illinois, and Washington
January 2006
Prepared for
Ivelisse Martinez-Beck
Child Care Bureau
Administration for Children and Families, HHS
1250 Maryland Avenue, SW
Portals Building, Suite 800
Washington, DC 20024
Prepared by
Ann Collins
Charles Michalopoulos
Jean I. Layzer
Table of Contents
Summary i
Part A Justification 1
A1 Explanation of the Circumstances That Make the Collection of Information Necessary 1
a.Massachusetts……………………………………...…………………………………...3
b. Illinois……….…..………………………………………………..……………...…….3
c. Washington……….………………………………………………...………………….5
A2 How the Information Will Be Used, by Whom, and for What Purpose 6
a. Massachusetts………………………………………………………………………….6
b. Illinois………………………………………………………………………………...10
c. Washington…………………………………………………………………………...15
A3 Use of Improved Technology to Reduce Burden……………………………………….23
A4 Efforts to Avoid Duplication 24
A5 Efforts to Minimize Burden on Small Businesses or Other Small Entities 24
A6 Consequences if the Information is Not Collected 24
A7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5 (D) (2) 26
A8 Efforts to Consult with Persons Outside the Agency 26
A9 Payments to Respondents 27
A10 Assurances of Confidentiality Provided to Respondents 28
A11 Justification for Questions of a Sensitive Nature 291
A12 Estimates of Respondent Burden 29
A13 Estimates of the Cost Burden to Respondents 31
A14 Estimates of the Cost to the Federal Government 31
A 15 Reasons for Any Program Changes or Adjustments 32
A 16 Plans for Tabulation, Statistical Analysis and Publication 32
a. Massachusetts 32
b. Illinois………………………………………………………………………………..35
c. Washington…………………………………………………………………………..43
A17 Display of Expiration for OMB Approval 49
A18 Exception to the Certification Statement Identified in Item 19.0 of Form OMB 83-I 49
Part B Collection of Information Using Statistical Methods 50
B1 Sample Universe, Sampling Method and Expected Response Rates 150
a. Massachusetts 50
b. Illinois 51
c. Washington…………………………………………………………………………...52
B2 Data Collection Strategy 55
a. Massachusetts 55
b. Illinois 56
c. Washington 58
B3 Methods to Maximize Response Rates 162
B4 Tests of Procedures 64
B5 Individuals Consulted on the Statistical Aspects of the Design 65
References……………………………………………………………………………………………67
Summary
The Evaluation of Child Care Subsidy Strategies is designed to provide Federal, state, and local policymakers with information about the role of subsidy programs and policies in helping low-income families obtain and retain work and in improving outcomes for children. The goal of the study is to determine how differences in certain aspects of child care subsidy policies or quality-improvement efforts are related to outcomes for parents, children, and child care providers. The Evaluation of Child Care Subsidy Strategies consists of three studies, one in Massachusetts, one in Illinois, and one in Washington. OMB clearance is sought for each study site.
a. Massachusetts
The Massachusetts experiment will test the effects of a curriculum designed to address language development and the development of pre-literacy skills for very young children in family child care settings. Study participants are approximately 350 family child care providers who belong to one of 16 state-supported family child care networks and care for subsidized children. Each of these providers has approximately two children under the age of three in their homes. Half of the providers (i.e., the treatment group) will be randomly assigned to use Learningames, a research-based early childhood curriculum that can be easily adapted for use in family child care homes. These providers will be supported by network home visitors who are trained in the Learningames approach. The other half of the providers (i.e., the control group) will continue to offer care in their usual manner and be supported by the network’s standard training and technical assistance.
The major research questions include:
What is the effect of a research-based developmental curriculum (Learningames) on provider’s behavior and interactions with children on the language and literacy environment of the home?
What is the impact of the intervention on children’s language and pre-literacy skills?
The study will include an implementation analysis and an impact analysis. Data sources include observations of the family child care setting, at baseline and three other points in time, assessments of children taken at two points in time, questionnaires for providers at two points in time, and a one-time questionnaire for home visitors. Providers began using Learningames in the fall of 2005. The intervention and evaluation will be completed in the fall of 2007. Reports will be issued in 2008.
P
art
A
Justification
A1 Explanation of the Circumstances That Make the Collection of Information Necessary
Investment in child care by the Federal government and by individual states increased substantially in the years after the passage of the Personal Responsibility and Work Opportunity Reconciliation Act. The legislation provided greatly increased Federal resources to states to provide child care assistance, authorizing some funds specifically for child care and also allowing states to transfer Temporary Assistance for Needy Families (TANF) funds to the Federal child care program (and to spend TANF funds directly on child care subsidies). The Child Care and Development Fund (CCDF), created as part of the legislation, combined four of the existing child care funding programs into a single block grant to states, giving them much more flexibility to decide how child care funds should be expended. In FY 2004, Federal and state spending on child care totaled $9.4 billion: Federal CCDF funding (including TANF funds transferred into CCDF) reached $6.9 billion; state spending totaled $2.5 billion. In addition, direct TANF spending on child care was $1.4 billion. As a result of increased Federal funding combined with steady increases in the states’ contributions to the subsidy program, many more low-income families with working parents are able to receive help in paying for child care. In addition, the CCDF stipulates that states must set aside 4% of their CCDF funding for efforts to expand the supply or improve the quality of child care; many states allocate more than this minimum amount. States face a considerable challenge in trying to use their child care funds as effectively as possible, both to support parent’s employment and to improve child care quality to ensure children’s safety and enhance their development.
In related efforts, states are working to meet the goals of President Bush’s initiative, Good Start, Grow Smart, to enhance the school readiness of young children. Universal pre-kindergarten (UPK) is one of the strategies that many states use to meet the presidential mandate. Typically, states are implementing UPK through the existing system of schools and child care centers. However, much of the care for young children, especially infants and toddlers, is provided in family child care homes. States can and do use the CCDF quality-set aside funds to attempt to improve the quality of family child care, but they do so with scant information about the effectiveness of their efforts. The majority of the research available about efforts to enhance the school readiness of children from low-income families focuses on center-based early childhood programs that serve primarily preschool-age children. A research focus on center-based programs for three- and four-year old children does not reflect the widespread recognition that the very early years are also a critical period of development. A focus on the school readiness of children in center and pre-school classrooms leaves out the many children who are cared for in family child care, before they reach the age where they may be in settings supported by UPK. The work of Hart and Risley (1996), among others, suggests that efforts need to be made early in children’s lives to enrich their language environments because this are of development is key to school readiness and later success in school.
Furthermore, despite the substantial increases in funding for subsidies over the last seven years, more recently some states have faced budget constraints that have had an impact on the subsidy program. Regardless of their fiscal situation, states must still make choices about how to allocate resources and target subsidies to meet multiple objectives. They do so directly and indirectly through a host of decisions about child care policies and their implementation. These include: determining the level of state resources and matching requirements for counties (where applicable); setting eligibility guidelines and setting priorities for subsidies (including priorities attached to serving TANF vs. non-TANF families); deciding how and to what extent the availability of subsidies will be publicized; developing co-payment scales; and developing fee schedules and payments for providers. None of the child care research conducted over the past three decades has systematically examined the effectiveness of different child care subsidy policies or programs.
Recognizing the need for carefully-designed research that would provide useful information to states and communities, the Child Care Bureau and the Office for Planning, Research and Evaluation of the Administration for Children and Families (ACF) within the US Department of Health and Human Services commissioned a research effort designed to expand our knowledge about child care subsidies and quality-improvement efforts. In September 2001, a contract was awarded to Abt Associates Inc. to conduct a multi-site, multi-year Evaluation of Child Care Subsidy Strategies. To carry out the study, Abt Associates and its subcontractors—Manpower Demonstration Research Corporation (MDRC), Columbia University’s National Center for Children in Poverty (NCCP), and Moore & Associates, Inc.— worked closely with state and local partners in four sites to design and implement experimental studies that are tailored to their needs and interests, as well as the interests of policymakers in general.
The Evaluation of Child Care Subsidy Strategies will provide Federal, state, and local policymakers with information about the role of subsidy programs and policies in helping low-income families obtain and retain work and in improving outcomes for children. The goal of the study is to determine how differences in certain aspects of child care subsidy policies or quality-improvement efforts are related to outcomes for parents, children, and child care providers/caregivers. Outcomes of interest include the stability of parental employment and earnings, parent and child well-being (especially the development of children’s language development and literacy skills), availability of care, and child care quality. The study will address this goal through rigorous evaluation using a set of three random assignment experiments that will test aspects of subsidy policies or evaluate quality-improvement efforts. Three states and one locality have agreed to participate in the study—Illinois, Washington, Massachusetts, and Miami-Dade County in Florida. This request for OMB approval is for the experiments that will be conducted in Massachusetts, Illinois, and Washington.1
Authorization for the CCDF is part of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), Public Law 104-193 (42 USC 1305). Authorization for research related to the CCDF is part of the appropriations legislation for the Departments of Labor and Health and Human Services (PL 109-149).2
a. Massachusetts
In Massachusetts, many licensed family child care providers who accept subsidies for children in their care operate under the auspices of family child care networks (called “systems” by the state). There are more than 50 such networks in the state, varying greatly in size and geographic coverage. Networks in Massachusetts receive state funds to provide technical assistance to their member providers in part through regular visits by home visitors/mentors. These visits are intended to provide general support and information to providers to improve the quality of the care they offer. In 2003, the state developed family child care regulations that stipulated that all licensed family child care providers must use a developmental curriculum. However, little guidance has been offered about how to identify and select a curriculum, and state agency staff believes that the mandate has largely been ignored or overlooked, although mentors could, in principle, help providers do so. Further, little research, thus far, has focused on the linkages between the use of CCDF quality funds and state activities aimed at improving the quality of programs and of the care children experience.
To address the need for tested approaches that enhance children’s language development and pre-literacy skills, the Commonwealth of Massachusetts has identified a strong research-based curriculum—Learningames— that can easily be adapted for use in a family child care setting. The state is interested in experimentally testing the effectiveness of Learningames in family child care homes linked to family child care networks. The state views the proposed experimental test of Learningames as an opportunity to provide both system administrators and providers with evidence-based guidance on how to address the state mandate as well as complement their current efforts on a new UPK initiative, by focusing on younger children in family child care settings. The need for such a study is clear. In Massachusetts the most recent research on the quality of child care (Marshall, et al, 2001 and 2003) indicates that, on average, the quality of care in family child care is considerably lower that of center-based programs. Indeed, only 30% of family child care homes met accepted standards for good quality child care.
The evaluation in Massachusetts includes 350 child care providers who are members of 16 family child care networks. Each provider is caring for at least two children under the age of 26 months at the outset of the study. Half of the providers were assigned to receive training on the implementation of Learningames, materials and mentoring support; the remaining providers will continue to receive the usual ongoing training and support. Recruitment for the study began in the spring of 2005 and random assignment was completed by June 30th. Outcomes of interest are changes in caregiver behavior and changes in children’s language development and literacy skills. The data collection includes observations of the family child care home environment and child assessments. It will occur at three points in time: in June 2006 (after receiving OMB approval); in January 2007; and January/February 2008. In addition, a short caregiver survey will be administered twice; in June 2006 and January 2008. Baseline data will include observations of homes and extant child assessment data collected by family child care network staff. Finally, home visitors will be asked to complete a brief questionnaire once, in June 2006.
A2 How the Information Will Be Used, by Whom, and for What Purpose
a. Massachusetts
In cooperation with the State Lead Child Care Agency, the Massachusetts experiment will test the effects of the Learningames curricula to address language development and the development of pre-literacy skills for very young children in family child care settings. Learningames is an approach built on evidence that children learn best in one-on-one interactions with a caregiver who is nurturing and responsive to the child and who also provides rich language stimulation. If caregivers in home-based settings are given appropriate tools, the intimate interactions that are possible in home-based care provide an ideal opportunity for promoting children’s oral language, communication skills, and early phonological awareness. The curriculum is well suited to support the caregiving practices that have been linked to improving and providing good quality care, a goal of the CCDF. An earlier version of Learningames was first used in the highly-successful Abecedarian program, as an approach for teachers to use with children and their parents. It has been adapted for use in Even Start, a two-generation family literacy program, both with the teachers in the center-based early childhood programs and with parents to use at home with their children from birth to eight years. In its revised form, Learningames expands the number of games and activities to reflect the newest research on the importance of language and emergent literacy. It is appropriate for use with parents, family child care providers and early childhood teachers.
What makes the Learningames curriculum suitable for family child care is the fact that it is less concerned with a particular set of lessons and content than with the relationship between the caregiver and the children. It focuses on teaching caregivers to be (a) responsive and nurturing and, (b) capable of using observation of the child and appropriate stimulation to move the child from his/her developmental level to a higher level of functioning. The stimulation is organized around a set of simple games that the provider can play with the child one-on-one, as a way to encourage the provider to listen to the child, talk to the child, respond to the child’s questions and actions, and help the child develop.
Learningames supports caregivers in providing an effective learning environment for children. The model is organized around three strategies that are used sequentially, forming a responsive interaction between caregiver and child. First, the caregiver notices what the child is doing as well as the child’s interests and developmental level. Second, based on what he or she notices, the caregiver then nudges the child in an appropriate way to help move his/her understanding to a higher level. Once the child responds, the adult begins to narrate, to track what the child is doing and/or to guide the child’s behavior in new directions. As the adult notices changes in the child’s behavior the cycle begins again and may repeat itself many times even during a short interaction. To help caregivers find opportunities to engage in these types of interactions with children, Learningames has a set of about 200 simple activities (each presented on a card with both a picture of a caregiver and child in that activity and easily-understood suggestions about how to initiate and extend the activity). The 200 “Learningames” focus on simple, everyday activities to provide opportunities for the Notice-Nudge-Narrate cycle. In addition, the activities are carefully constructed to promote children’s oral language development, communication skills, and early understanding of sounds and letters.
Mentors are trained over a three-day period in how to work with providers both to implement Learningames and to address other problems and needs they may have. Providers and mentors each attend separate, initial group training sessions after which mentors visit the homes every two weeks, spending one to two hours with each provider. During this visit, mentors introduce new games, model their use and encourage providers to try using the approach with children. They answer questions and discuss problems or issues the provider may have and suggest solutions.
Giving providers simple games to play individually with children, and training providers about the importance of using language and of encouraging children to use language in an interactive cycle, Learningames incorporates the most recent research on what supports children’s development and delivers it in a way that seems well-suited to the family child care environment.
Study Components
The study includes two components:
An Implementation Study to document the process in which Learningames is implemented in family child care homes in Massachusetts and describe challenges and barriers to implementation that are encountered in this setting; and
An Impact Study to estimate the effects of Learningames on the quality of care provided in family child care homes and children’s language and pre-literacy skills.
Implementation Study
When combined with a well-designed impact study, a comprehensive Implementation study is an indispensable evaluation component. In the overall context of the experiments, the research goals of the Implementation analysis are to:
Describe the intervention;
Describe the degree to which the intervention was implemented as planned;
Document relevant contextual factors; and
Help interpret the findings of the impact study.
The principal task of the Implementation Study is to describe the intervention. Because observed impacts are the result of differences in services actually experienced by treatment and control group providers and children (i.e., the fidelity of the implementation), those experiences must be documented. There will be three sources of information on fidelity/degree of implementation: home visitors that are implementing Learningames will use a five-point scale with definitions at each point; as part of the implementation, providers will keep feedback logs on which games they used with which children during the week; and, as part of the independent data collection, Abt observers will also complete a simple observational measure of fidelity.
In addition, we will investigate challenges to implementation and possible reasons for differences in implementation through informed discussions during regular meetings with child care systems staff and home visitors and through a Home Visitor Questionnaire. The Questionnaire, which home visitors for both the treatment and control groups will be asked to complete, will collect information on caseload size, frequency duration and purpose of home visitors, education and training, and language of the home visitor.
Information on actual implementation of Learningames will help in the interpretation of impact findings. For example, if Learningames fails to bring about its expected impacts on providers and children in Massachusetts, the information collected through the Implementation Study should help us distinguish among three possible reasons: Learningames was not implemented as planned and was not a true test of the intervention; contextual factors counteracted the behavioral influence of the demonstration; or the demonstration was implemented well and in a favorable context, but failed to change behavior in expected ways. Clearly, if there are no impacts, each of these reasons conveys different policy information.
Impact Study
The impact study will address the following research questions:
What is the impact of a research-based developmental curriculum (Learningames), designed to enhance the quality of care and tailored to the needs of family child care provider, on providers’ behavior and interactions with children and on the language and literacy environment of the home?
What is the impact of the intervention on children’s language and pre-literacy skills?
Family child care homes participating in the evaluation will be randomly assigned to one of two groups. The treatment group will receive Learningames, in addition to generalized technical assistance provided by family child care networks. Control homes will receive only the generalized technical assistance. We will then estimate the impact of Learningames on both providers and children by comparing the Learningames group and the control group on a set of key outcomes described below. The process of random assignment ensures that the groups of providers are the same across all measured and unmeasured characteristics that could be related to the study outcomes. This means that the estimates of the impacts of Learningames will be unbiased; statistically significant differences on the study outcomes that favor the Learningames group will provide convincing evidence that the implementation of the curriculum caused these positive differences.
The logic of the impact study is that Learningames will change what providers do with children, which in turn will improve children’s developmental outcomes. In addition, it is assumed that there will be differences in how well caregivers implement the Learningames approach, depending on the caregiver’s background characteristics. To assess all parts of this logic model, the study will collect three kinds of data will be collected for the evaluation: observations of provider behavior with children, a brief interview with the provider about her education and experience, and assessments of children’s development.
The provider observations will use the QUEST Caregiver Rating Scale, a standardized coding system for rating the quality of early childhood settings, including either family child care or center care. The Caregiver Rating Scale is based on the most recent research on instructional practices that are associated with children’s development and learning. The rating scale focuses on caregiver warmth/responsiveness and on caregiver support for the child’s development in four critical domains—cognitive development, especially language development and early literacy; emotional development; social development; and physical development.
The QUEST describes six main aspects of caregiver interactions in the home:
Caregiver with Children
Caring and responding (items 1-10)
Using positive guidance and discipline (items 11-19)
Supervision (items 20-23)
Does no harm (items 24-28)
Supporting Social Emotional Development (items 29-36)
Supporting Play (items 37-40)
Supporting Cognitive Development
Instructional style (items 41-45)
Learning activities and opportunities (items 46-56)
Supporting Language Development and Early Literacy (items 57-67)
Television and Computers (items 68-69)
The QUEST is completed based on a minimum of 2.5 to 3 hours of observation in the home. Providers will be observed up to four times over the two years, before the intervention and three additional times. A subset of observations will use an additional rating scale – the Family Day Care Rating Scale (FDCRS) – for the principal purpose of enabling a comparison of the two ratings.
Outcome data on the children will be collected using parts of two instruments: 1) the items related to the auditory subscale of the Preschool Language Scale-Fourth edition (PLS-4) and 2) the first six items of the Bracken Basic Concept Scale--Revised (BBCS-R) The PLS-4 auditory subscale measures children’s receptive language. It is appropriate for children from birth to six years. The norms include children’s total language, auditory comprehension, standard scores, percentile ranks, and language age equivalents. There are English and Spanish language versions. The subtest for the PLS-4 takes between 15 and 30 minutes per child. For the Bracken, we would use the first six items (the School-Readiness Subtest). There are also English and Spanish language versions. These items relate to colors, letters, numbers, shapes, size, and quantity. These six items on the Bracken take between 5 and 15 minutes to administer. The maximum total per child assessment time to do the above-described components of each of the instruments would thus be 45 minutes (i.e., a maximum of 30 minutes for the PLS-4 auditory component and a maximum of 15 minutes for the Bracken School-Readiness Component).
After OMB clearance has been obtained, a provider questionnaire will be administered. It will include questions about the provider’s background, the level and types of education and training obtained, languages spoken, and motivation for being a child care provider. Information from the provider questionnaire will be used as covariates for the impact study. A second, shorter questionnaire will be administered at 24 months of the study, to document additional education and training obtained by providers, beyond the Learningames intervention, over the two years. Finally, for baseline information on children’s developmental status, we will use extant data that are currently collected by the home visitors employed by the family child care networks in the course of their regular work.
A3 Use of Improved Technology to Reduce Burden
a. Massachusetts
The use of improved technology has been incorporated into the data collection wherever possible to reduce respondent burden. For instance, information about the number of subsidized children enrolled in the homes and the duration of their enrollment will be obtained from centralized databases maintained by each family child care system.
A4 Efforts to Avoid Duplication
There are no other studies currently underway to examine the effects of Learningames or any other curriculum for family child care providers in the Commonwealth of Massachusetts. There are no other studies currently underway to examine the effects of providing subsidies in Illinois to families who would otherwise be ineligible for them according to current eligibility limits in Illinois. There are no other studies currently underway to examine the effects of reducing parent co-payments in the State of Washington.
A5 Efforts to Minimize Burden on Small Businesses or Other Small Entities
a. Massachusetts
Every attempt has been made to reduce the burden placed on family child care providers participating in the experiment. To this end, information on child care quality and fidelity of implementation of Learningames will be collected through direct observation rather than interviews or questionnaires that address provider behavior.
A6 Consequences if the Information is Not Collected
a. Massachusetts
Currently, states expend funds on initiatives to improve family child care, but have little guidance on what constitutes an effective intervention. Compared with center-based care, for which there is an expanding body of knowledge based on rigorous experimental research, family child care has been relatively neglected, except for descriptive studies. This study will provide reliable information on the effectiveness of a research-based intervention that is particularly suited to family child care.
Observational data on caregivers and the home environment is essential in understanding the impact or lack of impact on children of the intervention, since the caregiver’s behavior mediates those impacts. If the intervention results in behavior and interactions that promote children’s development, we might expect to see a positive impact on children’s performance on standardized tests of language and preliteracy skills. If, on the other hand, the intervention fails to make changes in caregivers that are greater than those that occur in the control group as a result of conventional assistance, we would not expect a measurable difference in the outcomes for children in the two groups of homes. We do not expect to see immediate or substantial change in caregiver behavior. The observations are repeated over the course of the study so that we can understand the rate at which and the ways that caregiver behavior changes, with and without an intervention.
Background information on caregivers will be collected once to provide information necessary to construct covariates for the regression models used in the impact analysis.
Child outcome data will be gathered through administration of standardized assessments at three points in time. In this case, the repeated measures are necessary to ensure that we capture information on children who leave the home before the end of the two-year-period of the study. While these data collections will generally occur at set intervals, like the observations, we will ask providers to let us know when a child is leaving so that we can schedule an assessment if the regularly scheduled data collection will miss that child.
A7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5 (D) (2)
All three proposed data collections are consistent with the guidelines set forth in Section 1320.5 (D) (2).
A8 Efforts to Consult with Persons Outside the Agency
An announcement of the Administration for Children and Families’ (ACF) intent to seek approval to collect this information provided an opportunity for public comment on this study. This announcement was published in the Federal Register, September 21, 2005, Volume 70, Number 182, pages 55402-55403 and specified a 60-day period for comment ending November 20, 2005. No comments or suggestions were received in response to this notice. A copy of the relevant Federal Register announcement is provided in Appendix A.
Several individuals were consulted in developing the design for the studies and identifying the types of data to be collected. Their feedback was obtained through telephone conversations, on-site meetings of the full study staff, and meetings with individual consultants. The names and affiliations of these individuals include:
Bobbie Weber
Coordinator
Oregon Child Care Research Partnership
Oregon
State University
236 NW 28th Street
Corvallis, OR 97330-5307
David Edie
State Technical Assistance Specialist
10 S Blackhawk Ave
Madison, WI 53705-3317
Deanna T. Schexnayder
Associate Director
Ray Marshall Center for the Study of Human Resources
LBJ School of Public Affairs
The University of Texas at Austin
3001 Lake Austin Blvd., Suite 3.200
Austin, TX 78703
512-471-2193 (voice)
Robinson Hollister
Professor: Econometrics, Labor and Social Economics, Health Economics
Swarthmore College
500 College Avenue
Swarthmore, Pennsylvania 19081
Marcia Meyers
Associate Professor
University of Washington
School of Social Work and
Daniel J. Evans School of Public Affairs
4101 15th Ave NE
Seattle, WA 98105
206-616-4409
Louise Stoney
NCCIC Technical Assistance Specialist
308 Thais Road
Averill Park, NY 12018
518-674-5635
Ann D. Witte
Professor of Economics
Wellesley College
Wellesley, MA
A9 Payments to Respondents
a. Massachusetts
Family child care homes will be given a $20 gift certificate for children’s educational materials each time researchers conduct an observation and assess children in their home, to thank them for accommodating the researchers in their homes. Every effort will be made to minimize the disruption in their homes caused by the data collection.
A10 Assurances of Confidentiality Provided to Respondents
Abt Associates is fully committed to protecting the anonymity of respondents at all points in the data collection and analysis Implementation. The following data handling and reporting procedures will be used to maintain the privacy of all individual respondents:
a. Massachusetts
Each family child care provider and child who participates in the study will be assigned a unique identification number which will be used throughout the study. All data collected will be attached to and stored with this number, rather than to any name or other identifying information.
Files linking ID numbers to the names of individuals will be kept in a locked file to which access is restricted. Access to this information will be limited to field staff/interviewers who need this information to schedule interviews and to Abt senior staff who will monitor the data collection.
The importance of maintaining confidentiality will be stressed during data collection training. All Abt staff will be required to sign a statement that affirms their understanding of the assurance of confidentiality and pledges to maintain that confidentiality.
Coding documents and computer files will refer to respondents by their ID numbers only. No names or other identifying information will appear on data files. Access to all data bases will be protected by passwords and restricted to staff involved in the data analysis.
No data will ever be reported by the contractor in any form that can be identified with individual respondents.
A11 Justification for Questions of a Sensitive Nature
For all three studies, the interviews that will be conducted do not contain any questions of a sensitive nature.
A12 Estimates of Respondent Burden
a. Massachusetts
Exhibit A12.1 presents estimates of respondent burden for the Massachusetts study.
Exhibit A12.1 Respondent Burden |
|||||||||||||
Year |
Number of Family Child Care Providers |
Interviews Per Year |
Hours Per Interview |
Hours Total Burden: Provider Interviews |
Number of Children |
Assessments Per Year |
Average Hours Per Assessment |
Hours Total Burden: Assessments |
Numbers of Home Visitors |
Interview Per Year |
Hours Per Interview |
Hours Total Burden: Home Visitor Interviews |
Total Burden: Interviews + Assessments |
2006 |
350 |
1 |
.16 |
56 |
700 |
1 |
.5 |
350 |
64 |
1 |
.16 |
10 |
416 |
2007 |
350 |
0 |
0 |
|
700 |
1 |
.5 |
350 |
|
|
|
|
350 |
2008 |
350 |
1 |
.16 |
56 |
700 |
1 |
.5 |
350 |
|
|
|
|
406 |
Total |
350 |
2 |
.16 |
112 |
700 |
3 |
.5 |
1,050 |
|
|
|
|
1,172 |
The estimated annual burden for respondents in the Massachusetts study is 391 hours (total burden divided by number of years: 1,172/3).
A13 Estimates of the Cost Burden to Respondents
For all three studies, there are no direct monetary costs to respondents other than their time to participate in the study.
A14 Estimates of the Cost to the Federal Government
The information collection activity and associated forms have been developed in the performance of DHHS contract number 233-01-0012. The period of performance of the project is from September 30, 2001 through September 30, 2008. The costs associated with the data collection activity for which clearance is requested are as follows:
a. Massachusetts
Development of Data Collection Instruments $0
Data Collection $854,000
A15 Reasons for Any Program Changes or Adjustments
All three studies are new projects.
A16 Plans for Tabulation, Statistical Analysis and Publication
a. Massachusetts
Analysis Plan
There are two primary sources of data to be analyzed in this study: (1) observation measures of providers’ behaviors and interactions with children and (2) measures of children’s language development. Questions about the impact of Learningames on provider behavior and child outcomes will be answered by estimating the mean difference between the treatment and control groups on the outcome measures at each post-assessment. One of the primary benefits of a randomized experiment is that it produces unbiased estimates of program impacts. Because providers have been randomly assigned, on average the treatment and control groups will be the same across all dimensions except for the presence or absence of the Learningames curriculum. As a result, any differences in average outcomes can be attributed to the implementation of Learningames. It should be noted, however, that this estimate could become biased because of poor implementation of the randomized design. Therefore, it is critical to take steps to ensure that the fidelity of the design is preserved, both in the execution of the study and analytically. Four common sources of bias are (a) post-assignment attrition, (b) crossovers of group members from treatment and control groups, (c) contamination of the control group, and (d) treatment group non-participation or under-participation (Shadish, Cook & Campbell, 2002). While the study is designed to minimize the burden on providers, it is possible that homes could drop out of the study after being assigned to a condition, particularly as this is a longitudinal study. This would result in both a loss of statistical power, and, in the case of non-random attrition, would bias estimates. Abt Associates has an impressive record of minimizing such attrition, and has established strategies for retaining participants throughout the course of longitudinal studies.
Crossover and contamination, the second and third potential threats, both result in control providers or children being exposed to the treatment either by switching groups or by being in close proximity to treatment children or their providers. This seems unlikely in this study, both because providers have little contact with each other and because the agency staff assigned to visit treatment providers will not visit any control providers. This will eliminate any temptation to introduce elements of Learningames to control providers. It is possible that a child could switch from a control provider’s home to a treatment home during the study. We will identify the children in each home, as well as any previous homes they attended, so that we can include a measure of crossover in our final analyses.
The final threat to the design is of greater concern. While the goal is for each child in a treatment home to receive the complete Learningames curriculum, for a variety of reasons (e.g., absences, differences in provider practices, timing of the child’s enrollment etc.), it is possible that there will be some children who will not receive the full Learningames treatment. If there are a large number of untreated or under-treated children in our treatment group, we will underestimate the true impact of Learningames. Therefore, we will examine whether variation in program effects across children is related to differences in the implementation of the program itself, including child attendance and provider practices as well as other variables related to the fidelity of the implementation of the curriculum.
Estimating Impacts on Children and Providers. In each home in the sample, there will be a single provider and a small number of children ranging in age at baseline from 2 to 36 months. We expect the number of children to range from one to four, and these children will therefore be clustered within homes.
To estimate the impact of Learningames on children while accounting for the fact that they are clustered within child care homes, we will fit a separate, two-level hierarchical linear model for each child outcome of interest. The model will include baseline covariates at both the child and provider level, including demographic characteristics and other characteristics known to be associated with study outcomes.3
The model is specified as follows:
Level-1 (child):
where:
is the outcome measure (e.g., PLS-4; Bracken) for child i, in
home j,
is the mean of the outcome in home j,
BLij is the measure of the outcome at baseline for child i, in home j,
is the coefficient associated with the baseline measure,
are a set of child-level covariates (1-h) for child i, in
home j,
are the coefficients associated with those covariates, and
is the unique error term associated with child i, in home j.
Level-2 (provider/home):
where:
is
the mean of the outcome (e.g., PLS-4 subscale; Bracken subscale) in
control homes,
LGj is a school-level group indicator variable equaling 1 for Learningames homes and 0 for control homes,
is
the mean difference between the Learningames and control homes
on the outcome measure,
are a set of provider level covariates (1-h) for home j,
are the coefficients associated with those covariates, and
is the unique error term associated with home j.
The coefficient associated with the Learningames
indicator (
)
can then be directly interpreted as the impact of Learningames
on the outcome measure. If this coefficient is positive and
significant, we will conclude that Learningames has had a
positive impact on the children in family child care homes.
To estimate the impact of Learningames on providers, we will fit separate OLS regression models for each provider outcome. These models will also control for baseline measures as well as provider demographics and other characteristics known to be related to study outcomes, such as the size of the child care home.
The model is specified by the following equation:
where:
is the outcome measure for provider i,
is the mean of the outcome for control providers,
LGi is an indicator variable equaling 1 for providers in the Learningames group and 0 for those in the control group,
is the difference between the Learningames and control group
on the outcome measure,
BLj is the measure of the outcome at baseline for provider i,
is the coefficient associated with the baseline measure,
are a set of provider-level covariates (1-h) for provider i,
are the coefficients associated with those covariates, and
is the unique error term associated with provider i.
The coefficient
,
can then be directly interpreted as the impact of Learningames on
the outcome measure, controlling for a set of provider
characteristics known to be related to the study outcomes. We will
then conduct a hypothesis test to assess whether this estimate is
statistically different from zero in favor of the Learningames
group (a one-tailed hypothesis test).
Study Schedule
The planned time schedule for the study is as follows:
Expected OMB approval May 2006
Recruitment and random assignment begins May 2005
Random assignment ends July 2007
First data collection: provider observations June 2006
Second data collection provider observations +
Child assessments May/June 2007
Final data collection January 2008
Final Report December 2008
A17 Display of Expiration for OMB Approval
A space for the OMB approval number and expiration date is indicated at the top of the cover page for each instrument submitted.
A18 Exception to the Certification Statement Identified in Item 19.0 of Form OMB 83-I
None.
Part
B
Collection of Information Using Statistical Methods
B
1 Sample
Universe, Sampling Method and Expected Response Rates
a. Massachusetts
Sample Universe
The target population includes family child care providers who are licensed, part of a network, and stable (i.e., in business for at least two years). The study is being conducted with a sample of family child care providers from family child care networks in the state who have indicated interest in and the capacity to implement Learningames. A statewide sample of such family child care homes is desired in order to obtain results that are applicable to the state as a whole.4 Choosing a sample from only part of the state would yield results that are representative of those parts of the state, but substantial differences in economic and personal circumstances of family child care providers and families in different parts of the state would mean the results would be of less use to the State. However, in order to increase efficiency and reduce costs we will try to cluster the sample of providers within a relatively small number of regions of the state. The study will include children in family child care homes enrolled in the home who, at the start of the study, are 36 months of age or younger.
Sampling Method
Within each region included in the sample, we will recruit family child care networks that can contribute at least 10 homes to the study (e.g., they have 10 homes that will volunteer to participate, and each has been in business for two years and cares for two children under 36 months of age). The number of networks participating in the study will vary by region. Randomization will occur within each family child care network so that all participating agencies are guaranteed to have half of their homes be in the Learningames group.
Within these providers’ homes, we will study the development of preschool children. Since the study is longitudinal, following the same providers over two years, we will include in the study children in the homes who are 36 months old or younger who either (a) are in the homes at the outset of the evaluation or (b) who enroll in the homes during the first 18 months of the study. This “rolling” sampling strategy will help increase our chances of having an adequate sample of children for the impact analyses on child outcomes. We will close study enrollment to new children six months before the end of the evaluation period so that all children evaluated at the final assessment point will have been in the home at least six months.
The children will be in the family child care homes for differing amounts of time. Some children will enter the home during the study period and others will leave. At the end of the two years of the evaluation, the analyses of child impacts will first analyze the average-age standardized score on the measure of language development across all children clustered within the home. Second, we will examine the impacts for different age groups of children, assuming the final sample includes sufficient numbers of children in the relevant age categories. We propose to divide the sample into four age groups, based on age of child at the completion of the study or at the last testing point before the child leaves the home:
under 12 months,
12-23 months,
24-35 months, and
36 – 60 months.
Sample size is determined by our desire to measure child outcomes as well as provider behavior. We will net approximately 350 providers, 175 treatment and 175 control. We assume that each provider will have at least two children in the sample. This sample size allows us to detect effects on children and on providers of 0.23 standard deviations.5
B2 Data Collection Strategy
a. Massachusetts
Four kinds of measures will be collected for the evaluation: systematic observations of provider behavior; standardized assessments of children’s development; a provider questionnaire; and a home visitor questionnaire.
Exhibit B2.1 shows the categories of data to be collected, data sources, time-period for collection and analyses in which they will be used.
Exhibit B2.1 Overview of Data Needs and Data Sources |
|||
Data Needs |
Sources of Data |
Time Collected |
Analyses for Which Data Are Used |
Child characteristics Age, gender, home language, length of time in care setting |
|
|
|
|
|
|
|
Provider characteristics Age, ethnicity, education, training, experience, job motivation |
|
|
|
|
|
|
|
Home environment Health and safety, support for cognitive, language, social-emotional development, equipment materials |
|
|
|
|
|
|
|
Provider behaviors and interactions
|
QUEST caregiver rating |
|
|
Child outcomes
|
|
|
|
|
|
|
|
Home visitor characteristics Education, training, experience, caseload size, frequency and duration of home visits, job responsibilities
|
|
|
|
Observations of Providers
Baseline data will be collected by the staff of the family child care networks using the QUEST form and trained by Abt staff. Study staff, hired by Abt Associates, will collect similar observation data using QUEST and FDCRS six months after the intervention begins, and again at 12 and 24 months. At each observation point, providers will be for approximately 2.5-3.0 hours. The observations will use a standardized rating system. All observers trained to reliability by the Abt staff.
Child Assessments
Baseline information on the developmental status of children in the study will be drawn from extant data collected by the participating family child care systems for children who are in the homes at the outset of the study, and for children who enter the homes at a later date and up to six months before the study ends. Similar data will be obtained for children who are under 36 months and enroll in the family child care home after the study begins and up to six months before the end of the study. The evaluation team will collect assessment data at three points over the two years, on the same schedule as for the observation data. These assessments will use subscales from two standardized measures, the PLS-4 auditory subscale and the Bracken School-Readiness Subscale described in an earlier section. For those children 3 years and older, the test will be administered individually to the children by study staff, at the family child care homes.
Provider Questionnaire
A provider questionnaire will be administered by Abt study staff in June 2006 and January 2008. The initial questionnaire will obtain information on the background and educational and training experience, and motivation of the providers. The second questionnaire will focus specifically on additional education and training obtained over the two years, beyond that offered by Learningames.
Home Visitor Questionnaire
A questionnaire for home visitors will be distributed by Abt study staff in June 2006. The questionnaire will collect data on education and training, caseload size, job responsibilities, frequency and duration of home visits.
b. Illinois
Exhibit B2.2 presents a summary of our data collection strategy. Our main sources of data are extant administrative data and documents, the parent interview as described in sections above, and unstructured interviews with state officials and child care experts. Because we are using extant data or will be speaking with fewer than nine people in any category for the Implementation study, and because we are not using a structured format for these interviews, we are not asking for review for the Implementation study component of the data collection.
Exhibit B2.2
Overview of Data Needs and Data Sources
|
|||
Data Needs |
Sources of Data |
Time Collected |
Analyses for Which Data Are Used |
Family and household characteristics (e.g., family size, number of parents, number and ages of children) |
|
|
|
|
|
|
|
Employment and educational characteristics (e.g., number of employers, employment hours and schedules, earnings, school attendance) |
|
|
|
|
|
|
|
Family income (e.g., total household income, child support received, household income from employment) |
|
|
|
|
|
|
|
Public assistance use and costs (e.g., use of TANF cash assistance, use of food stamps, administrative costs of subsidy receipt) |
|
|
|
|
|
|
|
Child care characteristics (e.g., number of children receiving child care, type of subsidized arrangements, schedule of arrangement, child care subsidy costs, administrative costs, family costs) |
|
|
|
|
|
|
|
Planning and start up (e.g., demonstration design, rationale, target groups, intended impacts, planning Implementation, start-up experiences, etc.) |
|
|
|
|
|
|
|
Demonstration operations (e.g., client flow through random assignment, levels and patterns of participation) |
|
|
|
|
|
|
|
Site-related contextual factors (e.g., local child care market conditions, local economic conditions, expectations about subsidy use among low-income families) |
|
|
|
c. Washington
Exhibit B2.3 aligns the categories of data with our data sources and provides the time period during which they will be collected.
Exhibit B2.3 Overview of Data Needs and Data Sources |
|||
Data Needs |
Sources of Data |
Time Collected |
Analyses for Which Data Are Used |
Family and household characteristics (e.g., family size, number of parents, number and ages of children) |
|
|
|
|
|
|
|
Employment and educational characteristics (e.g., number of employers, employment hours and schedules, earnings, school attendance) |
|
|
|
|
|
|
|
Family income (e.g., total household income, child support received, household income from employment) |
|
|
|
|
|
|
|
Public assistance use and costs (e.g., use of TANF cash assistance, use of food stamps, administrative costs of subsidy receipt) |
|
|
|
|
|
|
|
Child care characteristics (e.g., number of children receiving child care, type of subsidized arrangements, schedule of arrangement, child care subsidy costs, administrative costs, family costs) |
|
|
|
Data Collection Strategies
The major sources of data include the parent interview, administrative data and other extant information, and interviews with staff at Washington Department of Social and Human Services (DSHS) and the State’s regional offices. Each of these is discussed briefly below.
Parent Interviews
As stated earlier, parent interviews will be conducted by telephone at 8, 16, and 24 months after random assignment. We do not plan to interview people in their homes. We will attempt to interview about 2,500 families and expect to interview 2,000 families (for an 80 percent response rate). Interviews will be divided about equally between the treatment and control groups. The interview will be a vital source of information for the impact and benefit-cost analyses and parts of it may also be used for the Implementation Study. The interview will provide us with more detailed information about family characteristics than is available in the baseline data, as well as changes that have occurred in some of these characteristics (e.g., the birth of a child, an additional adult moving into the household) since random assignment began. The survey will also be the study’s primary source of information about child care and employment characteristics over the course of the two-year period.
We believe that we can capture changes in employment and child care with sufficient accuracy through a telephone interview. If, at the completion of the first interview at 8 months, it becomes clear that a higher percentage of in-person interviews will be necessary, we would be able to adjust our data collection plan.
Administrative Data
Records from various public assistance programs will be used for the impact and benefit-cost analyses. In general, these records will be used to quantify participants’ use of various forms of public assistance. In addition, information on employment and earnings from Unemployment Insurance records will augment data from parent surveys.
For each automated system, data will be provided one year (understanding data system limitations) prior to and two years following random assignment. If additional funding is obtained, we may seek additional follow-up data. In that case, we would ask for consent for release of identifying information when families are surveyed at the 24-month point. The automated systems include the following:
Child care subsidy amounts and provider information;
TANF authorized grant amounts and dates;
Food Stamps authorized amounts and dates; and
Unemployment Insurance (UI), quarterly wages (earnings), and employer ID numbers.
Data Collection for the Implementation Study
The Implementation Study will rely on information from the baseline, administrative, and survey data (Exhibit B2.4). In addition, this part of the subsidy evaluation will rely on a range of open-ended interviews and document reviews. These are described briefly below.
On-Site Data Collection
Open-ended interviews, as well as the collection of various documents, will take place on-site through two field visits over the course of the demonstration. During the visits, researchers will conduct individual and small-group interviews with State DCCEL and DSHS management and staff, and local DSHS management and staff. Researchers will also use both visits to observe demonstration operations.
Exhibit B2.4 Data Collection Strategies for the Implementation Study |
|
Data Source |
Collection Strategy |
Demonstration providers Parents |
Small group open-ended interviews Follow-up surveys (as part of the impact analysis) |
State DSHS staff |
Individual and small group open-ended interviews |
Local DSHS staff |
Individual and small group open-ended interviews |
DSHS administrative data |
Periodic files provided by DCCEL (as part of impact analysis) |
DSHS statistical reports |
Periodic requests to DCCEL |
Demonstration plans and design |
Requests for DCCEL planning documents; MOAs between Abt Associates and DCCEL |
Subsidy system policy manuals and eligibility forms |
Requests to DCCEL |
Census information |
U.S. Census |
BLS labor market data |
BLS publications (hard copy and online) |
Individual and Small Group Open-Ended Interviews
Much of the descriptive information about intervention design, planning, and implementation, as well as about the context in which the demonstration will operate, will come from individual and small-group interviews with key informants during the first site visit to Washington. The Implementation Study will include open-ended interviews with the following informant state and local subsidy and public assistance agency staff.
Researchers will use interview guides that will be developed for each type of informant. The open-ended interviews will be conducted individually or in small groups of up to three informants. An advantage of small-group interviews (where possible) is that although one respondent may forget one or more details, or may answer incorrectly, informants in small groups usually correct one another and can fill in details others may leave out. Because we are primarily interested in "getting the story right," we will try to organize small-group interviews, where possible.
The interview guides will be organized by topic area for each type of informant. Within each topic area, the guides will include basic questions and probes designed to stimulate discussion and more complete information for each topic area. The use of detailed interview guides insures some level of uniformity across researchers and informants. Also, the guides as annotated by interview notes provide a structure to data collection that readily organizes field notes for analysis and reporting.
Another useful practice in conducting open-ended interviews is to ask respondents the reasons and/or evidence for their judgmental answers. First, this may force informants to think more carefully about their responses and qualify them in the light of their grounds for holding their opinions. Second, it allows the researcher to weigh the informant's opinion against the strength of the evidence used to support it.
Subsidy Agency Statistical Reports
Extant subsidy agency statistical reports will be used to help characterize the child care subsidy market in the demonstration sites. We expect such reports to provide basic information about: subsidy use, including numbers of families, children, and providers; mean subsidy amounts; types of care used.
Subsidy System Policy Manuals and Eligibility Forms
We will collect demonstration site subsidy system manuals and eligibility forms as our primary source of information about subsidy eligibility criteria, subsidy levels, and co-payment amounts and collection processes. The manuals and eligibility forms will also allow some insight into the initial eligibility and recertification processes, although information about those operations will also be collected in the open-ended interviews at demonstration sites.
Census Information
Census Bureau information will be used as a primary data source for information about site demographic and socio-economic characteristics. Using data from census tracts that most closely overlap with the demonstration sites, the Implementation Study will summarize information about demonstration site ethnicity, household number and composition, number of families with children, distribution of children by age, and other relevant contextual factors.
Bureau of Labor Statistics Labor Market Data
The BLS is an important source of data about local area labor markets, wage rates, industrial mix, employment/population ratios, unemployment data, and other labor market factors. The BLS data are organized by major metropolitan areas and the larger standard statistical metropolitan areas (SMSA). The BLS data will be important in characterizing the low-income labor market facing many subsidy families.
B3 Methods to Maximize Response Rates
a. Massachusetts
The data collection strategies planned for the study involve observations in the family child care home and direct assessment of children. Early in the study, providers will be asked to complete a brief questionnaire about their educational background, experience and motivation. Since the response burden for providers is very little (7-10 minutes), and since home visitors will assist Abt staff in collecting any missing questionnaires, we expect a response rate for the questionnaire of better than 90%. There is, however, a burden imposed by the presence of observers and assessors; if not addressed with sensitivity, this could, over time, affect provider willingness to allow data collection in their home.
Using past experience as a guide, we propose several strategies to address this issue. First, in scheduling visits to the home, we will emphasize that the visit will occur on a morning that is convenient for the provider, and that their schedule and preferences will be decisive in scheduling a visit. The date and length of the visit will be confirmed in a letter, which will also set out expectations for what will happen during the visit. Data collection staff will telephone providers the day before the visit to confirm the schedule since, in any child care setting, unscheduled events can throw off the provider’s schedule. If this occurs, we will reschedule the visit at a time that is convenient for the provider.
Second, at the end of each visit, we will give each provider a $20 gift certificate to compensate her for the disruption in her schedule occasioned by the data collection.
Finally, as part of our validation efforts, we will telephone a sample of providers visited by each data collector to ensure that the visit went as planned, that the data collector explained what she was doing, answered questions, and was respectful and unobtrusive. For all other providers, we will send a thank you card with a toll-free number they can call if they have any concerns about the data collection.
In addition to these strategies, early in the study, each provider will receive a library of 12 children’s books. We will maintain contact with providers through holiday cards and newsletters.
We expect that these efforts will be successful in maintaining providers’ cooperation. However, there are many reasons why we might experience attrition from the study that have to do with providers’ own lives. Providers may leave the study because they have decided not to continue providing care, because of a family or personal emergency or for reasons beyond our control. If their reasons for leaving the study have to do with the demands of the study, we will work with home visitors and system staff to negotiate a solution. We have planned for approximately 15% attrition. If attrition increases beyond this rate, we plan to refresh the sample by adding new providers. We would randomly assign these providers within systems to either Learningames or the control group, following the same procedures as those initially used.
We expect children to leave the child care home in the course of the study and will replace these children with new entrants under three years of age. While we hope to obtain two assessments on each child, the design does not call for a longitudinal study of specific children. We will continue to recruit age-eligible children into the study until six months before the study ends. Our plan is to have essentially continuous data collection and to have providers notify us if a child is leaving the home. This will allow us maximum flexibility in assessing children and reduce non-response because of brief absence or permanent attrition. At the same time, we expect to have no more than three measurements of each child, for the purposes of calculating burden.
b. Illinois
Survey data will be collected at three points in time. All families in the treatment and control group (a total of 2,000) families will be contacted to be interviewed. Our goal is to achieve an 80% response rate at the first survey wave, conducted approximately 8 months after random assignment (1,600 respondents), 75% of the sample at Wave 2 at 18 months (1,500 respondents) and 70% at Wave 3 at 24 months (1,400 respondents). For each wave, we will attempt to reach the entire study sample, excluding those who ask not to be contacted further. For example, for Wave 2, we will not exclusively attempt to contact the 1,600 respondents who participated in the Wave 1 interview but will use the total study group of 2,000, with the exception of those who refused to be contacted further. While we are estimating a response rate for each wave of the study, we estimate that the overall response rate will be close to 80%; that is, 80% of the sample will respond to at least one of the three survey waves.
In order to increase the likelihood of obtaining this rate, the evaluation team will ensure that the contact information from study participants is accurate and of high quality. The contact information provided by the study participants will include their own address and telephone numbers as well as similar contact information of relatives and friends who are likely to know the participant's whereabouts and do not cohabitate with the respondent. In addition, where it is pertinent, the team will use contact information that it can obtain from public assistance records for those who use TANF, food stamps, or Medicaid over the course of the study period. Contact information will be entered into a centralized sample database that will be used for data tracking and management purposes.
In addition to ensuring that we have high-quality contact information, the evaluation team will use a number of interim tracking methods to ensure that we continue to have up-to-date information. The evaluation team will provide the study participants with a toll-free number to call should they move or get a new phone number. To ensure that the number is on hand we will print it on both a refrigerator magnet and a coffee mug. We will also give sample members a pre-addressed, postage-paid postcard that they may send with any updated address or telephone information. Finally, sample members will be mailed "tracking" letters at points prior to their interviews times. Families who return these postcards will receive a $5 voucher or gift certificate. If these letters are undeliverable, the team will engage in a number of efforts to locate the proper address and telephone number. All respondents who complete an interview will receive a $20 voucher or gift certificate.
Using data from UI wage records and other public records, we will be able to gather basic information about the non-respondents. If necessary, we will be able to construct weights to address non-response. We do not expect that there will be differential response rates between the treatment and control groups.
c. Washington
Survey data will be collected at three points in time. All 2,500 families who are selected to be in the interview sample (drawn evenly from the treatment and control groups) will be contacted to be interviewed. Our goal is to achieve an 80% response rate at the first survey wave, conducted approximately 8 months after random assignment (2,000 respondents), 75% of the sample at Wave 2 at 18 months (1,875 respondents) and 70% at Wave 3 at 24 months (1,750 respondents). For each wave, we will attempt to reach the entire interview sample, excluding those who ask not to be contacted further. For example, for Wave 2, we will not exclusively attempt to contact the 2,000 respondents who participated in the Wave 1 interview but will use the total interview group of 2,500, with the exception of those who refused to be contacted further. While we are estimating a response rate for each wave of the study, we estimate that the overall response rate will be close to 80%; that is, 80% of the sample will respond to at least one of the three survey waves.
In order to increase the likelihood of obtaining this rate, the evaluation team will ensure that the contact information from study participants is accurate and of high quality. The contact information provided by the study participants will include their own address and telephone numbers as well as similar contact information of relatives and friends who are likely to know the participant's whereabouts and do not cohabitate with the respondent. In addition, where it is pertinent, the team will use contact information that it can obtain from public assistance records for those who use TANF, food stamps, or Medicaid over the course of the study period. Contact information will be entered into a centralized sample database that will be used for data tracking and management purposes.
In addition to ensuring that we have high-quality contact information, the evaluation team will use a number of interim tracking methods to ensure that we continue to have up-to-date information. The evaluation team will provide the study participants with a toll-free number to call should they move or get a new phone number. To ensure that the number is on hand we will print it on a refrigerator magnet and a coffee mug. We will also give sample members a pre-addressed, postage-paid postcard that they may send with any updated address or telephone information. Finally, sample members will be mailed "tracking" letters at points prior to their interviews times. Families who return these postcards will receive a $5 voucher or gift certificate. If these letters are undeliverable, the team will engage in a number of efforts to locate the proper address and telephone number. All respondents who complete the first interview will receive a $10 voucher or gift certificate; $15 for completion of the second interview; and $20 for completion of the third interview. The incentive will not affect participants’ potential benefits for public benefits.
Using data from UI wage records and other public records, we will be able to gather basic information about the non-respondents. If necessary, we will be able to construct weights to address non-response. We do not expect that there will be differential response rates between the treatment and control groups.
B4 Tests of Procedures
a. Massachusetts
The observation measures and provider questionnaire have all been tested and used in other large-scale studies with similar populations and so do not require pretesting. The same is true for the standardized child assessments. However, to ensure that our plan for collecting the data is realistic and does not impose undue burden on the provider, we will pretest the data collection procedures in nine family child care homes early in 2006. The results of the pretest will be sent to OMB, with a description of any recommended changes in procedures.
For the children in the study, we want to obtain permission from the maximum number of parents to allow their child to participate in the standardized assessments. We will work closely with the providers to have them help us contact and convince parents of the importance of the study and the low risk of negative consequences for their child. We have a hotline that parents and providers will be able to use to call with questions or concerns at any time during the study.
b. Illinois
We will pre-test the parent telephone interview survey with nine respondents. The results of the pretest will be sent to OMB, with a description of any recommended changes in wording or administration of the survey.
c. Washington
The parent telephone interview survey used in the Illinois study will also be used in the Washington study. The results of the pre-test conducted in the Illinois study apply to the Washington study as well.
B5 Individuals Consulted on the Statistical Aspects of the Design
The information for all three studies is being collected by Abt Associates Inc. and its subcontractor, Moore & Associates, on behalf of the Administration for Children and Families (ACF), U.S. Department of Health and Human Services. With ACF oversight, Abt Associates is responsible for study design, data collection, analysis, and report preparation.
a. Massachusetts
The project staff responsible for the design include the project director (Jean Layzer) the deputy project director (Ann Collins), and the director of analysis (Barbara Goodson).
b. Illinois
The project staff responsible for the design include the project director, Jean Layzer (Abt Associates); the deputy project director, Ann Collins (Abt Associates), and the co-leads for analysis, Nancy Burstein (Abt Associates) and Charles Michalopoulos (MDRC).
c. Washington
The project staff responsible for the design include the project director, Jean Layzer (Abt Associates); the deputy project director, Ann Collins (Abt Associates) and the director of analysis, Charles Michalopoulos (MDRC).
For all three studies, outside consultants reviewed the statistical aspects of the design. These include:
Robinson Hollister
Professor: Econometrics, Labor and Social Economics, Health Economics
Swarthmore College
500 College Avenue
Swarthmore, Pennsylvania 19081
Marcia Meyers
Associate Professor
University of Washington
School of Social Work and
Daniel J. Evans School of Public Affairs
4101 15th Ave NE
Seattle, WA 98105
206-616-4409
Ann D. Witte
Professor of Economics
Wellesley College
Wellesley, MA
References
Anderson, S., Ramsburg, D., & Rothbaum, B. (2003). Illinois study of license-exempt child care: Interim report. Report to the Child Care Bureau, U.S. Department of Health and Human Services. Urbana, IL: School of Social Work, University of Illinois at Urbana-Champaign.
Blau, D. M., “The Economics of Means-Tested Child Care Subsidies,” paper prepared for an
NBER volume on Means-Tested Transfers, February, 2000.
Burstein, N., Layzer, J., and Cahill, K. (2001). National Study of Child Care for Low-Income
Families. Patterns of Child Care Use Among Low-Income Families. Cambridge, MA: Abt
Associates, Inc.
Capizzano, J., Adams, G., and Sonenstein, F. (2000). Child Care Arrangements for Children Under
Five: Variations Across States. Washington DC: Urban Institute.
Casper, L. What Does It Cost to Mind Our Preschoolers? Census Bureau, Current Population
Reports, Household Economic Studies, P70-52.
Chaplin, D., Robins, P.K., Hofferth, S., Wissoker, D., and Fronstin P., “The Price Sensitivity of Child Care Demand: A Sensitivity Analysis,” unpublished manuscript, January 1999.
Collins, A., Layzer, J., Kreader, J., Werner, A., and Glantz, F. (2000). National Study of Child Care for Low Income Families: State and Community Substudy Interim Report. Cambridge, MA: Abt Associates Inc.
Council of Economic Advisors. (December, 1997). The Economics of Child Care. White Paper. Washington, DC: Author.
Hart, B. & Risley, T.R. (1995) Meaningful differences in the everyday experience of young American children. Baltimore, MD: Paul H. Brookes Publishing Co.
Marshall, N. L., Creps, C.L., Burstein, N. R., Glantz, F. B., Robeson, W. W., and Barnett S. (2001). The Cost and Quality of Full Day, Year-round Early Care and Education in Massachusetts: Preschool Classrooms. Wellesley Center for Women and Abt Associates Inc.
Marshall, N. L., Creps, C.L., Burstein, N. R., Cahill, K., Robeson, W. W., Wang, S. Y., Schimmenti, J., and. Glantz, F. B (2003). Family Child Care Today. Wellesley Center for Women and Abt Associates Inc.
Meyers, M., Peck, L., Davis, E., Collins, A., Kreader, J.L., George, A., Schexnayder, D., Schroeder, D., Olson, J. (2002). The Dynamics of Child Care Subsidy Use. A Collaborative Study of Five States. New York, NY: The National Center for Children in Poverty, Mailman School of Public Health, Columbia University.
Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin.
Siegel, G.L., and Loman, L.A. (1991). Child Care and AFDC Recipients in Illinois: Patterns,
Problems, and Needs. St. Louis, MO: Institute of Applied Research.
1 In the fourth site, Miami-Dade County, we are using extant data on children and teachers collected by the Early Learning Coalition of Miami-Dade/Monroe Counties as part of its ongoing Assessment and Improvement Initiative to address research questions about the impact of the experimental treatment. Abt is collecting observation data on each study classroom, but there is no data collection burden and thus no request for clearance.
2 The Labor-HHS appropriations bill states that “$9,920,000 shall be for use by the Secretary for child care research, demonstration, and evaluation activities.”
3 Although random assignment means that in expectation the characteristics of the children and homes in the treatment and control groups will be the same, in our particular sample of children and homes there may in fact be differences between the two groups on observable characteristics. Therefore, we will include demographic and other child and provider characteristics in our models to control for any observable differences between the two groups.
4 Not all family child care providers in the state are associated with networks. This limits the generalizability of the findings to providers who are linked to networks and receive the support and monitoring provided by network staff. However, this subset of providers, who receive child care subsidies, is of particular policy interest to the state. We will use additional extant data from the networks, and from Abt’s Cost-Quality Study of Family Child Care in Massachusetts, to investigate the differences between the study sample of providers and the wider universe of providers in Massachusetts.
5 We are purposefully looking for whether or not Learningames produces positive changes in provider and child outcomes. Therefore, we will conduct a one-tailed test using α1=0.05. We have also assumed that the analyses will include baseline measures that explain 25% of the variation in study outcomes.
Abt
Associates Inc. OMB Clearance Request
| File Type | application/msword |
| File Title | Abt Single-Sided Body Template |
| Author | stewartg |
| Last Modified By | Computer Network Services |
| File Modified | 2007-10-03 |
| File Created | 2007-10-03 |
© 2024 OMB.report | Privacy Policy