Pretesting Data Collection (OMB 0970-0355)
Supporting Statement Part A for OMB Approval
Measurement Development: Quality of Family-Provider Relationships in Early Care and Education
The Office of Planning, Research, and Evaluation (OPRE) and the Office of Head Start (OHS) of the Administration for Children and Families (ACF), within the U.S. Department of Health and Human Services (HHS), are proposing a data collection activity as part of the development of a measurement tool to assess relationships between families and providers of early care and education for children aged birth to five years. The major goal of this project is to develop a measure of the quality of family-provider relationships that will be (1) applicable across multiple types of early care and education settings and diverse program structures, including Head Start; (2) sensitive across cultures associated with racial, ethnic, and socioeconomic characteristics; (3) reliable in both English and Spanish; and (4) appropriate for program evaluation. As a step in developing this measure, OPRE and OHS request permission to conduct two iterative rounds of cognitive interviews with Head Start Family Service Workers (FSW) and families they serve. The purpose of the cognitive interviews is to help improve item wording and ensure that items are applicable to and well understood by FSWs and families. Previous rounds of cognitive testing were approved and completed through this generic clearance (0970-0355) as part of the FPRQ project.
The information collected will be used for internal purposes only.
A growing literature on early care and education indicates that the family-provider relationship is an important domain in early care and education settings. Specifically, research has highlighted the value of the interactive role that families and programs play in fostering positive developmental outcomes of children in these settings (Dunst, 2002; Johnson, 2000; Mendez, 2010). The benefits of early care and education are especially important for children living in poverty who encounter disproportionate risks to meeting academic and social demands needed to excel in school (Bornstein & Bradley, 2003). The nation’s largest and most comprehensive response to the needs of preschool children is the Head Start program. Head Start is a social welfare initiative to give young children growing up in poverty the skills and experiences necessary for subsequent school success (Improving Head Start for School Readiness Act of 2007). Head Start offers young children an enriched preschool experience and families a range of health and social service interventions, including family case management. Indeed, Head Start employs a two-generation approach, focusing on the welfare of both children and their parents, in order to achieve positive outcomes for the family overall (Henrich and Gadaire, 2008). An important component of two-generation approaches is case managers, called FSWs, who work directly with families (Duch, 2005). FSWs also serve as liaisons between the Head Start program and its families and are critical to the achievement of Head Start’s social service agenda. It is often the collaboration between parent, teacher, and FSW that creates an optimal supportive network both for the child at school and the parenting at home (Webster-Stratton, Reid, & Hammond, 2001). For this reason, understanding the FSW-family relationship is imperative to evaluating the effectiveness of Head Start. However, most of the research on the quality and effectiveness of Head Start and the family-provider relationship focuses on the educational intervention in the classroom, including teacher and setting characteristics (e.g., Abbott·Shim, Lambert, & McCarty, 2000), rather than the quality and effectiveness of the FSW and the social services provided to the families (Zigler et al, 1997). Greater focus should be brought to the FSW-family relationship if the quality of the family-school connection is to be fully understood.
While there are a number of federal surveys that collect data on the early care and educational experiences of families and children, such as the National Survey of Early Care and Education and the National Household Education Survey, none include measures that tap into multiple dimensions of family-provider relationship quality that are applicable across diverse populations, focus on the role of FSWs in care settings, or are appropriate for use in program evaluation. The Family-Provider Relationship Quality (FPRQ) project will develop a measure to address these gaps. The new FPRQ measure will be a tool that federal, state and local government agencies can use to gather valid and reliable information about the quality of family-provider relationships as well as a tool that can be used for program evaluation. As part of this project, two FSW surveys will be developed to assess the quality of the FSW-family relationship. One survey will be completed by FSWs, and the other will be completed by parents/guardians regarding their relationship with their FSW.
The proposed data collection activity is part of the fourth step in the process of developing the FPRQ measure.
First, we began with an extensive review of the literature and of extant survey measures, and developed a conceptual model of family-provider relationships to guide our work.
Second, focus groups (conducted under OMB Formative Generic Clearance 0970-0356) with parents and providers were used to assess the extent to which our conceptual model matched the perceptions and experiences of our target populations, and to help guide item development. The focus groups found that, for the most part, the FPRQ conceptual model and definitions of the elements within the model accurately reflected provider and parent perceptions of strong family-provider relationships. In particular, both parents and respondents spontaneously agreed with the elements within the attitudes, knowledge, and practices constructs in the conceptualized model, and generally agreed with the environment construct after they were prompted to provide their opinions. In sum, focus groups findings confirmed the constructs in the conceptual model and helped to streamline the construct definitions.
Third, we conducted an extensive review of existing items, honed our definitions (with the help of the focus group findings), and revised and developed new items when necessary to create new measures of family-provider relationships.
As a fourth step, we tested how well the measures we developed work using cognitive interviews (conducted under OMB Pretesting Data Clearance #0970-0355) with parents, providers (including a small number of FSWs) and directors.
This new round of cognitive interviews will gather targeted information from FSWs and families about the FSW-family relationship specifically. The information obtained through the cognitive interviews will be used to finalize items for the FSW surveys.
Cognitive interviews offer an ideal vehicle for identifying problems with item wording and questionnaire design, and for understanding respondents’ information retrieval and response formation (Willis, 2005). Cognitive interviews in this project will ensure that the items are clear, easily understood, and interpreted the way they were intended. Additionally, cognitive interviews will ensure that the questions developed are applicable to the FSW-family relationship. In addition to determining ease of comprehension, recall of information, and response formation, the cognitive interviews will also identify other issues affecting the accuracy of the information collected in the surveys, such as formatting issues (e.g., skip patterns that are confusing), instructions, and flow of the survey.
This step in the project will result in sound and reliable measures that will tap into multiple domains of FSW-family relationships, and can be used with FSWs, Head Start parents, and for program evaluation.
Cognitive Interviews
Cognitive interviews will be conducted to tap into parents and FSWs’ understanding of items designed to measure FSW-family relationships, and their perspectives about what should or should not be included in such measures. The cognitive interviews will cover the general topics of parents and FSWs’ understanding of and reaction to items that measure the FSW-family relationship, including relationship practices, knowledge, and attitudes that affect FSW-family relationships.
Two iterative rounds of cognitive interviews will be conducted with 9 parents and 36 Family Service Workers. Data collected from the cognitive interviews will be used to identify problematic items for the FPRQ FSW measures and help develop new items if necessary. Specifically, cognitive interviews will help identify comprehension or wording issues, issues with information recall, response formation issues, and response mapping issues. Additionally, participants’ reactions to and feedback on items will help guide the selection of questions that are applicable to the FSW-family relationship. Data from cognitive interviews will also provide an opportunity to identify language and key terms parents and FSWs use to define, discuss, and think about the FSW-family relationship. Identifying common terms across the cognitive interviews will aid in discerning item wording that is applicable to the role of the FSW within the Head Start program. The survey instruments and the cognitive interview protocols for parents and FSWs are presented in Appendices A and B.
Whenever possible, advanced technology will be used to collect and process data to reduce respondent burden and make data processing and reporting more timely and efficient. A digital audio recorder will be used in all cognitive interviews. (Before using the audio recorder, participants’ verbal consent to be audio recorded will be obtained.) To reduce participant travel burden and to get a geographically diverse sample, we will conduct at least two-thirds of the interviews via telephone. We will send participants materials via email when possible. We will also text a reminder with the date, time, and location of the cognitive interview, unless participants indicate a preference to receive this information via regular mail (see Appendix C).
Every effort has been made to determine whether similar measurement tools exist by searching various databases (e.g., national and scholarly), reviewing existing early care and education quality measures, and consulting with experts in the field. Our review of extant literature uncovered family-provider relationship measures; however, none measured multiple domains of family-provider relationships, assessed the FSW-family relationship, or were appropriate for program evaluation. We have also consulted with experts in the early care and education field and they concur that the field lacks appropriate and psychometrically sound (i.e., socially desirable) measures that assess the quality of FSW-family relationships and are applicable for use in program evaluation.
We will only collect data from FSWs from Head Start programs and from families served by Head Start. Although these FSWs and families are not considered small businesses/entities, we will attempt to reduce the impact on them by conducting cognitive interviews on days, at times (e.g., evenings and weekends), and in locations that are convenient to them. This will help to ensure that FSWs’ and families’ participation does not conflict with their other responsibilities. Also, the impact, if any, on small businesses or other small entities will be reduced by the voluntary nature of the data collection.
To minimize the potential burden, participants will only be asked to volunteer to participate in a single cognitive interview. Less frequent data collection would only be possible by not collecting any data at all.
There are no special circumstances requiring deviation from these guidelines. As such, this request fully complies with regulation 5 CFR 1320.5.
The first Federal Register notice for ACF’s generic clearance for information gathering was published in the Federal Register, Volume 76, page 34078 on June 10, 2011. The agency did not receive any comments in response to the Federal Register notice for the generic clearance.
The second Federal Register notice was published in the Federal Register, Volume 76, page 53682 on August 29, 2011.
The FPRQ project has benefited from consultation with many outside experts, including attendees of the “Family-Sensitive Caregiving and Family Engagement Working Meeting: Identifying and Measuring Common Concepts,” a meeting that was sponsored by OPRE in June 2010, and the FPRQ Technical Work Group.
Non-federal attendees of the Family-Sensitive Caregiving and Family Engagement Working Meeting were:
Gina Adams, Urban Institute
Don Bailey, RTI International
Juliet Bromer, Erikson Institute
Concha Delgado-Gaitan, Consultant
Carl Dunst, Smoky Mountain Research Institute
Jay Fagan, Temple University
Nikki Forry, Child Trends
Anne Henderson, Consultant, Annenberg Institute for School Reform
Lee Kreader, National Center for Children in Poverty
Michel Lahti, University of Southern Maine
Laurie Linscott, Michigan State University
Tammy Mann, United Negro College Fund
Lisa McCabe, Cornell University
Christy McWayne, Tufts University
Diane Paulsell, Mathematica Policy Research
Toni Porter, Bank Street College of Education
Eva Marie Shivers, Indigo Cultural Center
Amy Susman-Stillman, University of Minnesota
Bobbie Weber, Oregon State University
The FPRQ Technical Work Group is comprised of the following experts in the fields of measurement development, family-provider relationships, and early care and education:
Catherine Ayoub, Harvard University
Carl Dunst, Smoky Mountain Research Institute
Julia Henly, University of Chicago
Judith Jerald, Save the Children
Elena Lopez, Harvard University
Doug Powell, Purdue University
Lori Roggman, Utah State University
Julia Mendez, University of North Carolina at Greensboro
Suzanne Randolph, University of Maryland
For in-person interviews, we will review the consent form with participants, have them sign the form acknowledging their willingness to participate in the study, and ask them to provide their verbal consent for the interview to be audio recorded. For interviews conducted by phone, we will review the consent forms with participants, request their permission to audio record the consent process, and obtain verbal consent. The consent form will explain to participants the extent to which their privacy will be protected as part of the study (see Appendix D and Appendix E). Specifically, participants will be assured, verbally and in consent forms, that their names will not be documented on final reports, their responses will not be shared with others outside of the study team, and their personally identifiable information will not be linked to their responses. Identifiable information will only be collected prior to the start of the cognitive interview and will not be linked to data collected during the cognitive interview. In order to protect participants’ privacy, a study-specific identification code will be assigned to each participant and will be used for all data collected.
All information collected will be kept private to the fullest extent required by law. More specifically, Child Trends (the subcontractor collecting data for the cognitive interviews) will maintain the security of the data and the privacy of participants by storing electronic data (i.e., electronic computer files, audio electronic files) on a restricted access drive. Following the completion of each cognitive interview, Child Trends project staff will transfer the audio recording over to the secure drive and delete it from the portable recorder. Hard copies of completed recruitment materials, screener interviews, and consent forms will be stored in locked files in locked offices at Child Trends, and will be kept separate from cognitive interview data files (such as transcriptions). The one exception will be the recruited participant sheet, an electronic file that lists all recruited participants and provides their study ID. In addition to being stored on a restricted access drive, this document will also be encrypted and password protected. Child Trends will further institute procedures to ensure the security of data transfer by immediately transfering data onto the secure drive and deleting it from e-mail files.
No sensitive questions will be asked as part of this data collection.
The total annualized hours for this data collection activity is estimated to be 37 hours.
|
TABLE A.1 ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST |
|||||||
|
|
|||||||
Respondent |
Respondent N |
Annual Number of respondents (annualized over 3 yr generic clearance period) |
Number of responses per respondent |
Average burden hours per response |
Total burden hours (annualized over 3 yr generic clearance period) |
Average Hourly Rate |
Total Annual Cost (Dollars) (annualized over 3 yr generic clearance period) |
|
COGNITIVE INTERVIEWS |
|
|
|
|
|
|
|
|
FSW Instruments |
|
|
|
|
|
|
|
|
Ineligible |
72 |
24 |
1 |
0.09 |
2 |
$10.07 |
20.14 |
|
Eligible |
36 |
12 |
1 |
2.25 |
27 |
$10.07 |
271.89 |
|
Parent Instruments |
|
|
|
|
|
|
|
|
Ineligible |
18 |
6 |
1 |
0.09 |
1 |
$15.55 |
15.55 |
|
Eligible |
9 |
3 |
1 |
2.25 |
7 |
$15.55 |
108.86 |
|
|
|
|
|
|
|
|
|
|
TOTAL OVERALL |
135 |
|
1 |
|
37 |
|
416.43 |
|
|
*Note: We will use a recruitment matrix that includes quotas (the maximum number of participants with particular characteristics that we will accept into the sample). Once quotas are filled, no more volunteers with characteristics of the filled quota will be accepted. This strategy will ensure sample diversity and will help us narrow the field of volunteers. |
|||||||
Estimates of Annualized Costs. There is an estimated annualized burden over the 3-year generic clearance period to respondents of $416.43.
For parent respondents, an average hourly salary of approximately $15.55 is assumed based on the Bureau of Labor Statistics (BLS) estimates for median hourly wages for high school graduates. For FSW respondents, an average hourly salary of approximately $10.07 is assumed based on BLS estimates for median hourly wages for child care workers.
There will be no direct cost to the respondents other than their time to participate in the study.
There will be no capital, operating, or maintenance costs to the respondents.
This OMB package is for cognitive interviews specifically for FSWs for the FPRQ project. As stated previously, the purpose of the FPRQ project is to develop measures of the quality of family-provider relationships, including FSW-family relationships, that will be (1) applicable across multiple types of early care and education settings and diverse program structures (including Head Start); (2) sensitive across cultures associated with racial, ethnic, and socioeconomic characteristics; and (3) appropriate for use as a program evaluation tool.
Given the qualitative nature of the data collected, the analysis of the cognitive interviews will not be conducted using descriptive statistics. Instead, a summary document will be prepared for the agency’s internal use. All information collected is for internal use only and will be used solely to inform the development of the new measure of family-provider relationship quality.
The goal is for data collection to start in November. All contacts with potential participants for the purpose of collecting data for the cognitive interviews will occur within a four month period following OMB approval. The goal is for this to occur between November 2013 and February 2014 (see Table A.2). Child Trends will recruit participants from different Head Start programs in the Washington, DC metropolitan area and across other large metropolitan and rural areas in the United States, such as Chicago, IL, Detroit, MI, Seattle, WA, and Los Angeles, CA. Approximately six to eight weeks will be allotted for recruitment and data collection in each round. Two weeks in between rounds will be used for conducting data collection debriefing meetings, data analyses, and edits to the instruments as needed based on the findings from the first round of cognitive interviews. We can expect to conduct our first cognitive interview for Round 1 within a few days of commencing recruitment. Therefore, if data collection begins in November, Round 1 will be completed in November/early December. Round 2 data collection then could begin in January 2014 and be completed in early February.
TABLE A.2
Activity |
Timeline |
||
COGNITIVE INTERVIEWS |
|
|
|
Round 1 |
Commencement* |
Duration |
Completion* |
Recruitment and Data Collection |
November 2013 |
6-8 weeks |
December 2013 |
Debriefing Meetings and Data Analysis |
December 2013 |
2 weeks |
January 2014 |
Round 2 |
Commencement* |
Duration |
Completion* |
Recruitment and Data Collection |
January 2014 |
6-8 weeks |
February 2014 |
Debriefing Meetings and Data Analysis |
February 2014 |
2 weeks |
March 2014 |
*Please note these dates are approximate and may change based on actual dates of OMB approval.
|
|||
The OMB number and expiration date will be displayed at the top of the first page of the consent form that will be given to each participant in the cognitive interviews. We will read the consent form along with the OMB number and expiration date at the start of each cognitive interview.
No exceptions are necessary for this data collection.
References
Abbott-Shim, M., Lambert, R., & McCarty, F. (2000). Structural model of Head Start classroom quality. Early Childhood Research Quarterly, 15(1), 115-134.
Bornstein, M.H., & Bradley, R.H. (Eds.) (2003). Socioeconmic status, parenting, and child development. Mahwah, NJ: Erlbaum.
Duch, H. (2005). Redefining parent involvement in Head Start: a two-generation approach. Early Child Development and Care, 175(1), 23-35.
Dunst, C. J. (2002). Family-centered practices: Birth through high school. Journal of Special Education, 36, 139-147.
H.R. 1429--110th Congress: Improving Head Start for School Readiness Act of 2007. (2007). In www.GovTrack.us. Retrieved September 4, 2013, from http://www.govtrack.us/congress/bills/110/hr14
Henrich, C. C., & Gadaire, D. M. (2008). Head Start and parent involvement. Infants & Young Children, 21(1), 56-69.
Johnson, B. H. (2000). Family-centered care: Four decades of progress. Families, Systems, & Health,18(2), 137-156.
Mendez, J. L. (2010). How can parents get involved in preschool? Barriers and engagement in education by ethnic minority parents of children attending Head Start. Cultural Diversity and Ethnic Minority Psychology, 16(1), 26-36.
Webster-Stratton, C., Reid, M. J., & Hammond, M. (2001). Preventing conduct problems, promoting social competence: A parent and teacher training partnership in Head Start. Journal of clinical child psychology, 30(3), 283-302.
Willis, G. B. (2005). Cognitive interviewing: a tool for improving questionnaire design. Thousand Oaks, California: Sage Publications, Inc.
Zigler, E. F., Finn-Stevenson, M., & Stem, B. M. (1997). Supporting children and families in the schools: The school of the 21st century. American Journal of Orthopsychiatry, 67(3), 396-407.
Appendices
Appendix A: FPRQ Cognitive Interview Instruments for Ineligible Parents
Appendix B: FPRQ Cognitive Interview Instruments for Ineligible Family Service Workers
Appendix C: FPRQ Cognitive Interviews - Email and Text Message Reminder of Interview
Appendix D: FPRQ Cognitive Interview Instruments for Eligible Parents
Appendix E: FPRQ Cognitive Interview Instruments for Eligible Family Service Workers
| File Type | application/msword |
| Author | jwest |
| Last Modified By | Paul Margie |
| File Modified | 2013-10-25 |
| File Created | 2013-10-25 |