FPRQ Cognitive Interviews - Generic Clearance Part A 22March12 track changes

Pretesting Data Collection (OMB 0970-0355)

Supporting Statement Part A for OMB Approval

Measurement Development: Quality of Family-Provider Relationships in Early Care and Education

March 2012

A. JUSTIFICATION

The Office of Planning, Research and Evaluation (OPRE) and the Office of Head Start (OHS) of the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), are proposing a data collection activity as part of the development of a measurement tool to assess relationships between families and providers of early care and education for children aged birth to five years. The major goal of this project is to develop a measure of the quality of family-provider relationships that will be (1) applicable across multiple types of early care and education settings and diverse program structures (including Head Start); (2) sensitive across cultures associated with racial, ethnic, and socioeconomic characteristics;(3) reliable in both English and Spanish; and (4) appropriate for program evaluation. As a step in developing this measure, OPRE and OHS request permission to conduct three iterative rounds of cognitive interviews with parents of children aged birth through five years,early care and education providers, as well as early care and education center directors. The purpose of the interviews is to help improve item wording and ensure that items are applicable to and well understood by a diverse population of providers, directors, and parents, as well as applicable to diverse child care settings. ACF contracted with Child Trends and Westat to carry out these data collection activities.

The information collected will be used for internal purposes only and will not be released to the public.

A.1. Circumstances Making the Collection of Information Necessary

A growing literature on early care and education indicates that the family-provider relationship is an important domain in early care and education settings. Specifically, research has highlighted the value of the interactive role that families and programs play in fostering positive developmental outcomes of children in these settings (Dunst, 2002; Johnson, 2000; Mendez, 2010). In addition, positive family-provider interactions are hypothesized to be associated with improved family and parental well-being as well (AAP, 2003; Bailey et al., forthcoming; Kaczmarek et al., 2004; Trivette et al., 2010). Given these research findings and considering that about half of preschool-aged children in the United States are enrolled in at least one non-parental care arrangement (Iruka & Carver, 2006), it is important to have valid and reliable measures of family-provider relationships.

While there are a number of federal surveys that collect data on the early care and educational experiences of families and children, such as the National Survey of Early Care and Education and the National Household Education Survey, none include measures that tap into multiple dimensions of family-provider relationship quality that are applicable across diverse populations and care settings or are appropriate for use in program evaluation. The Family-Provider Relationship Quality (FPRQ) project will develop a measure to address these gaps. The new FPRQ measure will be a tool that federal, state and local government agencies can use to gather valid and reliable information about the quality of family-provider relationships as well as a tool that can be used for program evaluation.

The proposed data collection activity is the fourth step in the process of developing the FPRQ measure. First, we began with an extensive review of the literature and of extant survey measures, and developed a conceptual model of family-provider relationships to guide our work. Second, focus groups (conducted under OMB Formative Generic Clearance 0970-0356) with parents and providers were used to assess the extent to which our conceptual model matches the perceptions and experiences of our target populations, and to help guide item development. The focus groups found that, for the most part, the FPRQ conceptual model and definitions of the elements within the model accurately reflect provider and parent perceptions of strong family-provider relationships. In particular, both parents and respondents spontaneously agreed with the elements within the attitudes, knowledge, and practices constructs in the conceptualized model and generally agreed with the environment construct after they were prompted to provide their opinions. In sum, focus groups findings confirmed the constructs in the conceptual model and helped to streamline their definitions.

Third, we conducted an extensive review of existing items, honed our definitions (with the help of the focus group findings), and revised and developed new items when necessary. Now that we have an initial set of items, we need to test how well they work using cognitive interviews. Specifically, the cognitive interviews will gather information from parents of children aged birth through five years old participating in a non-parental care arrangement, early care and education center directors, as well as early care and education providers from various care settings including Head Start, preschool, community-based child care centers and family-based care settings. The information obtained through the cognitive interviews will be used to revise the items before they are pilot tested and tested in a large field test, for which a full OMB package will be submitted.

Cognitive interviews offer an ideal vehicle for identifying problems with item wording and questionnaire design and for understanding respondents’ information retrieval and response formation (Pressler et al., 2004). Cognitive interviews in this project will ensure that the items are clear, easily understood and interpreted the way they were intended. Additionally, cognitive interviews will ensure that the questions developed are applicable for a racially, ethnically and economically diverse populations and across different types of child care settings and programs as we will be able to gauge how participants from diverse groups interpret and understand the items. In addition to determining ease of comprehension, recall of information and response formation, the cognitive interviews will also identify other issues affecting the accuracy of the information collected in the surveys, such as formating issues (e.g., skipping patterns are confusing), instructions, and flow of the survey.

This step in the project will help us identify items that are working as intended as well as identifying problematic items and ways in which those items can be revised before they are further tested in a pilot test study. As such, this step in the project will ultimately help us develop a sound and reliable measure that will tap into multiple domains of family-provider relationships that can be used across care settings serving families of various backgrounds and for program evaluation.

A.2. Purpose and Use of the Information Collection

Cognitive interviews will be conducted to tap into parents’, providers’, and directors’ understanding of items designed to measure parent-provider relationship, and their perspectives about what should or shouldn’t be included in such measures.

The cognitive interviews will cover the general topics of parents’, providers’, and directors’ understanding of and reaction to items that measure:

• parent-provider relationship practices;

• environmental features that support quality parent-provider relationships;

• attitudes that affect parent-provider relationships; and,

• provider’s knowledge that affects parent-provider relationship.

A total of three iterative rounds of cognitive interviews will be conducted with 102 English- and Spanish-speaking early care and education providers, directors, and parents of children ages birth through five years. As stated above, data collected from the cognitive interviews will be used to identify problematic items for the FPRQ measure and help develop new items if necessary. Specifically, cognitive interviews will help identify comprehension or wording issues, issues with information recall, response formation issues, and response mapping issues. Additionally, participants’ reactions to and feedback to items will help guide the selection of questions that are applicable for diverse care settings and families. Data from cognitive interviews will also provide an opportunity to identify language and key terms parents, early care and education providers, and directors use to define, discuss, and think about family-provider relationships. Identifying common terms across the cognitive interviews will aid in identifying the wording of items that is applicable across care settings and families of diverse backgrounds. The survey instruments and recruitment materials as well as the cognitive interview protocols for parents, ECE providers, and ECE directors are presented in Appendices A-F and O-R.

A.3. Use of Improved Information Technology and Burden Reduction

Whenever possible, advanced technology will be used to collect and process data to reduce respondent burden and make data processing and reporting more timely and efficient. A digital audio recorder will be used in all cognitive interviews. (Before using the audio recorder, participants’ verbal consent to be audio recorded will be obtained.) To reduce participant travel burden and to get a geographically diverse sample, we will conduct at least one-third of the interviews via telephone. We will send participants via email and/or text a reminder with the date, time and location of the cognitive interview, unless they indicate a preference to receive this information via airmail (see Appendix G).

A.4. Efforts to Identify Duplication and Use of Similar Information

Every effort has been made to determine whether similar measurement tools exist by searching various databases (e.g., national and scholarly), reviewing existing early care and education quality measures, and consulting with experts in the field. As we reviewed the extant literature, we did find family-provider relationship measures; however, none measured multiple domains of family-provider relationships nor were they applicable to diverse care settings and groups or appropriate for program evaluation. We have also consulted with experts in the early care and education field and they concur that the field lacks appropriate and psychometrically sound (i.e. socially desirable) measures that assess the quality of family-provider relationships and are flexible to diverse care settings and family backgrounds and applicable for use in program evaluation.

A.5. Impact on Small Businesses or Other Small Entities

It is possible that we will be collecting data from family-based service providers as well as center-based providers and directors who could be considered small businesses/entities. To reduce the impact on these settings, we will conduct cognitive interviews on days, during times of the day (e.g., evenings and weekends), and in locations convenient to them. This will help to ensure that the participation of service providers and directors from these settings does not conflict with their work responsibilities. Also, the impact, if any, on small businesses or other small entities will be reduced by the voluntary nature of the data collection.

A.6. Consequences of Collecting the Information Less Frequently

To minimize the potential burden, participants will only be asked to volunteer to participate in a single cognitive interview. Less frequent data collection would only be possible by not collecting any data at all.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances requiring deviation from these guidelines. As such, this request fully complies with regulation 5 CFR 1320.5.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

The first Federal Register notice for ACF’s generic clearance for information gathering was published in the Federal Register, Volume 76, page 34078 on June 10, 2011. The agency did not receive any comments in response to the Federal Register notice for the generic clearance.

The second Federal Register notice was published in the Federal Register, Volume 76, page 53682 on August 29, 2011.

The FPRQ project has benefited from consultation with many outside experts, including attendees of the “Family-Sensitive Caregiving and Family Engagement Working Meeting: Identifying and Measuring Common Concepts”, a meeting that was sponsored by OPRE in June 2010, and the FPRQ Technical Work Group.

Non-federal attendees of the Family-Sensitive Caregiving and Family Engagement Working Meeting were:

• Gina Adams, Urban Institute

• Don Bailey, RTI International

• Juliet Bromer, Erikson Institute

• Concha Delgado-Gaitan, Consultant

• Carl Dunst, Smoky Mountain Research Institute

• Jay Fagan, Temple University

• Nikki Forry, Child Trends

• Anne Henderson, Consultant, Annenberg Institute for School Reform

• Lee Kreader, National Center for Children in Poverty

• Michel Lahti, University of Southern Maine

• Laurie Linscott, Michigan State University

• Tammy Mann, United Negro College Fund

• Lisa McCabe, Cornell University

• Christy McWayne, Tufts University

• Diane Paulsell, Mathematica Policy Research

• Toni Porter, Bank Street College of Education

• Eva Marie Shivers, Indigo Cultural Center

• Amy Susman-Stillman, University of Minnesota

• Bobbie Weber, Oregon State University

 

And the FPRQ Technical Work Group is comprised of the following experts in the fields of measurement development, family-provider relationships, and early care and education:

• Catherine Ayoub, Harvard University

• Carl Dunst, Smoky Mountain Research Institute

• Julia Henly, University of Chicago

• Judith Jerald, Save the Children

• Elena Lopez, Harvard University

• Doug Powell, Purdue University

• Lori Roggman, Utah State University

• Julia Mendez, University of North Carolina at Greensboro

• Suzanne Randolph, University of Maryland

A.9. Explanation of Any Payment or Gift to Respondents

All participants who complete a cognitive interview will be given $50 as a token of appreciation for their participation and time spent during the interview. Child Trends has used similar incentive amounts in past studies with similar populations of parents and child care providers. For instance, parents who participated in cognitive interviews for the Redesign of the National Household Education Survey (NHES OMB Control No. is 1850-0803) received a $60 incentive. Most recently, parents and childcare providers who participated in focus groups for this study (OMB Control number 0970-0356) received $50 as a token of appreciation for their time and effort. Child Trends has found that this incentive amount helps to reduce overall recruitment costs and effort as well as facilitate the recruitment of hard-to-reach populations (e.g., racial/ethnic minorities, low-income parents, etc). This is supported by research that demonstrates that response rates increase as incentive amounts increase. For example, Kulka, Eyerman, and McNeeley (2005) conducted a study that examined the use of monetary incentive in federal surveys and they found that recruitment efforts decreased as incentive amounts increased and lower recruitment efforts resulted in lower cost per completed interview. Given our past experience with our successful recruitment efforts in tight data collection timelines and the empirical findings highlighted above, we believe that a $50 incentive amount will result in less recruitment efforts while maintaining the cost effectiveness of the study.

A.10. Assurance of Privacy Provided to Respondents

As part of the consent process for the cognitive interviews, we will have participants sign a consent form acknowledging their willingness to participate in the study. On the consent form, individuals will be made aware of the extent to which their privacy will be protected as part of the study (see Appendices A-F). Specifically, participants will be assured, verbally and on consent forms, that their names will not be documented on final reports, that their responses will not be shared with others outside of the study team, and that their personally identifiable information will not be linked to their responses during the cognitive interviews. Identifiable information will only be collected prior to the start of the cognitive interview and will not be linked to data collected during the cognitive interview. In order to protect participants’ privacy, a study-specific identification code will be assigned to each participant and will be used for all study materials.

All information collected will be kept private to the fullest extent required by law. More specifically, Child Trends (the subcontractor collecting data for the cognitive interviews) will maintain the security of the data and the privacy of participants by storing electronic data (i.e., electronic computer files, audio electronic files) in a restricted access drive. Following the completion of each cognitive interview, Child Trends project staff will transfer the audio recording over to the secure drive and delete it from the portable recorder. Hard copies of completed recruitment materials or screener interviews will be stored in locked files in locked offices at Child Trends separate from cognitive interview data files (such as transcriptions). Child Trends will also institute procedures to ensure the security of data transfer. Child Trends will immediately transfer the data onto the secure drive and delete it from the e-mail files.

A.11. Justification for Sensitive Questions

No sensitive questions will be asked as part of this data collection.

A.12. Estimates of Annualized Burden Hours and Costs

The total annualized hours for this data collection activity is estimated to be 234.53 hours.

TABLE A.1

ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST
Respondent	Respondent N	Number of responses per respondent	Average burden hours per response	Total burden hours	Average Hourly Rate	Total Annual Cost (Dollars)
Parent Instruments
Ineligible	23	1	0.09	2.07	$15.55	32.19
Eligible	36	1	2.25	81	$15.55	1,259.55
Center Director Instruments
Ineligible	6	1	0.08	.48	$17.90	8.59
Eligible	11	1	2.25	24.75	$17.90	443.03
ECE Provider Instruments
Ineligible	31	1	0.08	2.48	$10.07	24.97
Eligible	55	1	2.25	123.75	$10.07	1,246.16
TOTAL	162	1		234.53		3,014.49
*Note: We will use a recruitment matrix that includes quotas (the maximum number of participants with particular characteristics that we will accept into the sample).  Once quotas are filled, no more volunteers with characteristics of the filled quota will be accepted. This strategy will ensure sample diversity and will help us narrow the field of volunteers.

Estimates of Annualized Costs. There is an estimated annualized burden to respondents of $3,014.49.

For parent respondents, an average hourly salary of approximately $15.55 is assumed based on the Bureau of Labor Statistics (BLS) estimates for median hourly wages for high school graduates. BLS estimates were also used to estimate the hourly wages for program directors ($17.90) and child care workers ($10.07).

There will be no direct cost to the respondents other than their time to participate in the study.

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There will be no capital, operating, or maintenance costs to the respondents.

A.14. Annualized Cost to Federal Government

The total and annualized cost to the federal government for these data collection activities under the terms of the contract to develop a measure to assess family-provider relationships is estimated to be $170,999. This figure includes direct and indirect costs and fees.

A.15. Explanations for Program Changes or Adjustments

Focus groups related to this study were previously approved under OMB Control No. 0970-0356. This OMB package is for the next step (cognitive interviews) of the FPRQ project. As stated previously, the purpose of the FPRQ project is to develop a measure of the quality of family-provider relationships that will be (1) applicable across multiple types of early care and education settings and diverse program structures (including Head Start); (2) sensitive across cultures associated with racial, ethnic, and socioeconomic characteristics; and (3) appropriate for use as a program evaluation tool.

A.16. Plans for Tabulation and Publication and Project Time Schedule

Given the nature of the data collected, the analysis of the cognitive interviews will not be conducted using descriptive statistics. Instead, a summary document will be prepared for the agency’s internal use. All information collected is for internal use only and will solely be used to inform the development of the new measure of family-provider relationship quality.

All contacts with potential participants for the purpose of collecting data for the cognitive interviews will occur between April 2012 (or upon OMB approval) and May 2013 (see Table A.2). Child Trends will recruit participants from different communities, programs, and child care centers in the Washington, DC metropolitan area and across other large metropolitan areas and rural areas in the United States. Approximately three months will be allotted for recruitment and data collection in each round. Two months in between rounds will be used for conducting data collection debriefing meetings, data analysis, and edits to the instruments as needed based on the findings from the cognitive interviews. Given this timeline, recruitment efforts for Round 1 are scheduled to begin in March. We can expect to conduct our first cognitive interview for Round 1 within a few days of commencing recruitment. Data collection for Round 1 will be completed in July, 2012. Round 2 data collection is then set to begin in September, 2012 and will be completed in , December 2012. Finally, Round 3 data collection will begin in February, 2013 and will be completed in May, 2013.

TABLE A.2

Activity	Timeline
Round 1	Commencement*	Duration	Completion*
Recruitment and Data Collection	April 2012 (or upon OMB approval)	(3 months)	July 2012
Debriefing Meetings and Data Analysis	July 2012	( 2 months)	September 2012
Round 2	Commencement*	Duration	Completion*
Recruitment and Data Collection	September 2012	(3 months)	December 2012
Debriefing Meetings and Data Analysis	December 2012	(2 months)	February 2013
Round 3	Commencement*	Duration	Completion*
Recruitment and Data Collection	February 2013	(3 months)	May 2013
Debriefing Meetings and Data Analysis	May 2013	(2 months)	July 2013
*Please note these dates are approximate

There are no plans for tabulating and publishing the information gathered from this pretest process. The information that is collected will be for internal use only; however, information might be included as a methodological appendix or footnote in a report containing data from a larger data collection effort.

A.17.   Display of Expiration Date for OMB Approval

The OMB number and expiration date will be displayed at the top of the first page of the consent form that will be given to each participant in the cognitive interviews. We will read the consent form along with the OMB number and expiration date to participants at the start of each cognitive interview.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this data collection.

References

American Academy of Pediatrics (2003). Family-centered care and the pediatrician’s role. Pediatrics, 112(3), 691-696.

Bailey, D. B., Raspa, M., Humphreys, B. P., & Sam, A. M. (forthcoming). Promoting Family Outcomes in Early Intervention. Manuscript submitted for publication.

Dunst, C. J. (2002). Family-centered practices: Birth through high school. Journal of Special Education, 36, 139-147.

Johnson, B. H. (2000). Family-centered care: Four decades of progress. Families, Systems, & Health,18(2), 137-156.

Iruka, I.U. & Carver, P.R. (2006). Initial Results from the 2005 National Household Education Surveys (NHES) Early Childhood Program Participation Survey (NCES 2006-075). Washington, DC: U.S. Department of Education, National Center for Education Statistics (NCES).

Kaczmarek, L. A., Goldstein, H., Florey, J. D., Carter, A., & Cannon, S. (2004). Supporting families: A preschool model. Topics in Early Childhood Special Education, 24(4), 213-226.

Mendez, J. L. (2010). How can parents get involved in preschool? Barriers and engagement in education by ethnic minority parents of children attending Head Start. Cultural Diversity and Ethnic Minority Psychology. 16(1), 26-36.

Trivette, C. M., Dunst, C. J., & Hamby, D. W. (2010). Influences of family-systems intervention practices on parent-child interactions and child development. Topics in Early Childhood Special Education, 30(1), 3-19.

Appendices

Appendix A: FPRQ Cognitive Interview Instruments for Ineligible Parents

Appendix B: FPRQ Cognitive Interview Instruments for Eligible Parents

Appendix C: FPRQ Cognitive Interview Instruments for Ineligible Center Directors

Appendix D: FPRQ Cognitive Interview Instruments for Eligible Center Directors

Appendix E: FPRQ Cognitive Interview Instruments for Ineligible ECE Providers

Appendix F: FPRQ Cognitive Interview Instruments for Eligible ECE Providers

Appendix G: Email/Text message reminder of interview

Appendix O: FPRQ Cognitive Interviews – Center Director Cover Letter and FAQs

Appendix P: FPRQ Cognitive Interviews – Parent Cover Letter and FAQs

Appendix Q: FPRQ Cognitive Interviews – ECE Provider Letter and FAQs

Appendix R: Cognitive Interviews – Protocol for Recruitment Materials

1