Study of Coaching Practices in Early Care and Education Settings (SCOPE)
OMB Information Collection Request
New Collection
Supporting Statement
Part A
March 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, DC 20201
OPRE Points of Contact:
Wendy DeCourcey, Ph.D. (Federal Project Officer)
Tracy Carter Clopet, Ph.D. (Contract Project Specialist)
This page has been left blank for double-sided copying.
CONTENTS
A1. Necessity for the Data Collection 1
Study Background 1
Legal or Administrative Requirements that Necessitate the Collection 2
A2. Purpose of Survey and Data Collection Procedures 2
Overview of Approach 2
Overview of Purpose and Research Questions 4
Study Design 5
Universe of Data Collection Efforts 6
Descriptive study data collection activities 6
Case study data collection activities 7
A3. Improved Information Technology to Reduce Burden 8
A4. Efforts to Identify Duplication 9
A5. Involvement of Small Organizations 9
A6. Consequences of Less Frequent Data Collection 9
A7. Special Circumstances 10
A8. Federal Register Notice and Consultation 10
Federal Register Notice and Comments 10
Consultation with Experts Outside of the Study 10
A9. Incentives for Respondents 10
A10. Privacy of Respondents 10
A11. Sensitive Questions 11
A12. Estimation of Information Collection Burden 11
A13. Cost Burden to Respondents or Record Keepers 13
A14. Estimate of Cost to the Federal Government 16
A15. Change in Burden 16
A16. Plan and Time Schedule for Information Collection, Tabulation and Publication 16
Analysis Plan 16
Time Schedule and Publication 17
A17. Reasons Not to Display OMB Expiration Date 18
A18. Exceptions to Certification for Paperwork Reduction Act Submissions 18
References 19
TABLES
Table A.1. Description of data collection activities 3
Table A.2. Research questions and data sources 4
Table A.3. Data source, mode, length and timing 8
Table A.4. SCOPE expert panel membership 10
Table A.5. Total burden requested under this information collection 12
Table A.6. SCOPE honorarium structure 14
Table A.7. OMB approved projects with similar honoraria and populations 14
Table A.8. SCOPE 2018 schedule of data collection 17
APPENDICES
Appendix A. Coaching session observation protocol
Appendix B. 60-Day Federal Register Notice and Comments
Appendix C. Study recruitment materials
Appendix D. Mathematica Confidentiality Pledge
ATTACHMENTS (STUDY INSTRUMENTS)
Attachment 1. State coaching informant interview protocol
Attachment 2. ECE setting eligibility screener
Attachment 3. Center director survey
Attachment 4. Coach survey
Attachment 5. Teacher/FCC provider survey
Attachment 6. Center director case study semi-structured interview protocol
Attachment 7. Coach case study semi-structured interview protocol
Attachment 8. Teacher/FCC provider case study semi-structured interview protocol
Attachment 9. Coach supervisor case study semi-structured interview protocol
The Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS) seeks approval to collect descriptive information for the Study of Coaching Practices in Early Care and Educational Settings (SCOPE) project. The goal of this information collection is to identify how professional development coaching practices for early care and education (ECE) providers are implemented and vary in ECE classrooms serving children supported by Child Care and Development Fund (CCDF) subsidies or in settings that receive Head Start grants. This request is for conducting three data collection activities. First, we will collect information on coaching taking place in different states in spring 2018 (pending OMB approval) from state-level entities. Second, we will conduct one round of survey data collection with ECE center director, coaches, teachers, and family child care (FCC) providers in fall 2018 through winter 2019. Third, we will conduct case studies to better understand factors that influence the coaching approaches identified through the survey data collection. The case studies will include semi-structured interviews, a coaching session observation, and collection of coaching logs. The case studies will occur in fall 2019 through winter 2020.
In September 2016, the Office of Planning, Research, and Evaluation (OPRE) at ACF awarded a contract to Mathematica Policy Research to carry out the SCOPE project. Coaching has emerged as one of the most common approaches to professional development in ECE (Aikens et al. 2016) because of its potential to improve teachers’ practice. Key federal policies also suggest a role for coaching in improving teacher practice. The new Head Start Program Performance Standards require use of coaching as an approach for supporting teachers. Similarly, the Child Care and Development Block Grant (CCDBG) Act of 2014 requires States to establish professional development and training requirements for child care in such areas as implementing early learning guidelines, responding to challenging behaviors, or engaging families. Guidance on meeting these requirements identifies coaching as a possible strategy. At the state level, coaching is often deployed through Quality Rating and Improvement Systems (QRIS).
This study aims to advance understanding of implementation of core features of coaching in ECE classrooms, and how features may vary by key contextual factors and implementation drivers. This study will focus on coaching used for delivering professional development services to ECE teachers and caregivers to improve knowledge and practice in center-based classrooms and family child care homes serving preschool-age children. At the classroom/teacher level, the approach to coaching can vary by dosage, content, activities (e.g., assessment, observation, goal-setting, modeling, reflection), materials (e.g., video, written plans, resource documents) and other features (Isner et al. 2011; McGroder et al. 2014; Snyder et al. 2015; and Tout et al. 2015). However, there is no consensus as to which of these features are core to the practice of coaching. Additionally, little is known about how these features of coaching are implemented, combined, and tailored to the needs of teachers or how contextual factors or implementation drivers may mediate or moderate the effects of coaching. Ultimately, SCOPE findings will improve coaching practice in the ECE field and determine which coaching features are ripe for more rigorous evaluation.
There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.
Descriptive Study: The study team will carry out a descriptive study in a sample of seven states. Sampled states will have one or more classroom-based coaching models offered to centers and FCC homes that are serving preschool-age children and that receive Head Start grants or serve children who receive CCDF subsidies. All of the data that will be collected are listed below and summarized in Table A.1. We propose the following data collection activities to be carried out in spring 2018 through winter 2019 (after OMB approval):
State coaching informant interviews (Attachment 1): Obtain information about coaching approaches and about the available administrative data on coaching through interviews with individuals knowledgeable about coaching in the state.
ECE setting eligibility screening (Attachment 2): Determine if a setting is eligible and if personnel are willing to participate in the study.
Surveys for the descriptive study:
A center director survey (Attachment 3)
A coach survey (Attachment 4)
A teacher/FCC provider survey (Attachment 5)
As part of the descriptive study, we will also request administrative data in its existing format. We will not request specific data elements or for the administrative data to be presented in any specific format. This does not impose burden.
Case Studies: We will also conduct case studies in twelve ECE settings selected from the seven states (some of the settings may have participated in the descriptive study and others may not have). Site visit data collection for the case studies would occur from fall 2019 through winter 2020 and consist of the following activities (after OMB approval):
Semi-structured interview protocols (Attachments 6-9) with center directors, teachers/FCC providers, coaches, and coach supervisors.
Teacher/FCC provider survey (Attachment 5): The teachers and FCC providers in the case studies will be asked to complete the descriptive study teacher/FCC provider survey.
Collection of coaching logs (whenever available). There is no burden associated with collecting the coaching logs. The coaching logs are preexisting materials used by some coaches when they provide coaching to teachers.
Coaching session observations (Appendix A). There is no burden associated with the coaching session observations.
Table A.1. Description of data collection activities
Activity |
Data need |
Respondent |
Mode |
Schedule |
Descriptive Study Data Collection Activities |
||||
State coaching informant interview protocol |
Information about coaching approaches and about the available administrative data on coaching, including the range of coaching providers and approaches occurring in states and what ECE settings may be receiving classroom coaching. |
State coaching informants knowledgeable about coaching and coaching providers |
1-hour telephone interview and documents and electronic records collection |
Spring to Summer 2018 |
ECE setting eligibility screener |
Characteristics of the ECE setting and the coaching taking place in that setting; characteristics of participants in coaching |
Center director or FCC provider |
15-minute telephone interview |
Fall 2018 |
Center director survey |
Center context, supports for and challenges to coaching and professional development, structural and process features of coaching |
Center director |
30-minute web-based survey |
Fall 2018 to Winter 2019 |
Coach survey |
Coach characteristics, supports for coaching (implementation drivers), and process and structural features of coaching |
Coaches |
30-minute web-based survey |
Fall 2018 to Winter 2019 |
Teacher/FCC provider survey |
Teacher and FCC provider characteristics, supports for coaching (implementation drivers), structural and process features of coaching, coaching outputs (such as attitudes and beliefs), FCC context |
Teachers and FCC providers |
35-minute web-based survey |
Fall 2018 to Winter 2019 |
Case Study Data Collection Activities |
||||
Case study semi-structured interview protocols |
How decisions are made about coaching and what features are included; supports for coaching; organizational context and climate for coaching (e.g., organizational culture, philosophies, staffing and turnover, languages and cultures of families served); individual experiences as part of the coaching process; barriers/supports for coaching |
Center directors, FCC providers, teachers, coaches and coach supervisors in 12 locations |
30 to 90-minute In-person interviews |
Fall 2019 to Winter 2020 |
Teacher/FCC provider survey |
Teacher/FCC provider characteristics, supports for coaching (implementation drivers), structural and process features of coaching, coaching outputs (such as attitudes and beliefs), FCC context |
Teachers and FCC providers in the 12 case study locations |
35-minute web-based survey |
Fall 2019 to Winter 2020 |
Coaching logs |
Frequency of interactions between teachers/FCC providers and coaches, the focus of these interactions, and the coaching strategies employed |
Study team |
Collected in pre-existing format from coaches if coaching logs are completed as part of their feedback sessions; No associated burden |
Fall 2019 to Winter 2020 |
Coaching session observation |
Process features of coaching (coaching activities and relationship-building activities); structural features (target of coaching and content); and outputs (teacher/FCC provider engagement, actual dosage, and perceptions of the coach-teacher/FCC provider relationship) |
Study team |
Direct observations; No associated burden on those observed |
Fall 2019 to Winter 2020 |
The purpose of this collection is to understand the core features of coaching for improving classroom practice and how those coaching features are implemented on the ground in programs that receive funding from Head Start or serve children who receive CCDF subsidies (the ECE settings targeted by SCOPE). In meeting this purpose, we will answer the research questions shown in table A.2.
Table A.2. Research questions and data sources
Research questions (RQs) |
Data sources |
RQ1: How do prevalence, implementation, combining, and tailoring of core features of coaching vary across ECE classrooms? (a) How do core features vary by implementation drivers and contextual factors? (b) How are features pulled apart and/or combined to meet the needs of programs and teachers? (c) How do implemented features align with programs’ intentions? (d) Which features are ripe for more rigorous evaluation? |
Web-based surveys of ECE center directors, coaches, teachers, and FCC providers Available administrative data and documents |
RQ2: What program or systems-level contextual factors influence the implementation of coaching? (a) Which of the features identified in the conceptual model are evident in practice? And what combination of features is evident in practice? (b) Which of the supports identified in the conceptual model are evident in practice? (c) What program or systems-level contextual factors influence the implementation of coaching? (d) What outputs from the coaching (for example, teacher/FCC provider engagement in the coaching session, perceptions of coach-teacher/FCC provider relationship) are evident in practice? |
Observation of coaching sessions Semi-structured interviews with ECE center directors, coaches, coach supervisors, teachers, and FCC providers and teacher/FCC provider web-based survey Collect coaching logs |
To meet the government’s objectives for the descriptive study, sampling will be purposive to ensure variation in coaching contexts, sources of coaching, and other important variables.
We will begin by selecting states for the study. We propose focusing on states that offer a defined classroom-based coaching approach, and using that as a starting point for identification of other coaching (both defined and less defined) in the state. We identify a “defined” model as one in which there is documentation describing the model or process that a coach follows when interacting with providers (e.g., a coaching manual or protocol). The study will include coaching provided by a variety of entities in each state (for example, professional networks, states/state-funded entities, private coaching purveyors or businesses, independent consultants). The study will include coaching provided by these external providers and coaching provided by staff internal to an ECE program.
State selection will proceed in two phases. In the first phase, we will narrow the list of potential states to about nine by (1) seeking nominations from 3-6 experts and stakeholders with knowledge about classroom-coaching efforts occurring across the United States and (2) by reviewing publicly available information on coaching in states (for example, CCDF Plans, the Quality Performance Report (QPR), the QRIS Compendium, public websites).
The second phase of state selection will take place after OMB approval. Using the state coaching informant interview protocol (Attachment 1), we will reach out directly to coaching informants in each of the nine states being considered for the study. We will begin by contacting the CCDF administrator or Head Start State Collaboration Office director in each of the nine states to describe the SCOPE project and determine whether the state meets our criteria to include in the study. Based on those initial conversations, and depending on the state’s structure for providing coaching, we will contact additional coaching informants (in particular, those who fund or provide coaching) using the state coaching informant interview protocol (Attachment 1). Based on what we learn, we will select seven states for the study that have adequate variation in coaching approaches and that have coaching taking place in all the settings of interest for this study (FCCs and centers receiving Head Start grants or serving children with CCDF subsidies) so that we can answer the study’s research questions.
Once we have selected the seven states for the study, we will proceed to select the study sample. Within each state, we will seek to obtain pre-existing administrative data from organizations conducting coaching on the characteristics of the coaches and on the ECE settings receiving coaching, including whether those settings receive funding from Head Start or CCDF. We will not request specific data elements or for the administrative data to be presented in any specific format. The specific data sources will depend on the range of organizations providing the coaching identified in the state, but could include QRIS offices, state-level organizations funding coaching, or local organizations providing coaching. To learn about ECE settings that serve children who receive CCDF subsidies, we will contact CCDF agencies. We will also use Head Start Program Information Report (PIR) data on funded Head Start programs in the seven study states to select a sample of Head Start programs that vary in size, location, and inclusion of programs with FCC homes.
Once we have gathered information about ECE settings receiving classroom-based coaching in each state, we will contact them to screen for eligibility with the ECE Setting Screener (Attachment 2). To be eligible, the ECE setting must be (1) currently operating, (2) receiving funding from Head Start and/or serve children who receive CCDF subsidies, and (3) receiving classroom-based coaching for teachers of preschoolers/FCC providers caring for preschoolers. If eligible, we will assess the setting’s and coach’s willingness to participate in the study and gather information about the preschool teachers, FCC providers, and their coaches. Part B of the Supporting Statement contains more detailed information about the design of the sample.
Table A.2 provides a crosswalk between the SCOPE research questions and data sources (Attachments 1-9). For the web-based surveys, to the extent possible we drew on questions used in prior studies including First 5 LA’s Los Angeles Early Educators Advance study (LA Advance), the Early Learning Mentor Coach study (ELMC), the Head Start Family and Child Experiences Survey (FACES) 2014, the National Survey of Early Care and Education (NSECE), Head Start Classroom-based Approaches and Resources for Emotion and Social Skills Promotion (Head Start CARES), and the Texas School Ready (TSR) End-of-Year Survey as well as standardized measures of particular constructs. The web survey instruments (Attachments 3-5) are annotated to identify sources of questions from existing studies as well as questions we developed for this study. The coaching observation protocol is based on the instrument used for the Success By 6® (SB6) evaluation. Below we briefly describe each of the instruments we plan to use in the study.
State coaching informant interview protocol (Attachment 1). We will conduct telephone interviews to learn about the range of coaching being offered in the state. We will learn about the characteristics of different coaching approaches and which settings in the state may be receiving that coaching. This information will be collected during recruitment and sampling of states and, within those states, from organizations funding or providing coaching. We estimate conducting 1-hour interviews with approximately 45 informants familiar with coaching provided in their state.
ECE setting eligibility screener (Attachment 2). We will conduct a 15-minute telephone interview to gather and/or confirm information about the ECE setting, the coaching taking place in that setting, and the characteristics of participants in the coach process. These calls will also assess willingness to participate in the study. We estimate a total of 173 respondents.
Center director survey (Attachment 3). We will conduct a 30-minute web-based survey with sampled center directors. The sample will consist of 60 center directors. The survey will include questions addressing program context, supports for coaching (competency, leadership and organizational support), structural features of coaching, funding sources and demographic information.
Coach survey (Attachment 4). We will conduct a 30-minute web-based survey with sampled coaches. The sample will consist of 90 coaches who provide professional development coaching to ECE teachers and FCC providers. The survey will include questions about coaching characteristics, supports for coaching (competency, leadership and organizational support), process features of coaching, outputs and demographic information.
Teacher/FCC provider survey (Attachment 5). We will conduct a 35-minute web-based survey with a sample of 160 respondents1 (120 teachers in the 60 ECE centers and 40 FCC providers from the FCC homes that are participating in the descriptive study). The survey will focus on coaching characteristics, supports for coaching (competency, leadership and organizational support), structural and process features of coaching, outputs, outcomes, and demographic information. For FCC providers, the survey will include a module of questions about the FCC context (for example, about funding sources).
We will also request administrative data in its existing format. We will not request specific data elements or for the administrative data to be presented in any specific format. This does not impose burden.
Semi-structured interviews with ECE center directors, FCC providers and teachers, coaches, and coach supervisors (Attachments 6-9, respectively). The interviews will address how programs selected coaching features, such as the coaching content, the target of coaching, the coaching process, intended dosage, the format, and how coaching is connected with other professional development. Interviews will cover the supports for coaching and selection of features, and how they relate to resources, policies, standards, and regulations, center/FCC organizational culture, philosophies, staffing and turnover, and languages and cultures of the families served. Interviewers will also collect information on the coaches’ experiences implementing coaching and teachers’ and FCC providers’ experiences receiving coaching, as well as barriers to coaching. In addition, the 12 teachers and FCC providers participating in the interviews will complete the teacher/FCC provider survey to provide additional information on coaching experiences (Attachment 5).
Coaching observations. We will conduct observations of one coaching feedback session between coaches and teachers or FCC providers in the 12 sites (Appendix A). These direct observations will measure process features of coaching, such as coaching activities and relationship-building activities; structural features, such as the target of coaching and content; and outputs, such as teacher/FCC provider-coach rapport or engagement. We will record with whom the coaches interact, the activities (such as assessment, goal-setting, feedback, and modeling), and the content covered (such as behavior management, or curriculum topics such as literacy or mathematics). There is no burden associated with the coaching observations.
Coaching logs. We will request any pre-existing coaches’ logs of their coaching sessions with teachers and FCC providers from the 12 coaches who participate in the case study if the coaches are keeping logs. We will use coaching logs to measure the frequency of interactions between teachers/FCC providers and coaches, the focus of these interactions, and the coaching strategies employed. There is no burden associated with collecting the coaching logs.
The data collection plan is designed to obtain information in an efficient way that minimizes respondent burden. When feasible, we will gather information from existing data sources, using the most efficient methods available. Table A.3 provides information on the source, mode, length, and timing for each data collection activity.
We will ask ECE center directors, coaches, teachers, and FCC providers to complete a web-based survey. The web-based surveys will enable respondents to complete the data collection instrument at a location and time of their choice, and its built-in editing checks and programmed skips will reduce the level of response errors. We will also offer participants the choice to respond to the survey by phone if they prefer.
We will ask state coaching informants to provide electronic copies of records. Whenever possible, states and programs will upload the files to a secure Mathematica File Transfer Protocol (FTP) site. We will also request administrative data in its existing format. We will not request specific data elements or for the administrative data to be presented in any specific format. This does not impose burden.
Table A.3. Data source, mode, length and timing
Data source |
Mode, length, and timing |
Respondent group |
State coaching informant interview protocol |
1 hour telephone interviews with state coaching informants and available administrative data requested from states and coaching programs in Winter/Spring 2018 |
State coaching informants knowledgeable about coaching |
ECE setting eligibility screener |
15-minute telephone interview to determine if the setting meets eligibility requirements in fall 2018 |
Center directors and FCC providers |
Center director web-survey |
30-minute web-based survey, with telephone option if respondent requests it, administered in Fall 2018/Winter 2019 |
Center directors |
Coach web-survey |
30-minute web-based survey, with telephone option if respondent requests it, administered in Fall 2018/Winter 2019 |
Coaches |
Teacher/FCC provider web-survey |
35-minute web-based survey, with telephone option if respondent requests it, administered in Fall 2018/Winter 2019 (for the descriptive study) and Fall 2019/Winter 2020 (for the 12 teachers and FCC providers in the case studies) |
Teachers and FCC providers |
Case study semi-structured interview protocols |
30 to 90 minute in-person interviews with each respondent in Fall 2019/Winter 2020 |
Center directors, FCC providers, teachers, coaches and coach supervisors |
There is no other current or planned effort to collect information on how coaching features are implemented and vary in center-based classrooms and FCC homes serving preschool-age children and supported by CCDF subsidies or Head Start grants.
None of the study instruments ask for information that can be obtained from alternative data sources including administrative data. We will use publicly available data and existing administrative information as much as possible, primarily for constructing the sample frame and for identifying some program characteristics. The design of the study instruments ensures minimal duplication of data collected across instruments and does so only in cases where we need the perspective of more than one type of respondent to answer specific research questions. For example, the survey of teachers and FCC providers will include a subset of the questions administered to coaches related to features of coaching to establish what their own coaching experiences entail.
Most of the child care centers and all of the FCC homes included in the study will be small organizations, including community-based organizations and other nonprofits. We will minimize burden for respondents by offering a web-survey that respondents can complete at their convenience and by restricting the length of the web-survey. We will schedule the in-person case study interviews at times and locations that are convenient for the respondents selected to be interviewed.
The web-surveys are a one-time data collection activity. These data collection activities are necessary to ACF to gain a better understanding of the core structural and process features of coaching, how those features are combined, and how implementation drivers and contexts influence these features. Study findings will both provide the basis for a rigorous evaluation of core features of coaching and improve coaching practice in the ECE field.
There are no special circumstances for the proposed data collection efforts.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on December 22, 2017; Vol. 82; No. 245; Page 60746 and provided a sixty-day period for public comment. A copy of this notice is attached as Appendix B. During the notice and comment period, two comments were received. The comments and ACF responses are attached in Appendix B.
We consulted with experts to complement the knowledge and experience of the team (Table A.4). Consultants included researchers with expertise in professional development services in ECE programs and child care more broadly. Throughout the study we will continue to work with expert consultants. We also consulted key stakeholders (for example, individuals from professional organizations) who offer national perspectives on classroom-based coaching.
Table A.4. SCOPE expert panel membership
Name |
Affiliation |
Kathleen Artman Meeker |
College of Education, University of Washington |
Juliet Bromer |
Herr Research Center for Children and Social Policy, Erikson Institute |
Bridget Hamre |
Center for Advanced Study of Teaching and Learning, Curry School of Education, University of Virginia |
Annemarie Hindman |
College of Education, Temple University |
Lisa McCabe |
Bronfenbrenner Center for Translational Research, Cornell University |
Douglas Powell |
Department of Human Development & Family Studies, College of Health and Human Sciences, Purdue University |
Rebecca Bulotsky Shearer |
Department of Psychology, University of Miami |
Christina Weiland |
School of Education, University of Michigan |
There are no incentives for respondents in this data collection.
Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law (this is described for respondents in the recruitment materials included in Appendix C and in each of the data collection instruments).
As specified in the contract signed by ACF and Mathematica (referred to as the Contractor in this section), the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ personally identifiable information. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. All Mathematica employees sign a Mathematica Confidentiality Pledge (Appendix D) that emphasizes the importance of confidentiality and describes employees’ obligations to maintain it.
As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system and establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information that ensures secure storage and limits on access. Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.
There are no sensitive questions in this data collection.
Table A.5 provides an estimate of time burden for the data collections, broken down by instrument and respondent. These estimates are based on our experience collecting information from states and administering surveys to center directors, coaches, teachers, and FCC providers. We expect the total annual burden to be 157 hours.
Table A.5. Total burden requested under this information collection
Instrument |
Total Number of Respondents |
Annual Number of Respondents |
Number of Responses Per Respondent |
Average Burden Hours Per Response |
Annual Burden Hours |
Average Hourly Wage |
Total Annual Cost |
State coaching informant interview protocol |
45 |
23 |
1 |
1 |
23 |
$34.52 |
$793.96 |
ECE setting screener (center directors and FCC providers)a |
173 |
87 |
1 |
0.25 |
22 |
$22.91 |
$504.02 |
Center director survey |
60 |
30 |
1 |
0.5 |
15 |
$34.52 |
$517.80 |
Coach survey |
90 |
45 |
1 |
0.5 |
23 |
$23.45 |
$539.35 |
Teacher/FCC provider surveyb |
172 |
86 |
1 |
0.58 |
50 |
$11.30 |
$565.00 |
Center director semi-structured interview protocol |
12 |
6 |
1 |
1.5 |
9 |
$34.52 |
$310.68 |
Coach semi-structured interview protocol |
12 |
6 |
1 |
1 |
6 |
$23.45 |
$140.70 |
Teacher/FCC provider semi-structured interview protocol |
12 |
6 |
1 |
1 |
6 |
$11.30 |
$67.80 |
Coach supervisor semi-structured interview protocol |
12 |
6 |
1 |
0.5 |
3 |
$23.45 |
$70.35 |
Estimated Annual Burden Total |
157 |
|
$3,509.66 |
||||
a The average hourly wage of $22.91 for the ECE setting screener is the average of the wage for center directors ($34.52) and FCC providers ($11.30).
b The total and annual number of respondents for the teacher/FCC provider survey includes the 160 teachers and FCC providers who will complete this survey for the descriptive study and the 12 who will complete this survey for the case studies.
We expect the total annual cost to be $3,509.66 for all of the instruments in the current information collection request.
Average hourly wage estimates for deriving total annual costs are based on Current Population Survey data for 2016 (Bureau of Labor Statistics 2016). For each instrument included in Table A.5, we calculated the total annual cost by multiplying the annual burden hours and by the average hourly wage.
For state coaching informants administrators, center directors, coaches, and teachers we use the median usual weekly earnings for full-time wage and salary workers of age 25 and older. We divided weekly earnings by 40 hours to get hourly wages.
We propose to offer respondents in the descriptive study and case studies an honorarium to acknowledge their efforts and to be respectful of their time spent participating in data collection activities. To develop honoraria amounts we considered average hourly wages across members of the target population as well as experiences in other data collections with the same target population. In general, we aim to provide honoraria that are both similar to the hourly wage averaged cross members of the target population and that are at or below the amounts used in prior data collections. Considering the average hourly wages, the length of the data collection activities, and the potential disruption to the schedules of the targeted respondents from participation, we developed the honorarium structure in table A.6.
Table A.6 shows the proposed honoraria amounts as well as the average hourly wage for members of the target population (Bureau of Labor Statistics 2016) and the proposed length of each data collection activity. Average hourly wages for members of the target population range from $11.30 (for teachers and FCC providers) to $34.52 (for center directors); the average across these amounts is $23.09 (which is slightly higher than the $20 honorarium being offered to study participants for most of the study’s data collection activities). Respondents will need to complete the descriptive study data collection activities outside of working hours because of their child care responsibilities. We have identified a higher honorarium for FCC providers for the descriptive study because they tend to work longer hours than center teachers and may not have the support of another adult in their setting (Moiduddin et al. 2015), so it will be more challenging for them to set aside the time to complete the survey. For the case studies, we propose to offer honoraria to individual respondents and to the ECE settings in which we conduct the case studies. The case study semi-structured interviews with ECE center directors, FCC providers and teachers, coaches, and coach supervisors will occur at the ECE setting during the work day. This will require disrupting participants’ schedules and may require other staff to cover for a respondent participating in the semi-structured interview. We will also need the assistance of the ECE setting to schedule the semi-structured interviews and coaching observations.
In addition to average salaries among members of the proposed target population, we also considered the approach used in other data collections with the same target population when developing the honorarium structure. These data collections are shown in table A.7. Looking across these studies, the SCOPE honoraria typically fall at the same level or below those in other studies for activities of the same or similar length. For example, in FACES 2014 (OMB 0970-0151), teachers completed the 5E-Early Ed pilot survey which took approximately 20 minutes to complete; they received $20. For the Q-CCIIT psychometric field test (OMB 0970-0392), teachers and FCC providers received a $25 gift card for completing a 20-minute questionnaire. In the National Survey of Early Care and Education 2012 (OMB 0970-0391), center-based providers (center directors) and FCC providers received $35 for completing an approximately 30-minute survey. While this amount is higher than what SCOPE is offering to teachers and center directors, it is similar to what SCOPE is offering to FCC providers (SCOPE is offering $40 to FCC providers for a 35-minute survey, while this study offered $35 for a 30-minute survey). For Baby FACES 2009 (OMB 0970-0354), programs (the ECE setting) received $500 for the program’s overall participation in the study, which is the same approach as the $250 we propose to offer to the ECE settings for their participation in the case study semi-structured interview and for providing support with coordinating the onsite interviews and observations.
Table A.6. SCOPE honorarium structure
Data collection activity |
Respondent |
Length of activity |
Average hourly wage |
Honorarium |
Center director survey |
Center directors |
30 minutes |
$34.52 |
$20 |
Coach survey |
Coaches |
30 minutes |
$23.45 |
$20 |
Teacher/FCC provider survey |
Teachers |
35 minutes |
$11.30 |
$20 |
Teacher/FCC provider survey |
FCC providers |
35 minutes |
$11.30 |
$40 |
Case study data collection coordination |
ECE setting |
1.5 daysa |
n.a. |
$250 |
Case study semi-structured interview |
Center directors |
90 minutes |
$34.52 |
$20 |
Case study semi-structured interview |
Coaches |
60 minutes |
$23.45 |
$20 |
Case study semi-structured interview |
FCC providers and teachers |
60 minutes |
$11.30 |
$20 |
Case study semi-structured interview |
Coach supervisors |
30 minutes |
$23.45 |
$20 |
a The 1.5 days represents the total amount of time the project team would be on site at ECE settings for the case study data collection; this does not refer to burden for any one respondent. The burden for each case study respondent can be found in the rows of this table that refer to the case study semi-structured interviews, and in table A.5.
Table A.7. OMB approved projects with similar honoraria and populations
Project |
Respondent |
Average hourly wage |
Honorarium |
Mode |
Length |
Timing |
Response rate |
Q-CCIIT psychometric field test (OMB 0970-0392) |
Center-based caregivers and FCC providers |
$11.30 |
$25 |
Paper-and-pencil hardcopy survey |
15 minutes |
Fall 2012 |
97% |
FACES 2014 OMB (0970-0151) |
Head Start teachers |
$11.30 |
$20 |
Web survey with paper-and-pencil hardcopy option |
10 minutes |
Spring 2015 |
88% |
Baby FACES 2009 (OMB 0970-0354) |
ECE setting |
n.a. |
$500 for the ECE setting each year of the study |
Classroom observations, paper-and-pencil teacher-child reports, and program director interviews |
Classroom observations (2.5 hours); Teacher-child reports (10 minutes); Program director interview (45 minutes) |
Spring 2009, spring 2010, spring 2011, spring 2012 |
95% to 100%a |
National Survey of Early Care and Education 2012 (0970-0391) |
Listed home-based providers including FCC providers |
$11.30 |
$35 |
Web-based survey, with paper-and-pencil option if respondent requests it |
30 minutes |
Spring 2012 |
81% |
National Survey of Early Care and Education 2012 (0970-0391) |
Center-based providers (directors) |
$34.52 |
$35 |
Web-based survey, with paper-and-pencil option if respondent requests it |
30 minutes |
Spring 2012 |
74% |
FACES 2014 (OMB# 0970-0151) |
Head Start teachers |
$11.30 |
$10 |
Web-based survey, with paper-and-pencil option if respondent requests it |
10 minutes |
Fall 2014 and spring 2015 |
Fall 98% Spring 95% |
ECE ICHQ (OMB# 0970-0499 |
Teachers and center directors |
Teacher: $11.30
Director: $34.52 |
$10 |
Web-based survey, with paper-and-pencil option if respondent requests it |
15 minutes |
Fall 2017 |
80% (in process) |
a Response rates for classroom observations were 95% (2009) and 98% (2010). There were no observations conducted in 2011 and 2012; response rates for teacher-child reports were 95% (2009), 96% (2010), 96% (2011), and 98% (2012); response rates for the program director interview was 100% 2009, 2010, and 2011. Program director interviews were not conducted in 2012.
The total cost for the data collection activities under this current request will be $846,117. Annual costs to the Federal government will be $423,059 for the proposed data collection.
This is a new data collection.
The instruments included in this OMB package will yield data that we will analyze using quantitative and qualitative methods. We will carefully link the research questions guiding the study with the data collected, constructs measured, and analyses undertaken. To describe the core features of coaching, implementation drivers, and contextual factors, we will use data from the web surveys. We will construct summary variables and scales and use the appropriate techniques to assess the quality and psychometric properties of these constructed variables. The constructed variables will typically involve combining information from several sources or multiple items within a single data collection instrument. For example, we will construct scales for relationship building based on teacher and coach surveys or scales for organizational culture/climate based on teacher, coach, and director surveys. Analytic approaches for answering SCOPE research questions include descriptive statistics (means and percentages), factor analysis, cluster analysis, discriminant analysis, and Hierarchical Linear Modeling (HLM) analysis.
To describe key factors that influence coaching and the coaching process, we will use data from the case studies. To begin, two trained qualitative researchers will code and analyze the data to identify emerging themes. Researchers will begin by using a collective and individual process of open coding, which allows for codes or themes to emerge. Researchers will meet to review codes and reconcile any discrepancies through consensus. Initial themes may begin to emerge during these discussions and will be documented. During the analysis process, the initial codebook will be adjusted as necessary to reflect the discussion. This coding process will continue in a similar manner until all transcripts are coded. Once the data are coded, the team will analyze the data by research question, gauge consistency across respondents within sites, and identify themes. We will also use the coaching observations and logs to calculate frequencies to integrate into the analysis. For coaching observations, quantitative information (e.g., the presence or absence of coaching features during the observation period, frequency of observed behaviors, ratings of the interactions) will be entered into a dataset, verified, and analyzed. We will use the analysis of the coaching logs to supplement the information we gain from the observations and interviews, further identifying themes and trends in the coaching activities and features. We will enter the quantitative information from the logs into a dataset and calculate descriptive information.
In the Supporting Statement Part B, we describe the analytic approaches for each research question specifically.
Table A.8 contains the timeline for the data collection and reporting activities. After obtaining OMB approval, recruiting of states and state coaching informants will begin in spring or summer of 2018, and recruiting of ECE settings and coaching participants will begin in fall or winter 2018. Data collection in centers and FCC homes will follow and is expected to occur between September 2018 and March 2019 for the web-based surveys and December 2019 to March 2020 for the on-site coaching observations and staff interviews. Mathematica will produce several publications based on analysis of data from these activities.
We will prepare brief profiles of coaching in the states in which state coaching information interviews are conducted.
We will prepare a set of tables describing findings from all surveys. The intention is to quickly produce findings that can be used by the government.
We will prepare a final report on the surveys that includes the information from the descriptive tables along with more narrative explanation of the findings. The format of the report will be accessible to a broad audience and will use graphics and figures to communicate key findings.
We will prepare an overview of themes that emerge from the case studies and supporting evidence so that the findings can also be quickly used by the government.
We will prepare a final report on the case studies that includes information on themes and supporting evidence along with a narrative explanation of the findings. The format of the report will be accessible to a broad audience and focus on conveying key findings.
We will produce briefs of varying lengths on specific topics of interest to the government. These briefs will be accessible to a broad audience.
Table A.8. SCOPE 2018 schedule of data collection
Activity |
Timinga |
Recruitment |
|
State recruitment/coaching informant interviews |
Spring/Summer 2018 |
Program (center and FCC home) recruitment |
Fall 2018 |
Data collection |
|
State coaching informant interview |
Spring/Summer 2018 |
Center director survey |
Fall 2018/Winter 2019 |
Coach survey |
Fall 2018/Winter 2019 |
Teacher/FCC provider survey |
Fall 2018/Winter 2019 |
Case study on-site classroom observations, staff interviews, and teacher/FCC provider surveys |
Fall 2019/Winter 2020 |
Analysis |
|
Data processing and analysis of survey results |
Spring/Summer 2019 |
Data processing and analysis of case study classroom observations, staff interviews, and teacher/FCC provider surveys |
Spring/Summer 2020 |
Reporting |
|
Profiles of coaching for states participating in coaching informant interviews |
Summer 2018 |
Data tables on web surveys |
Summer 2019 |
Final report on web survey data collection |
Summer/Fall 2019 |
Summary of themes from case studies |
Summer 2020 |
Final report on all data collection activities |
Summer/Fall 2020 |
Briefs on specific topics |
TBD |
aAfter obtaining OMB approval
All instruments will display the expiration date for OMB approval.
No exceptions are necessary for this information collection.
Aikens, N., L. Akers, and S. Atkins-Burnett. “Professional Development Tools to Improve the Quality of Infant and Toddler Care: A Review of the Literature.” OPRE Report 2016-96. Washington, DC: Office of Planning, Research & Evaluation, Administration for Children & Families, U.S. Department of Health and Human Services, 2016.
Bureau of Labor Statistics. “Usual Weekly Earnings of Wage and Salary Workers: Third Quarter 2016.” USDL-16-2025. Washington, DC: Bureau of Labor Statistics, October 2016.
Isner, T., K. Tout, M. Zaslow, M. Soli, K. Quinn, L. Rothenberg, and M. and Burkhauser. “Coaching in Early Care and Education Programs and Quality Rating and Improvement Systems (QRIS): Identifying Promising Features.” Washington, D.C.: Child Trends, February 2011.
McGroder, S. M., E.C. Howard, M. Fishman, V.E. Rankin, and F.K. Helsel. “Putting the Pieces Together: A Program Logic Model for Coaching in Head Start from the Descriptive Study of the Head Start Early Learning Mentor Coach Initiative.” No. 2014-06. U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation, 2014.
Moiduddin, Emily, Elizabeth Cavadel, Yange Xue, and Owen Schochet. “A Portrait of Early Educators in First 5 LA’s Workforce Development Programs: Their Characteristics and Views on Professional Development.” Issue brief submitted to Frist 5 LA. Washington, DC: Mathematica Policy Research, October 2015.
Snyder, Patricia A., Mary L. Hemmeter, and Lise Fox. “Supporting Implementation of Evidence-Based Practices through Practice-Based Coaching.” Topics in Early Childhood Special Education, vol. 35, no. 3, 2015, pp. 133-143.
Tout, K., D. Epstein, M. Soll, and C. and Lowe. “A Blueprint for Early Care and Education Quality Improvement Initiatives: Final Report.” Child Trends, March 2015.
This page has been left blank for double-sided copying.
1 Table A.7 identifies 172 respondents for the teacher/FCC provider survey. That total includes the 160 teachers and FCC providers who will complete this survey for the descriptive study (discussed here) and the 12 who will complete this survey for the case studies (discussed under case study data collection activities).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Study of Coaching Practices in Early Care and Education Settings OMB Supporting Statement Part A |
| Subject | OMB |
| Author | Emily Moiduddin |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |