Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Supporting and Strengthening the Home Visiting Workforce (SAS-HV) :
Online Pretest of Draft Reflective Supervision Measure
Pre-testing of Evaluation Data Collection Activities
0970 - 0355
Supporting Statement
Part B
June 2023
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Nicole Denmark
Shirley Adelstein
Part B
B1. Objectives
Study Objectives
The purpose of the Supporting and Strengthening the Home Visiting Workforce (SAS-HV) project is to advance understanding of how to support and strengthen the early childhood home visiting workforce. One focal area of the project is the use of reflective supervision in early childhood home visiting. The Maternal Infant Early Childhood Home Visiting (MIECHV) legislation mandates federally funded home visiting programs maintain high-quality supervision practices, but relatively little is known about specific supervision practices used in the field. This research seeks to address a key gap by developing and testing a measure of reflective supervision that is practice-relevant and useful for research.
Prior phases of this project developed a conceptual model of reflective supervision; reviewed current research, measures, and practice; completed concept mapping data collection and solicited input from the technical workgroup (TWG), practitioner workgroup (PWG), and home visiting model representatives; and developed a draft set of items and response options for the measure1. As a next step in the measure development process, the Administration for Children and Families (ACF) is seeking approval to pretest the draft measure with a sample of home visiting supervisors to use in preliminary descriptive analyses.
Generalizability of Results
This study is intended to produce descriptive data and participant feedback on the draft reflective supervision measure to guide refinements prior to larger scale pilot testing and validation. The study is not intended to promote statistical generalization to other sites or service populations.
Appropriateness of Study Design and Methods for Planned Uses
The design includes web-based pretesting. The web-based pretesting will include the draft reflective supervision items and brief feedback questions. The use of a web-based pretesting approach is appropriate to collect a) measure responses from a small sample of participants for descriptive analyses and b) experiences, observations, and feedback on the clarity, acceptability, and perceived value of the measure. Web-based pretesting with feedback probes can identify similar issues with measure items as cognitive interviews and can be accomplished more quickly, with less burden, and can produce a preliminary dataset of responses.2
This data is not intended to be a representative sample of home visiting supervisors; however, it will allow us to learn from a subset of practitioners to further inform measure development.
As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
B2. Methods and Design
Target Population
We propose to gather information from home visiting supervisors. We aim to obtain variation in participant characteristics, roles, length of time working as home visiting supervisors, home visiting program models implemented, and type of families served to understand a range of perspectives in relation to reflective supervision. Inclusion criteria for participants includes at least one year of experience with reflective supervision in the home visiting context. The universe of home visiting practitioners is not known, although estimates suggest more than 2,400 home visiting supervisors3 are currently working in the field. We will select up to 50 total participants, focused on supervisors implementing home visiting models that are eligible for MIECHV funding.
This sample size is consistent with that of other studies that have used measure pretesting for measure development.4 The sample will include individuals who respond to a recruitment announcement sent to the Home Visiting Applied Research Collaborative (Appendix 1), complete a screening questionnaire (Instrument 1), and who are eligible for participation based on inclusion criteria. Because participants will be purposively selected, they will not be representative of the population of supervisors within the home visiting field.
Respondent recruitment
We will recruit participants to the measure pretesting through the Home Visiting Applied Research Collaborative’s (HARC) Practice-Based Research Network (PBRN). The purpose of HARC’s PBRN is to provide a national network for conducting collaborative, field-initiated studies with local home visiting programs, regardless of the model of home visiting being used. The PBRN is a voluntary network of hundreds of local home visiting programs, including individual sites and state and local networks.
If more individuals respond than are planned for the data collection activities, we will select participants purposively with the goal of achieving diverse and balanced representation across the categories listed below:
Race and ethnicity
Geographic location
Home visiting model(s) implemented
Types of communities served (urban, suburban, rural, frontier)
Number of families served by home visiting program
Primary language of families served by the home visiting program
Length of time as a home visiting supervisor
If more respond than we can accommodate, we may randomly select participants within a given category.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
Data collection includes completing the reflective supervision measure and accompanying feedback questions. The development of the draft measure was guided by a systematic review of existing elements of reflective supervision and informed by the Study Team’s knowledge of existing research and literature on reflective supervision. Elements of reflective supervision represented in the measure were first refined in collaboration with technical and practitioner workgroup members. The Study Team then conducted a concept mapping process with practitioners and researchers, including a group interpretation meeting. Informal cognitive interviews were conducted to identify issues with interpretation, definitions, and response options. The web-based measure pretesting will include the full item pool with targeted feedback questions for each section of the reflective supervision measure.
B4. Collection of Data and Quality Control
The Study Team will collect measure pretesting data from individuals that consent to participate. ACF has contracted with James Bell Associates and partners Johns Hopkin University and University of Colorado Anschutz Medical Campus to conduct the study. James Bell Associates will lead all data collection and analysis activities in collaboration with Johns Hopkin University and University of Colorado Anschutz Medical Campus.
Recruitment of supervisors into the study will involve outreach to potential participants through Home Visiting Applied Research Collaborative recruitment announcements (Appendix 1). Interested home visiting supervisor participants will complete a screening questionnaire (Instrument 1). The Study Team will then send a link via email to selected participants to complete the web-based measure pretesting (Instrument 2). The measure and accompanying feedback questions will be programmed into Qualtrics, web-based survey software.
To monitor measure pretesting for quality and consistency, the Study Team will review reports on survey completion rates and item completion rates periodically during the data collection period.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
Based on experience recruiting participants via the Home Visiting Applied Research Collaborative, we anticipate being able to successfully recruit the desired number of participants described in section B2: Respondent Recruitment. We assume a minimum of 60% of the individuals who are invited to participate will complete the web-based measure pretesting. To account for this response rate, the study team will identify up to 30 alternate participants in addition to the 50 individuals selected for participation. The study team will reach out to the pool of alternate participants as needed, to reach a total of 50 participants. The measure pretesting is not designed to produce statistically generalizable findings and participation is wholly at the participant’s discretion. Response rates will not be calculated or reported.
Non-response
As participants will not be randomly sampled and findings are not intended to be representative, nonresponse bias for the web-based measure pretesting not be calculated. Participant demographics will be documented and reported in written materials associated with the data collection, however, we will not have demographic information on the universe of home visiting supervisors to compare with.
B6. Production of Estimates and Projections
Data will not be used to generate population estimates, either for internal use or dissemination.
B7. Data Handling and Analysis
Data Handling
To minimize errors in survey data processing and analysis, we will review all data during initial processing before data is approved and becomes part of the final data set. Additionally, the Qualtrics web-based software employs quality controls to ensure only valid responses are allowed and to minimize missing responses.
Data Analysis
The following descriptive methods will be used to analyze the data.
Descriptive quantitative methods: We will analyze the responses to the measure pretesting using statistical software. Analyses of measure items will include:
Descriptive statistics to check for item variability and skewness
Preliminary reliability and validity analyses as appropriate, i.e. item correlations (Cronbach’s alpha) within hypothesized elements (factors)
These analyses will provide preliminary evidence of whether items detect variation in reflective supervision practices and whether items developed to represent a particular element of reflective supervision group together. The use of these analyses, however, will be limited by the small size of the pretest sample and will not generally provide precise estimates. Instead, they will provide preliminary indicators that can help focus subsequent design decisions to be pilot tested with a larger developmental sample in the next phase of the project. A separate information collection request will be submitted to OMB for this next phase of testing and data collection.
Analyses of closed-ended feedback questions accompanying the items in the draft measure will include descriptive statistics (frequencies and cross tabulations by select participant characteristics, i.e. race or ethnicity, and amount of reflective supervision experience). These responses will also provide indications of home visiting supervisors’ perspectives on the acceptability and relevance of the reflective supervision measure.
Data Use
Data will be used to refine and possibly reduce items for the measure of reflective supervision for home visiting. No dissemination products will be developed based on the study findings. However, findings will inform development of a manual to be used for internal purposes and future testing of the measure. In future publications regarding the measure, we will describe how the current information collection contributed to the draft measure.
B8. Contact Persons
The information for this study is being collected by James Bell Associates, Johns Hopkin University, and University of Colorado Anschutz Medical Campus on behalf of ACF. Principal Investigator Allison West (awest25@jhu.edu), Project Director Mariel Sparr (sparr@jbassoc.com) and Task Lead Nancy Whitesell (nancy.whitesell@ucdenver.edu) led development of the study design plan and data collection protocols and will oversee collection and analysis of data.
The agency responsible for receiving and approving contract deliverables is:
The Office of Planning, Research, and Evaluation (OPRE),
Administration for Children and Families (ACF)
U.S. Department of Health and Human Services
The Federal project officers for this project are Nicole Denmark and Shirley Adelstein.
Attachments
Appendix 1: Home Visiting Applied Research Collaborative Recruitment Announcement for Web-based Measure Pretesting
Instrument 1: Screening Questionnaire for Web-based Measure Pretesting
Instrument 2: Web-Based Measure Pretesting Questions
1 Information collection activities were approved by OMB under OMB #0970-0355 on August 19, 2022, with the title Supporting and Strengthening the Home Visiting Workforce (SAS-HV).
2 Lenzner, T., and Neuert, C.E. (2017). Pretesting survey questions via web probing – Does it produce similar results to face-to-face cognitive interviewing? Survey Practice 10 (4). https://doi.org/10.29115/SP-2017-0020.
3 National Home Visiting Resource Center. (2022). 2022 Home Visiting Yearbook. James Bell Associates and the Urban Institute.
4 Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: a primer. Frontiers in Public Health, 6, 149.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mariel Sparr |
| File Modified | 0000-00-00 |
| File Created | 2024-07-25 |