SSA GenIC Pretesting - SAS-HV : Online Pretest of Draft Reflective Supervision Measure v2

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Supporting and Strengthening the Home Visiting Workforce (SAS-HV): Online Pretest of Draft Reflective Supervision Measure

Pre-testing of Evaluation Data Collection Activities

0970 – 0355

Supporting Statement

Part A

June 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Nicole Denmark

Shirley Adelstein

Part A

Executive Summary

Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (0970-0355).

Progress to Date: A concept mapping study design with home visiting practitioners and researchers was conducted to inform the development of a measure of reflective supervision for home visiting. The data collected from a survey and group interpretation meeting identified what end users of the measure view as key elements of reflective supervision in the home visiting context. Results were used to assemble a pool of items that will be tested in this information collection. Previous data collection activities were approved under a different pretesting generic OMB clearance.  

Description of Request: This is a request to conduct a pretest survey of a reflective supervision measure with up to 50 home visiting supervisors. The data collected from the pretest of the measure will help assess clarity, acceptability, and perceived value of the draft items. We will also calculate basic descriptive statistics (central tendency and variability) of the measure to examine item performance. Results of this phase will inform further refinement and/or removal of items that are not clear or demonstrate limited variability.

We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

The Maternal Infant Early Childhood Home Visiting (MIECHV) legislation mandates federally funded home visiting programs maintain high-quality supervision practices, and reflective supervision is endorsed in the most recent MIECHV formula funding guidance. This refers to reflective supervision for home visitors to support their work with families. Reflective supervision is a specific type of professional development that is intended to help home visitors (1) develop knowledge, skills, and key competencies to carry out their roles and (2) support and help restore home visitor professional well-being.¹ Despite strong theoretical support for reflective supervision, there is limited understanding of how it is implemented in practice and limited evidence of effectiveness. This is due, in part, to a lack of valid, reliable measures of reflective supervision. Valid, reliable measures of reflective supervision are important for advancing research on the role of reflective supervision in supporting home visitors’ work with families.

The purpose of the Supporting and Strengthening the Home Visiting Workforce (SAS-HV) project is to advance understanding of how to support and strengthen the early childhood home visiting workforce. A prior phase of the SAS-HV project developed a conceptual model of reflective supervision and reviewed current research, measures, and practice to identify gaps in knowledge. A second phase of the project elicited key elements of reflective supervision from practitioners and researchers using a concept mapping study design². This current phase will address a key gap identified by testing and further developing the reflective supervision measure. This collection is a necessary step to develop a measure of reflective supervision for home visiting that will be primarily used for research and informed by practitioner experience and perspectives, with promising secondary use for practice.

There are no legal or administrative requirements that necessitate the collection. The Administration for Children and Families (ACF) is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The purpose of this information collection is to test the preliminary item pool and response categories to identify how to improve the reflective supervision measure for larger-scale pilot testing and validation. In the current phase of the measure development process, ACF is seeking approval to pretest the measure with home visiting supervisors to assess the measure for clarity, acceptability, and perceived value, and to identify items in need of refinement. Our process for screening potential participants according to established inclusion criteria will help ensure participants reflect the characteristics of potential end users of the measure.

This proposed information collection meets a primary goal of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions

We plan to seek input from a broad, diverse array of home visiting supervisors, which will be analyzed to inform the refinement of the draft measure. The research questions we are seeking to address are as follows:

• Do draft items function as intended in a sample of home visiting supervisors?

• What are home visiting supervisors’ perceptions of the draft reflective supervision measure, in terms of its clarity, acceptability, and perceived value?

Study Design

In earlier project phases, the Study Team identified a list of reflective supervision elements then completed concept mapping data collection with practitioners (home visitors and supervisors) and researchers to sort, rate, and interpret these data. The Study Team then identified items for inclusion in the measure. This identification process involved a careful review of existing measures and input from the technical workgroup (TWG), practitioner workgroup (PWG), and home visiting model representatives. These items and input were integrated with the concept mapping findings. The Study Team also drafted response options for each item with consultation from the TWG, while considering measurement best practices and implications for analyzing data and psychometric properties of the measure. In this phase of the project, the Study Team will pretest the item pool and response options with a sample of home visiting supervisors to use in preliminary descriptive analyses. Results will inform future testing with a larger sample (which will be submitted as a separate information collection request).

This phase includes two main information collections; Table 1 provides a summary overview of these activities:

1. Screening questionnaire: In the questionnaire, participants will be asked to provide information about their race, ethnicity, primary language, and years of experience with home visiting supervision and reflective supervision practices. They will also be asked to provide information about the location of the home visiting program they work for, number of families served by the home visiting program, race and ethnicity of families served, primary language of families served, whether they work in a tribal home visiting program, and program model(s) implemented. Demographic information is necessary to ensure that participants from certain populations and participants that work with certain populations, including people of color and others who have been historically underserved and marginalized, are included in study activities and their voices are heard.  

2. Web-based measure pretesting with feedback questions: Participants will be asked to 1) complete the current draft of the reflective supervision measure 2) complete embedded questions on their experience, observation, and feedback on the draft measure.

Table 1. Data Collection Activities

Data Collection Activity	Instruments	Respondent, Content, Purpose of Collection	Mode and Duration
Screening for participation	Screening questionnaire	Respondents: Home visiting supervisors. Content: Potential participant demographics and experience with home visiting supervisory role and reflective supervision; information about the home visiting programs with which they are associated. Purpose: Information collected will be used to determine which individuals meet inclusion criteria and will be invited to participate. We will also use information to select a sample that is diverse in terms of participant and program characteristics.	Mode: Online/web-based survey Duration: 5 minutes
Web-based pretesting of reflective supervision measure (including responding to feedback questions)	Web-based survey	Respondents: Home visiting supervisors. Content: Full item pool of reflective supervision measure and embedded feedback questions. Purpose: Descriptive analysis of survey results will permit checking of item variability and skewness. These analyses will provide preliminary indicators that can help focus subsequent changes to be implemented prior to piloting the measure with a larger developmental sample. Feedback on the experience with the measure will inform the instructions and approach to piloting.	Mode: Online/web-based survey Duration: 1 hour

Other Data Sources and Uses of Information

Prior project work informed the development of a list of reflective supervision elements that were used in concept mapping activities in the prior phase of the project. An item pool was subsequently developed from the concept mapping, a review of literature and existing measures, and cross-walked with home visiting model, state, and Tribal expectations, and guidelines.

Building upon the information gathered through initial activities, the Study Team is now preparing to pretest the measure by surveying a small sample of home visiting supervisors.

A3. Use of Information Technology to Reduce Burden

Participants will be recruited via announcements sent through the Home Visiting Applied Research Collaborative listserv. If they are interested in participation, they will have the option to complete a screening questionnaire to determine eligibility. Surveys will be completed using a secure web platform (Qualtrics) to reduce participant burden.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

Information to achieve purposes stated in A.2 does not exist for the field of home visiting. There currently is not information available to evaluate the current item pool measuring the key elements of reflective supervision. This pretesting data is needed to develop a relevant, feasible, acceptable, culturally responsive, and useful measure for home visiting.

A5. Impact on Small Businesses

The web-based survey will include individual staff at state and territory local implementing agencies (LIAs) and Tribal Home Visiting programs, which may be small businesses. The requested information is the absolute minimum necessary for the intended use of the data. The Study Team will minimize the burden on individuals by keeping the measure survey as short as possible.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to submit a request to OMB for review of the overarching generic clearance for pre-testing activities. This notice was published on January 5, 2021; Volume 86, Number 2, page 308, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received. A second notice in the Federal Register announcing the agency’s submission of the overarching generic clearance for pre-testing activities for OMB’s review. This notice was published on May 21, 2021; Volume 86, Number 97, page 27624, and provided a thirty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

The Study Team consulted with a small group of research and evaluation experts, home visitors, and those who support home visitors (program managers and supervisors) on the initial list of reflective supervision elements³. This consultation helped refine and pare down the list of elements that were used to develop the item list used in these pretesting activities, thus reducing burden to study participants.

Practitioner workgroup members include:

Heather Smith

Tiara Smith

Stephanie Massey

Carri Chischilly

Sierra Guches

Lindsey Hackney

Kehaulani Fernandez

Sanquinita Martin

Amanda Ray

Technical workgroup members include:

Sherryl Scott Heller

Jon Korfmacher

Dawn Nixon

David Schultz

Angela Tomlin

Edward Watkins

Maria Elena Oliveri

A9. Tokens of Appreciation

Honoraria will be provided for participants to share their expertise and experiences in the area of home visiting. See section A13 for additional information.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

For the purposes of study recruitment, we will collect participant name, physical location they work in, and email. For enrolled participants we will retain records of participant name, physical location, and email. Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

As specified in the contract for this project, the Study Team will protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Study Team will ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

Data Security and Monitoring

The Study Team has developed a Data Security Plan that assesses all protections of respondents’ personally identifiable information. The Study Team will ensure that all its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

As specified in the evaluator’s contract, the Study Team will use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Study Team will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Study Team will: ensure that this standard is incorporated into the Study Team’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations.

A11. Sensitive Information ⁴

Instrument 1 (screening questionnaire) asks respondents to provide information about race, ethnicity, and language data for themselves and the families they work with. We request this information in support of OPRE’s commitment to incorporating racially equitable approaches into research⁵ and the Executive Order Advancing Racial Equity and Support for Underserved Communities Through the Federal Government (January 21, 2021). Specifically, we will use this information to ensure that participants from certain populations and participants that work with certain populations, including people of color and others who have been historically underserved and marginalized, are included in study activities.  

A12. Burden

Explanation of Burden Estimates

Screening questionnaire: We anticipate up to 125 participants for the measure survey will complete an electronic screening questionnaire. The questionnaire will take about 5 minutes to complete. We do not anticipate variance in response time by respondent type.

Web-based reflective supervision measure pretesting: Up to 50 participants will be asked to complete the measure with embedded feedback questions. The survey will take about 60 minutes to complete. We do not anticipate variance in response time by respondent type.

Estimated Annualized Cost to Respondents

The estimated total annual cost to respondents is approximately $2,325.93 (see burden table below). This cost to respondents is based on the average wage of social and community services managers (occupation code 11-9151). Estimates come from the 2022 Bureau of Labor Statistics report on Wage Estimates (retrieved from https://www.bls.gov/oes/current/oes_nat.htm).

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Screening questionnaire	125	1	0.09	11	$38.13	$419.43
Web-based measure pretesting	50	1	1	50	$38.13	$1,906.5
Total	125			61	$38.13	$2,325.93

A13. Costs

The Study Team proposes providing $35.00 in honoraria, in the form of a $35.00 gift card for their time spent providing expert guidance given their professional expertise in reflective supervision. The honorarium amount of $35.00 nears the average hourly wage (see hourly wage in burden table above) for respondents and reflects a time commitment of approximately one hour per respondent. The Presidential Memorandum on Restoring Trust in Government through Scientific Integrity and Evidence-Based Policy Making⁶, as well as the ACF Evaluation Policy⁷ discuss community engagement and inclusion in research. Consistent with these guidance documents, and to ensure involvement from a variety of people with diverse professional experiences in the home visiting field, we plan to offer all participants an honorarium. The honorarium is intended to offset costs of providing expert guidance on the reflective supervision measure, such as staff time away from other necessary work, or other expenses that might otherwise prevent individuals from participating in the study. In some instances, Federal data collections have found that providing honorarium to respondents increases response rates or reduces nonresponse bias. Previous research indicates that providing an honorarium improves response rates and decreases nonresponse bias, especially from minority respondents.⁸ Additionally, prior research within home visiting has found that the honorarium amount proposed for this study results in higher response rates.⁹

A14. Estimated Annualized Costs to the Federal Government

The estimated annualized costs are based upon full-time equivalent time, operational expenses (such as equipment, overhead, printing, and staff support), and other expenses which would not have been incurred without this collection of information.

Activity	Estimated Cost
Survey administration and monitoring	$15,000.00
Analysis	$10,0000.00
Reporting and documentation	$5,000.00
Total/Annual costs over the request period	$30,000.00

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella clearance for pre-testing (0970-0355).

A16. Timeline

Data collection activities will occur within a 5-month period after OMB approval. Data analysis will occur within 4 months of completing data collection.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendix 1: Home Visiting Applied Research Collaborative Recruitment Announcement for Web-based Measure Pretesting

Instrument 1: Screening Questionnaire for Web-based Measure Pretesting

Instrument 2: Web-Based Measure Pretesting Questions

1 West, A., & Madariaga, P. (2022). Reflective supervision: A planning tool for home visiting supervisors (OPRE Report No. 2022-138). Office of Planning, Research, and Evaluation; Administration for Children and Families; U.S. Department of Health and Human Services.

2 Information collection activities were approved by OMB under OMB #0970-0355 on August 19, 2022, with the title Supporting and Strengthening the Home Visiting Workforce (SAS-HV).

3 Since input was collected from fewer than ten individuals, these activities were not subject to the Paperwork Reduction Act.

4 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

5 OPRE's Commitment to Incorporating Racially Equitable Approaches into its Research, Evaluation, Evaluation Technical Assistance, and Related Data Practices | The Administration for Children and Families (hhs.gov)

6 https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/27/memorandum-on-restoring-trust-in-government-through-scientific-integrity-and-evidence-based-policymaking/

7 https://www.acf.hhs.gov/opre/report/acf-evaluation-policy

8 Singer, E., & Ye, C. (2013). The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science, 645(1), 112–141. https://doi.org/10.1177/0002716212458082 

9 Geyelin Margie, N. & Nerenberg, L. (2019). MIHOPE incentive experiment results: 15-month follow-up. Memo submitted to Josh Brammer and Margo Schwab, Office of Information and Regulatory Affairs (OIRA), Office of Management and Budget (OMB). https://omb.report/icr/201907-0970-010/doc/93761901.pdf 

11