SSA - Measuring Self- and Co-Regulation in SRAE Programs Phase 1

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Measuring Self- and Co-Regulation in Sexual Risk Avoidance Education Programs

Pre-Testing of Evaluation Data Collection Activities

0970 – 0355

Supporting Statement

Part A

February 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Tia Brown

Calonie Gray

MeGan Hill

Part A

Executive Summary

• Type of request: This Information Collection Request is for a generic information collection under the umbrella generic, Pre-Testing of Evaluation Data Collection Activities (0970-0355).

• Description of request: We do not intend for this information to be used as the principal basis for public policy decisions.

• Time sensitivity: ACF will use the data to assess the preliminary effectiveness of the instrument at measuring youth’s self-assessment of self-regulation skills before and after they participate in a Sexual Risk Avoidance Education (SRAE) program in which facilitators use co-regulation strategies in a classroom setting. ACF would like to pre-test the measures under an existing SRAE evaluation contract, which ends in September 2023. To complete this work on schedule, the pre-test data collection should begin in early 2023.

A1. Necessity for Collection

In 2013, the Office of Planning, Research, and Evaluation (OPRE) of the Administration for Children and Families (ACF), U.S. Department of Health and Human Services (HHS), began developing an important research agenda focused on self-regulation. This agenda generated a set of foundational resources and contributed to theory and practice.^1,2 Self-regulation, defined as a developmental process that includes executive function, self-control, and regulation of emotions, is as an important skill that impacts quality of life for children and their families. Thus, it relates directly to the programs that ACF administers and evaluates. This work has grown and evolved over the past decade, as outlined in ACF’s co-regulation learning agenda. It has led to an understanding that youth develop self-regulation through co-regulation, “the interactive process by which caring adults (1) provide warm, supportive relationships, (2) promote self-regulation through coaching, modeling, and feedback, and (3) structure supportive environments.”¹

This generic information collection request supports the development and testing of a survey measure to assess youth’s self-regulation skills before and after they participate in programming in which facilitators use co-regulation strategies. This pre-test data collection would support ACF’s goal of prioritizing the development of reliable and valid co- and self-regulation measures. There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

This proposed information collection meets the primary goals of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures. Data collected under this generic information collection request will be used to develop a self-regulation measure that, if found to be valid and reliable through pre-testing, would advance ACF’s learning agenda on co- and self-regulation.

The information collected is meant to contribute to the body of knowledge on ACF program design and intended outcomes. It is not intended to be used as the principal basis for a decision by federal decision makers, and it is not expected to meet the threshold of influential or highly influential scientific information. Limitations will be described in written products associated with this pilot study.

Research Questions or Tests

ACF proposes the following guiding questions for this pre-test:

1. Are the survey items valid and reliable for measuring youth self-regulation?

2. How long does it take youth to complete the survey items?

3. Do the youth respondents seem to easily understand the questions and response options?

Study Design

This survey measures pre-test has two phases. In both phases, the survey instruments will be administered in a classroom as part of a program that uses the co-regulation strategies. These programs take place at ACF-funded Sexual Risk Avoidance Education (SRAE) grant recipients’ classroom sites. The survey items will be administered before the program and again after the program. To minimize the burden on staff and youth, the items will be administered at the same time youth complete the required SRAE entrance and exit performance measure surveys.³

During the first phase, the study team will administer two versions of the survey, purposively assigning versions to ensure variation by program and facilitator. Having two versions will enable the study team to pre-test more items while reducing the burden on individual respondents. Each youth participant will complete the same version of the survey before and after the program. The study team will use the survey data from the two survey versions to identify a core set of items to use in a refined survey.

During the second phase, the study team will pre-test the refined survey with a new group of youth participants. Each youth participant in the second phase will complete this same survey before and after the program. The study team will use results from the second pre-test to further refine the survey; results are not intended to be representative of, or generalizable to, a given subpopulation.

The study team will recruit approximately 450 youth participating in programming offered by as many as four selected SRAE grant recipients that use facilitation strategies or curricula (for example, Love Notes) that include the concept of self-regulation.

Table A.1 summarizes the study design, including the three data collection instruments, their content and respondent types, and the mode and duration of each data collection activity.

Table A.1. Study design summary

Data collection activity	Instruments	Respondent, content, purpose of collection	Mode and duration
Phase 1. Youth survey	Instrument 1. Youth Self-Assessment Pre- and Post-Program Survey Version A Instrument 2. Youth Self-Assessment Pre- and Post-Program Survey Version B	Respondents: 300 youth, ages 14–19 Content: Questions about aspects of cognitive, emotional, and behavioral regulation. For example, questions about goal setting, inhibition, and controlling emotions. Purpose: To measure aspects of youth self-regulation skills.	Mode: Paper or electronically Duration: 10 minutes
Phase 2. Youth survey	Instrument 3. Youth Self-Assessment Pre- and Post-Program Survey Version C	Respondents: 150 youth, ages 14–19 Content: Questions about aspects of cognitive, emotional, and behavioral regulation. For example, questions about goal setting, inhibition, and controlling emotions. Purpose: To measure aspects of youth self-regulation skills.	Mode: Paper or Electronically Duration:10 minutes

Other Data Sources and Uses of Information

Findings from the pre-test data collection will inform the development of new measures for youth self-assessment of self-regulation skills and behaviors pre- and post-program. The study team is also developing a classroom observation tool to identify and assess the use of co-regulation strategies. Only the study team members will pre-test the observation tool. Thus, the burden table does not include this activity.

A3. Use of Information Technology to Reduce Burden

The survey instruments (Youth Self-Assessment Pre- and Post-Program Survey Version A, Youth Self-Assessment Pre- and Post-Program Survey Version B, and Youth Self-Assessment Pre- and Post-Program Survey Version C) will be delivered electronically or using a paper mode, depending on how each grant recipient administers its SRAE performance measures survey and its mode preference for this proposed pilot.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The study team has scanned available measures of self-regulation and identified a set of items aligned with ACF’s research priorities. The survey incorporates items from measures used in other contexts. Some items might be adapted slightly to meet the needs of this study, and the full survey measure developed for this pre-test will be new.

A5. Impact on Small Businesses

The programs participating in the study will be small, nonprofit organizations. The study team will request information required only for the intended use. The burden for respondents will be minimized by restricting the survey length to the required minimum and by scheduling this pre-test data collection during the data collection activities associated with the SRAE performance measures entrance and exit surveys administered to youth. Depending on each grant recipient’s preference, some facilitators might be asked to oversee the classroom data collection activities during a normal course period. Otherwise, the study team will administer the survey. Facilitators will not be asked to work additional time to complete data collection activities.

A6. Consequences of Less Frequent Collection

This is a one-time data collection with two phases. First, we propose testing two versions of the survey measures, and then based on those results, conducting a final test of the revised items. This approach will ensure that the final set of items are tested with a variety of students receiving programming in several locations and with different facilitators.

A7. Now subsumed under 2(b) (above) and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intent to submit a request to OMB for review of the overarching generic clearance for pre-testing activities. This notice was published January 5, 2021 (Volume 86, Number 2, page 308), and provided a 60-day period for public comment. During the notice and comment period, no substantive comments were received. A second notice in the Federal Register announced the agency’s submission of the overarching generic clearance for pre-testing activities for OMB’s review. This notice was published May 21, 2021 (Volume 86, Number 97, page 27624), and provided a 30-day period for public comment. No comments were received during the notice and comment period.

Consultation with Experts Outside of the Study

Although no experts have been engaged yet for the proposed measures pilot, the study team plans to engage two external experts to inform development of the survey measure: Desiree Murray (senior research scientist at the Center for Health Promotion and Disease Prevention at the University of North Carolina) and Velma McBride Murray (professor, Peabody College and School of Medicine, Vanderbilt University). We will also share a list of measures with no more than nine experts from Mathematica who specialize in measurement, youth self-regulation, and co-regulation, and with federal staff, such as Aleta Meyer and Calonie Gray.

A9. Tokens of Appreciation

Participants will not receive tokens of appreciation.

A10. Privacy: Procedures to protect privacy of information while maximizing data sharing

Personally Identifiable Information

We will not maintain information from the pilot data collection in a paper or electronic system that would enable users to retrieve data by an individuals’ personal identifier. The youth self-assessment will not collect personally identifiable information and will not be linked to the performance measures data. To track youth responses between their pre-program and post-program surveys, we will use a series of self-identification survey questions at the beginning of the pre-program survey to develop unique self-generated identification codes. The post-program survey will then ask those same self-identification questions, and the responses to those questions will be used to link responses at the individual level. These identifiers will not be linked to any personally identifiable information and will be maintained separately from the SRAE entrance and exit performance measures data to ensure the two sets of data are not linkable.

In programs that require active or passive consent for the program or performance measures, the study team will work with the site to adapt their site consent form (as needed) to also include the youth survey for this pilot data collection. Sites will manage the consent form process, and youth and parent names will not be shared with the study team or be linkable to the pilot data set.

Assurances of Privacy

The study team will follow the guidelines of the grant recipients and their local institutional review board (IRB) regarding the need for parental consent or youth assent. All youth will be informed that participation is voluntary, the study team will not link their data to them, and information collected will be kept private to the extent permitted by law.

As part of the overall project, the study team discusses privacy issues during training sessions with staff who work on the project. The contractor, Mathematica, requires all staff to complete online security awareness training when they are hired and to participate in refresher training annually. Training topics include the security policies and procedures outlined in Mathematica’s corporate security manual. All records containing data will be transferred using a secure file transfer protocol site. As specified in the contract, Mathematica will protect respondents’ privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. In addition, the study leaders at Mathematica will conduct project-specific trainings for all staff who work on the study to communicate expectations and procedures for privacy, informed consent, and data security.

The study will be reviewed by Mathematica’s IRB, the Health Media Lab, and the IRBs overseeing the data collection of the selected programs. Outreach and data collection will not begin until the project has received IRB approval.

Data Security and Monitoring

As specified in the contract, the contractor shall protect respondents’ privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information. The contractor has developed a data security plan that assesses all protections of respondents’ personally identifiable information. The contractor will ensure all employees and subcontractors (at all tiers), including employees of each subcontractor who perform work under this contract and subcontract, receive training on data privacy issues and comply with all requirements. All Mathematica staff must sign an agreement to: (1) maintain the privacy of any information from individuals, businesses, organizations, or families participating in any projects conducted by Mathematica; (2) complete online security awareness training when they are hired; and (3) participate in a refresher training annually.

As specified in the evaluator’s contract, the contractor will use encryption compliant with the Federal Information Processing Standard (Security Requirements for Cryptographic Module, as amended) to protect all sensitive information during storage and transmission. The contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the standard. The contractor will incorporate the standard into its property management and control system and establish a procedure to account for all laptop and desktop computers and other mobile devices and portable media that store or process sensitive information. The contractor will secure any data stored electronically in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor’s data safety and monitoring plan includes strategies for minimizing to the extent possible including sensitive information on paper records and for protecting any paper records, field notes, or other documents that contain sensitive information to ensure secure storage and limits on access.

No information will be given to anyone outside the study team and ACF.

A11. Sensitive Information ⁴

No sensitive information will be collected in this study.

A12. Burden

Explanation of Burden Estimates

In Table A.2 we summarize the estimated reporting burden and costs for each instrument. The survey estimates include time for respondents to review the instructions and to complete and review their responses. The study team expects the total annual burden to be 153 hours for all the instruments in this information collection request. Figures are estimated as follows:

• Phase 1. Youth Self-Assessment Pre- and Post-Program Survey Version A. The survey will be administered to youth participating in SRAE programming (N = 150), just before starting the SRAE program (pre), and again at the end of the SRAE program (post). The time between the pre- and post-survey is typically less than one month. Therefore, we anticipate 100 percent of the youth who are administered the SRAE performance measures survey at program entry and exit will also complete the Youth Self-Assessment Survey (Version A). The survey will be completed on paper or electronically and is expected to take approximately 10 minutes.

• Phase 1. Youth Self-Assessment Pre- and Post-Program Survey Version B. The survey will be administered to a different set of youth participating in SRAE programming than those receiving Version A. Some items in Version B will differ from those in Version A. As with Version A, Version B will be administered (N = 150) just before starting the SRAE program (pre), and again at the end of the SRAE program (post). The time between the pre- and post-survey is typically less than one month. Therefore, we anticipate 100 percent of the youth who are administered the SRAE performance measures survey at program entry and exit will also complete the Youth Self-Assessment Survey (Version B). The survey will be completed on paper or electronically and is expected to take approximately 10 minutes.

• Phase 2. Youth Self-Assessment Pre- and Post-Program Survey Version C. Following data analysis from Phase 1 of the data collection, a new survey instrument, Version C, will be developed, taking the best items from both Versions A and B. This revised survey will be administered to a different set of youth participating in SRAE programming than those who received Versions A or B. This instrument will similarly be administered (N = 150) just before starting the SRAE program (pre), and then again at the end of the SRAE program (post). The time between the pre- and post-survey is typically less than one month. Therefore, we anticipate 100 percent of the youth who are administered the SRAE performance measures survey at program entry and exit will also complete the Youth Self-Assessment Survey (Version C). The survey will be completed on paper or electronically and is expected to take approximately 10 minutes.

Estimated Annualized Cost to Respondents

The study team expects the total annual cost to be $1,110 for all instruments in the current information collection request. We estimate the average hourly wage for youth respondents at $7.25, based on the federal minimum wage. For each instrument listed in Table A.2, the study team calculated the total annual cost by multiplying the annual burden hours by the current federal minimum wage ($7.25), as listed in Table A.2.

Table A.2. Total burden requested under this information collection

Instrument	Number of respondents (total over request period)	Number of responses per respondent (total over request period)	Average burden per response (hours)	Total annual burden (hours)	Average hourly wage rate	Total annual respondent cost
Instrument 1. Youth Self-Assessment Pre- and Post-Program Survey Version A	150	2	.17	51	$7.25	$370
Instrument 2. Youth Self-Assessment Pre- and Post-Program Survey Version B	150	2	.17	51	$7.25	$370
Instrument 3. Youth Self-Assessment Pre- and Post-Program Survey Version C	150	2	.17	51	$7.25	$370
Total	450			153	$	$1,110

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The estimated total cost to the federal government for this study is $246,000 (Table A.3). This includes costs for collecting, processing, and analyzing the data, and preparing reports.

Table A.3. Estimated total cost by category

Cost category	Estimated costs
Field work	$226,000
Publications and dissemination	$20,000
Annual total	$246,000

A15. Reasons for changes in burden

This request is for an individual information collection under the umbrella clearance for pre-testing (0970-0355).

A16. Timeline

Table A.4 contains the timeline for collecting, analyzing, and reporting data. The study team expects to collect data from Youth Self-Assessment Pre- and Post-Program Survey Versions A and B during winter 2023. The study team will conduct preliminary analysis on the data collected to inform the design of Youth Self-Assessment Pre- and Post-Program Survey Version C, which will be administered in late spring 2023. Subsequent analysis will be conducted in summer 2023 and completed by September 2023.

Table A.4. Schedule for pre-test data collection and reporting

Activity	Timing a
Data collection and analysis, Phase 1
Youth Self-Assessment Pre- and Post-Program Survey Version A	Winter 2023
Youth Self-Assessment Pre- and Post-Program Survey Version B	Winter 2023
Data Analysis Phase 1	Winter and spring 2023
Data collection and analysis, Phase 2
Youth Self-Assessment Pre- and Post-Program Survey Version C	Spring 2023
Data Analysis Phase 2	Spring and summer 2023
Reporting	Summer 2023

^a After obtaining OMB approval

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instruments

Instrument 1. Youth Self-Assessment Pre- and Post-Program Survey Version A

Instrument 2. Youth Self-Assessment Pre- and Post-Program Survey Version B

1 McKenzie, K.J., A. Meyer, et al. “Co-Regulation and Connection in Human Services: Developing a Learning Agenda.” OPRE Insights Blog, March 29, 2022. https://www.acf.hhs.gov/opre/blog/2022/03/co-regulation-connection-human-services-developing-learning-agenda

2 Office of Planning, Research, and Evaluation. “Self-Regulation and Toxic Stress Series.” Administration for Children and Families, U.S. Department of Health and Human Services, n.d. https://www.acf.hhs.gov/opre/project/self-regulation-and-toxic-stress-series. Accessed December 20, 2022.

3 As approved under OMB Control Number 0970-0536, expiration date 12/31/2023.

4 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

9