SSB - Measuring Self- and Co-Regulation in SRAE Programs Phase 1

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Measuring Self- and Co-Regulation in Sexual Risk Avoidance Education Programs

Pre-Testing of Evaluation Data Collection Activities

0970 – 0355

Supporting Statement

Part B

February 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Tia Brown

Calonie Gray

MeGan Hill

Part B

B1. Objectives

Study Objectives

The proposed pre-test data collection supports the development and testing of a survey measure to assess youth’s self-regulation skills before and after they participate in programming in which facilitators use co-regulation strategies. This data collection will further ACF’s learning agenda on co- and self-regulation by supporting ACF’s goal of prioritizing development of reliable and valid co- and self-regulation measures. The findings from this data collection will inform the development of new survey measures of youth self-regulation before and after they receive sexual risk avoidance education (SRAE) programming. New survey measures are critical to ACF’s co-regulation learning agenda¹ and will contribute to the body of knowledge on ACF program design and intended outcomes.

Generalizability of Results

The proposed pre-test is intended to inform instrument development by producing descriptive findings about the reliability and validity of a survey measure for the focus population. It is not designed to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

This proposed information collection meets the primary goals of ACF’s generic clearance for pre-testing (0970-0355): to develop and test information collection instruments and procedures. Data collected under this generic clearance will be used to inform the development of a self-regulation measure, which, if found to be valid and reliable through pre-testing, will further ACF’s learning agenda on co- and self-regulation.

The survey instruments will be administered before and after an SRAE program using facilitation strategies and/or SRAE curricula (for example, Love Notes) that teach self-regulation to youth. The programs using the co-regulation strategies include ACF-funded SRAE grant recipients’ classrooms. The survey items will be administered before and after the program. They will be administered at the same time youth complete the normal entrance and exit performance measure surveys (OMB Control Number 0970-0536, expiration date 12/31/2023), reducing burden on both youth and survey administrators. This design and data collection approach is appropriate to the quality of data needed to assess the measures being piloted.

Because this is an initial pilot study, the data collected will not be representative and cannot be used to directly assess student outcomes. This information is not intended to be used as the principal basis for public policy decisions, and is not expected to meet the threshold of influential or highly influential scientific information. Limitations will be described in written products associated with this pilot study.

B2. Methods and Design

Target Population

The target population for this generic information collection request is youth.

Sampling and Site Selection

The study team will invite up to four grant recipients to participate in the pre-test of the self-regulation survey. Eligible SRAE grant recipients are those that implement the Love Notes curriculum in classroom settings in a way that includes teaching about co-regulation strategies. These grant recipients will be identified through a process that includes a review of their programming (that is, their curricula and facilitation plans) and data previously collected by the study team for the National Descriptive Study-Early Implementation Study of SRAE programs (OMB Control Number 0970-0530, approved July 2019), which asked SRAE grant recipients about their program design, implementation, curricula, and the settings in which they provide programming (for example, in middle or high schools, in or out of schools, in juvenile detention centers, etc.).

ACF will prioritize selection of: (1) grant recipients known to be implementing programs with the highest number of youth during winter and spring 2023; (2) grant recipients whose existing procedures for administering the SRAE program performance measures would allow for the addition of a new survey; and (3) grant recipients that have parental consent in place and/or are exempt from collecting consent. Where active parental consent or youth assent is needed, the study team will work with the grant recipient and its Institutional Review Board (IRB) to ensure its specific consent requirements are met.

We expect that across the grant recipients, approximately 450 youth will participate across two phases of survey administration.

During the first phase, the study team will recruit a total of approximately 300 youth. The team will administer two versions of the survey and purposively assign the distribution of the versions to ensure variance by site and facilitator. Each youth participant will take the same version of this survey once before and once after the program. We expect about half of the youth will complete Version A of the survey (Instrument 1. Youth Self-Assessment Pre- and Post-Program Survey Version A), and the other half of the youth will complete Version B of the survey (Instrument 2. Youth Self-Assessment Pre- and Post-Program Survey Version B). Exact sample sizes will depend on the number of youth being served by the grant recipient.

During the second phase of data collection, a third group of youth will complete a revised survey (Instrument 3. Youth Self-Assessment Pre- and Post-Program Survey Version C). Each youth participant will take the same version of this survey once before and once after the program. The study team will recruit with the aim of approximately 150 youth taking Version C of the survey.

All youth participating in the programs offered by grant recipients will be eligible to complete the survey provided they assent and that the consent requirements of the site (and its IRB) have been met. We anticipate that youth program participants will be middle and high school students between the ages of 14 and 19. Program participants involved in data collection will be from a convenience sample. Consequently, they may not be representative of the population all SRAE grant recipients serve.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument(s)

The survey instrument tested under this proposed pilot will collect information about youth perceptions of their own emotional regulation (such as identifying and controlling their emotions), behavioral regulation (such as cooperation, communication, and prosocial behaviors), and cognitive regulation (such as goal setting and planning, and decision making).

To develop the surveys, the study team first conducted a measures scan to identify existing measures of youth self-regulation. The scan included published literature; prior ACF-funded projects; and existing measures compendia. The study team also consulted substantive experts for their input on the existing measures and the areas of focus for this survey.

After conducting the scan, the study team identified a set of items from existing measures that covered self-regulation skills relevant to the SRAE programming, including topics in emotional, behavioral, and cognitive regulation. These items were used to develop Instrument 1 and Instrument 2. By having some of the same items and some different items on those two instruments, the study team can test more items while being mindful of participant burden. The study team will use data from the two instruments to select, refine, and narrow the set of items for Instrument 3. Each survey will take about 10 minutes to complete.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection. The study team will coordinate data collection with each site to determine its preferred data collection approach.

The youth will be middle and high school students participating in existing SRAE programs in classroom settings. Just before the start of the program, youth will complete the self-regulation survey (Instrument 1. Youth Self-Assessment Pre- and Post-Program Survey Version A). After the last session of training, they will complete the same survey again. A separate group of youth will complete Version B of the self-regulation survey (Instrument 2. Youth Self-Assessment Pre- and Post-Program Survey Version B), before and after the program. Data from Instruments 1 and 2 will inform modifications to the survey instrument to create Version C (Instrument 3. Youth Self-Assessment Pre- and Post-Program Survey Version C). As a second phase of data collection, a new group of youth will complete Version C before and after the program. As noted, the study team will aim to recruit approximately 150 youth participants for each instrument, but actual numbers may differ depending on how many youth the selected programs serve.

Mathematica staff will work with each grant recipient to coordinate the distribution and collection of the surveys at each administration period. Depending on the preferences of the grant recipients, the program facilitators may distribute and collect the surveys, or Mathematica study team members may go to the site to support the collection. To ensure an efficient data collection process at each program, Mathematica staff will participate in a data collection training. The training will cover the expectations and process for working with the programs and their facilitators so that the data are collected consistently and that all procedures around data privacy and security are followed. Collected data will be immediately reviewed to determine if issues around quality exist (for example, that the number of completed surveys matches to the number of youth in the classroom we expect to complete and that the surveys contain few item-level missing data). Senior project staff will monitor the response rates daily during data collection. The study team will work closely with facilitators and study team members going to the site to continually address data collection issues as they arise.

Table B.1 lists all data collection activities proposed for this pre-test study.

Table B.1. Data collection activities

Data collection	Administration plans
Youth Self-Assessment Pre- and Post-Program Survey Version A	Total participants	Approximately 150
	Mode	Paper-based or electronic
	Time	10 minutes
	Frequency	2
Youth Self-Assessment Pre- and Post-Program Survey Version B	Total participants	Approximately 150
	Mode	Paper-based or electronic
	Time	10 minutes
	Frequency	2
Youth Self-Assessment Pre- and Post-Program Survey Version C	Total participants	Approximately 150
	Mode	Paper-based or electronic
	Time	10 minutes
	Frequency	2

All study team members will receive training to ensure that any data collected from facilitators and youth are collected in a consistent and high-quality manner. Lastly, to ensure quality and consistency, the study team will meet often to discuss program activities and troubleshoot issues as they arise.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The youth will be administered the survey measures during their SRAE program classes, at the same time they take the scheduled SRAE performance measures entrance and exit surveys. Given this, we anticipate high response rates. The surveys are not designed to produce statistically generalizable findings, and participation is wholly at the respondent’s discretion.

Non-Response

Participants will not be randomly sampled, and findings are not intended to be representative of the larger universe of SRAE participants. Consequently, we will not calculate non-response bias. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

Data collected for this measure development phase of the study will document: (1) whether the measure questions are appropriate for middle and high school–age youth participating in SRAE programs where facilitators are using co-regulation strategies in classroom settings, (2) whether the questions can be interpreted by respondents, and (3) how long the measure takes to complete. The study team will also examine the measure’s reliability and validity. The goal of collecting this information is to build information to help design a survey measure to support ACF’s goal of prioritizing the development of reliable and valid co- and self-regulation measures, thereby furthering ACF’s learning agenda on co- and self-regulation. The data will not be used to generate population estimates, either for internal use or for dissemination.

B7. Data Handling and Analysis

Data Handling

No personally identifiable information will be collected during this study. Survey instruments, regardless of mode, will include several questions for respondent self-identification at the beginning, which will be used to create unique identifiers. The unique identifiers will be used to match pre-program and post-program survey responses for each participant. At sites that prefer paper surveys, those will be collected and sent via Federal Express to Mathematica’s Survey Operations Center for receipting, data entry, and secure storage. The survey data, once entered into the data entry system, will be stored on Mathematica’s network, which is accessible only to the study team. All paper and electronic data will be destroyed at the end of the study.

Data Analysis

This project will not employ complex data analytic techniques. For each survey, we will conduct standard descriptive analyses, including measures of central tendency and ranges. If the sample size permits, the study team will conduct an exploratory factor analysis to discover whether particular items might group together conceptually. The study team will also conduct standard validity and reliability analyses. If possible, the study team will also examine relations between the self-regulation survey items and the co-regulation observation scores.

Data Use

Findings will be used to assess whether the survey instrument is feasible to implement and understandable to youth. Analyses will assess the preliminary effectiveness of the instrument at measuring youth’s self-assessment of self-regulation skills before and after participating in an SRAE program where facilitators are using co-regulation strategies in a classroom setting. The findings from this pre-test study will inform the development of the self-regulation survey, which would support ACF’s learning agenda on co- and self-regulation. The study team will develop an internal ACF memorandum describing the pre-test methodology and results of the pre-test, including changes to the measure after the first phase of data collection, findings from the final version of the survey, recommendations for future measure development and refinement, and suggestions for further use of the measure.

B8. Contact Person(s)

Table B.2 lists the federal and contract staff responsible for the study, including their affiliation and email address.

Table B.2. Staff responsible for study

Name	Affiliation	Email address
Calonie Gray	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Calonie.Gray@acf.hhs.gov
MeGan Hill	Family and Youth Services Bureau Administration for Children and Families U.S. Department of Health and Human Services	Megan.Hill@acf.hhs.gov
Tia Brown	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Tia.Brown@acf.hhs.gov
Susan Zief	Mathematica	SZief@mathematica-mpr.com
Elizabeth Cavadel	Mathematica	ECavadel@mathematica-mpr.com
Avery Hennigar	Mathematica	AHennigar@mathematica-mpr.com
Tiffany Waits	Mathematica	TWaits@mathematica-mpr.com
Lourdes Fernandez	Mathematica	LFernandez@mathematica-mpr.com

Attachments

Instruments

Instrument 1. Youth Self-Assessment Pre- and Post-Program Survey Version A

Instrument 2. Youth Self-Assessment Pre- and Post-Program Survey Version B

1 McKenzie, K.J., Meyer, A., and OPRE Self-Regulation Learning Agenda Team. “Co-Regulation and Connection in Human Services: Developing a Learning Agenda.” U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research and Evaluation. https://www.acf.hhs.gov/opre/blog/2022/03/co-regulation-connection-human-services-developing-learning-agenda.

7