SSB - SRAENE CIS GenIC

Alternative Supporting Statement Instructions for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Sexual Risk Avoidance Education National Evaluation: Formative Evaluation for the Program Components Impact Study

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

August 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

MeGan Hill

Tia Brown

Part B

B1. Objectives

Study Objectives

The Program Components Impact Study (CIS) seeks to refine and test improvements to one or more components of Sexual Risk Avoidance Education (SRAE) programs to ultimately improve youth outcomes. This request is specific to a formative evaluation phase of the CIS and aims to: (1) gather evidence to inform the direction of a future large-scale evaluation of the effectiveness of using co-regulation strategies, (2) inform guidance and TA resources to support successful replication of co-regulation strategies by additional programs if it is rigorously evaluated, and (3) help to further ACF’s co-regulation learning agenda.

Specifically, this formative evaluation phase of the study will document the implementation of a previously piloted SRAE facilitation strategy, co-regulation (See the Proof of Concept Pilot Study approved under the umbrella generic for Formative Data Collections for ACF Program Support; OMB #: 0970-0531), which proved promising in a Proof of Concept Pilot study, but requires further information about its implementation before conducting a larger scale replication. This formative evaluation phase of the study also seeks to assess associations of the co-regulation strategies with proximal youth outcomes, which is important for understanding whether improving any one component of a program – in this case, facilitation – can bolster youth outcomes. Supporting Statement Part A, Section A1 provides the background for this formative phase of the study, and Section A2 discusses the purpose and use of the data.

The findings from this formative evaluation phase will be used to support the development of TA resources, such as guidance documents and videos, that can be made available to the public to support further replication and testing of the strategy and inform development of ACF research specifically related to integrating co-regulation strategies into the SRAE grant program improvement efforts. The study research questions are described in Supporting Statement Part A, Section A2.

Generalizability of Results

This study is intended to present an internally valid description of SRAE program facilitation using the co-regulation strategy in chosen sites and is not meant to promote statistical generalization to other program, sites, or service populations. Data collected under this generic information collection request will be used to refine SRAE program delivery strategies, furthering ACF research focused on this area.

Appropriateness of Study Design and Methods for Planned Uses

As discussed in Supporting Statement Part A, Section A2, this formative phase of the study is designed to further learn about the implementation of the co-regulation strategies at a larger scale than the Proof of Concept Pilot study. We will assess the implementation of co-regulation strategies to understand the successes and challenges associated with broader implementation of the strategies to build youth’s self-regulation skills. These co-regulation strategies have been used in two SRAE programs under the Proof of Concept Pilot study and showed promise. However, further information about implementation is needed before large-scale replication. Thus, this formative phase is designed to further the research conducted under the Proof of Concept Pilot Study by using a more rigorous design at a larger scale than used during the proof of concept pilot phase, to learn more from larger and broader implementation before the strategy is disseminated to all SRAE grantees. Data collected in this study are not intended to be representative.

As discussed in Supporting Statement Part A, Section A2, the proposed formative study and associated data collection aims to collect feedback on the co-regulation facilitation strategies being implemented in up to nine sites. Facilitators working in these sites will be trained to implement the co-regulation strategies in their SRAE programs. The study team will collect data from the training and throughout implementation to inform what facilitators need to implement with fidelity when the strategy is disseminated to all SRAE grantees. Details about the specific data collection activities are described in Section B2.

As noted in Supporting Statement Part A, this information is not intended to be used as the principal basis for public policy decisions, and it is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target populations for this generic information collection request are facilitators of SRAE programs and youth receiving SRAE programs in-school settings, during the school day. To identify the target populations, we will begin with the SRAE grantees and their sub-recipient providers. Eligible sites are those that deliver SRAE programming in schools with program facilitators and that implement the Love Notes curricula because it is aligned with the focus on co-regulation and self-regulation. The Love Notes curriculum is the most prevalent curriculum across all SRAE grantees.¹ These sites will be identified through a process that includes a review of their program plans and discussions with ACF. An early review of this information suggests that there will be approximately 50 eligible sites.

Sampling and Site Selection

ACF will invite grantees meeting the eligibility criteria described above to volunteer to be a part of this study. If more than nine grantees express interest, ACF will prioritize grantees known to be implementing programs with the highest number of youth during fall 2022 and spring 2023 for selection.

The study team anticipates that each of the nine sites will employ approximately 4 facilitators, for a total of 36 facilitators in the study (9 sites * 4 facilitators per site = 36 facilitators). These assumptions are primarily based on a review of grantee applications. We expect all facilitators within a site to participate in the study. We plan to conduct two focus groups with SRAE program youth at each of the 9 sites, for a total of 18 focus groups. Each focus group will have no more than 10 youth per group, for a total of up to 180 youth across the 9 sites (9 sites * 2 groups per site * 10 youth per site= 180). These 36 facilitators and youth participating in SRAE programming at their sites comprise the eligible sample for the studies.

All youth program participants in the facilitators’ classrooms will be eligible to participate in the focus groups provided they assent and have parental consent to participate. We assume that 20 percent of youth will have parental consent and will show up to participate in the focus groups. We anticipate that youth program participants will be high school students between the ages of 14 to 18. Program participants involved in data collection will be from a convenience sample; they may not be representative of the population all SRAE programs serve.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The study team adapted the instruments from those used in the pilot study of the co-regulation strategy and similar protocols and surveys used for other ACF-funded studies, including projects to examine educator training for youth-serving Healthy Marriage and Relationship Education grantees and to strengthen the implementation of two-generation programs serving parents and children in the same family.²

The facilitator pre- and post-training surveys (Instruments 1 and 2) collect knowledge-based information from the facilitator training. The facilitator interview protocol (Instrument 4) covers topics related to the use of the strategy, perceived effectiveness of the strategy, and suggestions for improvement. These instruments were tested during the Proof of Concept Pilot Study and revisions have been made based on learning from the pilot. The facilitator follow-up survey (Instrument 3) was not tested during the proof of concept phase and is a newly developed survey to collect systematic data post-training on facilitators’ use of the co-regulation strategies in the classroom. The facilitator implementation log (Instrument 5) is a newly developed instrument to learn about facilitators’ adherence to the co-regulation strategies and any adaptations they make. The youth focus group protocol (Instrument 6) covers a range of topics related to youth’s perceptions of the facilitation strategies and the programming, including their own and other youth’s engagement and interest in the classes and topics covered. This instrument was not tested during the proof of concept phase. Appendix C: SRAENE CIS Surveys Crosswalk and Research Questions shows the survey question for each of the instruments, mapped to the research question, and indicates the item source.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection. Three Mathematica study team members (one lead and two supports) will be assigned to each of the nine sites to support the data collection activities. Each team of three will conduct or oversee surveys and qualitative data collection with the SRAE program facilitators and the youth program participants.

The program facilitators will participate in training on the co-regulation strategies (described in Supporting Statement A, Section A2). Just prior to the start of training, facilitators will complete a pre-training web-based survey (Instrument 1. Facilitator Pre-Training Survey) and then following the last session of training they will complete a post-training web-based survey (Instrument 2. Facilitator Post-Training Survey). At the end of the fall semester 2022, and then again at the end of the spring semester in 2023, facilitators will be asked to complete a facilitator follow-up survey (Instrument 3). Each semester, facilitators will also partake in a one-on-one, semi-structured interview (Instrument 4. Facilitator Interview Protocol). Throughout the fall and spring semesters, facilitators in the study will be asked to complete an implementation log at the end of each day they implement the program, across a total of 16 selected weeks (Instrument 5: Facilitator Implementation Log). Data collection with youth will occur through in-person focus groups at the program sites. Youth will be recruited for the focus groups through their SRAE program. Facilitators will distribute parental consent forms and collect them. Youth interested in participating, with the consent of their parents and their own written assent, will be allowed to participate in the focus groups. Data will be collected using a structured focus group protocol (Instrument 6. Youth Focus Group Protocol).

Table B.1 lists all data collection activities proposed for the Formative Evaluation for the Program Components Impact Study.

Table B.1. SRAENE Formative Evaluation for the Program Components Impact Study: Data collection activities

Data Collection	Administration plans
Facilitator Pre-Training Survey	Total participants	36
	Mode	Web
	Time	10 minutes
	Frequency	1
Facilitator Post-Training Survey	Total participants	36
	Mode	Web
	Time	5 minutes
	Frequency	1
Facilitator Follow-up Survey	Total participants	36
	Mode	Web
	Time	5 minutes
	Frequency	2
Facilitator Interview	Total participants	36
	Mode	In-person
	Time	60 minutes
	Frequency	2
Facilitator Implementation Log	Total participants	36
	Mode	Web
	Time	3 minutes
	Frequency	Daily, during selected weeks
Youth Focus Groups	Total participants	180
	Mode	In person
	Time	60 minutes
	Frequency	1

Facilitator data collection activities. The study team will use a web-based survey platform to collect the pre- and post-training and the follow-up survey data from all program facilitators. The facilitators will receive email invitations containing a link to their secured surveys (Appendix A. Facilitator Survey Invitation Email). The pre-training survey will take place prior to the start of training, and the post-training survey will take place following the last session of the training. At the end of each semester, the facilitators will take a follow-up survey, via a link received by email.

The facilitator interviews will take place once in the fall semester and once in the spring semester. The interview data will be collected in person, during a site visit by two study team members. Interviews can also be conducted virtually if necessary. The study team will work with the facilitators ahead of the site visit to schedule the interview time and location.

The facilitators will complete an implementation log daily during a selected 16-week period across the fall and spring semesters (8 weeks per semester), following each day of SRAE program implementation. The log will be web-based, and facilitators will receive a link and reminder to complete the log sent to their email.

Data collection from youth program participants. The youth program participants will have the opportunity to participate in a focus group to discuss both their impressions of the program and its facilitation. Program facilitators will send paper versions of consent forms for parents and guardians home with youth (Appendix B. Parent Consent and Youth Assent Forms). We expect that the proposed approach of sending consent forms to all students will yield sufficient numbers of returned and consented forms to allow for a focus group of up to 10 youth participants per group.

The youth focus groups will be conducted in all study sites, and youth will be selected into the focus groups based on both their parent’s consent to participate and their own assent, and other factors as discussed in Section B2. The focus groups will be conducted in person on the school grounds, during the school day (e.g., during students’ lunch period), but can be conducted virtually if necessary. Two study team members will conduct each focus group, with one team member moderating and the other member taking notes.

All study team members will receive training to ensure that any data collected from the facilitators and the youth are collected in a consistent and high-quality manner. With participant permission, interviews and focus groups will be recorded, with one study team member taking notes during all of them. Lastly, to ensure quality and consistency, the study team will meet often to discuss program activities and troubleshoot issues as they arise.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The surveys and qualitative data collection activities are not designed to produce statistically generalizable findings, and participation is wholly at the respondent’s discretion.

NonResponse

Participants will not be randomly sampled, and findings are not intended to be representative. Consequently, we will not calculate nonresponse bias. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

Data collected for this formative evaluation phase of the study will document the implementation of the co-regulation strategies at a larger scale than the Proof of Concept Pilot study, building evidence for promising practices related to the feasibility of training and implementing a specific facilitation strategy, critical for furthering ACF’s research agenda on co-regulation, assessing the need for a future rigorous study of the strategies, and supporting technical assistance to the SRAE grant recipients. The data will not be used to generate population estimates, either for internal use or for dissemination.

B7. Data Handling and Analysis

Data Handling

No personally identifiable information will be shared outside of the study team. Survey data and qualitative data, including typed notes and audio recordings, will be stored on Mathematica’s network, which is accessible only to the study team, and destroyed at the end of the study.

Data Analysis

This project will not employ complex data analytic techniques. To analyze qualitative data—such as notes from the facilitator interviews and youth participant focus groups—we will use standard qualitative analysis techniques such as thematic identification. For the facilitator surveys, we will conduct standard qualitative analysis of responses to open-ended items; and calculate ranges, averages, and simple descriptive statistics for the quantitative questions.

Data Use

Data collected will be used to assess the implementation of the co-regulation strategies at a scale larger than the Proof of Concept Pilot study, including describing how the facilitators integrate the training into their instruction, identifying what additional supports they need to do so with fidelity, and examining the association of the strategies with youth proximal outcomes.

The findings from this formative evaluation phase will inform the planning of a possible future effectiveness evaluation of the co-regulation strategies, as well as support the development of TA resources, such as guidance documents and videos, that can support successful replication in a future study. The primary purpose of the information collected is not publication, but findings may be incorporated into materials that are made publicly available. For example to contextualize the plans for successful replication for a future study, ACF may reference the findings from this formative evaluation phase.

B8. Contact Persons

In Table B.1, we list the federal and contract staff responsible for the study, including their affiliation and email address.

Table B.2. Individuals Responsible for Study

Name	Affiliation	Email address
Calonie Gray	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Calonie.Gray@acf.hhs.gov
MeGan Hill	Family and Youth Services Bureau Administration for Children and Families U.S. Department of Health and Human Services	Megan.Hill@acf.hhs.gov
Tia Brown	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Tia.Brown@acf.hhs.gov
Susan Zief	Mathematica	SZief@mathematica-mpr.com
Heather Zaveri	Mathematica	Hzaveri@mathematica-mpr.com
Tiffany Waits	Mathematica	TWaits@mathematica-mpr.com

Attachments

Appendices

Appendix A: Study Notification and Reminder Materials

Appendix B: Youth Focus Group Consent and Assent Forms

Appendix C: SRAENE CIS Surveys Crosswalk and Research Questions

Instruments

Instrument 1. Facilitator Pre-training Survey

Instrument 2. Facilitator Post-training Survey

Instrument 3. Facilitator Follow-up Survey

Instrument 4. Facilitator Interview Protocol

Instrument 5. Facilitator Implementation Log

Instrument 6. Youth Focus Group Protocol

1 Neelan, T., DeLisle D., & Zief, S. (2022). The Title V Competitive and General Departmental Grantees’ Sexual Risk Avoidance Education Program Plans (OPRE Report No. #2022-91). Washington, DC: Office of Planning, Research, and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

2 Self-Regulation Training Approaches and Resources to Improve Staff Capacity for Implementing Healthy Marriage Services for Youth (SARHM, OMB Clearance #0970-0355, approved June 2018); Next Steps for Rigorous Research on Two-Generation Approaches (OMB Clearance #0970-0356, approved July 2020); and Strengthening the Implementation of Marriage and Relationship Programs (SIMR, OMB Clearance #0970-0531, approved October 2021).

11