Supporting Statement B - NWS Substudy 04.05.23

Alternative Supporting Statement Instructions for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

SRAENE Substudy: Delivering SRAE Programming to Youth in Juvenile Justice Detention Centers and Foster Care Group Homes

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

April 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Calonie Gray

MeGan Hill

Tia Brown

Part B

B1. Objectives

Study Objectives

The objective of this proposed formative data collection is to collect data from Sexual Risk Avoidance Education (SRAE) grantees and providers about their experiences implementing programs in juvenile justice detention centers and foster care group homes, including their successes, challenges, and lessons learned. The information collected through this study will be used to inform the development of future descriptive research focused on delivering SRAE programming to youth in these two community settings. Specifically, data from this formative study will fill current gaps in knowledge about the unique implementation experiences among these providers and inform ACF’s research agenda for further studies of SRAE programming provided to youth in these settings. Additionally, the study findings will inform future technical assistance offerings to grantees and providers implementing SRAE programs in these settings.

The study research questions are described in Supporting Statement Part A, Section A2.

Generalizability of Results

This study is intended to present an internally valid description of SRAE grantees and providers delivering SRAE programming to youth in these two specific settings and is not meant to promote statistical generalization to other program, sites, or service populations. Data collected under this generic information collection request will be used to inform the development of a descriptive research study focused on SRAE programming within community settings.

Appropriateness of Study Design and Methods for Planned Uses

We will conduct one-on-one asynchronous virtual interviews. Respondents will log in to an online platform and answer posted interview questions moderated by the study team at Mathematica¹. The flexibility of a self-administered instrument combined with the benefits of study team moderation to probe and follow up on open-ended responses enables rapid collection of qualitative data from a diverse set of grantees and providers.

The study design lends itself well to this population, as grantee and provider staff often have competing demands and might find it difficult to set aside time during the day for a phone interview. The chat board format allows respondents to complete the interview at their convenience over a five-day period. In addition, the design is appropriate to collect the data needed to meet the study objectives. The interview protocol (Instrument 1) includes a mix of closed- and open-ended questions; the open-ended questions enable the study team to collect more in-depth qualitative data to capture the diversity of grantee and providers’ experiences adapting SRAE programming to serve youth in juvenile justice detention centers and foster care group homes. The chat board format lets the study team moderate across interviews, following up with individual respondents as needed to ensure they collect the data necessary to address the study questions. We describe key limitations of the design in Supporting Statement Part A, Section A2, and will include them in internal written products associated with the study.

This information is not intended to be used as the principal basis for public policy decisions, and it is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The population of interest for this study is a subset of all organizations funded through the SRAE grant program, that is, only those grantees and providers who: 1) deliver direct SRAE services, and 2) serve youth in juvenile justice detention centers and/or foster care group homes. Respondents of interest for the interviews are the grantee administrators or program directors, whomever the grantee identifies as the most knowledgeable respondent.

To identify the target populations, we will use survey data collected under the SRAENE Nationwide Study (NWS) (OMB Control Number 0970-0596; Exp Date 10/31/2025). The NWS collects data from SRAE grantees, including state-level grantees and their providers, and direct service provider grantees. The surveys ask the grantee administrator or program director about settings in which youth receive SRAE programming, including whether they provide SRAE programming to youth in juvenile justice detention centers, and/or foster care group homes. Grantees and providers eligible for this study are those that indicate in the NWS grantee or provider survey that they serve youth in one or both of these settings. Once the programs have been selected, the Administration for Children and Families (ACF) will review and approve prior to the start of data collection. An early review of data from the NWS surveys and performance measures suggests that there will be approximately 75 eligible programs.

Sampling and Site Selection

Once the programs have been selected and approved, we will invite up to 75 of the programs selected to complete the interview. Respondents involved in data collection will be from a convenience sample; they may not be representative of the population all SRAE programs serve.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

The study team developed the Interview protocol for this study, in collaboration with ACF, to address four research questions that reflect the information needs for ACF regarding SRAE programming conducted by grantees serving youth in the two settings of interest. The NWS survey² collects high-level quantitative data on the populations SRAE programs serve and the settings, including juvenile justice detention centers and foster care group homes, where grantees serve youth. The NWS survey data provide prevalence rates, but do not provide in-depth, contextual data that this study will collect from providers serving youth in these specific settings. The three primary topics covered by this study’s data collection protocol include: (1) how grantees and providers serve youth in juvenile justice detention centers and foster care group homes, including how they identify a curriculum and customize SRAE programming, how they recruit, enroll, and engage youth, how they train and support staff, and how they build partnerships; (2) the extent to which grantees and providers utilized different strategies when serving youth in juvenile justice detention centers and foster care group homes compared to when they served youth in other settings; and (3) the successes and challenges grantees and providers have experienced with their programs. The interview protocol includes a combination of open-ended items that allow providers to provide rich qualitative responses about their experiences and close-ended items to meet the study objectives.

The study design reduces measurement error in two ways: (1) study leads can oversee all interview moderators in the chat board, ensuring consistency in grantee and provider probes and follow-ups and clarifying unclear or ambiguous responses, and (2) the online platform exports a transcript of all comments to interviewers, preventing errors in notetaking and transcription that could occur during a telephone or in-person interview.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection.

Mathematica will conduct the interviews using QualBoard, a chat board platform where respondents can see and respond to questions. The interview will not be a real-time chat; respondents can log on at any time convenient for them. Individual respondents will not be able to see other participants’ responses. Qualboard stores responses in a database only accessible to the study team and responses will be used as an interview transcript.

Data collection will occur on a rolling basis. We will stagger the data collection over a five-week period, to limit the number of respondents accessing the board at one time. Smaller groups will also enable the study team to more carefully review and moderate interviews. We will release the interviews in groups of roughly 10 to 15 respondents each week. We expect respondents to complete the interview within one week, but some may require up to two weeks.

Before data collection, the study team will send an email inviting grantee or provider program directors to participate in an interview (Appendix A: Study Notification and Reminder Materials). The invitation will explain the purpose of the interview and detail the data collection mode. It will include a proposed week of data collection, to ensure respondents will be available during their data collection window. Following the initial invitation, respondents will receive log-in credentials via email to access the QualBoard site. The contractor will follow up with interview nonrespondents to encourage their participation by highlighting the importance of the interviews. Federal project officers might also contact nonrespondents to encourage their participation. In addition, to encourage participation and to provide flexibility, upon request, we will allow respondents to complete the protocol via telephone with a trained member of the study team. The assigned moderator will add the data collected over the phone to Qualboard.

A trained member of Mathematica’s study team will conduct each interview. Questions from the interview protocol will be programmed into chat boxes in QualBoard, and respondents will type in their answers. Interview moderators will be trained on the interview protocol, and the project lead will review all moderators’ sessions for consistency in engagement with respondents. Throughout the sessions, moderators will review responses. When necessary, they will encourage more in-depth responses and probe respondents to elaborate on responses. When the moderator asks a follow-up question, the respondent will receive an email notification that follow-up on their interview has been requested.

Table B.1. shows the total expected number of participants, the data collection mode and expected time to complete the interview, and that this is a one-time data collection request of the participants.

Table B.1. SRAENE Formative Evaluation for the Program Components Impact Study: Data collection activities

Data Collection	Administration plans
Interview protocol for SRAE grantees and providers delivering SRAE programming to youth in juvenile justice detention centers and foster care group homes	Total participants	50
	Mode	Web
	Time	60 minutes
	Frequency	1

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The interviews are not designed to produce statistically generalizable findings.

At the time of their application, grantees were notified that participation in any of the SRAENE activities is mandatory. However, study respondents will be informed that they can skip questions that they do not feel comfortable answering. Responses will be monitored and the study team and/or the Family and Youth Services Bureau (FYSB) project officers will send reminders to nonrespondents. The study team will also offer telephone interviews for grantees and providers if they prefer that mode over the QualBoard interface.

NonResponse

Participants will not be randomly sampled, and findings are not intended to be representative. Consequently, we will not calculate nonresponse bias. Respondent grantee type will be documented and reported in written data collection materials.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Interview data retrieved from QualBoard will be saved on a secure drive accessible only to Mathematica study team members. Direct export of the QualBoard data to the secure drive will result in minimal processing. Interviewers will review transcripts to fix spelling and grammar issues, fill in missing words, and explain unclear terms or phrases in preparation for qualitative coding and analysis.

Data Analysis

Qualitative data from the interview transcripts will be reviewed for overarching themes and lessons on each of the key topics explored through the virtual discussions. The study team will develop a coding scheme based on the research objectives and interview topics. Moderators will apply the coding scheme to the transcripts and conduct a thematic analysis of the responses under each topic. The task leads for the interviews will monitor coding and thematic analysis across the team to ensure accuracy and consistency.

Data Use

Data collected will be used to improve ACF’s understanding of understanding of how SRAE programs are implemented in juvenile justice and foster care group home settings.

The findings from this formative data collection will inform the development of future descriptive research on delivering SRAE programs to youth in community settings, specifically to youth engaged in the justice and child welfare systems. It will fill key gaps in knowledge and lay the foundation for developing a research agenda for future, rigorous exploration into how SRAE programming is implemented and experienced by youth within community settings. The findings will also support future SRAE technical assistance planning and resources for grantees serving youth in these settings. The primary purpose of the information collected is not publication, but findings may be incorporated into materials that are made publicly available. For example, to contextualize the study design plans for a future study, ACF may reference the findings from this formative phase.

B8. Contact Persons

In Table B.2, we list the federal and contract staff responsible for the study, including their affiliation and email address.

Table B.2. Individuals Responsible for Study

Name	Affiliation	Email address
Calonie Gray	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Calonie.Gray@acf.hhs.gov
MeGan Hill	Family and Youth Services Bureau Administration for Children and Families U.S. Department of Health and Human Services	Megan.Hill@acf.hhs.gov
Tia Brown	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Tia.Brown@acf.hhs.gov
Susan Zief	Mathematica	SZief@mathematica-mpr.com
Betsy Keating	Mathematica	EKeating@mathematica-mpr.com
Theresa Neelan	Mathematica	TNeelan@mathematica-mpr.com
Bess Fleischman	Mathematica	BFleischman@mathematica-mpr.com
Tiffany Waits	Mathematica	TWaits@mathematica-mpr.com

Attachments

Appendices

Appendix A: Study Notification and Reminder Materials

Instruments

Instrument 1. Interview Protocol for SRAE Grantees and Providers Delivering SRAE Programming to Youth in Juvenile Justice Detention Centers and Foster Care Group Homes

1 ACF has contracted with Mathematica to carry out this information collection.

2 The NWS survey is approved and conducted under OMB Control No: 0970-0956.

6