SSB Pretesting GenIC - Youth Programs –ACF Youth Participant Entry and Exit Survey Pretest

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Administration for Children and Families Youth Programs – Youth Participant Entry and Exit Survey Pretest

Pre-testing of Evaluation Data Collection Activities

0970 - 0355

Supporting Statement

Part B

June 2025

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Caryn Blitz, Tia Brown

Part B

B1. Objectives

Study Objectives

The Administration for Children and Families (ACF) is currently engaged in efforts to improve Personal Responsibility Education Program (PREP) and Sexual Risk Avoidance Education Program (SRAE) performance measures. These efforts include a focus on improving youth participant entry and exit survey questions by simplifying language, eliminating skip patterns, and shortening the length of the surveys to increase youth comprehension and survey completion rates.

This information collection request includes pretesting revised participant entry and exit surveys with youth who have or are currently participating in PREP or SRAE programs. The proposed pretests include (1) administering the draft revised entry and exit surveys to youth, and then (2) conducting in-person focus groups. The pretests will explore how youth understand the surveys, the relevance of the language used, and their usefulness in real world application. The pretest will also provide estimates of the time required to complete the revised surveys. ACF is not interested in and will not document the survey responses, just the process youth went through to answer the questions and how long it took to complete the survey. Pretests will be conducted with both English- and Spanish-speaking youth.

ACF proposes the following guiding questions for the pretests with youth:

• Do the youth respondents easily understand the questions and response options?

• Do the items accurately and adequately capture youth youth’s characteristics, experiences, and behavioral intentions?

• How long does it take for youth to complete the survey?

Generalizability of Results

The proposed pretests are intended to inform instrument development for revisions to the PREP and SRAE participant surveys (approved under OMB #: 0970-0497 and 0970-0536, respectively) by obtaining feedback from program youth. It is not designed to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

Data collected under this generic clearance will be used to inform the revisions of PREP and SRAE participant surveys. The focus groups will help ACF understand how youth interpret the survey items, including how they understand and operationalize the items, and also assess real-world application.

Group cognitive interviews have been recommended over traditional one-on-one cognitive interviews with a teen population (Norris et al 2014)¹, with the researchers noting that the brainstorming process typical of focus groups allowed them to “find and fine tune alternative wording for items that were unclear or confusing,” and that the measures they revised through this process were completed without difficulty in a later study. This research team also employed red and green cards to signify likes and dislikes in part of their group interviews and found that their population of teen girls was very engaged in making use of the cards; our team plans to employ this same tactic with the focus groups.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. Limitations will be described in any written products associated with these pretesting activities. 

B2. Methods and Design

Target Population

The target populations for this generic information collection request are youth ages 12 through 19 who have or are participating in PREP or SRAE programs.

Sampling and Site Selection

ACF will identify PREP and SRAE program sites from which to recruit youth for the pretests. All youth participating in PREP or SRAE programs at the selected sites will be eligible to participate in the pretests, provided they assent and that the parental consent requirements of the site and its Institutional Review Board (IRB), if any, are met.² The study team will coordinate with the program sites to collect parental consent and youth assent for those under 18, or participant consent for those 18 or older (Appendix A).

We expect that across the sites, approximately 72 youth will agree to participate in the pretests. We anticipate that the youth program participants will be middle and high school students between the ages of 12 and 19. Program participants involved in data collection will be from a convenience sample and may not be representative of the population that all PREP and SRAE programs serve.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Listening sessions with grant recipients were conducted under the formative generic clearance (0970-0356) to identify challenges youth were having with the entry and exit surveys and to obtain suggested revisions. Grant recipients reported that youth completing the entry and exit surveys experience challenges with the reading level being too advanced, questions that include double-negatives, survey fatigue due to survey length, and following skip patterns. Listening sessions were also held with Family and Youth Services Bureau (FYSB) federal project officers and leadership to obtain additional suggested revisions and to ensure the program office’s data needs could still be met while reducing the survey length. The findings from these listening sessions informed the current draft surveys (Instruments 1 – 8), which will be further refined through the survey pretests with PREP and SRAE youth program participants.

To develop Instrument 9, the Focus Group Protocol, we considered the overall goals of the pretests; namely, to develop an understanding of how clear and understandable the survey questions are to youth, and collect youth feedback on ways to improve the surveys. Aligned with current best practices for conducting focus groups, we developed high-level questions to ask each group of youth to consider thinking across all of the questions in the surveys, as well as a set of more specific questions for some of the revised survey items.

B4. Collection of Data and Quality Control

ACF is contracting with Mathematica for this data collection, and Mathematica will oversee all data collection and ensure quality control measures are in place and followed. To ensure an efficient and standardized data collection process, Mathematica study staff will participate in a project-specific data collection training. The training will cover all aspects of the data collection. All staff will be trained on best practices for collecting high-quality data and procedures around data privacy and security.

The pretests will be conducted in person, on site, for select SRAE and PREP programs. Mathematica will collect data by administering hard-copy participant entry and exit surveys to consented youth. Youth will be asked to make note of any items they find confusing and any words or phrases they do not understand. Youth can use the paper surveys during the focus group to refer to during the discussion, but the surveys will be destroyed after the focus group ends. ACF is not interested in and will not document the survey responses, just the process youth went through to answer the questions and how long it took to complete the survey. Once all youth in a group have completed the surveys, Mathematica staff will conduct a 60-minute focus group discussion about their experience completing the surveys, using a data collection protocol (Instrument 9). Discussions with Spanish-speaking youth will be led by moderators who are fluent in Spanish. A notetaker from Mathematica will take live notes, and with participant permission, the sessions will be audio recorded to ensure accurate collection of data. Following each discussion group, the moderator and notetaker will debrief on the session together. The notetaker will finalize the notes, which will be reviewed by the moderator to ensure accuracy and completeness.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

We will track the total number of youth at each site who provided consent to participate in the pretests, along with those who completed the pretests.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the participant pretest data collection.

B6. Production of Estimates and Projections

The goal of the proposed pretests is to collect information from youth to support ACF’s effort in revising PREP and SRAE participants entry and exit surveys. The data collected will document (1) whether the questions are age appropriate, and (2) whether and how the questions can be interpreted by respondents, and 3) the relevance of the language used.

The data will not be used to generate population estimates, either for internal use or for dissemination. Policy decisions will not be made based on the data collected for these pretests, as the data are not representative.

B7. Data Handling and Analysis

Data Handling

For the pretests, we will collaborate with select grant recipients within ACF to recruit youth who are currently enrolled in, or recently completed, a PREP or SRAE program. For that recruitment effort, ACF or the grant recipients may need to provide first names and email addresses to the contractor to coordinate data collection. The Mathematica staff will work together with sites to manage the consent form process, and youth and parent names will not be linked to the focus group data. Immediately after completing the focus groups, all names and contact information will be destroyed. Data collected from the focus group discussions will not be linked by personally identifiable information (PII).

Names will not be included on the hard copy surveys. The focus group discussions will be audio-recorded for notetaking, with the consent of all group participants. We will wait to begin recording the discussion until after everyone has introduced themselves. The transcribed notes will not include any names.

All notes and recordings will be stored on Mathematica’s secure network. No one outside the study team will have access to the data. Only Mathematica staff working directly on this project have access to the project folder on the network where recordings will be saved. All recordings will be destroyed as soon as they have been transcribed and notes will be destroyed per contract requirements. All paper surveys will be destroyed after completing the pretest.

Data Analysis

This pretesting activities will not employ complex data analytic techniques. We will not be analyzing the survey data, since the purpose of the pretests is to examine the process of completing the surveys, not to assess the survey responses.

The study team will analyze the focus group data qualitatively, using feedback from the participants to assess and improve the clarity and comprehension of the participant entry and exit surveys. This will include coding the discussion notes to identify recurring themes or patterns across the items, as well as comments that may indicate potential issues with a particular item. The analysis will focus on understanding the context and reasons behind any challenges in comprehension or responding to the survey items and will determine what revisions can be made to improve the clarity and relevance of the items.

Data Use

ACF is currently engaged in efforts to improve the PREP and SRAE participant entry and exit survey questions. Findings from the pretests will inform these efforts.

After conducting the pretests, the study team will develop an internal ACF memorandum describing the pretest methodology and results, including any proposed changes to the surveys, and justification for the proposed changes.

ACF will use the results internally to inform revisions to entry and exit surveys administered to youth participants in PREP and SRAE programs. The pretests will also provide estimates of the time required to complete the revised surveys.

B8. Contact Persons

Table B.2 lists the federal and contract staff responsible for the study, including their affiliation and email address.

Table B.2. Staff responsible for study

Name	Affiliation	Email address
Caryn Blitz	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Caryn.Blitz@acf.hhs.gov
Tia Brown	Office of Planning, Research, and Evaluation Administration for Children and Families U.S. Department of Health and Human Services	Tia.Brown@acf.hhs.gov
Brian Goesling	Mathematica	bgoesling@mathematica-mpr.com
Lara Hulsey	Mathematica	lhulsey@mathematica-mpr.com
Melissa Thomas	Mathematica	mthomas@mathematica-mpr.com
Maya Reid	Mathematica	mreid@mathematica-mpr.com

Attachments

Appendix A: Consent and Assent Forms – English

Appendix A: Consent and Assent Forms –Spanish

Instrument 1: SRAE Participant Entry Survey for high school and older youth – English

Instrument 1a: SRAE Participant Entry Survey for high school and older youth – Spanish

Instrument 2: SRAE Participant Entry Survey for middle school youth – English

Instrument 2a: SRAE Participant Entry Survey for middle school youth – Spanish

Instrument 3: SRAE Participant Exit Survey for high school and older youth – English

Instrument 3a: SRAE Participant Exit Survey for high school and older youth – Spanish

Instrument 4: SRAE Participant Exit Survey for middle school youth – English

Instrument 4a: SRAE Participant Exit Survey for middle school youth – Spanish

Instrument 5: PREP Participant Entry Survey for high school and older youth – English

Instrument 5a: PREP Participant Entry Survey for high school and older youth – Spanish

Instrument 6: PREP Participant Entry Survey for middle school youth – English

Instrument 6a: PREP Participant Entry Survey for middle school youth – Spanish

Instrument 7: PREP Participant Exit Survey for high school and older youth – English

Instrument 7a: PREP Participant Exit Survey for high school and older youth – Spanish

Instrument 8: PREP Participant Exit Survey for middle school youth – English

Instrument 8a: PREP Participant Exit Survey for middle school youth – Spanish

Instrument 9: Focus Group Discussion Protocol – English

Instrument 9a: Focus Group Discussion Protocol – Spanish

1 Norris AE, Torres-Thomas S, Williams ET. Adapting cognitive interviewing for early adolescent Hispanic girls and sensitive topics. Hisp Health Care Int. 2014;12(3):111-9. doi: 10.1891/1540-4153.12.3.111. PMID: 25239207; PMCID: PMC4459492. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4459492/

2 If a participating site does not have a local IRB, data collection activities will be covered under Mathematica’s IRB, Health Media Labs (HML).

8