SSA Pretesting GenIC - Youth Programs –ACF Youth Participant Entry and Exit Survey Pretest

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Administration for Children and Families Youth Programs –Youth Participant Entry and Exit Survey Pretest

Pre-testing of Evaluation Data Collection Activities

0970 – 0355

Supporting Statement

Part A

June 2025

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Caryn Blitz, Tia Brown

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Pre-testing of Evaluation Data Collection Activities (0970-0355).

• Progress to Date: The Administration for Children and Families (ACF) is currently engaged in efforts to improve Personal Responsibility Education Program (PREP) and Sexual Risk Avoidance Education Program (SRAE) performance measures. PREP Performance Measures are currently approved under OMB #0970-0497 and SRAE performance measures are currently approved under OMB #0970-0536. These efforts include a focus on improving youth participant entry and exit survey questions by simplifying language, eliminating skip patterns, and shortening the length of the surveys to increase youth comprehension and survey completion rates.

• Description of Request: This information collection request is to pretest draft survey questions aimed at providing youth data for performance measures across PREP and SRAE. The proposed pretests include separate focus groups with PREP and SRAE youth participants to explore how youth understand the survey questions and the relevance of the language used. Listening sessions with grant recipients were conducted under the formative generic clearance for research (0970-0356) to identify challenges youth were having with the surveys and to obtain suggested revisions. Listening sessions were also held with Family and Youth Services Bureau (FYSB) federal project officers and leadership to obtain additional suggested revisions and to ensure the program office’s data needs could still be met while reducing survey length. The findings from these listening sessions informed the current draft surveys, which will be further refined through pretests of the surveys with PREP and SRAE youth program participants.

Data collected in the study are not intended to be generalized to a broader population, nor do we intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: ACF requests approval of the proposed data collections as soon as possible so the final, revised surveys can be included in a subsequent information collection revision request, which will need to be approved by December 2025.

A1. Necessity for Collection

The Administration for Children and Families (ACF) is currently engaged in efforts to improve the participant entry and exit surveys used by the Personal Responsibility Education Program (PREP) and Sexual Risk Avoidance Education Program (SRAE) to obtain performance measures data on youth characteristics, a range of adolescent behaviors, and the perceived influence of the programs on behavioral intentions at program exit. PREP Performance Measures are currently approved under OMB #0970-0497 and SRAE performance measures are currently approved under OMB #0970-0536. Challenges with the surveys reported by grant recipients include: the reading level being too advanced, questions that include double-negatives, survey fatigue due to survey length, and confusion with skip patterns.

A2. Purpose

Purpose and Use

This information collection will inform revisions to entry and exit surveys administered to youth program participants in PREP and SRAE programs. There are two key purposes of the pretests. The first purpose has to do with youth thought process and comprehension. By having youth respond to the survey questions, we can better understand what they were thinking about when they answered the question, how they interpreted the question (including instructions and response options), how easy or difficult it was for them to answer the question, and whether they think other youth would be able to understand the question. The second purpose of the pretests is to better understand how the survey questions flow (question order), and to determine how long it takes youth to complete the survey, which is required to calculate respondent burden. Because we are proposing changes to previously approved instruments, it is important to get a more accurate estimate of the average time per respondent to ensure we are being responsive to grant recipients’ concerns about the previous surveys’ length.

Findings from this data collection will ultimately inform the development of the final PREP and SRAE youth participant entry and exit surveys. ACF will use the results internally to inform subsequent information collection requests, which will be submitted to OMB for review and approval. ACF is not interested in and will not document the actual survey responses, just the process youth went through to answer the questions and how long it took to complete the survey.

This proposed information collection meets the primary goals of ACF’s generic clearance for pretesting (0970-0355): to develop and test information collection instruments and procedures.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Guiding Questions

ACF proposes the following guiding questions for this pretest:¹

• Do the youth respondents easily understand the questions and response options?

• Do the items accurately and adequately capture youths’ characteristics, experiences, and behavioral intentions?

• How long does it take for youth to complete the survey?

Study Design

The proposed pretests include (1) administration of in-person draft revised performance measures entry and exit surveys to a subset of PREP and SRAE youth program participants, and then (2) conducting in-person focus groups with youth to evaluate the readability of the items and real-world application.

ACF will identify PREP and SRAE programs appropriate for pretests. Where possible, the study team will balance geography, community type (rural, urban, suburban), and populations. The study team will work directly with programs to recruit youth for the pretests. Youth who provide assent and (for those under 18) parental consent will be eligible to participate in pretesting.

Table A.1 summarizes the study design, including the data collection instruments, their content and respondent types, and the mode and duration of each data collection activity.

Table A.1. Study design summary

Data Collection Activity	Instruments2	Respondent, Content, Purpose of Collection	Mode and Duration
Administer youth participant survey	Instrument 1: SRAE Entry Survey Version for High School or Older Youth Instrument 2: SRAE Entry Survey Version for Middle School Youth Instrument 3: SRAE Exit Survey Version for High School or Older Youth Instrument 4: SRAE Exit Survey Version for Middle School Youth Instrument 5: PREP Entry Survey Version for High School and Older Youth Instrument 6: PREP Entry Survey Version for Middle School Youth Instrument 7: PREP Exit Survey Version for High School and Older Youth Instrument 8: PREP Exit Survey Version for Middle School Youth	Respondents: Former and current SRAE and PREP program participants, ages 12 to 19 Content: Questions pertaining to participants’ demographic and behavioral characteristics and perceptions of program effects Purpose: To assess clarity and understanding of entry and exit survey items and real world application	Mode: In-person Duration: 0.117 to 0.167 hour per instrument
In-person focus groups with youth participants in ACF programs	Instrument 9: Focus Group Discussion protocol	Respondents: Former and current SRAE and PREP program participants, ages 12 to 19 Content: Questions around comprehension of items, how youth processed their responses, ease of response, and thoughts on revisions. Purpose: To assess clarity and understanding of entry and exit survey items and real world application	Mode: In-person Duration: 1 hour

Other Data Sources and Uses of Information

This request covers all data to be collected for the purpose of revising the youth entry and exit surveys for the SRAE and PREP programs. Listening sessions with grant recipients were conducted under the formative generic clearance for research (0970-0356) to identify challenges youth were having with the surveys and to obtain suggested revisions. Listening sessions were also held with Family and Youth Services Bureau (FYSB) federal project officers and leadership to obtain additional suggested revisions and to ensure the program office’s data needs could still be met while reducing the survey length. The findings from these listening sessions informed the current draft surveys, which will be further refined through survey pretests with PREP and SRAE youth program participants.

A3. Use of Information Technology to Reduce Burden

Survey administration and focus group discussions with youth will be conducted in person. After obtaining permission from each participant, the Mathematica study team will audio-record all discussion groups with youth. These recordings will ensure that information is accurately captured. The Mathematica study team will use the recordings to supplement the notes taken during the focus group discussions.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The revised surveys have not yet been pretested or fielded, ensuring no possibility of duplication with other data collection efforts. ACF believes that pretesting these surveys with a small group of SRAE and PREP program participants will inform and ensure that youth will more easily understand and complete the final surveys, thereby providing more useful and accurate performance measures information to grant recipients and ACF staff.

A5. Impact on Small Businesses

Youth participating in the pretests will be recruited from ACF grant programs run by small, nonprofit organizations. The study team will work closely with program staff to schedule the pretest activities and restrict information to that only necessary for the pretesting activities. The pretests will be scheduled in conjunction with scheduled program activities to help reduce burden on youth and the program staff. The study team will be flexible in case some programs require a different scheduling process.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to submit a request to OMB for review of the overarching generic clearance for pre-testing activities. This notice was published on June 20, 2024 (89 FR 51888) and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received. A second notice in the Federal Register announcing the agency’s submission of the overarching generic clearance for pre-testing activities for OMB’s review. This notice was published on August 26, 2024 (89 FR 68444) and provided a thirty-day period for public comment. During the notice and comment period, no comments were received.

Consultation with Experts Outside of the Study

We consulted with the following individuals and groups to inform the identification of survey questions for pretesting: Dr. Brett Brown, Office of Planning, Research, and Evaluation Division of Data and Improvement; FYSB federal project officers (FPOs) and leadership; and PREP and SRAE grant recipients. The listening sessions (held under the formative generic clearance for research (0970-0356)) included staff from 29 PREP and 42 SRAE grant recipients across the various funding streams.

A9. Tokens of Appreciation

To recognize the value of the input provided by participants, help to facilitate participation, and improve the quality of the data for this effort, we plan to provide participants with a token of appreciation.

Focus group data are not intended to be representative in a statistical sense, in that they will not be used to make statements about the prevalence of experiences of youth participating in PREP and SRAE. However, it is important to secure enough participants to provide quality data and capture the experiences of youth participating in different SRAE and PREP programs. The proposed revisions to the entry and exit surveys are due in part to feedback grant recipients received from youth participating in these programs, whose comments highlighted challenges and concerns with the existing surveys. Obtaining feedback on the proposed revisions from youth that reflect the larger population of interest is essential in determining how well the revised items work for youth participating in the PREP and SRAE programs. It is important that within the group of program participants, we also gather feedback from youth of varying ages, to fully capture how well the items work for youth of all ages served by these programs.

To ensure recruitment of a sufficient number of youth and to recognize the importance of information provided by program participants, we propose offering participant $50 for their participation in the pretest data collection activities (completion of two surveys and participation in a 60-minute focus group discussion). Youth will be asked to participate on their own time, outside of regular programming, and providing a token of appreciation should help facilitate participation. This token of appreciation also affirms that the contributions from those with experience in the programs of interest are as valuable as those of other experts.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

We will collaborate with FYSB to recruit youth who are currently enrolled in, or recently completed, a PREP or SRAE program. For that recruitment effort, ACF may need to provide first names and email addresses to the contractor to coordinate data collection. The sites will manage the consent form process, and youth and parent names will not be linkable to the survey or focus group data. Surveys will not collect youth name, site or program name, or other PII. Immediately after completing the focus groups, all names and contact information will be destroyed. Data collected from the focus group discussions will not be linkable by the PII.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by a personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

Participants and their parents or guardians will be informed that paper surveys will not collect any identifying information. Youth can use the paper surveys during the focus group to refer to during the discussion, but the surveys will be destroyed after the focus group ends. ACF is not interested in and will not document the survey responses, just the process youth went through to answer the questions and how long it took to complete the survey. Participants and their parents or guardians will also be informed that focus groups will be audio-recorded for notetaking, and that they may opt out of these recordings (see Appendix A). Groups will only be recorded if all participants in the group agree to being recorded. The participant consent form also includes language explaining privacy risks associated with participating in a focus group discussion. The study team will begin recording the discussions after everyone has introduced themselves so that names will not be included in the recording or transcribed notes.

Youth will be informed that participation is voluntary, nothing they say will be identified as coming from them, and they can refuse to answer any questions they do not wish to answer.

All notes and recordings will be stored on Mathematica’s secure network. No one outside the study team will have access to the data. Only Mathematica staff working directly on this project have access to the project folder on the network where recordings will be saved. All audio recordings will be destroyed as soon as they have been transcribed and notes will be destroyed per contract requirements.

The pretesting plans will be reviewed by Mathematica’s Institutional Review Board (IRB), the Health Media Lab, and the IRBs overseeing the data collection of the selected programs participating in the pretests, if needed.³ Outreach and data collection will not begin until Mathematica has received IRB approval for the pretests.

Data Security and Monitoring

The contractor shall protect respondents’ privacy to the extent permitted by law and will comply with all federal and departmental regulations for private information The contractor will ensure all employees receive training on data privacy issues and comply with all requirements. All Mathematica staff must sign an agreement to (1) maintain the privacy of any information from individuals, businesses, organizations, or families participating in any projects conducted by Mathematica; (2) complete online security awareness training when they are hired; and (3) participate in a refresher training annually.

The contractor will use encryption compliant with the Federal Information Processing Standard (Security Requirements for Cryptographic Module, as amended) to protect all sensitive information during storage and transmission. The contractor will securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the standard. The contractor will incorporate the standard into its property management and control system and establish a procedure to account for all laptop and desktop computers and other mobile devices and portable media that store or process sensitive information. The contractor will secure any data stored electronically in accordance with the most current National Institute of Standards and Technology requirements and other applicable federal and departmental regulations. In addition, the contractor’s data safety and monitoring plan includes strategies for minimizing risk, to the extent possible including sensitive information on paper records and for protecting any paper records, field notes, or other documents that contain sensitive information to ensure secure storage and limits on access.

No information will be given to anyone outside the study team and ACF.

A11. Sensitive Information ⁴

The goal of this data collection is to inform the revisions to the PREP and SRAE youth participant entry and exit surveys, which ask about youth sexual behaviors and other risk behaviors. As such, some of the surveys’ questions are of a sensitive nature. However, ACF is not interested in and will not document the actual survey responses, just the process youth went through to answer the questions and how long it took to complete the survey. Youth will be asked to provide feedback on the clarity of the questions and response options. This feedback will allow ACF to develop final surveys that are relevant to the population and designed in a way that accurately captures information important for PREP and SRAE program monitoring and continuous quality improvement.

Youth focus group members will consist of current and/or past SRAE and PREP program participants. These youth will have been exposed to sensitive topics through their program participation and will have completed a version of the entry and exit surveys that included the sensitive questions. We seek to obtain youth feedback on those questions because we have revised them and their response categories and changed the flow in the survey. For example, based on grant recipient feedback, we deleted the skip patterns for sexual behavior questions in the exit surveys, resulting in stand-alone questions. Feedback is required on survey question and response category comprehension, survey flow, and response times.

All pretest participants will be informed that their participation is voluntary and that they can refuse to answer any questions they feel uncomfortable answering. This will be noted on the parent consent and youth assent form (Appendix A). Group moderators will reiterate this verbally at different points throughout the focus group discussions, as scripted in the Focus Group Discussion Protocol (Instrument 9). Additionally, participants will be informed that the goal of the pretests is about their understanding of the survey questions, not the content of their specific answers.

A12. Burden

Explanation of Burden Estimates

Table A.2 summarizes the estimated burden for each instrument. The following provides an overview of burden estimates. See Supporting Statement B for additional information about the target population and sampling.

The following eight survey instruments will be used in pretests followed by a group discussion using the protocol in Instrument 9:⁵

1. Instrument 1: SRAE Entry Survey Version for High School or Older Youth

• Participants: Former and current SRAE program participants who are in the age range served by high schools in their community or older. We estimate pretesting this version with 18 youth.

• Time to complete: 7 minutes

2. Instrument 2: SRAE Entry Survey Version for Middle School Youth

• Participants: Former and current SRAE program participants aged 12 and older who are enrolled in middle school. We estimate pretesting this version with 18 youth.

• Time to complete: 7 minutes

3. Instrument 3: SRAE Exit Survey Version for High School or Older Youth

• Participants: Former and current SRAE program participants who are in the age range served by high schools in their community or older. We estimate pretesting this version with 18 youth.

• Time to complete: 10 minutes

4. Instrument 4: SRAE Exit Survey Version for Middle School Youth

• Participants: Former and current SRAE program participants aged 12 and older who are enrolled in middle school. We estimate pretesting this version with 18 youth.

• Time to complete: 10 minutes

5. Instrument 5: PREP Entry Survey Version for High School and Older Youth

• Participants: Former and current PREP program participants who are in the age range served by high schools in their community or older. We estimate pretesting this version with 18 youth.

• Time to complete: 7 minutes

6. Instrument 6: PREP Entry Survey Version for Middle School Youth

• Participants: Former and current PREP program participants aged 12 and older who are enrolled in middle school. We estimate pretesting this version with 18 youth.

• Time to complete: 7 minutes

7. Instrument 7: PREP Exit Survey Version for High School and Older Youth

• Participants: Former and current PREP program participants who are in the age range served by high schools in their community or older. We estimate pretesting this version with 18 youth.

• Time to complete: 7 minutes

8. Instrument 8: PREP Exit Survey Version for Middle School Youth

• Participants: Former and current PREP program participants aged 12 and older who are enrolled in middle school. We estimate pretesting this version with 18 youth.

• Time to complete: 7 minutes

9. Instrument 9: Youth Focus Group Discussion Protocol

• Participants: Former and current PREP and SRAE program participants aged 12-19 who participate in the pretest of the entry and exit surveys. We estimate using the focus group protocol with 72 youth.

• Time to complete: 60 minutes

Estimated Annualized Cost to Respondents

We estimate the average hourly wage for youth respondents at $7.25, based on the federal minimum wage from the U.S. Bureau of Labor Statistics, Occupational Employment Statistics (OES).⁶ For each instrument listed in Table A.2, the study team calculated the total annual cost by multiplying the annual burden hours by the current federal minimum wage ($7.25), as listed in Table A2.

Table A.2. Total burden requested under this information collection

Instrumentb	No. of Respondents (total over request period) a	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/AnnualBurden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Instrument 1: SRAE Entry Survey Version for High School or Older Youth	18	1	0.117	2.1	$7.25	$15.27
Instrument 2: SRAE Entry Survey Version for Middle School Youth	18	1	0.117	2.1	$7.25	$15.27
Instrument 3: SRAE Exit Survey Version for High School or Older Youth	18	1	0.167	3.0	$7.25	$21.79
Instrument 4: SRAE Exit Survey Version for Middle School Youth	18	1	0.167	3.0	$7.25	$21.79
Instrument 5: PREP Entry Survey Version for High School or Older Youth	18	1	0.117	2.1	$7.25	$15.27
Instrument 6: PREP Entry Survey Version for Middle School Youth	18	1	0.117	2.1	$7.25	$15.27
Instrument 7: PREP Exit Survey Version for High School or Older Youth	18	1	0.117	2.1	$7.25	$15.27
Instrument 8: PREP Exit Survey Version for Middle School Youth	18	1	0.117	2.1	$7.25	$15.27
Instrument 9: Youth Focus Group Discussion Protocol	72	1	1	72	$7.25	$522.00
Totals	72			90.6		$657.20

1. Youth will complete both the entry and exit survey for their program and age group but participate in only one focus group discussion.

2. A Spanish translation of each instrument is available.

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The estimated total cost to the federal government for this study is $250,000 (Table A.3). This includes costs for planning, recruitment, collecting, processing, and analyzing the data, and preparing memorandums.

Table A.3. Estimated total cost by category

Cost Category	Estimated Costs
Field Work (planning, recruitment, and data collection)	$242,500
Publications/Dissemination (analysis)	$7,500
Total/annual costs	$250,000

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella clearance for pre-testing (0970-0355).

A16. Timeline

The study team expects to conduct the pretests as soon as possible following OMB approval with a goal to complete the pretest effort by fall 2025. This would allow the team to analyze the focus group data, revise instruments, and finalize a revision request for the full Performance Measures information collections requests in time for approval in December 2025 (the current expiration date for OMB #0970-0536 is December 31, 2025).

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendix A1: Consent and Assent Forms (English)

Appendix A2: Consent and Assent Forms (Spanish)

Instrument 1a: SRAE Entry Survey Version for High School or Older Youth (English)

Instrument 1b: SRAE Entry Survey Version for High School or Older Youth (Spanish)

Instrument 2a: SRAE Entry Survey Version for Middle School Youth (English)

Instrument 2b: SRAE Entry Survey Version for Middle School Youth (Spanish)

Instrument 3a: SRAE Exit Survey Version for High School or Older Youth (English)

Instrument 3b: SRAE Exit Survey Version for High School or Older Youth (Spanish)

Instrument 4a: SRAE Exit Survey Version for Middle School Youth (English)

Instrument 4b: SRAE Exit Survey Version for Middle School Youth (Spanish)

Instrument 5a: PREP Entry Survey Version for High School or Older Youth (English)

Instrument 5b: PREP Entry Survey Version for High School or Older Youth (Spanish)

Instrument 6a: PREP Entry Survey Version for Middle School Youth (English)

Instrument 6b: PREP Entry Survey Version for Middle School Youth (Spanish)

Instrument 7a: PREP Exit Survey Version for High School or Older Youth (English)

Instrument 7b: PREP Exit Survey Version for High School or Older Youth (Spanish)

Instrument 8a: PREP Exit Survey Version for Middle School Youth (English)

Instrument 8b: PREP Exit Survey Version for Middle School Youth (Spanish)

Instrument 9a: Focus Group Discussion Protocol (English)

Instrument 9b: Focus Group Discussion Protocol (Spanish)

1 Both questions will be addressed through the focus groups. The pretest participants’ responses to the survey questions will not be analyzed.

2 A Spanish translation of each instrument is available.

3 Programs that participate in the pretests may be governed by their own IRBs. Some of these IRBs may accept Mathematica’s IRB approval and waive their own standard processes, but others may require that Mathematica complete their own standard IRB application process as well.

4 Examples of sensitive topics include (but not limited to): social security number; sexual behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

5 A Spanish translation of each instrument is available.

6 Characteristics of minimum wage workers, 2023: BLS Reports: U.S. Bureau of Labor Statistics

14