SSB_PREP PMs_0970-0497_023021203_clean_rev 060523

Personal Responsibility Education Program (PREP) Performance Measures

OMB Information Collection Request

0970-0497

Supporting Statement

Part B

JUNE 2023

Submitted by:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Caryn Blitz, Tia Brown

Part B

B1. Objectives

Study Objectives

The objective of the Personal Responsibility Education Program (PREP) performance measures (PM) effort is to document how PREP-funded programs are operationalized in the field and assess program outcomes. This performance measures effort calls for the collection and submission of performance measures data by State PREP (SPREP), Tribal PREP (TPREP), Competitive PREP (CPREP), and Personal Responsibility Education Innovative Strategies (PREIS) grantees. The Family and Youth Services Bureau (FYSB) and Office of Planning, Research and Evaluation (OPRE) in the Administration for Children and Families (ACF), at the U.S. Department of Health and Human Services (HHS), seek approval for the continued collection of performance measures data to (1) track how grantees are allocating their PREP funds; (2) assess whether PREP objectives are being met (for example, in terms of the populations served); and (3) help drive PREP programs toward continuous improvement of service delivery. In addition, ACF will use this information to continue fulfilling its reporting requirements to Congress and OMB concerning the PREP initiative. ACF will also continue sharing grantee and provider-level findings with each grantee to inform grantees’ own program improvement efforts.

Generalizability of Results

These data are intended to present internally valid descriptions of PREP; they do not promote statistical generalization to other sites or service populations. The study will continue to include information on all PREP grantees, subrecipient program providers, and participants who respond to data collection.

Appropriateness of Study Design and Methods for Planned Uses

The PREP PM data collection is designed to describe the implementation and outcomes of the PREP program. The PM data collected through this study will continue to provide needed information to ACF and grantees for effective program management. Entry and exit surveys of youth participating in PREP are needed to collect information on the demographic and behavioral characteristics of program participants, their experiences in the program, and their perceptions of program effects. Administrative data from grantees and their subrecipient program providers are needed to understand the structure and features of PREP programs, participant numbers, implementation supports, and staff perceptions of quality challenges and needs for technical assistance. Given that the collected information provides the basis of performance measures, the study requires data on the universe of grantees, programs, and participants. As noted in Supporting Statement A, the information is not intended for use as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

The target population for the PREP PM data collection includes all PREP grantees, their subrecipient program providers, and youth participants. The numbers may vary somewhat by year, but they currently include 51 SPREP, 8 TPREP, 27 CPREP, and 12 PREIS grantees. The estimated number of subrecipients is 360 across all grantees. Grantees are expected to serve approximately 369,474 participants over the three-year OMB clearance period, for an average of about 123,158 new participants per year. Program participants are youth in middle school through age 20. The data collection instruments to be used by each target population follow:

• Instrument 1 (a through c): Participant entry survey–youth participants

• Instrument 2 (a through c): Participant exit survey–youth participants

• Instrument 3 (a and b): Performance Reporting System Data Entry Form–grantees

• Instrument 4 (a and b): Subrecipient Data Collection and Reporting Form–subrecipient providers

Sampling and Site Selection

The PREP PM data collection will continue to include all PREP grantees, subrecipient program providers, and participants. ACF will use the performance measures data to monitor and report on progress in implementing PREP programs and to support continuous quality improvement of the programs. In addition, grantees will use the information to inform improvements to their program(s). The study must include all PREP grantees so that (1) the measures reflect the program’s complete scope and (2) the data can be used to promote program improvement among all grantees.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

In April 2017, OMB approved the collection of PREP performance measures data through participant entry and exit surveys and the grantee and subrecipient reporting forms (OMB #0970-0497). Since then, OMB has approved several nonsubstantive change requests, such as the following:

• In August 2019, OMB approved a nonsubstantive change request to revise the participant entry and exit surveys (Instruments 1 and 2) based on cognitive pre-test results. The implemented changes include removing sensitive items, modifying existing items related to adulthood preparation subjects, and adding items to capture information relevant to the success sequence for poverty prevention.

• In April 2020, OMB approved a nonsubstantive change request for further revision of the participant entry and exit surveys (Instruments 1 and 2) and the Performance Reporting System Data Entry Form (Instrument 3) informed by grantee feedback on the 2019 surveys. Changes included producing two different versions of the entry and exit surveys for middle school-age groups (Instruments 1a and 2a) and high school– and older age groups (Instrument 1b). Middle school versions removed or revised items focused on sexual activity and pregnancy for use with SPREP, TPREP, and CPREP participants. Edits to the Performance Reporting System Data Entry Form (Instrument 3) reflected changes to the entry and exit surveys.

• In August 2020, OMB approved a nonsubstantive change request to add measures related to effects of the COVID-19 pandemic on PREP programming and data collection.

• In January 2021, OMB approved a nonsubstantive change request to revise the pandemic-related measures.

• In November 2021, OMB approved a nonsubstantive change request to (1) reduce the number of pandemic-related measures and (2) streamline the participant entry and exit surveys (Instruments 1 and 2) to ensure that all items directly align with legislation, reduce burden, and improve clarity.

The current submission reflects all the changes previously reviewed and approved by OMB.

This Information Collection Request seeks to (1) extend the previously approved data collection and (2) revise the previously approved performance reporting system data forms (Instruments 3 and 4) to no longer require grantees to aggregate participant survey data to the program level for submission as of survey data collected in 2023–2024. For data collected in 2022-2023, grantees and subrecipients will continue to report aggregate data using the current data reporting system (Instruments 3a and 4a). For data collected in 2023-2024 and beyond, grantees and subrecipients will use the revised data reporting system (Instruments 3b and 4b) and report respondent level data from Instruments 1 and 2. These instruments were developed based on (1) the previous versions and (2) our experience with individual-level submission of similar participant survey data as a component of the performance measures for another ACF program.¹

Each of the four data collection instruments addresses each of the study’s objectives described in Section B1 above. Instruments 1 and 2 capture information on the characteristics of the youth participating in the program and their perceptions of program effects, and Instruments 3 and 4 capture information on grantees’ and subrecipients’ implementation of PREP programs. Both types of data will continue to be used to monitor program implementation and outcomes, guide program improvement, and provide status and progress updates.

B4. Collection of Data and Quality Control

Instruments 1–2: Participant entry and exit surveys. As in earlier rounds of PREP performance measures data collection, each grantee and its subrecipients will make decisions regarding procedures for collecting the participant entry and exit surveys. Some grantees have elected to work with local evaluators that administer the surveys for performance measures purposes; the local evaluators could decide to use paper-and-pencil or web-based surveys. For grantees not working with local evaluators, most subrecipients’ program facilitators administer the entry and exit surveys with paper and pencil in group or individual settings, with a slightly smaller percentage administering the surveys by web.² Grantees inform their individual program participants that participation is voluntary and that they may refuse to answer any or all of the questions in the entry and exit surveys.

Instruments 3–4: Performance Reporting System Data Entry Form and Subrecipient Data Collection and Reporting Form. Grantees will report individually on levels of participant attendance, reach, and dosage (Instrument 3 and 4). Providers, which could be grantees or subrecipients, will continue to collect data on these measures (Instruments 3 and 4). Grantees and subrecipients, through their administrators, will continue to collect administrative data on program features and structure, allocation of funds, and staff perceptions of quality challenges. Grantees will prepare and submit their final data sets to ACF through the web-based data submission portal. The Performance Reporting System Data Entry Form (Instrument 3) contains the list of all non-survey data elements that grantees will report, including those collected from among their subrecipients. Data elements included in Instruments 3a and 4a will be used to report data collected through 2022-2023. Instruments 3b and 4b contain the list of data elements grantees and subrecipients will report in subsequent years, starting with 2023-2024.

The timing of participant survey data collections will be customized for each site depending on the start and end dates of each cohort of participants. Performance measures data will continue to be submitted to the portal twice a year.³

Grantees have been successfully collecting performance measures data on an ongoing basis since 2017 under this ICR and since 2013-2014 under a prior ICR (OMB #0970-0398), and ACF has used the data in its reporting and public-facing products. ACF’s contractor will continue to provide training and technical assistance to ensure that grantees and program providers understand the measures, instruments, and data collection processes.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

In Table B5.1, we describe the respondents and expected response rates associated with the performance measures data collection.

Instruments 1–2: Participant entry and exit surveys. As in earlier rounds of PREP performance measures data collection, the administration of entry surveys to all participants at enrollment and the administration of exit surveys during final program sessions will maximize the response rates for participant surveys. Where feasible, exit surveys will be administered on an individual basis to program exiters who are absent during final sessions when the surveys are completed. The response rate for both surveys is expected to be 95 percent. These estimates reflect the experiences in administering performance measures since 2013–2014, with the acknowledgment that disruptions related to the COVID-19 pandemic depressed response rates in recent years.

Instruments 3–4: Performance Reporting System Data Entry Form and Subrecipient Data Collection Reporting Form. To reduce grantee burden and maximize grantee response rates, ACF provides common data element definitions across PREP program models, and grantees submit data in a uniform manner through the web-based data submission portal (see Instruments 3–4). Given that the collection and submission of data for performance measures is a funding requirement of the PREP grants, the grantee and subrecipient response rates are expected to be 100 percent.

Table B5.1. Annual respondent universe and expected response rates for the study of performance measures

Data collection	Type of respondent	Annual number of respondents	Expected response rate	Annual expected respondents
Instrument 1: Participant entry survey	Youth participant	123,158	95%	117,000
Instrument 2: Participant exit survey	Youth participant	112,352	95%	106,734
Instrument 3: Performance Reporting System Data Entry Form	Grantee Administrator	98	100%	98
Instrument 4: Subrecipient Data Collection and Reporting Form	Subrecipient Administrator	360	100%	360
Estimated totals		235,968		224,192

Nonresponse

As in earlier rounds of the collection of PREP performance measures data, analyses will be based on respondents’ provision of data for a given measure, with no imputation or weight adjustments to address missing data.

Given that participants will not be randomly sampled and that findings are not intended to be representative, nonresponse bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

ACF and PREP grantees will continue to primarily use the performance measures data internally, although they will use the data as the basis for ACF’s annual report to Congress on the progress of the PREP program, with end-of-cohort reports made available to the public.

The performance measures data will continue to be collected from all PREP grantees, program providers, and youth participants. The analyses will include computation of statistics such as percentages and means based on respondents; we will not produce estimates intended to apply to any broader population.

B7. Data Handling and Analysis

Data Handling

Grantees and their subrecipients will continue to collect performance measures data. In some cases, grantees will have engaged local evaluators to assist with performance measures data collection. Grantees will continue to submit these data into the web-based data submission portal, which will continue to be maintained by ACF’s contractor, Mathematica. Mathematica will use the data to analyze PREP performance data and to generate performance measurement reports for ACF.

The entry screens of the portal will continue to include a series of automated validity checks to identify some types of errors as data are entered. Error messages alert grantees to inconsistencies between data elements, values beyond the expected range, and similar issues and provide an opportunity for the grantee to correct such errors before data submission. The system also conducts automated checks to ensure entry of the full set of performance measures.

Mathematica conducts additional quality checks to identify remaining issues. Cases with unresolved data issues may be omitted from analyses that rely on problematic data elements. If suspect data are included in any tabulations, caveats will be included alongside the reported data.

Data Analysis

As in earlier rounds of PREP performance measures data, the contractor will analyze PREP performance data to generate performance measurement reports for ACF and other audiences. Core analyses will include computation of means and sums of continuous numeric measures (such as number of participants served) and calculation of frequency distributions of categorical and character variables (such as program models implemented). The shift from grantees submitting aggregate participant survey data to individual-level survey responses will allow ACF to conduct more detailed analyses. Cross-tabulations will explore potential relationships between variables. For example, when participant survey data are submitted at the individual level rather than aggregated, we can examine whether participants’ perceptions of program effects differ by demographic characteristics. We will examine changes in the performance measures data over time. Analyses will continue to be conducted separately for each of the four PREP funding streams (SPREP, TPREP, CPREP, and PREIS).

Data Use

In addition to the availability of the data to ACF and grantees in the Dashboard, the performance measures data will continue to be used to develop fact sheets, end-of-cohort reports, and briefs. End-of-cohort reports synthesize performance measures data across years and include summaries of the data collection and analysis methods as well as appropriate caveats and data limitations. ACF also uses the data for annual reports to Congress and OMB.

B8. Contact Persons

• Caryn Blitz, OPRE, ACF, Caryn.Blitz@acf.hhs.gov

• Tia Brown, OPRE, ACF, Tia.Brown@acf.hhs.gov

• Lara Hulsey, Mathematica (contractor), lhulsey@mathematica-mpr.com

• Lauren Murphy, Mathematica (contractor), lmurphy@mathematica-mpr.com

• Kim McDonald, Mathematica (contractor), kmcdonald@mathematica-mpr.com

Attachments

Instrument 1: Entry survey for high school and older youth

1a: Entry survey for middle school youth

1b: Entry survey for high school and older youth (PREIS/TPREP)

1c: Entry survey for middle school youth (PREIS/TPREP)

Instrument 2: Exit survey for high school and older youth

2a: Exit survey for middle school youth

Instrument 3a: Performance Reporting System Data Entry Form (2022-2023)

Instrument 3b: Performance Reporting System Data Entry Form (2023+)

Instrument 4a: Subrecipient Data Collection and Reporting Form (2022-2023)

Instrument 4b: Subrecipient Data Collection and Reporting Form (2023+)

1 Sexual Risk Avoidance Education Program Performance Analysis Study (SRAE PAS) (OMB #0970-0536)

2 In 2021-2022, approximately 65% of programs administered the surveys by paper and 60% administered the surveys by web. Note that administration modes are not mutually exclusive.

3 Measures of structure, cost, and support for program implementation will be submitted once per year, and all other measures will be submitted twice per year.

6