Attachment A Overview

ATTACHMENT A
OVERVIEW OF THE PREGNANCY ASSISTANCE FUND (PAF) EVALUATION: FEASIBILITY AND DESIGN STUDY (FADS)

OVERVIEW OF THE PREGNANCY ASSISTANCE FUND (PAF) EVALUATION: FEASIBILITY AND DESIGN STUDY (FADS)

In March 2010, Congress authorized the Pregnancy Assistance Fund (PAF) Competitive Grants Program as part of the Patient Protection and Affordable Care Act (ACA). The PAF grant program is a key element of the federal strategy to support teens and young adults who are having or raising a child and their families. Administered by the Office of Adolescent Health (OAH), the PAF grant program funded a second cohort of 17 grantees—states, tribes, and tribal entities—in summer 2013 to develop and implement programs focused on an array of outcomes, including increasing access to and completion of secondary and postsecondary education, improving child and maternal health, reducing the likelihood of repeat teen pregnancies, increasing parenting and co-parenting skills, decreasing intimate partner violence, and raising awareness of available resources. To promote positive outcomes, grantees may implement a wide variety of services for expectant and parenting teens, women, fathers, and their families.

The PAF evaluation will have two core components: a rigorous assessment of program impacts and implementation of two or three grantees, and a descriptive examination of program design of all 17 grantees. The PAF evaluation will help the federal government, grantees, and local service providers learn more about program design, implementation, and impacts.

Preliminary PAF evaluation efforts, including instrument development, will be conducted through the PAF Feasibility and Design Study (FADS). The purpose of the FADS is to design rigorous impact evaluations in two or three sites, develop data collection materials for all aspects of a future evaluation, and conduct telephone interviews with grantees about the program design decisions and early implementation experiences. Information collected through the FADS will also be used to provide funding agencies with information to inform the structure and components of programs for expectant and parenting teens and their families, so that future rigorous program evaluation will also be possible.

The FADS comprises a design and implementation analysis and an impact study as its two primary data collection components. The design and implementation analysis will describe grantees’ program design and factors that influenced their decision making. The impact study will use an experimental design to test the effectiveness of PAF-funded services on outcomes related to education, sexual behaviors, parenting, and health.

The objective of the Pregnancy Assistance Fund (PAF) Feasibility and Design Study (FADS) is to establish a foundation for a rigorous impact and implementation evaluation. Specifically, FADS will: (1) assess design options for implementation and impact evaluation, (2) document how PAF programs are operationalized in the field, (3) identify and enter into agreements with two or three sites for the evaluation, (4) provide assistance to sites on their sub-award process to support a rigorous evaluation framework, (5) develop all evaluation instruments and obtain clearance, and (6) pilot baseline data collection. As described below, the evaluation will have two main components: the design and implementation analysis and the impact study.

Design and Implementation Analysis. During the FADS, all PAF grantees will participate in telephone interviews about program design and early implementation experiences. These interviews will be conducted toward the end of the first grant year.

The specific questions asked during each interview will vary, depending on 1) what is already known about the respondent’s program design decisions (e.g. we will not ask questions for which we already have answers, based on Mathematica’s prior review of program documents and administrative data) and 2) the discretion of the interviewer, who will adapt his or her questions based on the respondent’s answers, while still touching on key themes across interviews.

A specific protocol will be developed for each interview in advance of the call. This overall approach has been used for the PREP evaluation, and has demonstrated benefit of reducing burden for respondents, because each interview will be tailored for the specific respondent who is being interviewed.  

The two or three grantees selected for the impact evaluation will also participate in, during a future evaluation, a more in-depth implementation study. The in-depth implementation study will take a detailed look at program operations along four key aspects: (1) inputs required for implementation to succeed and be sustained, (2) contextual factors that influence implementation, (3) quality of program implementation, and (4) participants’ responsiveness to service.

Impact Study. Using an experimental design, the PAF evaluation will test the effectiveness of services to impact educational, health, sexual behavior, and parenting outcomes. During the FADS, the study team will identify and work with two or three grantees to decide which services components will be evaluated, which participants will be included, and which outcomes will be measured. In addition, the FADS team will work with grantees to develop a plan for random assignment. Finally, the FADS team will work with the selected sites to design a process for collecting study data, including evaluation consent, a baseline survey, and two follow-up surveys.

In each of the two or three sites selected for the impact study, all eligible youth will be considered for enrollment in the study. The evaluation team will then work collaboratively with each site to identify youth for the study and obtain consent.

Active written consent will be obtained from youth older than 18 and parental permission will be obtained for those younger than 18. For youth older than 18, the informed consent process will be integrated with baseline data collection.

The baseline survey will be administered to all consented youth shortly after study enrollment. Youth assent will be collected at the beginning of the baseline administration. The mode of data collection will depend on the program setting. We will offer various modes for completing the baseline survey. These modes are likely to be computer-assisted telephone interviewing (CATI), a web-survey, or a self-administered paper and pencil instrument (PAPI). We will work with grantees to assess the best baseline survey mode given the context, with sensitivity to respondent literacy levels and access to technology.

For CATI completes, data collectors will be assigned a project cell phone that will be handed off to respondents during the intake process. Respondents will use the phone to call Mathematica’s Survey Operations Center (SOC) and complete the survey over the phone with a trained interviewer. Data collectors will also be given a toll-free number for the Survey Operations Center that they can give to respondents to call in and complete the survey over the phone at their convenience. Additionally, SOC staff can make calls to the respondent to complete the survey over the phone. When completing the survey through CATI, the interviewer (and data collector, when applicable) will ensure the respondent is in a secure, private place to respond to the survey questions.

We will also offer the option of completing the survey over the web. If a web-based survey is used, respondents will be provided a unique PIN/password. We will also provide them with a toll-free number to call should they have any issues with the web survey.

A paper and pencil (PAPI) version of the baseline survey will also be available for anyone wishing to complete the survey using the self-administered instrument. Data collectors will keep PAPI versions on-hand and distribute them to respondents, as needed. The completed surveys will be returned to staff in sealed envelopes

OAH is currently requesting OMB approval for the collection of (a) program design and early implementation data collected through telephone interviews with 17 grantees and (b) baseline data in up to three impact sites through self-administered questionnaires. Approval for additional data collection activities through follow-up surveys and the in-depth implementation study will be requested in a later submission.