Ss_b

Part B: Statistical Methods for the Collection of 24-Month Follow-up Survey Data- Pregnancy Assistance Fund Study

OMB Control Number 0990-0424

July 2016

Submitted to: U.S. Department of Health and Human Services Office of Adolescent Health Office of the Director Department of Health and Human Services 1101 Wootton Parkway, Suite 700 Rockville, MD 20852 Project Officer: Amy Farb

Part B: Statistical Methods for the Collection of 24-Month Follow-up Survey Data- Pregnancy Assistance Fund Study OMB Control Number 0990-0424 July 2016

Contents

PART B INTRODUCTION 1

B1. Respondent Universe and Sampling Methods 1

B2. Procedures for Collection of Information 5

B3. Methods to Maximize Response Rates and Deal with Non-Response 6

B4. Test of Procedures or Methods to be Undertaken 7

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 7

TABLES

Table B1.1. Minimum Detectible Impacts for California 4

ATTACHMENTS

ATTACHMENT A: OVERVIEW OF THE PAF EVALUATION

ATTACHMENT b: QUESTION BY QUESTION SOURCE LIST FOR THE 24-Month FOLLOW-UP SURVEY

ATTACHMENT C: SOURCES REFERENCED FOR THE 24-Month FOLLOW-UP SURVEY

ATTACHMENT D: PERSONS CONSULTED ON INSTRUMENT DEVELOPMENT AND/OR ANALYSIS OF THE PAF 24-month follow-up SURVEY

ATTACHMENT E: CONFIDENTIALITY PLEDGE

ATTACHMENT F: ANALYSIS PLAN

ATTACHMENT G: PAF 24-MONTH FOLLOW-UP 60-DAY NOTICE

INSTRUMENTS

Instrument 1: PAF 24-MONTH FOLLOW-UP sURVEY: cALIFORNIA

Instrument 2: PAF 24-MONTH FOLLOW-UP sURVEY: TEXAS

PART B INTRODUCTION

Previous Information Clearance Requests Approved by OMB. OMB has previously approved two ICRs related to this evaluation (ICR #201406-0990-001):

• August 30, 2014 – OMB approved the instruments associated with two data collection efforts: (1) telephone interviews with all 17 current Pregnancy Assistance Fund grantees; and (2) collection of baseline data for the impact study in two sites through a baseline survey (OMB Control # 0990-0424).

• January 7, 2016 – OMB approved a revision of the above approved package to include the 12-month follow-up data collection effort for the impact study in two sites (OMB Control #0990-0424) extending the expiration date to January 31, 2019.

Current Information Clearance Request. In this submission, OAH is requesting a revision to the existing approval to add the 24-month follow-up survey instruments to be used in the two impact sites: (1) Pregnancy Assistance Fund 24-Month Follow Up Survey – California (Instrument 1), and (2) Pregnancy Assistance Fund 24-Month Follow Up Survey – Texas (Instrument 2). These surveys are very similar to the baseline and 12-month follow-up surveys approved for this evaluation, and the two are nearly identical, except for some minor differences to reflect differences in the interventions. The California survey contains additional items to measure changes in youth resiliency, a primary focus of the program in California. The Texas survey does not contain such resiliency items, but does contain items measuring parenting and relationship skills, a focus of the program in Texas.

B1. Respondent Universe and Sampling Methods

There are three sites participating in the PAF Study. Two of these sites (California and Texas) will use an experimental design and primary data collection through surveys of youth, including the 24-month follow-up survey which is the focus of this ICR. OAH has selected two program sites to participate in an experimental impact study, one of which is a current Pregnancy Assistance Fund grantee (California). OAH has selected a third program site to participate in a quasi-experimental design evaluation. This third site, in Washington, DC, is a pilot test for using a quasi-experimental design and existing administrative data provided through data use agreements with three local public agencies. Youth will not be surveyed in Washington, DC. The two study sites that require ongoing data collection (California and Texas) are described below, and also in Appendix A, Overview of the PAF Evaluation. Appendix A also contains a description of the Washington, DC pilot site.

1. California Department of Public Health, Division of Maternal Child and Adolescent Health (MCAH).

California MCAH is currently an OAH Pregnancy Assistance Fund grantee. They are using their grant to introduce Adolescent Family Life Program Positive Youth Development (AFLP PYD) across their program providers throughout the state. These program providers are currently implementing an older version of the program – AFLP. AFLP PYD differs from the original AFLP in three ways: 1.) Development of structured materials for the case managers to use during interactions with youth, including home visits; 2.) Case managers carry fewer cases and therefore, youth receive double the amount of AFLP dosage via home visits; 3.) Case managers utilize the positive youth development framework, which promotes youth resiliency and self-sufficiency via motivational interviewing and techniques. This study will primarily address the impact of AFLP-PYD, as compared to AFLP, on outcomes such as subsequent pregnancy, improving contraception use, and supporting school completion.

The evaluation involves 13 program providers across the state. Within two of the larger providers, approximately 500 expectant or parenting females will be randomly assigned as individuals to either AFLP (the business as usual condition) or AFLP-PYD (the enhanced treatment condition). Across the remaining 11 providers, we have assigned clusters to either AFLP or AFLP-PYD. A cluster is the entire provider among the smallest providers (10 providers, 10 clusters), or specific geographic locations served by larger providers (2 providers¹, 3 clusters). We randomized a total of 13 clusters, and expect to enroll approximately 800 expectant and parenting females across them. Sample enrollment will occur over a 24-month period.

As of the end of June 2016, 1,064 participants have completed the baseline survey in California and the baseline survey response rate was 98 percent of all youth who have provided study consent in California. All California baseline surveys should be complete by the end of 2016. The 12-month follow up survey began in February 2016 and responses will be attempted for a total of six months. The first cohort has not yet been released for five months and has an 82 percent response rate.

The evaluation sample is expected to be primarily Hispanic (~80 percent) and low-income (~75 percent of the sample eligible for Medicaid). At enrollment, approximately 55 percent of the sample is expected to be pregnant (and not yet parenting) and 43 percent parenting (and not pregnant). A small percentage (~2 percent) may be pregnant and parenting.

2. The Houston Department of Health and Human Services (HDHHS)

The HDHHS will implement Steps to Success, which was developed by Healthy Families San Angelo (HFSA) as an enhancement to Healthy Families America home visiting services. The primary component of Steps to Success is a structured home visiting model that covers parenting, contraception, employment, relationships, and finances.

HFSA developed Steps to Success based on research on key risk factors for repeat pregnancies among adolescent mothers. This research pointed to the importance of encouraging these young mothers to use long-acting contraceptives (a key element of the Steps to Success approach) as essential to delaying repeat pregnancy. It also suggested that promoting more positive relations with the baby’s father and encouraging these young mothers to stay in school were both promising avenues for reducing the risk of rapid repeat pregnancy. Steps to Success aims to promote both these goals.

Clients are accepted into the program either during the pre-natal period or early post-partum period. Home visits occur weekly initially and transition to monthly visits as appropriate based on the needs of the family. These visits are provided for two years after the baby is born. Home visitors have a maximum case load of twenty-five clients at a time. The study will test the impact of Steps to Success, as compared to a “business as usual” control condition, on outcomes such as delaying a subsequent pregnancy, improving contraception use, supporting school completion, and improving parenting skills.

The evaluation involves randomly assigning eligible and interested young women to Steps to Success or a control group that will have access to any existing community resources (such as healthcare and other related services at WIC clinics) but not to Steps to Success.

We expect to enroll and randomize 720 young mothers over a 24-30 month period. The sample is expected to be primarily Hispanic (~75 percent) and low-income (100 percent having qualified for Medicaid). At enrollment, approximately 50 percent of the sample is expected to be pregnant (and not yet parenting) and 50 percent parenting (and not pregnant).

As of the end of June 2016, 232 participants have completed the baseline survey in Texas and the baseline survey response rate was 100 percent of all youth who have provided study consent in Texas. All Texas baseline surveys should be complete by the end of 2017. The 12-month follow-up survey will begin in July 2016.

Statistical Power. The statistical power for each site is described separately below. In both California and Texas, youth will be surveyed three times – at the time of study enrollment (baseline survey, previously approved under OMB Control # 0990-0424), 12-months later (as approved by the January 2016 revision to 0990-0424), and 24-months later (the focus of this revision)² The primary mode of survey completion for the 24-month follow-up survey, the focus of this ICR, will be a web survey. Nonrespondents to the web survey will be given an opportunity to complete the survey using CATI.

The sites will be analyzed separately, therefore statistical power analyses are reported separately.

California: An overall impact will be calculated as a weighted average of the impacts from the two designs (individual random assignment and clustered random assignment). We will use inverse variance weights in our benchmark analysis and sample size weights as a sensitivity analysis. At the time of the 24-month follow-up, we expect to retain 75 percent of the sample or 975 youth. For a prevalence rate of 25 percent (such as a subsequent pregnancy during the follow-up period), we can detect a 8.2 percentage point difference between the two groups; and, for a prevalence rate of 50 percent (such as receiving a diploma during the follow-up period), we can detect a 10.3 percentage point difference between the two groups. If the response rate is instead 70 percent, we can detect an 8.4 percentage point difference on an outcome with a prevalence rate of 25 percent, and a 10.5 percentage point difference on an outcome with a prevalence rate of 50 percent. Examining impacts by particular sub-groups (such as whether expecting or parenting at program enrollment, or whether primary language is English or Spanish) will be considered exploratory, as the study is not considered sufficiently powered to detect impacts on those samples. Given the risk profile of the population, the findings from this study will have policy relevance for the field without sub-group analysis.

Table B1.1 reports minimum detectible impacts on two illustrative outcomes—one with 50 percent prevalence and one with 25 percent prevalence. Separate estimates are presented for the two components of the evaluation (individual randomization and cluster randomization) as well as for the overall study (in which the overall impact is calculated as a weighted average of the impacts from the two study components).

Table B1.1. Minimum Detectible Impacts for California

	Percentage Point Impacts for Illustrative Binary Outcomes
Study Component	50 percent prevalence rate	25 percent prevalence rate
75 Percent Response Rate
Individual Randomization (2 sites; 375 youth)	12.8	10.3
Cluster Randomization (13 sites; 600 youth)	18.9	14.9
Full Study	10.3	8.2
70 Percent Response Rate
Individual Randomization (2 sites; 350 youth)	13.2	10.6
Cluster Randomization (13 sites; 560 youth)	19.1	15.0
Full Study	10.5	8.4

Notes:   Sample sizes account for survey nonresponse. Figures assume that the sample is evenly divided between the program and control groups and that covariates explain 20 percent of the variance at the individual level and 40 percent at the cluster level. We assume an ICC of 0.06. The figures also assume a two-tailed t-test with 80 percent power and a 95 percent confidence interval. 

Texas. At the time of the 24-month follow-up, we expect to retain 75 percent of the sample, or 540 youth. For a prevalence rate of 25 percent (such as a subsequent pregnancy during the follow-up period), we can detect a 9 percentage point difference between the two groups; and, for a prevalence rate of 50 percent (such as receiving a diploma during the follow-up period), we can detect a 11 percentage point difference between the two groups. If the response rate is 70 percent (504 youth), we can detect very similar percentage point differences on outcomes with prevalence rates of 25 and 50 percent, respectively. Given the small sample size, we do not anticipate conducting any subgroup analyses.

Limitations. The sites are not meant to be representative of the general Pregnancy Assistance Fund program for expectant and parenting youth. Site selection has focused on programs that (1) are large enough to support an impact study, (2) are implementing programs in a way that is amenable to random assignment or a quasi-experimental design, and (3) address priority gaps in the existing research literature on evidence-based approaches to assist pregnant and parenting youth. OAH acknowledges that the limitations of the PAF Study. OAH does not intend to use the results of separate program evaluations in California and Texas to generalize to the effectiveness of similar programs nationally. Each of the selected programs has a different approach for serving pregnancy and parenting teens, and each approach is similar to approaches used across the country. However, since each site was purposefully selected for its ability to support the design of a rigorous impact evaluation, the results cannot be generalized to the broader population of similar programs. Still, the results will add value to a non-existent knowledge base. These three separate evaluations will provide some foundational knowledge on “what works” for pregnant and parenting teens, evidence that can, in the future, be expanded with replication studies in other contexts and settings.

B2. Procedures for Collection of Information

In each of the two sites selected for the experimental impact study (California and Texas), all eligible youth will be considered for enrollment in the study (discussed in Section B.1). Each site will be responsible for providing the evaluation team with a list of eligible youth on an ongoing basis, as programs enroll individuals over time and not entire cohorts of youth at any one time. The evaluation team will then work collaboratively with each site to identify youth for the study and obtain consent.

Mathematica thoroughly and efficiently trained staff to ensure they can properly inform study participants. In California, study intake is performed by program staff trained in person on data collection procedures by Mathematica. In Texas, study intake is performed by professional data collectors working for a subcontractor to Mathematica (Decision Information Resources) and trained by Mathematica. We created a study description to ensure that accurate and consistent information is available, and trained staff on explaining the study, reviewing the study description, answering questions about the study, and administering consent and the baseline survey. This process and consent forms have been approved by OMB on August 30, 2014 (OMB Control # 0990-0424).

The 12-month follow-up survey will be administered to consented sample members approximately 12 months after study enrollment and completion of the baseline survey. The data collection plan for the 12-month follow-up survey is the same across the two sites (California and Texas) and also reflects sensitivity to issues of efficiency, accuracy, and respondent burden. This process was approved by OMB on January 7, 2016 as a revision to the original OMB request (OMB Control #0990-0424).

The 24-month follow-up survey will mirror the 12-month survey processes and will be administered to consented sample members approximately 24 months after study enrollment and completion of the baseline. The 24-month survey is not dependent on completion of the 12-month follow-up survey. As discussed in Part A of this ICR, we will offer two modes for completing the 24-month follow-up survey. These modes will be a web-survey that will be smart phone compatible and computer-assisted telephone interviewing (CATI). We will use email and text messages with links to the web survey and toll-free telephone number should respondents prefer to complete the survey by telephone or have any issues with the web survey.

For those opting to complete the survey over the web, respondents will be provided a unique short URL with imbedded user ID and password to access the survey from either type of device. Respondents will first be asked to verify their name and date of birth before being allowed to proceed. We will advise respondents to complete the survey in a private location. We will also provide them with a toll-free number to call should they prefer to complete the survey by telephone or have any issues with the web survey. The web survey will also include a link to email the project team with questions or issues.

For those who do not call in or complete the web survey, we will make outbound calls from Mathematica’s Survey Operations Center (SOC). When a respondent is reached, a SOC telephone interviewer will use computer assisted telephone interviewing (CATI) to complete the survey. If a respondent is not reached, the SOC telephone interviewer will leave a message whenever possible and provide a toll-free number the respondents can use to call and complete the CATI survey. When completing the survey through CATI, the interviewer will direct the respondent to be in a secure, private place to respond to the survey questions.

Instruments 1 and 2 contain the 24-month survey for each site separately – California and Texas. These surveys are very similar to the baseline and 12-month follow-up surveys approved for this evaluation, and the two are nearly identical to each other, except for some minor differences to reflect differences in the interventions. The California survey contains additional items to measure changes in youth resiliency, a primary focus of the program in California. The Texas survey does not contain such resiliency items, but does contain items measuring parenting and relationship skills, a focus of the program in Texas. A question by question list of sources for the 24-month follow up survey is found in Attachment B, and a description of the sources referenced is found in Attachment C.

B3. Methods to Maximize Response Rates and Deal with Non-Response

OAH expects to achieve a response rate of 75 percent for the 24-month follow-up survey. This rate has been achieved for two sites with similar populations, program, and data collection modes – one site that is participating in the Personal Responsibility Education Program (PREP) Multi-Component Evaluation and the other that is participating in the Evaluation of Adolescent Pregnancy Prevention Approaches, both of which are conducted by Mathematica. We can expect to achieve these completion rates for the PAF study at the 24-month follow-up period for several reasons.

First, we have planned two mailings six months and nine months after the 12-month follow-up survey. Both mailings will remind the respondent about the upcoming 24-month survey and encourage them to update their contact information, as necessary.

Second, at the time of their release for the survey, we will send them an advance letter that provides them with the information necessary to complete the survey over the web. Additional telephone, email and text prompts to youth and parents will be conducted as needed.

Finally, $25 gift cards will be provided to respondents to encourage participation in the survey. This is consistent with other evaluations, such as the Personal Responsibility Education Program Multi-Component Evaluation (PREP), in which respondents are using phone to complete a survey, and is consistent with the amount approved for PAF on the consent forms approved by OMB on August 30, 2014 (OMB Control # 0990-0424).

As discussed above, the evaluation team anticipates high response rates (75 percent) to the 24-month follow-up survey. Even so, the team will take steps to understand the nature of any non-response and to account for the threat that it may pose for the validity of the study’s impact estimates. Using data from the baseline survey, evaluation team members will first test for statistically significant differences across demographic and baseline outcome variables between respondents and nonrespondents. Any such differences will be controlled for in the analyses by using non-response weights. The team will also test for differences between the research groups in their baseline characteristics and control for these differences using covariates when estimating program impacts (see Attachment F).

B4. Test of Procedures or Methods to be Undertaken

OAH and other offices within HHS (OPRE, ASPE) have made it a priority to align measures in the baseline and follow-up surveys across evaluations of similar programs and populations. As discussed in Part A of this information collection request, many of the items included on the 24-month PAF follow up survey are taken from the approved baseline and 12-month follow-up surveys and from similar surveys OMB has already approved for use in the ongoing Evaluation of Adolescent Pregnancy Prevention Approaches (PPA), the Teen Pregnancy Prevention Replication Study, and the Personal Responsibility Education Program (PREP) Multi-Component Evaluation³. To date, 11,654 PPA follow-up surveys have been administered, 2,526 to expectant and parenting young women; the Replication Study first follow-up surveys has been administered to 6,511 adolescents and the second follow-up survey to 6,491 adolescents; and the PREP follow-up survey has been administered to 4,271 youth, including 413 expectant and parenting young women.

The plan for the impact analysis is described in Attachment F.

B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Data collection for the 24-month follow-up survey will be led by OAH’s prime contracting organization, Mathematica Policy Research. Mathematica’s subcontractor, Decision Information Resources, will conduct field locating efforts in both California and Texas for study participants who do not respond to invitations to complete the 24-month survey on-line or on the telephone. All analysis will be conducted by Mathematica Policy Research. Attachment D lists the individuals whom OAH consulted on instrument development, data collection, and analysis.

1 One of the large provider has a location doing individual random assignment and a small location that was randomized as a cluster.

2 The current ICR only pertains to the 24-month follow-up survey.

3 ACF received initial OMB approval for the PPA baseline survey on July 26, 2010 (OMB Control Number 0970-0360). In summer 2011, oversight of PPA was transferred to the Office of Adolescent Health (OAH) within the Office of the Assistant Secretary, and the project is now tracked with a different OMB Control Number (0990-0382). The OMB Control Number for the Teen Pregnancy Prevention Replication Study is 0990-0394. OMB approval for the PREP follow-up survey was received on May 8, 2013 (OMB Control Number 0970-0398).