Supporting Statement A 0990-0450

Part A: Justification for the Collection of Data—Sustainability Study of Programs Funded by OAH in 2010

June 2016

Submitted to:

U.S. Department of Health and Human Services

Office of Adolescent Health
Department of Health and Human Services

1101 Wootton Parkway, Suite 700
Rockville, MD 20852

Project Officer: Amy Farb
Contract Number: GS-10F-0050L/HHSP233201300416G

Submitted by:

Mathematica Policy Research

P.O. Box 2393
Princeton, NJ 08543-2393
Telephone: (609) 799-3535
Facsimile: (609) 799-0005

Project Director: Jean Knab
Reference Number: 40305.C65

CONTENTS

PART A: INTRODUCTION 1

A1. Circumstances making the collection of information necessary 3

1. Legal or administrative requirements that necessitate the collection 3

2. Study objectives 3

A2. Purpose and use of the information collection 5

A3. Use of information technology to reduce burden 5

A4. Efforts to identify duplication and use of similar information 6

A5. Impact on small businesses 6

A6. Consequences of not collecting the information/collecting less frequently 6

A7. Special circumstances 6

A8. Federal register notice and consultation outside the agency 7

A9. Payments to respondents 7

A10. Assurance of confidentiality 7

A11. Justification for sensitive questions 7

A12. Estimates of the burden of data collection 7

1. Annualized burden estimates 7

2. Estimates of annualized costs 8

A13. Estimates of other total annual cost burden to respondents and record keepers 8

A14. Annualized cost to the federal government 8

A15. Explanation for program changes or adjustments 8

A16. Plans for tabulation and publication and project time schedule 8

1. Analysis plan 8

2. Time schedule and publications 9

A17. Reason(s) display of OMB expiration date is inappropriate 10

A18. Exceptions to certification for Paperwork Reduction Act submissions 10

ATTACHMENTS

ATTACHMENT A: PERSONS CONSULTED ON INSTRUMENT DEVELOPMENT

ATTACHMENT B: EMAIL REQUESTING PARTICIPATION IN THE STUDY

INSTRUMENTS

INSTRUMENT #1: IN-DEPTH INTERVIEW MASTER TOPIC GUIDE

TABLES

A.1 Sustainability study objectives 5

A.2 Calculations of burden hours and cost for grantee staff 8

A.3 Proposed sustainability study time schedule 10

ABSTRACT

• Goal of study: Assess which former OAH-funded programs were sustained and identify the key factors that either enabled or hindered grantees’ ability to do so.

• How data will be used: The data will be incorporated into deliverables that highlight successes, challenges, and lessons learned about program sustainability.

• Methods of collection: The study will collect data using in-depth telephone interviews.

• Respondent population: Program administrators at up to 50 grantee organizations.

• Analysis techniques: The study will rely on descriptive qualitative analysis.

PART A: INTRODUCTION

In 2010, the Office of Adolescent Health (OAH) launched a number of initiatives to reduce and delay teen pregnancy and to support expectant and parenting young families.

The Teen Pregnancy Prevention (TPP) grant program was launched in spring 2010 as a key early piece of the federal government’s ongoing “evidence and innovation” agenda. The program provides approximately $100 million in annual competitive contracts and grants to public and private entities to fund medically accurate and age-appropriate programs to reduce teen pregnancy. The program features a “tiered-evidence” grant design that reserves most of the funding for grants to replicate programs with existing evidence of effectiveness (Tier 1). A smaller proportion of funding is reserved to encourage innovation in the field by implementing and rigorously testing promising new programmatic approaches (Tier 2). The first 75 Tier 1 replication grants and 16 Tier 2 innovative or promising grants were awarded in fall 2010 for programming to start in fall 2011.

A second OAH program, the Pregnancy Assistance Fund (PAF), is a competitive grant program providing $25 million annually over 10 years to states and tribal entities to develop and implement activities to support expectant and parenting teens, women, fathers, and their families. PAF grantees could apply for funding in four categories:

• Category 1: Programs supporting expectant and parenting student services at institutions of higher education

• Category 2: Programs supporting expectant and parenting teens, women, fathers, and their families at high schools and community service centers

• Category 3: Programs improving services for pregnant women who are victims of domestic violence, sexual violence, sexual assault, and stalking

• Category 4: Programs increasing public awareness and education services for expectant and parenting teens, women, fathers, and their families

The first PAF grants were awarded in 2010 to 17 grantees, including 14 states and 3 tribal entities, for up to four years.

In addition to the two grant programs described above, OAH and the Centers for Disease Control (CDC) allocated a portion of the Tier 2 funding to support nine state and community-based grantees to implement teen pregnancy prevention initiatives through their Communitywide Teen Pregnancy Prevention Initiatives Program (Communitywide Program). Grantees were expected to saturate their communities through innovative, multicomponent initiatives to reduce teen pregnancy, with a focus on reaching African American and Latino or Hispanic youth.

Consistent with their focus on evidence, OAH has undertaken a range of evaluation activities associated with each of the funding streams. The burden associated with other ongoing data collection activities regarding the grantees has been previously reviewed and approved by OMB under three different information collection requests (ICRs).

May 9, 2012—All TPP grantees collect data on a uniform set of performance measures and report them to OAH on a semiannual basis through an online system. The Office of Management and Budget (OMB) has approved the original instruments used to collect this data as well as an extension to this ICR on August 31, 2015 (OMB Control #0990-0392).

September 2, 2014—OMB approved the instruments associated with two data collection efforts: (1) collection of PAF descriptive implementation data for the Design and Implementation Analysis through telephone interviews, regarding how 17 PAF grantees are using grant funding to implement their programs; and (2) collection of baseline data for the experimental impact studies through a baseline survey (OMB Control #0990-0424).

September 8, 2014—OMB approved the instruments used to collect data on (1) program costs and (2) program impacts from a subset of OAH TPP Program grantees (OMB Control #0990-0425).

With this ICR, OAH seeks approval for the data collection instrument for a complementary study on the sustainability of the first cohort of former PAF, TPP, and CDC grantees. The proposed study will be a new and unique contribution to OAH’s portfolio of evaluation activities. The study aims to address five overarching research questions: (1) Which of the TPP, PAF, or CDC programs funded in 2010 were sustained, for how long, and in what form? (2) What are the key factors that affected their sustainability? (3) What methods did grantees use to sustain programs (for both those that were successful and those that were not)? (4) What types of technical assistance and resources did grantees receive to assist them in sustaining their programs, and from whom? and (5) What are the key lessons learned that can be used to help grantees in sustaining their programs in the future? OAH has contracted with Mathematica Policy Research to conduct this three-year (October 2015–September 2018) study.

Current Information Clearance Request. In this submission, OAH is now requesting OMB approval for one instrument related to the sustainability study: The In-Depth Interview Master Topic Guide. It is the only anticipated ICR for this study. The interview data will be combined with a review of previously collected performance measures data and documents.

A1. Circumstances making the collection of information necessary

1. Legal or administrative requirements that necessitate the collection

On March 23, 2010, the president signed into law the Patient Protection and Affordable Care Act (ACA), H.R. 3590 (Public Law 111-148), which authorized $25 million for each of fiscal years 2010 through 2019 for the Secretary of Health and Human Services (HHS) to establish and administer PAF “for the purpose of awarding competitive grants to States to assist expectant and parenting teens and women.¹”

Previously, on December 16, 2009, the president had signed the Consolidated Appropriations Act, 2010 (Public Law 111-117). Division D, Title II of the Act provides $110,000,000 for making competitive contracts and grants to public and private entities to fund medically accurate and age- appropriate programs that reduce teenage pregnancy and the federal costs associated with administering and evaluating such contracts and grants. The statute states that the funds shall be made available in two tiers: (1) not less than $75,000,000 for funding the replication of programs that have been proven effective through rigorous evaluation to reduce teenage pregnancy, behavioral risk factors underlying teenage pregnancy, or other associated risk factors; and (2) not less than $25,000,000 for funding research and demonstration grants to develop, replicate, refine, and test additional models and innovative strategies for preventing teenage pregnancy. A total of $9,800,000 of the $25,000,000 allocated to the second tier funds the Communitywide Program, a collaboration between OAH and the CDC. Any remaining amounts are to be made available for training and technical assistance, evaluation, outreach, and additional program support activities. The TPP currently operates under the authority contained in the Consolidated Appropriations Act, 2014.

The proposed sustainability study is a key piece of OAH’s broad and ongoing effort to comprehensively evaluate all of its funding efforts as required by the legislation and identify the key elements that will ensure sustainability of these programs after federal funding ends.

2. Study objectives

Program sustainability is an area in need of further exploration; specifically, there is a lack of knowledge about what it takes for programs to be sustained after grant funding ends and what support grantees need while funded to improve the likelihood of sustainability. Former TPP, CDC Community-wide, and PAF grantees funded in 2010 provide an excellent opportunity to assess the extent to which former OAH-funded programs are sustained and identify the key factors that either enabled or hindered grantees’ ability to do so. The PAF,TPP, and CDC Community-wide grants were made to a wide variety of grantees, including states, tribal entities, private entities, and community organizations spanning the majority of U.S. states. These former grantees provide a key opportunity to study program sustainability for up to three years after the end of a federal grant.

The proposed study will aim to meet five key objectives (Table A.1). The first objective is to assess which programs were sustained after federal funding ended, and in what form. For those programs that continued operating (that is, were sustained), we will also assess what it means to be sustained, and if and how the program may have changed in structure or scale in order to survive. We will explore and describe variation in the characteristics of the grantees, the programs, and the organizations that implement them.

The second objective of the study is to describe key factors and variables that positively or negatively influenced former grantees’ ability to sustain their programs, including the 8 factors outlined in the Sustainability Framework developed by OAH. These include the following:

• Programmatic factors: We will explore factors such as the flexibility of programs, the cost to train and implement, program length, required staff structure and delivery methods, program setting, and target population. We will also explore how findings from the program’s impact evaluation affected whether it was sustained, and if they were used for dissemination and/or replication.

• Environmental factors: We will examine the effect of factors such as policy changes, political shifts, assessment of local community needs, state or local legal requirements, engagement with key stakeholders, and community buy-in and/or demand.

• Organizational variables: We will explore how factors such as organizational size, leadership structure or changes, staff capacity, sources of funding, strategic partnerships, systems support, and operational budget affect sustainability.

The third objective is to describe the activities and processes in which former grantees were engaged to build sustainable programs. The study will examine the steps taken by former grantees during and after the grant period to ensure that their programs continued to operate after grant funding ends, as well as any strategies used to sustain program outcomes. The study team will also look at how the nature and timing of those strategies may have varied across funding streams and programs.

The fourth objective is to document and characterize the types of technical assistance, resources, and support former grantees received to sustain their programs. The study will explore the extent to which grantees used the technical assistance tools and materials developed by OAH or other sources to guide them in their efforts, specifically examining the benefits of the action steps outlined in the OAH Sustainability framework. The study will also document additional resources (such as webinars, discussions with other grantees, training) that may have been useful in improving the likelihood of programs being sustained.

Finally, the study will identify lessons learned and develop key illustrative examples highlighting successful strategies grantees were able to employ. We will also document and describe the common challenges grantees faced in their efforts to sustain programs after federal funding has ended.

To achieve the these objectives, the study will rely on data gathered through in-depth key informant interviews (Instrument #1) with up to 50 former grantees (reflecting a mix of characteristics, funding streams, and outcomes) to explore in greater detail the trajectories they followed to sustain their programs.

Table A.1. Sustainability study objectives

Objective	Document Review (Not covered under this ICR)	In-Depth Interview Master Topic Guide (Up to 50 respondents per round)
		Round 1	Round 2
1) Assess which federally funded programs were sustained, for how long, and in what form.		
2) Describe the key factors affecting sustainability of programs.			
3) Describe the methods grantees used to sustain programs.			
4) Describe the technical assistance and support grantees received to help them sustain their programs, and from whom.		
5) Describe the lessons learned and successful strategies for future grantees.			

A2. Purpose and use of the information collection

The sustainability study will collect and analyze data to understand program sustainability in the years after funding ends. Data will be obtained from the in-depth interviews, guided by the Master Topic Guide, with a subsample of up to 50 respondents. Data from existing documents submitted to OAH and the CDC will also contribute to this study.

OAH will use the data and findings from this study to identify key factors in program sustainability, the strategies that worked and did not work in sustaining programs over time, and the types of support and assistance former grantees required to sustain programs. The study team will conduct descriptive and qualitative analyses of notes produced during the in-depth interviews. The results of these analyses will be incorporated into an interim report, a final report, and up to eight case studies or briefs per round of interviews. These deliverables will clearly describe grantees’ sustainability efforts for all audiences and highlight key challenges, successes, and lessons learned for future funding and program implementation.

A3. Use of information technology to reduce burden

To help minimize the level of burden on participating grantees, the interviews will be administered via telephone, and only to the minimum number of respondents necessary to meet study objectives.

The study team will work closely with the grantee’s former OAH or CDC project officer, who will serve as a liaison between the grantees and Mathematica. The federal project officer will provide an initial introduction and be available to help the study team communicate and follow up with respondents to ensure a high response rate for the telephone interviews.

The In-Depth Interview Master Topic Guide will be administered as semi-structured interviews conducted by telephone. Grantees will be given time to review the interview request, and the study team will schedule the interviews at their convenience within the planned three-month interview period.

A4. Efforts to identify duplication and use of similar information

This study is the first and only ongoing effort to systematically collect and analyze data on the sustainability of grantees of federal teen pregnancy prevention programs or programs for expectant and parenting young families.

To prepare for data collection, the study team plans to incorporate information from existing documents submitted to OAH and the CDC, such as grantee annual progress reports, final reports, performance measure data, and sustainability plans. However, because these documents cover only the time period during which grantees were still funded, they do not provide adequate data or an accurate picture of post-funding program sustainability that will allow the study team to meet the study objectives.

A5. Impact on small businesses

Programs in some sites may be operated by small, community-based organizations. The study team will ensure that the burden on such sites is minimal and will collect data from the minimum number of respondents needed to meet the study objectives, based on grantee information and the results of previous data collection efforts.

A6. Consequences of not collecting the information/collecting less frequently

Evaluation project deliverables are expected to provide significant lessons about program sustainability that will be relevant for both OAH and for grantees. Collecting these data is crucial to closing an existing gap in OAH knowledge about how to support the sustainability efforts of current and future grantees, including the 2015–2020 TPP grantee cohort and the 2013–2016 PAF cohort.

Data will be collected from grantees on an annual basis. Collecting data at two time points after federal grants have ended, will allow OAH to understand how specific grantee activities have resulted in sustainable outcomes, both in the short and longer terms, up to three years following the end of federal funding.

A7. Special circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal register notice and consultation outside the agency

The 30-day Federal Register Notice will be posted with this submission.

The names and contact information of the persons consulted in the drafting and refinement of the sustainability study of OAH-funded grantees can be found in Attachment A.

A9. Payments to respondents

No payments to respondents are proposed for this information collection.

A10. Assurance of confidentiality

Data being gathered through this study is not sensitive in nature, and examines organizational or program practices and experiences. However, to protect individual privacy, rReporting of data gathered through this study will be in aggregate and will not reference include any personal identifying information such as staff name or attribute quotes to individuals or organization. Grantee and implementing organizations will also be de-identified in reports, tables, and figures to maintain confidentiality. Using an email template (Attachment B) prepared by the study team, the project officer will send an introductory email to all grantees describing the study, requesting their participation, and confirming the appropriate contact for data collection. Subsequent to the first contact, the study team will follow up with identified respondents. When scheduling the in-depth telephone interviews, respondents will receive information about privacy as well as a chance to opt out, should they want to do so. Finally, staff conducting the interview will reiterate privacy protection and consent details at the time of the interview.

All data will be transmitted and stored according to the level of security necessary for the sensitivity and identifiability of the data. Staff are trained to keep all electronic and hard copy interview notes in a secure location and instructed not to share any materials with anyone outside of the study team. Mathematica, the evaluation contractor, will store responses to all data collection instruments on secure network servers, with access limited to project staff on a “need-to-know” basis.

A11. Justification for sensitive questions

There are no sensitive questions in the instrument designed for the sustainability study. The questions focus on grantees’ program experiences and organizational practices, and do not touch on sensitive subjects.

A12. Estimates of the burden of data collection

1. Annualized burden estimates

OAH is requesting three years of clearance for data collection activities for the sustainability study. Table A.2 provides the estimated annual reporting burden calculations for staff for the data collection.

In-Depth Interview Master Topic Guide: The study will draw on a universal set of 72 TPP, PAF, and CDC Community-wide grantees who applied and received funding or cooperative agreements through OAH’s grant programs in 2010 (Cohort 1) but were not subsequently refunded. The study team will collect data from a subsample of up to 50 respondents during each round of in-depth interviews. Each respondent in the subsample will be asked to participate in up to two interviews during the three-year study, with each interview expected to last up to 1.5 hours. Assuming the maximum anticipated number of respondents for two rounds of interviews, the total annualized burden hours is estimated to be 50 hours.

2. Estimates of annualized costs

Assuming a wage rate of $30.99, the annualized cost of the burden of the proposed data collection is estimated to be 50 hours x $30.99 = $1549.50. This hourly wage rate represents the mean hourly wage rate for “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2012. This proposed information collection does not impose an additional financial burden on respondents other than the time spent answering the questions contained in the instrument.

Table A.2. Calculations of burden hours and cost for grantee staff

Instrument	Total number of respondents	Annual number of respondents	Number of responses per respondent	Average burden hours per response	Total burden hours	Average hourly wage	Total annualized cost
In-Depth Interview Master Topic Guide	50	17	2	1.5	50.0	$30.99	$1,549.50
Estimated Annual Burden Hours for Grantee Staff					50.0		$1,549.50

A13. Estimates of other total annual cost burden to respondents and record keepers

These information collection activities do not place any capital cost or cost of maintaining requirements on respondents.

A14. Annualized cost to the federal government

Mathematica, the data collection contractor, will conduct the data collection. The total annualized cost to the government will be $117,769.48. This total includes the cost of coordination between Mathematica and OAH, OMB applications, developing the data collection plans and instrument, conducting a pilot test of the data collection instrument with up to nine respondents, administering the final data collection instrument to all selected grantees, and data analysis.

A15. Explanation for program changes or adjustments

This is a new data collection.

A16. Plans for tabulation and publication and project time schedule

1. Analysis plan

The instrument included in this OMB package for the sustainability study will yield data that will be analyzed using qualitative and quantitative methods to assess the extent to which grantees that were funded in 2010 and not refunded subsequently, have sustained their programs, and identify the key factors that either enabled or hindered their ability to do so. A greater understanding of factors affecting program sustainability and how grantees addressed them is expected to ensure that future grant funding and the technical assistance provided to OAH grantees is useful and effective.

The research team will create a coding scheme consisting of conceptual categories and classifications linked to the evaluation research questions and different dimensions of sustainability. Team members will then use software (for example, Atlas.ti) to assign codes to specific text in the electronic file of interview notes. Coding the qualitative data in this way will enable the team to access data on a specific topic quickly, organize information in different ways to facilitate the identification of themes, and compile the evidence supporting them. As data collection proceeds, the team will refine the coding scheme to better align it with both themes and topics that emerge from the data, and with the research questions (Ritchie and Spencer 2002).² To facilitate analyses of patterns and themes across grantees, we will also code the interview notes with key grantee-level characteristics from the document review, such as funding source, status, and tier, as well as program type.

After all of the qualitative data have been coded, we will use the software to retrieve data on the research questions and subtopics to identify themes and triangulate across data sources and individual respondents. Much of the meaning of the data will be discerned through descriptive analyses—qualitative and quantitative—that organize data thematically; create summary statistics that characterize overall grantee sustainability efforts, as well as variations across and within sites; and highlight the similarities and differences among them (Patton 2002).³ We will also explore relationships across themes (for example, relationships between the types of sustainability challenges sites face and their program activities).

2. Time schedule and publications

OAH expects that the sustainability study will be conducted over the three-year period October 2015–September 2018. This request is for a three-year period and subsequent packages will be submitted as necessary for new collections or to extend collection periods; however, none is anticipated at this time. Table A.3 shows a schedule of the data collection efforts for the sustainability study, the focus for this ICR (Table A.3):

Table A.3. Proposed sustainability study time schedule

Instrument	Date of 30-day submission	Date clearance needed	Date for use in field
Sustainability study
In-Depth Interview Master Topic Guide	June 2016	August 2016	October 2016

By the end of 2018, OAH plans to produce an interim report, a final report, and up to 16 case studies or briefs summarizing its findings on program sustainability.

A17. Reason(s) display of OMB expiration date is inappropriate

The expiration date for OMB approval will be displayed on all data collection instruments.

A18. Exceptions to certification for Paperwork Reduction Act submissions

There are no exceptions to the certification statement.

www.mathematica-mpr.com

Improving public well-being by conducting high quality,
objective research and data collection

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ TUCSON, AZ ■ Washington, DC

1 See http://www.gpo.gov/fdsys/pkg/PLAW-111publ148/html/PLAW-111publ148.htm; Section 10212.

2 Ritchie, J., and L. Spencer. “Qualitative Data Analysis for Applied Policy Research.” In The Qualitative Researcher’s Companion, edited by A.M. Huberman and M.B. Miles. Thousand Oaks, CA: Sage Publications, 2002.

3 Patton, M.Q. Qualitative Research and Evaluation Methods: Third edition. Thousand Oaks, CA: Sage Publications, 2002.