Ss_b

TPP Tier 1B Design and Implementation Study Supporting Statement B

Supporting Justification for OMB Clearance of Teen Pregnancy Prevention Tier 1B Design and Implementation Study

Part B: Statistical Methods for TPP Tier 1B Design and Implementation Study Data Collection

May 2016

1. Respondent Universe and Sampling Methods

For the TPP Tier 1B Design and Implementation Study, we will document how each of the 50 grantees funded under this grant program are scaling-up efforts to strengthen and expand the reach of evidence-based TPP programs in their respective communities. This work starts with a systematic review of each grantee’s grant application to OAH. OAH anticipates that grantees will employ diverse strategies in working within their communities to scale up their initiatives. Because this information collection will contribute to the emerging knowledge base about community-wide efforts to scale up evidence-based programs (EBPs), mobilize community support, and establish linkages to youth-friendly health services at the community level, it will be important to document the variety of grantee approaches and challenges they have encountered as a result of local conditions and strategies. To document these features and experiences, a lead staff member in each grantee organization will be interviewed by phone as well as up to two key grantee partners (e.g., organizations that provide the EBPs to youth or lead the initiatives’ Community Advisory Group (CAG)). Partners to be interviewed will be selected based on the prominence and variety of their roles within each initiative in order to provide multiple perspectives on implementation.

To obtain more detail on implementation than can be gathered in a telephone interview, site visits with up to 15 grantees will be conducted to collect data that will illustrate in detail a variety of approaches and strategies for scaling up to the community level evidence-based approaches to teen pregnancy prevention. The 15 grantees will be selected and recruited based on a review of their grant applications, which will incorporate the degree to which grantees’ approaches to implementation are typical or novel and the variety of community contexts in which they operate. Each site visit will examine the drivers of implementation and investigate topics of particular interest to OAH. Data collection will cover all of the setting types (e.g., in-school middle school, community-based settings) served by the grantee, although it will not be necessary to interview one stakeholder from every individual site (e.g., each specific middle school or community-based location). We will aim to collect information from a representative subset if there are many setting types within a grantee project.

Within each participating grantee community, data will be collected from grantee and partner staff and community members with varying roles and responsibilities. They will have knowledge about the origins and operations of their project and the challenges it has encountered. Focus groups will be held with approximately eight youth participating in each Youth Leadership Council (YLC) and select CAG members who agree to participate (the number of focus groups will vary based on the number of YLCs and CAGs associated with each grant). The youth and adults will not be randomly selected for the focus groups. Up to 390 participants will be included in the data collection overall (including telephone interviews and site visits).

2. Procedures for the Collection of Information

We developed an overall topic guide (Attachment A) to organize data collection and documentation of the grantees’ projects. This topic guide identifies the information that will be gathered to document the grantee strategies, challenges, and resolutions. For the phone interviews, we have created an interview guide for discussions with the grantees and their partners (Attachment B). The site visit data collection will build on information from the phone interviews and probe in more detail into grantee strategies and decision-making. The specific topics that will be relevant and the individuals best positioned to provide information on each aspect of program implementation will vary from site to site, but will fall within the topics listed in the overall topic guide.

Topics to be explored in this data collection include: community and organizational context; partners and their roles; program models (EBPs) selected; settings and sites where the EBPs are implemented; strategies for scaling the EBPs and the initiative to community-wide coverage; use of the pilot year; monitoring and managing fidelity and quality; ensuring safe and supportive environments for youth; linkages and referrals to youth-friendly healthcare services; community mobilization and strategic dissemination; sustainability planning; and recruitment and retention of youth participants.

The most intensive data collection for the implementation study will take place in a site visit to each of the 15 selected grantee communities. The site visits will use semi-structured key informant interviews with grantee staff and staff from partner organizations; collection of any additional documents provided by grantees or their partners for later review; semi-structured observations of activities, classes and other services; and focus groups with youth and community stakeholders. Although the general topic guide will be tailored to the circumstances and design of each site’s project, we can project in general terms which of these data sources will be accessed to explore the major topics of interest (see Attachment A).

The visits will last up to four days, depending on the complexity of the program design and number of key partners. The site visits will occur half-way through the grant projects’ second year and first year of full implementation. This will allow the data to reflect grantees’ experiences with lessons learned from the pilot year. Each visit will be conducted by a two-person team. The team leader will be a senior project member who can communicate clearly, organize work effectively, provide strong analytical thinking, and remain objective and professional. The two-person approach to the site visits will increase the effectiveness of probing during interviews and the accuracy of information obtained. It also builds in flexibility to accommodate site schedules, allowing site visitors to split up and cover different interviews if the need arises.

Communication with Grantees and Communities

Prior to conducting the telephone interviews, the study team will send an introductory email to the project director for each of the Tier 1B grants, explaining the purpose of the telephone interviews, asking to schedule an interview, and asking about key partner staff with whom we should also speak (Attachment D). We will reach out to key partner staff with a similar introductory email explaining the purpose of the telephone interview and asking about their availability.

Once a telephone interview is scheduled, the study team will provide interview questions to the respondent, which will allow respondents to feel more comfortable and prepared for the interview, and to think about questions prior to the interview or look up information, if desired.

Training and Data Collection

Prior to the grantee telephone interviews, the data collection team will participate in a group training to provide a study overview and to review the data collection protocols in detail. Following the training, each telephone interview will be conducted by a two-person team: a lead interviewer and a note-taker. Most interviewers will have participated in the grant application review and will already be familiar with the Tier 1B program. Before each phone interview, the lead interviewer and note-taker will review the grantee’s application summary and any additional available information on the project. The lead interviewer will determine if additional probes are needed to address the data elements for a specific particular project, and tailor the probes for partner organizations as needed. For efficiency, in cases where the grantee is participating in the site visits, the site visit team will conduct the telephone interviews, reducing the need for repetition in subsequent in-person interviews.

Each telephone interview will last one and a half hours, on average. After the interview, the note-taker will finalize the interview notes and verify that all information is accurate and complete. The lead interviewer will review the information for clarity, consistency, and completeness.

The site visit teams will participate in training on recruitment and site visit protocols prior to the site visits to ensure consistency, quality, and efficiency. The site visit leader will create customized discussion guides to ensure that we collect the needed information in an efficient, consistent way from the most appropriate respondents. The site-specific plan will include a customized topic guide, which may elaborate on or provide “local language” versions of topic definitions, and may eliminate some topics if they are not relevant or already thoroughly explored (in the phone interviews). The plan will identify which information will be collected from which sources, key respondents who should be interviewed, and other sources that should be tapped. Study leaders will review the site visit plans and customized discussion guides for each site to help ensure consistency across sites and to facilitate inclusion of topics to inform issues that are emerging from early visits.

Interviews and focus group discussions may be audio-recorded and transcribed for accuracy. Following each site visit, the site team will complete a project summary, drawing on observation notes and interview and focus group transcripts (where applicable), presenting what was learned from all of the site visit activities. Follow-up communication may be necessary to confirm information and to answer questions that arise in the process of writing the summaries. The summaries represent an important preliminary analytic step that allows the site visitors to record impressions and interpretations as well as what they learned.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Phone interviews and site visits will be planned well in advance so that all identified respondents can participate in individual or group discussions. Furthermore, as a condition of their grants, grantees are required to participate in the federal evaluation conducted or commissioned by OAH. Based on our experience in the Federal TPP Replication study, we anticipate that refusals to participate and absences will be rare.

4. Tests of Procedures or Methods to be Undertaken

Many of the topics and probes in the implementation study protocols have been tested and refined in a previous study with similar respondents (the TPP Replication study).

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

The TPP Tier 1B Evaluation phone interviews and site visits will be overseen by staff of Abt Associates, the contractor selected in September 2015 to conduct the study. Listed below are the individuals whom OAH consulted on the collection and/or analysis of the data described in this document. They include the contractor staff for the evaluation and the OAH Contract Officer Representative (COR) for the evaluation.

Meredith Kelsey, Ph.D

Abt Associates

55 Wheeler Street

Cambridge, MA 02138

(617) 520-2422

Kimberly Francis, Ph.D

Abt Associates

55 Wheeler Street

Cambridge, MA 02138

(617) 520-2502

Jean Layzer, M.A.

Belmont Research Associates

42 Fairmont Street

Belmont, MA 02478

(617) 484-8189

Lesley Freiman, MPP

Abt Associates

55 Wheeler Street

Cambridge, MA 02138

(617) 520-2334

Carolyn Layzer, Ph.D

Abt Associates

55 Wheeler Street

Cambridge, MA 02138

(617) 520-3597

Amy Farb, Ph.D

Office of Adolescent Health

Office of the Assistant Secretary for Health

U.S. Department of Health and Human Services

1101 Wootton Parkway, Suite 700

Rockville, MD 20852

(240) 453-2836

Dr. Farb is the TPP Tier 1B Design and Implementation Study project officer and has overseen the development of the current interview and focus group instruments. Inquiries related to the TPP Tier 1B program or evaluations of it may be directed to Dr. Farb.

0