guide for community partners

APPENDIX F

Evaluation of Teen Pregnancy Prevention Replications

Discussion Guide for Use with PARTNER AGENCY STAFF

Program Name:

Program Location:

Sponsoring Organization:

Individual(s) Interviewed: (names and titles)

Contract Staff (as appropriate):

Date of Communication:

Instructions for Site Visitors

This visit has a set of very specific goals:

• To ensure that we have a complete and up-to-date understanding of the roles and responsibilities of the grantee and partners and their staff:

• To expand our understanding of aspects of readiness and preparation that would support strong replication of a program model (preliminary information has been abstracted from the grant proposal and other extant documents and is incorporated into the profile you received. Please make sure that you have read this and are thoroughly familiar with it so that you can probe for updated information and identify incorrect information);

• To understand the plan for replication of the program model, the adaptations that were approved and made, and the extent to which the replication was implemented as planned (again, you will have been given a summary of the replication plan as contained in the proposal and updated in the request for continuation funding. Your task will be to use discussion and direct observation to determine how the program is actually implemented and to determine the extent to which the various aspects of the replication were implemented as planned);

• To understand the local context in which the replication is being implemented (this includes the school or agency environment as well as the social structure ,behavioral norms, resources and services availability of the local community);

• To understand the ways in which the grantee changed or adapted aspects of the replication plan in response to local needs or pressures;

• To understand the challenges encountered in replicating the program model and in other aspects of implementation, the extent to which staff are able to address those challenges and the strategies they employ to address them; and

• To document the services provided to the members of the control group.

These goals will have been articulated in prior correspondence with the sites before the visit, but you should reiterate them at the beginning of any discussion with staff. You should hand the following statement to everyone you interview and an appropriate version of it to youth participants in focus groups.

Thank you for taking time to talk about (Name of Program). As you know, we are conducting the TPP Replication Evaluation. As an important part of that effort, we are visiting programs that are participating in the evaluation two or three times during their grant period, in an effort to understand and document the process of replicating an evidence-based model in real-world settings, the challenges that arise, and how staff on the ground respond to those challenges. The information we gather will serve two purposes: it will help future program operators and policymakers understand what is needed to replicate a program with fidelity, to implement it as intended, with the populations originally targeted; and it will be used to help us understand variations in program impacts, where they occur.

You may, of course, choose not to discuss any topic or end the discussion at any time. We will combine the information from this visit and subsequent visits, with information from your program documents and the performance and fidelity data you have collected, to create a narrative account of your program and the process of replicating the program model you selected. At the same time, we will combine the information about your program with information about other programs in the study to identify consistent themes that apply more generally across a range of program types and replication efforts. Neither your name nor the names of any individuals will be reported, and the notes we take about our discussions will not be shared with or provided to the federal government or anyone else except the members of the evaluation team.

A: Readiness/Preparation: Partner Organization

A1. Age, size, structure, mission of the partner organization. Type and scope of current operations. Role in program implementation. (Use profile and check for accuracy)

Probes: How is your organization involved in (name of project)? What role do you and your staff play in implementing the program?

A2. Prior experience with grantee organization, view of its position in community.

Probes: Have you worked with (grantee name) before? On what kinds of projects? How is the grantee organization viewed in the community – in terms of its mission, the accessibility of its programs and services, its ability to reach and serve needy populations?

A3. Prior experience with programming for youth, with sexual health programming

Probes: If not mentioned earlier, what experience has your organization had with youth programming, sexual health programming? How successful were these earlier efforts in terms of attracting and retaining the target population, ability to implement the intervention as planned, any outcomes measured? Any adverse reactions/opposition from community members?

A4. Selection of program model for replication

Probes: Were you involved in the choice of the program model to be replicated? What information did you use in determining the need for the program (problems in community, statistics on teen pregnancy, births, STIs)? What were the considerations in selecting (name of program model)? In what ways did it appear appropriate to the needs you identified? Did you foresee any challenges in implementing this program model – if so what were your concerns (agency policies, community opposition, school district concerns about aspects of the program)? What was your vision for the program and what it might accomplish or lead to?

A5. External support for the program

Probes: What resources, if any, were there in the community to support the program (sexual health services, youth programs as sources for referral into the program or sources for additional services)? Were there organizations or individuals in the community you felt could be counted on to support the program (school district or school staff, local government agencies, private agencies)?

D. Implementation: Putting the Program in Place (Ask only if involved in program delivery)

D1. Settings for the program

Probes: Were you able to implement the program in the number and type of schools (other settings) that you planned? What obstacles did you encounter? Were you able to overcome them? How?

D2. Staffing the program

Probes: Did you make any changes in program staffing as a result of the pilot year? What were they? What is the workload (case flow) for front-line staff? Is it more or less than you expected? What are the reasons for the difference? Have you lost any of your original staff? How many and over what period?

D3. Target population

Probes: Are you serving the youth you planned to serve, in terms of numbers, characteristics, risk factors? If not, what barriers to your original plan did you encounter? What outreach strategies have you developed to recruit participants? How do you recruit youth for the program? Have you encountered problems with retention? What strategies have you developed to improve retention?

D4. Schedule for program activities

Probes: How is the program delivered? In how many sessions, of what length, and over what period of time? What challenges to scheduling the program did you encounter? How does scheduling affect retention?

D6. Program components/activities

Probes: have you been able to implement all the components/activities required by the program model (as adapted for the replication)? If not, which ones have you had to drop or modify? What were the reasons for the change?

E. Implementation: Administrative and Supervisory Processes

E1. Working with grantee

Probes: Have you been able to work productively with the grantee? What problems or barriers did you encounter?

E2. Decision-making and problem-solving processes and strategies

Probes: Who is involved in making decisions about the program, solving problems that arise?

E3. Maintaining school and community support

Probes: Have you been involved in maintaining support for the program in schools (or community agencies)? What difficulties have you encountered?

F. Support for Staff Performance (ask only if partner is involved in delivering the program)

F1. In-service training for staff

Probes: Do you provide in-service training for your front-line staff? What type and amount do you provide? What about new staff … how are they trained?

F2. Consultation and coaching

Probes: In addition to any in-service training, who can front-line staff go to for advice, consultation? Does this happen as a regularly scheduled activity, or as needed?

F3. Monitoring, evaluation and feedback

Probes: Who is responsible for monitoring staff performance, in particular monitoring fidelity to the program model and effectiveness of delivery? How is that information used, in addition to reporting it to OAH? Is it used to provide feedback to front-line staff? Who provides the feedback and on what schedule? What has been staff reaction to the monitoring tools and any feedback? Do they find it helpful? Do they believe that the monitoring tools assess performance accurately?

G. Community Context

G1. External events that affected program implementation

G3. Community attitudes toward the problem of teen pregnancy

Probes: What are the prevailing attitudes towards adolescent sexual and other risk behaviors? What are the beliefs about teen pregnancy (i.e. a large problem, a manageable problem)? Are teen sexual behavior and pregnancy perceived as problems by members of community?

G4. Visibility of the program and community response

Probes: Is this program (highly) visible in the community? What is the level of community support for and/or opposition to the program from schools/school supervisors/community leaders? What are the sources of support for and/or opposition to the program from schools/schools supervisors/community leaders? Have you received any positive or negative messages about your program? Are there particular components of the program that are perceived positively or negatively by the community?

Abt Associates Inc. Appendix F: Discussion Guide for Use with PARTNER AGENCY STAFF