Staff and community member interview (Master Topic Guide)

Form Approved

OMB No. 0990-xxxx

Exp. Date XX/XX/20XX

Master Topic Guide

This guide provides a comprehensive list from which topics will be selected in developing customized site visit protocols for each PPA evaluation site. ¹ The first three major sections focus primarily on topics that are relevant to the program that is being implemented and evaluated, and the fourth major section identifies topics that focus on programs and services available to control group youths.

Some topics can be addressed in the first site visit, others need to be addressed in the first site visit and updated in the second visit, and some topics are relevant only in the second visit. The topics that we expect to address only in the first visit are preceded by an asterisk, while topics that we expect to address only in the second visit or using follow-up data are preceded by a “+”. All other topics will be addressed in the first visit and updated on the basis of the second visit.

The customized site visit protocols developed for each PPA evaluation site using the Master Topic Guide will include guides for interviews with program leaders and key staff, representatives of sponsoring school or community based organizations, program partners, key school staff, and community members. The topics will also be addressed, as appropriate in group discussions with front-line staff and youths participating in the program and where appropriate, other community services.

THE PROGRAM BEING EVALUATED

Program background

• *Name and version of the program being implemented

• *When, by whom, and how the program was developed

• *Program type according to the developer and program staff (e.g. abstinence until marriage, abstinence-based education, sexuality education, family planning education and outreach, STD/HIV education and prevention, relationship education, or youth development program)

• Core components of program –

  • Core content

  • Core pedagogy/instructional strategies

  • Core implementation logistics

• Planned adaptations to the original program model and reasons for adaptations

• *Prevalence of current and past use of this program (number of copies of the program materials sold, number of school districts and other organizations that have purchased the program)

• *Prior evidence of effectiveness

• *Rationale for including this program in the PPA evaluation

Site background

• *Stakeholders and their views on teenage pregnancy and how to achieve program goals

• *Organizational settings for program implementation

• *Number of program and control group locations participating in the evaluation, and how they are defined (school or community-based organizations)

• *Characteristics of the general population of youths in the site, and variations in characteristics across locations

• Other programming currently targeting similar outcomes in program and control group locations within the site

• *Previous programs aimed at preventing adolescent pregnancy that have been implemented in the evaluation locations and may have provided valuable experience

• Availability of family planning services for teens at site or in community (community clinics, school-based health center, etc)

Local priorities

• *Perception of the problem that the program is being implemented to address

• *Primary goals of the program

• *Outcomes targeted for change

• *Population targeted for change

Design and logic of the program

• *Program’s main messages

• *Program’s approach to improving the targeted outcomes

• *Mechanisms by which program’s key features are expected to improve outcomes

School needs and support

• *School needs and values considered in program selection and implementation (if program is implemented in school-based locations)

• Steps taken to gain or maintain school and district support for program

• Conflicts between schools and the program, and how they have been handled

Community needs and support

• Public sentiment about adolescent pregnancy and pregnancy prevention programs for adolescents

• Visibility of the program in the community

• Community members and organizations important to the program, and how their support is critical

• Steps taken to gain or maintain community support for the program

External events

• +Outside influences that may have affected the program outcomes targeted for change

Other programs

• Other programs and services with similar goals available to program youths

• +Number of program participants participating in these programs or receiving these services

PROGRAM IMPLEMENTATION

Lead organization

• *Organization responsible for implementing the program in the evaluation

• Formal and informal partners and their roles in implementing the program

• Whether a national, state, or regional organization sponsors or oversees program implementation, and if so, its role in implementing the program in the evaluation

Role of schools and other community agencies

• Whether the program staff coordinates with other community agencies to implement activities or provide services to participants; if so, describe

• Whether the program staff coordinates or links with school activities; if so, describe

Funding and Cost

• *Total budget for implementing the program

• *Budgeted cost per participant

• Sources and amounts of funding and significant in-kind donations

• Adequacy of program funding to support program implementation as planned

• Adjustments necessary to make it feasible to implement program with available funding

• +Whether actual cost was less than, about the same as, or higher than budgeted, and reasons for differences from budget

• +Major program costs (staffing, external staff training and technical assistance, program materials, other)

• +Time spent by staff (on training, on service delivery in an average week)

Facilities

• *Infrastructure (such as special facilities, space, or computer technology) needed to implement the program

• Infrastructure available

• +Modifications to program, if any, made because of differences in infrastructure needed and available

Staff structure and background

• Staff positions and minimum qualifications for frontline staff positions (request job descriptions for dedicated staff)

• Positions filled with staff members from partner organizations

• Role of volunteers

• Educational background and relevant experience of frontline and leadership staff

• +Number of staff who have left and been replaced since evaluation period began

• +Reasons for turnover

• +How turnover is managed

Training and technical assistance

• Pre-service and in-service training required for staff and volunteers in each position

• Participation in required training sessions

• Training offered on a voluntary basis

• Participation in voluntary training related to role in program

• +Usefulness of training

• Access to technical assistance in implementing the program

• +Sources, amount, and topics of technical assistance received

• Certification requirements, process, and cost

Outreach and recruitment

• Outreach strategies to recruit youth and/or identify youth who should participate; effectiveness of such strategies; reasons why strategies may not be effective; any steps taken to address challenges; results of modifications

• Eligibility criteria

• Procedures for identifying and enrolling eligible youths

• Recruitment and enrollment targets

• How recruitment and enrollment are tracked (MIS, other tools or procedures)

Numbers and characteristics of enrollees

• Number of program spaces (youths planned to serve)

• +Number of youths enrolled

• +Reasons for differences between numbers of spaces and youths enrolled

• Characteristics of youths enrolled

Parental permission

• *Parental permission requirements

• Parental permission procedures

• Proportion of potential participants whose parents did not give permission

• Concerns that led some parents to withhold permission

Essential program features

• Youth-focused features (such as adult-led instruction, peer-led instruction, individual counseling/mentoring/tutoring, large group events, online resources and activities, direct health services, community service/experiences…)

• Adult-focused features (such as parent involvement, parent training, train-the-trainer activities…)

• Community-focused features (such as special events, media campaigns, websites…)

For each key feature:

Timeline and Location

• *Planned timeline for implementation

• *Planned schedule of activities

• +Deviations from the planned timeline and schedule, and reasons for them

• Location(s) of program activities

Number, Frequency, and Duration of Sessions

• *Number, frequency, and duration of sessions prescribed by the program developer

• *Number, frequency, and duration of sessions planned by program leaders, and reasons for any deviations from program developer’s specifications

• +Deviations from the planned number, frequency, and duration of sessions, and reasons for deviations

Staffing

• Numbers and types of staff and volunteers involved, and their roles

Types of activities

• Range of activities

• Typical session

• +Observed session

Program materials

• Type and content of materials available from the program developer and used (curriculum materials, teacher/leader guides, assessment/monitoring tools)

• Type and content of supplemental materials used

• Topics and key messages conveyed

• Instructional strategies

Other types of participant contact

Unplanned adaptations of the program to fit context

• Changes to the curriculum, and why

• Changes to the schedule, duration, or frequency of sessions to accommodate context, and why

• Changes to recommended program staffing to fit context, and why

• Supplemental materials used, and why

• Other changes to respond to needs of target population, and why

Distinguishing features of program

• Relative to other related programs and services for adolescents, aspects of the program that are unique or special

IMPLEMENTATION FIDELITY AND QUALITY

Group sizes and adult:youth ratios

• *Maximum group sizes and adult:youth ratios planned during instructional activities

• +Actual group sizes and adult:youth ratios during instructional activities

• +Reasons for differences between planned and actual group sizes and ratios

• +Group sizes and adult:youth ratios during observed activities

Staff-youth interactions

• +Quality of interactions during observed activity

• +Nature of interactions as described by staff and youths

• +Whether staff are comfortable presenting the material

• +Whether youths are comfortable with material that is presented

Youth involvement

• Ways youth are involved in planning program activities

• Receptiveness of youths to the program

Service and participation tracking

• Attendance records

• Other ways of tracking youth participation or service receipt

• Outcomes measured, and how

Fidelity of implementation to core components (content/pedagogy/implementation logistics)

• Performance standards and monitoring by program developer

• Self-assessment process and changes made to program based on self-assessment

• Definition of and adherence to implementation benchmarks

Implementation challenges and successes

• +Challenges

• +Aspects of the program most difficult to implement, and why

• +Accomplishments most proud of, and why

• Program features or other factors that are believed to be important to program’s success

• +Lessons learned in implementing the program

Participation and engagement in program activities

• Mandatory and voluntary activities

• Minimum participation requirements and consequences for not meeting them

• Monitoring of participation

• Followup when attendance is low

• +Levels and patterns of attendance at key program activities

• +Definition and rates of program completion

• +Reasons for youth dropout

• +Engagement of participants in program activities

Parental involvement

• Opportunities for parent input to program

• Opportunities for parent participation in activities

• +Proportion of youths whose parents participate in these activities

1 In this and other PPA implementation study documents, the term “program” refers to the intervention that is being implemented. The term “site” refers to the organization and places in which the program (intervention) is being implemented and in which control group members are located. The term “location” refers to a specific place in which the program is implemented within a site.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0990-xxxx. The time required to complete this information collection is estimated to average one hour and thirty minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Health & Human Services, OS/OCIO/PRA, 200 Independence Ave., S.W., Suite 336-E, Washington D.C. 20201, Attention: PRA Reports Clearance Officer