Form IA Interview Topic IA Interview Topic IA Interview Topic Guide

RPG – Site Visit Topic Guide

RPG Cross-site Evaluation

REGIONAL PARTNERSHIP GRANTS (RPG) CROSS- SITE EVALUATION
TOPIC GUIDE FOR IMPLEMENTATION STUDY SITE VISIT INTERVIEWS
The core implementation study for the Regional Partnership Grants (RPG) cross-site evaluation will
include two multi-day site visits to each grantee. Visits will occur in years 2 and 4. Researchers will
interview grantee program directors, managers, supervisors, and frontline staff who work directly
with families during the site visits. Interviews will be conducted either one-on-one or in small
groups, depending on staffing structure, roles, and the number of individuals in a role.

Topic
Informant
Characteristics

Sub-Topic
Informant Characteristics
Job title
Education background/licensing qualifications
Years in current position and with agency
Role on RPG
Relevant prior experience with the RPG grant program, target population, and evidence-based
programs (EBPs) being implemented by the grantee
Pre-Implementation

Selection of
EBPs

Grantees’ prior experience with similar programs and how prior experience informed the RPG
design
Knowledge of evidence-based practices (EBPs)
Involvement of partners and other community organizations/stakeholders in the planning and
decision-making processes, and how concerns were addressed
Involvement of frontline staff in the planning and decision-making processes, and how concerns
were addressed
Key design decisions made during the planning phases and rationale for those decisions
Process by which grantees selected the planned interventions, including:
•

Community need to be addressed by the EBP

•

Needs and resource assessment (including need for and availability of: space,
technology, financial and other resources, including in-kind contributions by grantee
and/or partners)

•

Alignment with planned target population

•

Assessment of organization capacity/readiness

•

Whether other programs were considered

•

Champions for certain EBPs

•

Need for adaptation

• Alignment with grantee and partners’ goals and mission
Challenges encountered during the planning process and steps taken to address them

DRAFT

1

RPG – Site Visit Topic Guide

RPG Cross-site Evaluation

Topic
Referral
Processes to
RPG services

Sub-Topic
How and when grantee determined referral pathways
Sources of referrals, length of relationship with these referral sources, and how relationships
were established, relative size of enrollment from each referral source
Referral sources that consistently refer individuals that meet eligibility criteria and engage in the
RPG program
Process used by other agencies to refer potential participants to RPG
Any changes to outreach and referral strategies and why
Barriers and facilitators to establishing pathways and translating referrals into participation
Sustainability of referral pathways

Staff Selection
and Hiring and
Retention

Staffing structure for the RPG program, including frontline staff and those who support their
implementation (program directors, managers, and supervisors)
Responsibilities and expectations for each staff role
Timeline and process for hiring new staff or reassigning staff to fill RPG roles
Re-assignment of existing staff to implement RPG services and/or support implementation
Whether job postings specified specific qualifications required for implementation of EBPs
selected; recruitment methods used to identify likely qualified candidates; protocols and criteria
used to identify qualified candidates
Number of full-time equivalent (FTE) staff in RPG program devoted to administration and
direct service provision
Extent to which staff in the RPG program had other responsibilities in addition to RPG
Current staff vacancies; length of vacancies; efforts to fill vacancies
Extent of staff turnover since initiating program operations; reasons for staff turnover (or staff
retention); effects on remaining staff when turnover occurs; length of process to replace
departing staff
Effect of staff turnover on enrollment and service delivery; programmatic adjustments and
accommodations as a result of turnover
Efforts to prevent future turnover and retain current direct service staff and supervisors
Expectation of continued rate of turnover for sustainability
Likelihood of identifying individuals with necessary qualifications for sustainability and/or scaleup

Pre- and Inservice Training

Plan for and approach to providing supervision and training to direct service staff, including the
intended frequency, duration, and focus
Initial and in-service training plan for new and ongoing RPG program staff, including the
frequency, content, length, and format of training, and individual or organization providing the
training (includes whether the EBP’s developer or purveyor was involved in training, whether
training covers key components of EBPs and whether trainees were given time to practice
implementation with feedback)
Grantee’s ability to provide sufficient training to all necessary staff, at start-up and for
sustainability and/or scale-up
Staff perception on extent to which training(s) provided necessary information on theory of
intervention(s), goals of RPG, as well as competencies needed to implement
Whether staff received the planned level of initial and ongoing training and guidance

Implementation

Organizational structure for the RPG project

DRAFT

2

RPG – Site Visit Topic Guide

RPG Cross-site Evaluation

Topic
Teams

Sub-Topic
Development of implementation team; timing of development, relative to program
implementation
How grantee determined members of implementation team; qualifications established for team
membership; member characteristics
Roles and responsibilities of team and its members
Strongest advocate for RPG program and how demonstrated; role of advocate and how
individual emerged as advocate
Existence and role of advisory committee and/or steering committee
Duration of operation of implementation team; frequency of meetings; forms of communication
by team members;
Turnover of team membership; reasons for turnover; impact on implementation
Barriers and facilitators to fully installing implementation team in RPG project
Accomplishments of implementation team
Staff perception of usefulness of team
Sustainability of team for scale-up

Implementation
Plans

Development of plans and procedures used to ensure that all staff carry out program activities
as planned and in a consistent manner; what details were included in plan (e.g. types of tasks,
timeline for activities, staff responsible for tasks)
Modifications to the grantee’s RPG implementation plan that have occurred since
implementation began; reasons for modifications; whether they were planned or unplanned
Development of strategies to address barriers to the program’s ability to deliver high-quality
services
Staff perceptions of whether implementation plan was communicated sufficiently, executed
successfully, and useful in proactively identifying roadblocks to implementation
Barriers and facilitators to success of implementation plan
Sustainability of implementation plan as RPG programs adapt
Early and On-Going Implementation

Facilitative
administrative
support

Grantee oversight of RPG activities and partner services
Changes in the demonstration’s organizational structure
Changes in grantee, partner, or RPG program leadership staff that occurred during the
demonstration and may have impacted the direction of the RPG program
Strategies to reduce administrative barriers, develop communication and feedback protocols,
implement program improvement based on data or staff suggestions
Staff perception of availability of these strategies
Staff perception of administration’s commitment to supporting the implementation of EBPs
Sustainability of leadership approach
Facilitators and barriers to providing administrative support

Supervision and
Feedback

Whether protocols were established for providing feedback
Use of staff performance assessments for frontline staff
Sources of data for performance assessments
Facilitators and barriers to supervisory and feedback mechanisms

Technical

Grantee and partners’ capability to provide ongoing TA for duration of RPG and beyond

DRAFT

3

RPG – Site Visit Topic Guide

RPG Cross-site Evaluation

Topic
Assistance and
Coaching

Sub-Topic
Use of external TA providers, including curriculum developers, Children’s Bureau, and other
entities; whether staff have accessed these resources and, if so, helpfulness of the technical
assistance
Extent to which grantee believed TA providers were aware of and receptive to goals for RPG
Whether TA and coaching led to further adaptation of program model
Whether TA and coaching led to need for further training sessions
Topics on which staff needed more training and technical assistance
Plans for when, how, and why TA or coaching would be provided
Sustainability of TA and coaching networks, including financial and other resources

Internal
evaluation and
continuous
program
improvement

Grantee expectations about the quality of services delivered through RPG; how grantee defines
high quality delivery for core services, and why program defines service quality in this manner
Efforts to monitor service quality, adherence to curricula or other programming, client
engagement, participation, and participant outcomes; who completes monitoring; what is
monitored and how often; how information is used by staff
Strategies for identifying successes and challenges to implementation for purposes of continuous
program improvement
Use of improvement cycles or other continuous quality improvement strategies
Dissemination of RPG implementation to policymakers: frequency of exchanges, extent to
which such exchanges are purposeful and part of usual practice
Dissemination of RPG implementation to partners: frequency of exchanges, extent to which
such exchanges are purposeful and part of usual practice
Dissemination of RPG implementation to researchers, other practitioners: plans to publish
findings
Facilitators and barriers to ongoing evaluation and program improvement
Sustainability of systems needed to monitor and improve program implementation

Decision
Support Data
Systems

Use of data systems to monitor progress toward goals and partner performance
Plan for monitoring program performance and for tracking service delivery and quality,
adherence to curricula or other programming, client engagement and participation, and
participant outcomes
Data sources and frequency of data collection and analysis
How staff use the data to make program decisions
Dissemination plans for: partners/stakeholders, administrators, support staff, frontline staff;
perception of purpose of feedback dissemination
Staff perception of the relevance and usefulness of program data, management information
system
Sustainability of data systems
Barriers and facilitators to using systems and conclusions derived from data

Referral
Processes from
RPG services

Types of community services to which RPG program staff refer participants
Extent to which needed services are available and accessible in the community
Plan for conducting initial and ongoing assessments of participants’ needs and linking them to
appropriate services
Extent to which participants follow up on referrals and take up the services
Process for tracking referrals, how often progress is monitored, and who is responsible for

DRAFT

4

RPG – Site Visit Topic Guide

Topic
Interventions
with external
systems

RPG Cross-site Evaluation

Sub-Topic

monitoring

Strategies to engage external systems in provision of financial, organizational, or other resources
Types of external systems/organizations engaged
Staff perception of alignment of organizations with grantee’s goals for RPG services
Adherence/Fidelity

Fidelity

How grantee defines high quality delivery of core components of the EBP, and why grantee
defines quality in this manner
The extent to which staff adhere to the EBP guidelines (during service delivery)
Consistency with which services are provided, per EBP guidelines
Grantee expectations about the quality of services delivered through the EBP
Attitudes expressed by staff towards the use of the EBP
Extent to which program staff think that the EBP will improve outcomes
Staff understanding of EBP’s theory of change (how program services are linked to desired
outcomes)

Staff Attitudes
Toward
Implementation

Staff perceptions as to whether these multiple roles had an effect on their ability to implement
the program as designed
Extent to which RPG program staff “bought in” to the idea that providing substance abuse
treatment, family strengthening, parenting education, and/or in an integrated package would
improve participant outcomes
Staff perceptions of the EBP’s fit to the target population, strengths, and weaknesses
Staff perceptions about how well the model has worked in practice; benefits and challenges of
this approach to demonstration leadership
Staff perceptions of the strengths and weaknesses of the demonstration’s approach to leadership
Staff perceptions of the utility of an integrated approach to the provision of core services
Community, State, and National Context

State and Local
Context

DRAFT

State or local policies and policy climate, and how they impeded or supported program
development
Other state or community organizations providing parenting or employment services; how the
services provided by these organizations differ from the RPG program; whether and how these
services may have affected the RPG program; and use of these other services by participants
Role of the courts and willingness of family court judges to support and participate in RPG
Physical, social, and economic characteristics of communities in which RPG is offered
Unexpected events that altered RPG program activities; how they affected the program and how
they were addressed

5