Topic guide for staff and stakeholder interviews

INSTRUMENT 1

TOPIC GUIDE FOR STAFF AND STAKEHOLDER INTERVIEWS

This page intentionally left blank for double sided copying

STREAMS

TOPIC GUIDE FOR PROCESS STUDY STAFF AND STAKEHOLDER INTERVIEWS

The STREAMS process study will include semi-structured interviews with staff and stakeholders for the HMRE programs selected to participate. This topic guide identifies the range of topics that will be covered during interviews. The guide will be used to develop respondent-specific guides that include only topics relevant to particular types of staff and stakeholders. Respondent-specific guides will also convert topics to questions that will be asked during interviews.

Most interviews will be conducted during a multi-day site visit to each participating program. Visits will occur approximately half-way through enrollment for STREAMS. Administrative and supervisory personnel and staff who have direct interaction with participants will be interviewed as well as representatives of partner agencies and other stakeholders. Interviews will be either one-on-one or small group, depending on staffing structure, roles, and the number of individuals in a role. Interviews will last one hour, on average.

OMB #:XXX-XXXX

EXPIRATION: XX-XX-XXXX

Strengthening Relationship Education and Marriage Services (STREAMS) Staff and Stakeholder Interviews

GENERAL INSTRUCTIONS Thank you for speaking with me today. On behalf of the U.S. Department of Health and Human Services (HHS), we are interviewing staff at each STREAMS evaluation site. The purpose of this interview is to learn how your HMRE program operates, your role in the program, and your experiences with the program. Your participation will help HHS learn how to better provide HMRE services. Your name will not shared in our reports or any other publications resulting from this study, and your responses will remain private to the extent permitted by law. We want you to know that: 1. Your participation in this interview is voluntary. 2. The interview will take about 60 minutes to complete. 3. We hope that you will answer all the questions, but you may skip any questions you do not wish to answer. 4. The answers you give will be kept private to the extent permitted by law.

THE PAPERWORK REDUCTION ACT OF 1995

Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The information requested in this study will be used to document how programs receiving HMRF grant funding operate and describe participant outcomes. The data gathered will allow ACF to better monitor grantee progress and performance. In accordance with the requirements of the Privacy Act of 1974, as amended (5 U.S.C. 552a), ACF/OPRE established system of records titled: 09-80-0361 OPRE Research and Evaluation Project Records, HHS/ACF/OPRE. A Federal Register Notice (80 FR 17893) announced the system.

Construct	Site Visit Interview Topic
Implementation Drivers: Competency Drivers
Staff Recruitment and Selection
Specification of staff skills and abilities and selection criteria	Required staff skills and abilities; pre-requisites for employment Extent to which job descriptions define required skills, abilities, and staff expectations Prioritization of skills and abilities when selecting candidates
Method for recruiting candidates	Process for identifying candidates to fill program positions Sources for advertising open positions Challenges recruiting candidates
Staff responsible for recruitment and selection	Lead person responsible for coordinating staff recruitment and selection Challenges coordinating staff recruitment and selection
Process and protocols for interviewing candidates	Process for identifying interviewers who can assess applicant knowledge, skills, and abilities Interview protocol content, including: Assessment of required skills and capacity to complete job responsibilities Assessment of applicant’s ability to receive feedback professionally and use it for improvement Process for monitoring adherence to the interview protocol Extent to which program uses training, turnover, and fidelity data and exit interviews to assess the effectiveness of staff selection
Training
Focus of content covered during training and regularity of training	Training content, including: History, theory, philosophy, and values of the program Instruction on delivering curricula Opportunities to practice new skills (role plays, teach backs) and receive feedback Timing and frequency of initial and on-going training; availability of booster training sessions
Staff or agencies responsible for ensuring implementers receive training	Lead person responsible for coordinating training Challenges to ensuring regular training occurs Identifying trainers; trainer qualifications
Collection and use of training data	Collecting training assessment data (pre-post tests) Sharing training assessment data with supervisors and coaches to support trainee competency development Soliciting feedback from trainees to improve future training activities
Coaching/Supervision
Development and use of a coaching/supervision plan	Developing staff coaching or supervision plans Content of coaching or supervision plans, including: Coaching/supervision requirements Coaching/supervision frequency Coaching/supervision methods Coaching/supervision plan review
Staff responsible for providing coaching/ supervision	Lead person responsible for coordinating coaching/supervision Selection of coaches/supervisors; required skills and abilities Challenges coordinating coaching/supervision, identifying qualified coaches/supervisors, and providing coaching/supervision
Process for assessing coaching/supervision effectiveness and modifying coaching/supervision plan	Monitoring performance of coaching/supervision recipients (i.e. direct observations, video or audio recordings, feedback from colleagues and other stakeholders) Obtaining feedback on coaching/supervision (i.e. satisfaction surveys of coaching recipients; coaching observations) Providing feedback to coaches/supervisors Using monitoring data and feedback to improve coaching/supervision
Fidelity Assessment
Development and use of plan and protocol for monitoring fidelity of service delivery	Protocol for assessing fidelity to HMRE curricula including: Data sources and methods for assessing fidelity Content of fidelity measures; assessment of content, staff competence to deliver content, and implementation context Frequency of fidelity assessments Use of fidelity data to improve curriculum delivery Staff involved in developing fidelity assessment protocol and measures Reviewing protocol for assessing fidelity
Staff responsible for assessing fidelity	Lead person responsible for coordinating fidelity assessments Selection of staff to conduct fidelity assessments; required skills and abilities Challenges coordinating and conducting fidelity assessments
Agency support for using fidelity protocol	Extent to which the agency supports the use of consistent fidelity measures
Implementation Drivers: Organization Drivers
Decision Support Data Systems
Use of data systems to support and monitor intervention delivery and fidelity, and assess overall program performance	Existing data systems prior to STREAMS evaluation Data elements captured in pre-existing and nFORM systems: Outcome measures Participation data Fidelity measures Cost data
How data system supports decision making	Extent to which data collected are: Reliable (standardized protocols, trained data collectors) Valid (useful indicators of the concepts or practices being assessed) Reported in a timely manner (when/to whom the data are most useful) Built into regular practice routines Extent to which agency staff have access to the data
Staff responsible for managing data system	Lead person responsible for coordinating content, quality, and timeliness of data collection Selection and training of data entry staff Challenges with data collection, data entry, and data system management
Process for using data to support decision making	Staff that receive data reports and their use for program improvement Frequency and content of data reports Usability of data reports Sharing data reports and program improvement plans with key stakeholders
Facilitative Administration
Composition and role of leadership team	Leadership team within agency to guide HMRE program; how identified Role of leadership team
Program leaders actively facilitate use of implementation supports	Extent to which leaders and managers make changes in organizational roles, functions, and structures to facilitate implementation Extent to which leaders and managers make changes to organization policies and procedures to facilitate implementation of HMRE program Extent to which leaders and managers use data to inform decisions and planning
Process for facilitating meetings	Use of meetings within the organization and HMRE program for program management Staff roles and responsibilities during meetings Process for ensuring meeting follow-up is completed and progress continues within designated timelines
Process for receiving and using feedback from staff, participants, and stakeholders	Feedback from staff, stakeholders, and program participants on program management and operations; frequency of feedback How leaders and managers use feedback for program improvement
Systems Intervention
Engagement of larger delivery and funding systems to encourage program sustainability	Frequency and focus of meetings with ACF, other funders, and other service organizations Sharing information about facilitators and barriers to program quality with ACF, other funders, and other service organizations
Engagement of stakeholders and partners to support program	Development and use of communication plans with key stakeholders, including schools, agencies serving at-risk youth, referring organizations, and other partner agencies Challenges with stakeholder communication
Process for identifying partner organizations; role of organizations	Partner roles; criteria and process for selecting partners Challenges in the selection of and work with partners Factors that facilitate the selection of and work with partners
Strategies to obtain buy-in for program among stakeholders	Assessing stakeholder support for the HMRE program Building stakeholder support for HMRE program
Implementation Drivers: Leadership Drivers
Identification of contextual issues related to implementation	Staff responsible for identifying community needs and how HMRE program can address these needs Building community support for HMRE program; community barriers to and facilitators for implementation
Identification of barriers and challenges to implementation and staff involvement	Key challenges to program implementation; how addressed by program staff Support from agency leaders to address implementation barriers, including allocation of needed time and resources Communication process for raising and addressing implementation challenges Decision-making authority to address implementation barriers
Fidelity of Implementation: Consistency
Regularity of offering HMRE curriculum	Frequency, format, length, and intensity of HMRE curricula offered through individual sessions and full curriculum Formats used for delivering curriculum; reasons for offering multiple formats Challenges and successes with using different formats for delivering curriculum Process and frequency of starting new cohorts Challenges with offering curriculum with intended regularity and starting new cohorts
Regularity of content delivery included in HMRE curriculum	Plan for providing all content specified in HMRE curriculum Process for tracking whether facilitator delivers all content Challenges with delivering all content
Availability of ancillary services within the HMRE program	Types and providers of ancillary services available to program participants Frequency of participant engagement in ancillary services Monitoring participant take-up of ancillary services Frequency, mode, and purpose of contact between program staff and other service providers on behalf of the client Challenges connecting participants to and engaging participants in ancillary services; attempted strategies to address challenges and success of these strategies
Need for adaptation of HMRE curriculum or ancillary services; adaptations (if made)	Process for identifying needed adaptations to HMRE curriculum or ancillary services Working with curriculum developers to make adaptations; extent to which curriculum developers are included in the adaptation process Adaptations made to HMRE curriculum and ancillary services; rationale for changes
Lessons and challenges offering HMRE curriculum and content and ancillary services	Challenges to providing HMRE curriculum as designed by curriculum developer; strategies for overcoming challenges Challenges to providing supplemental HMRE content (parenting skills, financial management, conflict resolution, and job and career development) and ancillary services; strategies for overcoming challenges
Fidelity of Implementation: Satisfaction
Buy-in for HMRE program	Alignment between leadership and front-line staff on strategies for improving participant outcomes Views about how HMRE program advances organization’s goals How program components seem to contribute to achieving expected outcomes Gaps in services that may prevent participants from achieving expected outcomes
Enthusiasm for providing HMRE curriculum	Perceived importance of HMRE services to achieving expected outcomes Willingness of agency staff to implement program services
Enthusiasm for providing ancillary services	Perceived importance of ancillary services to achieving expected outcomes Willingness of agency staff to provide or refer participants to ancillary services
Fidelity of Implementation: Quality
Receipt of initial and booster trainings on HMRE curriculum	Receipt of initial and ongoing training and guidance on HMRE curriculum Access to or attendance at initial and ongoing trainings Perceived benefit of initial and booster trainings
Participation in coaching or support systems related to curriculum delivery	Availability of and participation in coaching meetings and other support activities Perceived availability of coaches and supervisors to provide support to program staff Perceived benefit of coaching meetings and other support activities
Knowledge of HMRE curriculum	Identification of core HMRE curriculum components such as topics and activities Ability to articulate program goals and goals of HMRE curriculum Understanding of how HMRE curriculum aligns with agency goals and mission
Knowledge of community resources available to participants requiring additional support	Ability to identify community resources to which participants can be referred
Participant Responsiveness
Enrollment and motivation for enrolling	How participants learn of and come to be in the program Factors that most often make a participant ineligible for the program
Participation rate for curriculum sessions, ancillary services, and entire program	Challenges to participants attending HMRE sessions at expected frequency Challenges to participants receiving ancillary services Challenges to participants receiving any services offered by the program Most common barriers to participation Strategies to support participation and reduce barriers
Average dosage for curriculum sessions, ancillary services, and entire program	Definition of program completion Challenges to participants receiving the intended dose of the HMRE curriculum and other services Strategies to support program completion
Satisfaction with program services	Services participants seem to respond to the most Services participants seem to respond to the least Services that are challenging to get participants to engage in
Knowledge and skill acquisition	Observed changes in participant behavior that indicates acquisition of new knowledge or skills Whether pre-and post-tests indicate acquisition of new knowledge and skills Examples of key skills or knowledge that participants have gained; demonstration of specific skills and information taught in the program
Community Context
HMRE program awareness	Extent to which this HMRE program is known in the community
Availability of other HMRE or similar programming	Whether there are similar services available in the community (i.e. availability of other HMRE or related curricula through other organizations) Whether other community agencies offer similar ancillary programming in the community