Appendix L. Instrument 8 Semi-structured program discussion guide

Appendix L. Instrument 8 Semi-structured
program discussion guide

Next Generation of Enhanced Employment Strategies Project

Master semi-structured discussion guide

Introduction & Consent

Thank you so much for meeting with me today. My name is __________ and I am with an independent research organization called Mathematica. We are conducting the Next Generation of Enhanced Employment Strategies (NextGen) Project for the Office of Planning, Research, and Evaluation (OPRE) within the U.S. Department of Health and Human Services, [FOR SITES FUNDED BY SSA: in partnership with the Social Security Administration]. The study will help the government learn more about how employment programs serving people facing complex challenges can help them secure a pathway toward economic independence.

For this study, we are interested in talking with program staff at all levels to learn about the services you provide to participants, any challenges you have encountered, and lessons you have learned. Your participation in this study is voluntary. Our conversation will take about [90 minutes if program leadership; 60 minutes if supervisor or key partner; 45 minutes if program staff].

According to the Paperwork Reduction Act, a government agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-XXXX and the expiration date is XX/XX/XXXX.

We will keep all of your individual responses completely private and use them only for research purposes. We will not share them with anyone outside our research team. We will not share them with your supervisors or any other program staff. We will combine responses for reporting purposes, and we will never report names or identify any responses with a particular person. You also may choose not to answer any question you do not want to answer.

I would like to record our conversation so I don’t miss anything. No one will listen to the recording except for people on the study team and the person who types it up. Is it okay with you if I record this conversation? If you want me to turn the recorder off for any reason or at any time, just say so. [INTERVIEWER: TURN THE RECORDER ON]

Okay, I have now turned on the recorder. Now that I have the recorder on, I need to ask you again, is it okay if I record this conversation? [Interviewer: Get verbal consent to tape on the tape recording.]

[Interviewer: State your name, the respondent’s first name only, the location, and the date.]

Public Burden Statement

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is xxxx-xxxx. The time required to complete this information collection is estimated to average 45 minutes for program staff, 60 minutes for supervisors and key partners, and 90 minutes for program leadership including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate to XXX. OMB expiration date xx/xx/xxxx.

Discussion questions by respondent type
	Respondent type
	Leadership	Supervisors	Front line staff	Key partners
A. Respondent background	X	X	X	X
B. Economic context	X			X
C. Structure and staffing of the intervention	X			X
D. Structure and staffing of the social enterprise	X
E. Intervention development and target population	X			X (as applicable)
F. Services provided by the lead organization		X	X	X
G. Services provided by partners		X	X	X
H. Engagement with employers	X	X	X
I. Intervention participants and counterfactual		X	X	X
J. Intervention monitoring and oversight	X	X		X
K. Sustainability and lessons learned	X	X	X (as applicable)	X

Interviewers: Bullet points indicate topics on which data must be collected. Italicized text indicates examples of questions that could be posed to respondent and probes to help elicit information. It is important to discuss each topic; the specific questions to ask will depend on the circumstances. In addition, many questions can be prepopulated with information we know from previous discussions with the program. For information that we are not sure is accurate or is likely to change over time, prepopulate the question and confirm it with the respondent.

A. Respondent background

1. Thanks for agreeing to speak with me today. To start off, tell me a little about your role and background with your organization.

• Job title and current job responsibilities

• Tenure with the organization and in current position

B. Economic context

1. I’d like to hear a little more about your local economy, and how it affects your organization.

• Main economic challenges. What do you think are the main economic challenges in this area? E.g., unemployment rate, large-scale layoffs, shifts in industry, lack of talent to fill available jobs, lack of education and training opportunities, poor public education system

C. Structure and staffing of the intervention

Interviewer: ask this module of all interventions, unless it is a social enterprise with no other component. For instance, if the intervention is a housing assistance program that also operates a social enterprise, ask questions in sections C and D. If it is purely a stand-alone business, skip section C and ask section D.

1. Tell me a little about this organization.

• Mission and goal of the organization, target population

• How intervention fits into the organization

• Hierarchy or other organizational structure [Interviewer: Collect an organizational chart for the organization and intervention.]

• Number of employees organization-wide

• Government agency, nonprofit, or for-profit organization? If for-profit, privately owned or publicly traded?

• Funding sources. Probes: government contracts/grants, other public funds, philanthropy, combination, other

2. Let’s talk now about the structure and staffing of the intervention specifically.

• Location(s) where services are delivered

• Intervention staffing

  • Number of staff

  • Roles and hierarchy within staff (managers, front-line, etc.)

  • Staff qualifications. What are the key qualifications needed and why?

  • Staff training. Training topics, duration, intensity, who is trained and when.

  • Time spent on intervention versus other activities. About what fraction of time do managers, supervisors, and frontline staff spend on the intervention versus other non-intervention duties? Does time spent vary over the course of the intervention, and if so, how?

• Volunteers. Number, responsibilities, approximate hours spent per week or month, training needs

D. Structure and staffing of the social enterprise

Interviewer: only ask these if the intervention is or includes a social enterprise.

1. I’d like to learn a little more about your organization and mission.

• Mission and goals of social enterprise, target population

• Hierarchy or other organizational structure of social enterprise [Interviewer: Collect an organizational chart for the social enterprise.]

• For-profit or nonprofit? If for-profit, privately owned or publicly traded?

• Line of business. What does the company make, produce, or sell? What services does it provide? Who are your customers? Do you receive government contracts? If so, extent to which social mission helps win those contracts.

• Market for product or service. How is the market for your product or service compared with last year? Growing, declining, about the same?

• Revenue. What was it last year?

• Locations. How many? Where located?

• Employees. How many?

• Funding sources. How does the social enterprise cover its costs? Does the revenue generated from sales of products and services cover costs? If not, how much does it cover? What are the other sources of revenue (e.g. government grants, public funds, philanthropy, venture capital, other loans)?

• Profitability and sustainability. What are some of the challenges to maintaining sustainability of the social enterprise?

2. Let’s talk in a little more detail about the staffing of the social enterprise—that is, administrative staff who are not “participants/employees” from the target population, and volunteers.

• Number of staff

• Roles

• Staff qualifications. What are the key qualifications needed and why?

• Staff training. Training topics, duration, intensity, who is trained and when.

• Time spent on social enterprise versus other activities. About what fraction of time do managers and supervisors spend on the social enterprise versus other duties? What else do they work on?

• Volunteers. Number, responsibilities, approximate hours spent per week or month, training needs

E. Intervention development and target population

Interviewer: this module applies to all interventions, including social enterprises.

1. What populations are targeted for this intervention and why?

• Characteristics. Probes: age, gender, income, disability status

• Challenges to employment. What are the most common challenges or barriers to employment the target population faces?

• [If targeting person with disabilities:] Disability types. What types of disabilities are most common?

2. Are participants often co-enrolled in other programs (such as TANF, WIOA, and VR)?

3. Let’s think back now to when this intervention began to be developed.

• Motivation. What motivated your organization to develop/become involved with this intervention? Probes: economic or political factors. Was the intervention modeled after an existing program or based on experiences of other programs/agencies/partners?

• Intervention developers and roles. Who led the development of the intervention? Who else contributed? Probe for details about amount and type of input provided in concept development and planning, and who was involved; what worked well in designing the intervention? What were the challenges?

• Evidence base and technical assistance used. Did you use evidence-based practices to develop the model? Did outside experts or consultants help shape the model?

• Intervention goals. What were the initial goals of the intervention? Have they changed over time? What led to or informed these changes?

• Timeline. About how long elapsed from the time you (collectively) decided to implement the intervention until you began delivering services? Any deviations from the planned timeline?

4. What lessons did you learn during intervention development? (We’ll talk more later about lessons learned from its implementation.) What worked well? What did not work well? What did you change in response to challenges? Did it work?

5. What were sources of financial support for intervention development? Probe for amount and types of financial or in-kind contributions (e.g., volunteers, equipment), from whom

6. What factors do you believe helped your organization successfully roll out the intervention and serve clients? Was there a champion(s) within the organization who was critical to making implementation successful?

7. What challenges did you face in implementing the intervention?

F. Services provided by the lead organization

Interviewer: This module applies to all interventions, including social enterprises. The information covered by this section in particular should be well known in advance of the descriptive study site visit from the formative evaluation activities conducted and technical assistance provided in preparation for the evaluation. Walk respondents quickly through details and confirm.

1. [If intervention is or includes a social enterprise:] Let’s talk more about the social enterprise.

• Recruitment, outreach, engagement activities. Describe in detail the approaches to recruiting employees from the target population, both before the study and after the study began.

• Referral sources. What are the most frequent sources of referrals of employees from the target population? Before and after the study began.

• Screening/hiring process. What is the screening/hiring process? What qualifications are you looking for? For what reasons do you screen people out?

• Work activities/roles. What kinds of job responsibilities do employees from the target population typically have? Are the jobs you offer transitional? That is, do you limit the amount of time that people can work at your organization?

• Orientation. Is there an orientation for new employees?

• Training: What training do you provide? Who provides it? Who receives it? When? For how long?

• Compensation: How much do you pay your employees? Do you provide fringe benefits? What are they?

• Support services: Do you offer any benefits or provide any services to employees in the target population, such as transportation, housing assistance? Who provides them, what is usual amount?

• Accommodations: What accommodations do you provide to employees with disabilities?

• Retention. Maximum tenure at enterprise, average length of tenure in enterprise, how often do you terminate staff and for what reason? How often do staff quit and what are some common reasons?

2. [Interventions that are not social enterprises:] Let’s talk more about the specific services [intervention name] provides to participants.

• Recruitment, outreach, participant engagement activities. Describe in detail the approaches to outreach both before the study and after the study began.

• Referral sources. What are the most frequent sources of referrals to the intervention? Before and after the study began.

• Intake/eligibility process. Probes: eligibility criteria, assessments conducted as part of intake, duration.

• Orientation. Probes: is there an orientation? What does it cover? When does it occur (e.g., before or after intake)?

• Assessment. Probes: are participants assessed after enrolling in the program? What are they assessed on (e.g., skills and interests, education level, challenges to employment)? What assessment tools are used? How is assessment information used?

• Service details. [A list of services will be pre-populated based on our understanding of the intervention. For each service, confirm the following information:]

  • Participant flow through services. Is there a standard set and sequence of services, or are services tailored to each participant’s needs? If tailored, how is that determined? Probe for extent to which participants can advance through different services (e.g., start with basic workshops, progress to advanced).

  • Details of each service. Probes: length and schedule (e.g., total hours or weeks/months), content of services, which staff deliver services/supervise participants.

  • Dosage. Probes: expectations around attendance, attaining skills, etc.

  • Number served. Actual versus intended participation. How many served at once and over a year.

  • Monitoring. Probes: frequency and mode of monitoring, consequences for not attending, whether intervention staff communicate with the lead organization about participant performance.

  • Completion and next steps. Probes: how completion defined, what typically happens to participants after they leave the program (e.g., are they placed in a job), follow-up contact (whether happens, type, mode, frequency).

3. [If provide case management:] Let’s talk specifically about case management services. That is, which staff help intervention participants navigate services and provide referrals and supports?

• Case managers. Who provides case management?

• Participant-case manager interactions. How often do case managers typically interact with participants? Meet with them in person? By telephone, text, or email?

• How frequently does case manager meet with other staff who are working with participant? Who do they meet? What is discussed?

• How frequently does case manager meet with employers? What do they discuss?

• Typical case manager caseload. How large? Is it manageable?

• Tools or model. Describe any tools or models you use (e.g., coaching, clinical case management, strength-based case management).

3. [If provide work experiences:] Let’s talk in a little more detail about work experiences [intervention name] offered to participants.

• Type: (1) internships or externships (paid or unpaid), (2) job shadowing, (3) apprenticeships, (4) job tours/mock interviews, (5) on-the-job training/work-based learning. Collect the following information (for each, if more than one):

• Screening. Do you screen participants or assess them to determine what work experiences they are a good fit for? What are the criteria?

• Pay. Do you pay the participant an hourly wage or give them a stipend? Do you provide the employer or participant a subsidy? How much?

• Educational/formal training: Is there a formal educational or occupational training component? What is it? What must participants do, how often?

3. [If provide education or training:] Let’s talk in more detail about the education or training offered.

• Type: (1) instructor-led training, (2) self-directed, (3) other. Collect the following information (for each, if more than one):

• Screening. Do you screen participants or assess them to determine what education or training they are a good fit for? What are the criteria?

• Pay. Do you pay the participant or give them a stipend? Do you provide the employer or participant a subsidy? How much?

• Mode. In person (one-on-one or group), online, text-based.

• Format. Lecture, contextualized learning (for example, courses that integrate basic skills and occupational skills instruction), hands-on, group study.

• Topic. Work-readiness, occupational skills, literacy/high school equivalency.

• Completion and credentials. What is the average completion rate? Do participants obtain certificates or credentials? What are the requirements (e.g., certification test)? How many participants on average receive certificates or credentials? What types of employers recognize these certificates or credentials or require them for hiring?

3. Are there any other services we haven’t discussed yet? If yes, revisit the bullets in F.2 to probe about service details.

4. Where do you typically refer participants and for what services?

• Frequency of referrals?

• What services?

• [If applicable:] Describe the collaboration and communication between the intervention and referral agencies about individual participants. Probes: How is participant-specific information shared, who coordinates participant services/care, do staff track follow-up care across organizations?

3. What support services are available to assist participants in overcoming challenges to program participation (and possibly employment)? Probe for housing, medical care, transportation, child care, payment for drivers’ licenses.

• Type and amount, how paid (e.g., directly to participant or to service provider)

3. Does [intervention name] offer in-kind or cash payments? Probes: what type are they (e.g., gas cards, diapers, bus passes, food)? How much? What do participants need to do to receive them? Are they effective? What evidence do you have?

4. How have service offerings evolved over time? Probes: Add any? Subtract any? Why?

5. Have you faced any challenges to implementing any of the services?

6. Which services do you think are most beneficial to participants? Why?

7. Are there services your clients need that you don’t currently provide? If yes, why don’t you provide these services? Are there partners who can provide these services? Do you typically refer participants to these partners for these services?

G. Services provided by partners

Interviewer: this module applies to all interventions that involve partners.

1. Describe your partnerships with other agencies, organizations, and employers involved in delivering the intervention or providing work experiences.

• Characteristics. Partner names, roles, locations.

• Services provided. What services are partners responsible for providing?

• Funding. How are the services funded?

• History. How and why were these partners selected? Was this partnership developed specifically for the intervention?

• Agreements. Formal or informal partnerships? If formal, what kind of agreement do you have (MOU, DUA, etc.)?

• Collaboration. How often do you meet? Who attends the meetings? What are the agenda topics?

• Changes over time. How has the partnership changed over time? Why?

2. Describe at a high level how you communicate with partners. Probes: Who is primarily involved, types and reasons for communication, frequency of communication, method (regular phone calls, meetings).

3. What are some of the benefits of these partnerships? What has worked well?

4. What have been some challenges of maintaining these partnerships? Probes: communication, collaboration, siloed decision-making, funding, inadequate service quality.

5. What are some lessons learned from working with your partners for this intervention?

H. Engagement with employers

Interviewer: This module applies to all interventions except those that are solely social enterprises.

1. I’d like to shift now to talk about employer involvement with [intervention name]—other than what we have already talked about.

• Roles. What roles, if any, do employers play in [intervention name], other than what we have already discussed? Hire participants? Refer participants? No involvement?

• Recruiting employers. What types of employers do you reach out to? How do you identify appropriate employers to reach out to? What is the message that you give to them? Which messages work? Are there messages that you have found do not work? What proportion of employers you reach out to are engaged?

• Types and diversity of employers. About how many different employers does [intervention name] work with? In mostly the same industries or a range?

• Successes and challenges. What have been the successes in engaging employers in [intervention name] and keeping them engaged? Engaging them at all? Providing supports for employed participants? What strategies have you found effective? What have been the challenges?

I. Intervention participants and counterfactual

Interviewer: This module applies to all interventions, including social enterprises.

1. Let’s shift now to talking in more depth about the participants. Describe a typical background of someone who comes here.

2. Thinking about the participants who come here…

• Participant strengths. Probe for some of the strengths of intervention participants, such as motivation, work ethic, desire for self-improvement. Which participants do you think are most successful here?

• Participant challenges. What challenges have you observed participants often face to being successful while here? Probe for needed work accommodations for disability and details. Which participants do you think are least successful?

• Solutions to participant challenges. What solutions, if any, does your intervention provide to address these challenges? Probe for details about how, cost, who pays.

• Reasons for non-completion. Probes: what are the common reasons for quitting or being dismissed?

3. Thinking in terms of the broader community, what other programs and service providers in the area serve similar populations to the ones you are serving?

• Probes: Names of organizations, what services they provide?

4. What supports exist in the community that can help meet the needs of the target population? For example, transportation, food banks, child care, clothing closets.

• Probes: Names of organizations, what services they provide?

• What are some challenges the target population faces in accessing community services and supports outside of the intervention?

J. Intervention monitoring and oversight

Interviewer: This module applies to all interventions.

1. [If not solely social enterprise:] Describe the performance targets/outcomes the intervention uses to measure progress toward its goals and the processes in place to measure intervention performance.

• Performance requirements. What requirements are you subject to? By whom?

• Key outcomes. What outcomes? How measured? How often measured and reviewed? Whether and how the information is used to inform program decisions or changes?

• Data systems. What systems? Who has access? What information is collected? How is the information used and how often?

• Partners. Is performance of partners tied to funding? If so, how is their performance monitored? Do they generally meet their performance targets? What happens if they don’t meet their targets?

• Staff who track and report. Intervention staff, partner staff?

2. [If social enterprise:] What financials or other metrics do you track on an ongoing basis? How do you use that information in running the business?

3. To what extent has the intervention or social enterprise met its goals? How do you know? Probes: What aspects of the intervention do you believe are most responsible for key outcomes? If haven’t met goals, what were the reasons and barriers? If have met goals, what helped them get there?

4. [If applicable:] To what extent is the intervention operating with fidelity to the original model? How do you know? Have you made any adaptations to the original model? If so, what adaptations have you made and why did you make them? Have the adaptations worked? Why or why not?

K. Sustainability and lessons learned

Interviewer: This module applies to all interventions, including social enterprises.

1. [If not solely social enterprise:] Are the intervention’s costs generally consistent over time, or do certain costs increase or decrease over time? Why?

• Probes: Changes based on the number of participants served, seasonality, fluctuating costs of inputs to services?

2. Earlier you mentioned funding sources were [fill in from earlier responses]. Has that been consistent over time?

• Probes: If not consistent, why not? And how has it affected intervention operations (e.g., type/number of participants served, intensity of services, length of services)?

• If additional funds were obtained, how would those be used?

3. [If intervention received Section 1110 funds:] As part of the evaluation, your program/social enterprise received some funds to expand service provision. How might you continue to fund these services once the evaluation ends?

4. What have been some of the key successes, challenges, and lessons learned from implementing this intervention/running this social enterprise? Let’s start with successes:

• Probes: Organizational, leadership, staff quality, staff turnover, recruitment, services, partnerships, funding, data, performance requirements, engaging employers.

5. [If a social enterprise that uses transitional employment:] What are the benefits and challenges of limiting the amount of time people can work for your organization?

6. Do you envision any challenges to continuing current operations (for example, increasing costs, needing to leverage new funding sources, staffing constraints, or local conditions)?

• Is your organization taking any action in anticipation of these challenges?

5. What are the three most important lessons you have learned while implementing and operating the intervention?

6. What advice would you give other organizations that want to do something like this?

Thank you! This conversation has been very helpful and we appreciate your assistance with our study. If you have any questions after today please feel free to contact me [provide a business card].