NextGen Attachment B Program Admin Guide.Final.clean

NEXT GENERATION OF ENHANCED

EMPLOYMENT STRATEGIES (NEXTGEN)

ATTACHMENT B

Discussion Guide for Program Administrator Interviews

NextGen Discussion Guide for Program Administrator Interviews

Introduction

I am calling from Mathematica Policy Research to ask you information about the program your organization administers. Your participation in this information-gathering interview is voluntary and, should you wish to participate, the information you provide will be kept private within the project team. This interview should take about an hour.

On behalf of the Office of Planning, Research, and Evaluation in the federal Administration for Children and Families, we are conducting a study of innovative, promising employment interventions for low-income individuals facing complex challenges to employment. “Complex challenges to employment” is a broad concept that could encompass, for example, physical and mental health conditions, substance misuse, a criminal history, or limited work skills and experience. The study is called the Next Generation of Enhanced Employment Strategies Project (NextGen Project). It will help us learn more about how employment programs can help low-income people move toward economic independence.

If asked: The Office of Planning, Research, and Evaluation is also conducting a companion study, Building Evidence on Employment Strategies for Low-Income Families Project (BEES). The projects are both designed to test innovative employment interventions that are rooted in the best available research and aim to help low-income people move toward economic independence. The projects are coordinating closely. Depending on the types of interventions we find that warrant evaluation, the two projects may focus on different interventions or different populations.

If a program that might qualify for Section 1110 funds: Additionally, our project [and the BEES project if above noted] is working closely with the Social Security Administration (SSA) to incorporate a focus on employment-related early interventions for individuals with current or foreseeable disabilities who have limited work history and are potential applicants for Supplemental Security Income (SSI).

We plan to conduct a rigorous, random assignment study of about nine interventions. The study will examine the implementation of the intervention and estimate its impacts on employment and other outcomes. We will also study the program’s costs. We have identified [PROGRAM NAME] as operating an intervention to consider for inclusion in the evaluation. Right now, we are collecting information on interventions that might be a good fit for the study; we will select programs to invite to participate in the study later. We would like to ask you some questions about your program and intervention to determine whether they are well suited to be included in the evaluation. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. This information collection has been approved under OMB information collection request 0970-0356, which expires on June 30, 2021.

Participating in an evaluation has benefits to you, other programs like yours, and the clients you all serve. We have some funds available to work with some programs to refine their operations to get the interventions working as well as possible before evaluation. All programs participating in the evaluation will receive some funding to support their participation in a rigorous evaluation. For interventions focused on people with current or foreseeable disabilities who are potential applicants for SSI, there may be additional funds that can be used to bolster the intervention—for example, by adding an employment service component to an existing intervention or by increasing the services already being provided.

Programs that participate in random assignment studies can benefit by learning about the effectiveness of their interventions and how to improve them. They will build their capacity to collect and use data to inform their operations. They also gain an objective assessment of their intervention that policymakers and funders will trust. If the study shows the intervention is effective, it might be easier to pursue additional funding.

Before we begin, do you have any questions about the project?

Program and intervention overview and objectives

1. Please describe [INTERVENTION NAME].

2. What is the entity that administers the intervention? Who are key partners? What is the setting? Probe for TANF, SNAP, Vocational Rehabilitation, workforce system, behavioral health treatment setting, mental or physical health treatment setting, employer.

3. Do you work with an employer or employers? If yes, which ones?

4. What are the objectives of the intervention?

5. What is the target population?

6. How do you expect the intervention to affect client outcomes?

7. What would you count as a success for this intervention?

Intake and services

8. How do participants usually find out about the intervention? Probe for referral from human service agency, court, health treatment center, etc.

9. Walk me through what happens when someone first expresses interest in participating. How are they determined eligible? How are they enrolled?

10. What are the main services your program offers to participants? Probe for:

• Assessments

• Education

• Occupational skills training

• Life or soft skills training

• Job search assistance

• Work experience

• Case management

• Coaching

• Benefits counseling

• Service coordination

• Support services

• Referrals

• Financial support or incentives (probe for how much)

• Substance abuse treatment

• Mental, emotional, or physical health treatment

• Expungement or legal assistance

11. How do you decide which services are provided to each participant, and in what order?

12. What are the most common services provided to participants?

13. [If program includes case management or coaching] How often do the participants meet with their case manager or coach? How long on average are the meetings? Where do the meetings take place?

14. Do you subsidize employment? How much?

15. Do you support an employee while working? How?

16. How long does the program last? On average, how long do participants participate?

17. Are there other places in your community where participants could receive similar services? What are some of those places? How are they similar to or different from your program?

Implementation

18. How long has your organization been implementing [INTERVENTION NAME]?

19. How is the intervention funded?

20. Who are the staff that work directly with participants?

• What are their backgrounds and qualifications?

• What type of training do they receive?

• How many participants does each staff member work with? What is their caseload?

21. What have been the program’s successes?

22. What have been the primary challenges? How were those challenges addressed?

Program size

23. How many participants do you currently serve in a year?

24. About how many new participants join the program each month?

25. Of those who start the intervention, about what fraction complete it?

26. Is the intervention at capacity, or could you serve more people? How many more? What would it take to increase recruitment for and enrollment in the intervention?

Future plans

27. Do you expect to make changes to the intervention? What types of changes and when?

28. Do you have funding to expand the intervention? To whom? When? Where?

29. Do you anticipate any challenges to expanding the intervention?

Evaluation

30. Has [INTERVENTION NAME] ever been evaluated, as far as you know?

• If so, who conducted the evaluation?

• Did it involve an evaluation of its effectiveness?

• What outcomes were measured and when?

• What were the findings?

• Is there a report available that describes the evaluation’s methods and findings?

31. Do you currently collect data on services the participants receive, and/or participants’ outcomes? If so, how do you use those data?

32. [If intervention/program seems promising and an evaluation seems possible] As I mentioned at the start of the interview, participating in an evaluation can greatly benefit a program. You will get feedback on whether your intervention worked that will help you improve it. Many staff say they learn a lot about their program and how it can be improved just by participating in a study. By helping us learn about your intervention, you will also be helping other programs and the clients they serve. Would you consider participating in an evaluation?

33. If we were to conduct an evaluation, we would need to randomly assign intervention applicants to either receive intervention services or not. Suggest to the respondent how we would see this working. Would your program be willing to conduct this type of evaluation? What are your concerns?

Next Steps

Thank you very much for your time today. We are currently conducting interviews with several programs to determine which ones are candidates for evaluations in the NextGen Project. We will follow up with you in a few weeks to discuss further possibilities.