Interview Guide for Comparison Sites

Instrument 2

Interview Guide (Treatment and Comparison Sites) – track changes

Updated June 2024

1. INTERVIEW GUIDE FOR PATHWAYS SITES

Introductory script

Thank you for your time and for agreeing to participate in this interview. I’m _______________, and I work for Mathematica, an independent social policy research company.

We are conducting a study for the Office of Planning, Research, and Evaluation (OPRE) in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services. The goal is to understand how [name of implementing site] is delivering Pathways to Success services. We want to understand the services [name of agency] is delivering and the factors that helped or hindered [name of agency]’s success in delivering those services to youth.

I would like to talk to with you about your experience with and perceptions of Pathways. Your responses will be kept private to the extent permitted by law and used only for research purposes. I will not share your comments with anyone other than members of the Mathematica study team. Our team will use your responses, along with other information we collect about Pathways, to help describe what influenced your experience delivering services under Pathways. When we report information from this study to OPRE and ACF, we will not attribute any comments to you or your organization, and no individuals will be quoted by name.

In the future, information from this study may be securely shared with qualified individuals to help learn more about the experiences of young adults who have been in foster care. The information that is shared will be de-identified, meaning it would only include a study ID number and not your name.

All of my questions are open-ended. There are no right or wrong answers. You may choose not to answer any questions you don’t want to answer. You are the expert on your experience with the program, and I would like to learn from you during the discussion.

Your participation in this discussion is voluntary. Being part of this discussion is up to you, and it will not affect your employment. Nothing you share will be shared with your employer.

I would like to record this discussion. We will transcribe the recording to help summarize information from all the interviews we are conducting. No one besides the transcription service and members of the Mathematica study team will have access to or listen to the recording. Recordings and notes will be destroyed after the end of the contract. If you want to say anything that you do not want recorded, please let me know, and I will be glad to pause the recorder.

Do you have any concerns about being part of this interview or to my recording our discussion?

We have many topics to cover. At times, I may need to move the conversation along to be sure we cover everything. The discussion will last no more than one and a half hours, and we will not take any formal breaks. Please feel free to get up at any time if you need to.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0445, and it expires XX-XX-XXXX.

If you would like more information about this study or have any concerns about your privacy related rights, please contact Dr. M.C. Bradley at Mathematica at 855-888-2092 or by email at SYSIL@mathematica-mpr.com.

Do you have any questions before I turn on the recorder?

Warm-Up

To start, would you please describe your role/involvement in Pathways to Success?

[Follow-up]

How long have you been in that role?

How long have you been involved in the Pathways to Success program?

Recruitment and enrollment

[These questions will be asked of leadership, supervisors and navigators]

I’d like to start with questions about how youth get to Pathways to Success.

1. [Round 1] Can you walk me through how you identify youth who are potentially eligible for Pathways? [Round 2] In the past year, have you made any changes to how you identify youth who are potentially eligible for Pathways? If yes, what are those changes?

1. [Round 1] How do you identify youth?

2. What helps to make the identification process work well?

3. What has been challenging about identifying youth (probe on challenges reaching the target population)

   1. How did you address/overcome this challenge?

2. [Round 1] Now can you tell me about your role in identifying potentially eligible youth for risk factors for homelessness? [Round 2] In the past year, have you made any changes to how you identify potentially eligible youth for risk factors for homelessness? If yes, what are those changes?

1. What helps to make the screening process work well?

2. What has been challenging about screening youth for enrollment (probe on challenges enrolling the target population)?

   1. How did you address/overcome this challenge?

3. How have youth responded to the screening assessment?

3. [Round 1] Once you identify eligible youth, what is the enrollment process like? [Round 2] In the past year, have you made any changes to the enrollment process? If yes, what are those changes?

1. [Round 1] What are the steps of the enrollment process?

2. What helps to make the enrollment process work well?

3. What has been challenging about the enrollment process?

   1. How did you address/overcome these challenges?

4. What have been youths’ reactions to the Pathways enrollment process?

1. [Round 1] How do you present youth with the opportunity to enroll in Pathways?

2. What has helped with successful enrollment?

3. Are there points at which youth drop out of the enrollment process? If so, why?

Pathways service components

[These questions will be asked of navigators]

Now I want to talk to you about the components of the Pathways to Success program. First, I will ask about the core components of Pathways to Success. By core components, I mean the Pathways services and supports that are provided to all enrolled youth. Then I will ask about other components that might not be provided to all youth.

5. Overall, how is Pathways going?

6. Please tell me about engaging youth in a coach-like way. How is that going?

1. What does it mean to you to engage youth in a coach-like way?

2. What has helped with engaging youth in a coach-like way?

3. What has been challenging about engaging youth in this way?

   1. How did you address/overcome [each challenge]?

4. How have youth responded to working with Navigators in a coach-like way? [Probe on factors that facilitate and challenge youth in working with navigators in a coach-like way.]

7. Please tell me about helping youth establish goals. How is that going?

1. What has helped you support youth to establish goals?

2. What has been challenging about helping youth establish goals?

   1. How did you address/overcome [each challenge]?

3. How have youth responded to establishing goals? [Probe on factors that facilitate and challenge youth in developing goals.]

8. Next, will you please tell me about how you support youth who are in crisis situations (such as their immediate safety is threatened and/or housing instability)? [Ask for examples of crisis situations that Pathways youth have brought to their attention]

1. What has helped Navigators provide support to youth who are in crisis?

2. What has been challenging about providing support to youth who are in crisis?

   1. How did you address/overcome [each challenge]?

3. How have youth responded to the support that is provided to them when they are in crisis?

9. Next, I have some questions about the assessment tools you may use to gather information about youth for case planning purposes. [Ask for examples of assessment tools, such as the Youth Connections Scale. Navigators may use other evidence-based assessment tools.]

1. What has helped to complete the assessments or use this information to support youth?

2. What has been challenging about completing the assessments or using information from the assessments to support youth?

   1. How did you address/overcome [each challenge]?

3. How have youth responded to the assessments?

Now, I have questions about the other services and supports that might not be provided to all youth.

10. Have you worked with youth to secure and maintain housing? If yes, how is that going? [Ask for examples of working with youth to secure and maintain safe and stable housing.]

1. What has helped to support youth in securing and maintaining housing?

2. What has been challenging in working with you to secure and maintain housing?

   1. How have you addressed/overcome [each challenge]?

3. How have youth responded to your working with them to secure and maintain housing?

11. Have you provided youth with “flexible funds”? If yes, how is that going? [Ask for examples of when they provided funds or resources to youth and the reasons for providing these resources.]

1. What has helped to you to effectively use flex funds?

2. What has been challenging about using flex funds?

   1. How did you address/overcome [each challenge]?

3. How have youth responded to receiving the flex funds or resources paid for with flex funds?

12. Have you referred youth to other services? If yes, how is that going? [Ask for examples of when they referred youth to other services, the services to which they referred youth, and the reasons for the referral.]

1. What has helped make referring youth to other services successful?

2. What has been challenging about referring youth to other services?

   1. How did you address/overcome [each challenge]?

3. How have youth responded to referrals to other services?

13. Have you supported youth to plan or participate in county-led Permanency Roundtables (PRT) or Community Roundtables (CRT)? [Ask for examples of how they have supported youth with Permanency/Community Roundtables.]

1. What has helped youth to plan or participate in roundtables?

2. What has been challenging about roundtables?

   1. How did you address/overcome [each challenge]?

3. How have youth responded to the roundtables?

14. Have you helped youth who are graduating from Pathways build connections with other supports in the community that they can rely on after they graduate from Pathways? [Ask for examples of community supports to which they have helped youth build connections.]

1. What has helped to make these connections and transitions successful?

2. What has been challenging about identifying connections and transitioning youth to these supports?

   1. How did you address/overcome [each challenge]?

3. How have youth responded to your attempts to connect them to other supports in the community?

15. Now, thinking about all the services and supports provided to youth enrolled in Pathways, how do you determine the services that are provided to each youth?

[Ask for examples, and probe on barriers and facilitators.]

16. Out of everything we’ve talked about (for example, housing, financial help, supports during crises), which Pathways services and supports do youth and young adults seem to need most often?

17. Of the services and supports you provide, which are most helpful for youth and young adults enrolled in Pathways?

18. How many youth and young adults from your caseload have graduated from Pathways or are close to graduating? What are the graduation criteria for Pathways?

1. How long, on average, did you work with these youth?

2. How are services different for youth who are close to graduating than those who recently enrolled in Pathways?

3. What skills or tools do you hope youth learn through Pathways and take with them after they graduate from Pathways?

19. How has providing Pathways changed how you engage youth and young adults on your caseload?

1. What, if any, changes have you seen with Pathways over your time working as a Navigator? For example, changes with the services or supports that are provided and how they’re provided?

20. What do you think youth and young adults have going on in their lives that competes for their time and energy to engage in Pathways?

Partnerships that support Pathways

[These questions will be asked of leadership, supervisors and navigators.]

21. [Round 1] Who are the organizations that you partner with to provide Pathways services? [Round 2] In the past year, have the organizations that you partner with to provide Pathways services changed? If yes, how?

1. Which services do the partner organizations provide?

2. How have these partnerships improved your ability to serve Pathways youth?

3. What factors help facilitate a good relationship between you and your partners?

4. Have you experienced any challenges with your partners?

   1. If so, can you describe these challenges?

   2. Can you describe how [each challenge] has been overcome?

Service, resource, and policy context

[These questions will be asked of leadership, supervisors and navigators, unless indicated.]

Next, I’d like to speak with you about the services and resources that are generally available to youth in this area.

22. [Round 1] First, what are some of the main service needs of the youth enrolled in Pathways? [Round 2] In the past year, have the main service needs of the youth enrolled in Pathways changed? [Ask for examples, affordable housing, mental health counseling, access to transportation.]

23. [Round 1] Now that you’ve described the needs of youth, can you describe the extent to which Pathways meets these needs? [Round 2] In the past year, has anything changed about the how Pathways meets the needs of youth? If yes, how?

24. What needs do youth have that are not being met by Pathways?

1. What services and supports could Pathways provide to meet those needs?

2. In your opinion, why is Pathways to Success unable to meet these needs?

3. What are the challenges to providing supports and services to help youth meet those needs?

4. What changes would be needed to meet these needs?

5. Who provides services to meet those needs? And when do youth and young adults access these services most often?

25. [Leadership only] [Round 1] How do federal, state, or local policies affect how your agency serves youth? [Ask for examples.]

6. Do [policies/regulations] make it easier to meet the needs of youth? If so, how?

7. Do [policies/regulations] make it challenging to meet the needs of youth? If so, how?

8. Do [policies/regulations] result in gaps in services for youth? If so, what are the gaps?

Implementation experience

[These questions will be asked of leadership, supervisors and navigators, unless indicated.]

Now I’m going to ask you about your experiences putting Pathways to Success into operation and the factors that have contributed to or inhibited implementation.

26. [Leadership/supervisors only] [Round 1] How did you work with [state-level Pathways leadership] to prepare for Pathways implementation?

1. What went well in working with [state-level Pathways leadership] to prepare for Pathways implementation?

2. What was challenging about working with [state-level Pathways leadership] to prepare to implement Pathways?

27. [Leadership/supervisors only] [Round 1] How do you staff Pathways in [this agency]? [Round 2] In the past year, have you made any changes to how you staff Pathways in [this agency]? If yes, what are the changes?

1. [Round 1] Who is responsible for providing direct services to youth?

1. How many staff provide services to Pathways to Success youth?

2. How many staff work full time? Part-time?

28. [Leadership/supervisors only] [Round 1] Will you tell me about your experiences with recruiting and preparing/training staff (Navigators) to support Pathways? [Round 2] Have you made any changes to how you recruit and train staff (Navigators) to support Pathways?

1. What went well in your efforts to hire and prepare/train Navigators to support Pathways to Success?

2. What was challenging about hiring and preparing/training Navigators to support Pathways to Success?

29. Has there been turnover within your organization that has affected the operation of Pathways?

1. If so, how have you dealt with this turnover?

2. In general, what factors help to retain staff?

3. [If not addressed] What are some of the reasons that staff turnover has occurred?

Program resources

[These questions will be asked of leadership in agencies implementing Pathways]

30. What are the primary resources needed to implement Pathways? [Probe on staff, space, materials, financial resources, etc.]

1. Have you experienced any challenges obtaining the resource needed for implementation?

31. What funding sources are used to provide financial incentives to youth for participation?

Continuous quality improvement

[These questions will be asked of leadership, supervisors and navigators]

32. [Round 1] How, if at all, do you monitor the implementation of Pathways? [Round 2] In the past year, have you made any changes to how you monitor the implementation of Pathways? If yes, what are the changes?

1. Did you identify implementation goals? If so, how?

   1. Who is responsible for setting and monitoring goals?

2. How do you monitor progress toward these goals?

   1. Have you identified benchmarks? If so, what benchmarks have you identified?

   2. How often do you review progress toward goals?

33. Have you made changes to the Pathways model as a result of monitoring implementation?

1. If so, what changes were made?

2. How did you measure change?

3. If so, how has performance changed in light of these changes?

34. What has facilitated your ability to monitor implementation?

35. What challenges have you experienced monitoring implementation?

Wrap-up

Is there anything else you would like to share about Pathways to Success that we have not discussed?

May we contact you in the future to participate in a future product about Pathways that will be made public?

2. Interview Guide for Comparison Sites

Introductory script

Thank you for your time and for agreeing to participate in this interview. I’m _______________, and I work for Mathematica, an independent social policy research company.

We are conducting a study for the Office of Planning, Research, and Evaluation (OPRE) in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services. The goal is to understand how [name of agency] implemented the Chafee Foster Care Program. We want to understand the services the Chafee Foster Care Program is delivering and the factors that helped or hindered [name of agency]’s success in delivering those services to youth.

I would like to talk to with you about your experience with and perceptions of the Chafee Foster Care Program. Your responses will be kept private to the extent permitted by law and used only for research purposes. I will not share your comments with anyone other than members of the Mathematica study team. Our team will use your responses, along with other information we collect about the Chafee Foster Care Program, to help describe the services delivered. When we report information from this study to OPRE and ACF, we will not attribute any comments to you or your organization, and no individuals will be quoted by name.

In the future, information from this study may be securely shared with qualified individuals to help learn more about the experiences of young adults who have been in foster care. The information that is shared will be de-identified, meaning it would only include a study ID number and not your name.

All of my questions are open-ended. There are no right or wrong answers. You may choose not to answer any questions you don’t want to answer. You are the expert on your experience with the program, and I would like to learn from you during the discussion.

Your participation in this discussion is voluntary. Being part of this discussion is up to you, and it will not affect your employment. Nothing you share will be shared with your employer.

I would like to record this discussion. We will transcribe the recording to help summarize information from all the interviews we are conducting. No one besides the transcription service and members of the Mathematica study team will have access to or listen to the recording. Recordings and notes will be destroyed after the end of the contract. If you want to say anything that you do not want recorded, please let me know, and I will be glad to pause the recorder.

Do you have any concerns about being part of this interview or to my recording our discussion?

We have many topics to cover. At times, I may need to move the conversation along to be sure we cover everything. The discussion will last no more than one and a half hours, and we will not take any formal breaks. Please feel free to get up at any time if you need to.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is 0970-0445, and it expires XX-XX-XXXX.

If you would like more information about this study or have any concerns about your privacy related rights, please contact Dr. M.C. Bradley at Mathematica at 855-888-2092 or by email at SYSIL@mathematica-mpr.com.

Do you have any questions before I turn on the recorder?

Warm-Up

To start, would you please describe your role/involvement at [name of agency]?

[Follow-up]

How long have you been in that role?

Design of comparison services

[These questions will be asked of leadership, supervisors and case managers.]

Now I have some questions about the services that you provide to youth, especially through the Chafee Foster Care Program for Successful Transition to Adulthood.

1. [Round 1] Can you walk me through how you enroll youth who receive Chafee Foster Care Program services? [Round 2] In the past year, have you made any changes to how you enroll youth who received Chafee Foster Care Program services? If yes, what are the changes?

1. [Round 1] How do you identify youth?

   1. What does [agency] do to reach out to youth?

   2. What organizations refer youth to [agency]?

   3. How do youth get recruited and enrolled into the program?

2. [Round 1] What is the process for screening youth for eligibility to receive Chafee Foster Care Program services?

   1. Are there specific eligibility criteria youth need to meet in order to receive program services?

2. [Round 1] Can you describe the services offered to youth under the Chafee Foster Care Program? [Round 2] In the past year, have you made any changes to the services offered to youth under the Chafee Foster Care Program? If yes, what are the changes?

[For each service listed below, probe on the following to generate a detailed narrative about the service/support provided]

2. What does the service entail?

3. Who provides the service?

4. How often does each youth receive the service? (i.e., hours)

5. For how long does each youth receive the service?

1. What services are provided to youth by your agency?

   1. Meeting with youth one on one?

   2. Helping youth establish goals?

   3. Helping youth when they are having an emergency?

   4. Helping youth find stable housing?

   5. Providing financial support to youth?

   6. Organizing roundtables or meetings with others to support youth in achieving their goals?

   7. Connecting youth with other services and supports?

   8. Other services?

3. Of the services and supports you provide, which do you think are most helpful for youth and young adults enrolled in Chafee services?

4. What do you think youth and young adults have going on in their lives that competes for their time and energy to engage in Chafee services?

5. [Round 1] How do you determine the services that are provided to each youth? [Round 2] In the past year, have you made any changes to how you determine the services that are provided to each youth? If yes, what are the changes?

1. [Round 1] What is the process for assessing youth needs?

2. [Ask for examples, and probe on barriers and facilitators.]

6. [Round 1] How do you determine which staff provide services to youth? [Round 2] In the past year, have you made any changes to how you determine which staff provide services to youth? If yes, what are the changes?

1. [Round 1] What criteria do you use to assign youth to staff?

2. [Round 1] Do certain staff only provide certain services?

   1. If so, which staff provide which services?

   2. If so, how do you determine which staff provide which services?

3. [Round 1] What is the average caseload for staff?

7. [Round 1] Can you describe the process for referring youth to other services? [Ask for examples.] [Round 2] In the past year, have you made any changes to the process for referring youth to other services?

1. [Round 1] How do you identify the need to refer youth to other services?

2. [Round 1] How do connect youth with other services (youth’s initiative, referral, warm hand-off, etc.)?

3. [Round 1] To what extent do you communicate with external service providers about youth’s progress?

8. Who are the organizations that you partner with to provide services?

1. How long have these partnerships been in place?

2. Which services do the partner organizations provide?

Service, resource, and policy context

[These questions will be asked of leadership, supervisors and case managers.]

Next, I’d like to speak with you about the services and resources that are generally available to youth in this area, especially youth and young adults transitioning out of foster care.

9. [Round 1] First, what are some of the main service needs of the youth and young adults you serve who are transitioning out of foster care (for example, affordable housing, mental health counseling, access to transportation)? [Round 2] In the past year, have the main services needs of youth and young adults who are transitioning out of foster care changed? If yes, how?

10. [Round 1] Now that you’ve described the needs of youth, can you describe the extent to which you believe your agency and other organizations in your area are able to meet these needs? [Round 2] In the past year, has anything changed about the how your agency or other services in the area meet the needs of youth? If yes, how?

11. What are the needs that youth have that are not being met, if any?

1. What services, if any, are needed to fill those gaps?

2. What are the challenges to providing those services?

3. Who provides services to meet those needs? And when do youth and young adults access these services most often?

12. May we contact you in the future to participate in a future product about Chafee services that will be made public?

[These questions will be asked of leadership in agencies delivering comparison services]

Finally, I’d like to talk with you about the key child welfare policies and regulations that affect youth in your area.

13. [Round 1] How do federal, state, or local policies affect how your agency serves youth who are transitioning out of foster care? [Ask for examples.]

1. Do [policies/regulations] make it easier to meet the needs of youth? If so, how?

2. Do [policies/regulations] make it challenging to meet the needs of youth? If so, how?

3. Do [policies/regulations] result in gaps in services for youth? If so, what are the resulting gaps?