DCI-2: Topic Guide on Program Implementation

O MB Control No.:xxxx-xxxx

Expiration Date: xx/xx/xxxx

HEALTHY MARRIAGE/

RESPONSIBLE FATHERHOOD PROGRAM

INSTRUMENT DCI-2

Grantee Staff Topic Guide on Implementation

FaMLE Cross-Site Project research staff will use this instrument to guide semi-structured discussions with a subset of HMRF program staff during site visits. Respondents will participate in only one of the interviews shown in the topic guide, depending on their role in the organization (supervisor, case worker, facilitator, etc). The information gathered will provide information for ACF and other programs to use when designing future programs.

THE PAPERWORK REDUCTION ACT OF 1995

Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The information requested in this survey will be used to document how programs receiving HMRF grant funding operate and describe participant outcomes. The data gathered will allow ACF to better monitor grantee progress and performance. In accordance with the requirements of the Privacy Act of 1974, as amended (5 U.S.C. 552a), ACF/OPRE established system of records titled: 09-80-0361 OPRE Research and Evaluation Project Records, HHS/ACF/OPRE. A Federal Register Notice (80 FR 17893) announced the system.

Famle cross-site Project—Implementation Study

ORganizational leadership Interview Guide

Interviewer instructions: Before beginning the interview, tailor this guide to focus on the additional details we need about the grantee. In particular, be sure to probe for details on populations (such as youth aging out of foster care) and policy areas (such as connections to child support and intimate partner violence services) that are a high priority interest for OFA.

Thank you for taking the time to speak with us today. Mathematica Policy Research is conducting a study for the U.S. Department of Health and Human Services, called the Fatherhood and Marriage Local Evaluation and Cross-Site project, or the FaMLE Cross-Site Project. As part of this effort, we are gathering information to develop an in-depth understanding of how organizations like yours design, implement, and operate Responsible Fatherhood/Healthy Marriage programs. As part of this study, Mathematica is interviewing staff at all levels in a selected number of organizations that provide RF/HM programs.

We expect the current interview to take one hour and your participation is voluntary. Any information you share will be kept private. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB # for this collection is 0970-XXXX and the expiration date is XX/XX/XXXX.

During this interview, we would like to hear about [GRANTEE ORGANIZATION] and how the [RF/HM PROGRAM] fits into this larger organization. We’d also like to discuss your opinions about program successes and things other organizations should know if they want to develop a similar program. We have a large number of topics to cover and, at times, we may need to move our conversation along to ensure we get to all of them. We will use the information you share with us to write a summary of what we have learned about your program, which we will publish in a future report about your program and others, but we will not attribute any of your comments to you in our reports. We would like to record our discussion today, however, only members of our team will have access to this recording and we will destroy it after we complete our analysis. Do we have permission to record this conversation? [OBTAIN PERMISSION AND START RECORDING IF PERMISSION GRANTED.]

A. Respondent Background and Responsibilities

Let’s start by discussing your background and responsibilities.

1. What is your official job title?

2. How long have you been working at [GRANTEE ORGANIZATION] in this position? When did your involvement in the [RF/HM PROGRAM] begin?

3. What types of job experiences did you have before becoming a [JOB TITLE] for the [RF/HM PROGRAM]? What did you do immediately prior to assuming this position?

4. Describe for me your overall role/involvement in [RF/HM PROGRAM]?

B. Participant characteristics; Delivery (including customization), dosage, quality of program content; Integration of organizational partners

5. What are the greatest needs of the fathers/couples served?

6. What are the greatest strengths of the fathers/couples served?

7. Please describe the core services provided by [RF/HM PROGRAM] and how these services have changed since the program started in [FILL IN].

   1. Parenting services, including curriculum and individualized services

   2. Relationship services, including curriculum and individualized services

   3. Employment services, including curriculum and individualized services

   4. Other core services, including curriculum and individualized services

   5. What changes did you intentionally make to these programs?

   6. What changes were unintentional, and why did these changes occur?

8. How does [RF/HM PROGRAM] work with partner organizations in the community to provide the program?

   1. Which partner organizations contribute to the program?

   2. How does each organization contribute to the program?

   3. Were there preexisting relationships with any of these partners?

   4. How often do you have contact with these partners, and what do you typically communicate about?

   5. How is each partner organization beneficial to the program?

   6. Which organizations contribute referrals to the program? Have there been changes in referral sources? When and why?

C. Staff recruitment, characteristics, training, and retention

9. Please describe the process you use to identify, interview, and hire new program staff. Especially, what level of experience are you looking for when hiring, and have you been able to find staff with these qualifications?

10. How, if at all, have the roles and responsibilities of the program staff changed over time? How has the management structure changed over time, if at all?

If changes occurred:

1. When did these changes occur?

2. What led to these changes?

11. In what ways do staff believe that each of these services are valuable to fathers/couples?

    1. Parenting services

    2. Healthy marriage/ relationship services

    3. Economic stability services

12. Which services or strategies, if any, are staff more skeptical are helping fathers/couples achieve their intended outcomes?

13. Which services, if any, are staff uncomfortable with or resistant to providing?

14. In your view, to what extent are staff satisfied that the services delivered by the program are the “right mix” in terms of range, type and intensity to help fathers/couples achieve their intended outcomes?

D. Adherence/modifications to implementation plans/logic model; Attainment of program targets

I’d like to switch over to talking about changes you’ve made to the implementation plans and logic model to the [RF/HM program], and how you monitor the performance of this program.

15. Next, please describe the program’s definition of high-quality service delivery and how quality is monitored?

16. To what extent is consistent delivery of services by all staff expected and monitored?

17. What improvement in outcomes is expected for program participants and how is participant progress towards these outcomes monitored?

18. [Interviewer: Refer to notes on attainment of program targets.] According to the progress reports you’ve submitted, you are [ON TRACK/NOT ON TRACK] to meet your program targets. Going forward, do you think you will meet, exceed, or fall short of any planned targets?

    1. [IF EXCEED] What factors do you think will lead to your program exceeding its targets?

    2. [IF FALL SHORT] What factors do you think will lead to your program falling short of its targets? How do you think this has affected your program outcomes?

[PROBE: Factors may include staffing and turnover, service delivery supports, and organizational partnerships.]

19. What is the process for developing or revising policies and procedures for service delivery? What about the program logic model?

    1. What changes have you made, and why have you made changes?

How have these changes affected your program outputs and outcomes?

2. Have you considered alternatives to policies and procedures that you did not pursue? What were these, and why did you decide not to pursue these ideas?

E. Implementation drivers and successes; Implementation challenges and strategies for mitigating

We have covered a lot of information today related to your role as a leader for [ORGANIZATION NAME] and I have learned a lot about the way in which your program operates and some of its challenges and successes. We have just a few more questions to wrap-up our discussion.

20. Can you summarize for me now what you think have been the key factors that have facilitated the successful implementation of your program? What are the major challenges that have hindered how well the program operates? How have these challenges affected the program’s ability to (1) deliver high-quality services, (2) obtain high levels of participant engagement, and (3) achieve the sought outcome and benefits for participants?

21. To what extent do you believe program staff takes ownership for the success of the [RF/HM PROGRAM]?

22. What else would you like to see change in your program to improve it or make it more successful?

23. Are there lessons you have learned or is there advice you would like to share with other policymakers and practitioners interested in designing and implementing high quality and effective RF/HM programs?

Famle cross-site Project—Implementation Study

PROJECT DIRECTOR/PROGRAM MANAGER Interview Guide

Interviewer instructions: Before beginning the interview, tailor this guide to focus on the additional details we need about the grantee. In particular, be sure to probe for details on populations (such as youth aging out of foster care) and policy areas (such as connections to child support and intimate partner violence services) that are a high priority interest for OFA.

Thank you for taking the time to speak with us today. Mathematica Policy Research is conducting a study for the U.S. Department of Health and Human Services called the Fatherhood and Marriage Local Evaluation and Cross-Site Project, or the FaMLE Cross-Site Project. As part of this effort, we are gathering information develop an in-depth understanding of how [GRANTEE ORGANIZATION] designed, implemented, and operates [RF/HM PROGRAM]. As part of this study, Mathematica is interviewing staff at all levels in [GRANTEE ORGANIZATION] and at some partner organizations that work with [RF/HM PROGRAM] clients.

During this interview, we would like to hear about your experiences with [RF/HM PROGRAM], your opinions about program successes, and things other organizations should know if they want to develop a similar program. Our discussion will focus on your role as the [PROJECT DIRECTOR/PROGRAM MANAGER]. We have a large number of topics to cover and, at times, we may need to move our conversation along to ensure we get to all of them. We will use the information you share with us to write a summary of what we have learned about your program, which we will publish in a future report about your program and others, but we will not attribute any of your comments to you in our reports. We would like to record our discussion today, however, only members of our team will have access to this recording and we will destroy it after we complete our analysis. Do we have permission to record this conversation? [OBTAIN PERMISSION AND START RECORDING IF PERMISSION GRANTED.]

A. Respondent Background and Responsibilities

Let’s start by discussing your background and responsibilities.

1. What is your official job title?

2. Briefly describe your primary responsibilities for the [RF/HM PROGRAM].

3. How long have you been working at [GRANTEE ORGANIZATION] in this position? When did your involvement in the [RF/HM PROGRAM] begin?

4. What types of job experiences did you have before becoming a [JOB TITLE] for the [RF/HM PROGRAM]? What did you do immediately prior to assuming this position?

5. What are your primary responsibilities for the [RF/HM PROGRAM]?

6. Do you have other responsibilities outside of those you carry out for the [RF/HM PROGRAM]?

   1. What are these other responsibilities?

   2. How much of your time is allocated to your role for the [RF/HM PROGRAM] versus these other responsibilities?

7. How, if at all, has your role or responsibilities changed since the implementation of the [RF/HM PROGRAM]? What led to these changes?

B. Marketing, recruitment, and engagement strategies

8. Please describe all the methods you use to identify and reach out to prospective participants and get them to come through your doors.

Probe: To what extent are each of the following methods used?

1. Print-based outreach, which methods (Obtain any handouts, brochures, or other material that describes the program for potential applicants)

2. Media-based outreach, which methods

3. Outside organizations to provide referrals

4. Intra-organizational referrals

5. Co-location or outstationing RF/HM program staff at referral source locations

6. Canvassing and one-on-one recruiting in neighborhoods, businesses, and areas where target population is likely to concentrate

7. Encourage former participants to tell others about the program

8. [Interviewer: Refer to notes on marketing, recruitment, and engagement strategies.] Are the program’s outreach and recruitment strategies different from what was originally planned?

<If so>

1. What has changed over time?

2. When did each change occur?

3. What led to each change?

<If not>

4. What has contributed to consistency in the outreach and recruitment strategies?

8. Which strategies are you finding to be the most successful and why do you think the strategies work well?

8. How, if at all, have the eligibility requirements changed?

<If yes>

1. When did this change occur?

2. What led to this change?

<If no>

3. What has contributed to consistency in the eligibility requirements?

8. How, if at all, has the approach to client assessment for program eligibility changed since originally planned?

8. How, if at all, has the approach to client assessment for determining service needed changed since originally planned?

8. I’d like to talk about strategies used to foster fathers/couples’ participation in services.

   1. How do strategies differ for fostering initial compared to ongoing participation? How did you select these strategies? Has [GRANTEE ORGANIZATION] used these strategies in the past?

   2. What strategies are most successful and least successful at engaging fathers/couples initially and in the long-term? Which strategies/activities appear to be the most helpful in encouraging initial participation and why? For encouraging ongoing participation?

8. Overall, how interested are prospective fathers/couples in [RF/HM PROGRAM]? What methods are most effective at generating interest and getting fathers/couples to enroll?

C. Participant characteristics; Strategies for program enrollment and retention; Motivation for enrolling; Satisfaction with program services

8. [Interviewer: Refer to notes on participant characteristics.] How, if at all, is the population that you currently serve different from the population that you planned to serve when you applied for the grant? We’re interested in any demographic, socioeconomic, and geographic changes in the population, and also any changes in special groups that may be served, such as youth, reentering ex-offenders, or veterans. Please feel free to comment on any other characteristics you think are pertinent.

<If yes>

1. When did this change occur?

2. What led to this change?

3. Do you think this change in the population affected program delivery?

<If no>

4. What has contributed to consistency in the target population?

8. What criteria must an individual or couple meet to be eligible to participate in [RF/HM PROGRAM]?

8. What are the most common reasons that make a person ineligible for [RF/HM PROGRAM]?

8. Beyond the program’s eligibility requirements, does the [RF/HM PROGRAM] target any specific populations of fathers/couples?

20. Once fathers/couples enroll, how quickly are fathers/couples expected to begin participating? How is this expectation conveyed to enrollees?

20. What are the most common reasons fathers/couples give upfront for why they want to enroll in [RF/HM PROGRAM]? What aspects of your program are they most interested in? What are their expectations going into the program and what do they hope to gain from participating in the program?

22. What are the greatest needs of the fathers/couples served?

22. What are the greatest strengths of the fathers/couples served?

22. What are fathers/couples’ biggest barriers to participation in program services?

    1. What have you found to be the most common reasons why those enrolled in the program do not go on to participate? What about those who engage in the beginning but then drop out?

    2. How have you addressed these barriers?

22. What have you found to be the most common reasons why those enrolled stay in the program? How about those in the program do not go on to participate? What about those who engage in the beginning but then drop out?

22. How do you assess the overall level of satisfaction with program services? How satisfied do you think participants were with services?

    1. Were fathers/couples more engaged in or satisfied with certain aspects of your program? If so, which aspects? Do you have a sense for why this was more appealing?

D. Delivery (including customization), dosage, and quality of program content

Now we’re going to spend some time discussing how you work with fathers/couples to address employment and other case management needs.

27. [Interviewer: Refer to notes on delivery (including customization), dosage, and quality of program content.] How, if at all, are the services now offered different from or consistent with what was originally planned? When and why were changes made? What contributed to consistency in these areas?

    1. Parenting services, both curriculum-based and individual

    2. Relationship services, both curriculum-based and individual

    3. Employment services, both curriculum-based and individual

    4. Other additional services, both curriculum-based and individual

    5. What changes did you intentionally make to these programs?

    6. What changes were unintentional, and why did these changes occur?

    7. Did fathers/couples receive or participate in as much of each program as you had intended?

       1. If not, what caused differences?

       2. Were these changes intentional?

          1. If so, why did your program make these changes?

          2. If not, what factors led to unintentional changes?

27. Have there been any changes in how your staff screen for, identify and address intimate partner violence and/or child maltreatment? What were the reasons for these changes?

E. Staff recruitment, characteristics, training, and retention

Now let’s discuss program staffing—we want to learn about both the program’s current staffing plan and the program’s experience with turnover.

27. [Interviewer: Refer to notes on staff recruitment, characteristics, training, and retention.] How, if at all, has the staffing and supervisory plan for [RF/HM PROGRAM] changed while operating under this current cycle of grant funding or since the program began operating? When did these changes occur? What led to these changes?

27. Let’s discuss each staff position in your organization chart.

<ASK THE FOLLOWING QUESTIONS ABOUT EACH POSITION, EMPHASIZE POSITIONS THAT PD/PM DIRECTLY SUPERVISES>

1. How, if at all, have the roles and responsibilities of [POSITION] changed over time?

2. How, if at all, has the type and level of education sought for [POSITION] changed over time?

3. Have you experienced staff turnover? How have you managed staff turnover? What strategies have you used to retain staff?

4. What skills and qualities have you found to be useful for staff serving as [POSITION]?

27. What is your typical process for identifying, interviewing, and hiring new staff?

27. What experience and education level do you seek for each position? Have there been changes to the level of education and experience that you seek?

27. How, if at all, have the roles and responsibilities of the program staff changed over time? How has the management structure changed over time, if at all?

If changes occurred:

1. When did these changes occur?

2. What led to these changes?

27. What kinds of training do new program staff receive?

27. How much turnover has been experienced for staff in [RF/HM PROGRAM]? Are there any current staff openings for [RF/HM PROGRAM]?

27. To what do you attribute this turnover/lack of turnover? What are the main reasons staff have left, or been terminated?

27. When turnover has occurred, how long does it typically take to fill an open position and what challenges are experienced when filling a position?

27. When turnover has occurred, what have been the primary operational challenges experienced by [RF/HM PROGRAM]?

PROBE: Were aspects of service delivery altered? How was the workload for the vacant position managed? Were any staff or cases reassigned?

27. What steps or strategies have been undertaken to retain current staff and prevent future turnover?

27. In what ways do staff believe that the following services are valuable to fathers/couples?

    1. Parenting services

    2. Healthy marriage/ relationship services

    3. Economic stability services

27. Which services or strategies, if any, are staff more skeptical are helping fathers/couples achieve their intended outcomes?

27. Which services, if any, are staff uncomfortable with or resistant to providing?

27. In your view, to what extent do staff think that the services delivered by the program are the “right mix” in terms of range, type and intensity to help fathers/couples achieve their intended outcomes?

F. Integration of organizational partners

44. Interviewer: Refer to notes on integration of organizational partners.] How are the current organizational partners the same as or different from what was originally planned?

    1. FOR NEW PARTNERS: When did [RF/HM PROGRAM] begin working with this partner organization and why was the partnership established?

    2. FOR CONTINUING PARTNERS: What has contributed to continuity with this partner over time?

44. For each current organizational partner, what agreement exists between the partner organization and [GRANTEE ORGANIZATION] to specify how the partner will contribute to [RF/HM PROGRAM]?

    1. Is there an MOU between the partner organization and [GRANTEE ORGANIZATION]?

    2. Does the partner organization receive any funding from [GRANTEE ORGANIZATION] to support their role?

46. How have your organizational partnerships helped or not helped [GRANTEE ORGANIZATION] achieve its program goals?

46. How often do you personally have contact with your partners, and what is the substance of your communications?

    1. Is there a designated liaison at [GRANTEE ORGANIZATION] that is primarily responsible for being the point of contact with your partners?

    2. If so, how does the liaison communicate with the partners, and what is the substance of the communications?

46. How, if at all, are the referral sources now used different from what was originally planned? When did the [RF/HM PROGRAM] begin receiving referrals for these sources?

46. How, if at all, has [RF/HM PROGRAM] revised the approach used for forging a relationship with a referral source? Why did the approach change?

46. How, if at all, has the referral process been revised over time? Why did the referral process change and when did the change occur?

G. Adherence/modifications to implementation plans/logic model; Attainment of program targets

Now that we’ve talked quite a bit about how the [RF/HM PROGRAM] is implemented, we’d like to get a sense of how program operations are monitored and your assessment of the quality of program services.

51. [Interviewer: Refer to notes on adherence/modifications to implementation plans/logic model.] How has your logic model changed since you applied for the RF/HM grant? Why were these changes made?

    1. [IF YES] How do you think these changes affected your program outputs or outcomes?

    2. Did you consider, but not pursue, any alternative approaches? What were these, and why did you select not to pursue these ideas?

52. [Interviewer: Refer to notes on attainment of program targets.] Which aspects of the program do you monitor for consistent delivery?

    1. Is data used to adjust program delivery? If so, how so?

    2. How is consistent delivery defined for these aspects of the program?

    3. How do you manage staff, partners, and other resources in reaching your targeted outcomes?

    4. What are key challenges and barriers that impede staff from delivering these aspects of the program in a consistent manner?

    5. What do you do if you find substantial variation in how staff delivery these aspects of the program?

    6. How have changes in delivery helped or not helped [GRANTEE ORGANIZATION] achieve its program goals?

52. How is [RF/HM PROGRAM] doing relative to benchmarks on the following targets? To what do you attribute to this result?

    1. Outreach and recruitment

    2. Enrollment

3. Participation

54. How are data used for monitoring progress on outreach and referrals monitored? Which data are used?

    1. How often is progress monitored and who is responsible for this?

    2. How much do you attribute your performance on outreach and referrals to any of the following:

       1. Staffing

       2. Service delivery supports

       3. Organizational partnerships

54. How are data used for monitoring progress on enrollments? Which data are used?

    1. How often is progress monitored and who is responsible for this?

    2. How much do you attribute your performance on enrollments to any of the following:

       1. Staffing

       2. Service delivery supports

       3. Organizational partnerships

54. How are data used for monitoring progress on participation? Which data are used?

    1. How often is progress monitored and who is responsible for this?

    2. How much do you attribute your performance on participation to any of the following:

       1. Staffing

       2. Service delivery supports

       3. Organizational partnerships

H. Implementation drivers and successes; Implementation challenges and strategies for mitigating

We have covered a lot of information today related to your role as a program director for [RF/HM PROGRAM] and I have learned a lot about the way in which your program operates and some of its challenges and successes. We have just a few more questions to wrap-up our discussion.

54. Would you summarize for me what you think have been the key factors that have facilitated the successful implementation of your program? What are the major operational challenges encountered by the program? How have these challenges affected the program’s ability to (1) deliver high-quality services, (2) obtain high levels of participant engagement, and (3) achieve the sought outcome and benefits for participants?

54. To what extent do you believe program staff takes ownership for the success of the [RF/HM PROGRAM]?

54. Are there lessons you have learned or is there advice you would like to share with other policymakers and practitioners interested in designing and implementing high quality and effective RF/HM programs?

Famle cross-site Project—Implementation Study

Supervisor Interview Guide

Interviewer instructions: Before beginning the interview, tailor this guide to focus on the additional details we need about the grantee. In particular, be sure to probe for details on populations (such as youth aging out of foster care) and policy areas (such as connections to child support and intimate partner violence services) that are a high priority interest for OFA.

Thank you for taking the time to speak with us today. Mathematica Policy Research is conducting a study for the U.S. Department of Health and Human Services, the Fatherhood and Marriage Local Evaluation and Cross-Site Project, or the FaMLE Cross-Site Project. As part of this effort, we are gathering information to develop an in-depth understanding of how [GRANTEE ORGANIZATION] designed, implemented, and operates [RF/HM PROGRAM]. As part of this study, Mathematica is interviewing staff at all levels in [GRANTEE ORGANIZATION] and at some partner organizations that work with [RF/HM PROGRAM] clients.

During this interview, we would like to hear about your experiences with [RF/HM PROGRAM], your opinions about program successes, and things other organizations should know if they want to develop a similar program. Our discussion will focus on your role as a supervisor. We will not be asking detailed questions about the services provided through the program, as those topics will be covered in other interviews. We will use the information you share with us to write a summary of what we have learned about your program, which we will publish in a future report about your program and others, but we will not attribute any of your comments to you in our reports. We would like to record our discussion today, however, only members of our team will have access to this recording and we will destroy it after we complete our analysis. Do we have permission to record this conversation? [OBTAIN PERMISSION AND START RECORDING IF PERMISSION GRANTED.]

[Note to interviewer: Modify protocol as needed for group interviews.]

A. Respondent Background and Responsibilities

Let’s start by discussing your background and responsibilities.

1. What is your official job title?

2. How long have you been working at [GRANTEE ORGANIZATION] in this position? When did your involvement in the [RF/HM PROGRAM] begin?

3. What are your primary responsibilities for the [RF/HM PROGRAM]?

4. Do you have other responsibilities outside of those you carry out for the [RF/HM PROGRAM]?

If yes:

1. What are these other responsibilities?

2. How much of your time is allocated to your role for the [RF/HM PROGRAM] versus these other responsibilities?

B. Staff recruitment, characteristics, training, and retention; Delivery (including customization), dosage, and quality of program content

Now let’s talk about the staff you supervise and the extent of staff turnover in the [RF/HM PROGRAM].

5. What is the typical process for identifying, interviewing and hiring new program staff?

6. How many staff do you currently supervise?

7. What are their position titles and primary responsibilities?

8. [IF OVERSEE CASE MANAGERS] What is the typical caseload for your case managers?

9. How, if at all, have the roles and responsibilities of the staff you supervise changed since the beginning of the [RF/HM PROGRAM]?

If changes occurred:

1. When did these changes occur?

2. What led to these changes?

10. To what extent do you think [RF/HM PROGRAM] staff have “bought in” to the goals of [RF/HM PROGRAM]?

11. In what ways do staff believe that the following services are valuable to fathers/couples?

    1. Parenting services

    2. Healthy marriage/ relationship services

    3. Economic stability services

12. Which services or strategies, if any, are staff more skeptical about their usefulness for the fathers/couples served? Which services or strategies, if any, are staff uncomfortable with or resistant to implementing? What are their concerns?

13. For each type of staff you supervise, please describe the training they received as new staff for [RF/HM PROGRAM].

    1. What issues or topics were covered?

    2. What formats were used (e.g., group format, on-line/self-administered)?

    3. How long was the training?

    4. Who provided this training (organization/individual)?

    5. What is your assessment of how well the training provided to new staff prepares them to effectively carry out their job responsibilities and duties?

    6. In which areas or ways could it have been better?

14. Next, for each type of staff you supervise, please describe the ongoing or follow-up training that they receive for [RF/HM PROGRAM].

    1. How often is ongoing or follow-up training provided?

    2. What issues or topics were covered?

    3. What format was used (e.g., group format, on-line/self-administered)?

    4. How long was the training?

    5. Who provided this training (organization/individual)?

    6. What prompted you/ [GRANTEE ORGANIZATION] to provide this training?

    7. What is your assessment of how well the ongoing training provided to staff supports them in carrying out their job responsibilities and duties?

15. Are there other trainings you feel that your staff need that you/ [GRANTEE ORGANIZATION] have/has not been able to offer? Explain.

16. Are there currently any staff openings for the [RF/HM PROGRAM]? If so, which positions and why are the positions open?

16. Over the past 12 months, which staff positions within the [RF/HM PROGRAM] have experienced turnover or transitions?

16. For each staff position with turnover or transitions:

    1. Please explain the extent of turnover or transitions.

    2. When did the vacancy occur?

    3. What was the reason for the vacancy?

    4. What happened to the position after it was vacated?

    5. If filled, how long did it take to fill the position?

16. What challenges has the [RF/HM PROGRAM] encountered when filling vacant positions?

16. What operational challenges has the [RF/HM PROGRAM] experienced as a result of staff turnover/transition?

16. When turnover has occurred, what steps or strategies were implemented to handle the workload while the position was vacant?

    1. Were staff roles or responsibilities modified? If so, how?

    2. Were cases reassigned, even temporarily? If so, how?

    3. Were aspects of the program altered? If so, please explain.

    4. Did outreach or enrollment slow temporarily? If so, to what extent?

16. Overall, what effect has staff turnover had on remaining staff?

16. What steps or strategies have you undertaken to retain current staff and prevent future turnover?

C. Implementation drivers and successes; Implementation challenges and strategies for mitigating

We have covered a lot of information today related to your role as a supervisor for [RF/HM PROGRAM] and I have learned a lot about the way in which your program operates and some of its challenges and successes. We have just a few more questions to wrap-up our discussion.

16. Would you summarize for me what you think have been the key factors that have facilitated the successful implementation of your program? What are the major challenges that have hindered how well the program operates? How have these challenges affected the program’s ability to (1) deliver high-quality services, (2) obtain high levels of participant engagement, and (3) achieve the sought outcome and benefits for participants?

16. To what extent do you believe program staff takes ownership for the success of the [RF/HM PROGRAM]?

16. What kinds of additional services do you think would be beneficial to add to the ones your program already offers?

16. What else would you like to see change in your program to improve it or make it more successful?

16. Are there lessons you have learned or is there advice you would like to share with other policymakers and practitioners interested in designing and implementing high quality and effective RF/HM programs?

famle CROSS-SITE PROJECT—Implementation Study

OUTREACH AND RECrUITMENT STAFF Interview Guide

Interviewer instructions: Before beginning the interview, tailor this guide to focus on the additional details we need about the grantee. In particular, be sure to probe for details on populations (such as youth aging out of foster care) and policy areas (such as connections to child support and intimate partner violence services) that are a high priority interest for OFA.

Thank you for taking the time to speak with us today. Mathematica Policy Research is conducting a study for the U.S. Department of Health and Human Services called the Fatherhood and Marriage Local Evaluation and Cross-Site Study, or the FaMLE Cross-Site Project. As part of this effort, we are gathering information to develop an in-depth understanding of how [GRANTEE ORGANIZATION] designed, implemented, and operates [RF/HM PROGRAM]. As part of this study, Mathematica is interviewing staff at all levels in [GRANTEE ORGANIZATION] and at some partner organizations that work with [RF/HM PROGRAM] clients.

During this interview, we would like to hear about your experiences with [RF/HM PROGRAM], your opinions about program successes, and things other organizations should know if they want to develop a similar program. Our discussion will focus on your role in conducting outreach and recruitment for [RF/HM PROGRAM]. We will use the information you share with us to write a summary of what we have learned about your program, which we will publish in a future report about your program and others, but we will not attribute any of your comments to you in our reports. We would like to record our discussion today, however, only members of our team will have access to this recording and we will destroy it after we complete our analysis. Do we have permission to record this conversation? [OBTAIN PERMISSION AND START RECORDING IF PERMISSION GRANTED.]

[Note to interviewer: Modify protocol as needed for group interviews.]

A. Respondent Background and Responsibilities

Let’s start by discussing your background and responsibilities.

1. What is your official job title?

2. What are your primary responsibilities for the [RF/HM PROGRAM]?

3. Do you have other responsibilities outside of those you carry out for the [RF/HM PROGRAM]?

If yes:

1. What are these other responsibilities?

2. How much of your time is allocated to your role for the [RF/HM PROGRAM] versus these other responsibilities?

4. What are the key qualifications and skills associated with your job position? What experience and skills do you believe are most important for someone in your position to be effective?

5. How, if at all, has your role or responsibilities changed since the implementation of the [RF/HM PROGRAM]? What led to these changes?

B. Marketing, recruitment, and engagement strategies; Participant characteristics

6. [Interviewer: Refer to notes on participant characteristics.] How, if at all, is the current target population different from what was originally planned? We’re interested in any demographic, socioeconomic, and geographic changes in the population, and also any changes in special groups that may be served, such as youth, reentering ex-offenders, or veterans. Please feel free to comment on any other characteristics you think are pertinent.

<If yes>

1. When did this change occur?

2. What led to this change?

3. Do you think this change in the population affected program outreach, recruitment or enrollment?

<If no>

4. What has contributed to consistency in the target population?

6. Are there any particular geographic areas that you target for recruiting [RF/HM PROGRAM] enrollees?

Probes: single county/multiple counties, particular zip codes, other geographic configuration

6. What is the rationale for focusing recruitment efforts in that area/those areas?

6. Overall, how do prospective fathers/couples hear about the [RF/HM PROGRAM]?

   1. To what extent do staff use one-on-one recruitment as the primary method?

   2. Do partnerships with other organizations contribute significantly to outreach efforts? That is, do they provide referrals that you then follow up on?

6. Please describe all the methods you use to identify and reach out to prospective participants and get them to come through your doors.

Probe: To what extent are each of the following methods used? How much have these efforts changed since you applied for the RF/HM program grant?

1. Print-based outreach, which methods (Obtain any handouts, brochures, or other material that describes the program for potential applicants)

2. Media-based outreach, which methods

3. Outside organizations to provide referrals

4. Intra-organizational referrals

5. Co-location or outstationing RF/HM program staff at referral source locations

6. Canvassing and one-on-one recruiting in neighborhoods, businesses, and areas where target population is likely to concentrate

7. Encourage former participants to tell others about the program

6. [Interviewer: Refer to notes on attainment of program targets.] We’d like to talk about data you’ve reported on your recruitment efforts. Which of your methods are most and least successful? How about particularly with regards to enrollment and retention?

C. Strategies for program enrollment and retention

6. [Interviewer: Refer to notes on strategies for program enrollment and retention.] We have some information on your plans for enrollment, but could you provide a description of the process that is used for assessing eligibility and enrolling individuals into [RF/HM PROGRAM]?

   1. Does the process involve more than one meeting?

   2. If so, do you find that most fathers/couples come back to subsequent appointments to complete the intake process?

   3. Is attrition a problem?

   4. What efforts are made to help eligible fathers/couples to get through the entire intake process?

   5. At what point and for what reasons would you stop attempts to get the prospective participant to complete the intake process?

6. What criteria must an individual meet to be eligible to participate in [RF/HM PROGRAM]? Have you made any changes to the eligibility criteria? What kinds of information do you obtain about the individual’s circumstances and characteristics prior to enrolling them into [RF/HM PROGRAM] and why is it obtained?

6. What are the most common reasons that make a person ineligible for [RF/HM PROGRAM]?

6. How are individuals screened or assessed as part of the intake and eligibility determination process? Have you made any changes to the assessment procedures?

6. Is the screening and/or assessment formal or informal? Do you use any formal assessment instruments (obtain a copy of screening and assessment instruments)?

6. How, if at all, is the information obtained at this point in the enrollment process used to determine which program component or services a participant will subsequently be directed to?

6. What information about the program’s services and expectations are provided to prospective fathers/couples during enrollment?

   1. What do you tell potential participants about what is encouraged or expected of them in terms of their attendance and participation if they enroll?

   2. What do you tell them about the services that are provided?

   3. What do you tell them about the participation incentives or supports you provide (e.g., gift card for completing intake, attendance, transportation assistance)?

6. Are there any steps or aspects of the screening and enrollment process that have been difficult or challenging for you to carry out?

6. Are there any steps or aspects of the screening and enrollment process that you think are difficult or challenging for those interested in participating in the program?

6. Have there been any changes to the screening and enrollment process? What are they? Why were they made? Did they improve the process? If so, how?

6. What are the greatest needs of the fathers/couples served?

6. What are the greatest strengths of the fathers/couples served?

D. Motivation for enrolling; Delivery (including customization), dosage, and quality of program content

6. What are the most common reasons fathers/couples give upfront for why they want to enroll in [RF/HM PROGRAM]? What aspects of your program are they most interested in? What are their expectations going into the program and what do they hope to gain from participating in the program?

6. How interested are prospective fathers/couples in [RF/HM PROGRAM] and what methods are most effective at generating interest and getting fathers/couples to enroll?

6. What are the characteristics of the people who you think are most likely to participate in [RF/HM PROGRAM]?

6. Which fathers/couples do you think are most likely to benefit from participating in [RF/HM PROGRAM]? What types of qualities and characteristics do these fathers/couples possess? Are any fathers/couples not good candidates; if so, why?

6. Have there been any changes in how your staff screen for, identify and address intimate partner violence and/or child maltreatment? What were the reasons for these changes?

E. Staff recruitment, characteristics, training, and retention

29. Are there currently any openings for staff in outreach or intake positions?

If yes:

1. For how long has this position been vacant?

2. What effect has this had your workload and job responsibilities?

29. Has there been any turnover in outreach/intake staff since the grant program began?

If yes:

1. When did this vacancy occur? How long did it take to fill the job position?

2. What effect did this have your workload and job responsibilities?

29. What effect, if any, did the current vacancy or past turnover have on any aspect of the [RF/HM PROGRAM’s] recruitment process, this could involve outreach, referral, intake, or enrollment?

29. Have there been any other vacancies or turnover within the program or organization that you think have had an impact on the recruitment process, including outreach, referral, intake, and enrollment?

    1. Which position(s)?

    2. In what ways?

29. What experience or qualities do you think are important for someone to be effective in your role of recruiter/outreach/intake?

29. What type of training did you receive when you first were hired in your current position?

    1. What issues or topics were covered?

    2. What formats were used?

    3. How long was the training?

    4. Who provided this training (organization/individual)?

    5. What is your assessment of the quality and usefulness of the initial training and guidance you were provided? How could it have been better?

29. What ongoing or follow-up training have you received?

    1. What issues or topics were covered?

    2. What format was used?

    3. How long was the training?

    4. When and how often is it provided?

    5. Who provided this training (organization/individual)?

    6. What is your assessment of the quality and usefulness of the initial training and guidance you were provided? How could it have been better?

29. Have you received any technical assistance or guidance or other resources from outside entities, including individuals who are working with RF/HM grantees like yourself around the country?

For each type of training/TA received:

1. What is your assessment of the quality and usefulness of the ongoing training and guidance you have been given?

2. How could it have been better?

29. Are there any kinds of trainings or resources that you would like to have access to that you do not have access to? Please describe the types of training or resources you think you need.

29. In what ways do you and other outreach and recruitment staff believe that the following services are valuable to fathers/couples?

    1. Parenting services

    2. Healthy marriage/ relationship services

    3. Economic stability services

29. Which services or strategies, if any, are you more skeptical are helping fathers/couples achieve their intended outcomes?

29. Which services, if any, are you uncomfortable providing?

29. In your view, to what extent do staff believe that the services delivered by the program are the “right mix” in terms of range, type and intensity to help fathers/couples achieve their intended outcomes?

F. Integration of organizational partners

Now I’d like to ask you some additional questions about your referral sources and process.

29. What organizations and programs commonly refer individuals to your program, and are these different partners than originally planned for the [RF/HM PROGRAM]?

29. When did [RF/HM PROGRAM] or [GRANTEE ORGANIZATION] develop a relationship with these organizations or programs? Have any of these relationships been developed or strengthened since [RF/HM PROGRAM] grant started in 2011/2012?

29. Do you personally work with any of these organizations or programs to obtain referrals?

If yes:

1. What is the nature and purpose of your interactions?

2. Are you co-located or outstationed to an organization for recruitment? If so, how often does outstationing occur and how long are you at the organization?

3. Do you make informational presentations to staff at referring organizations?

4. Do you make informational presentations to clients served by the referring organizations?

29. How formalized is the process between referring organizations and [RF/HM PROGRAM] for making referrals? Are there established criteria or other procedures to help guide whether a potential father/couple is an appropriate referral? Is there an MOU with the referring organization? Is the organization compensated in some way for providing referrals (i.e., are they a subcontractor to the grantee?)

29. Which of these referral sources have generated the most referrals? The least?

29. Which referral sources consistently refer fathers/couples that meet eligibility criteria? Which referral sources consistently refer father/couples who engage in [RF/HM PROGRAM]? Is there variation across referral sources in prospective participants’ understanding and expectations of the services provided by [RF/HM PROGRAM]?

29. What are the key messages prospective participants receive about [RF/HM PROGRAM] through outreach? What aspects of the program are emphasized?

29. Have you or the referring organization staff tried different ways of describing the program to prospective participants? Which approaches appear to be most effective?

29. What methods have provided the most referrals? What methods have produced referrals that consistently meet eligibility criteria, are most likely to go ahead and enroll in the program and engage in program activities?

29. Why do you think these referral sources/methods have been the most successful? Why do you think other methods have been less successful?

29. What difficulties, if any, have been encountered with the referral process to date?

29. What are the major challenges your program has encountered in working with referral sources to identify potential participants?

G. Attainment of program targets

29. [Interviewer: Refer to notes on attainment of program targets.] During an average week, how many people are enrolled in the program?

29. Do you have targets or quotas for outreach, recruitment, and enrollment?

    1. What are they? What happens if you do/do not meet them?

    2. Are staff given incentives for enrolling a specific number of people?

    3. How do you use data to monitor your progress on these targets or quotas?

29. What do you think are the primary reasons that account for shortfalls in meeting outreach, recruitment, and enrollment targets? What accounts for your successes in surpassing targets?

H. Implementation drivers and successes; Implementation challenges and strategies for mitigating

We have covered a lot of information today related to your role in recruiting and enrolling fathers/couples into [RF/HM PROGRAM] and I have learned a lot about the way in which your program operates and some of its challenges and successes. We have just one final question to wrap-up our discussion.

57. Would you summarize what you think most contributes to successfully working with fathers/couples as an [OUTREACH/RECRUITMENT STAFF MEMBER]? Would you summarize challenges that are most common in your role, and how you deal with them? To what extent do you take ownership for the success of the [RF/HM PROGRAM]?

Famle cross-site Project—Implementation Study

Case Manager/employment specialist Interview Guide

Interviewer instructions: Before beginning the interview, tailor this guide to focus on the additional details we need about the grantee. In particular, be sure to probe for details on populations (such as youth aging out of foster care) and policy areas (such as connections to child support and intimate partner violence services) that are a high priority interest for OFA.

Thank you for taking the time to speak with us today. Mathematica Policy Research is conducting a study for the U.S. Department of Health and Human Services, called the Fatherhood and Marriage Local Evaluation and Cross-Site Project, or the FaMLE Cross-Site Project. As part of this effort, we are gathering information to develop an in-depth understanding of how [GRANTEE ORGANIZATION] designed, implemented, and operates [RF/HM PROGRAM]. As part of this study, Mathematica is interviewing staff at all levels in [GRANTEE ORGANIZATION] and at some partner organizations that work with [RF/HM PROGRAM] clients.

During this interview, we would like to hear about your experiences with [RF/HM PROGRAM], your opinions about program successes, and things other organizations should know if they want to develop a similar program. Our discussion will focus on your role working with participants on employment and/or other case management needs. We will use the information you share with us to write a summary of what we have learned about your program, which we will publish in a future report about your program and others, but we will not attribute any of your comments to you in our reports. We would like to record our discussion today, however, only members of our team will have access to this recording and we will destroy it after we complete our analysis. Do we have permission to record this conversation? [OBTAIN PERMISSION AND START RECORDING IF PERMISSION GRANTED.]

[Note to interviewer: Modify protocol as needed for group interviews.]

A. Respondent Background and Responsibilities

Let’s start by discussing your background and responsibilities.

1. What is your official job title?

2. What are your primary responsibilities for the [RF/HM PROGRAM] with respect to case management or employment services? Do you perform this work on a full-time or part-time basis?

3. Do you have other responsibilities for the [RF/HM PROGRAM], outside of case management or employment services? (e.g., Do you also facilitate workshops, conduct outreach or enrollment?) If so, how much time is allocated to these other responsibilities relative to case management/employment?

4. Do you have other responsibilities at this organization, outside of those you carry out for the [RF/HM PROGRAM]?

If yes:

1. What are these other responsibilities?

2. How much of your time is allocated to your role for the [RF/HM PROGRAM] versus these other responsibilities?

5. How, if at all, has your role or responsibilities changed since the implementation of the [RF/HM PROGRAM]? What led to these changes? When did these changes occur?

B. Strategies for program enrollment and retention; Participant characteristics

6. [Interviewer: Please refer to your notes about participant characteristics.] How, if at all, is the population that you currently serve different from the population that you planned to serve when you applied for the grant? We’re interested in any demographic, socioeconomic, and geographic changes in the population, and also any changes in special groups that may be served, such as youth, reentering ex-offenders, or veterans. Please feel free to comment on any other characteristics you think are pertinent.

<If yes>

1. When did this change occur?

2. What led to this change?

3. Do you think this change in the population affected program delivery?

<If no>

4. What has contributed to consistency in the target population?

6. Are fathers/couples expected to attend an orientation prior to their formally starting [RF/HM PROGRAM]?

<If yes>:

1. Is it a group orientation, one-on-one orientation or a mix of both? If a mix, under what circumstances are orientations one-on-one vs. group?

2. When and where does orientation take place and how frequently is it held?

3. How long is orientation?

4. Which staff members attend orientation?

5. What are the goals and purposes of orientation?

6. What topics does orientation cover? What are the key messages and information conveyed?

7. Who leads orientation? What other staff/organizations attend some or all of the orientation? What functions do they perform?

8. What happens if a father/couple does not attend orientation? What effort is made to get him to attend another orientation session?

6. Do you conduct any assessments of the needs of fathers/couples (these assessments may be formal or informal)?

<If yes, for each type of assessment ask>:

1. What kinds of information do you obtain about the father/couple?

2. When does this assessment take place? Who conducts it and how frequently is it completed?

3. Is this assessment formal or informal?

4. Do you use a specific assessment instruments or methods? (Obtain a copy of all screening and assessment instruments, if available.)

5. How do you use the information obtained from the assessment?

6. How is the information used to identify needs for services offered by outside agencies?

7. Which assessments do you find to be the most useful and why? Which, if any, assessment do you find to be unhelpful? How do these assessments fall short of providing the intended information?

8. Are methods other than assessment used to determine the program activities and services that a father/couple should be offered? If so, what methods and what factors are considered when making that determination?

9. Have there been any changes in your assessment practices and procedures since beginning to offer services in the grant cycle? If yes, please describe.

10. To what extent do you think these changes have improved your capacity to assess the needs of fathers/couples and/or link him/them to needed services?

9. After you have identified what services the father/couple needs, how do you link him/them to these services? What types of organizations and services do you most frequently refer fathers/couples to?

9. What are the gaps in the availability of needed services and resources for fathers/couples?

9. What strategies are used to encourage recently enrolled participants to engage in the activities or services they have been offered/signed up for?

9. Do you use any kinds of incentives to encourage newly enrolled fathers/couples to participate in program services? To encourage participants to continue to stay engaged and complete program activities?

   1. What incentives are provided? When and how often are they provided?

   2. When are fathers/couples told about these incentives?

   3. Are incentives contingent upon something that participants must do, such as attendance?

   4. How important do incentives appear to be for encouraging initial or ongoing participation?

   5. Which incentives seem to be more effective than others?

   6. Beyond providing an incentive for initial enrollment or participation, are there additional benefits to providing incentives?

9. Which strategies/activities appear to be the most helpful in encouraging initial participation? For encouraging ongoing participation?

C. Integration of organizational partners

9. When did [RF/HM PROGRAM] or [GRANTEE ORGANIZATION] develop a relationship with your partner organizations or programs? Have any of these relationships been initiated or developed further since [RF/HM PROGRAM] grant started in 2011/2012?

9. Do you personally work with any of these organizations or programs to obtain or give referrals?

If yes:

1. What is the nature of your interactions?

2. What have been the successes and challenges in your relationship with these partner organizations?

9. What is your referral process when a father/couple needs services that are not available within the [RF/HM PROGRAM]? Are referrals verbal or written? Are referrals provided to the father/couple or to the organization to which they are being referred, or both? How does your process vary by type of organization/service?

9. Do you follow up with the father/couple or the organization to which you referred the father/couple to determine whether there was follow-through?

9. [Interviewer: Refer to notes on strategies for program enrollment and retention and ask question if there is a difference.] According to what you reported, you expected to give [INSERT NUMBER] referrals, and you actually gave [INSERT NUMBER]. Do you have thoughts on what accounts for this difference?

9. Which referral sources consistently refer fathers/couples that meet eligibility criteria and engage in [RF/HM PROGRAM]? Is there variation across referral sources in prospective participants’ understanding and expectations of the services provided by [RF/HM PROGRAM]?

D. Motivation for enrolling; Satisfaction with program services

Now, we want to turn to discuss the program’s strategies for engaging fathers/couples in its services.

20. What are the most common reasons fathers/couples give upfront for why they want to enroll in [RF/HM PROGRAM]? What aspects of your program are they most interested in? What are their expectations going into the program and what do they hope to gain from participating in the program?

20. What have you found to be the most common reasons why those enrolled in the program do not actually participate? What about those who engage in the beginning but then drop out?

20. Is there a point at which you conclude that a nonparticipating father/couple is unlikely to ever attend? At what point and under what circumstances do you cease efforts to encourage the participant to (re)engage with the program?

20. What are the most common reasons that fathers/couples persist or stay in the program?

    1. How do you assess the overall level of satisfaction with program services? How satisfied do you think participants were with services?

E. Participant characteristics; Delivery (including customization), dosage, and quality of program content; Attainment of program targets

Now we’re going to spend some time discussing how you work with fathers/couples to address employment and other case management needs.

24. [Interviewer: Refer to notes on delivery (including customization), dosage, and quality of program content.] Can you walk me through how fathers/couples progress through [RF/HM PROGRAM], from the point at which they have completed the intake process to when they are no longer considered to be an active participant? It will be helpful to just start with an overall picture of the client flow.

    1. In general, how structured is the program in terms of the sequence and timing of activities and services offered? Is there a predetermined sequence of activities, a preferred sequence of activities, or does it largely depend on the individual’s preferences and needs?

    2. Do fathers/couples typically receive the number of classes or workshops that is planned by the program? Why or why not?

    3. At what point do you meet and start working with the father/couple? How long after the intake process is completed does this occur? What is the purpose of your initial meeting with the father/couple?

    4. How often do you meet with the father/couple? How long is your typical meeting? Is this different from what was expected, and if so, what accounts for the difference?

    5. What is your typical caseload?

    6. We have data on your participation targets and benchmarks for case management. Why do you think you have [NOT MET/EXCEEDED] your targets?

    7. Do you track data to monitor your targets for case management? How do you do this?

24. Have there been any changes in how your staff screen for, identify and address intimate partner violence and/or child maltreatment? What were the reasons for these changes?

24. How do you work and coordinate with other staff in [RF/HM PROGRAM]? How frequently and under what circumstances do you share information with the group facilitators, [other] employment specialists, [other] case managers, or other staff (specify)?

27. What is the primary purpose of these meetings?

27. What are the most common issues or topics you address in your one-on-one dealings with fathers/couples?

Probes: Helping to address issues, questions and concerns they have about parenting or relationships? Helping them access needed employment services? Providing individual-level employment services, such as help with resume development, job search, or work preparation skills?

27. How much time do you spend communicating with fathers/couples in one-on-one meetings versus other modes of communication (e.g., phone, email)?

27. How do you connect with fathers/couples on an interpersonal level? How important is it to build a relationship with them, establish a rapport and a sense of trust?

**ASK THE FOLLOWING QUESTIONS, IF STAFF ARE INVOLVED IN ASSISTING FATHERS/COUPLES WITH PREPARING FOR OR OBTAINING EMPLOYMENT **

There are many ways that programs may assist fathers/couples with preparing for and obtaining employment. We want to learn about the specific ways [RF/HM PROGRAM] works with fathers/couples around employment.

27. How do you work with fathers/couples to assist them in preparing for work and obtaining employment?

    1. Is this assistance different from what was planned? What accounts for the difference?

2. Describe the assistance you provide to participants related to career assessment, job readiness skill development, or resume writing.

   1. What specific job readiness skills are targeted and how are they addressed?

   2. How do you work with participants to develop a resume?

3. Describe how you assist participants with job search activities.

   1. Do you teach participants how to search for jobs or provide job leads directly?

   2. What are your expectations for follow-up on job leads by participants?

4. Does [RF/HM PROGRAM] offer subsidized employment to any participants?

   1. How many subsidized placements does [RF/HM PROGRAM] have funding for annually and what is the funding source for these placements?

   2. How many subsidized placements does [RF/HM PROGRAM] actually make in a year?

   3. In the past year, how many placements have you personally facilitated?

   4. How do you identify that an individual is appropriate for a subsidized employment slot?

   5. How do participants obtain a subsidized employment slot?

   6. What type of job is completed with a subsidized employment placement?

   7. How has [RF/HM PROGRAM] developed relationships with employers to facilitate subsidized employment placements?

   8. How long are wages subsidized and at what wage?

   9. How often do subsidized employment placements transition into permanent positions for the participant?

5. Do you work to actually place participants in jobs, other than subsidized employment slots (if relevant)?

   1. What types of jobs are you able to identify for participants? Are the positions usually permanent or temporary, full-time or part-time?

   2. How do you identify suitable jobs for specific individuals?

   3. How often do you find participants jobs?

   4. What are the most common barriers to employment faced by the participants you serve?

   5. How do you work around these barriers when finding suitable jobs?

6. Do you assist participants in dealing with problems that may arise on a job?

   1. What types of job-related problems do you assist fathers with?

   2. How frequently do you intervene for a participant on job-related problems?

   3. How does your intervention assist the participant on job-related problems?

F. Staff recruitment, characteristics, training, and retention

Now let’s talk about the qualities you think are important to work effectively with fathers/couples. I’d also like to discuss your perceptions regarding staff transitions in [RF/HM PROGRAM].

Recruitment

32. What is the general process for identifying, interviewing, and hiring case managers?

32. What experience or qualities do you think are most important for someone to be effective when working with participants on employment and case management needs?

1. What type and level of education and experience do you think is most important for someone who works with individuals on employment and other case management needs? Have these characteristics changed since the program was initiated?

Education: high school/GED, AA, BA or BSW, MA or MSW, or LCSW

Experience: social work, low-income families, sales, recruitment, former participant in [RF/HM PROGRAM]

2. What skills and qualities have you found to be useful for working with participants on employment and/or other case management needs?

3. Has the case manager role and responsibilities changed since the program was initiated? If so, in what ways?

32. In what ways do you and other case managers and employment specialists believe that the following services are valuable to fathers/couples?

    1. Parenting services

    2. Healthy marriage/ relationship services

    3. Economic stability services

32. Which services or strategies, if any, are you more skeptical about helping fathers/couples achieve their intended outcomes?

32. Which services, if any, are you uncomfortable providing?

32. In your view, are the program’s services the “right mix” in terms of range, type and intensity to help participants achieve their intended outcomes?

Retention

32. Are there currently any case management or employment-related staff openings?

<If yes>:

1. What position is vacant?

2. For how long has this position been vacant?

3. What effect has this had on your workload and job responsibilities?

32. Has there been any turnover in case management or employment-related staff since the grant program began?

<If yes>:

1. In what position did a vacancy occur?

2. When did this vacancy occur? How long did it take to fill the job position?

3. What effect did this have on your workload and job responsibilities?

32. What effect, if any, did the current vacancy or past turnover have on any aspect of the services provided by [RF/HM PROGRAM]?

    1. Have there been any other vacancies or turnover within the program or organization that you think have had an impact on overall program operations or service delivery?

       1. Which position(s)?

       2. In what ways?

Training

32. Please describe the training you received as new staff for [RF/HM PROGRAM].

1. What issues or topics were covered?

2. What formats were used (e.g., group format, on-line/self-administered)?

3. How long was the training?

4. Who provided this training (organization/individual)?

5. What is your assessment of how well the initial training you received prepared you to effectively carry out your job responsibilities and duties?

6. In which areas or ways could your initial training have been better?

32. Next, please describe any ongoing or follow-up training you have received with respect to your position at [RF/HM PROGRAM].

1. How often is ongoing or follow-up training provided?

2. When was the last follow-up training?

3. What issues or topics were covered?

4. What format was used (e.g., group format, on-line/self-administered)?

5. How long was the training?

6. Who provided this training (organization/individual)?

7. What prompted you/ [GRANTEE ORGANIZATION] to participate in this training?

8. What is your assessment of how well the ongoing training you received supported you in carrying out your job responsibilities and duties?

9. In which areas or ways could your ongoing training have been better?

32. Are there other trainings you feel that you need that [GRANTEE ORGANIZATION] has not been able to offer? Explain.

G. Implementation drivers and successes; Implementation challenges and strategies for mitigating

We have covered a lot of information today related to your role as a case manager for [RF/HM PROGRAM] and I have learned a lot about the way in which your program operates and some of its challenges and successes. We have just a few more questions to wrap-up our discussion.

44. Would you summarize what you think most contributes to successfully working with fathers/couples as a [CASE MANAGER/EMPLOYMENT STAFF]? Would you summarize any challenges that are most common in your role, and how you deal with those challenges? To what extent do you take ownership for the success of the [RF/HM PROGRAM]?

Famle Cross-Site Project—Implementation Study

Workshop Facilitator Interview Guide

Interviewer instructions: Before beginning the interview, tailor this guide to focus on the additional details we need about the grantee. In particular, be sure to probe for details on populations (such as youth aging out of foster care) and policy areas (such as connections to child support and intimate partner violence services) that are a high priority interest for OFA.

Thank you for taking the time to speak with us today. Mathematica Policy Research is conducting a study for the U.S. Department of Health and Human Services, the Fatherhood and Marriage Local Evaluation and Cross-Site Project, or the FaMLE Cross-Site Project. As part of this effort, we are gathering information to develop an in-depth understanding of how [GRANTEE ORGANIZATION] designed, implemented, and operates [RF/HM PROGRAM]. As part of this study, Mathematica is interviewing staff at all levels in [GRANTEE ORGANIZATION] and at some partner organizations that work with [RF/HM PROGRAM] clients.

During this interview, we would like to hear about your experiences with [RF/HM PROGRAM], your opinions about program successes, and things other organizations should know if they want to develop a similar program. Our discussion will focus on your role facilitating workshops for [RF/HM PROGRAM]. We will cover all types of workshops offered, which we understand may address topics related to parenting, healthy relationships, and/or economic stability. We will use the information you share with us to write a summary of what we have learned about your program, which we will publish in a future report about your program and others, but we will not attribute any of your comments to you in our reports. We would like to record our discussion today, however, only members of our team will have access to this recording and we will destroy it after we complete our analysis. Do we have permission to record this conversation? [OBTAIN PERMISSION AND START RECORDING IF PERMISSION GRANTED.]

[Note to interviewer: Modify protocol as needed for group interviews.]

A. Respondent Background and Responsibilities

Let’s start by discussing your background and responsibilities.

1. What is your official job title?

2. How long have you been working at [GRANTEE ORGANIZATION] in this position? When did your involvement in the [RF/HM PROGRAM] begin?

3. What types of groups do you facilitate for [RF/HM PROGRAM]? Do you have other roles within [RF/HM PROGRAM] in addition to facilitating groups?

If yes:

1. What are those responsibilities?

2. How much of your time is allocated to your role as a facilitator for the [RF/HM PROGRAM] versus these other responsibilities?

3. Do you have other professional responsibilities in other organizations or work as a facilitator for other groups?

4. Do you work as an employee of [GRANTEE ORGANIZATION] or do you work as a contractor for [GRANTEE ORGANIZATION] in providing workshop facilitation?

   1. If an employee, do you work full-time or part-time for [GRANTEE ORGANIZATION]?

   2. If a contractor, do you have a different job in addition to facilitating workshops for [GRANTEE ORGANIZATION]? What is that other job?

5. What education and prior job experience did you have when you assumed the position of facilitator for [RF/HM PROGRAM]?

6. Had you been a group facilitator previously? If yes, were the workshop topics and participants similar to those at [RF/HM PROGRAM]? If different, how?

If facilitated groups previously or currently facilitating groups elsewhere:

1. What types of groups did you facilitate previously?

2. How are the parenting and/or relationship classes you lead for [RF/HM PROGRAM] here different from other experiences you have had running groups?

7. What are the key qualifications and skills associated with your job position? What experience and skills do you believe are most important for someone in your position to be effective?

8. How, if at all, has your role or responsibilities changed since the implementation of the [RF/HM PROGRAM]? What led to these changes?

B. Strategies for program enrollment and retention; Motivation for enrolling; Satisfaction with program services; Participant characteristics

Ask next questions for each type of group activity offered.

9. [Interviewer: Refer to notes on participant characteristics.] How, if at all, is the population that you currently serve different from the population that you planned to serve when you applied for the grant? We’re interested in any demographic, socioeconomic, and geographic changes in the population, and also any changes in special groups that may be served, such as youth, reentering ex-offenders, or veterans. Please feel free to comment on any other characteristics you think are pertinent.

<If yes>

1. When did this change occur?

2. What led to this change?

3. Do you think this change in the population affected program enrollment?

<If no>

4. What has contributed to consistency in the target population?

10. What are the primary reasons that participants attend [FATHERHOOD/ PARENTING/RELATIONSHIP/JOB PREPAREDNESS] workshops offered through [RF/HM PROGRAM]? What is their primary motivation?

    1. What do clients hope to obtain from participating in the [FATHERHOOD/ PARENTING/RELATIONSHIP/JOB PREPAREDNESS] sessions?

    2. What do you tell them they will gain?

11. Based on your experience, what topics of [CURRICULUM NAME] seem to resonate well with fathers/couples and seem to help them the most?

12. What other aspects of the group sessions do fathers/couples seem to value and derive the most benefit from?

13. Can you provide any examples of changes in knowledge, attitudes, or behavior that you have seen in fathers/couples over the course of their participation in these groups? How common is this change among participants?

14. What has been the experience with no-shows and attrition? What are the most common reasons fathers/couples give for not coming to the sessions?

15. What efforts do you make to encourage high attendance and engagement? What efforts are made by other types of staff? What strategies, conducted by you or by other types of staff in the program, appear to be the most effective?

If provide incentives [NOTE TO INTERVIEWER – INCENTIVES NEED NOT BE FINANCIAL]:

1. Do you emphasize the participation incentives during the group sessions? Which incentives appear to be most effective in motivating fathers/couples to attend sessions and remain engaged in the program?

C. Delivery (including customization), dosage, and quality of program content; Integration of organizational partners

Ask next questions for each type of workshop offered and each curriculum mentioned.

16. [Interviewer: Refer to notes on delivery (including customization), dosage, and quality of program content.] It is my understanding that the [RF/HM PROGRAM] uses the [CURRICULUM NAME] to provide [FATHERHOOD/PARENTING/RELATIONSHIP/JOB PREPAREDNESS]. Is that correct?

17. We’d like to understand the topics covered in the curriculum and how you pace and sequence the topics. Is the sequence established, or do you have discretion over the sequence? If you have discretion, how do you determine the sequence?.

For parenting curricula:

1. Is information regarding child support presented?

2. What child support issues are covered?

3. What child support related issues are of greatest interest to participants?

18. How closely do you follow the curriculum?

    1. How much flexibility is there to adapt or modify the curriculum?

    2. Do you tailor the class content or format for the people who attend? How and why?

    3. Do you tend to spend more time on some topics than others? Which topics and why?

    4. Is there anything you regularly add to the curriculum to make it more useful or relevant for the participants?

19. Since [RF/HM PROGRAM] began to offer [CURRICULUM NAME], have there been any changes in the modules covered or topics addressed? When and why were these changes made?

20. Are there any aspects of the curriculum you would like to see modified? What aspects and why do you think this should be modified?

21. To your knowledge, are any modifications being planned or under consideration? If so, please describe.

22. Beyond covering the curriculum topics, what are the critical elements for making sessions of [CURRICULUM NAME] effective and engaging?

23. What workshops or curriculum group sessions does [RF/HM PROGRAM] offer to fathers/couples? Which of these workshops do you provide?

24. For each workshop offered, please answer the following:

    1. What is the format and structure of the group sessions?

    2. Where are the group sessions held?

    3. How many fathers/couples attend a group session, on average?

    4. What would be the ideal number of fathers/couples in attendance at this group?

    5. For RF programs only: How often does another person attend the workshop along with the father? If so, who does that person tend to be?

    6. For HM programs only: How often do couples attend the workshop as a couple? What happens when only one member of the couple shows up? Who serves as the member’s partner during couple exercises?

    7. How many staff generally attend each group session?

    8. Which types of staff attend and what are their roles?

If co-facilitate groups:

1. How do co-facilitators share the facilitation role?

2. Who comprise the co-facilitator pairings? Are all pairings male-female? Are any pairings married couples?

3. Are there any particular advantages or disadvantages with pairing facilitators?

9. How often do groups meet, for how long, and over what period of time?

10. Have different formats been tried?

If more than one format:

1. What are the pros and cons of these different formats?

2. Have you found that one format (number of hours, timing, etc.) works better than another?

11. When and where do you first meet other fathers/couples attending a group session? If you meet fathers/couples prior to the first group, what is the purpose of the meeting?

12. What is the process for scheduling/inviting people to attend a group workshop? Who handles that? Is there a limit on the number of people who can attend? How long will a father wait to begin attending a group workshop?

13. Has the process for scheduling/inviting people to attend a group session changed since the RF/HM grant program began? If so, how and why?

25. Some programs are structured so that participants are expected to attend a specific sequence of workshop sessions with the same group of people, while others allow participants to attend any ongoing sessions not necessarily in sequence or with the same group. Which approach is used at this program? What do you think are the benefits and drawbacks of this approach?

26. What additional parenting/relationship services are available to fathers/couples that are provided by [GRANTEE ORGANIZATION], including case management, employment assistance, and other services?

27. What role, if any, do you play in informing fathers/couples about these services, encouraging them to seek these services, and helping link fathers/couples to these services within the [GRANTEE ORGANIZATION]?

28. What additional parenting/relationship services are available to fathers/couples that are provided by other organizations in the community?

29. What role, if any, do you play in informing fathers/couples about these services, encouraging them to seek these services, and helping link fathers/couples to these outside services?

D. Staff recruitment, characteristics, training, and retention

30. What experience or qualities do you think are important for someone to be effective as a workshop facilitator?

31. In what ways do you and other case managers and employment specialists believe that the following services are valuable to fathers/couples?

    1. Parenting services

    2. Healthy marriage/ relationship services

    3. Economic stability services

32. Which services or strategies, if any, are you more skeptical are helping fathers/couples achieve their intended outcomes?

33. Which services, if any, are you uncomfortable providing?

34. In your view, to what extent do staff think that the services delivered by the program are the “right mix” in terms of range, type and intensity to help fathers/couples achieve their intended outcomes?

35. How do you work and coordinate with other staff in [RF/HM PROGRAM]? How frequently and under what circumstances do you share information with the other group facilitators, employment specialists, case managers, or other staff?

36. Has there been any staff turnover, either among the group facilitator positions or other program staff that have had an impact on the structure and staffing of group workshops? If so, please describe.

37. Have there been changes in the workshop facilitator’s role and responsibilities? If so, what are these changes and why were they made? When did the change(s) occur?

38. What training have you received on how to deliver [CURRICULUM NAME]?

    1. When did you receive this training?

    2. What issues or topics were covered?

    3. What formats were used (e.g., group format, on-line/self-administered)?

    4. How long was the training? (i.e., number of hours)

    5. Who provided this training (organization/individual)?

    6. Were you required to become “certified” to present the curriculum? If so, what did this involve?

    7. What is your assessment of how well this training prepared you to provide [CURRICULUM NAME]?

    8. In which areas or ways could it have been better?

39. Do you receive any ongoing training or technical assistance on delivering [CURRICULUM NAME]?

    1. What has been the focus of this training?

    2. When did you receive it?

    3. Who provided it?

40. What is your assessment of the quality and usefulness of the initial and ongoing training and guidance you have been given? How could it have been better?

41. Are there any kinds of trainings or resources that you would like to have access to that you do not have access to? What additional training or resources would you find useful for being an effective facilitator?

E. Adherence/modifications to implementation plans/logic model and Attainment of program targets

42. [Interviewer: Refer to notes on attainment of program targets.] How does your expected enrollment and retention benchmarks compare to your actual enrollment? If your actual enrollment and retention [EXCEEDS/DOES NOT MEET] your targets, to what do you attribute that difference?

    1. How is high quality and effective service delivery defined for groups workshops offered within [RF/HM PROGRAM]?

    2. What outcomes are used to gauge success? How do you use data to monitor these outcomes?

    3. How are you doing towards providing services that meet this definition?

43. To what extent are facilitators expected to provide group workshops in a consistent manner?

    1. Which aspects of group workshops are monitored for consistent delivery and how does monitoring happen?

    2. Why is consistent delivery important for these aspects of group workshops?

    3. How is consistent delivery defined for these aspects of group workshops?

    4. What are key challenges and barriers that impede you from delivering group workshops in a consistent manner?

    5. How has consistent delivery affected the program implementation outputs or outcomes?

    6. Have you considered any alternative approaches to the workshop that you decided not to pursue? What were these approaches, and why did you dismiss them?

F. Implementation drivers and successes; Implementation challenges and strategies for mitigating

We have covered a lot of information today related to your role as a workshop facilitator for [RF/HM PROGRAM] and I have learned a lot about the way in which your program operates and some of its challenges and successes. We have just one more question to wrap-up our discussion.

44. Would you summarize for me what you think most contributes to successful delivery of [CURRICULUM NAME]? What challenges have you faced in delivering the curriculum and how have you dealt with those challenges? To what extent do you take ownership for the success of the [RF/HM PROGRAM]?

Famle cross-site Project—Implementation Study

PArtner Organization Interview Guide

Interviewer instructions: Before beginning the interview, tailor this guide to focus on the additional details we need about the grantee and the program. In particular, be sure to probe for details on populations (such as youth aging out of foster care) and policy areas (such as connections to child support and intimate partner violence services) that are a high priority interest for OFA.

Thank you for taking the time to speak with us today. Mathematica Policy Research is conducting a study for the U.S. Department of Health and Human Services, called the Fatherhood and Marriage Local Evaluation and Cross-Site Project, or the FaMLE Cross-Site Project. [GRANTEE ORGANIZATION] is participating in FaMLE. We are gathering information to develop an in-depth understanding of how [GRANTEE ORGANIZATION] designed, implemented, and operates [RF/HM PROGRAM]. In addition to interviewing staff from [RF/HM PROGRAM], we want to learn how staff from partner organizations, such as yours, work with the [RF/HM PROGRAM].

During this interview, we would like to hear about the services your organization provides to the clients of [RF/HM PROGRAM]. We’ll also discuss your opinions about the [RF/HM PROGRAM’s] successes and things other organizations should know if they want to develop a similar program. We will use the information you share with us to write a summary of what we have learned about this program, which we will publish in a future report about this program and others, but we will not attribute any of your comments to you in our reports. We would like to record our discussion, however, only members of our team will have access to this recording and we will destroy it after we complete our analysis. Do we have permission to record this conversation? [OBTAIN PERMISSION AND START RECORDING IF PERMISSION GRANTED.]

[Note to interviewer: Modify protocol as needed for group interviews.]

A. Respondent Background and Responsibilities

Let’s start by discussing your background and responsibilities.

1. What is your official job title?

2. How long have you been working at [PARTNER ORGANIZATION] in this position? When did your involvement in the [RF/HM PROGRAM] begin? [If involvement began after grant was awarded: Did [PARTNER ORGANIZATION] begin working with [RF/HM PROGRAM] before you became involved? When?]

3. What types of job experiences did you have before becoming a [JOB TITLE] for the [PARTNER ORGANIZATION]? What did you do immediately prior to assuming this position?

4. Describe your overall role/involvement in [RF/HM PROGRAM].

B. Delivery (including customization), dosage, and quality of program content

Now, we want to explore your role within [RF/HM PROGRAM] and how [PARTNER ORGANIZATION] works with [RF/HM PROGRAM].

5. Please describe the core services provided by [RF/HM PROGRAM] and how [PARTNER ORGANIZATION] contributes to delivering these services.

   1. Parenting services (including information or assistance with child support issues)

   2. Relationship services (including intimate partner violence prevention or treatment)

   3. Employment services (including related services such as GED or occupational skills training)

   4. Other services (clothing for jobs, transportation, etc.)

6. Prior to [PARTNER ORGANIZATION’s] involvement with [RF/HM PROGRAM], what was [PARTNER ORGANIZATION]’s experience and investment in delivering parenting, healthy marriage, or employment services?

7. How do the services provided by [PARTNER ORGANIZATION] fulfill or support the mission of [PARTNER ORGANIZATION]?

8. How do the services provided by [PARTNER ORGANIZATION] fulfill or support the mission of [RF/HM PROGRAM]?

C. Integration of organizational partners

Next, we’d like to learn about [PARTNER ORGANIZATION].

9. Please describe the mission of and core services provided by [PARTNER ORGANIZATION].

10. What is the organization’s primary target population?

11. What sources typically fund programs offered at [PARTNER ORGANIZATION]?

12. When, how, and why did [PARTNER ORGANIZATION] become involved in providing services to fathers/couples in [RF/HM PROGRAM]?

13. Did you have an expectation for the number of referrals you would receive from the [RF/HM PROGRAM], or a general expectation for the volume of work you would be taking on from this program? Could you describe this expectation?

14. What agreement exists between [PARTNER ORGANIZATION] and [GRANTEE ORGANIZATION] to specify how [PARTNER ORGANIZATION] will enhance the services provided by the [RF/HM PROGRAM]?

    1. What are the roles and responsibilities of [GRANTEE ORGANIZATION] for [RF/HM PROGRAM]?

    2. Is there an MOU between [PARTNER ORGANIZATION] and [GRANTEE ORGANIZATION]?

    3. Does [PARTNER ORGANIZATION] receive any funding from [GRANTEE ORGANIZATION] to support this role?

15. How often and through what methods are you in communication with a staff member from [RF/HM PROGRAM]? Which staff member or members do you usually communicate with?

16. What topics are typically discussed during your communication with staff members from [RF/HM PROGRAM]?

17. Is information about specific clients ever shared between [PARTNER ORGANIZATION] and [RF/HM PROGRAM]?

<If so>

1. What type of client information is shared and how often is it shared?

2. How is this information shared?

3. How does sharing this information benefit each organization?

4. What, if any, additional information do you think would be useful to share across organizations and how would sharing this information be useful?

<If not>

5. Did either [PARTNER ORGANIZATION] or [RF/HM PROGRAM] face barriers with sharing information between organizations? What were these barriers and how were they addressed?

6. Do you think there is a need to share client information across organizations?

7. What information would it be useful to share?

8. Why do you think this information is not shared?

18. How might the communication between [PARTNER ORGANIZATION] and [RF/HM PROGRAM] be strengthened (if needed), and how would this improve your ability to provide services to fathers/couples in [RF/HM PROGRAM]?

D. Marketing, recruitment, and engagement strategies

19. Do you make referrals to [RF/HM PROGRAM]? [IF NO, SKIP TO SECTION E.]

If yes:

1. Are you co-located or outstationed to [GRANTEE] or another organization for recruitment? If so, how often does outstationing occur and how long are you at the organization?

2. Do you make informational presentations to your clients about [RF/HM PROGRAM]?

19. How formalized is the process between your organization and [RF/HM PROGRAM] for making referrals? Are there established criteria or other procedures to help guide whether a potential father/couple is an appropriate referral? Is there an MOU with the [GRANTEE ORGANIZATION]? Is your organization compensated in some way for providing referrals (i.e., are you a subcontractor to the grantee?)

19. What methods have provided the most referrals? What methods have produced referrals that consistently meet eligibility criteria, are most likely to go ahead and enroll in the program and engage in program activities?

19. What are the key messages prospective participants receive about [RF/HM PROGRAM] through your outreach? What aspects of the program are emphasized?

19. Have you tried different ways of describing the program to prospective participants? Which approaches appear to be most effective?

19. Why do you think these referral sources/methods have been the most successful? Why do you think other methods have been less successful?

19. What difficulties, if any, have been encountered with the referral process to date?

19. What are the major challenges your organization has encountered in trying to make referrals to the [RF/HM PROGRAM]?

E. Implementation drivers and successes; Implementation challenges and strategies for mitigating

We have covered a lot of information today related to your role as an organizational partner for the [RF/HM PROGRAM] and I have learned a lot about the way in which you help the program operate, and some of the program’s challenges and successes. We have just a few more questions to wrap-up our discussion.

19. Would you summarize what you think are the key factors that have facilitated the partnership between [PARTNER ORGANIZATION] and [RF/HM PROGRAM]? What would you like to see changed in your partnership with [RF/HM PROGRAM] to improve it or make it more successful?

19. To what extent do you and other [PARTNER ORGANIZATION] staff take ownership for the success of the [RF/HM PROGRAM]?

19. How do you think this partnership and the services provided by [PARTNER ORGANIZATION] strengthen [RF/HM PROGRAM]? How do you think the RF/HM program helps to strengthen your organization?

19. Are there lessons you have learned or is there advice you would like to share with other policymakers and practitioners interested in designing and implementing high quality and effective RF/HM programs that may involve partnerships across organizations?