Site Visit Interview Guide: Evaluator

DRAFT Site Visit Interview Guide for Evaluator

Version: June 17, 2019

Thank you for agreeing to speak with me today. My name is [name], and I work at John Snow Inc., or JSI, which has been contracted by the HIV/AIDS Bureau within the Health Resources and Services Administration to build the Ryan White HIV/AIDS Program Compilation of Best Practices and Interventions (i.e., Best Practice Compilation). This Best Practice Compilation will be housed on TargetHIV.org and allow organizations to learn about and potentially replicate intervention strategies shown to positively influence HIV care continuum outcomes. We are speaking with you because your organization submitted an emerging strategy to include in the Best Practice Compilation. Thank you! As you may know, intervention strategies submitted by Ryan White HIV/AIDS Program (RWHAP) recipients and subrecipients are scored based on pre-defined criteria for inclusion. The purpose of today’s interview is to learn more about your emerging strategy to ensure the submission form captured the appropriate and most useful information. We also want to get your feedback on the submission form content, scoring criteria, and the Best Practice Compilation summary produced by the information you submitted.

I was planning to record the interview for notetaking purposes only. Is that fine with you?

Interviewee and Organizational Background

Let’s start with introductions and a quick background of your organization.

1. What is your name and title?

2. What are the primary services your organization currently provides to people with HIV? Which of these services are funded by the RWHAP?

Intervention Strategy

Thank you for that background. Now, we would like to learn more about the emerging strategy you submitted. Based on your submission form, your emerging strategy involves [brief description of the emerging strategy]. I want to make sure that what we ask on the submission form accurately captures information about your emerging strategy and we are not missing anything that might be useful to other RWHAP recipients and subrecipients as they consider and plan for implementing your intervention strategy. Let’s walk through the various aspects of the form, and you can provide additional detail.

Service Delivery Model

First, I want to make sure I understand the emerging strategy’s service delivery model.

1. Could you walk me through the intervention strategy itself? [If an implementation guide/manual is available, walk through aspects of implementation.]

   1. How is an individual enrolled or referred into the intervention strategy?

      1. Does the intervention strategy intentionally target a specific population? If so, what is the target population and how are individuals identified? If not, what are the characteristics of the clients who tend to participate?

   2. What services are provided/activities implemented through the intervention strategy?

   3. Who provides the services (e.g., staffing model), where, and at what frequency?

   4. Does the intervention strategy have a specific duration for a given individual or is it ongoing?

   5. How many people have been served so far? Do you have a target goal?

2. What are the short and long-term goals of the intervention strategy? How do these goals relate to the steps along the HIV care continuum? [If the recipient/subrecipient has a Logic Model, walk through the aspects of the model.]

3. What is innovative about your intervention strategy? We define innovation as not being widely implemented by RWHAP providers and/or one that is related to an emerging HAB priority or area of focus.

Development and Sustainability

Now, let’s talk about how your emerging strategy was developed, its resource requirements, and your plans for sustainability.

1. When did your organization start implementing this intervention strategy?

2. How did you identify the service need?

3. How was the intervention strategy designed? Was it based on a pre-existing intervention strategy? Did the intervention strategy come from leadership or did you need to gain leadership buy-in for implementation? If the latter, how did you achieve buy-in? What stakeholders were involved in the design?

4. Why did you think this intervention strategy would best meet your client need? Was there any evidence supporting this? If so, what?

5. What were the resource requirements for getting the intervention strategy up and running (e.g., physical space, staff time, supplies, equipment/technology)? How did you fund this startup phase? What was the process for hiring staff and/or obtaining necessary supplies, space or equipment/technology?

6. What are the resource requirements for maintaining the intervention strategy (e.g., staff, supplies)? How are you funding ongoing maintenance? What are your strategies for retaining staff and ensuring you continue to have the supplies needed?

7. Were there any barriers to development and implementation? If so, what were/are they? How did you address them? What advice or lessons learned would you like to share with other RWHAP recipients and subrecipients that might implement a similar intervention strategy?

8. How sustainable is the intervention strategy in the long-run? Do you have a steady and sustainable funding source? Do you face any non-financial sustainability issues, such as staff retention?

9. If the intervention strategy is no longer operating, why not?

Success/Results

We would also like to hear about your process for assessing success of the intervention strategy. Once again, you provided some of this information already through the submission form, so we can walk through what you submitted, and you can add detail.

1. How are you assessing the success of the intervention strategy?

2. Have you been assessing impact at multiple points in time? How often?

3. What are your initial findings, if any?

4. [If impact] Do you think other RWHAP recipients and subrecipients could experience similar levels of impact if they implemented a similar intervention strategy?

   1. Why or why not?

   2. Are there any aspects about your intervention strategy that might make it hard to replicate to achieve similar outcomes (e.g., unique client population, funding source, implementing staff, leadership buy in)?

5. How are evaluation findings shared with program staff? Are they used for continuous quality improvement purposes? Could you provide an example of how the intervention strategy was changed based on findings?

[If the provider indicates that the intervention strategy in undergoing a more extensive evaluation or has plans to expand the evaluation, probe with the relevant questions below.]

6. Who conducted (is conducting) the evaluation (e.g., external evaluator, internal quality assurance staff)?

7. Please briefly describe the quantitative evaluation methodology (e.g., pre/post, uncontrolled studies, trend analysis, observational studies).

8. What are the quantitative metrics you’re using to evaluate the impact? Why and how did you select those metrics?

9. What are the data sources (e.g., survey data, electronic health record (EHR))? What are some of the challenges associated with collecting, managing or analyzing those data?

10. Are you also collecting qualitative data through client or staff interviews or focus groups, for example?

Feedback on Form, Criteria, and Content

Now, I’d like to get your direct feedback on the submission form, criteria and content of the Best Practice Compilation.

Submission Form

1. How long did it take you to fill out the submission form? Were there any aspects that were particularly burdensome? Were there any questions that were difficult to answer?

2. Were there any aspects of the instructions that were confusing? If so, how would you edit the instructions for clarify?

3. Based on what we discussed about your intervention strategy, it appears that the submission form captured [information] well, but did not capture [other information] well. Is there anything else you would like to change about the form to better capture the spirit and impact of your intervention strategy and help other RWHAP recipients and subrecipients replicate it?

Criteria

Now, let’s talk through the criteria and how your emerging strategy was scored. We want to make sure that the emerging strategies included in the Best Practice Compilation meet our definition. They should be innovative practices that are not yet supported by published evidence or an extensive evaluation, but have demonstrated effectiveness in the field. They should address emerging priorities and areas of focus. We’ve scored your emerging strategy based on these criteria. It’s possible that the scoring doesn’t accurately reflect your intervention strategy because we may have misunderstood information submitted, or key information was missing from the submission form. [If the intervention strategy should have been categorized as evidence-based or informed, instead of emerging, discuss reasons for the confusion.]

[Walk through each question on the scoring criteria form, discussing the score and why it was selected. For each score, ask the below questions.]

1. Does this score seem right to you? Why or why not?

2. What changes would you suggest for the criteria to more accurately reflect your intervention strategy?

Content of the Best Practice Compilation

Finally, we would like to talk through the content of the Best Practice Compilation – how the information about your emerging strategy is accessed by various filters and displayed.

1. Recipients and subrecipients can search for intervention strategies based on filters/search terms related to population reached, type of intervention strategy, outcome, etc. [Show the interviewee a mockup of the compilation.] Do these filters and categories make sense? Is there anything else you would like to see to help you find intervention strategies of interest?

2. Your emerging strategy is categorized by [categories]. Does that seem right to you? If not, how would you update the categories?

3. Here is the profile that the Best Practice Compilation generates for recipients and subrecipients about your emerging strategy. [Show the interviewee a mockup of the profile.] Do you feel the profile does a good job capturing the essence of your emerging strategy? Why or why not? What changes would you make?

4. Now, imagine you have identified a different intervention strategy on the Best Practice Compilation that you are considering implementing. Do you have any suggestions for additional content? If so, what additional information would help you replicate intervention strategies you identify through the Best Practice Compilation?

5. Do you have any suggestions for the intervention strategy profile layout to help you locate key information more easily?

Thank you so much for sharing more about your emerging strategy and providing feedback on the Best Practice Compilation. Is there anything else you’d like to add?

3