Form 1 Attachment A -- In-person Preliminary Interview Guide fo

Attachment A -- In-person Preliminary Interview Guide for LHS Leaders

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

Learning Health System Product Evaluation

Early Leadership Interview Guide (1)

For the interviewer:

The purpose of this interview is to learn about the experiences of Learning Health System (LHS) leadership and staff who are involved in: deciding to participate in this project; selecting the product(s); allocating resources to the project; and leading implementation of the product(s). Topics covered in the interview are LHS leaders’ and staff’s perceptions of the advantage and acceptability of the product(s), adoption in the LHS, appropriateness, feasibility, cost, and compatibility.

The in-person interviews, each lasting 60 minutes, will take place 1 to 3 months after the LHS has begun implementation of the product.

LHS and implementation product (e.g., Johns Hopkins, triage tool):

LHS leader or staff person name/role:

Interviewer Name:

Date:

Introduction

Thank you for agreeing to talk with us today. My name is ________________ and I will be conducting the interview. I am joined by _______________, who will be taking notes. We work with the American Institutes for Research (AIR), a not for profit organization that is supporting the Agency for Healthcare Research and Quality (AHRQ) in developing, implementing, and evaluating evidence-based products. The purpose of the products is to help learning health systems use the findings from Evidence-based Practice Center (EPC) systematic reviews.

The information you provide today will help AHRQ improve the uptake of products like the [PRODUCT NAME(S)] in learning health system settings.

Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, # 07W41A, Rockville, MD 20857.

In our conversation today, we would like to learn about your initial decision to choose the [PRODUCT NAME(S)] and your early experiences with the product. Specifically, we are interested in learning about how and why you chose the [PRODUCT NAME(S)], how you allocated resources to implement the [PRODUCT NAME(S)], and how implementation of the [PRODUCT NAME(S)] is going.

We are not evaluating you or your performance in any way. We are simply interested in hearing your perspectives and opinions. We have a series of questions to help guide our discussion. There are no right or wrong answers to these questions. If there are any questions that you do not feel knowledgeable about or do not feel comfortable answering, just let us know and we will move on.

With your permission, we would like to record today’s discussion. Our research staff will have access to the transcript from the interview. If, at any point, you would like to go off the record, please let us know and we will stop the recording. We will restart the recording only after you have allowed us to do so.

Do you have any questions before we begin?

Do we have your permission to record the discussion? [If yes, turn on recorder.]

Interview¹

We would like to start with a few questions about you and your role.

1. Please tell us about your main role and responsibilities at [LHS].

   1. How long have you been involved with [LHS] in this capacity?

   2. How did you become involved in the [PRODUCT NAME(S)] implementation?

1. Experience with other products

To begin with, I would like to know about your experiences implementing other products.

1. Have you or has your system used or reviewed any EPC Program products? [implementation update]

2. If so, what products did you use? [implementation update]

1. What was your experience with them? [implementation update]

2. How did you use those products? [implementation update]

3. Based on your experience so far, how does [PRODUCT NAME(S)] compare with these other products? [relative advantage, acceptability, and compatability]

2. Implementation plan update (Leaders who are involved with planning only)

We would like to get a sense of any changes made to the implementation plan and how implementation of [PRODUCT NAME(S)] is going. (Succinctly summarize and confirm what was learned as part of early planning and coaching calls.)

1. Is this summary of your organization’s product implementation approach correct? [implementation update]

1. Do you have anything to add to the summary? [implementation update]

2. If more than one product was chosen: What were the factors you considered when deciding to select both [PRODUCT NAME and PRODUCT NAME]? [relative advantage, acceptability, and compatability]

3. How do you see the products complementing each other?² [relative advantage, acceptability, and compatability]

2. How did the format and content of the [PRODUCT NAME(S)] influence your implementation approach? [implementation update]

1. How did previous product roll outs influence your approach to implementation for [PRODUCT NAME(S)]? [implementation update]

3. Have any changes occurred to the implementation plan from what was originally decided?

1. If so, what are those changes? [implementation update]

Probe: Changes to staff involved with the implementation, introducing the product(s), roles of the people who will be targeted to use the content of the product(s), or how the [PRODUCT NAME(S)]’s content will be introduced to end-users.

4. How has the implementation progressed? [implementation update]

1. When was/were the [PRODUCT NAME(S)] introduced to the implementation team? [implementation update]

2. How was/were [PRODUCT NAME(S)] introduced to the implementation team? To other groups? [implementation update]

3. Experiences with the product and fit to LHS

Now we’d like to ask you a few questions about your and your system’s early experiences and impressions of the [PRODUCT NAME(S)].

1. What were your early impressions of the product? [relative advantage, acceptability, and compatability]

1. Did you have strong feelings about the product(s) one way or another? [relative advantage, acceptability, and compatability]

2. What aspects of [PRODUCT NAME(S)] appealed to you the most? Why? [relative advantage, acceptability, and compatability]

3. What aspects of the [PRODUCT NAME(S)] appealed to you the least? Why? [relative advantage, acceptability, and compatability]

4. In general, how understandable is the information conveyed by [PRODUCT NAME(S)]? Why? [relative advantage, acceptability, and compatability]

2. How well has/have the product(s) supported your need for information on this topic so far?

1. Did [PRODUCT NAME(S)] include the right types of information? [relative advantage, acceptability, and compatability]

2. Was the information in the right amount? [relative advantage, acceptability, and compatability]

3. How well are you able to interpret and apply the evidence to your system’s circumstances? [feasibility]

4. What challenges, if any, have been encountered in using the [PRODUCT NAME(S)]? [adoption]

1. If there have been challenges, have you experienced these challenges with other evidence resources? [adoption]

2. If there have been challenges, are these challenges specific to certain users? [adoption]

5. What challenges, if any, have been encountered in using the product content? [adoption]

1. If there have been challenges with the product content, have you experienced these challenges with other resources? If so, please describe them. [adoption]

6. How well does this product/do these products fit the needs of your organization? [appropriateness]

1. How compatible is/are the product(s) with how your system usually accesses evidence to support decisionmaking? [relative advantage, acceptability, and compatability]

2. Is the content of the product(s) relevant to your patient population? [appropriateness]

1. If so, how? [appropriateness]

2. If not, why not? [appropriateness]

3. Does [PRODUCT NAME(S)] help solve a problem or a need that your system has? [appropriateness]

1. If so, how? [appropriateness]

2. If not, why not? [appropriateness]

7. Are there other products that are better suited or less suited to your system’s information needs compared to this one? [appropriateness]

1. If so, what are they? [appropriateness]

2. Did you or others access other related evidence use products after reviewing the product content? If yes, explain. [adoption]

8. Have you or others accessed other related EPC Program products after reviewing the content of [PRODUCT NAME(S)]? [adoption]

1. If so, what products? [adoption]

4. Use of product by others

In this next set of questions, we’d like to ask you about how this product could be used by others.

1. What groups do you see as the primary consumers of the evidence supplied by this product? [appropriateness]

1. How do you expect them to use the product(s)? [adoption]

2. Under which circumstances would you share the [PRODUCT NAME(S)] with this group or other health system leaders, clinicians, and patients? [feasibility]

3. Have you shared the content of the [PRODUCT NAME(S)] in other ways than sharing the product itself? [adoption]

1. If so, how and with whom? [adoption]

4. What was their reaction to the product(s)? [adoption]

1. Did they have any questions or concerns? [adoption]

5. What discussions, if any, are you having about the product(s)? [adoption]

2. What groups do you see as the primary consumers of the content within this product rather than the product itself? [adoption]

1. How do you expect them to use the product content? [adoption]

3. Will your health system act in any way as a result of reviewing the product(s) and content?

1. If so, how and in what regard? [actionability]

2. What content led to the decision to act or not to act? [actionability]

3. What next steps are usually followed after system leaders and staff have reviewed evidence? [adoption]

4. What else will be taking place in this case, if anything? [adoption]

E. Resource cost

Finally, we’d like to ask you a few questions about the time and training needed to use the [PRODUCT NAME(S)].

1. What types of resources are required to review the product(s)? [implementation cost]
   Probe: This might include staff time, meeting time, and program planning.

1. Was this more or less or about the same as you expected? [implementation cost]

2. What types of resources are required to share the product(s) with others? [implementation cost]

2. Was this more or less or about the same as you expected? [implementation cost]

3. What resources, if any, are saved as a result of having the product(s) and content? [implementation cost]

4. Do you feel that there are any other benefits from having the product(s) available or drawbacks? [implementation cost]

1. If so, what are they? [implementation cost]

2. If not, why? [implementation cost]

F. Wrap up

We have just a few more general questions before we wrap up.

1. Is there anything about the implementation and the [PRODUCT NAME(S)] that you would like to add that we have not covered?

2. What did you originally think about the EPC Program systematic reviews?

1. Has using [PRODUCT NAME(S)] changed the way you think about EPC Program systematic reviews?

2. If so, how?

3. Do you have any questions for us?

Thank you so much for your time.

1 The interviewer will tailor the questions to the product being piloted. If more than one product is selected, the interview questions will be asked of each product. In cases where the products are used for different purposes, the interviewer will refer to the product that is most applicable to the question.

2⁵ The interviewer will tailor the questions to the product being piloted. If more than one product is selected, the interview questions will be asked of each product. In cases where the products are used for different purposes, the interviewer will refer to the product that is most applicable to the question.