Form C1 Appendix C1. PD Interview Guide

Evaluation of Learning Health Systems K12 Training Program

Appendix C1. Program Director (PD) Discussion Guide

May 18, 2020

Program Director Discussion Guide

Estimated Time: 60 minutes

Introduction

Thank you for taking the time to speak with us today as a part of the evaluation being conducted by 2M Research (2M) on behalf of the Agency for Healthcare Research and Quality (AHRQ). The purpose of this study is to learn more about your experience as an LHS K12 program director. We also will be speaking with scholars and their health system advisors about their experiences. This information will be used by AHRQ to ensure that program goals are met, to improve upon the program, and to understand the value of the LHS K12 training program to its stakeholders.

During this interview, you will be asked a series of questions about your perspectives on the LHS K12 training program. Our discussion should last approximately 60 minutes. There are no right or wrong answers—we just want to know about your experiences with the LHS K12 training program. You can refuse to answer any questions you do not want to answer. Your participation in this study is voluntary, and you may stop participating at any time. There will be no negative consequences if you choose to stop or if you choose not to participate at all.

All interview data will be treated in a secure manner and will not be disclosed, unless otherwise compelled by law. The information 2M collects will be aggregated and summarized for reporting to AHRQ and the public. Information about your institution’s program and comments from other scholars and staff will be aggregated. All analyses will be conducted using a de-identified data file.

There are no risks to those who participate. The benefit of participating in the evaluation is that your experiences will help AHRQ understand the design and conduct of the training programs, as well as the successes and barriers encountered while implementing the training program.

If you have questions about this study after this interview, please contact Dr. Amy Windham, the evaluation lead, at awindham@2mresearch.com or 703-214-1512.

Before we begin, we would like to have your permission to audio record our discussion so that we can ensure our notes are accurate and complete. The audio recordings will be deleted once the project is complete.

Can we audio record this conversation?

Do you have any other questions before we begin?

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is XXXX-XXXX. Public reporting burden for the collection of information is estimated to average 60 minutes per response. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer, Attention: PRA, Paperwork Reduction Project (XXXX-XXXX), AHRQ, 540 Gaither Road, Room #5036, Rockville, MD 20850.

Respondent Background

Let’s go ahead and get started! These first few questions are about you.

1. Would you start by telling me a little bit about your role and position with respect to the LHS K12 training program?

Probe: How long have you been in this role?

Probe: How many LHS K12 scholars have you worked with?

Probe: Can you describe the health systems that the LHS K12 scholars in your program have worked in?

Attitudes Toward the Value of Research Carried Out in Health Systems

I would like to ask you a few questions about your opinions about the use of research in the health systems that the LHS K12 scholars work in. When we talk about research, this could include clinical research, health services research, health systems research, or operations-related research, etc. When we talk about “health system,” we are referring to the health system or systems the scholars worked in. We understand that your scholars may have worked in different health systems, so you may need to answer in relation to each of them.

3. In your opinion, how important is it for research carried out in health systems to reflect system priorities?

3. Do you think leaders in the health systems of your program’s scholars share this view about research, or do you think they have a different view?
   
   
   

Probe: In what ways do you think it differs?

3. To what extent were health system priorities reflected in the research carried out by LHS K12 scholars in your program?

Use of LHS K12 Training Program Research by the Health System

Next, I’m going to ask you some questions about how research conducted by LHS K12 scholars was used by the health systems.

3. How did the scholars’ research impact patient care in their health systems?

Probe: Can you give some examples?

3. How did the scholars’ research inform quality improvement efforts (in their health systems)?

Probe: Can you give some examples?

3. How did the scholars’ research inform operations (in their health systems)?

Probe: Can you give some examples?

3. Did the scholars’ research result in reduced cost of care (in their health systems)?

Probe: Can you give some examples?

Patient/Family Engagement in Research

My next questions relate to patient and family participation in research.

3. In your opinion, how important is it that patients and their families participate in research carried out in a learning health systems research approach?

Probe: If the respondent asks for examples, the following are examples of how patients and their families may participate in research:

• Determining research topics or research questions

• Providing input on the execution of research projects

• Reviewing research results and providing input on their meaning or implications
  
  
  

3. How did the health systems support or facilitate patient and family engagement in scholars’ research?

3. How did the LHS K12 training program impact the capacity of health systems to engage patients and families in scholars’ research?
   
   
   

Probe: If yes, in what ways was capacity impacted?

3. How did the LHS K12 training program impact health system leaders’ attitudes toward patient and family engagement in research?
   
   
   

Probe: If yes, how have those attitudes changed?

Other System Stakeholder Engagement

The next few questions are about other health system stakeholders who were involved with the LHS K12 training program. Other stakeholders include stakeholders other than system leaders such as clinical staff or lab technicians, and others.

3. Were there key stakeholders within the health systems who were engaged in scholars’ research?

Probe: If yes, what were the roles of these key stakeholders? (e.g., clinical staff, nurses, technicians)

If no, skip to question 16

3. How were these stakeholders engaged in research?

Probe:

• Did they help determine research topics or research questions?

• Did they provide input or facilitate the execution of research projects?

• Did they review research results and provide input on their meaning, implications, or implementation?

LHS K12 Training Program Impact on the Health System

The next questions are about the impacts of the LHS K12 training program on the health system and its leaders. By health systems leaders, we mean C-suite leaders or other personnel responsible for quality and/or safety, strategy, or operations.

3. How did the LHS K12 training program shift health system leaders’ views on how research should be conducted in health systems?

Probe: If no: why do you think there was no shift?

3. How did the LHS K12 training program shift health system leaders’ views of how research results should be used by health systems?

Probe: If no: why do you think there was no shift?

3. How did the LHS K12 training program impact the extent to which system priorities were reflected in research conducted in the system?

Probe: What are some examples?

3. How did LHS K12 training program promote research focused on quality improvement?

Probe: If yes, in what ways did the training program promote research on quality improvement?

3. Did the LHS K12 training program impact the design or execution of other research carried out in health systems—that is, research carried out by non-LHS K12 scholars?

Probe: If yes, would you describe these changes?

3. What evidence have you seen that health system leaders are more engaged in the dissemination of research as a result of the LHS K12 training program?

3. What aspects of the LHS K12 training program contributed to changes in how research is conducted or how research results are used by health systems?

Sustainability of LHS K12 Training Program

We only have a few more questions. The last questions are about the LHS K12 training program moving forward.

3. Would you like to see the LHS K12 training program sustained or continued in your institution?

Probe: Why or why not?

3. What sources of funding would be required to sustain it?
   
   
   

4. Besides funding, what other resources would be required to sustain it?
   
   
   

5. Do you envision health systems funding or co-producing LHS researchers/scientists in the future?

Probe: Why or why not?

3. Do you envision health systems providing other resources or financial support for LHS research?

Probe: Why or why not?

3. What changes to the LHS K12 training program would make it more responsive to health systems’ needs or priorities?
   
   
   

4. What changes to health systems are needed to improve LHS research’s potential value to the system?

Closing

Is there anything else you would like to share with us regarding your experience as an LHS K12 program director?

We appreciate the time you have taken to answer our questions. Your input is very valuable. If you have additional questions following this interview or if you think of anything else you would like to add, please feel free to email us.