Supporting Statement

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number:0935-0179)

TITLE OF INFORMATION COLLECTION:

Stakeholder Interviews for Task 4 of AHRQ’s ACTION III Diagnostic Safety Capacity Building Contract (TORFP: 75P00119R00265)

PURPOSE:

The MedStar Health Research Institute was awarded a contract with AHRQ to provide program support and expertise related to improving diagnostic safety and quality across five distinct contract tasks. Task 4 of the contract is to develop, pilot test and promote a resource to engage patients and families in the diagnostic process in order to reduce diagnostic errors. To execute this task, the contractor has assembled an interprofessional team to develop a patient and family resource that will provide both patients and clinical practices with simple approaches to close prevalent gaps in care leading to diagnostic failures.

A qualitative approach to data collection (e.g., individual interviews and/or focus groups) and analysis will be used to pilot test the resource and provide useful insights on stakeholders’ perceptions and opinions, identify barriers and facilitators to resource adoption, receptivity to the resource by stakeholders, as well as to explore general stakeholder feedback to enhance the usability of the resource. The approach will not consist of statistical surveys that yield quantitative results that can be generalized to the population. Pilot testing will occur at up to 20 primary care sites and the feasibility of implementation will be assessed at the stakeholder and practice levels. The information collected from stakeholders will be used to revise the resource in order to promote widespread adoption.

DESCRIPTION OF RESPONDENTS:

Respondents will include stakeholders representing patients, family members, clinicians, practice staff and administrators.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[ ] Usability Testing (e.g., Website or Software [ ] Small Discussion Group

[ ] Focus Group [X] Other: Stakeholder Interviews and/or Focus Groups

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: Margie Shofer

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [ ] Yes [X ] No

2. If Yes, is the information that will be collected included in records that are subject to the Privacy Act of 1974? [ ] Yes [ ] No

3. If Applicable, has a System or Records Notice been published? [ ] Yes [ ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [ X ] Yes [ ] No

The incentive is to compensate the patient participant for the burden of the information collection, particularly for participants from low income populations. A $25 stipend will be offered for a 60-minute focus group for patients and family members only. From prior direct experience, failure to offer any incentive of an amount lower than this will hamper the ability to successfully recruit the number of patients required for the proposed data collection and will result in the significant under-representation of patients from vulnerable communities and AHRQ priority populations.

Category of Respondent: (the options here are Public Sector or Private Sector, or both)

Both

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time	Burden
Practice Champions Medical and Health Services Managers (Code 11-9111)	20	1.75	35
Patients Misc. Healthcare Worker (Code 29-9090)	160	1	160
Clinicians Family Medicine/General Practitioners (Code 29-1062)	120	0.75	90
Practice Staff HC Support Occupations (Code 31-000)	120	1	120
Practice Administrators Medical and Health Services Managers (Code 11-9111)	60	1	60
Patient Care Team Two HC Support Occupations (Code 31-000) positions	200	0.5	100
Totals			565

FEDERAL COST: The estimated annual cost to the Federal government is $23,630.7

Grade	Number of Hours	Value
Medical and Health Services Managers (Code 11-9111)	95	$5,194.6
Misc. Healthcare Worker (Code 29-9090) [patient equivalent]	160	$3,537.60
Family Medicine/General Practitioners (Code 29-1062)	90	$8,424.90
HC Support Occupations (Code 31-000)	120	$2,427.60
Two HC Support Occupations (Code 31-000) positions	100	$4,046.00
Total	565	$23,630.70

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [ ] Yes [ X] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

Practices will be recruited from the networks of our four contract partners (MedStar Health, Baylor, Clinical Directors Network, and National Nurse-led Care Consortium). These partners represent the following types of practices:

1) MedStar Health - 100+ academic and community practices in urban, inner-city, rural, and suburban locations across Maryland, Washington, DC and Virginia

2) Baylor - academic practices in urban, rural, and suburban locations in Texas

3) Clinical Directors Network - represents more than 1,000 practices across 17 US states with more than two-thirds of the practices being Federally Qualified Health Centers

4) National Nurse-led Care Consortium – represents over 10,000 practices across 50 states and Washington, DC that are nurse-led (Nurse Practitioner or Doctor of Nursing Practice). The network also represents Migrant Clinic Networks.

Respondents (clinicians, staff, patients and families) will be recruited from within the 20 sub-contractor practices who are pilot testing the materials. 

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[ ] Web-based or other forms of Social Media

[X ] Telephone

[X ] In-person

[ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used? [ X ] Yes [ ] No

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Appendix A – Practice Descriptive Characteristics

Appendix B – Pilot Test Evaluation Protocol for Patients and Family Members

Appendix C –Organizational Readiness for Implementation Change (ORIC)

Appendix D – Pilot Test Interview Protocol for Ambulatory Care Providers

Appendix E – Pilot Test Evaluation Protocol for Practice Staff

Appendix F – Pilot Test Evaluation Protocol for Practice Administrators

Appendix G– Observation Tool

Appendix A – Practice Descriptive Characteristics

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

Diagnostic Safety Capacity Building – Patient and Family Resource

Please complete the following information about your practice:

General Information About Your Practice

Practice Name
Location (City, State)
Select one:	Urban Inner City Rural Suburban Other (Specify)
Contact Person
Medical Director
Number of	Physicians	__________
	Nurse Practitioners	__________
	Nurses	__________
	Medical Assistants	__________
	Pharmacists	__________
	Social Workers	__________
	Case Managers Other Practice Staff	__________ __________
	Other (specify)	__________
Total Number of Patients Served by Practice
Payer Mix (Indicate % of Patients)	Self-Pay Medicare Medicaid Private Insurance Uninsured Other	_________% _________% _________% _________% _________% _________%
Race (indicate % of patients)	White Black or African American American Indian or Alaska Native Asian Native Hawaiian or Other Pacific Islander	_________% _________% _________% _________% _________%
Ethnicity (indicate % of patients)	Hispanic or Latino Not Hispanic or Latino	_________% _________%

Information about Patient Safety and Quality Improvement Activities of the Practice

	Yes	No
Does your practice routinely conduct a patient safety culture survey?	☐ Please specify which survey you use: _____________ Date of the last survey ________	☐
Is your practice part of a larger healthcare system?	☐ Please indicate which health system you are affiliated with: ___________________	☐
Is your practice currently working on any other practice improvement strategies?	☐	☐
Does your practice have or use the services of a practice facilitator?	☐	☐

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

Appendix B – Pilot Test Evaluation Protocol for Patients and Family Members

AHRQ – Building Diagnostic Safety Capacity

Patient Focus Group Script – Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct up to twenty (n=20) focus groups (one per practice) with patients during Pilot Testing. We will aim to recruit patients in the following manner:

• 6-8 patients, family members, and/or caregivers per primary care practice.

• Patients/family members who have been exposed to the intervention

• A diverse group of patients in terms of age, sex, race/ethnicity, income, education, and current self-reported health status.

Recruitment Criteria

Patients and family members who have been exposed to the intervention (i.e. attended the primary care practice during the period of implementation) will be eligible to participate in the focus groups. MHRI team members will work with the practice coordinators to identify and recruit patients to participate in the focus groups.

Focus Group Goals

The goals of the focus groups will be to:

• Obtain feedback on the intervention patient-focused materials

• Obtain feedback on the barriers and facilitators encountered for each intervention

• Obtain feedback on satisfaction with interventions

• Obtain feedback on receptivity and enhancements to intervention to improve adoption

Focus Group Materials

• Copies of the patient-focused materials implemented by the practice during the period of implementation

• Informed consent documents

• Paperwork for processing the Participant stipends

• Digital recorder

Focus Group Location

Focus groups will be conducted at a location within the practice’s community. Locations may include libraries and/or community centers.

Participant Stipends

Upon arriving at the focus group location and after the completion of the informed consent process, all participants will complete the required paperwork (W9) to receive the stipend for participation. The stipend for participation will be $25.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the focus group.

Proposed Agenda – Patient Focus Group

Focus Groups are approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	20 minutes
Enhancements	10 minutes
Closing	5 minutes
Total	60 minutes

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

AHRQ - Building Diagnostic Safety Capacity

Patient Focus Group Script – Pilot Test Evaluation

[bracketed text will depend on interviewee and topic]

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the focus group about the strategy to improve patient and family engagement to improve diagnosis.

• My name is [INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT STAFF] and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with engaging with your care team on diagnosis.

• Nothing that you say or share today will impact your treatment or care from your doctor’s/Nurse Practitioner’s office. We will not share your name or anything that you say with them.

• Do you have any questions before we begin?

GROUND RULES

• We want to hear from everyone and want to hear your honest opinions. There are no wrong answers.

• If you have something to add to the conversation, please feel free to jump in. We do have a lot to cover so we will try not to spend too much time on any one topic.

• It is important that when you do jump in that we try to make sure that we only have one person talking at any time. This will help us hear everyone’s thoughts and opinions.

• Any questions?

So, let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

BRIEF INTRODUCTIONS

I would like to start with some brief introductions. We will go around the table. As we go around, please tell us all your first name and something about yourself like your favorite hobby or television show.

BACKGROUND

That is great! I am really pleased to meet you all. I would like to jump right in and start asking for your impressions about improving the process of diagnosis.

DIAGNOSIS

1. How would you describe the process of discussing your healthcare symptoms with your doctor/nurse practitioner?

   1. What goes well?

   2. What could go better?

   3. How would you change the encounter if you could to improve the process of getting a diagnosis?

   4. Can you describe what might help you to better engage with your clinicians to help share your symptoms?

2. How do you think that patients or their family members can engage with their healthcare team to improve the process of diagnosis?

   1. What about asking questions? Is there something that might help you ask questions easier during your encounter?

   2. What do you do to prepare for your visit?

   3. Do you ever forget to ask the doctor or nurse something during your visit?

Your doctor’s office has been implementing a new strategy to improve diagnosis. We would like to ask you about your experiences with some of the materials you may have seen in your doctor’s office.

3. What is the first thing that comes to mind when you see this [Agenda Setting Tool/Poster]?

   1. Have you seen these materials before?

      1. Where exactly did you see them?

      2. Did you pick them up or ask anyone about them?

         1. Why? Why not?

   2. Did you think they were helpful? Why/Why not?

   3. What would make them more useful to you?

4. Can you describe for me how you used/would use the [Agenda Setting Tool/Poster]?

   1. Did you find it helpful?

   2. What would have made it better?

   3. Did you feel like you needed more information about why you should use it?

   4. Were you able to get help if you needed it?

5. If you could change one thing about the materials, what would that be?

   1. If nothing to change:

      1. Did you personally get a chance to use these tools?

      2. Did your family member?

      3. Can you describe their experience?

6. What about the format of the [Agenda Setting Tool/Poster]? Is there a better way for us to think about presenting the materials?

Is there anything else you would like to share about your experiences with the materials? If not, let’s move on to learning more about your experiences with the Agenda Setting Tool/Poster.

7. We want to get some more information about your thoughts on [Agenda Setting Tool/Poster]. Where did you first hear about the [Agenda Setting Tool/Poster]?

   1. Who first brought it to your attention, your doctor, another organization?

   2. How long ago, or when did you first get introduced to the [Agenda Setting Tool/Poster]?

8. When you were first given the [Agenda Setting Tool/Poster] what did you think?

   1. How did you feel about using it?

   2. How did you end up using the [Agenda Setting Tool/Poster], or did you end up not using it after all?

   3. If no, what prevented you from using it? Time? Challenges with the materials?

   4. Do you plan on using the [Agenda Setting Tool/Poster] in the future? If yes, how. If no, why not? Is there something we could do to help you with making it more usable?

9. How easy or challenging did you find the [Agenda Setting Tool/Poster] to be?

10. Specifically thinking about the clinical encounter and getting a diagnosis, how did you think this the [Agenda Setting Tool] addressed safety for you?

11. And what about patient engagement, how did the [Agenda Setting Tool/Poster] help you to engage or increase your engagement in your care?

CLOSING

12. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project!

Appendix C – Organizational Readiness for Implementation Change (ORIC)

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

Organizational Readiness for Implementation Change (ORIC) – Practice Leader/Administrator/Practice Champion

MedStar Health Research Institute (MHRI) will conduct an interview with up to twenty (n=20) practice champion/administrator/provider to assess practice readiness to change. The information is collected at the practice level, not the individual level.

Recruitment Criteria

Any practice champion/administrator/provider that has practice level line of sight on organizational and operational priorities may complete the practice-level survey.

Location and Schedule

The change readiness survey will be completed online. The survey will take approximately 12 minutes to complete.

Informed Consent Procedures

Informed consent will be completed online with a survey cover page.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 15 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

Organizational Readiness for Implementing Change (ORIC)

	1	2	3	4		5
	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree		Agree
People who work here feel confident that the organization can get people invested in implementing this change.					1 2 3 4 5
People who work here are committed to implementing this change.					1 2 3 4 5
People who work here feel confident that they can keep track of progress in implementing this change.					1 2 3 4 5
People who work here will do whatever it takes to implement this change.					1 2 3 4 5
People who work here feel confident that the organization can support people as they adjust to this change.					1 2 3 4 5
People who work here want to implement this change.					1 2 3 4 5
People who work here feel confident that they can keep the momentum going in implementing this change.					1 2 3 4 5
People who work here feel confident that they can handle the challenges that might arise in implementing this change.					1 2 3 4 5
People who work here are determined to implement this change.					1 2 3 4 5
People who work here feel confident that they can coordinate tasks so that implementation goes smoothly.					1 2 3 4 5
People who work here are motivated to implement this change.					1 2 3 4 5
People who work here feel confident that they can manage the politics of implementing this change.					1 2 3 4 5

Appendix D – Pilot Test Interview Protocol for Ambulatory Care Providers

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

AHRQ – Building Diagnostic Safety Capacity

Provider Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews and/or focus groups with providers from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 120 cognitive interviews with ambulatory care providers (up to 6 providers per practice x 20 practices); each interview will last approximately 45 minutes

Recruitment Criteria

We will aim to recruit providers and practice staff in the following manner:

• Providers and practice staff who have been exposed to the PFE Resource

• Diversity among practice staff and providers

MHRI staff will work with the practice coordinators to identify individuals to participate in the interviews.

Interview Goals

The goals of the focus groups/interviews will be to:

• Obtain feedback on the intervention Guide materials

• Obtain feedback on the barriers and facilitators encountered for the Guide

• Obtain feedback on satisfaction with the Guide

• Obtain feedback on receptivity and enhancements to the Guide to improve adoption

Materials

• Copies of the PFE Resource materials

• Informed consent documents

• Documentation for participant stipends

• Digital recorder

Location

Interviews will take place at the primary care practice at a time convenient to the provider and/or practice staff members. Interviews may also take place over the phone to enhance ability to recruit and retain clinicians.

Informed Consent Procedures

Participants will complete the informed consent process prior to starting the interview.

Participant Stipends

None.

Each interview will take no more than 45 minutes.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 45 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

AHRQ – Building Diagnostic Safety Capacity

Provider Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

WELCOME AND INTRODUCTION

• Thank you for agreeing to speak with me!

• My name is [INSERT NAME OF INTERVIEWER] and I am here to ask you a few questions about the AHRQ’s Patient and Family Engagement (PFE) resource to improve diagnosis.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and we are sure that all of your comments are accurately.

• TODAY/TONIGHT I will be asking you questions about your experiences with the AHRQ’s Patient and Family Engagement (PFE) resource to improve diagnosis.

• Everything you say here will be kept confidential and included as part of our assessment of the feasibility of implementing the AHRQ’s PFE resource to improve diagnosis into practice. We will not share your name or attribute any of your words directly to you.

• Do you have any questions before we begin? Ok, great. Let’s get started.

DIAGNOSIS

1. How would you describe the impact of the AHRQ’s PFE resource on the diagnostic process in your personal practice?

   1. Why do you think that it was improved/not changed?

   2. Can you share with me any thoughts on what could be improved as part of the PFE resource that could make it more effective?

      1. What about changes to the patient poster? Agenda setting tool?

      2. How about the implementation planning toolkit?

      3. Training toolkit? What worked well? What would you do to improve it?

   3. How difficult was it to implement the strategy of allowing the patient to have the first minute of the clinical visit to tell their story?

      1. What impact do you feel this strategy had?

      2. Did you learn anything more from it compared to your usual practice?

      3. What was the patient’s response to this approach? How did you handle that?

2. In your opinion as a healthcare provider (doctor, nurse practitioner), what would you say are the most important things for patients to know or think about when it comes to improving diagnosis?

   1. Can you describe how this PFE Resource helped you to achieve this?

   2. What could or should we do differently to make this more effective?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

3. From your experience, how did the PFE Resource materials support engagement from the patient and or their family in the diagnostic process?

   1. Can you describe any barriers to engagement that you observed?

   2. How would you describe the level of engagement you had with patients and families after you implemented the PFE Resource?

   3. From your perspective, what part of the intervention made the greatest impact?

Now I would like you to consider the materials that you have in front of you for this next series of questions.

4. Can you describe for me how you used the (Planning toolkit, patient resources, training toolkit)?

   1. Did you find it helpful?

   2. Was the process difficult to follow?

   3. Was the training and education about how to use the PFE Resource appropriate? What changes would you make to improve it?

   4. Did you feel like you needed more information about how and/or why you should use the PFE resource?

5. What about the format of the materials? Is there a better way for us to think about presenting the materials?

   1. To patients? To clinicians? To Administrators?

   2. What about an electronic version? If you had this on your phone or another electronic mobile device would that help?

Is there anything else you would like to share about your experiences with the materials? If not, let’s move on to learning more about your experiences with the PFE Resource.

6. When you were first given the PFE Resource what did you think?

   1. How did you feel about using it?

   2. Can you describe your practice’s implementation process?

   3. How did you make decisions on which elements of the PFE Resource to use?

7. How easy or challenging did you find the PFE Resource implementation to be?

   1. Were any elements of the PFE Resource that were easier or harder to implement? If yes, can you describe which ones and what made them more challenging?

   2. What can we do to make them more effective? Easier to implement? More relevant to your practice’s workflow or patient population?

8. What was the best thing about the Resource from your perspective?

9. What was your least favorite thing about the Resource?

10. What would you have changed about the Resource to make it more user friendly?

11. What do you think your patients felt about having these tools available to them?

    1. Can you give me an example of a patient experience that was positive?

    2. How about a negative one? Is there anything that could have been done to make it a more positive experience for that patient?

12. What about your practice staff? Where they on board with the change?

    1. Did they like the new approach?

    2. Was it difficult to get buy-in from them?

13. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project!

Appendix E – Pilot Test Evaluation Protocol for Practice Staff

AHRQ – Building Diagnostic Safety Capacity

Staff Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews and/or focus groups with staff from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 120-160 cognitive interviews with ambulatory care staff (6-8 staff members per practice x 20 practices); each interview will last approximately 60 minutes

Recruitment Criteria

Ambulatory care practice staff who have been exposed to the PFE Resource within their practice will be eligible to participate in the focus group. MHRI team members will work with the practice champions to identify practice staff to participate in the focus groups. We will aim to recruit practice staff in the following manner:

• Staff members who were involved in the implementation of the PFE Resource or how have had experience using the PFE Resource

Focus Group Goals

The goals of the focus groups will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Focus Group Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Focus Group Location

Focus groups will be conducted at a location within the practice or within the practice’s community.

Participant Stipends

None.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the focus group.

Proposed Agenda – Practice Staff Focus Group

Focus Groups will be planned for approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	15 minutes
Enhancements	15 minutes
Closing	5 minutes
Total	60 minutes

AHRQ – Building Diagnostic Safety Capacity

Staff Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the focus group about your experiences with the Agency for Healthcare Research and Quality’s Resource for improving patient and family engagement in the diagnostic process!

• My name is [ INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT STAFF and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with using the PFE Resource and your practice’s experience implementing it.

• Nothing that you say or share today will be shared in an identifiable way with your practice but represented together along with nine other practices in a report to the Agency for Healthcare Research and Quality. We will not share your name or anything that you say with them in a personally identifiable way.

• Do you have any questions before we begin?

GROUND RULES

• We want to hear from everyone and want to hear your honest opinions. There are no wrong answers.

• If you have something to add to the conversation, please feel free to jump in. We do have a lot to cover so we will try not to spend too much time on any one topic.

• It is important that when you do jump in that we try to make sure that we only have one person talking at any time. This will help us hear everyone’s thoughts and opinions.

• Any questions?

So let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

I would like to begin by asking you all some questions about diagnostic error and the diagnostic process in your practice.

DIAGNOSIS

1. How would you describe the impact of the PFE Resource on the safety of the diagnostic process in your practice?

1. Why do you think that it was improved?

2. What strategies within the PFE Resource (agenda setting tool, poster, one-minute of patient talking uninterrupted) do you feel had the greatest impact on the process? Can you elaborate on why you think it had that impact?

3. Given your experience with the PFE Resource, what was most effective in your practice?

   1. Did you do anything to make changes to the materials or approaches to fit your practice? Can you describe those changes? Did that work? Why/Why not?

   2. Now that you have had some experience with the PFE Resource in your practice, what would you change about it? Why? What do you think that change would achieve?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

2. From your experience, how did the PFE Resource support engagement from the patient and or their family in the process of getting an accurate and timely diagnosis?

• Can you describe what you observed when patients/family members used the agenda setting tool? The poster?

• Were you able to observe the patient and/or family owning that first minute of the visit to tell their diagnosis story?

• How would you describe the level of engagement you had with patients and families after you implemented the PFE Resources?

Now I would like you to consider the materials that you have in front of you for this next series of questions.

3. I would like to focus on the practice’s implementation of the PFE Resource. Were you part of the implementation team? Can you describe for me how your practice advised the team about implementation?

• Did you use the quick start implementation guide? Do you have any feedback about what was missing in the implementation guidance? Anything that should have been there that would have been helpful? What about things in it that weren’t helpful?

• What could have made the implementation guide more useful?

4. Can you describe for me how your practice implemented the patient-facing resources? Specifically, the agenda setting tool and the poster. Did you use both?

• How did the patient’s respond to this new role?

• Can you describe the intended workflow for the patient materials? Who was responsible for orienting the patient? How was this accomplished?

  1. Was the process disruptive to the practice’s workflow?

  2. Did you feel like you had enough education and training on how to orient patients to the tools? What would have helped that we didn’t think about?

     1. How did you overcome this?

     2. What strategies did you use to help integrate the agenda setting tool into your practice?

     3. How did you track its effectiveness?

• Did you feel like you needed more information about how to or why you should use it?

5. What about the format of the materials? Is there a better way for us to think about presenting the materials?

6. When you were first given the PFE Resource what did you think?

• How did you feel about using it?

• Are there plans for your practice to continue using the PFE Resource in the future?

  1. What changes is your practice making to accommodate this sustained implementation?

  2. Why do you think that the practice is not continuing the implementation?

  3. Is there anything that would make it more sustainable/feasible to maintain the implementation?

7. What was the best thing about the PFE Resource?

8. What was your least favorite thing about the PFE Resource?

9. What would you have changed about the PFE Resource?

10. What about costs of implementing the PFE Resource? Was that a challenge at all for you?

1. Can you describe how much the time and/or effort it took from yourself or your staff to implement?

2. Was this a barrier?

3. Do you believe that this is sustainable for your practice?

   1. If yes. Can you describe what makes this approach important enough for you to continue using it?

   2. If no. Can you describe why you wouldn’t consider continuing to use it?

11. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project and will help us to make important improvements to the PFE Resource!

Appendix F – Pilot Test Evaluation Protocol for Practice Administrators

AHRQ – Building Diagnostic Safety Capacity

Practice Administrators – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews with administrators from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 20-60 cognitive interviews with practice administrators (1-3 practice administrators per practice x 20 practices); each interview will last approximately 60 minutes

Recruitment Criteria

Ambulatory care practice administrators who have been exposed to the PFE Resource within their practice will be eligible to participate in the interview. MHRI team members will work with the practice champions to identify practice administrators to participate in the interviews. We will aim to recruit practice administrators in the following manner:

• Practice administrators involved in the decision making around implementation of the PFE Resource or how have had experience using the PFE Resource

Interview Goals

The goals of the interviews will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Interview Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Location

Interviews will be conducted at a location within the practice or within the practice’s community. Interviews may also be conducted virtually via the telephone.

Participant Stipends

None.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the interview.

Proposed Agenda – Practice Administrators Interview

Interviews will be planned for approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	15 minutes
Enhancements	15 minutes
Closing	5 minutes
Total	60 minutes

AHRQ – Building Diagnostic Safety Capacity

Practice Administrators – Patient and Family Engagement Resource Pilot Test Evaluation

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the interview about your experiences with the Agency for Healthcare Research and Quality’s Resource for improving patient and family engagement in the diagnostic process!

• My name is [ INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT TEAM] and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with using the PFE Resource and your practice’s experience implementing it.

• Nothing that you say or share today will be shared in an identifiable way with your practice but represented together along with nine other practices in a report to the Agency for Healthcare Research and Quality. We will not share your name or anything that you say with them in a personally identifiable way.

• Do you have any questions before we begin?

So let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

I would like to begin by asking you all some questions about diagnostic error and the diagnostic process in your practice.

DIAGNOSIS

12. Prior to the signing up for the pilot test, what was your practice doing in terms of improving diagnosis and diagnostic error mitigation?

    1. Was this a strategic priority for your organization?

    2. What were the key factors that underpinned your decision to join the pilot test?

    3. Were any of the materials provided at the time of recruitment helpful to this process?

    4. What information could we have provided to make this intervention more attractive to you as a practice administrator and leader?

13. How would you describe the impact of the PFE Resource on the safety of the diagnostic process in your practice?

4. Why do you think that it was improved?

   1. Are you actively measuring and tracking these improvements?

   2. Are there other improvements that are harder to track? If yes, what would those be?

5. What strategies within the PFE Resource (agenda setting tool, poster, one-minute of patient talking uninterrupted) do you feel had the greatest impact on the diagnostic process within your practice?

   1. Can you elaborate on why you think it had that impact?

6. What was the most effective part of the PFE Resource within your practice? Can you describe why that is?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

14. As an administrator, what was the impact of the PFE Resource on your clinical team?

    1. What about your practice staff? Did the implementation result in changes in productivity or engagement from your staff? Clinicians? Others?

    2. What about burden? Was this unduly burdensome for your practice to implement?

       1. Compared to other quality improvement activities, how would you rate this activity in terms of burden for implementation? What about sustainability?

    3. Does your practice intend to continue using the PFE Resource?

       1. If yes, what parts of the resource did you find most impactful? How do you know this?

       2. What parts of the process/resource do you intend to keep?

       3. For those parts of the resource that you will continue, are you planning on making any changes to them?

       4. What would make the PFE Resource more useful? Easier to implement? Easier to sustain?

15. Do you believe that the PFE Resource impacted your patients in a positive way?

    1. Were they more or less engaged with the clinicians?

    2. Were the patient resources helpful or did they hinder patient-clinician interactions?

    3. Did patients give you any feedback about the resources or the approach?

       1. How did they describe their experiences?

       2. Where did they provide this feedback to you?

       3. What about measures of patient engagement? Did you see any changes in CAHPS scores or other satisfaction or experience surveys?

16. I would like to focus on the practice’s implementation of the PFE Resource. Can you describe for me how your practice advised the team about implementation?

• Did you use the quick start implementation guide? Do you have any feedback about what was missing in the implementation guidance? Anything that should have been there that would have been helpful? What about things in it that weren’t helpful?

• What could have made the implementation guide more useful?

17. As a practice administrator, was the format of the materials appropriate for how your practice conducts process and quality improvement implementations?

    1. Did it fit within your patient and family engagement strategies? Complement them? Were they at odds with them in any way?

    2. What changes to the materials might support greater adoption and more sustained implementation? Would you be willing to try that?

18. What about costs of implementing the PFE Resource? Was that a challenge at all for you?

    1. How will cost of the implementation influence the decision to sustain the program after the pilot test?

       1. What are the greatest influencing factors with respect to sustainability?

          1. Payment? Cost? Change fatigue? Interest?

       2. How could the PFE Resource better address these issues to make it more attractive to your clinic stakeholders?

19. How much staff time and resources did the implementation of the PFE Resource take? Is this sustainable? What would you say to other practice administrators who are contemplating implementing this resource?

20. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project and will help us to make important improvements to the PFE Resource!

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

Appendix G – Observation Tool

AHRQ – Building Diagnostic Safety Capacity

Patient and Family Engagement Resource Pilot Test Evaluation

Practice Observation Tool

The practice observation tool will be used during the practice site visit to document evidence of the PFE Resource implementation within the practice. This will be conducted by a project team member and have minimal burden on the practice and respondents. The practice observation tool is a brief checklist to audit materials and evaluate fidelity to the intended implementation of each of the resources into everyday practice. Patient-clinician interactions will be observed at the discretion of the patient, clinician, and practice. We aim to observe up to 10 patient-clinician interactions per practice for evidence of within encounter adoption of the PFE Resources.

MedStar Health Research Institute (MHRI) will conduct direct observations of patient-clinician encounters for up to 10 encounters per practice (n=200 encounters). Each encounter will engage one patient, one clinician, and one or more staff members (burden estimate based on 2 staff). Total time for the encounter burden was estimated at 20 minutes per encounter.

Recruitment Criteria

Ambulatory care practice administrators who have been exposed to the PFE Resource within their practice will be eligible to participate in the interview. MHRI team members will work with the practice champions to identify practice administrators to participate in the interviews. We will aim to recruit practice administrators in the following manner:

• Practice administrators involved in the decision making around implementation of the PFE Resource or how have had experience using the PFE Resource

Interview Goals

The goals of the interviews will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Interview Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Location

Practice observations will be conducted within the practice that is implementing the resource.

Participant Stipends

None.

Burden Hours: 100 patients x 30 minutes per patient

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the interview.

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

AHRQ – Building Diagnostic Safety Capacity

Patient and Family Engagement Resource Pilot Test Evaluation

Practice Observation Tool

PFE Resource Component	Observed	Description of Fidelity/Implementation in Practice
Patient facing poster	☐	Encounter Room Waiting Room Hallway Other: [Define]
Patient Agenda Tool – Preparation for Encounter	☐	Given at check in Given at exam room Given in waiting room Other
Patient One Minute Story	☐	Patient given one minute to speak at start of encounter Patient interrupted after [xxxx] seconds Patient not given opportunity Patient refused opportunity Family member spoke on behalf of patient Family member engaged in story telling with patient Other? [describe]
Patient Agenda Tool - Encounter	☐	MA acknowledged the Agenda Tool Clinician acknowledged the Agenda Tool Clinician used Agenda Tool in visit Other? [describe]

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 20 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.