5 Appendix E

Appendix E – Pilot Test Evaluation Protocol for Practice Staff

AHRQ – Building Diagnostic Safety Capacity

Staff Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews and/or focus groups with staff from up to twenty (n=20) practices. Evaluation will be completed within 3-6 months after implementation of the Patient and Family Engagement Resource.

• 120-160 cognitive interviews with ambulatory care staff (6-8 staff members per practice x 20 practices); each interview will last approximately 60 minutes

Recruitment Criteria

Ambulatory care practice staff who have been exposed to the PFE Resource within their practice will be eligible to participate in the focus group. MHRI team members will work with the practice champions to identify practice staff to participate in the focus groups. We will aim to recruit practice staff in the following manner:

• Staff members who were involved in the implementation of the PFE Resource or how have had experience using the PFE Resource

Focus Group Goals

The goals of the focus groups will be to:

• Obtain feedback on the intervention PFE Resource materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the PFE Resource materials to improve adoption and implementation

Focus Group Materials

• Copies of the PFE Resource

• Informed consent documents

• Documentation for Processing of Participant stipends

• Digital recorder

Focus Group Location

Focus groups will be conducted at a location within the practice or within the practice’s community.

Participant Stipends

None.

Informed Consent Procedures

Participants will complete the informed consent process at the time of arrival to the focus group.

Proposed Agenda – Practice Staff Focus Group

Focus Groups will be planned for approximately 60 minutes each.

Agenda
Introduction	5 minutes
Background	10 minutes
Review Materials	10 minutes
General Experience with Intervention (satisfaction/barriers/enablers)	15 minutes
Enhancements	15 minutes
Closing	5 minutes
Total	60 minutes

AHRQ – Building Diagnostic Safety Capacity

Staff Interviews – Patient and Family Engagement Resource Pilot Test Evaluation

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the focus group about your experiences with the Agency for Healthcare Research and Quality’s Resource for improving patient and family engagement in the diagnostic process!

• My name is [ INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT STAFF and HE/SHE will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with using the PFE Resource and your practice’s experience implementing it.

• Nothing that you say or share today will be shared in an identifiable way with your practice but represented together along with nine other practices in a report to the Agency for Healthcare Research and Quality. We will not share your name or anything that you say with them in a personally identifiable way.

• Do you have any questions before we begin?

GROUND RULES

• We want to hear from everyone and want to hear your honest opinions. There are no wrong answers.

• If you have something to add to the conversation, please feel free to jump in. We do have a lot to cover so we will try not to spend too much time on any one topic.

• It is important that when you do jump in that we try to make sure that we only have one person talking at any time. This will help us hear everyone’s thoughts and opinions.

• Any questions?

So let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

I would like to begin by asking you all some questions about diagnostic error and the diagnostic process in your practice.

DIAGNOSIS

1. How would you describe the impact of the PFE Resource on the safety of the diagnostic process in your practice?

1. Why do you think that it was improved?

2. What strategies within the PFE Resource (agenda setting tool, poster, one-minute of patient talking uninterrupted) do you feel had the greatest impact on the process? Can you elaborate on why you think it had that impact?

3. Given your experience with the PFE Resource, what was most effective in your practice?

   1. Did you do anything to make changes to the materials or approaches to fit your practice? Can you describe those changes? Did that work? Why/Why not?

   2. Now that you have had some experience with the PFE Resource in your practice, what would you change about it? Why? What do you think that change would achieve?

Thank you for sharing. I now want to speak with you a little more about patient engagement.

2. From your experience, how did the PFE Resource support engagement from the patient and or their family in the process of getting an accurate and timely diagnosis?

• Can you describe what you observed when patients/family members used the agenda setting tool? The poster?

• Were you able to observe the patient and/or family owning that first minute of the visit to tell their diagnosis story?

• How would you describe the level of engagement you had with patients and families after you implemented the PFE Resources?

Now I would like you to consider the materials that you have in front of you for this next series of questions.

3. I would like to focus on the practice’s implementation of the PFE Resource. Were you part of the implementation team? Can you describe for me how your practice advised the team about implementation?

• Did you use the quick start implementation guide? Do you have any feedback about what was missing in the implementation guidance? Anything that should have been there that would have been helpful? What about things in it that weren’t helpful?

• What could have made the implementation guide more useful?

4. Can you describe for me how your practice implemented the patient-facing resources? Specifically, the agenda setting tool and the poster. Did you use both?

• How did the patient’s respond to this new role?

• Can you describe the intended workflow for the patient materials? Who was responsible for orienting the patient? How was this accomplished?

  1. Was the process disruptive to the practice’s workflow?

  2. Did you feel like you had enough education and training on how to orient patients to the tools? What would have helped that we didn’t think about?

     1. How did you overcome this?

     2. What strategies did you use to help integrate the agenda setting tool into your practice?

     3. How did you track its effectiveness?

• Did you feel like you needed more information about how to or why you should use it?

5. What about the format of the materials? Is there a better way for us to think about presenting the materials?

6. When you were first given the PFE Resource what did you think?

• How did you feel about using it?

• Are there plans for your practice to continue using the PFE Resource in the future?

  1. What changes is your practice making to accommodate this sustained implementation?

  2. Why do you think that the practice is not continuing the implementation?

  3. Is there anything that would make it more sustainable/feasible to maintain the implementation?

7. What was the best thing about the PFE Resource?

8. What was your least favorite thing about the PFE Resource?

9. What would you have changed about the PFE Resource?

10. What about costs of implementing the PFE Resource? Was that a challenge at all for you?

1. Can you describe how much the time and/or effort it took from yourself or your staff to implement?

2. Was this a barrier?

3. Do you believe that this is sustainable for your practice?

   1. If yes. Can you describe what makes this approach important enough for you to continue using it?

   2. If no. Can you describe why you wouldn’t consider continuing to use it?

11. Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project and will help us to make important improvements to the PFE Resource!