Form #3 Form #3 Usability Testing Protocol

I

Form Approved
OMB No. 0935-XXXX
Exp. Date XX/XX/20XX

II. Protocol to Test Usability of the QI Tools

To be administered by phone or in person at the end of implementation.

1. BASIC BACKGROUND

• Interviewer to fill in prior to the interview.

1) Hospital Name [XXXX e.g., St. Elizabeth’s]

2) Hospital System Name: [XXXX e.g., Trinity]

3) User’s position: [XXXX e.g., nurses, physicians]

4) Implementation Phase: [XXXX e.g., baseline, early, mature]

5) Main AHRQ QI(s) targeted: [XXXX e.g., CABG mortality rate]

6) Intervention Strategy: [XXXX e.g. need intervention types here]

7) Hospital Type: [XXXX e.g., teaching, community]

8) Other important hospital selection characteristics: [XXXX e.g. rural, safety-net, etc]

II. INFORMED CONSENT & INTRODUCTION

• Interviewer to review read this section with aloud to each study participant.

This research is being conducted to learn how hospitals can work with the AHRQ Quality Indicators for quality improvement activities and how they may use the draft toolkit being developed in this project. This interview will allow us to collect information on your experience of the usability of this toolkit so that we can make improvements.

This interview is voluntary and anonymous, and the information you provide is not linked to you. Your responses will be kept confidential to the extent permitted by law, including AHRQ’s confidentiality statute, 42 USC 299c-3(c). RAND will keep confidential the identities of those who participate in the interview and will not attribute any comments to any specific individuals. In the notes from the interview, we will not record individuals’ names associated with comments made. If you want to know more about this study, please call Donna Farley, Ph.D., Principal Investigator, at 412-683-2300, ext. 4633.

Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 540 Gaither Road, Room # 5036, Rockville, MD 20850.

III. CONTEXT OF USE QUESTIONS

1) Who used the toolkit?

2) Please describe the process of using the toolkit.

• Probes:

  • Did different users use different sections?

  • Was the toolkit used by teams or individuals?

  • What types of computing or other resources were used with the toolkit?

3) How much prior experience did the user(s) have with hospital quality improvement?

4) How much prior experience did the user(s) have with the AHRQ QIs? With other areas covered by the toolkit?

5) How much and what type of coaching did users receive?

• Probes:

  • Coaching from UHC?

  • Coaching from employees of your hospital or hospital system?

  • Coaching from others outside of your hospital system?

IV. USABILITY QUESTIONS – INDIVIDUAL TOOLS

• Interviewer to use the grid for this section of the interview. Interviewer will sequentially cover each toolkit section (left column), asking questions with codable responses (first grid) followed by questions with narrative questions (second grid).

Questions on Individual Tools

Tool	1) Did you use [TOOL]? (Y/N) Skip to next row if N	2) How useful was [TOOL] in enabling you to achieve your goals effectively? Poor to Excellent (1 to 4)	3) How much time, money, and mental effort was required to use [TOOL]? Little to Much (1 to 4)	5) Do you recommend: keep [TOOL] as is, modify it, or drop it? (K, M, D)	If #5=M: How should tool be modified? If #5 = D: Why drop tool?
A. Getting ready for change
A.1. PSI/IQI Fact Sheet	Y N	1 2 3 4	1 2 3 4	K M D
A.2. PowerPoint presentations for Board and staff	Y N	1 2 3 4	1 2 3 4	K M D
A.3. Self-Assessment Tool	Y N	1 2 3 4	1 2 3 4	K M D
B. QIs and hospital data
B.1. Calculating QI rates with AHRQ software	Y N	1 2 3 4	1 2 3 4	K M D
B.2. Example of AHRQ QI software output	Y N	1 2 3 4	1 2 3 4	K M D
B.3. PowerPoint on data, trends, and rates	Y N	1 2 3 4	1 2 3 4	K M D
C. Identify improvement priorities
C.1. Prioritization Matrix	Y N	1 2 3 4	1 2 3 4	K M D
D. Implementation methods
D.1. Implementation Process Overview	Y N	1 2 3 4	1 2 3 4	K M D
D.2. Team Charter and Goals	Y N	1 2 3 4	1 2 3 4	K M D
D.3. Identifying Best Practices	Y N	1 2 3 4	1 2 3 4	K M D
D.4. Gap Analysis Tool	Y N	1 2 3 4	1 2 3 4	K M D
D.5. Implementation Planning	Y N	1 2 3 4	1 2 3 4	K M D
D.6. Measurement of Implementation Progress	Y N	1 2 3 4	1 2 3 4	K M D
D.7. Project Evaluation and Debriefing	Y N	1 2 3 4	1 2 3 4	K M D

Tool	1) Did you use [TOOL]? (Y/N) Skip to next row if N	2) How useful was [TOOL] in enabling you to achieve your goals effectively? Poor to Excellent (1 to 4)	3) How much time, money, and mental effort was required to use [TOOL]? Little to Much (1 to 4)	5) Do you recommend: keep [TOOL] as is, modify it, or drop it? (K, M, D)	If #5=M: How should tool be modified? If #5 = D: Why drop tool?
E. Cost effectiveness & ROI
E.1. Cost Effectiveness Analysis	Y N	1 2 3 4	1 2 3 4	K M D
E.2. Return on Investment Analysis	Y N	1 2 3 4	1 2 3 4	K M D
F. Monitor progress, sustainability
F.1. Measure selection and development	Y N	1 2 3 4	1 2 3 4	K M D
F.2. Reporting process and report formats	Y N	1 2 3 4	1 2 3 4	K M D
G. Existing Improvement guides
G.1. List of existing guides	Y N	1 2 3 4	1 2 3 4	K M D

Questions on Individual Tools (cont)

	Challenges Encountered		What did you like about this tool?
Tool	What challenges encountered in using the tool?	How you addressed the challenges?
A. Getting ready for change
A.1. PSI/IQI Fact Sheet
A.2. PowerPoint presentations for Board and staff
A.3. Self-Assessment Tool
B. QIs and hospital data
B.1. Calculating QI rates with AHRQ software
B.2. Example of AHRQ QI software output
B.3. PowerPoint on data, trends, and rates
C. Identify improvement priorities
C.1. Prioritization Matrix
D. Implementation methods
D.1. Implementation Process Overview
D.2. Team Charter and Goals
D.3. Identifying Best Practices
D.4. Gap Analysis Tool
D.5. Implementation Planning
D.6. Measurement of Implementation Progress
D.7. Project Evaluation and Debriefing
E. Cost effectiveness & ROI
E.1. Cost Effectiveness Analysis
E.2. Return on Investment Analysis
F. Monitor progress, sustainability
F.1. Measure selection and development
F.2. Reporting process and report formats
G. Existing Improvement guides
G.1. List of existing guides

V. USABILITY QUESTIONS – TOOLKIT OVERALL

• Interviewer to ask questions first for toolkit overall, then ask for differential responses across the tools.

  1. What coaching do you believe is needed for hospitals to use the toolkit?

  2. Which tools need the most/least coaching?

Tool	Level/Type of Coaching
A. Getting ready for change
A.1. PSI/IQI Fact Sheet
A.2. PowerPoint presentations for Board and staff
A.3. Self-Assessment Tool
B. QIs and hospital data
B.1. Calculating QI rates with AHRQ software
B.2. Example of AHRQ QI software output
B.3. PowerPoint on data, trends, and rates
C. Identify improvement priorities
C.1. Prioritization Matrix
D. Implementation methods
D.1. Implementation Process Overview
D.2. Team Charter and Goals
D.3. Identifying Best Practices
D.4. Gap Analysis Tool
D.5. Implementation Planning
D.6. Measurement of Implementation Progress
D.7. Project Evaluation and Debriefing
E. Cost effectiveness & ROI
E.1. Cost Effectiveness Analysis
E.2. Return on Investment Analysis
F. Monitor progress, sustainability
F.1. Measure selection and development
F.2. Reporting process and report formats
G. Existing Improvement guides
G.1. List of existing guides

3. How useful was the toolkit, overall?

4. Which tools were the most/least useful?

Tool	Usefulness of the Tool
A. Getting ready for change
A.1. PSI/IQI Fact Sheet
A.2. PowerPoint presentations for Board and staff
A.3. Self-Assessment Tool
B. QIs and hospital data
B.1. Calculating QI rates with AHRQ software
B.2. Example of AHRQ QI software output
B.3. PowerPoint on data, trends, and rates
C. Identify improvement priorities
C.1. Prioritization Matrix
D. Implementation methods
D.1. Implementation Process Overview
D.2. Team Charter and Goals
D.3. Identifying Best Practices
D.4. Gap Analysis Tool
D.5. Implementation Planning
D.6. Measurement of Implementation Progress
D.7. Project Evaluation and Debriefing
E. Cost effectiveness & ROI
E.1. Cost Effectiveness Analysis
E.2. Return on Investment Analysis
F. Monitor progress, sustainability
F.1. Measure selection and development
F.2. Reporting process and report formats
G. Existing Improvement guides
G.1. List of existing guides

VI. WRAP-UP QUESTIONS

1) Are other tools needed that are not currently in the toolkit?

2) What advice do you have for other hospitals on how to successfully use the toolkit?

3) We’re at the end of the interview. Do you have any questions or any other comments?

Thank you very much for your time. We appreciate it.

2