Form #2 Form #2 Focus Group Discussion Guide

Agency for Healthcare Research and Quality Evaluation of the National Guideline Clearinghouse™ Generic Focus Group
Discussion Guide

The AFYA/Lewin Team

2/26/2010

Agency for Healthcare Research and Quality

National Guideline Clearinghouse (NGC) Evaluation

Focus Group Discussion Guide

DRAFT

This document can serve as the basis for the focus group discussion guide. Currently, the core questions included in this guide assume that the focus group will include both users and non-users of the NGC. As documented in the notes throughout the guide, some questions will be specific to those who are users.

The flow of topic areas is similar to the flow of questions contained in the NGC Evaluation Survey. The first 20 minutes of the focus group discussion will pertain to the participants’ use of guidelines and, in the case of NGC Users, the participants’ experience with NGC. The second component of the focus group discussion has been tailored to the issues most relevant to the specific focus group user type (i.e., guideline developer, clinician, informatics specialist,, and medical librarian).

---------------------------------------------------------------------------------------------------------------------

Screening questions would have already been asked to the participants. The moderator will have this information on the group before beginning the focus group.

Screener questions previously asked to focus group participants:

1) Do you use clinical guidelines in your work?

a. All participants answered Yes.

2) How many times per year, on average, do you use clinical guidelines in your work?

3) Are you aware of the National Guideline Clearinghouse?

a. The focus group may contain both people who are and are not aware of the NGC

4) Have you used the NGC to access information on clinical guidelines?

a. The focus group may contain both users and non-users of the NGC.

---------------------------------------------------------------------------------------------------------------------

Participants:

Welcome/Background [5 Minutes]:

• Moderator introduction

• NGC overview

• Purpose of the evaluation

• Importance of their perspective/How it will be incorporated into the evaluation

• Focus group logistics

• Opportunity for questions

• Brief introductions of participants

Discussion Questions

Background/Screening [5 minutes]

First, as you know, in preparation for this focus group we have previously gathered background information on your use of clinical guidelines and the NGC. In order to help me moderate the focus groups, I am going to ask similar questions as our starting point.

Question: Each of you uses clinical guidelines in your work. Can you briefly describe how and how often you use clinical guidelines in your work?

Probe: potential reasons they use clinical guidelines may include:

- Finding clinical practice guidelines - School assignment

- Comparing guidelines - Professional knowledge building

- Development of a clinical practice guideline - Supporting clinical decision-making

- Development of quality measures - Reducing errors/malpractice (risk management)

- Academic/medical research - Determining coverage of services

- Rationalizing and controlling healthcare expenditures

Question: Are you aware of the National Guideline Clearinghouse (NGC)?

b. How did you learn about the NGC?

c. How long have you been aware of the NGC?

Use of NGC & Other Guideline Sources [10 minutes]

Now, we will discuss how you use the NGC and other guideline sources when you need to locate information on clinical guidelines.

Question: Of the times you have needed to locate information on clinical guidelines, what percentage of the time do you use the NGC for this information?

a. For those times when you do not use the NGC for clinical guideline information, where do you access the information?

Probe: Potential other sources may include:

• Pubmed/Medline

• From a search using a general purpose Web search engine (e.g., Google)

• Point-of-care Web-based resources (e.g., DynaMed, Essential Evidence Plus, EMedicine)

• Medscape

• U.S. Preventive Services Task Force

• National Institute for Health and Clinical Excellence (NICE)

• Guidelines Advisory Committee (Canada)

• Guidelines International Network (G-I-N)

• Institute for Clinical Systems Improvement (ICSI)

• Colorado Clinical Guidelines Collaborative (CCGC)

• Milliman Care Guidelines®

• Expert Consensus Guidelines® (EKS®)

• Medical Specialty Society guidelines (e.g.,AACE, AAFP, AAN, AAO, AAOS, AAP, ACC, ACCP, ACEP, ACP, ACPM, ACR, AGA, ASCO, ASGE, IDSA, SAGES)

• Professional or Disease Specific Society guidelines (e.g., AASM, AARC, ACS, ADA, AHA, ASPEN, BTF, HFSA, NKF, SCCM)

• Government guidelines (e.g., CDC, NHLBI, SAMHSA, TFCPS, DoD/VHA)

• Commercial products (e.g., UpToDate; ACP Pier) (please specify)

• Other government entities

Question: How long have you been using the NGC website? (NGC users only)

Question: How often would you say you have used the NGC website in the past 12 months? (NGC users only)

a. For those who have not visited the NGC in the past 12 months, why not?

Probe: Potential reasons may include:

- No need to locate clinical guideline information

- Need the information, but go without it;

- Need the information, but use other guideline source instead of NGC

Question: Thinking of the guideline source that you frequent most often other than the NGC, how often have you used the source in the past 12 months?

Effectiveness and Influence of NGC and Other Guideline Sources [20 minutes]

Now that we have discussed how and how often you use the NGC and other guideline sources to support your work with clinical guidelines, we are going to discuss how well these tools fulfill your needs regarding clinical guidelines.

Question: We previously discussed how you use clinical guidelines. Summarize how they use guidelines from Question 1. Keeping these thoughts in mind, to what extent does the NGC site fulfill these needs? How? (NGC users only)

Probe: If there are ways in which the participants use clinical guidelines, but do not use the NGC to assist in that activity, why not?

Question: When thinking of how you use clinical guidelines, to what extent does another guideline source besides the NGC fulfill these needs? How?

Question: Overall, how would you rate your satisfaction with the NGC compared to the other guideline source you frequent most often? (NGC users only)

Options to consider in your response, may include: less satisfied with the NGC, satisfaction about equal, more satisfied with the NGC

Probe: Why more/less satisfied with the NGC?

Question: Do you find the NGC criteria for guideline inclusion appropriate?

Probe: Why/why not? Too stringent/loose?

The four criterions for NGC guideline inclusion are:

a. The guideline must contain systematically developed statements that include recommendations that assist in making decisions in specific circumstances.

b. Guideline must be produced, sponsored, or supported by an organization, association, society, or government agency. An individual without sponsorship cannot submit a guideline.

c. There must be documentation that verifies a literature search and a review of existing evidence was performed as part of developing the guideline.

d. Full text must be available in English.

Question: Do you feel the NGC saves you time regarding your work with clinical guidelines? (NGC users only)

b. If yes, how much?

c. How does it save you time?

Question: Do you trust the information provided by the NGC (both NGC content and the guidelines which are included in NGC)? (NGC users only)

Probe: Why/why not?

Question: Thinking of the guideline source you most frequently use, do you trust the information provided? If NGC is the primary source, what is it about NGC that makes it a most trusted source?

Probe: Why/why not?

Probe. More/less than information from NGC?

Question: Clinical guidelines are used for variety of purposes. What is your perception of the influence of NGC on:

a. Achievement of organization goals?

b. Development of actionable knowledge?

c. Individual practice and learning?

d. Research?

e. The work of Quality Improvement Organizations?

Question: Generally, clinical guidelines are expected to contribute to the improvement of quality in healthcare, promote efficiency, reduce healthcare costs and improve clinical decision making. What is your perception of NGC’s impact on:

a. Healthcare quality (both in general and specifically pertaining to the type of key informant?

b. Efficiency?

c. Healthcare costs?

d. Clinical decision making?

STAKEHOLDER SPECIFIC QUESTIONS AT THIS POINT IN THE DISCUSSION GUIDE. [40 minutes]

Guideline Developer

Question: Do you (or your organization) submit guidelines for inclusion in the NGC?

Probe: Why/why not?

a. If yes, approximately what percentage of developed guidelines do you submit to NGC?

Question: For those of you that do submit your guidelines to the NGC, how would you rate the following components of the NGC process? Options to consider in your don’t know, poor, fair, neutral, good, excellent

• Your experience with the submission process

• Providing copyright

• Preparation of the NGC Summary and abstraction of your organization's guidelines

• Verification process (per each guideline summary and/or annually)

• Dissemination of your organization’s guidelines

• Response to FDA warnings

Question: How, if at all, does NGC influence your organization’s guideline development program?

Probe: guideline topic selection, guideline development methodology, documentation and reporting of guidelines, guideline update frequency, collaboration with other guideline developers

Now that we have discussed how the NGC has influenced your organization’s guideline development program, the next three questions address how the NGC has influenced other related activities.

Question: For those of you whose organizations have created implementation tools that can be used to facilitate implementation of your organization’s guidelines, how, if at all, has NGC influenced this activity?

Question: For those of you whose organizations have created quality measures that can be used to assess the implementation of guidelines, how, if at all, has NGC influenced this activity?

Question: For those of you whose organizations have developed guidelines in a format that can be integrated into electronic medical records or other decision clinical support tools, how, if at all, has NGC influenced this activity?

Question: To the best of your knowledge, does your organization use other AHRQ-sponsored products to inform its guideline development activities? Examples include:

• Systematic evidence reviews

• Technology assessments

• Comparative effectiveness reviews

• Technical briefs

• Policymaker guides

• DEcIDE project reports

• CERT project reports

• Other products from AHRQ’s Effective Health Care Program

Probe: Why/why not? How?

Clinicians/Students

Question: How, if at all, does NGC influence your delivery of care to patients and your use of clinical practice guidelines?

Probe: Clinical decision-making processes, ongoing learning efforts, your implementation of guidelines, your organization’s implementation of guidelines.

Question: Would content from the NGC Web site, delivered to you at the point of care, be useful to you?

Probe: Why/why not? If yes, in what form?

Question: Is there a need to broaden the scope and type of guidelines (e.g., credentialing, privileging, ethical, procedural, training) included in NGC?

Probe: Why/why not?

Question: If continuing medical educations were available through NGC, would you take advantage of it?

Probe: Why/why not?

Medical librarians

Question: How, if at all has NGC influenced the following activities?

• Your data collection processes

• Your approach to identifying guidelines

• Your ability to meet your client needs regarding evidence-based clinical practice guidelines

• Your ability to identify “high quality” guidelines

• Your ability to identify “current guidelines

Are there other activities that NGC has influenced? How?

Question: NGC currently uses the UMLS Metathesaurus to index the guideline summary content included on the site to support browsing and searching. Should NGC use other controlled medical vocabularies for this purpose?

Probe: Why/why not? Which ones?

Informatics Specialist

Question: Do you currently USE or DEVELOP Web 2.0 applications (e.g., blogs, wikis, RSS,) to communicate about NGC guidelines?

Probe: Why/why not? How do you use these tools?

Question: Do you utilize NGC data as an input in developing clinical information or decision support systems?

a. If not, why not?

b. If yes, what type of data and what type of “Application Programming Interface (API)” would you like NGC to provide?”

Probe: Why/why not? How? If not, can you think of some ways that NGC content might be used?

Question: Do you utilize any of NGC’s RSS downloads?

Probe: Examples include: NGC content inventory (e.g., NGC Summaries, Expert Commentaries, and Guideline Syntheses or “what’s new” file. Why/why not? How? Could other NGC RSS feeds be useful for informatics specialist?

Question: Do you utilize the NGC Search Form Feature which allows developers to create search interfaces with NGC on external sites?

Probe: Why/why not? How? If not aware of this feature (describe), now that you are, would you use it?

Question: Would having an NGC output for individual guidelines available as an XML file according to the Guideline Elements Model (GEM) be useful to you in your work?

Probe: Why/why not? How?

Additional Considerations [10 minutes]

Question: The NGC is undergoing a redesign. We would like to discuss your thoughts and potential use of the following enhancements:

• Subject-specific email alerts regarding new and updated guideline releases

• Ratings of a guideline’s quality and/or methodologic rigor (e.g., AGREE score)

• NGC user forums (e.g., blogs, bulletin boards) that promote discussion, education, and collaboration on clinical guidelines?

• NGC summary content formatted for mobile device (e.g., cell phone, pocket PDAs) viewing

• Additional data exporting options

• The ability to download references into a citation manager utility (e.g., Endnote)

• The ability to limit searches of NGC to data contained in specific fields in the NGC summary

• The ability to export the entire NGC database

• The ability export targeted elements of the entire NGC database

• The ability to search archived guideline content

• Access to archived guidelines

Probe: Would you like to see any other additional functionality?

Question: Are there any specific changes that AHRQ could make to improve your organization’s experience with NGC?
Probe: What kinds of changes?

Question: Based on our discussion today, did you gather new knowledge of the NGC?

a. What did you learn?

b. Does this new knowledge of the NGC increase your likelihood of using it in the future?

Question: Are there any final comments or questions?

This concludes our focus group. Thank you for your time and input. AHRQ greatly appreciates your participation.

NGC Focus Group Discussion Guide 10 | Page