Supporting Statement Part A -- Evaluation of the NGC 8-23-2010

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. This Act further states that AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

The mission of the Agency for Healthcare Research and Quality (AHRQ) is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health system practices, including the prevention of diseases and other health conditions. 42 U.S.C. § 299(b). AHRQ supports the dissemination of evidence-based guidelines through its National Guideline Clearinghouse™ (NGC).

The NGC serves as a publicly accessible Web-based database of evidence-based clinical practice guidelines meeting explicit criteria. The NGC also supports AHRQ’s strategic goal on effectiveness: to improve health care outcomes by encouraging the use of evidence to make informed health care decisions. The NGC is a vehicle for such encouragement. The mission of the NGC is to provide physicians, nurses, and other health professionals, health care providers, health plans, integrated delivery systems, purchasers and others an accessible mechanism for obtaining objective, detailed information on clinical practice guidelines and to further their dissemination, implementation and use.

AHRQ proposes to conduct a comprehensive evaluation of the NGC. NGC’s 10-year anniversary is an ideal time to perform this evaluation and build on the site trends AHRQ has already indentified, including growth from 70,000 to 700,000 visits per month, 600 to approximately 40,000 email subscribers, 250 to 2,370 guidelines represented, and 50 to nearly 300 participating guideline developer organizations from July 1999 to July 2009.

Three primary data sources were used to inform the evaluation design: (1) an environmental scan of published and unpublished (“grey”) literature on guideline use and dissemination to identify what is known about the NGC's influence to date on its various stakeholder groups; (2) a comprehensive analysis of the NGC project data (e.g., Annual Project Reports; Annual NGC Customer Satisfaction Surveys); and (3) input from a group of individual experts in guideline development, evaluation, dissemination, and implementation who formed the evaluation's Participant Evaluation Team (PET).

The objectives of the NGC evaluation are to gain a better understanding of how:

the NGC is used
the NGC supports dissemination of evidence-based clinical practice guidelines and related documents
the NGC has influenced efforts in guideline development and guideline implementation & use
the NGC can be improved

To achieve the objectives of this project the following data collections will be implemented:

NGC evaluation survey – administered to a convenience sample of both users and non-users of the NGC (see Attachments B and C),

Focus groups -- conducted with guideline developers, medical librarians, informatics specialists, clinicians, and students (see Attachment D), and

Key informant interviews -- conducted with influential individuals in medical societies, health plans, and quality improvement organizations as well as medical librarians, researchers, and informatics specialists who produce, use, and disseminate guidelines (see Attachment E).

Questions in the survey, focus group, and key informant discussion guides will focus on the effectiveness of NGC in areas of dissemination, implementation, and use of evidence-based clinical practice guidelines, and relative to other available guideline sources. For example, measures to be gathered through the instruments include the level of trust of the NGC, the use of the NGC relative to other guideline sources, and the influence of the NGC on various stakeholder groups. In addition, the instruments will be used to measure the use of other guideline resources which are used by non-NGC users.

This study is being conducted by AHRQ through its contractor, AFYA, Inc. and The Lewin Group (AFYA/Lewin), pursuant to AHRQ’s statutory authority to conduct and support research and disseminate information on healthcare and on systems for the delivery of such care, including activities with respect to clinical practice. 42 U.S.C. 299a(a)(4).

2. Purpose and Use of Information

The purpose of this project is to formally evaluate the Agency for Healthcare Research and Quality’s (AHRQ’s) National Guideline Clearinghouse™ (NGC).

The purpose of the survey component of the overall NGC evaluation is two-fold:

To obtain feedback from a relatively large number of individuals representing key stakeholders of the NGC initiative, regarding overall awareness of the National Guideline Clearinghouse (this will assess awareness of NGC by its intended audience: clinicians, nurses, health care professionals, students, guideline developers, etc)
For those individuals who describe themselves as aware of NGC, to characterize their use of NGC, as well as NGC’s influence on their work, organization initiatives, or guideline development efforts, etc., and their suggestions for enhancements

We will also conduct focus groups and key informant interviews to obtain qualitative information that will be used to elaborate on the information gathered from the survey questionnaire.

The data collected via the survey, focus groups, and key informant interviews is intended to:

Assess NGC’s past performance in meetings its objectives, and guide its future efforts to achieve its goals;
Assess the extent to which NGC influences guideline development, implementation, and use; and
Provide AHRQ with valuable information that will help them understand how its multi-million dollar investment in NGC has shaped health care quality and how additional investments can continue to influence improvements in health care quality

The final product for this evaluation will be a report that summarizes the impact of the NGC on guideline development, implementation, and use. The final report will include the background, methodology for primary data collection, primary data collection results, comparison of results to other guideline sources, and a discussion of key findings. We will also discuss the limitations to the study and provide recommendations, based on the findings, regarding enhancements to the NGC.

3. Use of Improved Information Technology

The NGC evaluation survey will be web-based; a preliminary version can be seen at https://www.surveymonkey.com/s/ahrqsurvey. Using an online system for data collection rather than a paper-based survey makes completing and submitting the survey less time-consuming for respondents and facilitates data analysis. Any skip patterns included in the survey (i.e. questions that are only appropriate for a portion of the respondents) will be automatically programmed into the Web-based form of the survey, thereby reducing the number of irrelevant questions to which a given respondent may be subject to and making the overall survey more concise and brief. In addition, the contractors can also ensure that important items are not inadvertently skipped or ignored by setting software requirements to ensure proper completion of the survey based on specific respondent selections.

Focus groups and key informant interviews cannot benefit from the use of improved information technology.

4. Efforts to Identify Duplication

AHRQ currently collects feedback from NGC users via its Annual NGC Customer Satisfaction Survey, a web-based questionnaire on the NGC Web site. The Annual NGC Customer Satisfaction Survey is administered under the Agency-wide generic OMB clearance for customer satisfaction surveys (OMB Control Number 0935-0106). This survey is used to assess customer opinion on the relevance, content, utility, and ease of use of the NGC Web site, and is specifically directed to NGC users. This survey focuses on specific features of the NGC Web site, and is not redundant with our proposed evaluation effort (see Attachment F – 2008 Annual NGC Customer Satisfaction Survey) which goes beyond user satisfaction and aims to address issues such as overall awareness of the Web site (seeking both current, previous, and non-users of the Web site), how the NGC has influenced activities of various stakeholder groups, and to what degree the NGC is meeting its mission.

The objective of this project is to assess the NGC’s role in clinical guideline development, use, and dissemination by collecting outcome measures not currently gathered through the Annual NGC Customer Satisfaction Survey. While the three data collection instruments used in this project have been designed to minimize redundancy with the data already collected, we do aim to anchor the two surveys in order to compare the survey respondents. For example, both surveys ask respondents to identify their stakeholder group and geographical residence for the purpose of enhancing AHRQ’s understanding of the linkages between specific stakeholder groups and/or regions of the country, and NGC’s role in guideline development, use, and implementation.

5. Involvement of Small Entities

No small businesses will be involved in this study.

6. Consequences if Information Collected Less Frequently

This request is for a one-time data collection effort.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on September 17^th, 2010 for 60 days (see Attachment G). No comments were received.

8.b. Outside Consultations

To help guide the overall evaluation of the NGC, AFYA/Lewin established a Participatory Evaluation Team (PET). The PET is similar to a Technical Expert Panel, but is used in participatory evaluation design. The five members of the PET were drawn from the stakeholder groups and communities that are directly affected by the program under review. The PET for this project has three main roles: to provide feedback, as members of the user community, on the evaluation design; to assist us in reaching out to members of stakeholder groups to serve as focus group participants or key informants; and to provide contextual validity to various components throughout the project.

We sent the proposed evaluation methodology as well as the three data collection instruments (survey, focus group, and key informant discussion guides) to the PET to obtain their input. Overall, all members of the PET thought the evaluation survey was comprehensive and complete. The feedback received from the PET, although minor, included suggestions on appropriate questions for different user groups and identifying potential key informants. The AFYA/Lewin Team made changes to three data collection instruments to reflect the minor suggestions made by the PET.

9. Payments/Gifts to Respondents

Respondents of the Web-based survey, focus group participants, and key informants will not receive any gifts or payment in exchange for their participation.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.

Information that can directly identify the respondent, such as name and/or social security number will not be collected.

11. Questions of a Sensitive Nature

This project includes no questions of a sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the respondents’ time to participate in this evaluation. The NGC evaluation questionnaire will be completed by approximately 40,220 persons and will require 10 minutes to complete for users of the NGC and about 2 minutes for non-users. For the purpose of calculating respondent burden an average of 8 minutes is used and reflects a mix of users and non-users with most respondents expected to be users.

Eleven different focus groups consisting of 9 persons each will be conducted and are expected to last 90 minutes each. Key informant interviews will be conducted with 30 individuals and will last about 60 minutes. The total annual burden hours are estimated to be 5,542 hours.

Exhibit 2 shows the estimated annualized cost burden based on the respondents’ time to participate in this project. The total annual cost burden is estimated to be $185,712.

Exhibit 1. Estimated annualized burden hours

Data Collection Method	Number of Respondents	Number of responses per respondent	Hours per response	Total Burden hours
NGC Evaluation Survey	40,220	1	8/60	5,363
Focus Groups	99	1	1.5	149
Key Informant Interviews	30	1	1	30
Total	40,349	NA	NA	5,542

Exhibit 2. Estimated annualized cost burden

Data Collection Method	Number of Respondents	Total Burden hours	Average Hourly Wage Rate*	Total Cost Burden
NGC Evaluation Survey	40,220	5,363	$33.51	$179,714
Focus Groups	99	149	$33.51	$4,993
Key Informant Interviews	30	30	$33.51	$1,005
Total	40,349	5,542	NA	$185,712

*Based upon the mean of the average wages for healthcare practitioner and technical occupations (29-0000) presented in the National Compensation Survey: Occupational wages in the United States, May 2009, U.S. Department of Labor, Bureau of Labor Statistics.

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Annualized Cost to the Government

Exhibit 3 shows the estimated total and annualized cost to the government for this one year project. The total cost is estimated to be $350,000 to conduct the one-time survey, 11 focus groups, and 30 key informant interviews and to analyze and present their results. This amount is the contract total for AFYA’s contract with AHRQ to evaluate the NGC. This amount includes the costs for project development and management ($70,000 or 20% of the entire contract amount); data collection activities ($105,000 or 30% of the entire contract amount); data processing and analysis ($70,000 or 20% of the entire contract amount); and administrative support activities and reporting ($105,000 or 30% of the entire contract amount).

Exhibit 3. Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development and Management	$70,000	$70,000
Data Collection Activities	$105,000	$105,000
Data Processing and Analysis	$70,000	$70,000
Administrative Support and Reporting	$105,000	$105,000
Total	$350,000	$350,000

15. Changes in Hour Burden

This is a new collection of information

16. Time Schedule, Publication and Analysis Plans

Data Collection and Analysis	Timeframes
Key Informant Interviews
Interviews for Federal Government staff	Between August 2010 – December 2010
Conduct remaining key informant interviews of non Federal staff following OMB clearance	To be completed within 3 months of receiving OMB clearance
Focus Groups
Conduct preliminary focus group with up to nine individuals	Between August 2010 – December 2010
Conduct remaining focus groups following OMB clearance	To be completed within 3 months of receiving OMB clearance
Survey
Administer survey following OMB clearance	1 Month following OMB clearance (or after mid January 2011); to be conducted for 4 weeks
Analyze data from all instruments	Within 1 month following collection of all data
Prepare final report	Within 2 months following collection of all data

Survey Analysis

Quantitative data from survey respondents will be imported into Microsoft Excel (for data cleaning and free text response classification and review) and SPSS (for analysis). Quantitative data will be obtained through multiple-choice and Likert-type scaled responses. Qualitative data will also be captured through several open-ended questions.

Data Cleaning: Following the completion of the survey, data will be imported into a Microsoft Excel file for data cleaning. The survey administrator will manually screen for inadequately completed survey responses. If more than half of the required item responses are missing on a questionnaire, the participant who submitted it will be excluded from the analysis. However, if the respondent provides more than three quarters of the required responses, then the missing responses may be imputed using imputation techniques (such as mean response imputation or hot-deck imputation method). We will determine the exact imputation techniques upon the analysis of reasons for receiving the missing responses. If the respondent provides less than three quarters, but more than half, of the required responses, missing items will not be imputed, but the responses included will be deemed appropriate.

During the data cleaning phase, we will also examine and categorize text responses for each of the questions with “other” text response options. If a text response could be classified clearly into one of the predefined categories, we will recode the response to that category.

Data Analysis: The survey data will be downloaded into Microsoft Excel and imported into SPSS for statistical analysis. Analyses will be performed for three sub-groupings of the data: the dataset including individuals responding that they are not aware of NGC in the initial screening question; individuals who are aware of NGC but have never used the NGC Web site; and NGC Web site users. Additional analyses will also be performed by stakeholder group, such as physician and researcher.

Data analysis will include descriptive statistics (frequencies, proportional frequencies, means, modes, standard deviations, and the number of non-responders) as well as formal statistical comparisons using cross tabulations of the data (chi square tests, and various measures of association [Gamma statistic for ordinal comparisons and Cramer’s V for comparison of nominal/categorical variables]).

Focus Group and Key Informant Interview Analysis

We will analyze results obtained from the focus groups sessions and the key informant interviews using content analysis, looking for themes regarding NGC use, how it can be improved, value relative to other guideline sources, and other themes regarding NGC quality and effectiveness. We will use software such as MindJet Mind Manager and InVivo for our qualitative data analysis, data integration/triangulation efforts. For those questions that align with the survey, we will incorporate the responses into our descriptive statistics where appropriate.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

Attachments:

Attachment A: Healthcare Research and Quality Act of 1999

Attachment B: NGC Evaluation Questionnaire

Attachment C: NGC Evaluation Survey e-mail Notification

Attachment D: Focus Group Discussion Guide

Attachment E: Key Informant Interview Discussion Guide

Attachment F: 2008 Annual NGC Customer Satisfaction Survey

Attachment G: Federal Register Notice

File Type	application/msword
File Title	SUPPORTING STATEMENT
Author	Jenice James
Last Modified By	william.carroll
File Modified	2010-11-23
File Created	2010-08-23