Supporting Statement Part A -- Development & Evaluation of AHRQs QI Indicators Toolkit 5-21-2010

SUPPORTING STATEMENT

Part A

Development and Evaluation of AHRQ’s Quality Indicators Improvement Toolkit

Version: May 19, 2010

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

A. Justification 3

1. Circumstances that make the collection of information necessary 3

2. Purpose and use of information 7

3. Use of Improved Information Technology 8

4. Efforts to Identify Duplication 8

5. Involvement of Small Entities 8

6. Consequences if Information Collected Less Frequently 8

7. Special Circumstances 9

8. Consultation outside the Agency 9

9. Payments/Gifts to Respondents 9

10. Assurance of Confidentiality 9

11. Questions of a Sensitive Nature 10

12. Estimates of Annualized Burden Hours and Costs 10

13. Estimates of Annualized Respondent Capital and Maintenance Costs 11

14. Estimates of Annualized Cost to the Government 11

15. Changes in Hour Burden 12

16. Time Schedule, Publication and Analysis Plans 12

17. Exemption for Display of Expiration Date 13

List of Attachments 13

A. Justification

1. Circumstances that make the collection of information necessary

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care. The reauthorization of the Agency for Healthcare Research and Quality (AHRQ) in 1999 established the Agency as a leader in support of research designed to improve the quality of healthcare, reduce its costs, promote patient safety and reduce medical errors, and broaden access to effective services.

An important part of AHRQ’s mission is to disseminate information and tools that can support improvement in quality and safety in the U.S. health care community. This proposed information collection supports that part of AHRQ's mission by developing and evaluating a toolkit that will enable hospitals to effectively use AHRQ's Quality Indicators (QIs).

AHRQ has developed sets of QIs that can be used by the Agency and others to document quality and safety conditions at U.S. hospitals. Two sets of QIs will be used in this proposed toolkit: the Inpatient Quality Indicators (IQIs) and the Patient Safety Indicators (PSIs). The IQIs contain measures of volume, mortality, and utilization for common medical conditions and major surgical procedures. The PSIs are a set of measures to screen for potentially preventable adverse events that patients may experience during hospitalization. These QIs have been previously developed and evaluated by AHRQ, and are in use at a number of hospitals throughout the country. The QIs and supportive documentation on how to work with them are posted on AHRQ’s Web site at www.qualityindicators.ahrq.gov. Many of the QIs have been endorsed by the National Quality Forum through its consensus review process.

Values for each QI can be estimated for a given hospital by applying computations in SAS programs developed by AHRQ to the hospital’s pre-existing inpatient encounter data. To identify potential areas for improving the quality and safety of the care that a hospital provides, the hospital can use these data to examine its current performance on each QI measure, changes in its performance over time, and how its performance compares to that of other hospitals. However, despite the availability of the QIs as tools to help hospitals assess their performance, many U.S. hospitals have limited experience with the use of such measurement tools, or in using quality improvement methods to improve their performance as assessed by these measures.

As shown in Figure 1, an alpha version of the Quality Indicators Improvement Toolkit will be developed, which then will be field tested by six hospitals. (See Attachment C for principles and outline of the Toolkit contents.) During the field test, the proposed evaluation described herein will assess the usability of the Toolkit for hospitals, and it will examine their experiences in implementing interventions to improve their performance on the AHRQ QIs, as well as effects on trends in the hospitals’ AHRQ QI values. Using results from the evaluation, the alpha Toolkit will be revised to yield a final Toolkit that will be effective in supporting hospitals’ quality improvement efforts.

Figure 1. Logic Model of the AHRQ Quality Indicator
Improvement Toolkit Project

The evaluation is the part of this project that involves primary data collection, which will include data on hospital improvement experiences and on the usability of the Toolkit. As an integral part of their quality improvement work, the six participating hospitals will assess their progress in improving performance on the QIs by developing data on trends in values for the Quality Indicators. They will calculate the QI rates by applying the QI SAS programs to their inpatient encounter data. These are the data that AHRQ will consider in assessing effects of the Toolkit and improvement activities on QI values.

The proposed data collection to be carried out in evaluating the alpha version of the Toolkit is an essential part of the Toolkit development process because it will gather important information from hospitals, as end users of the product, on what is necessary to ensure that the Toolkit is both useful and readily usable by them. Only by having this direct feedback from the participating hospitals can a Toolkit be developed that effectively supports hospitals in applying quality improvement efforts related to performance on the AHRQ QIs.

The Quality Indicators Improvement Toolkit is to assist hospitals in both using the QIs and improving the quality and safety of the care they provide, as measured by those indicators. As such, the Toolkit is to include: (1) instruction on how a hospital can apply the QIs to its inpatient data to estimate rates for each indicator; (2) methods the hospital can use to evaluate these QI rates for identifying opportunities for improvement; (3) strategies for implementing interventions (or evidence-based best practices); (4) methods to measure progress and performance on the QIs; (5) tools for evaluating the cost-effectiveness of these changes; and (6) discussion of the value of using the QIs for quality improvement as well as potential challenges and barriers to quality improvement efforts that incorporate the QIs and how to help overcome them. The Toolkit is not a survey or data collection instrument and it is not designed for use by AHRQ or its contractors. Rather, once developed, the Toolkit will contain a set of implementation support tools that hospitals across the country can use in their quality improvement processes.

In the field test of the alpha Toolkit, the six (6) participating hospitals will apply the tools it contains in their quality improvement activities aimed at improving their performance on the AHRQ QIs. They first will use their current values for the IQIs and PSIs to identify priorities for improvement actions, and then will develop and carry out an implementation plan designed to bring about those improvements. In each step of this process, they will make use of tools in the Toolkit designed to support the relevant implementation step.

The evaluation of the Toolkit will consist of two main activities. First, primary data will be collected on both the hospitals’ implementation activities and their assessments of the tools. Second, the evaluation will examine what impact the Toolkit has on hospital performance, as measured by the AHRQ IQIs and PSIs. The evaluation will draw upon pre-existing encounter (claims) data on hospitals’ inpatient stays, which will be analyzed by the participating hospitals as an integral part of the hospitals’ quality improvement implementation processes. These data are generated at all hospitals as the data records used for billing for hospital services.

The following four data collection instruments will be used in the evaluation:

1) Pre/post-test interview protocol -- consisting of both open and closed ended questions will be administered prior to implementation of the Toolkit and again post implementation. The purpose of this data collection is to obtain data on the steps the hospitals took to implement actions to improve performance on the QIs; their plans for making process changes; and their experiences in achieving changes and perceptions regarding lessons learned that could be shared with other hospitals. (See Attachment D).

2) Update protocol – consisting of both open and closed ended questions will be administered three times during the study (quarterly during the implementation year). The purpose of this data collection is to capture longitudinal data regarding hospitals’ progress in implementing changes, successes and challenges, and plans for subsequent actions. These data will include descriptive information on changes over time in the hospitals’ implementation actions and how they are using the Toolkit, as well as experiential information on the perceptions of participants regarding the improvement implementation process and its effects. It also ensures the collection of information close to pertinent events, which avoids the recall bias associated with retrospective reporting of experiences. (See Attachment E).

3) Usability testing protocol – also consisting of both open and closed ended questions will be administered once at the end of the evaluation period. The purpose of this data collection is to gather information from the hospitals on how they used each tool in the Toolkit, the ease of use of each tool, which tools were most helpful, suggested changes to improve each tool, and suggestions for other tools to add to the Toolkit. This information will be used in the revisions of the Toolkit following the end of the field test. (See Attachment F).

4) AHRQ QI data collection tool – used to collect the IQI and PSI measures calculated by the hospitals both prior to implementation of the Toolkit and again post implementation. The purpose of this data collection is to determine if the hospitals’ implementation actions, including use of the toolkit, had a measurable impact on the QI measures. (See Attachment G)

The development and evaluation of the Quality Indicators Improvement Toolkit will be conducted by AHRQ's contractor, the RAND Corporation, under contract number HHSA290200600017I. RAND has subcontracted with the University HealthSystem Consortium (UHC) to partner in the development of the Toolkit and field testing of it with hospitals as they use the Toolkit in carrying out initiatives designed to improve performance on the QIs.

2. Purpose and Use of Information

All the information obtained from the proposed data collection will be used to strengthen the Quality Indicators Improvement Toolkit before finalizing and disseminating it to hospitals for their use. First, information will be collected from the six hospitals participating in the Toolkit field test about their experiences in implementing performance improvements related to the AHRQ QIs, which will be used to prepare experiential case examples for inclusion in the Toolkit as a resource for other hospitals. Second, feedback will be elicited from them about the usability of the Toolkit, which will be applied to modify and refine the Toolkit so that it is as responsive as possible to the needs and priorities of the hospitals for which it is intended.

3. Use of Improved Information Technology

This collection of information will not involve the use of automated or electronic collection techniques. All of the interviews will be conducted either by telephone or in person during site visits to hospitals.

4. Efforts to Identify Duplication

As stated above, two types of information will be collected in the interviews: information on hospitals’ experience in implementing quality improvements related to the AHRQ QIs and feedback from the hospitals about the usefulness and usability of the Toolkit. Both sets of information will be unique to this project. The information on hospitals’ implementation experience is unique because the experience will relate specifically to their work with the AHRQ QIs and related improvement actions. Although it is known from published materials and previous experience that various factors influence implementation success, the instruments used here will collect data on the specific status of these hospitals on these factors, which also is unique. The feedback on the Toolkit is completely unique to this project because it is tailored specifically to improving the products being developed in the project.

5. Involvement of Small Entities

We do not believe that any of the participating hospitals would be considered a small business.

6. Consequences if Information Collected Less Frequently

This data collection activity is a one-time collection done as part of the evaluation for this Toolkit development project. If the data collection were not performed, activities in support of AHRQ’s Congressionally mandated activities to improve the quality and safety of health care for the country will be severely hindered, because the evaluation information is essential to developing a Toolkit that will effectively support performance improvements by hospitals on the AHRQ Quality Indicators.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on December 31^st, 2009 for 60 days. One comment was received. This comment, along with AHRQ's response is shown below:

Public Comment from Jean Public:

From: jean public [mailto:usacitizen1@live.com]
Sent: Thursday, December 31, 2009 12:53 PM
To: Lefkowitz, Doris C. (AHRQ); info@taxpayers.net; media@cagw.org
Subject: public comment on federal register

 

i do not think this project is effective or cost effective. i think this project can be shut down. and sunset. jeanupblic 15 elm st florham park nj07932

AHRQ's Response:

Quality and patient safety has received considerable public attention since the publication of two landmark reports by the Institute of Medicine, To Err is Human: Building a Safer Health System (2000) and Crossing the Quality Chasm: A New Health System for the 21^st Century (2001).  The federal Agency for Healthcare Research and Quality has developed the Quality Indicators (QIs) that can be used for measuring and improving quality and patient safety.  These indicators cover more than a dozen of common medical conditions and major surgical procedures as well as 18 patient safety events.  For real change to happen, it is important for hospitals to implement specific interventions that incorporate the use of these measures.  With the purpose to develop, test, and implement a toolkit that will help hospitals apply the QIs for ongoing assessment and improvement of quality and patient safety, this project will effectively contribute to the improvement of health care for all Americans.  We believe the cost estimates for completing this project are reasonable.

8.b. Outside Consultations

RAND and its subcontractor, UHC, consulted with representatives of the hospitals that will be participating in the evaluation to obtain their views and suggestions regarding the evaluation data collection process, and will consult with them as the evaluation interviews are scheduled to minimize disruption of their work processes. Further, these data collection methods have been used by RAND in previous studies; they are proven methods that will be applied in this evaluation.

9. Payments/Gifts to Respondents

No payments or gifts will be provided to the hospitals participating in the evaluation.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose. The responses will be aggregated with those of other respondents before any information is reported to any other party outside of the research team (see Attachment H).

11. Questions of a Sensitive Nature

The evaluation does not include any questions that would be considered sensitive.

Verbal informed consent will be obtained from all interview participants before starting each interview, in compliance with the requirements of RAND’s IRB, which has approved our informed consent and data privacy provisions (see Attachments E, F, H and I). The interview guides include explicit instructions for the interviewers to read the informed consent and data privacy provisions aloud to the interviewees.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the respondents' time to participate in this information collection. Three protocols will be used to collect data from respondents in interviews that will take one hour each. The pre/post-test interview protocol will be administered twice – at the beginning and end of the field-test year. The pre-test interviews will be performed as one-hour group interviews conducted with the six hospitals’ implementation teams at the start of the year. At the end of the year, post-test interviews will be performed as one-hour group interviews with three of the hospitals and during site visits with the other three hospitals. At each site visit, data will be collected through one-hour interviews with the hospital’s implementation team as well as through other group interviews performed separately with each of the key stakeholder groups – physicians, nurses, clerks, and others. The additional data from the stakeholder groups will allow triangulation of variations in perceptions and experiences among different groups, of which the implementation teams might not be aware.

The quarterly update protocol will be administered quarterly to 2 hospital staff members from each hospital during the year (in months 3, 6, and 9). The usability testing protocol will be administered to 4 staff members once at the end of the evaluation period. The AHRQ QI data collection tool will be used both pre- and post-implementation to collect the QI measures. The total burden is estimated to be 360 hours.

Exhibit 2 shows the estimated annualized cost burden associated with the respondents' time to participate in the evaluation. The total cost burden is estimated to be $10,545.

Exhibit 1. Estimated annualized burden hours

Form Name	Number of hospitals	Number of responses per hospital	Hours per response	Total burden hours
Pre/Post-Test Interview Protocol	6	26	1	156
Quarterly Update Protocol	6	6	1	36
Usability Testing Protocol	6	4	1	24
AHRQ QI Data Collection Tool	6	2	12*	144
Total	24	NA	NA	360

* Includes time to program and run the computer programs necessary to produce the measures.

Exhibit 2. Estimated Annualized Cost Burden for Hospitals

Form Name	Number of hospitals	Total burden hours	Average hourly wage rate*	Total cost burden
Pre/Post-Interview Protocol	6	156	$27.46	$4,284
Quarterly Update Protocol	6	36	$27.46	$989
Usability Testing Protocol	6	24	$27.46	$659
AHRQ QI Data Collection Tool	6	144	$27.46	$4,613
Total	24	360	NA	$10,545

* Based upon the mean of the average wages, National Compensation Survey: Occupational wages in the United States, March 2009, “U.S. Department of Labor, Bureau of Labor Statistics.” Used as an overall average wage rate across the various types of staff involved in the quality improvements.

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs are defined to include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs (including capital and maintenance costs) to respondents other than their time to participate in the study (shown in Exhibit 2).

14. Estimates of Annualized Cost to the Government

Exhibit 3 shows the estimated total and annualized cost of this project to the government. The estimated total cost for the evaluation work is $209,827 over the two-year year project, with an annualized total cost of $104,914. These costs were developed based on estimates of staff days required, to which administrative expenses are applied, and based on airfare, hotel, and per diem costs for staff travel for the site visits at the end of the evaluation.

Exhibit 3. Estimated Cost of the Evaluation to the Government

Cost Component	Total Cost	Annualized Cost
Protocol Development	$40,278	$20,139
Data Collection Activities	91,104	45,552
Data Analysis	45,252	22,626
Publication of Results	24,370	12,185
Travel for Site Visits	8,823	4,412
Total	$209,827	$104,914

15. Changes in Hour Burden

This is a new information collection.

16. Time Schedule, Publication and Analysis Plans

Data collected will be analyzed using standard qualitative data analysis methods, by aggregating responses for each topic area and comparing and contrasting the information provided by respondents at the participating hospitals. For analyzing implementation experiences, a focus will be given to key successes, challenges or barriers encountered, progress toward achieving intended improvements, prospects for sustainability, and advice to others pursuing similar improvements. For the feedback on the Toolkit usability, comments and suggestions for each tool will be aggregated and summarized, including identification of conflicting reactions as well as concrete suggestions for improving the tool. The analysis of effects on the QIs will be done by calculating differences in each hospital’s scores for each QI and testing the significance of those differences.

The results of the evaluation will be used to refine and finalize the Toolkit, which then will be disseminated for use by hospitals across the country. In addition, a report will be prepared for AHRQ, presenting the full results of this work, and articles will be prepared and submitted for publication in health-related peer-review and/or social science research journals. The table below presents the project’s current schedule:

Task/Activity	Timeline and Proposed Date of Completion
Submit 60 and 30 day notice for interviews Submit OMB package for interviews	December 31st, 2009 March 11th, 2010
Conduct pre-implementation interviews	July 2010
Hospitals start implementing improvements and using Toolkit	August 2010
Conduct quarterly update interviews	November 2010, February 2011, May 2011
Conduct post-implementation interviews	July 2011
Conduct interviews on tool usability	July 2011
Revise Toolkit based on evaluation results	September 2011
Dissemination of Toolkit	October 2011
Report and journal articles	November 2011

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Attachment A: Healthcare Research and Quality Act of 1999

Attachment B: Federal Register Notice

Attachment C: Toolkit Principles and Outline of Contents

Attachment D: Pre/post-test interview protocol

Attachment E: Quarterly update protocol (includes informed consent statement)

Attachment F: Usability testing protocol (includes informed consent statement)

Attachment G: AHRQ QI data collection tool

Attachment H: Informed consent form for Pre/Post Interview

Attachment I: RAND Human Subjects Protection Committee (IRB) Approval Notice

13