SUPPORTING STATEMENT
Part A
Establishing Comparative Data for the
Medical Office Survey on Patient Safety
Version: May 6, 2009
Agency of Healthcare Research and Quality (AHRQ)
TABLE OF CONTENTS
A. JUSTIFICATION
A. JUSTIFICATION
A.1. Circumstances Making the Collection of Information Necessary
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients,
consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
AHRQ requests that the Office of Management and Budget approve, under the Paperwork Reduction Act of 1995, AHRQ’s intention to (1) collect information needed to establish reliable comparative data for the Medical Office Survey on Patient Safety and (2) collect descriptive information on barriers and facilitators to survey participation by medical offices, and on the utility/value of survey data in improving ambulatory patient safety.
The ambulatory Medical Office Survey on Patient Safety (MO-SOPS), an adapted version of AHRQ’s Hospital Survey on Patient Safety Culture (HSOPSC), was developed in 2005 to measure specific factors of patient safety culture in the ambulatory setting. A pilot study (OMB #0935-0131) assessed and refined the psychometric properties of specific survey items, and a final version of MO-SOPS is now ready for public dissemination (see Attachment B). However, in order for the survey to be useful to ambulatory medical offices in identifying areas of relative strength and weakness in patient safety culture, reliable comparative data to which a practice’s responses can be compared need to be established.
AHRQ has determined, through discussions with potential end-users of MO-SOPS including leaders of physician and other provider groups, that an ambulatory practice is unlikely to have confidence in an MO-SOPS comparative data unless it is based on responses derived from offices with similar characteristics. Office characteristics perceived to have a potential effect on MO-SOPS responses include such factors as provider mix (single specialty/multi-specialty), size of practice, and use of electronic information technology. A separate Practice Characteristics Survey to collect standardized information about these and other practice characteristics has been developed and was tested and refined as part of the pilot study (see Attachment C).
AHRQ’s overall goal is to generate and make available to each MO-SOPS end-user comparative data summary measures of patient safety culture based on survey responses from ambulatory practices with similar characteristics. Toward this end, AHRQ intends to administer MO-SOPS to a purposive sample of ambulatory medical offices across the country that have been selected on the basis of a set of practice characteristics. This purposive sample will provide preliminary comparative data that can be accessed by medical offices until a permanent comparative database can be established from the responses of a wide variety of medical offices throughout the country. An alternative method of initially populating the comparative database would be to make the surveys available to the public immediately and wait until a sufficient number of practices with the targeted characteristics have submitted responses. However, this approach is unacceptable to AHRQ since (1) no comparative data would be available to provide feed back to the first wave of responders, and (2) the amount of time that may be required before sufficient numbers of practices with the desired range of practice characteristics voluntarily provide survey responses is unpredictable and likely to be excessive. In addition, AHRQ intends to collect from these practices evaluative information about administrative barriers and facilitators to survey participation as well as a description of how the office used (or plans to use) the survey results to enhance patient safety culture (see Attachment D).
This project is being conducted pursuant to AHRQ’s statutory mandates to (1) promote health care quality improvement by conducting and supporting research that develops and presents scientific evidence regarding all aspects of health care, including methods for measuring quality and strategies for improving quality (42 U.S.C. 299(b)(1)(F) and (2) conduct and support research on health care and on systems for the delivery of such care, including activities with respect to quality measurement and improvement (42 U.S.C. 2991(a)(2).
Survey items included in MO-SOPS allow the calculation of composite measures of specific aspects of patient safety culture in the ambulatory office from the perspective of the practice’s providers and staff. Examples of aspects of patient safety culture addressed by MO-SOPS that are potentially modifiable include the extent to which an office demonstrates openness of communication and responds non-punitively to error. All physicians, non-physician providers and employed staff in the practice are asked to complete the survey, and summary measures for each component of patient safety are fed back to the practice while protecting the identity of individual respondents. It is intended that the practice will use these results to identify areas for patient safety culture improvement or to assess the impact of ambulatory medical office patient safety culture improvement initiatives. Preliminary comparative data that practices will accept as reliable reference points (in that they are derived from the survey responses of practices with similar characteristics) will be developed through this information collection.
The Comparative Database will be used for the following purposes:
1) Comparison--to allow medical offices to compare their patient safety culture survey results to other medical offices in the U.S.
2) Internal Assessment and Learning--to enable medical offices to identify their strengths and areas with potential for improvement in patient safety culture
3) Trending--in Year 2 of the database and beyond, trending data will be presented to describe changes in patient safety culture over time for medical offices that submit data more than once
4) Research and Analysis--de-identified data from the database will be made available to researchers for projects examining relationships involving patient safety culture survey data in medical offices. Note: Researchers will be required to submit proposals for approval by AHRQ before de-identified survey data will be made available to them. Medical offices submitting to the database will sign a data use agreement indicating their consent to release their de-identified data for research purposes.
AHRQ hopes to market the MO-SOPS to a wide audience of ambulatory medical practices across the country in the future. Information about the various ways, and the extent to which, practices actually use (or intend to use) MO-SOPS will be important to this effort. More information about factors that have facilitated or limited the participation of practices in the survey process and the value/utility of composite survey results can inform ways of improving the future administration of the survey and will be collected as part of the post-survey evaluation.
Achieving high response rates to the MO-SOPS from physicians and staff of sampled offices will be critical if the benchmarking database is to be adequately populated. Several recent studies have indicated that traditional methods of paper-based surveying continue to deliver better response rates, compared to electronic/web-based methods, among medical professionals, especially in busy office settings. (Raziano DB, Jayadevappa R, Valenzula D et al. E-mail versus conventional postal mail survey of geriatric chiefs. Gerontologist 2001;41:799-804; VanDenKerkhof EG, Parlow Jl, Goldstein DH et al. Anesthesiologists are less likely to respond to an electronic, compared to a paper questionnaire. Can J Anaesth. 2004;51:449-54) Investigators postulate that busy clinicians are frequently interrupted in the process of responding to surveys and that paper formats more easily accommodate stop-and-start approaches to survey completion. MO-SOPS and the Office Characteristics Survey will therefore be administered to physicians and staff in paper format, with standard non-response follow-up techniques such as reminder postcards and distribution of a second survey.
The proposed post-survey evaluation is distinct from the MO-SOPS and Office Characteristics surveys in that it includes several items that require (or allow) a free-text response. AHRQ estimates that hand written responses to such items could increase by more than five minutes the time required to complete each evaluation (compared to typed responses). To avoid this additional burden, evaluations will be conducted via the internet for all offices having access to high speed internet connections. In a previous study conducted in similar practice sites, 100% of offices reported access to such connections. AHRQ considers the benefit of reducing the burden to responders to outweigh the risk of a reduced response rate for this part of the project. In addition, an internet-based approach will benefit AHRQ in assuring more consistent legibility of free-text responses than would be expected with hand-written responses.
The earlier pilot testing of the MO-SOPS and Office Characteristics Survey instruments generated responses from 97 ambulatory practices. However, these responses can not be used to populate the comparative database since several items or factors included in the earlier versions of both instruments have been either modified or dropped in the final versions as a result of an analysis of pilot data on item non-response, variability, reliability and construct validity.
As discussed above in A.3, the proposed mixed-method approach (paper-based surveys for MO-SOPS and Office Characteristics Survey; internet-based evaluations) is AHRQ’s attempt to balance the needs for maximizing survey response rates while limiting as much as possible the burden on individual providers and office staff respondents.
This information collection will be a one-time collection.
The data collection efforts will be consistent with the guidelines at 5 CFR 1320.5(d)(2).
A. Federal Register Notice
As required by 5 CFR 1320.8(d), a notice was published in the Federal Register on September 19, 2008 (Vol 73, No. 183, p.54402-3) for 60 days. One comment was received and is summarized below:
Tax payers should not have to pay for this project. Put a tax on the very profitable health care industry to pay for this.
Attachment A summarizes the Federal legislation authorizing AHRQ to support projects which have the potential to improve patient safety in U.S. healthcare settings.
B. Outside Consultation
In developing this project, AHRQ has consulted with the leaders of three practice-based research networks who are experts in ambulatory health care and have recently been funded to study issues related to ambulatory patient safety. One of these leaders, Dr. Lyle Fagnan of the Oregon Health Sciences University, has consulted with a group of nine clinicians whose practices are potential end-users of MO-SOPS. The names of the expert consultants and clinician end-users are shown below:
Content Experts
John Hickner, MD, MSc, Professor, Department of Family Medicine, University of Chicago, Chicago, Illinois
Lyle Fagnan, MD, Clinical Professor, Oregon Health and Science University, Portland, Oregon
Joann Sorra, PhD, Survey methodologist, Westat
Potential End-Users of MO-SOPS
John T. Lynch, MPH, Director, Connecticut Center for Primary Care, Hartford, Connecticut
Barcey Levy, MD, family physician, Iowa City, Iowa
Shersten Killip, MD, family physician, practice patient safety coordinator, Louisville, Kentucky
Joyce Weinhandl, RD, MBA, coordinator of practice quality improvement, Minneapolis, Minnesota
Katy Duncan Smith, practice enhancement assistant for 10 primary care practices, Oklahoma City, Oklahoma
There is no plan to remunerate individuals or practices for responses to either the MO-SOPS or the Practice Characteristics Survey.
Individuals and organizations contacted will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Individuals and organizations contacted will be further assured of the confidentiality of their replies under 42 U.S.C. 1306, and 20 CFR 401 and 4225 U.S.C.552a (Privacy Act of 1974). In instances where respondent identity is needed, the information collection will fully comply with all respects of the Privacy Act.
Information that can directly identify the respondent, such as name and/or social security number, will not be collected.
There are no sensitive questions asked of the respondents.
Table A.12-1 shows the estimated burden hours for the medical offices’ time to participate in this one-time data collection. It is anticipated than an average of 10 persons (about 3 physicians and 7 staff) in each of the approximately 400 medical offices will respond to the survey, resulting in a maximum of 4000 responses (approximately 1,200 physicians and 2,800 staff). The Medical Office Survey on Patient Safety (MO-SOPS) and post-survey evaluation will be completed by both physicians and staff, while the Office Characteristics Survey will be completed by the office manager at each of the participating medical offices. Standard techniques such as using a cover letter of support from the medical office, reminder postcards, and distribution of a second survey will be used to achieve the target response rate.
The MO-SOPS survey and Office Characteristics survey each require approximately 15 minutes to complete. All staff will be asked to complete the MO-SOPS, however only the office manager will need to complete the Office Characteristics Survey. Additionally, the Post-Survey Evaluation, which will take an estimated 15 minutes to complete, will be completed by only one person per practice, either the office manager or the lead clinician. It is estimated that the total annualized respondent burden for completing the surveys will be 1,200 hours (Table A.12-1).
Table A.12-1: Estimated annualized burden hours
Survey Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
MO-SOPS |
400 |
10 |
15/60 |
1,000 |
Office Characteristics Survey |
400 |
1 |
15/60 |
100 |
Post-Survey Evaluation |
400 |
1 |
15/60 |
100 |
Total |
1,200 |
|
|
1,200 |
Table A.12-2 shows the estimated annualized cost burden based on the respondent's time to participate in this project. For the MO-SOPS and Post-Survey Evaluation the wage rate is the national average wage for “healthcare practitioner and technical occupations.” For the Office Characteristics Survey the hourly wage is the national average wage for “medical and health services managers.” National Compensation Survey: Occupational Wages in the United States 2006, U.S. Department of Labor, Bureau of Labor Statistics. Based on the burden hours and hourly rates of physicians and staff, the total annualized cost burden is estimated at $35,004.
TABLE A.12-1: ESTIMATED ANNUALIZED COST BURDEN
Survey Name |
Number of respondents |
Total burden hours |
Average Hourly Wage Rate |
Total Cost Burden |
MO-SOPS |
400 |
1,000 |
$27.44 |
$27,440 |
Office Characteristics Survey |
400 |
100 |
37.82 |
3,782 |
Post-Survey Evaluation |
400 |
100 |
37.82 |
3,782 |
Total |
1,200 |
1,200 |
n/a |
$35,004 |
A.13. Estimates of Annualized Respondent Capital and Maintenance Costs
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
The estimated annualized cost to the Federal Government for this contracted survey effort is approximately $340,000. These estimates include the costs associated with the project such as the preparation of survey administration procedures for both paper and electronic data collections, remuneration costs, labor costs, administrative expenses, costs associated with copying, postage, and telephone expenses, data management and analysis, and preparation of final reports. These costs are outlined in Table A.14-1.
Table A. 14-1 Estimated Cost to Federal Government |
||
Preparation of Surveys |
20,500 |
|
Remuneration (320 offices @$100/office) |
32,000 |
|
Labor Contractor
|
150,000 |
|
Labor Personnel |
93,000 |
|
Administrative Expenses |
4,000 |
|
Data Management and Analysis |
8,500 |
|
Annualized Total |
$308,000 |
|
This is a new collection of information.
A.16. Plans for Tabulation and Project Time Schedule Plans for Tabulation
As detailed in Supporting Statement B, practices eligible to participate in this project will be members of one of 14 practice-based research networks and will be stratified with regard to three characteristics: provider mix (single specialty vs multi-specialty); practice size (2-3 physicians vs 4 or more physicians), and HIT capacity. Initial stratification of each practice according to these characteristics will be based on the information available to the practice-based research network at the time the practice joined the network. This information, however, will be updated/verified (and, if necessary, the practice re-categorized) according to responses provided on the Practice Characteristics Survey. Provider mix and practice size will be confirmed based on responses to items 3a and 7 of the survey. Categorization of the practice as either HIT-enabled or not HIT-enabled will be determined based on practice response to item 4 (“To what extent has this medical office implemented each of the following electronic (computer-based) tools?”). Respondents will indicate whether the practice has fully implemented or is in the process of implementing each of six HIT itemized tools. Those practices indicating that they have fully implemented or are in the process of implementing four or more of the tools will be categorized as HIT-enabled. All other practices will be categorized as not HIT-enabled. AHRQ anticipates that the same approach will be used to categorize (and identify appropriate benchmarks for) practices that respond to MO-SOPS once the survey is made available on a public website. Comparative summaries corresponding to each set of practice characteristics will be made available on the website.
The method used to compute summary measures of patient safety for the Hospital Survey on Patient Safety Culture, as described by Sorra et al (2008)1, will be used to compute percent positive responses corresponding to patient safety culture composites for offices surveyed using MO-SOPS. Table A.16-1 lists 12 patient safety culture composites of survey responses. For each composite, percent positive responses will be calculated as follows:
For positively worded items, percent positive response is the combined percentage of respondents within a clinic who answered “Strongly agree” or “Agree” or “Always” or “Most of the time,” depending on the response categories used for the item. For example, for the item “People support one another in this work area,” if 50 percent of respondents within a clinic Strongly agree and 25 percent Agree, the item-level percent positive response for that clinic would be 50% + 25% = 75% positive.
For negatively worded items, percent positive response is the combined percentage of respondents within a clinic who answered “Strongly disagree” or “Disagree” or “Never” or “Rarely,” because a negative response on a negatively worded item indicates a positive response. For example, for the item “We have patient safety problems in this work area,” if 60 percent of respondents within a clinic Strongly disagree and 20 percent Disagree, the item-level percent positive response would be 80 percent positive (i.e., 80 percent of response do not believe they have patient safety problems in their work area.”
Table A.16-1: Patient safety culture composites
Patient Safety Culture Composite |
Definition: The extent to which… |
1. Communication openness |
Staff freely speak up if they see something that may negatively affect a patient, and feel free to question those with more authority |
2. Feedback & communication about error |
Staff are informed about errors that happen, given feedback about changes implemented, and discuss ways to prevent errors |
3. Frequency of events reported |
Mistakes of the following types are reported: 1) mistakes caught and corrected before affecting the patient, 2) mistakes with no potential to harm the patient, and 3) mistakes that could harm the patient, but do not |
4. Handoffs & transitions |
Important patient care information is transferred across clinic units and during shift changes |
5. Management support for patient safety |
Clinic management provides a work climate that promotes patient safety and shows that patient safety is a top priority |
6. Non-punitive response to error |
Staff feel that their mistakes and event reports are not held against them, and that mistakes are not kept in their personnel file |
7. Organizational teaming – Continuous improvement |
There is a learning culture in which mistakes lead to positive changes, and changes are evaluated for effectiveness |
8. Overall perceptions of patient safety |
Procedures and systems are good at preventing errors, and there is a lack of patient safety problems |
9. Staffing |
There are enough staff to handle the workload, and work hours are appropriate to provide the best care for patients |
10. Supervisor/manager expectations & actions promoting safety |
Supervisors/managers consider staff suggestions for improving patient safety, praise staff for following patient safety procedures, and do not overlook patient safety problems |
11. Teamwork across units |
Hospital units cooperate and coordinate with one another to provide the best care for patients |
12. Teamwork within units |
Staff support one another, treat each other with respect, and work together as a team |
These percentiles will be reported by specialty mix, size of practice and HIT capacity. General guidelines on the use of these percentiles and the margin of error associated with them will also be provided in online documents.
Project Time Schedule
The time schedule for the project is provided below:
Table A.16-2: Project Time Schedule
Activity |
Time Schedule (in relation to OMB) |
Approximate Dates |
Sampling Plan |
5 – 6 months prior to OMB approval |
September 2008 |
Practice Recruitment |
Approximately 4 months prior to OMB approval |
Beginning October 2008 (following approval of the sampling plan by AHRQ) |
OMB approval |
By April 1, 2009 |
Note: this accounts for the following periods of time: publication of 60-day FRN, 30-day FRN, up to 2 months of review of package at OMB
|
Administration of Surveys |
1-2 months after OMB approval |
May/June 2009 |
Survey Administration Report |
5 months after OMB approval |
September 2009 |
Data submitted to PBRN Resource Center |
5-6 months after OMB approval |
September/October 2009 |
Distribution of feedback reports to practices |
8 months after OMB approval |
December 2009 |
Report on survey process (based on web evaluation survey) |
10 months after OMB approval |
February 2010 |
Draft Final Report |
11 months after OMB approval |
March 2010 |
Final Report |
Approximately 12 months after OMB approval |
By March 31, 2010 |
AHRQ does not seek this exemption.
LIST OF ATTACHMENTS:
Attachment A: Healthcare Research and Quality Act of 1999
Attachment B: Medical Office Survey on Patient Safety (MO-SOPS)
Attachment C: Survey on Practice Characteristics
Attachment D: Post-survey Evaluation
1 See Sorra, J., Famolaro, T., Dyer, N., Nelson, D., and Khanna K. (2008). Hospital Survey on Patient Safety Culture 2008 Comparative Database Report. (Prepared by Westat, Rockville, MD, under contract No. 233-02-0087, Task Order 18). AHRQ Publication No. 09-0039. Rockville, MD: Agency for Healthcare Research and Quality. March 2008. The details of the percent positive score calculation are provided in Chapter 5.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | AHCPR |
| Last Modified By | wcarroll |
| File Modified | 2009-05-07 |
| File Created | 2009-05-07 |