SUPPORTING STATEMENT
Part A
Pilot Test of the Proposed Pharmacy Survey on Patient Safety Culture
Version: August 10, 2011
Agency for Healthcare Research and Quality (AHRQ)
2. How, by Whom, and for What Purpose Information Will Be Used 5
3. Use of Improved Information Technology 6
4. Efforts to Avoid Duplication 6
6. Consequences of Less Frequent Collection 7
7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2) 7
8. Applicability to 5 CFR 1320.8(d) 7
9. Remuneration of Respondents 7
10. Assurance of Confidentiality 8
11. Questions of a Sensitive Nature 8
12. Estimates of Annualized Hour Burden 8
13. Estimates of Total Annual Cost Burden to Respondents 9
14. Estimates of Annualized Cost to the Government 9
17. Exemption for Display of Expiration Date 12
List of Attachments 11
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
Furthermore, AHRQ shall conduct and support research to provide objective clinical information to pharmacists; improve the quality of health care through the prevention of adverse effects of drugs and the consequences of such effects; identify the causes of preventable health care errors and patient injury in health care delivery; develop, demonstrate, and evaluate strategies for reducing errors and improving patient safety; and disseminate such effective strategies throughout the health care industry [Section 912, (b) (2) (A) (ii) (II) and (iii) (II) and (c) (1) (2) and (3) (http://www.ahrq.gov/hrqa99b.htm)].
As the baby boomer population ages, the general U.S. population continues to grow, and as drug therapies for the treatment of chronic diseases become more efficacious, the expected increase in the number of prescriptions and demand for pharmaceutical products is likely to increase the potential for medication errors in community/retail pharmacies. In 2007, there were about 56,000 community/retail pharmacies, including about 22,000 traditional chain pharmacy companies, nearly 17,000 independent drug stores, about 9,300 supermarket pharmacies, and about 7,700 mass merchant pharmacies.1 Numerous reports substantiate the presence of medication errors in pharmacies. For example, one national observational study of prescription dispensing accuracy and safety in 50 pharmacies in the U.S. found a rate of about 4 errors per day in a pharmacy filling 250 prescriptions daily. This error rate translates to an estimated 51.5 million errors occurring during the filling of 3 billion prescriptions each year2.
Given the widespread impact of pharmacies on patient safety, the new Pharmacy Survey on Patient Safety Culture (Pharmacy SOPS) will measure pharmacy staff perceptions about what is important in their organization and what attitudes and behaviors related to patient safety are supported, rewarded, and expected. The survey will help community/retail pharmacies to identify and discuss strengths and weaknesses of patient safety culture within their individual pharmacies. They can then use that knowledge to develop appropriate action plans to improve their practices and their culture of patient safety. This survey is designed for use in community/retail pharmacies, which includes chain drugstores (e.g., Walgreens and CVS), supermarket pharmacies, independently owned pharmacies, and mass merchant pharmacies (e.g., Wal-Mart, Costco, Target), not for use in hospital pharmacies.
This research has the following goals:
1) Cognitively test and modify as necessary the Pharmacy Survey on Patient Safety Culture Questionnaire;
2) Pretest and modify the questionnaire as necessary;
3) Make the final questionnaire available for use in the public domain.
To achieve these goals the following activities and data collections will be implemented:
1) Cognitive interviews – Two rounds of interviews will be conducted by telephone with 10 respondents each. The purpose of these interviews is to refine the questionnaire’s items and composites. Each round will be conducted with a mix of pharmacists and non-pharmacist staff working in community/retail pharmacies throughout the U.S. The same interview guide will be used for each round (see Attachment B).
2) Pretest – The draft questionnaire will be pretested with all pharmacy staff in approximately 60 community/retail pharmacies (see Attachment C). The purpose of the pretest is to collect data for an assessment of the reliability and construct validity of the survey’s items and composites, allowing for their further refinement (see Part A, Section 16 for analysis plan description). The draft pretest questionnaire cover letter and follow-up reminder notice are included in Attachment D.
3) Pharmacy background questionnaire – This questionnaire will be completed by the pharmacy manager in each of the 60 pretest sites to provide background characteristics of the pharmacy, such as pharmacy type (independently owned or chain), type of chain (traditional drugstore, supermarkets, mass merchant), average number of prescriptions filled weekly, average number of hours the pharmacy is open on weekdays, etc (see Attachment E).
4) Dissemination activities – The final questionnaire will be made available in the public domain through the AHRQ website. This activity does not impose a burden on the public and is therefore not included in the burden estimates in Section 12.
This study is being conducted by AHRQ through its contractor, Westat, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
The information collected will be used to test and improve the draft survey items in the Pharmacy Survey on Patient Safety Culture Questionnaire described in Section #1. Psychometric analysis will be conducted on the pilot data to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items included in the survey. Because the survey items are being developed to measure specific aspects of patient safety culture in the pharmacy setting, the factor structure of the survey items will be evaluated through multilevel confirmatory factor analysis. On the basis of the data analyses, items or factors may be dropped.
The final survey instrument will be made available to the public for use in pharmacies to assess their safety culture from the perspectives of their staff. The survey can be used by pharmacies to identify areas for patient safety culture improvement. Researchers are also likely to use the pharmacy survey to assess the impact of pharmacies’ patient safety culture improvement initiatives. This pharmacy survey is an expansion of AHRQ’s suite of surveys on patient safety culture, which are available on the AHRQ Web site at (http://www.ahrq.gov/qual/patientsafetyculture)3. Those surveys have been used by hundreds of hospitals, nursing homes, and medical offices across the U.S. to assess patient safety culture. The pharmacy survey contains new and revised questions and composites that more accurately apply to the pharmacy setting.
Data collection will not involve the use of any information technology. This data collection effort is a pilot study designed to gather information to examine the psychometric properties (internal consistency reliability, response variability, etc.) of a paper-and-pencil pharmacy safety culture survey instrument. The use of information technology is not needed for this study and is not likely to result in a reduction of burden at this time.
There is the possibility that a pharmacy chain or pharmacy system might participate in the pilot and elect to administer a Web data collection survey instrument. However, the pilot study is designed to use primarily a paper survey mode because staff in pharmacies do not always have access to the Internet or a computer.
We consulted pharmacy experts and conducted a review of the literature, searching for surveys of all pharmacy staff that measure patient safety culture in pharmacies. We were unable to identify any validated surveys that do so.
A number of different types of surveys for the pharmacy setting currently exist. The various types include surveys of automated systems in pharmacies (many pharmacies are moving toward adoption of electronic tools and systems, including e-prescribing and robotics); surveys about the changing practice of pharmacy (to include more clinical services); surveys about workforce characteristics; surveys that focus on a single safety-related topic such as drug-interactions; and surveys to assess patient/customer satisfaction. All of these surveys, however, are administered only to pharmacists or customers and have specialized topics other than safety culture.
We did identify a checklist survey developed by the Institute for Safe Medication Practices that is more like a group self-assessment of the policies and practices that should ideally be implemented in the pharmacy setting. It is quite lengthy and is not designed to be a culture survey. We also found a British survey, partially adapted from the AHRQ Hospital Survey on Patient Safety Culture, that is designed for pharmacists to assess their perceptions about the work climate in their pharmacies. Again, it has not been tested with all staff working in pharmacies.
We also learned about a pilot safety culture survey that was designed for all staff working in
pharmacies in Indiana. The response rates in the pilot study, however, were low, particularly among pharmacy technicians, and the researchers were unable to analyze their data to assess the validity of the questionnaire. They did not have funding to pay incentives or to pretest their items appropriately.
In summary, we did not locate any validated surveys that assess patient safety culture in the pharmacy setting from the perspective of all pharmacy staff.
Some of the pharmacies participating in this pretest will be small businesses. The data collection instruments and procedures are designed to minimize burden on individual pharmacy staff respondents and will not have a significant impact on them.
This effort is a one-time pilot test.
The data collection efforts will be consistent with the guidelines at 5 CFR 1320.5(d)(2).
8.a. Federal Register Notice
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on (date and page number of 60 day notice) for 60 days (see Attachment F).
8.b. Outside Consultations
A number of pharmacy expert reviewers and consultants were contacted for background interviews on the need for a patient safety culture survey for the pharmacy setting as well as for review and comment on the draft survey domains and items—see Attachment G for a list of those consulted both within and outside the Agency thus far.
Cognitive Interview Respondents. To successfully recruit 20 cognitive interview participants, it is appropriate to offer a cash incentive. We propose a $100 cash remuneration for pharmacists and a $50 cash remuneration for pharmacy technicians and cashiers/clerks.
Pilot Study Respondents. In consultation with pharmacy experts with whom we are working to develop the pilot test data collection plan, we have determined that some type of incentive will be appropriate to recruit pharmacies to participate in the pilot study and to achieve adequate response rates for collecting enough data to conduct psychometric analyses to validate the instrument. All pharmacists, technicians, and cashiers/clerks working in the pharmacies will be asked to complete a 15-minute paper-based survey. In addition, the pharmacy manager will be responsible for the following tasks: completing a form about background characteristics of the pharmacy, putting together a list of pharmacy staff and staff positions, distributing surveys, promoting staff cooperation, and conducting nonresponse followup.
Because it is too expensive for us to consider an individual-level incentive, and given that pharmacists may expect more remuneration for completing a survey than other staff (which would be problematic), we have determined that a site-level incentive would be the best way to ensure site participation as well as providing some benefit to all the individuals within a pharmacy. As we did in the pilot study for the AHRQ Medical Office Survey on Patient Safety Culture, we plan to recruit a site-level point of contact (POC) in each pharmacy who will manage data collection at that pharmacy (compile sample information, distribute surveys, promote survey response, etc.). We propose sending a check for a set amount ($100 per pharmacy with 6 or fewer staff and $150 for pharmacies with more than 6 staff members) made out to the pharmacy. We will suggest that the pharmacy manager oversee the use of the funds in a way appropriate to that pharmacy. We will encourage them to use the incentive to provide for a lunch or something else that would benefit each potential respondent within the pharmacy. As we did in the medical office pilot study, we also propose making 50 percent of the site incentive contingent on a site response rate of 50 percent or higher.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Information that can directly identify the respondent, such as name and/or social security number will not be collected.
We do not believe there are questions of a particularly sensitive nature included in the survey, but if during cognitive testing sensitivities are discovered, such questions will be modified to ensure they are not of a sensitive nature.
Exhibit 1 shows the estimated annualized burden hours for the pharmacies’ time to participate in this research. Cognitive interviews will be conducted with staff at 20 pharmacies (approximately 10 pharmacists and 10 nonpharmacist staff) and will take about one hour and 30 minutes to complete. 627 staff from 60 pharmacies will participate in the pretest (an average of 10.45 staff from each pharmacy). The pretest questionnaire (the Pharmacy Survey on Patient Safety Culture) requires 15 minutes to complete. The pharmacy background questionnaire will be completed by the manager at each of the 60 pharmacies participating in the pretest and takes 10 minutes to complete. The total annualized burden is estimated to be 197 hours.
Exhibit 2 shows the estimated annualized cost burden associated with the pharmacies’ time to participate in this research. The total cost burden is estimated to be $4,948 annually.
Exhibit 1. Estimated annualized burden hours
Form Name/Activity |
Number of pharmacies |
Number of responses per pharmacy |
Hours per response |
Total burden hours |
Cognitive interviews |
20 |
1 |
1.5 |
30 |
Pretest |
60 |
10.45 |
15/60 |
157 |
Pharmacy background questionnaire |
60 |
1 |
10/60 |
10 |
Total |
140 |
na |
na |
Exhibit 2. Estimated annualized cost burden
Form Name/Activity |
Number of pharmacies |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Cognitive interviews |
20 |
30 |
$32.28 |
$968 |
Pretest |
60 |
157 |
$22.08 |
$3,467 |
Pharmacy background questionnaire |
60 |
10 |
$51.27 |
$513 |
Total |
140 |
197 |
na |
$4,948 |
*Based upon the mean of the average hourly wages for Pharmacists (29-1051; $51.27), Pharmacy Technicians (29-2052; $13.92), and Pharmacy Aides (31-9095; $10.74), National Compensation Survey: Occupational wages in the United States May 2009, “U.S. Department of Labor, Bureau of Labor Statistics.” The hourly wage for the cognitive interviews is a weighted average for 10 pharmacists, 8 pharmacy technicians and 2 pharmacy aides; the hourly wage for the pretest is a weighted average for 157 pharmacists, 235 pharmacy technicians and 235 pharmacy aides.
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. The only cost to the respondent will be that associated with their time to respond to the information collection, as shown in Exhibit 1.
Exhibit 3 shows the estimated total and annualized cost for this project. Although data collection will last for less than one year, the entire project will take about 3 years. The total cost for this project is approximately $320,818.
Exhibit 3. Estimated Total and Annualized Cost
Cost Component |
Total Cost |
Annualized Cost |
Project Development |
$65,340 |
$21,780 |
Data Collection Activities |
$62,831 |
20,944 |
Data Processing and Analysis |
$11,004 |
$3,368 |
Publication of Results |
$15,767 |
$5,256 |
Project Management |
$7,496 |
$2,498 |
Overhead |
$158,380 |
$5,293 |
Total |
$320,818 |
$106,939 |
This is a new activity.
As soon as OMB approval is received, pilot study activities will begin. The estimated time schedule to conduct these activities is shown below:
Two rounds of cognitive interviews (2.5 months)
Pilot study data collection (5 months)
Data analysis, feedback report production, and development of technical report (3 months)
Final survey and development of toolkit materials (2 months)
The final version of the Pharmacy Survey on Patient Safety Culture Questionnaire and accompanying toolkit materials will be made available in the public domain on the AHRQ Web site.
The goal of our data analysis will be to assess the multi-level psychometric properties of the Pharmacy Survey on Patient Safety Culture using a multi-staged approach to assess construct validity through factor analyses and intercorrelations among composites and to assess reliability.
Descriptive Statistics
The means, standard deviations, and response frequencies for the survey items will be examined to ensure that respondents and pharmacies exhibit adequate response variability on the survey items. In addition, items will be examined to ensure that there are low rates of missing data (lower than 8% missing response per item). Poorly functioning items will be identified.
Individual Level Factor Analysis
An individual level factor analysis will be conducted to initially examine whether groups of items intended to measure a specific patient safety composite are interrelated, ignoring the nesting of respondent data within pharmacies. Factor loadings for each item in an apriori composite will be considered as having an adequate contribution to a particular composite or factor if the strength of the item's relationship to that factor (i.e., its factor loading), is .40 or greater.
The percent of variance accounted for by the factor will also be examined. The more variance that is accounted for by a factor, the more justifiable it is to combine the items into a single composite score. At least 50% of the variance should be accounted for by the items in a composite.
Intraclass Correlations (ICCs) and Design Effects
Intraclass correlations (ICCs) will be computed for each composite. ICC's determine if substantial variation exists between groups compared to variation within groups. ICCs above .05 or 5% indicate that the between group variance is greater than expected by chance and imply that nesting in groups does have an effect on the responses of individuals.
Given that ICCs are likely to be inflated when there are many groups with few individuals within the groups (compared to few groups with many individuals within the groups), we will also examine design effects, which take into account within-group sample size. A design effect of 2 or more implies that group membership or nesting of individuals within groups does have an effect on the responses of the individuals and therefore multilevel modelling should be conducted to account for the multilevel nature of the data.
Multilevel Confirmatory Factor Analysis (MCFA)
Individuals responding to the survey are located within pharmacies. When data are nested in groups like this, results from an individual-level factor analysis may be biased or incorrect. Therefore, multilevel confirmatory factor analysis will be conducted on the survey’s a priori composites to examine the structure of the factors at the pharmacy level of analysis, taking into consideration that the data are nested.
An MCFA will be conducted to test the fit of the measurement model for the survey’s patient safety composites, taking into consideration the nested nature of the data at the pharmacy level of analysis. We will first evaluate the MCFA results by examining the item factor loadings on the composites. Factor loadings should be .40 or greater.
We will also examine overall model fit indices using standard fit statistics: the chi-square, comparative fit index (CFI), and the standardized root mean square residual (SRMR). For chi-square statistics, lower and non-significant chi-squares indicate good fit. The factor structure is determined to adequately fit the data if the CFI is at least .90. A value of zero for the SRMR indicates perfect fit, but a value less than .08 is considered a good fit.
Reliability Analysis
Reliability analyses will then be performed on the composites to examine whether individuals responded consistently to the items within each composite. Internal consistency reliability will be calculated using Cronbach's alpha. The minimum criterion for acceptable reliability is an alpha of at least .70.
Intercorrelations
Intercorrelations among the survey’s patient safety composites and with outcome measures included in the survey (e.g. Overall perceptions of patient and medication safety, and a rating of the pharmacy’s “grade” on patient safety) will also examined. Intercorrelations will be explored at the individual and pharmacy levels of analysis. While the composites should be correlated since they measure aspects of the patient safety culture, the intercorrelations should not be extremely high (.90 or higher) because very high intercorrelations indicate that the composites may not be unique enough to be considered separate constructs or measures. While there is no steadfast criterion about the magnitude of dimension intercorrelations and construct validity, in general, such correlations should be less than .80 for the composites to be considered unique and avoid problems with multicollinearity.
The above analyses will be used to determine which items and composites are poorer functioning and remove them from the survey to derive a final set of items and composites with good psychometric properties and reduce the overall length of the final survey. The Technical Expert Panel will be informed of the data analysis results and be allowed to weigh in on which items to retain or drop when the psychometric results do not provide enough guidance and decisions can be made based on the content value of the items alone.
The final survey will be made available in the public domain for use by pharmacies, pharmacy systems/chains, and researchers to assess patient safety culture.
No exemption is being requested.
List of Attachments
Attachment B – Cognitive Interview Guide
Attachment C – Draft Pretest Questionnaire
Attachment D – Pretest Cover Letter and Reminder Notice
Attachment E – Draft Pharmacy Background Questionnaire
Attachment F – Federal Register Notice
Attachment G – List of Patient Safety Expert Reviewers and Consultants
1 National Association of Chain Drug Stores, Inc. (2009). Facts & figures. Accessed May 28, 2009, from the http://www.nacds.org/user-assets/pdfs/Facts_Resources/2007/Retail_Outlets2007.pdf
2 Flynn, E.A., Barker, K. N., and Carnahan, B. J. (2003). National Observational Study of Prescription Dispensing: Implications for Practice, J Am Pharm Assoc, 43(2).
3The Hospital Survey on Patient Safety Culture was pilot tested (OMB # 0935-0115, exp 1/31/2004) and made available to the public in November 2004; the Nursing Home Survey on Patient Safety Culture was pilot tested (OMB # 0935-0132, exp. 7/31/2010) and made available to the public in November 2008; and the Medical Office Survey on Patient Safety Culture was pilot tested (OMB # 0935-0131, exp. 7/31/2010) and made available to the public in December 2008.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | AHCPR |
| File Modified | 0000-00-00 |
| File Created | 2021-02-01 |