Supporting Statement Part B -- Evaluation of Phase I Demonstration 10-23-2009

SUPPORTING STATEMENT

Part B

Evaluation of Phase I Demonstrations of the

Pharmacy Quality Alliance

Draft dated: March 30, 2009

Agency for Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 4

4. Tests of Procedures 5

5. Statistical Consultants 5

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The results of this study’s survey of pharmacists will provide insights into the most effective approach for disseminating pharmacy performance measures to pharmacists. Pharmacists in five demonstration sites will be surveyed, and the study’s analysis will illuminate what worked and what did not work in those sites. Study personnel are aware that results will not be scientifically generalizable but rather will provide significant and important “lessons learned.”

The respondent universe will consist of the pharmacists who are working at the pharmacies in the five demonstration sites and who participated in the demonstration projects. To determine our survey universe, we will request a list of the participating pharmacists from each demonstration project. If there are more than 100 participating pharmacists at a site, we will select a simple random sample of 100 of those pharmacists, and those pharmacists will be our survey sample. If there are fewer than 100 participating pharmacists at a site, we will include all of them in our survey sample. As shown in the table below, we expect a response rate of 75 percent in each demonstration.

Demonstration	Number of pharmacists in the sample frame	Sample size	Expected response rate	Expected number of responses
Highmark collaboration	138	100	75%	75
Outcomes Pharmaceutical Health Care collaboration	200	100	75%	75
Pharmacy Society of Wisconsin collaboration	125	100	75%	75
Purdue University School of Pharmacy collaboration	55	55	75%	41
University of Iowa collaboration	60	60	75%	45

NOTE: Totals across the demonstrations are not relevant for this study, because the data will only be analyzed within each demonstration. Data from different demonstrations will never be combined.

We expect these sample sizes to provide the degree of precision needed for our study. Because we are interested in qualitative lessons learned from implementing pharmacy performance reports, our focus is on identifying substantive (or large) differences across the demonstration projects, i.e., differences in aspects of implementation that produce large differences in outcomes such as pharmacists’ perceptions about the usefulness of the reports . The ability to identify small differences across the demonstration sites would not be useful for our purposes. We expect that the sample sizes above will be sufficient to enable us to identify large differences, but would not provide the precision to identify small differences.

2. Information Collection Procedures

As described above, the sample for the pharmacists survey will be a simple random sample of 100 pharmacists in each demonstration project, unless the demonstration project has fewer than 100 pharmacists, in which case all the pharmacists will be included in the sample. A single advance letter will be sent from the CNA team (CNA and Thomas Jefferson University) to the pharmacists on AHRQ letterhead (see Attachment E). The letter will notify the pharmacists that they will be receiving the mailed survey in approximately two weeks. It will also describe the purpose of the survey, set expectations regarding the amount of time required to complete the survey, and assure confidentiality.

The survey of pharmacists will be a self-administered paper survey and will be accompanied by a cover letter on AHRQ letterhead (see Attachment F), sent by the CNA team. It will include a postage paid business reply envelope for the convenience of the respondents.

Our estimation approach will be to compute simple means and proportions for each site and then determine which of the differences in pharmacy survey items are statistically significant across sites. Variance estimation will be straightforward because we will be using a simple random sample in each site. Because the study is not concerned with producing generalizable estimates, we will not be calculating any estimates that combine the data for all the sites.

3. Methods to Maximize Response Rates

This study will use a survey of pharmacists in five demonstration projects that are beginning to use pharmacy performance measures. Survey questions will include how well the pharmacists understand the performance measures and how clearly those measures are being reported to them.

To maximize the response rate, the survey will be designed to facilitate completion in the context of pharmacy workflow: (1) it will be relatively brief, requiring only about 30 minutes to complete; and (2) the pharmacist will be able to leave and return to the survey at any time if s/he is interrupted by other job responsibilities, such as tending to patients or to prescribing physicians. Interruptions are expected given the nature of pharmacy practice, so the ability to return to the survey is essential to obtaining the target number of responses.

In addition, to maximize the response rate, the CNA team will send a letter on AHRQ letterhead to the sampled pharmacists two weeks after sending the survey to remind the pharmacists about the survey (see Attachment G).

The information from this survey is not meant to be generalizable to all pharmacists. Instead, it will be used to help understand why one demonstration’s approach to disseminating pharmacy performance measures to pharmacists might have worked better than another demonstration’s. The response rates resulting from the methods described above are expected to be sufficient for that purpose.

4. Tests of Procedures

To help ensure the effectiveness of the survey of pharmacists, the study has pretested the survey questions using a group of five practicing pharmacists employed by a health system in Philadelphia. These pharmacists were chosen from the approximately 80 registered pharmacists employed by that health system and had responsibilities and training similar to that of the demonstration site pharmacists. The pharmacists in the pretest were provided with training materials developed by one of the demonstration sites and a mock performance report based on an actual de-identified demonstration site report.

Results from this pretest were used only for internal learning purposes to improve the utility of the survey questions and minimize respondent burden. The pretest results will not be combined with any data from the study’s actual survey, and they will not be publicly reported, as they were only for internal learning purposes.

5. Statistical Consultants

The following individuals have been consulted on quantitative statistical aspects of the design and/or will be collecting and/or analyzing the information.

• Joyce McMahon, PhD, Research Analyst, CNA, 703-599-4474

• Elizabeth Schaefer, MA, Research Analyst, CNA, 703-824-2992

• Laura Pizzi, PharmD MPH RPh, Associate Professor, , Jefferson School of Population Health, Thomas Jefferson University, 215-955-1159

• Erin Thomson, MPH, Research Analyst, Jefferson School of Population Health, Thomas Jefferson University, 215-955-3888

5