Attachment Q - Response to Public Comments FINAL 1 14 10

Attachment Q: Responses to Public Comments

on the Health IT Community Tracking Study 2009

In accordance with the Paperwork Reduction Act of 1995, 44 U.S.C. 3506(c)(2)(A), the Agency for Healthcare Research and Quality (AHRQ) published a 30-day notice in the Federal Register on September 1, 2009 regarding the project ‘‘Health IT Community Tracking Study 2009.’’

AHRQ received comments from the following organizations and individuals during this period (listed in alphabetical order):

• American Academy of Pediatrics (AAP)

• American Pharmacists Association (APhA)

• National Association of Chain Drug Stores (NACDS)

• Nelson King, PhD, Assistant Professor, American University of Beirut (King)

• Surescripts

AHRQ appreciates these parties’ interest in the project and their thoughtful comments. AHRQ also appreciates that all of the parties, while representing differing perspectives, express support for the study. They concur with the need for and value of the study to provide a better understanding of current electronic prescribing (e-prescribing) implementation in physician practices and pharmacies and to assist health care policy makers and other stakeholders in developing strategies to promote adoption and effective implementation of e-prescribing, particularly in pharmacy settings.

The parties also provided specific input on the study design. After careful consideration of all of the comments, AHRQ believes that the current qualitative design of the study will best address most of the issues raised. We have, however, made several revisions to the protocols and one change to an invitation letter in response to selected comments.

AHRQ provides a general response to the comments in Section 1 below, followed by responses to the each of the individual comments and suggestions. Because the parties sometimes comment on similar issues, AHRQ has grouped responses under the major components of the research design as follows: Section 2 - Respondent Selection, Section 3 - Protocol Topics and Questions, Section 4 - Data Collection Methods, and Section 5 – Dissemination. AHRQ has indicated changes to the protocols and the invitation letter where appropriate.

General Response

AHRQ believes the proposed qualitative design is well-suited to the exploratory nature of this study. Prior research has identified some barriers to using e-prescribing but has not explored in depth the sources and causes of the barriers and how physician practices and pharmacies have responded, which can uniquely be accomplished through qualitative research. Several key features of the proposed design allow AHRQ to address the concerns and suggestions in the submitted comments. These are:

• Flexibility to gather information about, and speak directly with, potential respondent organizations to identify eligible organizations and knowledgeable respondents.

• Use of semi-structured interview protocols with broad, open-ended questions allows researchers to identify the issues that are most important to respondents and lets respondents discuss the topics in their own words. Because this is not a survey, interviewers can explain the questions to respondents if something is not clear and can also probe on the meaning of the respondent’s answers and the source of the information they are providing if it does not appear to reflect the specific experience of their organization.

• Ability to inform respondents that if, at any time during the interview, they do not feel comfortable answering a question, or do not know the answer, the researcher will go to the next appropriate question. 

• Ability to collect important contextual information during interviews to guide later data collection and analysis through triangulation of findings and identification of additional study respondents.

As indicated in “Section 4. Tests of Procedures” in Supporting Statement Part B, interview protocols for physician practices and other respondent types were drawn from questions used successfully by AHRQ’s contractor, the Center for Studying Health System Change (HSC), and other researchers in previous studies on e-prescribing and ambulatory electronic health records. Because AHRQ has not identified any qualitative studies on pharmacies’ experiences with e-prescribing, the pharmacy protocol was cognitively tested with two individuals, including a pharmacist and a pharmacy school student with retail pharmacy experience with e-prescribing. In addition, the study design and interview protocols were reviewed by the project’s consultant, who is a professor, researcher, and pharmacist and who has substantial experience developing and fielding interview protocols, focus group discussion guides, and surveys for pharmacists and physicians on e-prescribing, including as part of the Centers for Medicare and Medicaid Services-funded pilot projects testing e-prescribing standards.

More generally, HSC has substantial experience in qualitative research. HSC researchers, including this study’s project director, have been conducting interviews with providers and other health care leaders using this same general qualitative design in the 12 study communities through HSC’s longitudinal Community Tracking Study (CTS) since 1996. They have published numerous articles in peer-review journals based on this data collection.

Respondent Selection

Physician practices

Comment(s): The American Academy of Pediatrics (AAP) (p. 1) notes that pediatricians face specific challenges to implementing e-prescribing such as e-prescribing systems that cannot calculate medication dosages for children. They recommend that the study include pediatric practices and, if this is not possible, that AHRQ consider a separate study of pediatric practices. They also recommend that the study include both solo/small group practices and large group practices.

Response: “Section 1: Respondent Universe and Sampling Methods” of Supporting Statement Part B details the study’s proposed approach for the purposeful selection of physician practices to vary in size and specialty. Unfortunately, the small number of participating physician practices (24) precludes guaranteeing stratification of respondents by specific specialties such as pediatrics. AHRQ, however, welcomes AAP’s offer to help identify potential participants in the study communities.

Comment(s): King (pp. 2, 5, 8) suggests clarifying the eligibility criteria for physician practice participation because some practice IT administrators/office managers that send prescriptions from their computer to pharmacies may not be aware if the prescriptions are being sent electronically through Electronic Data Interchange (EDI) or via computerized fax.

Response: AHRQ agrees with King that some potential respondents may not be aware of the difference and that the wording is challenging because there is no standard industry terminology that is widely understood.

Given the lack of widely-understood terminology, the advantage of a qualitative study is that AHRQ’s contractor, HSC, can engage in discussion to further clarify the eligibility criteria during recruitment. HSC’s experience is that there are knowledgeable individuals within practices who know this information. Even if a specific respondent is unaware of the difference between EDI and electronic fax, prior communication will ensure the appropriate organizations are invited to participate. As noted in “Section 1: Respondent Universe and Sampling Methods” of Supporting Statement Part B, AHRQ will use data from Surescripts to identify practices capable of EDI before recruitment is initiated. Data from Surescripts’ Web site identifies physicians and pharmacies that transmit prescriptions electronically via the Surescripts electronic prescription routing network, which is the largest in the United States.

However, even when a practice’s e-prescribing system sends a prescription electronically, it may be converted to fax for one of several reasons. For example, Surescripts indicates on their website that prescriptions may be converted to fax if the pharmacy is not yet enabled for e-prescribing, the medication is a controlled substance (where faxing is permissible), or a temporary loss of network connectivity has occurred. Which delivery method is used for any particular prescription—electronic or fax—may or may not be transparent to a practice that transmits prescriptions electronically.

To clarify the eligibility criteria in this regard, AHRQ modified Attachment C: Invitation Letter – Physician Practice by adding the words indicated in italics below:

To participate:

• Your practice should be using e-prescribing software currently, and,

• Your practice should regularly send prescriptions directly from your e-prescribing system to pharmacies electronically, instead of by fax. (We recognize that prescriptions sent electronically by your practice may sometimes be received at the pharmacy by fax. This does not affect the practice’s eligibility for participation in the study.)

Comment(s): King (p. 4) suggests “a contrast should be made between lead users and the reluctant users. Presumably the medical officer willing to talk to you is a lead user. You can’t ask this directly.”

Response: The questions in Interview Protocol 2 – Physician Practice Medical Director or Physician User are intentionally worded to ask how the practice and its physicians are using e-prescribing, creating opportunities for interviewers to probe on the extent to which there are differences among users. For example:

Q.AEX.C.01. Could you briefly walk us through how the e-prescribing system is used in your practice to write new prescriptions and transmit them to the pharmacy? For each step in the process, indicate who in your office uses the system (e.g. office staff, nurses, or physicians).

Q.AEX.C.A.01. Do physicians have access via the e-prescribing system to information on patient medication history from a third-party vendor, such as Surescripts, at the time they are writing a prescription?

Chain pharmacies

Comment(s): NACDS (pp. 1-2) articulates concern that interviewing the “Pharmacist-In-Charge” would “not provide the most helpful information in response to the question you plan to ask” and recommend that, for chain pharmacies, “a person at the corporate headquarters who has oversight and responsibility for e-prescribing operations” be interviewed “to provide a more informed response with respect to question concerning policies and strategies.”

Response: AHRQ is interested in learning about the first-hand experiences of e-prescribing users in the field, including physicians and pharmacists. Based on consultations with pharmacists and other experts—who are identified in “Section 8b. Outside Consultations” in Supporting Statement Part A —we identified the pharmacist-in-charge as the most knowledgeable respondent about hands-on experiences with e-prescribing in independent and chain pharmacies. Based on these recommendations and the ability of our pilot test respondents to adequately answer the protocol questions (see Section 4: Tests of Procedures in Supporting Statement Part B), we continue to believe that the pharmacist-in-charge is the appropriate respondent to interview regarding day-to-day experiences with e-prescribing.

AHRQ agrees with NACDS, however, that for chain pharmacies, corporate headquarter respondents should be interviewed to learn about corporate policies, strategies and experiences regarding e-prescribing. Time for interviews with such respondents has already been included in the study design. As specified in “Section 1: Respondent Universe and Sampling Methods” of Supporting Statement Part B, we will contact “other pharmacy respondent organizations including mail-order pharmacies, which are typically regional or national; national pharmacy chain headquarters, to learn about company-wide e-prescribing initiatives; and state pharmacy associations, which will provide a market perspective on pharmacy experiences with e-prescribing as well as insights into relevant state regulatory issues.” Exhibit 5. Target Respondent Organizations and Individual Respondent Types specifies the target allocation of respondents for this category. AHRQ welcomes NACDS’ offer to help identify potential respondents.

Mail-order pharmacies

Comment(s): King (p. 6) queries whether it is necessary to ask whether mail-order pharmacies operate in the CTS sites as is done in Q.AEX.I.01.b since mail order pharmacies operate across state lines.

Response: AHRQ will include a mix of national and regional mail-order pharmacies that operate in the 12 communities as indicated in “Section 1: Respondent Universe and Sampling Methods” of Supporting Statement Part B. The purpose of the question is to identify regional mail-order pharmacies that operate in the study sites.

State pharmacy associations

Comment(s): Surescripts (p. 6-9) and NACDS (pp. 1-2) raise concerns about the ability of state pharmacy associations to reliably respond to many of the questions in the Interview Protocol 4 because they are less likely to have hands-on experience with e-prescribing systems. NACDS recommends that AHRQ interview state pharmacy representatives with “extensive, hands-on” e-prescribing experience or “knowledge gained through systematic collection and analysis of data from a representative sample of their member pharmacists who are users of e-prescribing.” Surescripts recommends deleting a specific set of questions on Protocol 4, including questions about the extent to which pharmacies and physicians in the market use e-prescribing and questions on the experiences of pharmacies in the market with e-prescribing and their perceptions about the effects of e-prescribing.

Response: The proposed study explores physician practice and pharmacy experiences with e-prescribing in 12 communities located in 12 states. As noted in “Section 1: Respondent Universe and Sampling Methods” of Supporting Statement Part B, “State e-prescribing regulations vary across the 12 markets as does the relative market shares of different local, regional and national pharmacies.” To help better understand variations in physician practice and pharmacy experiences that arise from these types of market differences, AHRQ has proposed including state pharmacy association respondents, who can provide a “market vantage” perspective on each of the 12 markets being studied. As described further on p. 4, state pharmacy associations…”will provide a market perspective on pharmacy experiences with e-prescribing as well as insights into relevant state regulatory issues.” Furthermore, as noted on p. 3, AHRQ will request assistance from these respondents in identifying pharmacy respondents for the study.

Given the qualitative design of the study, AHRQ does not believe it is necessary to limit which state pharmacy association representatives participate or which questions are asked, as proposed by NACDS and Surescripts. HSC has been studying the 12 study communities since 1996 and has found value in including such “market vantage” respondents routinely in its studies. In consultation with experts, AHRQ identified the state pharmacy associations as the best candidates for the “market vantage” role. Subsequently, executives at two state pharmacy associations located in CTS sites were consulted to learn more about their role and to get their input on some aspects of study design, as noted in “Section 8b. Outside Consultations” in Supporting Statement Part A. AHRQ will work with the state pharmacy associations to identify the most appropriate respondent to cover the interview protocol topics, as indicated in Attachment E – Interview Letter – Other Respondent Types. It has been HSC’s experience that when contacting organizations to participate in a study, if the original contact is not the most knowledgeable person, HSC is typically referred to a more appropriate candidate, either within the organization or elsewhere. In the case of the state pharmacy association, possible examples include referrals to a member pharmacist or a pharmacist serving on the state’s pharmacy board.

The information collected from different respondent types in qualitative studies may be used for different purposes. The State Pharmacy Association Representative is solicited for broad information about community trends to provide market context for interviews with pharmacy and physician practice respondents and in interpreting results. The information provided by state pharmacy associations will not be aggregated with results from pharmacies, for example, in reporting specific experiences with e-prescribing. Furthermore, researchers take advantage of the qualitative format to follow up with respondents during interviews to understand the respondents’ perspectives and the reasoning behind generalizations. Such information is taken into account by the researchers in analyzing the data and triangulating among respondents. Lastly, it is clearly stated in the protocol introduction, which is read to respondents, that if respondents “at any time...do not feel comfortable answering a question, or do not know the answer” that the researcher will “move on.”

Comment(s): King (p. 6) also seeks clarification on what perspective the State Pharmacy Association representative is requested to provide, noting there are similar questions being asked of the Pharmacist-In-Charge (protocol 3), and querying if the respondent will be asked about the “views of the association in general” or “just one site.” King further asks if questions about how e-prescribing system features are used in pharmacies (Q.AEX.H.) are “directed at the pharmacy of the state representative.”

Response: As noted, this protocol is designed to elicit responses at the community and state levels, rather than asking specifically about the organization or its membership.

Comment(s): King (p. 6) subsequently suggests the interviewer ask “if surveys had been done on their members,” presumably to understand the source of the respondents’ information.

Response: As explained above, the qualitative nature of data collection allows the interviewer to probe on the rationale behind respondents’ generalizations.

Vendors

Comment(s): King (p. 7) questions if asking vendors for referrals to specific users “make[s] sense as most vendors sell across communities.”

Response: In HSC’s experience interviewing health IT vendors, there is a reasonable potential for respondents to know of users of their system in the CTS communities, or to have suggestions for other individuals to contact who can provide referrals.

Policy makers

Comment(s): King (p. 8) suggests interviewing policy makers using a new separate protocol to address any state/national issues raised by respondents.

Response: If, on review of the preliminary study findings, AHRQ feels it is appropriate to interview policy makers, these interviews can be allocated to the “Other E-prescribing Experts” category, using that protocol tailored for the specific respondents.

Sampling with replacement

Comment(s): King (p. 2) raises the concern that the study design does not provide “a mechanism to replace an insufficient response from one site with another.”

Response: AHRQ maintains that “sampling with replacement” is not an appropriate method for a qualitative study. To minimize insufficient responses, the researcher will guide the interview to ensure that the core topics are covered. While respondents may not be able to answer every question thoroughly, AHRQ does not anticipate these gaps will impact the ability to understand trends more generally across respondents. In the unlikely event that a core piece of information is missing that precludes rigorous analysis of a key study topic, the researcher can pursue limited follow-up by email to obtain needed information.

Protocol Topics and Questions

Recommendations to add topics, questions or specific question probes

Comment(s): The American Pharmacists Association (APhA) (pp. 2-3) recommends expanding the research questions to capture data on the quality of the prescriber and pharmacist user interfaces built into e-prescribing systems and the quality of the e-prescriptions that these systems generate. Specific suggestions include:

• Significant number of telephone calls from pharmacists to prescribers to address e-prescription errors and to seek clarifications.

• Administrative, workflow, and financial burdens on pharmacists/pharmacies to address errors and to seek clarifications.

• Lack of authority from the Drug Enforcement Administration (DEA) to allow e-prescribing of controlled substances.

• Design issues with computerized physician order entry (CPOE) systems (additional subtopics listed in letter on p. 2).

• Design issues with pharmacy operating systems (additional subtopics listed in letter on pp. 2-3).

Response: AHRQ believes that the current semi-structured interview design of the protocols will allow for adequate discussion on these important topics suggested by APhA. A hallmark of exploratory qualitative research is use of open-ended questions, which allow respondents to discuss their own experiences in their own words without skewing results towards issues that researchers perceive a priori to be important. We are confident that a set of core, open-ended questions in the discussion guide will elicit the information that APhA identifies as important if the respondents believe they are significant issues.

For example, most of the issues raised by APhA will be addressed in pharmacy responses to the following existing protocol questions (as well as corollary questions asked of physician practice respondents):

Q.AEX.H.01. Could you briefly walk us through how pharmacy staff receives new prescriptions generated from physician e-prescribing systems, enter them into the work queue and fill them? Please indicate the ways in which computer generated prescriptions are handled differently from non-computer generated prescriptions.

Q.AEX.C.F.03. What are the most common reasons physicians and pharmacists communicate about prescriptions that are computer-generated, whether sent electronically via electronic data interchange or by other means?

Q.AEX.H.06. What are the major factors, if any, facilitating the receipt and processing of electronic prescriptions?

Q.AEX.H.07. What are the major challenges, if any, to receiving and processing electronic prescriptions? [Probe on how, if at all, the pharmacy and staff [in this site] addressed these challenges.]

Q.AEX.D.02. What have been the perceived effects of using e-prescribing on pharmacy operations and patient care? Probe if necessary on:

Q.AEX.D.02.a. Overall pharmacy efficiency?

Q.AEX.D.02.b. Pharmacist and technician efficiency?

Q.AEX.D.02.c. Volume and type of phone communications with physician practices about new prescriptions and renewals?

The broad, open-ended questions in the protocols are grouped with more detailed probes and follow-up questions. It is HSC’s experience that respondents typically address most of these issues in response to the initial questions. The follow-up probes and questions serve, in many cases, as reminders for interviewers to follow up on consistently on key issues if they are not covered by the respondent and to facilitate coding the data when the specific topics are addressed in response to the initial question.

Given the broad interest among public and private stakeholders around e-prescribing of controlled substances, AHRQ has included a follow-up question to ensure that this topic is discussed. In the physician practice protocols (Interview Protocols 1 and 2), the following question is included after the question asking about the most common reasons for using methods to send prescriptions other than electronic transmission:

Q.AEX.C.F.01.g. How do the Federal Drug Enforcement Agency’s restrictions on the electronic transmission of prescriptions for controlled substances affect the method the practice uses to send prescriptions (e.g.if one of the patient’s prescriptions is for a controlled substance)?

Similar questions have also been added to Interview Protocols 3 - 7.

Comment(s): King similarly makes a number of suggestions about places to add additional probes or follow-up questions and at one point, suggests having a checklist of response categories on the protocol to aid the respondent. Specific suggestions include:

• King suggests additional “fallback question[s]” for Q.AEX.C.A. as well as other questions in the Q.AEX.C. series (pp. 3-4).

• King suggests listing the features the office manager described and confirming these are the ones in use (or at least available) with the physician respondent (p. 4).

• King suggests adding a new question Q.AEX.C.00. to get a sense of the knowledge of the physician respondent about the system (p.4).

• King suggests a series of questions about how the correct pharmacy information is entered into the e-prescribing system before the prescription is sent:

  • Q.AEX.C.F.01.g Who in the practice populates the patient’s choice of pharmacy? (p. 3)

  • Q.AEX.C.F.01.h What options does your e‐prescribing system offer in terms of choice of pharmacy? Different pharmacy for a particular drug? No pharmacy choice (patient decides later)? (p. 3)

  • Q.AEX.C.D.01.a. Does the physician have to fill in the preferred pharmacy information of the patient? If not, who does it? (p. 4)

  • Q.AEX.C.D.01.a. How does the physician decide on which pharmacy to send the prescription? How long does this take? (p. 4)

• King suggests a follow-up to Q.AEX.C.F.02.a., Does the practice want to know about whether the patient picked up the prescription? Doesn’t this introduce liability issues for the practice? (p. 03)

• King suggests that a checklist of questions to aid respondent be added for Q.AEX.C.B.03. (p.3) and Q.AEX.C.B.02. and Q.AEX.C.F.05. (p. 4).

• King suggests adding a question, Q.AEX.G.02.a., about the e‐prescribing features in the installed version (p.5).

• King comments that pharmacists are not asked directly about DDI/DAI (p. 6) and about medication histories (p. 6).

• King suggests elaborating on Q.AEX.K.05. about certification to ask about specific services being certified (p.6).

• King indicates that nothing was asked about integration of pharmacy e-prescribing functionality with the adjudication side of their systems (p. 7)

• King suggests asking e-prescribing vendors about the ability to override features and defaults programmed into the system (“hidden agendas”) (p. 7).

• King suggests asking other e-prescribing experts what set of features borrowed from each vendor would satisfy different types of users (p. 8).

• King suggests asking e-prescribing connectivity and content vendors whether the difference in formulary standards has been resolved (p. 8).

Response: As noted above, AHRQ maintains that the open-ended approach used in the proposed protocol questions is the best way to collect data from respondents on the issues that they consider most significant. Specific responses to some of King’s other suggestions for question modifications are as follows:

• King makes three comments regarding physician practice use of surrogates (pp. 3, 5). - Question Q.AEX.C.01.already has a probe regarding the role of office staff.

• King comments that in Q.AEX.D., there is no question on preventing prescribing error (p. 4) and also notes there is no “mention” of medication errors (pp. 4, 6). - Question Q.AEX.D.02.e. specifically probes on the effects of prescribing safety and quality.

• King suggests an additional question, Q.AEX.G.02.c. If the practice doesn’t have e‐prescribing, what will it take (cost, resources) to upgrade? (p. 5) - This information will be elicited by questions in series Q.AEX.G.04.

• King suggests asking e-prescribing vendors about what features are used and not used by their customers as the “usage experience” (p. 7). - This information will be elicited by question Q.AEX.C.F.06. to vendors about the benefits and challenges of e-prescribing reported to their customers among other questions.

• King suggests adding additional questions for vendors about certification (p. 7). – This information will be obtained from secondary sources prior to the interview.

• King suggests asking e-prescribing vendors how the system knows which is cheapest among multiple generics (p. 7). - This information will be elicited by question Q.AEX.C.B.01., Q.AEX.C.C.01. and relevant sub-questions.

• King suggests asking e-prescribing connectivity vendors for questions Q.AEX.C.A.04. and Q.AEX.C.B.03. “Can you share the data on how many transactions are attempted that data is incomplete when sent back to the physician?” (p. 8) - This information will be elicited by question Q.AEX.D.02. and relevant sub-questions.

Comment(s): Some of the issues APhA and King suggest exploring are outside the scope of this study, unless they are identified by respondents as challenges to effective e-prescribing use in physician practices and pharmacies or related to the perceived effects of using e-prescribing on pharmacy and practice operations and patient care. The issues AphA (pp. 2-3) mentions are:

• Confusion over what a “certified” e-prescribing software system/vendor means for the quality of an e-prescribing system.

• Unrealistic views of the quality of e-prescriptions in the current market.

• Unfair financial burdens on pharmacies for all transaction fees related to the transmission of e-prescriptions (original, two-way follow-up, and re-sent/corrected).

• Potential to reduce e-prescribing errors and improve patient safety if the indication of use/diagnosis is included on e-prescriptions.

• Ongoing need for feedback data on the quality of e-prescriptions not just the quantity being transmitted.

• E-prescribing should not create additional administrative or financial burdens on pharmacies or pharmacists.

• E-prescribing should be treated as more than just an electronic transaction (e.g. it is not simply a one-way transaction like an online bill payment). All efforts to increase the utilization of e-prescribing must ensure that systems are in place to protect patient safety and the ongoing health care interactions among the patient, prescriber, and pharmacist.

The issues King mentions that are out of the scope of the study unless they are raised by respondents are:

• Volume of prescriptions generated (p. 4).

• Insurance mix of patients visiting the physician practices and pharmacies (pp. 4-5)

• Appropriateness of pharmacy transaction fees related to the transmission of e-prescriptions (same topic raised by AphA) (pp. 6-9). King makes four suggestions for adding specific questions on this topic.

• What physicians are going to do with all this information that might be available to them (p. 4).

• Patients’ involvement in e‐prescribing (p. 4).

• Willingness of vendors to curtail competition for the sake of a national good such as collaborating on and adopting a more effective alert system (p. 7).

• Features in the vendor systems recommended in the literature (for manual prescribing) to reduce medication error (e.g., diagnosis) (p. 7).

• Reasons for and implications of e-scripts getting “pushed” from the prescriber to the pharmacy (p. 8).

• What is being done with the existing systems (e.g., manual prescribing with electronic adjudication) to better understand why stakeholders would invest in a new system (p. 9).

• Allocation of benefits among e‐prescribing stakeholders and how e-prescribing fits in with broader medication management systems (p. 9).

Recommendations to drop questions

Comment(s): NACDS (p. 2) and Surescripts (p. 2 and p. 5) suggest deleting the following question with follow-ups, which is asked of multiple respondents.

Q.AEX.H.04. Do you typically send other types of electronic messages besides renewal

authorization requests to practices with e-prescribing (e.g. delivery confirmation, change

requests)?

Q.AEX.H.04.a. If yes: How does pharmacy staff use this feature?

Q.AEX.H.04.b. If yes: How frequently is this feature used?

Surescripts (pp. 3- 4, 5, 9-10) also identifies other protocols where a similar question and follow-up is asked:

Q.AEX.C.F.02.a. Does the practice typically receive any (other) electronic communications from pharmacies that accept electronic prescriptions (e.g. delivery confirmation, notification of whether the patient picked up the prescription, change requests)?

Q.AEX.C.F.02.b. Does the practice typically use the e-prescribing system to respond to these notifications electronically via electronic data interchange (e.g. sending electronic renewal authorizations, denials, cancellations, changes)? (Note that Surescripts’ concern applies only to the latter three functions in the parentheses since the ability to send electronic renewal authorizations is currently functional).

NACDS and Surescripts suggest that, because two-way electronic communication between physician practices and pharmacies via Surescripts is currently limited to sending new prescriptions and renewal requests and authorizations, including questions about expanded functionality, for example, delivery confirmation, “would be confusing and/or misleading to the interviewee and of questionable value to the study.”

Response: AHRQ included these questions because efforts are underway to expand the capabilities of physician practices and pharmacies to communicate electronically via e-prescribing. First, because it was uncertain what functionality would be available by the time the data collection for this project would begin, the questions were included for completeness. Secondly, pharmacies may have other means of communicating electronically with physician practices besides using Surescripts, for example, pharmacies that have close affiliations with particular physician practices. However, to the extent that no respondents are currently using such functions, HSC’s experience is that they are most likely to respond by saying “No,” as did our pilot respondents. The flexible nature of qualitative questioning minimizes the possibility of confusion (compared with a survey) because, if a respondent lacks experience with a particular issue area, the researcher can explain the question to avoid confusion. As written, the question is clearly stated and is unlikely to cause confusion or be misleading, especially since the researcher will not initially read the information in parentheses—delivery confirmation and change requests. Some questions have such parenthetical information to help researchers explain the question, but are read only if needed to maximize the amount of time respondents have to respond to questions. Lastly, as stated above, respondents are not required to answer every question, and can ask to move on to another topic if they so desire.

Comment(s): NACDS and Surescripts also suggest deleting the following question “because it is a general pharmacy regulatory issue and not directly pertinent to electronic prescribing. Thus, it is not relevant to the purposes of the survey.”:

Q.AEX.H.05. If a prescription is written for a brand-name medication when therapeutically equivalent generic medications are available and “Dispense As Written” is not indicated, does the pharmacy typically make a generic substitution or consult the prescribing physician about the possibility of a substitution?

King (p. 5) comments that this same question, Q.AEX.H.05, is an “important question” as well as “one of most overlooked variables in studies advocating cost benefit of e‐prescribing” and recommends that it be rephrased into multiple questions.

Response: AHRQ has included this question because an important focus of the study is the effects of e-prescribing on generic prescribing. One of the potential benefits of e-prescribing is the ability to increase the rate of filled generic prescriptions, where clinically appropriate. AHRQ maintains the view that this question should be included because understanding pharmacy policies regarding generic substitutions generally (which may vary by study community) will provide valuable context to better understand how e-prescribing specifically affects pharmacist behavior regarding generic prescriptions. However, AHRQ disagrees with King that the question should be rephrased into multiple questions because, although it provides important context, it should not take priority over other questions more focused on use of e-prescribing.

Comment(s): Surescripts (pp. 2-11) also suggest the deletion of additional questions because “the stakeholders being interviewed have no direct experience that would allow them to reliably answer the question” including:

• On pp. 3-4, a series of questions asked of the two physician practice respondents about whether physicians have access to patient medication histories from a third-party vendor at the time a prescription is written, whether physicians have access to formulary data at the time a prescription is written, the vendors that provide these data as well as the e-prescribing system’s medication database, and how often the information is updated.

• On p. 5, questions which ask pharmacists to speak more generally about e-prescribing in their community, for example, which pharmacies have the largest share of the local market and which have particularly high or low rates of e-prescribing and what are the reasons some pharmacies are not accepting prescriptions electronically.

• On pp. 9-10, a question and a follow-up which ask Pharmacy IT and e-prescribing system vendors about what are the most common reasons physicians and pharmacists communicate about computer-generated prescriptions.

Response: AHRQ disagrees with the need to delete these questions. With respect to the first set of questions, HSC’s prior research experience indicates that physician practice IT administrative staff, medical directors and other e-prescribers are often familiar with the functionality of the e-prescribing software they use. For example, by design, a physician should be able to identify whether patient medication history is available that was not entered directly by the respondent themselves or other physicians in the practice. In fact, these topics, which are suggested to be outside the scope of respondents’ knowledge, arose in HSC’s prior study on e-prescribing with those same respondent types but were not sufficiently explored at that time. For example, multiple respondents mentioned that they believed formulary information was not reliably updated and described their practices’ work-arounds to address the issue. HSC also has extensive experience working with practices willing to participate in health information technology studies and they often identify respondents that together can cover the range of administrative and clinical questions that are of interest, so that questions not covered in one interview can typically be addressed in the other.

After reviewing the questions on third-party vendors, AHRQ has modified the questions on access to patient formulary information to be consistent with the questions on patient medication history. In the physician practice protocols (Interview Protocols 1 and 2), the following words in italics were added:

Q.AEX.C.B.01. Do physicians have access via the e-prescribing system to patient formulary information from a third-party vendor at the time they are writing a prescription?

A similar change was made in the e-prescribing vendor protocol (Interview Protocol 6).

With respect to the concerns about the ability of respondents to discuss differences among their peers raised in the second set of questions, in HSC’s experience with community-focused studies, respondents are often aware of these differences. In piloting the pharmacy protocol, HSC again found this to be true.

In the last set of questions, AHRQ believes it is reasonable to ask vendors about the issues that arise that require pharmacies and physician practices to communicate about e-prescriptions since vendors are likely asked to help address and resolve such problems when they arise. Reasons for pharmacy-physician practice communication could include, for example, the issues identified by APhA (p. 2) such as “wrong drug/dose/directions/pharmacy selection,” “transmitted with dual directions,” “transmitted with missing patient, prescriber and/or prescription information,” or “transmission fails to arrive at the pharmacy.”

Moreover, with any of the questions, knowing a particular respondent type routinely asks to move on to another topic itself provides useful insight into the state of e-prescribing. For example, if many physician respondents are unaware of whether patient-specific formulary data is available, it could suggest they infrequently refer to that information when writing the initial prescription.

It is also useful to note that information may be collected for different purposes and analyzed differently. For example, if respondents cannot discuss the medication database vendors they use, it would not hamper overall analysis of data regarding use of e-prescribing systems. However, it is still necessary to ask for this information because useful background information can be obtained from secondary sources regarding database features that could influence respondents’ experiences with e-prescribing. Similarly, information on peer organizations or market trends will be analyzed differently than information gathered about the experiences of the participating pharmacies and physician practices and will be used more generally to provide context for what is learned from other respondents. In both examples, the information can also be a valuable source for identifying potential study respondents such as e-prescribing content vendors and pharmacies with e-prescribing in local markets.

Comment(s): Surescripts (p. 9) suggests deleting another question on Interview Protocol 5—Pharmacy IT Vendor Representative. The question reads:

Q.AEX.H.08. What, if anything, do you hear from physician practices about the benefits or challenges of e-prescribing?

Response: Surescripts is correct that this question is worded inappropriately for this respondent type and AHRQ appreciates being notified of this error. The protocol has been revised by replacing this question with the following correctly-worded question:

Q.AEX.C.F.06. What, if anything, do you hear from pharmacists about the benefits or challenges of electronic prescribing?

Comment(s): King suggests the need to clarify some terms or makes specific wording suggestions including:

• King suggests explaining the term “stand‐alone” (p. 2)

• King suggests rephrasing Q.AEX.B.03.c as “what percentage of …” (p. 2)

• King suggests expanding questions Q.AEX.B.04. and Q.AEX.G.02 by asking “when was the e‐prescribing system last updated,” explaining what is meant— “just software update or system replacement” and asking “when do you plan to update the system next”, then having the researcher determine the interval rather than just asking the physician practice and pharmacy respondents how often the IT system is updated or upgraded. (pp. 2, 5)

• King states he does not understand Q.AEX.C.F.04. What are the major factors, if any, facilitating use of electronic prescribing with pharmacies? (p. 4)

• King suggests in Q.AEX.L.01. that “issues” should be replaced by “comments.” (p. 4)

• King suggests expanding question Q.AEX.F.03. to include other employees such as store clerks or interns. (p. 5)

• King queries whether it is necessary for the vendor to monitor all its customers and aggregate the data to answer Q.AEX.H.03.b. If yes: Approximately what proportion of all renewal requests to physician practices with e-prescribing are sent electronically via electronic data interchange? Approximately what proportion of responses to those electronic renewal requests is sent back electronically via electronic data interchange from those physician practices to the pharmacy? (p. 6)

Response: AHRQ does not believe any of the suggestions or concerns require making changes to the existing protocol questions. Nonetheless, AHRQ’s contractor, HSC, can clarify terms during the interview if necessary.

Comment(s): King (p. 3) suggests modifying Q.AEX.C.B.02 to say “inhibiting” rather than “facilitating” since earlier studies show that the specific formulary data is limited.

Response: AHRQ believes it is more appropriate to ask in a neutral manner about both the facilitators of and challenges to using formulary data. The question King references is followed by a corollary question, Q.AEX.C.B.03., which specifically asks about challenges. This approach is used consistently throughout the protocols in asking about the different e-prescribing features being studied.

Comment(s): King (p. 3) suggests moving the Q.AEX.C.F. series of questions, which cover physician practice-pharmacy communication including electronic transmission of prescriptions, to the technical detail questions asked earlier.

Response: The questions are appropriately placed. The purpose of these questions is to explore how practices use e-prescribing systems to communicate with pharmacies and how e-prescribing affects physician-pharmacy communication via electronic means or otherwise; they are not simply background technical questions. They follow the broad, open-ended questions that ask about how the e-prescribing system is used by the practice and the follow-up questions that apply to the earlier stages of writing prescriptions before they are sent to the pharmacy, such as use of information on medication history, formularies and generics.

Comment(s): King (p. 3) notes that some questions are missing including:

• Q.AEX.C.C.01.a.

• Q.AEX.C.F.01.a.

• Q.AEX.C.F.01.b.

• Q.AEX.D.02.b.

Response: These follow-up questions are intentionally excluded from the practice IT/office manager protocol. The questions ask how are certain features are used in actually writing prescriptions and are more appropriately included in the medical director/physician user protocol.

Comment(s): King (p. 3) states that “Q.AEX.C.C. presumes the physician will use generic information. The literature suggests that most physicians prescribe brand.”

Response: These questions do not presume that the physician will use the generic information. King refers to the specific questions from this series in Interview Protocol 1 – Physician Practice IT Administrator or Office Manager that ask what vendor provides the data and how often it is updated in the system. Additional questions from this series in Interview Protocol 2 – Physician Practice Medical Director or Physician User specifically ask how physicians use the information. (See the next comment and response for more information.)

Comment(s): King (p. 3) suggests adding a new question, Q.AEX.C.C.01.d. “Knowing your physicians, would they access generic medication information or prefer to just name the brand (and let the pharmacy choose the appropriate generic)?”

Response: This addition is not needed in Interview Protocol 1 – Physician Practice IT Administrator or Office Manager because the question is asked directly of physicians in Interview Protocol 2 – Physician Practice Medical Director or Physician User in a more neutral way as follows:

Q.AEX.C.C.01. In what ways, if any, does the e-prescribing system support generic prescribing by physicians?

Q.AEX.C.C.01.a. How are these features used in writing a patient’s prescriptions? [Probe on any differences between new prescriptions and renewals.]

Comment(s): King (p. 5) suggests that, in Q.AEX.H.02.a, it is important to also collect information on computer‐generated faxes as well.

Response: AHRQ agrees with this point. There are two questions and the second is currently worded to obtain information about computer-generated faxes.

Q.AEX.H.02.a. Approximately what proportion of all prescriptions at this location are prescriptions received electronically via electronic data interchange directly into the pharmacy’s computer system, not as computer-generated faxes? Approximately what proportion of all prescriptions at this location are prescriptions generated by physician e-prescribing systems but received via other means such as from patient, phone, manual or computer-generated fax, etc.?

General recommendations regarding protocols

Comment(s): King (pp. 2, 4-5) suggests prioritizing protocol questions because researchers may run out of time.

Response: The structure of the phone interview allows the researcher to focus on the most critical and relevant topics and prioritize questions based on participants’ expertise and knowledge as well as the interview pace. The protocol necessarily contains more questions than will be asked directly to ensure that adequate follow-up questions are available to researchers; for example, many probes are phrased in an “if yes:” and “if no:” format and would not be asked of the same respondent. As noted above, AHRQ’s contractor, HSC, has found in their experience that most of the key findings can be drawn from respondents’ answers to the core open-ended protocol questions, examples of which are provided above.

Comment(s): King (p. 1) suggests that some additional tailoring of protocol questions to respondent types would be helpful.

Response: As is typical in qualitative research, questions will be tailored to individual respondents as part of the interviewing process.

Comment(s): King (p. 1, 9) recommends that protocols be pilot tested.

Response: AHRQ does not feel that the protocols need further testing since the concerns raised by King will be addressed with the current design of the study. In addition, as noted above, interview protocols for physician practices and other respondent types were drawn from questions developed for previous studies on e-prescribing and ambulatory electronic health records and modified to reflect this study’s focus on particular e-prescribing features that have previously not been explored in depth. The pharmacy protocol was cognitively tested with two individuals. The protocols were also reviewed by the project consultant.

Data collection methods

Comment(s): King (pp. 1-2) suggests that “the protocol(s) in places seems better suited to an on‐line survey due to length of explanations” and that a “written (web-based) questionnaire” would “enrich the data.” King (p. 6) also suggests that state pharmacy associations should administer a survey to their member pharmacies or encourage them to answer questions online, rather than participating as respondents.

Response: Due to the nuance and complexity of e-prescribing systems and e-prescribing behavior, it is often necessary to ask follow-up, probing questions to truly understand participants’ responses. Most interview questions are thus open-ended to allow for in-depth exploration of issues. Precisely because of the probable length of explanations, a survey is not a viable data collection option for this exploratory study.

Comment(s): King (p. 7) later suggests AHRQ request permission for site visits from e-prescribing vendors.

Response: Site visits are outside the scope of the proposed data collection.

Comment(s): King (pp. 2-3, 8) makes several suggestions about providing information or questions in advance to respondents to free up interview time.

Response: AHRQ maintains this approach is not compatible with the proposed qualitative methodology. Additional responses to King’s comments are as follows:

• Include questions in the cover letter. - It is not customary to provide qualitative, open-ended research questions to respondents prior to the interview. AHRQ does not wish to increase respondent burden by encouraging them to research answers to questions ahead of time.

• Send the introductory information in advance instead of reading it at the beginning of the interview. - Respondents will receive written information about the study during recruitment and when their interview is confirmed, which they could refer to in the future if necessary. Information about the study topic as well as data collection procedures (e.g., confidentiality) is important to review by phone to ensure respondents’ comprehension and establish rapport.

• Ask for responses to background questions at the beginning of the protocol and all other questions before Q.AEX.C.01. in writing. - These questions will only be asked if background information is missing. HSC may already be familiar with the respondent organization, may have access to secondary information such as a website, or the respondent may volunteer this information during the screening process.

Comment(s): King (p. 5) comments that the 60 minutes allotted to the pharmacy respondent is disproportionate to the time allotted to the Office Manager (Protocol 1) and Physician (Protocol 2).

Response: In total, AHRQ is proposing to talk with physician practice respondents for 75 minutes (30 minutes with the IT administrator/office manager and 45 minutes with the physician respondent). The physician practice protocols together are slightly longer because there are more specific issues to be explored in understanding physician practice uses of specific e-prescribing features in depth.

Comment(s): King (p.6) later comments that 60 minutes is “too much” for the state pharmacy association representative interview.

Response: In HSC’s experience, it is prudent to allow an entire hour for “market vantage” interviews. The interviews will preferably be conducted at the beginning of the project to provide important background and context that requires in-depth explanation. If the discussion in fact takes less time than estimated, the estimated burden on this particular category of respondents would be lower than the estimate in “Section 12: Estimates of Annualized Burden Hours and Costs” of Supporting Statement Part A.

Comment(s): King (p.7) comments that the interviewer must be “tech-savvy.”

Response: AHRQ is confident that its contractor, HSC, has the level of expertise necessary to describe e-prescribing use among physicians and pharmacies. HSC has substantial experience conducting studies of health IT generally, and this study’s project director also has specific e-prescribing research experience. Additionally, the open-ended nature of the protocol questions ensures the interviewer can follow-up with respondents during the discussion to clarify any confusing terminology that may arise.

Comment(s): King (pp. 7-8) suggests that obtaining secondary material such as manuals and diagrams may be necessary to understand how pharmacies process new prescriptions generated from physician e-prescribing systems and that “screen captures, user manuals and demo versions” will be a necessary component to the e‐prescribing vendor and connectivity and content vendor protocols. King states, “Just asking for features

doesn’t tell you much.”

Response: AHRQ is confident that the semi-structured interview questions provide enough opportunity and structure to adequately explore the usability of e-prescribing features. The protocol questions do not ask for a list of features, but rather ask respondents to describe how physicians and pharmacists use the IT systems. Follow-up questions on specific features are included in the protocol as reminders to the interviewers to ask about specific functions if the respondent does not mention them. In addition, interviewers will review any publicly available information on the vendor’s products prior to the interviews. Respondents may also provide supporting documents, but AHRQ believes standardizing such requests would unnecessarily add to respondent burden, particularly because the questions have been successfully used in prior research on health IT, including e-prescribing.

Comment(s): King (p. 6) also suggests checking the Surescripts usage data for each of the 12 study sites, since it is “highly variable.”

Response: All relevant background information will be tracked as the study progresses.

Dissemination of Findings

Comment(s): NACDS (p. 1) and Surescripts (p. 2) comment that AHRQ has appropriately discussed the limitations of the qualitative study design but expressed concern that the limitations and applicability of the results be adequately acknowledged when the study is released. They recommend that the qualitative nature of the study be emphasized in any reports and other dissemination of findings.

Response: AHRQ and its contractor, HSC, follow the highest standards of the research community in describing research methods and interpreting findings in all dissemination efforts. Research reports and publicity documents are always subject to both internal and external review by experts. The study mentioned by Surescripts was published in a peer-review journal with a rigorous methods section and the associated HSC news release clearly stated the qualitative nature of the study, the number of interviews and types of respondents, and the time period of the data collection. Findings from this qualitative study were published in a timeframe consistent with comparable studies, and, as confirmed by other subsequently-published research, the findings were relevant at the time of publication, demonstrating the value of this type of exploratory qualitative approach in identifying and outlining the key facilitators of and challenges to e-prescribing.

16