REVISIONS AND RESPONSES TO PUBLIC COMMENTS
FROM FEDERAL REGISTER NOTICE REQUEST FOR INFORMATION
A PROTOTYPE CONSUMER REPORTING SYSTEM FOR PATIENT SAFETY EVENTS
Version: May 17, 2013
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
Introduction 3
1. Revisions to the design of the reporting system 3
2. Comments to the design that did not lead to revisions 5
3. Revisions to the intake form 9
4. Comments to the intake form that did not lead to revisions 12
5. Summary of revisions and changes in burden 15
6. Revisions to the supporting documents to the intake form 15
7. Themes of personal stories submitted 15
AHRQ received substantive public comments from 45 members of the public, some of which included supporting documentation. Please refer to Attachment I for the full text of the public comments and supporting documentation. The comments also included 64 personal stories. These comments and personal stories raised 37 issues in the wording of the intake form, two issues with wording in other supporting documentation to the intake form, and 69 individual design issues that we categorized into 18 types of design concerns. Below we have responded to each of the issues raised. First, we describe the revisions we made to the design of the system; second, we respond to the public comments that did not lead to revisions to the design; third, we describe the revisions we made to the intake form; fourth, we respond to the comments that did not lead to revisions to the intake form; fifth, we summarize how the revisions changed the calculated time of survey burden on respondents; and next, we discuss two revisions that we made to the supporting documents to the intake form. Lastly, we provide a table that categorizes the types of personal stories submitted.
Based on the 18 design concerns raised in public comments, we have proposed the following revisions to design of the reporting system and have provided clarifications to the design of the system to address the remaining 13 comments. Because the pilot system is still under development, the revisions are contingent on further modification in collaboration with the participating facilities in the pilot community. These revisions reflect clarifications of the intake and data management protocol, but not substantial changes in survey burden or intent.
Revision#1: A non-profit organization, physician organization, hospital association, and a consumer advocate requested clarification of the protocol for matching consumer reports to provider information and in this context, clarification of the protocol for ensuring the privacy of the patient. The following strategies will be used to protect patient privacy. First, no patient reports will be shared with professionals or facilities unless the patient has consented explicitly to such sharing and has designated the facility or provider that should receive the information. Second, all CRSPS staff will become members of the ECRI PSO. Under PSO statute, any material shared internally to the PSO regarding a report will include a header specifying that the included material is patient-safety work product (PSWP).
Patient or consumer reports will be matched through two separate procedures. Participating facility staff will attempt to find matches of consumer reports (with consumer permission) to incidents that have been reported to the facilities’ own internal systems. PSO staff will attempt to match incidents that have been reported by facilities to the PSO. In each case, incident reports will be matched based on the date of the event, setting, type of the event, and other details about what happened. Only if the consumer has consented to sharing names will names be used. (Previous experience with a Massachusetts study shows that such matching is feasible.) The results of the matching protocol conducted by the PSO would not be shared with the reporting health care facility or the reporting consumer. All data resulting from the matching attempts would be reported only in the aggregate with review to ensure that patient cannot be inadvertently identified.
Revision#2: A PSO service group, physician organization, and members of Congress asked how the system was designed to maintain the privacy of the consumer and the privacy of the provider. The comments asked for more explanation on how the privacy of the patient and provider is protected and whether the collected information would be available to medical malpractice proceedings. As described above, privacy protections will include the normal confidentiality and protections from legal discovery that are afforded to data collected as part of an AHRQ authorized research project. Information generated from an analysis conducted by the participating care delivery organizations or by the PSO will be marked as Patient Safety Work Product (PSWP); this protects the information, including provider identity, under the PSQIA regulation. All of the CRSPS staff will be members of the PSO. As noted, any public reports generated by the CRSPS or PSO will utilize only aggregated data with review to ensure that patient and provider cannot be identified based on a description of a single incident.
The CRSPS team will work with the pilot community to ensure that policies are in place to protect patients and will draft participation agreements covering the handling of data collected by the project. As covered entities, participating health care facilities will be required to handle consumer reported data as patient information under HIPAA regulation. Facilities will thus be unable to share it without patient consent. In addition, the CRSPS website will include a list of health care facilities that have no-blame policies for providers and that have signed a non-retaliation policy to protect patients from retaliation during and after the pilot project. In addition, consumers who provide information to CRSPS, maintain the right to use their own reported information for other purposes. Any CRSPS/PSO analysis using information that a health care facility provides would be PSWP under the PSO statute and therefore not discoverable by the patient, his/her representatives, or others.
Revision#3: Consumer advocates asked that the system protect patients against retaliatory actions by providers. Their proposals included a written policy and potentially incorporating an approach whereby retaliation could be tracked and monitored. To address the first concern, we will introduce the requirement as part of the pilot that participating facilities sign a non-retaliation policy (For an illustrative example of such a policy please refer to Attachment J) as a condition of participation in the pilot. We will post on the resource page of the CRSPS web site the list of the facilities and providers that have signed the non-retaliation policy. Current hospital complaint systems utilize this option, and the Joint Commission mandates it. On the idea of tracking retaliation, as part of the evaluation of the pilot system, we propose to make consumers aware that they have the option to contact the system if they believe they have been subjected to retaliation. We will also query participating facility advocates about potential reports of retaliation that may have occurred.
Revision#4: A physician organization hospital association, consumer advocate and a PSO service group raised the issue of protecting the provider from consumer reports that may contain false claims, unfounded complaints of adverse events or wrongly identify a provider. For the purposes of this project, the consumer reports are not considered formal complaints against providers. CRSPS staff will refer consumers with grievances to other reporting options or advocates. Participating facilities as part of the pilot will sign a no-blame policy for providers (Please refer to Attachment K for an example of the process that we would use to assist participating providers in the pilot to establish such a policy as well as an illustrative policy example). In addition, the CRSPS website would contain a list of health care facilities that have no-blame policies for providers. Finally, the results of the PSO matching exercise and any subsequent analyses will be reported only in the aggregate so providers would not be publicly identified by the system.
Revision#5: Several consumer advocates raised issues around feedback of the data to the public. They proposed that individual hospitals be named, that the system compare hospitals, and that data be stratified and analyzed by setting (e.g., institutionalized vs. inpatient vs. outpatient). Given that the CRSPS system will be working within the constraints and protections of a PSO and the intent of the CRSPS to serve as a safety improvement approach, public documents based on the CRSPS data will only be available in aggregated form. Individual consumer reports (which may name individual hospitals) would be covered under the research protocol and so are not discoverable. CRSPS agrees that it would be valuable to examine the data across setting. We have included in the analysis plan to examine the data by setting, including institutionalized, inpatient and outpatient as long as this can be done without unmasking patients or providers.
Revision#6: Nurses, PSO service group, consumer advocates and a non-profit organization all raised the issue of coordination with existing consumer reporting systems. We agreed that CRSPS needed a strategy for making information about other reporting systems available. CRSPS is intended to complement -- not compete with -- other reporting systems. So, upon exiting the intake form, the consumer will land on a reference page that contains several links to direct the consumer to other resources and information. One of the links will be a list of patient advocacy groups from the pilot community in which the consumer resides. Another link will include a list of organizations in the pilot community that enable consumers to report grievances or complaints. Another link will list other consumer reporting systems available to consumers for reporting other types of errors, such as MedWatch, which is designed for reporting errors related to drugs, biologics, medical devices, and special nutritional products and cosmetics. There is also a link provided to Frequently Asked Questions for consumers if they have any concerns (see Attachment C, CRSPS FAQs information sheet). The FAQs answer a series of questions that the consumer may have as well as provides directions to links or 800-numbers to answer their questions or concerns. It defines a grievance and a service complaint and several types of medication errors to help direct consumers to the correct system to report events.
Twelve types of design issues were raised by public comments for which we offer our rationale for not making changes to the intake form.
Comment#1: Consumer advocates asked that providers publish information about the types of adverse events reported as having occurred at the participating facilities. Consumer advocates specifically requested that providers respond to incidents by offering written protocols summarizing oversight of the procedures that may be related to reported adverse events and to maintain copies of the consumer report and its resolutions in patient files (along with a patient signature indicating the patient had received a copy).
These proposed ideas are not implementable because they would violate regulations concerning patient confidentiality and provider information, which is considered patient safety work product. The CRSPS database will not contain patient safety work product (PSWP). The participating facilities and consumers will have responsibility for consumer reported information that was shared based on consumer consent, but as covered entities under HIPAA facilities will have to manage any feedback or dialogue directly with patients and not through the CRSPS system. In choosing a pilot community, the CRSPS team will seek to engage organizations that will have systems for receiving consumer reports and will be responsive when patients or caregivers communicate concerns to CRSPS or to the facility directly.
Comment#2: Consumer advocates and nurses highlighted the benefit of following up with the consumers who report a patient safety event. One consumer advocate suggested that every consumer who makes a report should receive contact information for the patient ombudsman, liaison, or advocacy office maintained by the healthcare facility identified by the consumer while making a report. This idea is incorporated in the provision of a list of patient safety organizations and advocates in the pilot community. This list will be made available on the resource page or provided directly to anyone who calls the CRSPS 800-number, as is stated in the FAQs.
Consumer advocates also suggested that consumers who consent to forward a report to a provider should be informed regarding the providers' responses and any positive outcomes of their disclosure. In the context of the pilot project, as noted above current regulation does not permit sharing of patient safety work product (PSWP) outside of the PSO. To be responsive to the spirit of this suggestion, CRSPS will refer patients to the patient advocate or ombudsman’s office of the provider organization so that the patient can make such an inquiry directly to the provider.
Comment#3: Consumer advocates proposed that the pilot project pursue a formal sampling approach, rather than relying on consumers to volunteer reports. One commenter suggested that the reporting system should collect information from a probability sample of respondents (either a census or a calculated percentage of patients selected at random) intending that some will receive access to the CRSPS website intake form -- and then make a comparison between responders and non-responders. Another suggested that the pilot should include a mode test comparing a web administration to an on-site kiosk. These ideas could inform the evaluation planned for the option years of the contract. That evaluation will include discussion with the selected pilot sites to review the feasibility of various deployment options that could be part of the evaluation. If the selected community has the capabilities, we may be able to test varying recruitment and sampling strategies and modes of administration. However, the primary goal of the currently funded work is to produce an intake form that can be feasibly administered under a variety of recruitment and administration approaches.
Comment#4: Members of Congress, a hospital organization, a PSO service group and consumer advocates asked how the data from the reporting system would be used by the Federal Government, state regulators and by provider organizations to do quality improvement.
The current research pilot does not presuppose any one specific use of the data. Instead, it is designed to develop a tool that could be applied in various ways that could meet the needs of various users including provider organizations as well as federal and state regulators. The answer to the question of use is somewhat dependent on the findings of the pilot. If valid and reliable data on safety can be obtained using the pilot intake form, then plausible uses might include analysis of shared data by provider organizations to increase detection of adverse events and errors so that delivery systems can be redesigned for safety, the use of reports to initiate and augment root cause analyses, and the development and refinement of organizational safety data collection strategies.
Uses of the data within the pilot project will be negotiated with participating institutions and in line with the research, confidentiality, privacy, and data sharing protocols noted above. We anticipate the participant feedback on the usefulness of shared data and aggregate reports will be a key part of the evaluation that occurs during subsequent years. The results of the pilot project will help to inform the design of policies on future uses of such data by any interested health care stakeholder. Such protocols do not now exist and would need to be developed before the data could be shared outside of a research context. We believe this guidance will be an important product of the research.
Comment#5: A physician organization, members of Congress, hospital association, and nurses raised the issue of the system collecting inaccurate information and go on to say that patients and caregivers may be a poor source of information on errors since they lack the clinical judgment to distinguish errors from normal health care delivery and adverse events from anticipated complications. While patients and caregivers are not able to observe directly all of the adverse events and errors that occur, prior scientific work suggests that consumers are able to report reliably on some types of events that may represent errors (e.g., delayed or incorrectly administered tests or prescriptions) or adverse events due to treatment (e.g., allergic reactions, nerve damage, wrong site surgeries). It is not anticipated that consumers would be the only source of information about safety events, but one important source that may identify errors and adverse events that currently go undetected and a source of additional information about the impact of errors and adverse events.
Recognizing the known limitations of general reporting, the intake protocol will include review of all reports by members of an experienced team with clinical and patient safety expertise. This verification step will enable clinicians to assign a probability that a report may represent an error or adverse event, to classify the type of event and to identify any unresolved questions that the consumer could address in order to clarify the classification. We anticipate that a percentage of reports will be erroneous or incomplete at the time of this review. These reports will be analyzed to understand misreporting of patient safety information, but not included in calculated rates of reported error or adverse event.
Comment#6: A consumer advocate and a university law professor raised concern that the reporting database would not capture the full range of safety events and specifically would miss less serious events (such as near misses) and patient or caregiver distress, which may be an important consequence of medical error. We believe the intake form has been designed to elicit these types of concerns. Specifically, the intake form asks the consumer to report on any suspected error whether or not it caused harm, injury, or distress. It also uses the term “negative effect” which we have found in prior testing to capture a broad array of effects including emotional distress. In addition, at the review stage, clinician-reviewers will flag points that may require clarification by the consumer. On a follow up contact (with consumer consent) and this will offer the consumer another opportunity to report on both the circumstances surrounding a potential error and the distress that may have been associated with the event.
Comment#7: A hospital organization, a consumer advocate and a veteran raised issues around national implementation. However, the research pilot is not a national implementation plan. The design and implementation of a nationwide reporting system and whether veterans or the VA would be included in such a system is well outside the scope of this research pilot and a subject for future study if the intake form is determined to be feasible based on the pilot phase.
Comment#8: A physician organization, university law professor, hospital engagement network organization, PSO service organization and consumer advocate raised issues around identifying or classifying events and what the process was for defining and assessing the content of the reports. Specifically, how will the intake protocol differentiate severe and minor events, and whether it will score the preventability of events.
Based on prior research and our focus groups and cognitive interviews, consumers find it difficult to rate the severity and preventability of errors and negative effects. They can describe the personal impact of an event, but scoring implies a standard that is not widely shared and somewhat challenging to implement even in a clinical review process involving experts. Most consumers have a limited set of prior experiences to create a standard against which they can compare the severity and preventability of an event. In prior research, a scoring protocol was developed and employed by experienced clinical experts. The validity and reliability of this scoring system has been evaluated and refined.
Comment#9: The nurses association raised the issue that the CRSPS system does not address the potential for high costs associated with matching the reported error with the health care record. This matching could be particularly challenging and costly in paper-based or disparate electronic systems. We agree that at present the cost of such a matching protocol to facilities is unknown. The pilot study is designed deliberately to assess the time required to successfully match consumer reported events to documented events within care organizations (and to determine the yield of this activity) across various settings. These results would inform estimates of the cost and utility of such a matching protocol for future implementers.
Comment#10: A physician organization, members of Congress, a non-profit organization, hospital association, and consumer advocate raised questions about the public disclosure of data and sharing of this information. As noted above, the CRSPS pilot project will operate under research authorization and in relationship to a PSO umbrella. Safeguards against the inadvertent release of CRSPS data will assure that only aggregated or summary results will be shared with the public and this will be done in such a way that the risks of identification or disclosure are minimized.
Comment#11: A physician organization, hospital association and members of Congress raised the issue that such a reporting system may increase the risk of malpractice liability. Further, they asked whether the information collected from consumers could be used in medical malpractice lawsuits. Consumer reports to the CRSPS would have the status of protected research data, which has been protected from legal discovery to this point. Derivative PSO analyses and facility analyses conducted under the PSQIA are considered patient safety work product (PSWP) and are not discoverable under current interpretations of the PSQIA. Reporting to CRSPS does not deny the consumers the right to provide their own information to lawyers or any other parties. However, in a previous Massachusetts-based study of consumer reporting of safety events involving nearly one thousand patients reporting on hospital safety events, no lawsuits are known to have been instigated after consumers reported the information to researchers.
Comment#12: A non-profit organization, university law professor and consumer advocate raised issues about the deployment of CRSPS as a web technology. They requested that the intake form use newer technologies such as mobile devices (smartphones, PDAs), kiosks, ipads, and other Internet or email-based approaches. They also raised issues about the security of the selected browser (Internet Explorer) and the limited report printing option (Adobe File Reader). We recognize the browser and printing options are constraining. However, the budget constraints of the pilot limit the pilot to web and voice telephone reporting as the primary reporting modes. Kiosks or other modes may be introduced depending on the capabilities and interest of the participating organizations in the pilot community. Depending on results of the pilot additional modes of data collection should be explored in future projects.
Based on the public comments, we have proposed the following 19 revisions to the intake form. These revisions reflect changes in wording, the deletion of 2 items, and the revising of two items to be open-ended questions (instead of with response options to choose from), but no change in survey burden or intent. The time (30 seconds each) to answer the two items that were deleted will be used by the consumer to answer the two open-ended items that were previously response options. In general, we are mindful of the tradeoff between obtaining more detailed information from the consumer about events and the time and effort required of the consumer by the addition of any more items to the intake form. Our aim is to strike a balance between specificity and parsimony.
Revision#1: A physician organization proposed including whether and to what extent conversations between the physician and patient occurred about any changes in medications or treatment. We judged this topic as too specific to include on the intake form and more suitable for investigation through a root cause analysis conducted as appropriate. The more general topic of communication with clinicians is covered by question sequence 5.1.
Revisions#2: A PSO service group points out that the intake form refers to medical mistake, safety concern and negative effects interchangeably in the introduction section of the form. We agreed and have revised the Section 1 introductory language to refer only to medical mistakes and negative effects, rather than "injuries related to health care," which is not used elsewhere in the survey. In Section 6, we have changed all uses of the term "safety concern" to "mistake or negative effect."
Revision#3: A PSO service group pointed out that the question concerning the setting where a negative effect is first experienced includes response options that are not mutually exclusive. We agree and have revised the instructions for 3.2 and 4.4 to read "Please choose the one answer that fits best."
Revision#4: A physician organization proposed revisions to question 3.1.2.2 “Did the mistake with a test, procedure, or surgery involve any of the following? Please check all that apply” on the grounds that some of the responses to this question may be too subjective, such as “the test, procedure, or surgery was delayed unnecessarily” and “it took too long for the patient to get the results.” The same wait time may be interpreted by one patient as appropriate, while another may view it as inappropriate. In accord with general patient experience survey design principles, we revised the question 3.1.2.2 to remove “judgment” terms. We removed the word “unnecessarily” from answer choice ‘E’ and reworded answer choice ‘F’ to: “The test results were lost and the patient did not receive them.”
Revision#5: A physician organization asked us to change the phrasing of questions that ask patients about whether they received the wrong diagnosis or advice because patients lack the clinical training to judge whether diagnoses or changes in treatment represent errors or appropriate care. We agree with the commenter and have reworded the question 3.1.3.1 as: “In your opinion, what was the mistake with the diagnosis or medical advice?” Rather than providing answer choices, we are now providing a free text box. This modification will result in an increased burden of 30 seconds/survey.
Revision#6: A physician organization proposed revising the question, “Did the mistake with the diagnosis or medical advice involve any of the following? PLEASE CHECK ALL THAT APPLY”, reasoning that the answers would be subjective and variably interpreted by patients. This group recommended removal of the term “bad medical advice” since it is too vague to result in useful information. We agree with the commenter and reworded the question 3.1.3.1 as: “In your opinion, what was the mistake with the diagnosis or medical advice?” Rather than providing answer choices, the form now provides a free text box. Based on the text received in the pilot, we can consider redesigning response options. This modification will result in an increased burden of 30 seconds/survey.
Revision#7: A physician organization proposed revising the response scale to the question 3.3 that reads “Would you like to tell us the name and address of the health care doctor, nurse, or other health care provider (or the health care facility) involved in the mistake?” We agree that the original phrasing excludes the cases in which the respondent doesn’t know the name/address, but would give this information if they knew it. The revised question 3.3 (and 3.3.3) now has a response scale that includes: Yes; Yes, but I do not know the name and address of the provider; No, I do not know the name and address of the provider; and No, I do not want to tell you the name and address of the provider. This same issue was revised in 4.5 and 4.5.3.
Revision#8: A consumer advocate group did not like the phrasing of the question, “How did the patient find out that the mistake happened? Please choose the one answer that fits best” and its response choices: the patient noticed it; A friend or family member noticed it and told the patient; A doctor, nurse, or other health care provider told the patient about it; We simplified the question 3.5 (and 4.7) to read, “Did a doctor, nurse, or other health care provider tell you the mistake happened? This question focuses on the information likely to be of greatest interest to the participating organizations and clinicians: whether a doctor, nurse, or other health care provider told the consumer about the mistake.
Revision#9: A consumer advocate group and the AOA did not like the phrasing of the question and its follow up question that asks “Did a doctor, nurse or other health care provider make any special effort to help the patient handle the mistake?” with the options to answer Yes, No, Don’t know. If yes, the reporter is asked” Did it help?”. The commenter proposed instead an open-ended question allowing a consumer to explain how a provider helped the patient handle the mistake. Because of limited resources for the pilot study, an open-ended text response was considered infeasible so these two questions were dropped from the intake form. This modification will result in a decreased burden of 60 seconds/survey.
Revision#10: A consumer advocate proposed including questions that ask about encounters with healthcare professionals that resulted in "distress", not just those encounters that lead to "harm". After a careful review of the questions and given the inclusion of mistakes that do not lead to harm, we have revised the definition of negative effects to: “Negative effects can be physical or emotional….” In question 4.2, we ask whether the negative effect was emotional, physical, or both in order to clarify the intention of the questions.
Revision#11: A consumer advocate proposes that we more clearly state the option to share identifiable information, so that a consumer is not surprised after seeing all of the front-end language about confidentiality. We agree with the commenter and added language about the option of sharing identifiable information to the beginning of the form in the Introduction Section.
Revision#12: A PSO group indicates that when asking about the month and the year of when a mistake happened that we should not indicate that “Your best guess is fine.” Prior experience suggests that survey respondents struggle to identify exact dates and that asking them to do so increases cognitive burden. To conform with standard survey approaches, we have modified the wording to question 4.6.
Revisions#13: A PSO group indicated that the FAQs document is not patient friendly. We have revised the wording to be more patient friendly. We revised the FAQs and also item 4.9. The previous question was “Did it help?” with a yes, no, don’t know scale (in reference to special efforts by a doctor, nurse or other health care provider to help a patient handle a negative effect). The revised question: “ How helpful were they?” with a 5-point scale from Extremely helpful to Not at all helpful.
Revision#14: A PSO group indicated that the question “Could anything have been done differently to prevent this mistake or negative effect from happening?” was misplaced in the form as the last question in section 4. We agree and have moved this question to Section 5 (as question 5.1), which asks about contributing factors.
Revision#15: A physician, a physician organization, a PSO group and members of Congress were concerned about the contributing factors question (question 5.2 and 5.3 series), and specifically the negative phrasing of the response options and the vague nature of the communication response options included. We agreed that these items, drawn from prior staff reporting forms, needed to be reframed for consumers. The sequence has been completely revised based on principles developed by the Consumer Assessment of Healthcare Providers Survey (CAHPS) that include well-tested questions about the consumer experience of health care quality (and associated safety concepts). This section has been completely revised so as to address these concerns. We have revised the answer choices to the contributing factors questions (5.3 series) such that they are more specific and employ more positive phrasing.
Revision#16: A physician organization proposed qualifying the question, “Is there anything else that caused the mistake or negative effect to happen?” by adding “may have” before “caused”. We agree with the commenter and revised question 5.2 to, ”may have caused”. We also replaced the original question with “Why do you think this mistake or negative effect happened” to reflect the fact that the reporter has one helpful perspective – but not the sole perspective – on the mistake or negative effect.
Revision#17: A PSO group pointed out a problem in the numbering of section 5. We agreed and corrected the numbering in 5.4 and 5.4.1.
Revision#18: The PSO service group commented on whether the questions referring to sex, race, and Hispanic origin used current federal standards. We have revised these items (6.2, 6.4 and 6.5) according to the implementation guidelines Section 4302 of the Affordable Care Act for survey items and their inclusion in patient experience surveys.
Revision#19: A PSO group asked why the form asks the location where the consumer is filling out the intake form and recommended to include cell phone or kiosk as choices. Upon discussing this issue, we realized that the location was less important to know than “how” the consumer learned about the option to report. As a result, we are eliminating “Where are you filling out this report?” We retained the existing question “How did you learn about the Consumer Reporting System for Patient Safety?” (Question 6.8).
There were 17 public comments for which we offer our rationale for not making changes to the intake form.
Comment#1: A consumer advocate raised the issue that in the flyers and documentation that people would not know what ECRI stands for. The sample flyer (Attachment F) included ECRI, but this will be replaced with the organization name once the pilot community is established.
Comment#2: A hospital engagement network proposed adding a “score” related to the preventability and seriousness of the incident. Because of the cognitive burden and expected limitations of the validity and reliability of consumer determinations of severity and preventability, we did not implement this change. Scoring of severity and preventability will be conducted by clinical experts using a process that enables evaluation of validity and reliability.
Comment#3: A PSO service group questioned how the intake form would apply to children. We confirm that a consumer who wishes to report must be an adult (i.e. 18 years or older). An adult consumer can report on a safety concern involving a minor. The consumer would choose answer choice B, “ A Child” for question 1.1 that asks “Who is the patient with a safety concern?”.
Comment#4: A PSO service group indicated that it would be difficult to cross-reference a particular report to any other standard incident reporting with a hospital. We recognize this challenge. We plan to use date and time to cross-reference or match reports, though we appreciate the limitations of this approach. It is intended that the pilot will enable the implementation partner to work with the research team to develop an algorithm for efficient matching algorithm that will identify related reported incidents in the facilities' incident database.
Comment#5: A PSO service group suggested that the form does not allow for a medical mistake without associated harm. In fact, the intake form is structured to recognize the possibility of reporting a medical mistake without associated harm (so-called “near miss”). If a respondent indicates a "yes" to 2.2 ("Did a doctor, nurse, or other health care provider make a medical mistake or error in the patient's care?"), and "no" to 3.7 ("Did the patient experience any negative effects as a result of the mistake or error?"), then the paired responses are interpreted as a medical mistake without associated harm (a near miss). There are two types of near misses – those for which the mistake was intercepted and did not reach the patient and the type in which the mistake reached the patient, but no harm occurred. The intake form is not a substitute for a root cause analysis, which would be necessary to determine such distinctions.
Comment #6: A PSO Service group proposed that we require the reporter to put some text in each blank to make sure that we know they have read the questions. We do not consider intake form items mandatory given that this is a voluntary reporting form. In the pilot, we will assess the degree to which respondents skip items on the intake form or abandon completion of the intake form. This refers to the Programmer General Instructions on page 1.
Comment#7: A consumer advocate did not like the wording "heath care safety concern" or "negative effect" arguing these terms are not widely used and have little meaning to providers. Focus groups and prior published research demonstrated that consumers understand these terms. “Safety concern” was a newly developed term within this project that tested well with patients in a focus group and in cognitive testing. We revised these terms for consistency throughout the introductory text.
Comment#8: A PSO service group pointed out that we indicate in the introduction that the report should take 10-15 minutes to complete and that that contradicts the average of 25 minutes that is report in the Supporting Statement A and B. These are not contradictory statements. To clarify, the 25-minute average includes the estimated 10 minutes of follow-up by phone with a consumer who agrees to be contacted about any clarifications that their report requires. This is mentioned in the introductory text.
Comment#9: A PSO service group indicated that the AHRQ common formats have 6 types of “wrongs” for medication and that in our intake form (question 3.1) we only ask for two types plus a “something else” write in option. As we did elsewhere, we trimmed some response items for relevance and to maintain a shorter response length. Overall we aimed to strike a balance between specificity and parsimony.
Comment#10: A consumer advocate asked that we examine the data stratified by institutionalization (nursing home, hospice) vs. inpatient vs. outpatient. We believe that the institutionalized patients are included in the response option "somewhere else" (items 3.2 and 4.4). We are aiming to strike a balance between specificity and parsimony.
Comment#11: A PSO service group did not like the phrase “Your best guess is fine” in question 3.5. When asking about the month and year of when a mistake happened. They wanted to have information about the day of the week or the time of the day. We did not revise the question that asks about the month and year to find out when the mistake happened because month and year should suffice to match the consumer report with facility reported events. Furthermore, asking the consumer about specific days and times is a far more difficult cognitive task.
Comment#12: AOA proposed revising 3.6 that reads, “Did the mistake affect the patient financially?” to include an open-ended response for patients to explain the financial impact. In response we agree that the proposed question is interesting, but outside the scope of the pilot study and therefore not included in the intake form.
Comment#13: A PSO group indicated that we “Totally omitted any data on actual harm or injury and did not ask what the injury was.” We did not make any revisions because the question 4.3 does ask about the injury or negative effect and offers a series of options in addition to write-in options.
Comment#14: A PSO group points out that the form does not ask about medical devices. We did not revise the intake form question 4.1 to include medical devices for several reasons. First, we are aiming in the form to strike a balance between specificity and parsimony. In addition, consumers have a vague understanding of the range of “medical devices” raising concerns about the validity of answers, Consumers experiencing negative effects or injuries as a result of a medical device would use answer choice ‘F.’ (something else) with an open-ended response to describe the medical device.
Comment#15: A PSO group points out that question 4.2 asks about “What kind of negative effect did the patient experience?”. The response choices are: physical, emotional, or both. They suggested that emotional effect was “totally omitted”. However, the intake form does ask whether the negative effect was emotional. We decided not to ask detailed questions about emotional negative effects. In this same comment the PSO group raises the issue that the intake form does not offer the possibility to report a medical mistake and a negative effect. However, in the intake form if a respondent responds "yes" to question 2.2 ("Did a doctor, nurse, or other health care provider make a medical mistake or error in the patient's care?"), and "yes" to question 3.7 ("Did a negative effect take place as a result of the patient's care?"), then these responses are consistent with reporting of both a medical mistake and a negative effect.
Comment#16: A PSO group proposed adding “patient advocate” as one of the response options for the question that reads “Who did the patient tell about the mistake or negative effect”? We aim in this intake form to strike a balance between specificity and parsimony. We believe that a patient advocate would be included in the open-ended response option. We will monitor the responses to 5.4other, the open-ended response option, to see whether additional answer choices are needed.
Comment#17: A PSO group raised the issue of including the options of having heard about the system from public service radio or TV or an educational program (question 6.8). We have not been tasked in this pilot project to include the type of marketing described by the commenter.
Together, these revisions reflect both a decrease of 60 seconds, balanced by the revision to open-ended responses of two 30-second statements. Therefore, these modifications will not change the average burden/respondent.
Based on the public comments, we have proposed the following revisions to the supporting documents to the intake form. These revisions do not change the survey burden or intent.
Revision to Supporting Documents#1: A consumer advocate proposed revising the telephone follow up script when someone other than the respondent is reached to assure that there is no mention of patient safety (so as to maintain the confidentiality of the person who reported the event). We agree and have revised the follow-up script to not reference patient safety until the consumer who made the report is contacted directly.
Revision to Supporting Documents#2: A PSO Service group commented that in the introductory pages of the website, there was no mention of nurses in the group of people who can report. Upon reviewing the introductory pages (Attachment A) in response to this comment, we noticed several areas warranting revision pertaining to referencing nurses and details about the design of the intake form and the types of reports it will enable. We also reviewed the Frequently Asked Questions (FAQs) (Attachment C) and where appropriate added references to nurses.
The public comment received by AHRQ included 64 personal stories. These stories were categorized into 14 themes, as shown in Table 1. Table 1 shows the number of times each of the 14 themes is mentioned across the 64 stories and provides the percentage of stories that contain a given theme. For example, 78 percent of the personal stories, i.e. 50 of the 64 stories, mentioned a complaint about a bad doctor, nurse or system.
Table 1. Themes of Personal Stories, by Frequency of Mention
Theme |
Number of times Theme mentioned |
Percent of stories containing Theme |
Complaint -i.e. bad doctor/ nurse / system |
50 |
78% |
Support for patient safety reporting |
20 |
31% |
Inaction by other, general |
14 |
22% |
CRSPS Project "doing something" about medical errors |
14 |
22% |
Doctors / admins covering each other |
9 |
14% |
Inaction by health care system |
8 |
13% |
Inaction by government |
7 |
11% |
Suggestions for improving health care |
7 |
11% |
Results / records hidden |
6 |
9% |
Retaliation by doctors / system |
5 |
8% |
Skepticism of patient safety reporting |
4 |
6% |
Other |
4 |
6% |
General comment on project |
3 |
5% |
Complaint - medical insurance |
2 |
3% |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |