EHC Responses to OMB comments

AHRQ’s Responses to OMB’s Follow-up Comments

Re: Evaluation of AHRQ’s Effective Health Care Program

Feb 18, 2010

The Agency for Healthcare Research and Quality (AHRQ) appreciates the comments received from the Office of Management and Budget (OMB) on the proposed information collection under the project: “Evaluation of AHRQ’s Effective Health Care Program” (dated 12/23/2009). Below are our general responses to OMB’s comments, suggestions and requests. Point-to-point responses follow from the next page.

OMB suggested expanding the scope of the data collection efforts (i.e., interviews, on-line surveys) in several areas: expanding the representation of certain stakeholder groups in the interview, expanding the survey to the universe instead of a sample, conducting focus groups instead of individual interviews, and adding additional interview questions. AHRQ has taken these suggestions under advisement and has made the following changes:

• ARHQ will conduct eight additional interviews to achieve better representation of certain stakeholder groups identified by OMB (disease-specific advocacy organizations and consumer groups, provider groups, device and pharmaceutical companies, and health plans). In selecting interviewees, AHRQ will identify interviewees from an updated EHC program contacts database, and will emphasize the groups mentioned by OMB in the allocation of the total number of interviews so that the recommended groups are better represented.

• AHRQ will increase the number of survey respondents to the universe of the updated contacts database, which we expect to be around 400. As a result, we have revised Table 1: Universe for Online Survey, by Strata, in the supporting statement and have eliminated Table 2: Expected Respondents for Online Survey, by Strata, since the universe will now be surveyed.

• Additional interview questions have been added; please see the revised supporting statements.

Furthermore, OMB is interested in AHRQ providing greater opportunities for public input. AHRQ will make an effort to encourage public review and comment as appropriate. Moreover, AHRQ only intends to use this study for internal program evaluation purpose, and does not intend to use it in lieu of formal PART evaluation.

With regard to several more specific comments, AHRQ confirms that document review and international organization comparison will be conducted prior to the Appreciative Inquiry (AI) workshop. In fact, findings from all data collection efforts will be presented at the beginning of the workshop, and discussed with workshop participants. AHRQ will also revise the honorarium size for the AI workshop attendee per OMB request, although will re-assess the need to increase the honorarium size if the response rate becomes concerning. However, AHRQ does plan to compensate the attendees from non-DC metro areas for costs associated with travel, lodging, and per diem.

In addition, we re-evaluated the time needed to complete the on-line survey after adding additional questions, especially those open-ended, to the survey questionnaire based on OMB’s suggestions. It occurs that the revised survey will take about 5 minutes longer to complete so approximately 20 minutes total. Therefore, we made the corresponding changes to the public burden estimates in the supporting statements. If the response rate becomes concerning during the data collection stage, AHRQ will assess whether providing some incentives will boost the response rate.

Finally, we have updated some letters and emails about the evaluation to provide more background about the EHC program and the study.  These revised letters are included as attachments to the supporting statements.

August 18, 2009

The Agency for Healthcare Research and Quality (AHRQ) appreciates the comments received on the proposed information collection under the project: “Evaluation of AHRQ’s Effective Health Care Program (74 Fed. Reg. 34019, July 14, 2009)”. Thoughtful comments were submitted by five organizations: American Nurses Association (ANA), Mental Health American (MHA), National Health Council (NHC), Pfizer, and Pharmaceutical Research Manufacturers of America (PhRMA). All five organizations are supportive of AHRQ’s proposal to evaluate the Effective Health Care (EHC) Program’s governance structure, methods for engaging stakeholders, and approaches to setting priorities for the research conducted by the EHC program. Specific comments on the proposed evaluation and AHRQ’s responses to these comments are below:

1) Assess how research findings have informed decisions at the point of care.

The goal of the proposed project is to promote and structure the involvement of health care providers, communities, and consumers (both program users and contributors) in the EHC program’s governance. To accomplish this goal, the IMPAQ/Abt team will perform an evaluation of the EHC program governance by conducting key informant interviews, by administering an online survey, by preparing case studies, and by conducting an Appreciative Inquiry workshop.

As part of these data collection activities, questions will be asked about the impact of the Effective Health Care Program’s research. These questions will specifically inquire about the research’s impact on clinical practice, patients’ behavior, and health outcomes.

OMB follow-up: This comment underscores OMB’s recommendation to include more provider groups and consumer/patient organizations in the interview sample to assess how their knowledge of the governance structure—and their feelings about it—have affected the extent to which providers feel comfortable using EHC information to inform their clinical decision making and, ultimately, how that has affected patient care (both from the perspective of the provider and the patient).

Accepted. The study will include more provider groups and consumer/patient organizations in the interview sample.

2) Ensure inclusion of patient perspectives in evaluation.

The proposed sample for the key informant interviews and the survey includes EHC program general users and stakeholders. Patients are considered both EHC program general users and stakeholders, and their perspectives will be captured in these evaluation activities by inviting representatives from consumer and/or disease advocacy groups who represent patients’ perspectives.

Semi-structured key informant interviews will be used to understand the EHC program’s governance components and structure, from the vantage point of individuals governing the program, governed by the program, contributing to the program in various capacities, or affected by the program’s activities.

A structured, web-based online survey of EHC program Research Centers Staff and EHC program stakeholders will be used to gather information on the EHC program. The survey will provide a robust view of the EHC governance system by providing feedback from a broad group of individuals, including a subcategory of stakeholder “patients/consumers/advocacy organizations.” Specifically, the survey will collect data about these individuals’ engagement and involvement with the EHC program; perceptions of the program’s governance; experiences with the development, production, dissemination, and use of EHC products; and their beliefs regarding the quality and nature of the collaborative work, including public-private partnerships, being done within centers, across centers, and between centers and stakeholders.

OMB follow-up: This comment underscores OMB’s recommendation to include more provider groups and consumer/patient organizations in the interview sample to assess how their knowledge of the governance structure—and their feelings about it—have affected the extent to which providers feel comfortable using EHC information to inform their clinical decision making (both from the perspective of the provider and the patient).

Accepted. The study will include more provider groups and consumer/patient organizations in the interview sample.

3) Recognize that nurses, including advanced practice registered nurses, are important stakeholders in the EHC program.

The proposed sample for the key informant interviews and the survey includes provider groups/providers. Nurses, including advanced practice registered nurses, are considered provider groups/providers and their perspectives will be captured in theses evaluation activities.

Semi-structured key informant interviews will be used to understand the EHC program’s governance components and structure, from the vantage point of individuals governing the program, governed by the program, contributing to the program in various capacities, or impacted by the program’s activities.

A structured, web-based online survey of EHC program Research Centers Staff and EHC program stakeholders will be used to gather information on the EHC program. The survey will provide a robust view of the EHC governance system by providing feedback from a broad group of individuals. Specifically, the survey will collect data about these individuals’ engagement and involvement with the EHC program; perceptions of the program’s governance; experiences with the development, production, dissemination, and use of EHC products; and their beliefs regarding the quality and nature of the collaborative work, including public-private partnerships, being done within centers, across centers, and between centers and stakeholders.

OMB follow-up: This comment underscores OMB’s recommendation to provide more detail on how AHRQ plans to select which entities will be included in the interview sample. At the moment, AHRQ plans to interview just one provider group. Given interest by nurses, how will AHRQ decide between, say, the AMA and the ANA?

We will increase the number of interviewees by 8, increasing representation by the four specific subgroups recommended by OMB.

In terms of how to select interviewees, first, we will use the AHRQ EHC program’s contacts database to identify the potential interviewees for each user/stakeholder subgroup. Next, we will examine the number and diversity of candidate organizations represented for each of the subgroups (i.e., health plans, provider groups). Additionally, we will examine what role(s) these organizations have had with the EHC program (e.g., reviewed report, nominated topic, etc.) to the extent that these data are available in the contacts database. Then we will purposively sample for diversity of organization (e.g., physician as well as nursing associations) type within subgroups and in terms of what role(s) they’ve had with the EHC program. We will also select the requisite number for each subgroup plus at least 2 alternatives for recruiting purposes given that some individuals will decline participation or not respond to invitations to participate in the interviews.

4) Evaluate the extent to which the research projects initiated by AHRQ reflect the research mandate defined under Section 1013 of the Medicare Modernization Act.

As part of the evaluation, the contractors will be reviewing relevant program documents and data. Two types of documents will be reviewed and analyzed – those that will provide background and governance information, and those that will yield evaluative data. The documents will be classified using the following categories: output or document type; priority population addressed; priority disease/condition addresses; type of stakeholder addressed; and Tunis and Stryer Levels of Impact classification scheme.

The documentation information will be summarized based on these key components and will be used to inform the Agency’s and project team’s current understanding of the EHC program including information on whether the research projects initiated by AHRQ reflect the research mandate as defined under Section 1013. This information will also guide the development of a conceptual framework for the governance of the EHC program.

5) Include a comparison of the types of research topics proposed by stakeholders to the types of research initiated by AHRQ.

The purpose of the quantitative data analysis part of the evaluation is to provide evaluative data related to the EHC program’s activities, outputs, and impacts. This will involve analyzing the inventoried documents and downloaded data from AHRQ’s internal databases and the EHC program’s lists. These will provide quantitative summary descriptive statistics on process measures and intermediate outcome measures. This information may include type of research topics proposed by stakeholders, number of comparative effectiveness research reviews produced, percentage of awards that focus on each of the EHC’s priority conditions, and number and percentages of stakeholders in the program’s contact database and reasons for being contacted by the program.

6) Gauge the extent to which stakeholders understand how their input is considered by the program.

The project will gauge the extent to which stakeholders understand how their input is considered by the EHC program.

The key informant interviews will include questions about engaging stakeholders and may include the following:

• Who do you see as the EHC program’s stakeholders and users?

• What are the benefits of the EHC program and its products for your organization or stakeholder group?

• What role have you played in engaging stakeholders or stakeholder groups?

• What is the role of the EHC program Stakeholder Group in engaging stakeholders for the program, either with their own organizations and/or with other general stakeholders?

• Which, if any, stakeholder groups have not been sufficiently engaged in the EHC program?

• How does the program serve consumers with low levels of health literacy and numeracy, with limited English proficiency, or racial/ethnic diversity?

• How is the EHC program held accountable for engaging stakeholders, and by whom?

• How, if at all, does the EHC program Stakeholder Group hold the EHC program accountable for engaging stakeholders?

• From your perspective, is the EHC program’s approach for engaging stakeholders transparent?

As part of the survey, stakeholder will be asked to rate the extent to which they agree with the following statements:

• As a stakeholder, I feel that the EHC program’s mechanism/process of engaging me or my organization in setting up the research priorities is sufficient.

• As a stakeholder, I feel that I have the opportunity to be involved in the development of the EHC products.

• As a stakeholder, I feel that I have the opportunity to be involved in the translation of evidenced-based tools and materials into EHC products.

• As a stakeholder, I feel that I have the opportunity to be involved in the dissemination of the EHC products.

OMB follow-up: While the above questions address involvement in the EHC program and how respondents feel about that involvement, none of them seem to really address the issue how the EHC program actually uses the information that is provided to them. To be responsive to the commenter, we would request that AHRQ consider adding an additional question like, “Do you understand how your input is considered” to the key informant interview schedule.

Request accepted. Please see the revised supporting statements.

7) Evaluation should include the operation of the program’s EHC Stakeholder Group

The project will evaluate the operation of the program’s EHC Stakeholder Group.

The key informant interviews will include questions about the EHC Stakeholder Group and may include the following:

• What is the role of the EHC Stakeholder Group in engaging stakeholders for the program, either with their own organizations and/or with other general stakeholders?

• How, if at all, does the EHC Stakeholder Group hold the EHC program accountable for engaging stakeholders?

As part of the survey, respondents will be asked to rate the extent to which they agree with the following statements or question:

• The EHC Stakeholder Group is appropriately consulted by the EHC and your research center.

• The process used for selection of members for the EHC Stakeholder Group is transparent.

• The decisions and/or opinions of the EHC Stakeholder Group reflect accountability to the stakeholders in general.

• Do you feel that all stakeholders are well represented by the EHC Stakeholder Group?

8) Evaluate how the proposed activities in the AHRQ spend plan for comparative effectiveness research funds appropriated under the American Recovery and Reinvestment Act may/may not address any shortcomings identified in the evaluation.

A discussion of AHRQ’s spend plan for comparative effectiveness research funds appropriated under the American Recovery and Reinvestment Recovery Act will be included in the final synthesis of the evaluation activities and in the roadmap for future programmatic development.

9) Clarify how the evaluation will be making use of information from international programs.

Findings related to the governance of the EHC program in its current format, lessons learned from international comparisons, and effective governance principles identified in the literature will guide the development of alternative governance approaches for the roadmap for future programmatic development related to the dimensions of engaging stakeholders and priority setting.

10) Consider increasing the number of respondents to ensure the evaluation reflects a broad representative sample of users and stakeholders that interact with the program and use its reports.

AHRQ may consider increasing or decreasing the number of individuals surveyed, with an ultimate goal of collecting the useful data for developing the roadmaps in an efficient and effective manner.

OMB follow-up: This comment underscores OMB’s recommendation to survey at least the universe of stakeholders and users.

Request accepted. AHRQ will survey the universe of stakeholders and users in the most updated version of the EHC Program’s contacts database, which we expect to be around 400.

11) The evaluation should not be limited to an assessment of the program’s processes, but also its intended outcomes.

The goal of the proposed project is to promote and structure the involvement of health care providers, communities, and consumers (both program users and contributors) in the EHC program’s governance. To accomplish this goal, the IMPAQ/Abt team will perform an evaluation of the EHC program governance by conducting key informant interviews, by administering an online survey, by preparing case studies, and by conducting an Appreciative Inquiry workshop.

As part of these data collection activities, questions will be asked about the impact of the Effective Health Care Program’s research. These questions will specifically inquire about the research’s impact on other research, organization policy or guidelines, clinical practice, patients’ behavior, and health outcomes, including their level of significance.

12) The Agency and contractors should take necessary measures to provide respondents with some level of anonymity.

Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). Participants will be advised that participation is voluntary. They will be informed about the purposes for which the information is collected, and that, in accordance with this statute, individually identifiable information about them will not be used or disclosed for any other purpose.

Participants will not be identified in any publications of the study’s findings, including interim reports generated by the contractor, IMPAQ International, LLC, for AHRQ. All findings reported to AHRQ will be aggregate-level data to ensure the confidentiality of participants’ responses.

Identifying information, such as name, title, address, phone number, and email address, will be used only to send the participant a copy of the study’s findings and, in rare situations, to contact the participant for follow-up questions, if they are agreeable to this. Identifying information will be stored in locked file cabinets and secure electronic storage by the contractor, IMPAQ International, LLC. All identifying information will be destroyed upon completion of the study.

Key informant interviews and case studies will be audio recorded to improve data capture, only with the participants’ prior consent. Audio recordings will be used to create accurate transcripts of the interviews. Digital audio files will be saved in secure electronic storage and accessed only by password-protected computers. All audio files will be destroyed upon completion of the study.

13) The qualitative and quantitative research should also include former members of the EHC Stakeholder Group, current and former members of the National Advisory Council, and clinical and policy decision makers involved in making day-to-day decisions about patient care.

Key informant interviews and surveys will include a variety of EHC stakeholders including:

• EHC program Stakeholder Group members

• Employers & Health Related Business Groups

• Federal Partners

• Healthcare Industry

• Healthcare Providers

• Patients/ Consumers/ Advocacy Organizations

• Pharmacy & Therapeutics

• Policy Makers

• Professional Organizations

• Researchers

• Third Party Healthcare Payers

• The EHC program Stakeholder Group

OMB follow-up: The commenter seems to be highlighting the need to include former members of the EHC Stakeholder group. Does AHRQ plan to include them? If not, why not? Also, does AHRQ plan to include either current or former members of the National Advisory Council? If not, why not?

Several individuals in the EHC Program’s contacts database are also current or past members of the National Advisory Council (NAC). As such, we will add current or past NAC membership as an aspect on which to purposively sample.

14) Explore the use of an Internet survey, similar to that used by the Institute of Medicine Comparative Effectiveness Research (CER) Priorities Committee to reach a wide variety of stakeholders who can participate on a voluntary basis.

Improved electronic technology (e.g. web-based materials) will be used whenever possible to reduce the burden on the public and to reach a wide variety of stakeholders. The voluntary survey will be administered via a web-based/online instrument called Survey Monkey, an intelligent survey software designed to create professional online surveys quickly and easily – and at low cost. The software provides encryption capabilities for ensuring confidential responses and for the exporting of responses for future analyses. The software also allows for the identification of responders and non-responders without compromising the confidentiality of responses. Web-based surveys have several advantages over other types of surveys, including the ability to monitor the response rate in real-time and send customized reminder emails to participants, with easier-to-follow skip patterns, and ensure confidential and secure data.

OMB follow-up: This comment underscores OMB’s recommendation to include more respondents. It also raises the interesting possibility of using the same sampling frame used by the IOM CER priorities committee to reach a wide variety of stakeholders, including those who do not necessarily have an established relationship to the EHC program. This would lend more credibility to this study’s results and improve its internal and external validity.

Per OMB’s recommendation we have increased the number of interview respondents (by 8) as well as the survey to sample for the universe of the EHC program contacts database (i.e., ~400). While we find it is an interesting suggestion to follow the IOM’s approach used to identifying CER priorities, we have decided against a similar approach largely because the approach is not as relevant given the purpose of this study. Specifically, the IOM approach was aimed at developing a priority list of research topics to be undertaken with American Recovery and Reinvestment Act funding using broad stakeholder input, and identifying the necessary requirements to support a robust and sustainable CER enterprise.. Whereas, our study aims to specifically examine the EHC Program’s approach to engaging stakeholders and priority-setting, determine what is working well (or not) from multiple perspectives, and devise alternative strategies for the program to approach these governance areas to achieve its ultimate goals.

15) All survey instruments and draft reports should be made publicly available and should include a comment opportunity.

The analyses and findings produced in this project are intended primarily for internal use but may be shared with key government policy and management officials, AHRQ staff, and other members of the EHC program. This suggestion will be taken into consideration.

OMB follow-up: Since governance of the program affects the credibility of the program’s products and, therefore, whether and how it is used in clinical decision making and policy making, it does not seem like a bad idea to provide the public an opportunity to comment any governance proposals before they are implemented. After the “roadmap” is produced, will this be made available to the public for comment?

Request accepted. AHRQ will consider the suggestion and make an effort to encourage public review and comment.

In addition, three recommendations were made to the Effective Health Care Program, in general, and were not directly relevant to the proposal to evaluate the EHC Program’s governance structure, methods for engaging stakeholders, or approaches to setting priorities for research conducted by the EHC program. These suggestions are: to establish a position at AHRQ for a patient advisor on CER; to establish an explicit channel for AHRQ to educate patients on CER and, in turn, for patients to advise AHRQ; and to sponsor research designed to improve clinical decision making by both clinicians and patients. While these comments are not directed to the evaluation activities proposed, AHRQ will take these suggestions into consideration.

October 21, 2009

The Agency for Healthcare Research and Quality (AHRQ) appreciates the comments received from the Office of Management and Budget (OMB) on the proposed information collection under the project: “Evaluation of AHRQ’s Effective Health Care Program”. AHRQ’s specific responses to these comments are below:

Supporting Statement Part A

         P2 – please explain what “governance element” means (“Additional in-depth key informant interviews… that results from its governance element or approach, or about a specific, important governance element.”)

Response: Governance element in this study refers to a specific element, aspect, or dimension of the EHC program’s governance structure or approach that are the focus of this study. Examples of governance elements/aspects/dimensions of interest include: priority-setting, engaging stakeholders, transparency, and accountability.

         P2 – please clarify who precisely the “EHC Program Stakeholders” and “EHC Program Users” are. Is this a specific body of people, or stakeholders and users in general terms? How will each of interview participants be picked?

Response: EHC Program stakeholders and users do not refer to specific individuals; instead they refer to general stakeholders and users of the EHC program. To further clarify, EHC Programs stakeholders and users are groups or organizations that are known or anticipated to utilize the EHC program’s products (e.g., research reviews) for their organization or their individual interest. The following is the list of stakeholder and user groups that will be recruited to participate in an interview and the proposed number of proposed interviews with each type of organization:

Federal & State Policymakers and Programs (FDA, CDC, CMS, NIH, HRSA)	4
Private Sector (pharmaceutical companies, medical technology companies, biotech companies)	4
Disease-Specific Advocacy Organizations and Consumer Groups (MS Society, AARP)	4
Academicians (healthcare epidemiologists, medical school faculty, etc.)	2
Federal and State Public Health authorities, associations (Society of General Internal Medicine, the Academy of Managed Care Pharmacy, APHA, etc.)	2
Clinical Guideline developers (Joint National Commission, American Dental Association)	2
Health Plans (BCBS, Kaiser Permanente, United Health, etc.)	3
State regulating/licensing agency (JCAHO, etc.)	1
Data organizations (SHADAC, quality improvement organizations, etc.)	1
Provider groups (AMA, APhA, Pediatrics, etc.)	3
Pharmacy Benefit Management (e.g., Express Scripts, CVSCareMark)	1

A list of specific organizations for each of these organization types in the table above is only for illustration purposes, and will first be identified from the EHC Program’s “contacts database,” which includes a list of all contacts and the reasons for contact with the EHC Program (e.g., nominated a topic, reviewed a research review). If one of these organization types is not represented in the EHC Program’s database, then we will compile a list of potential organizations. From the compiled list of potential organizations to interview from each type we will select a diverse set of organizations in collaboration with AHRQ.

OMB follow-up: It seems that one of the main reasons why governance might be important is that it lends credibility to the end-product. Therefore, it is probably important to include more organizations from segments that have been particularly critical or skeptical of CER in general and perhaps EHC in particular. Device and pharmaceutical companies come to mind, as well as patient advocacy groups. Among patient advocacy groups, groups representing ethnic and racial minorities as well as groups representing patients with mental health issues have been particularly vocal about their concerns with CER.

Therefore, we would strongly suggest including more than 2 private sector organizations, and more than 2 disease-specific advocacy organizations and consumer groups in the interview sample. We would also suggest including at least 2 organizations for health plans and provider groups.

Request accepted. The study will include 2 more private section organizations, 2 more disease-specific advocacy organizations and consumer groups, 2 more health plans, and 2 more provider groups in the interview.

We would also need more detail on how AHRQ plans to pick these organizations, particularly as AHRQ is picking only 1-2 organizations from a much larger universe of organizations within each stakeholder category.

[SAME RESPONSE AS ABOVE]

First, we will use the AHRQ EHC program’s contacts database to identify the potential interviewees for each user/stakeholder subgroup. Next, we will examine the number and diversity of candidate organizations represented for each of the subgroups (i.e., health plans, provider groups). Additionally, we will examine what role(s) these organizations have had with the EHC program (e.g., reviewed report, nominated topic, etc.) to the extent that these data are available in the contacts database. Then we will purposively sample for diversity of organization (e.g., physician as well as nursing associations) type within subgroups and in terms of what role(s) they’ve had with the EHC program. We will also select the requisite number for each subgroup plus at least 2 alternatives for recruiting purposes given that some individuals will decline participation or not respond to invitations to participate in the interviews.

If cost is a concern, AHRQ may consider holding focus groups with several organizations representing each stakeholder group rather than selecting 1-2 for in-depth interviews (e.g. a focus group of Aetna, United, BCBS, etc. rather than picking just BCBS to represent the views of the entire insurance sector).

While we appreciate OMB’s suggestion to move to focus groups to address the extent of representation of specific groups, we respectively decline moving forward on this suggestion for three reasons. First, participants of focus groups should be homogeneous in some key aspects of import to the study, and while private sector organizations are homogeneous in some regards (i.e., similar organizational structures with the same goals), they may be vastly different in terms of their experience or encounters with the EHC program, which would arguably be the aspect on which we would want homogeneity. Second, the logistics of pulling together focus groups of key informants who are likely very busy for an in-person, or even a virtual focus group may be prohibitive. Lastly, this change would be a significant change in scope, and require extensively revising data collection protocols and obtaining additional resources. Again, we are increasing the representation of specific groups in the interviews as described above.

         P3 – please specify who will be participating in the Appreciative Inquiry workshop and how they will be selected.

Response: The Appreciative Inquiry (AI) workshop will include 20 participants, including:

• AHRQ staff

• EHC program staff

• AHRQ project officers of the EHC program centers (i.e., EPCs, DEcIDE, CERTS)

• Principal Investigators or other representatives from the research center networks

• Key program stakeholders or users

The IMPAQ/Abt team will work with the AHRQ Task Order Officer to identify workshop participants that are best qualified to meet the purpose of the workshop. The workshop is designed to facilitate consensus among decision-makers, to encourage EHC decision-makers to consider and discuss the proposed alternative governance models for engaging stakeholders and priority-setting, and roadmaps that should be further developed.

OMB follow-up: Will these workshops happen AFTER the document review and research on international organizations have taken place? We would encourage this sequencing, so that the workshop can comment on the findings of that research.

Yes, this is our planned sequence. However, we will conclude the document review and international comparison after the Workshop, in case there will be any follow-up work deemed necessary.

         P3 – please provide more information on what would be entailed in the document review and interviews with international organizations and how that information will feed into this study. For example, what kinds of documents will AHRQ be reviewing and how will they be analyzed? Which of the research questions outlined on page 2 will be answered with these analyses?

Response: The international organization interviews will focus on four key international organizations that have comparative effectiveness healthcare systems in place from the United Kingdom, Australia, Canada and Germany. Because these four organizations were instituted prior to the advent of the EHC program and because they have greater latitude over their own country’s health care system, they will provide a rich resource of information for this study. For example, interviews will help AHRQ understand the governance elements of those organizations to contrast with the EHC model, and then inform the development of the roadmap.

For the document review, the IMPAQ/Abt team will be looking at various background, organizational, and evaluative documents to identify their relation to priority-setting, governance structure, clinical adoption, and transparency.  Further, the team will be looking at what outputs (e.g. research review, summary guides) the EHC program produces, categorized by the national prioritized health conditions/diseases.

Both the document review and the international interviews provide supplemental information to the primary data collection activities (key informant interview, online survey and Appreciative Inquiry workshop), and also are exempted from the OMB clearance.

OMB follow-up: Even if the burden is exempt from the PRA, we still need to understand how this data will be used to inform the research being undertaken in this ICR. For example, will the document review and international interviews take place prior to the in-depth interviews and appreciative inquiry, so that the information obtained can be discussed in the in-depth interviews and appreciative inquiry? That sequencing will increase the utility of the in-depth interviews and appreciative inquiry and therefore we strongly recommend it.

Yes, the bulk of the document review and international interviews will take place prior to the in-depth interviews and appreciative inquiry. However, we do plan to conclude those two activities after the in-depth interviews and AI Workshop for necessary follow-up.

         P4 – while we support a mixed methods approach, it is not clear how the various methods being used will build off of and relate to each other. For example, it would seem important to include some questions about the international governance model “alternatives” (presumably derived from the interviews with the international organizations) in the key informant interviews to see what the key informants think about these alternatives. How will the survey methods feed into or build off of the key-informant/in-depth interviews? Will they be used to triangulate results, or will they be used to inform the questions asked in the interviews? Please clarify. The sequencing of the various instruments on table 4 would suggest that each phase of the study is meant to build off of the phase that came before it (an approach that we would highly support), but it is not clear that the instruments as they are written will provide the information AHRQ may need to build upon preceding results. For example, the questions in the survey, key informant interview guides, and the in-depth interview guides are very similar, so it is not clear what AHRQ hopes to gain from asking the same questions but in different formats.

Response: While the mixed methods used in this study are largely intended to triangulate data and findings from different sources and individuals, some methods will also build off one another. The key informant interviews will be conducted first, and therefore will subsequently inform the survey and in-depth key informant interviews, which are focused on specific examples of impact from the EHC program products. For example, based on the findings from the key informant interviews we will identify potential examples of impact to examine specifically for the in-depth interviews. The international comparison interviews will be conducted during these data collection efforts in order to clarify the governance elements of those organizations, and help identify what comparative information is needed about each of the international programs to inform the development of the governance alternative models.

Lastly, with regard to your question about whether informants will be provided an opportunity to comment on the alternatives, they will not. However, the Appreciative Inquiry workshop will involve the participants considering the alternatives and providing feedback on the most relevant or feasible alternatives to meet the EHC program’s goals.

         P5 – this ICR received a number of public comments, which we have forwarded to AHRQ. AHRQ may have received additional comments that did not come to OMB. Please provide a summary of all comments and a response to the comments, either in a separate document or under #8 of the supporting statement. If AHRQ chooses to use a separate document, please modify the response to #8 to clarify the number of the comments received and to explain that responses to comments are provided on a separate document.

Response: Fourteen thoughtful comments were submitted by five organizations: American Nurses Association (ANA), Mental Health American (MHA), National Health Council (NHC), Pfizer, and Pharmaceutical Research Manufacturers of America (PhRMA).  All five organizations are supportive of AHRQ’s proposal to evaluate the Effective Health Care (EHC) Program’s governance structure, methods for engaging stakeholders, and approaches to setting priorities for the research conducted by the EHC program.  Specific comments on the proposed evaluation and AHRQ’s responses to these comments are detailed in the separate attachment.

         P6 – the use of incentives are not meant to “confer distinction on, or to symbolize respect, esteem, or admiration for the participants…” Rather, they are meant to obtain a desired response/participation rate. Please explain what AHRQ believes would happen to participation/response if an incentive were not given or if a lower incentive amount were provided. Does AHRQ believe that respondents would choose not to participate, even though they appear to have established relationships with AHRQ?

Response: Each participant in the one-day Appreciative Inquiry workshop who is not a Federal employee will be provided a two hundred dollar (US $200) honorarium to compensate participants for their time participating in the workshop. AHRQ anticipates approximately 10 non-Federal participants for a total of approximately $2,000 in honoraria cost.  In addition, participants’ travel expenses and per diem will be reimbursed in accordance with General Services Administration (GSA) policies. The provision of honoraria is a standard industry practice to enhance participation rates.  Most anticipated workshop participants have established relationships with AHRQ and the Effective Health Care Program, and many of them have given their time voluntarily on multiple occasions.  However, AHRQ believes modest honoraria of $200 per day are warranted to both express appreciation for their continued participation and input into the program as well as to encourage their participation in the workshop and future Effective Health Care Program activities.  (NIH Manual 1130, Delegations of Authority, Acquisition #5, “Rates of Compensation (Honoraria) Under Professional Services Orders.”)”

OMB follow-up: We do not approve incentives on the basis AHRQ has presented in this response. For example, incentives are not intended to “compensate participants for their time” or to “express appreciation.” Rather, we approve incentives based on their impact on response rates. If a lower incentive were provided, it does not seem like it would have a significant impact on response rates, particularly given the established relationship many of these participants have with AHRQ and the EHC program. Therefore, we are prepared to approve a $75 incentive. Should AHRQ encounter difficulties with response rates, we would be happy to consider increased incentive amounts at that time.

Request accepted. AHRQ will revise the incentive amount to $75. However, AHRQ will re-assess the amount of the honorarium if the response rate becomes more concerning.

         P6 – please confirm that the statutory authorities listed would withstand a FOIA request. Please also clarify how the Privacy Act is applicable in this study. 

 

According to the OGC, AHRQ's confidentiality statute, Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c), will withstand a FOIA request. The Privacy Act is not applicable to this study and reference to it will be deleted.

Attachment C4

         Please define for the respondents what AHRQ means by “governance” and “governance element”  and “governance dimension” as these appear to be key concepts in this study but can be defined in any different ways.

Response: For clarification purpose, we will revise and add the following definition of “governance” to the beginning of the interview protocol:

Governance refers to the administrative structures, policies, processes, and decision rights and responsibilities that support the operations of the EHC program by providing strategic direction, allocating resources, coordinating or controlling activities, managing risks, and using resources responsibly and with accountability, and ensuring objectives are achieved. Governance describes both the process of decision-making and the process by which decisions are implemented.

Governance element in this study refers to a specific element, aspect, or dimension of the EHC program’s governance structure or approach that are the focus of this study. Examples of governance elements/aspects/dimensions of interest include: priority-setting, engaging stakeholders, transparency, and accountability.

Governance dimension in this study includes, but is not limited to:

• Vision and mission statements

• Strategic plans

• Administrative roles or job descriptions (e.g., the stakeholder group)

• Organization charts

• Bylaws or guidelines for operation (e.g., priority-setting, engaging stakeholders)

• Formal and informal communication channels and processes

• Funding mechanisms

• Steering committees, advisory groups, or expert panels

• Public-private partnerships and other collaborative work structures

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

         Please revise this protocol to be more neutral. For example, some stakeholders may believe that the impact of the EHC program has been negative, yet the protocol seems to steer respondents to providing examples of successes.

Response: The focus on successes was intentional because the evaluation framework was guided by Appreciative Inquiry; however, we have revised the protocol per your request to include questions about the negative impacts.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

        Since governance is the key concept for this study, placing the governance question at the very end of the interview seems a bit out of place. Can AHRQ explain why the governance question was placed at the end? Does AHRQ want examples of successes and impacts that may not have stemmed directly from governance?

Response: The in-depth key informant interviews are aimed at examining specifically the impact of an EHC Program product that resulted from a governance dimension (element or approach). Consequently, the interview questions are focused on examining the impact of the EHC program product and the specific governance dimension that contributed to the impact. Questions 3 and 6 on the original interview protocol submitted were about governance. Per the recommendation, we will move question 6 under question 3 (combining the two). We will add questions about impacts that may not have stemmed directly from governance.

Attachment C3

         Is there some reason why not all of the participants are asked specifically about the governance of the EHC program, given that this is the key purpose of this study? For example, the EHC Program Users and Stakeholders do not appear to be asked about governance.

Response: Yes. EHC Program Users and Stakeholders would not be expected to know how the program is governed, and therefore are not asked questions about governance.

OMB follow-up: Why would they not be expected to know how the program is governed? This appears to be one of the areas that often comes up when patient advocacy groups or minority health groups, for example, or medical device companies question the credibility of CER research.

Also, in response to public comments, AHRQ said “The goal of the proposed project is to promote and structure the involvement of health care providers, communities, and consumers (both program users and contributors) in the EHC program’s governance.” If the goal is to promote involvement of program users and stakeholders in governance, isn’t it important to assess what they know of—and how they feel about—the current governance structure?

We would strongly recommend inserting questions into the survey that assess this dimension specifically among EHC program users and stakeholders. One of the response categories could be “do not know” if, in fact, they do not know about the governance of the EHC program. A follow-up question might be “In your opinion, would knowing more about how the EHC program is governed increase the credibility of EHC program products?”

To clarify, the users and stakeholders of the EHC Program are actually being asked questions that indirectly relate to the EHC Program’s governance, specifically the program’s governing approach for i) engaging stakeholder and ii) setting priorities (e.g., topic nominations from the public), whereas the AHRQ and EHC Program Staff are explicitly asked about “program management, collaboration, and governance”, and as such, provided a definition of “governance” as part of the interview process. Similarly, the AHRQ and EHC program staff are asked questions about the approaches for engaging stakeholder and setting priorities governance as well.

Attachment E

         Please define for the respondents what AHRQ means by “governance” and “governance element” and “governance dimension” as these appear to be key concepts in this study but can be defined in any different ways.

Response: For clarification purpose, we will revise and add the following definition of “governance” to the beginning of the interview protocol:

Governance refers to the administrative structures, policies, processes, and decision rights and responsibilities that support the operations of the EHC program by providing strategic direction, allocating resources, coordinating or controlling activities, managing risks, and using resources responsibly and with accountability, and ensuring objectives are achieved. Governance describes both the process of decision-making and the process by which decisions are implemented.

Governance element in this study refers to a specific element, aspect, or dimension of the EHC program’s governance structure or approach that are the focus of this study. Examples of governance elements/aspects/dimensions of interest include: priority-setting, engaging stakeholders, transparency, and accountability.

Governance dimension in this study includes, but is not limited to:

• Vision and mission statements

• Strategic plans

• Administrative roles or job descriptions (e.g., the stakeholder group)

• Organization charts

• Bylaws or guidelines for operation (e.g., priority-setting, engaging stakeholders)

• Formal and informal communication channels and processes

• Funding mechanisms

• Steering committees, advisory groups, or expert panels

Public-private partnerships and other collaborative work structures

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

         We understand that statute requires the EHC program to provide “rational” assessments of therapeutics and other interventions, but the word “rational” seems a bit peculiar. Would “objective” or “credible” be more appropriate terms?

Response: Yes, we will revise the term to be “objective”.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

         Please add a question as to the usefulness/usability of the EHC program assessments. While the survey includes information about the utility of the TRANSLATIONS the program provides of its assessments, it seems worthwhile to also assess the utility of the ASSESSMENTS themselves (e.g. right after question #10).

Response: Yes, we will add this question.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

         Please clarify what the follow-up column refers to. For example, what does “18(e)” refer to in terms of being a “follow-up” to 18.

Response: Question 18(e) refers to the situation that if a respondent chooses “e” for question 18, s/he will be approached for question 19. We will delete the 18(e) next to a question 18, which is an error. Likewise, 21(a) and 21(b) next to question 21 will be moved to be next to question 20.

         Question #22 is very important and respondents may benefit from being able to provide an open-ended response. Would AHRQ consider adding an open-ended field in follow-up to this question?

Response: Yes, we will add an open-ended question.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

         Question 25 – 26: is there some reason why the EHC stakeholders are not asked these questions?

Response: Questions 25 and 26 are related to the topic nomination and selection process, and targeted to EHC Center staff. EHC stakeholders are not expected to be involved in these program functions.

OMB follow-up: Why would they not be expected to be involved? This appears to be one of the areas that often comes up when patient advocacy groups or minority health groups, for example, or medical device companies question the credibility of CER research (i.e. they believe their views are not taken into account adequately in the topic selection process). We would strongly recommend inserting questions into the survey that assess this dimension specifically among EHC program users and stakeholders.

Suggestion accepted. Please see the revised supporting statements.

         On questions 27-31, question 39, question 42: the ability of respondents to “check all that apply” might carry the risk of diluting the utility of the information AHRQ receives. How likely are respondents to check all of the boxes? And if so, how useful would that information be to AHRQ? Would it be more useful to compel respondents to prioritize their responses (e.g. rather than asking “which of the following EHC program priority conditions warrant additional research by the EHC program (check all that apply)?”, would a question like “which of the following EHC program priority conditions MOST warrant additional research (you may check up to 3 boxes)?” provide more useful information?)

Response: Thanks for the suggestion. We will revise those questions accordingly.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

         Question 27: please provide an “other” option as a response category.

Response: We will add that option.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

• Question 28: please revise this question as “Please, explain further what specific research is needed and why.”

Response: We will revise the question per your suggestion.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

         Questions 33-36: since the program staff are being asked to evaluate the worth of their own research, aren’t respondents likely to provide skewed/biased responses to these questions? Is there some reason why the stakeholders aren’t asked these questions (albeit in somewhat revised form)?

Response: The questions between 33 and 36 are targeted to collect impact-related information for the EHC program through on-line survey, we do recognize the potential bias of responses from the respondents which limits the use of the responses. We will remove these questions from the survey questionnaire.

         Question 42: the question asks respondents to select the products that have been “most helpful” and yet respondents are told to select all that apply. Would AHRQ receive better information if respondents were limited to a maximum of 3 options?

 Response: We will revise the question to be the “the most helpful 3”.

OMB follow-up: Thanks. Please send us the instrument with this revision.

Please see the revised supporting statements.

Part B Responses

Nov 6, 2009

The Agency for Healthcare Research and Quality (AHRQ) appreciates the comments received from the Office of Management and Budget (OMB) on the proposed information collection under the project: “Evaluation of AHRQ’s Effective Health Care Program”. AHRQ’s specific responses to the comments related to Part B Supporting Statements are below:

OMB Question: P2 – can AHRQ clarify who the EHC stakeholders are? Does this refer to a particular group of people with whom the EHC program has an established relationship? For example, “policy makers” seems like it could be a large, rather amorphous group of people who use EHC program information to make policy decisions (e.g. Congress, Federal government, State government, etc.). However, the table on this page suggests that there are 17 people who make up the universe of policy makers for purposes of the EHC program. Please clarify. To the extent these stakeholders represent people/organizations with established relationships with the EHC program, is there interest in broadening the universe to encompass people/organizations with a stake in EHC program information but who – perhaps because of their concerns with EHC program governance – have not established relationships with the EHC program?

Response: EHC Programs stakeholders and users are groups or organizations (and individuals) that are known or anticipated to utilize the EHC program’s products (e.g., research reviews) for their organization or their individual interest. EHC Program stakeholders and users in the table below (table 1 of the Part B supporting statement) is based on the EHC Program’s “contacts dataset,” which includes a list of all contacts and the reasons for contact with the EHC Program (e.g., nominated a topic, reviewed a research review). This table captures a list of stakeholder and user groups that will be recruited from the “contacts dataset” to participate in the on-line survey:

EHC Program Stakeholders
The EHC Program Stakeholder Group	18
Employers & Health Related Business Groups	6
Federal Partners	11
Healthcare Industry	41
Healthcare Providers	43
Patient/ Consumer/ Advocacy Organization	68
Pharmacy & Therapeutic	10
Policy Makers	17
Professional Organizations	78
Researchers	33
Third Party Healthcare Payers	18
Subtotal	335

In the current AHRQ “contacts database”, there are 17 individuals in the “policy makers” category. These are comprised of state Medicaid offices, state departments of health, and state and national level health planning and policy organizations.

Inclusion in the study is limited to a group of people with whom the EHC has an established relationship and who are therefore familiar with the EHC program. This decision was made with the idea that these stakeholders’ knowledge of the EHC program will allow them to answer most meaningfully, particularly with respect to program governance. Although input from stakeholders who do not have established relationships with the EHC program would provide additional insights, identifying respondents without a relationship but yet adequately familiar with the program would be difficult and would incur significant additional time, effort, and expense, potentially delaying the analysis. For this reason, the judgment was made to limit the scope of the study to stakeholders who have been involved with the EHC program.

OMB follow-up: This would appear to be a rather large study limitation. We would like assurances that AHRQ will be using these results for internal agency use and that the results are not generalizable and will not be presented as such. We would also like assurances that AHRQ will not use these findings in lieu of a more formal program evaluation for purposes of assessing program assessment (e.g. PART).

AHRQ intends to use this study’s findings for internal use only and the results will be presented as such. AHRQ will not use the findings from this study in lieu of a formal PART evaluation.

OMB Questions: P2 – please clarify the extent to which AHRQ plans to rely on and use strata-level results? As alluded to on page 4, the universe of some of the strata is small. How reliable will evaluation at the strata level be?

Response: We define the universe of on-line survey to be stakeholders with previous contacts with the EHC program, who are familiar with the program to provide meaningful responses. From that universe, we intended to draw sample respondents from each strata. We do recognize some strata’s universe is small, and we planned to contact all respondents for survey if the universe is <10, and contact at least 10 respondents if the universe is close to 10. Based on our sample selection design, the proportion of respondents for sample and universe ranges from 60% to 100%, which is reasonably high. We believe this proportion will generate representative results for the universe we identified in the study.

OMB Questions: P2 – given that the overall universe of respondents is relatively small, leading to even smaller universes within strata, has AHRQ considered surveying the universe of respondents rather than a random sample?

Responses: The decision to survey a random sample rather than the universe of respondents comes down to a question of resources. We asked our contractor for a rough estimate of the cost of broadening the survey to encompass the full universe of respondents, and their assessment is that the costs associated with the online survey would increase by about 100%, in addition to added costs associated with the IMPAQ team’s activities such as contacting and monitoring the communication with more individuals, analyzing, and summarizing more qualitative data from the free-text responses based on open-ended questions. Although the more comprehensive survey would provide richer data, the random sample will provide valid information and falls within our cost constraints.

OMB follow-up: While it may take more time to analyze the free text responses, it’s not clear how the administration of an online survey to 194 more individuals would increase the costs by 100%.One of the advantages of an instrument like Survey Monkey is that it is scalable. Moreover, most of the questions have fixed response categories. Therefore, the incremental cost of adding additional respondents should be very small. Please explain in more detail.

Response accepted. We will survey the universe instead of the sample, which is estimated to be about 400.

There will be very limited economy of scale to analyze the quantitative data collected from the survey, but not much of that is gained from the other aspects of the survey activities (e.g. managing the respondents in survey administration, and analyzing the free text responses from a significant number of open-ended questions). With the addition of more open-ended questions based on OMB’s suggestion, more qualitative data with free texts will needs to be analyzed and summarized.

OMB Questions: P2 – please explain what AHRQ plans to do if the 80% response rate is not reached.

Responses: Firstly, we will try the approaches described in the supporting statement to achieve a high response rate in the first place. Secondly, in the case that the response rate is less than 80%, we will remedy the problem with two approaches:

1. Use multiple contacts with non-respondents. We will send out multiple email reminders before the survey is closed. The reminders will come from AHRQ and the IMPAQ team jointly. Also, the non-response issue is not fully addressed, we will use telephone follow-up to contact non-respondents. The survey respondents have already established the relationship with the EHC program, so we expect the multiple contacts will increase the response rate.

1. Use statistical procedures to correct for the non-response issue. If the response rate is below 80% but reasonably high (e.g. above 75%), we will impute the missing values using imputation techniques (such as mean response imputation or hot-deck imputation method).  Item imputation will be performed only if the respondent answers 75% of all questions or more. For respondents who will answer fewer than 75% but more than 50% of the questions missing items will not be imputed; but reported responses will be included into the analysis where they are needed.  The pool of potential donors in the imputation will include respondents deemed to be similar to the respondent who provided incomplete responses, with respect to predetermined characteristics. We will determine the exact imputation techniques upon the analysis of reasons for receiving the missing responses. In some situations where good donors are not available, we will model the participant's response propensity using logistic regression model. Predicted response probabilities will then be used to create non-response adjustment classes within which the participant's weights will be adjusted for non-response.

OMB follow-up: While multiple contacts with non-respondents would seem like a good idea, it is not clear that imputation methods on a sample so small is appropriate. Please clarify.

 

The imputation will be done only if the response rate is 75% or more, which represents a minimum sample size of approximately 320, if the universe (approximately 400) is surveyed. Imputation has often been done on smaller samples and references can be provided, if needed.  We believe this size is even large enough to develop a logistic regression model for response propensity. 

23