Attachment G -- Research Topics and Sources

HHSA290200900019I Mathematica Policy Research

B.6

Appendix B. Topics Covered in Literature Review, Surveys, and Grantee Interviews

Topic	Literature Review	Surveys	Grantee Interviews
Partnerships Partnerships were a key requirement of all three types of grants. Knowing what types of partnerships are most effective and long-lasting (during the grant project period and beyond) and how partnerships can be made more effective will improve future AHRQ grant making and contribute to knowledge in the field related to improving care through coordination and data sharing. A specific focus on vendor partnerships is included as AHRQ believes these may be an important key to achieving improved outcomes from health IT implementations.	An association was found between prior collaborative experience among partners and reported success, but data limitations were severe. Types of organizations that were partners varied widely but could not be coded accurately. Likewise, the scale of the projects, which may be important to understanding success, could not be classified into meaningful categories for analysis with the information available.	All Surveys—Number of partners by type of organization Extent to which partners continued to work together after the grant Reasons for lapses in partnerships (focus on point-of-care organizations) Whether ARRA/HITECH altered more or less collaboration after its passage in 2009 Implementation and Planning—Leadership support among partnered organizations as a strength or weakness in planning Implementation—Level of participation and trust among partners and geographic distance between them as strengths or weaknesses of implementation Whether partners used strategies to maintain and enhance partnerships; if so, describe any effective strategies Health IT vendor support as a strength or weakness in implementation	All—How successful was the partnership aspect of the project and why Lessons learned about the strengthening of partnerships, what can undermine them, partner selection, and importance of partners to project success Implementation, and Planning (if subsequent implementation)—role and importance of the vendor relationship in influencing implementation successes
AHRQ Grant Process/Features To improve its grant process, AHRQ needs feedback on the usefulness of the co-funding and multi-partnership requirements as well as the sufficiency of the funding time and level in order to maximize project success. AHRQ needs input on whether the grantees believe planning grants are an important mechanism to prepare health care organizations for participating in large-scale research. AHRQ also needs feedback on whether it has successfully met an unmet need—whether the project would likely have taken place without the grant. In addition, AHRQ required projects to define “specific aims” up front; the agency needs feedback on whether this was helpful and how and why aims may have changed over the course of each project as well as the extent to which researchers may have been overly ambitious in the number and breadth of specific aims. Finally, the agency needs to better understand the benefits and drawbacks of the various features of the three different grant mechanisms that were used: the RO1 traditional grants for the value grantees, the cooperative agreement type for the implementation grantees, and the P-grants for the planning grantees.	Grantees did not discuss these topics in their reports.	All—Recommended funding level, time period, and co-funding policy if AHRQ were to undertake similar efforts in the future Planning—Importance of involvement with the planning grant to pursuing subsequent health IT implementation Whether the organization/partnership needed to continue planning after project period/funding ended, and if so, sources of support for this	All—Importance of the grant to advancing health IT for the involved organizations Grant requirements or processes that were particularly helpful to the project and any that impeded the project Anything about the grant structure or process that should be changed if AHRQ funds similar projects Grantees’ stage of thinking about health IT prior to grant funding Implementation and Planning—Odds of achieving similar accomplishments without the grant Implementation—Whether/how the 50/50 co-funding requirement resulted in more commitment to the project Whether defining “specific aims” up front for AHRQ was useful, and whether and why these aims changed over time
Planning for Health IT Identifying pathways to successful health IT planning (and pitfalls to avoid) will help AHRQ and others support health IT implementation more effectively. How researchers and Chief Information Officers and administrators at various healthcare organizations can effectively support health IT planning is another important topic for AHRQ given its focus on translating research into practice.	The literature review team was able to Identify frequently encountered barriers to planning and common strategies used to address barriers from the grant reports and other grantee literature. Due to lack of systematic data collection, the team could not tell how widespread the barriers were or their primary cause. Although some types of solutions to barriers were discussed, there was a lack of information in the source documents on strengths of or facilitators to planning. Researcher roles were not discussed. Pathways to successful health IT planning could not be identified.	Whether a list of planning issues (including those identified by the literature review) is a strength or weakness of the project, and if a weakness, what was the primary cause Up to three features of planning that contributed the most to success Information that will help generate insights about pathways to effective planning: - Who was involved in planning and selecting the health IT - Types of researchers and research organizations involved - Technical assistance sources critical to project success - Whether certain steps believed to be associated with successful planning and implementation were taken - Presence of various barriers and their impact on the planning process - Planning features that most contributed to success - Actual and recommended factors considered in purchasing health IT Facilitators tried during planning and whether they had a positive effect (list based on literature and team knowledge) Planning - Changes grantees would make to the planning process Implementation—Any important issues not included in the implementation plan that had to be addressed before implementation	Implementation—How much pre-implementation planning was done, and how important was it to implementation; most important resources for planning; whether parts of planning process were cut short due to resource constraints; whether important user needs were missed during planning Changes to original implementation plan and reasons for the changes All—Lessons learned (what, if anything, would grantees have done differently in planning) Survey followup—Discussion of survey responses regarding strengths, weaknesses, and effective facilitators to help develop pathways to success.
Implementation of Health IT Identifying pathways to successful health IT implementation (and pitfalls to avoid) will help AHRQ and others support health IT implementation more effectively. How researchers, Chief Information Officers and administrators at various healthcare organizations can contribute to successful implementation is another area of interest for AHRQ given its mission to translate research into practice. Within the general topic of implementation, AHRQ seeks particular insights into the various training and workflow/process redesign strategies that health care organizations used, as these are two topic areas on which more guidance to the field is needed.	The literature review team examined project characteristics associated with “success,” as summarized from the positive/negative direction of the measures each grantee chose to report. Analysis yielded more questions than answers, but it helped to frame the survey and interview questions to better illuminate project characteristics that may be associated with better results.	Value— Request the grantees point us to the source that best illustrates lessons learned from this project; key contact information requested, if applicable Implementation—Sources of support for implementation, besides AHRQ Whether health IT was implemented; if not, why not Role of research team in aspects of implementation and their influence, if any, on health IT use Descriptive information on training (items designed to see to what extent best practices from the literature and adult-learning theory were used) Extent and nature of workflow/process redesign Who provided technical assistance, and how much did it contribute to/inhibit successful implementation Planning—Whether health IT was subsequently implemented, and if so, sources of support for implementation; use and satisfaction with the health IT implemented; why health IT was not implemented, if it wasn’t	Implementation and Planning (if health IT was implemented—Parts of implementation that worked best and parts that were most difficult [For implementation grantees, probe using survey information. Probe on training to understand what training strategy was used, how effective they found it to be, and whether there is anything they would change about it. Probe on workflow/process redesign to understand how workflow/process redesign was done and when relative to health IT implementation, whether they see improvements to efficiency or quality of care as a result, and what if anything they would change about how they did it] Planning—If health IT was implemented, follow up on survey responses to understand what was implemented and how closely related it was to the implementation plan developed during the planning period If not implemented, discuss reasons why not
Common and Unique Barriers and Facilitators to Implementation Across Types of Health IT and Care Settings	Barriers, facilitators, and lessons learned were summarized from the grantee final reports. Due to lack of systematic data collection, the prevalence of barriers and extent of facilitator use (and the impacts of both) could not be understood.	Value—Whether a list of implementation issues was a strength or a weakness with respect to influencing project outcomes Implementation—Whether a list of implementation issues was a strength or weakness and, if a weakness, severity of impact on implementation Whether a list of potential facilitators was tried and the impact on the project	Implementation (and Planning, if Subsequently Implemented Health IT)—What facilitators worked especially well to help advance the project; whether these positives were anticipated or discovered later; any reason to think these things would work especially well for specific health IT, populations, or settings; cost of or savings from these facilitators Most important barriers faced; whether anticipated or surprised by them; reasons to think they are more or less serious depending on the specific health IT, the population, or care setting; strategies used to overcome them [For implementation grantees, survey information will be used to probe on barriers and facilitators]
Tool to Help Facilitate Better EHR Implementation in Rural Hospitals	Grantee reports lacked in-depth information, and literature by itself does not support development of a practical tool.	Specific issues related to EHR implementation in rural hospitals were not covered	Implementation Projects Involving EHRs in Rural Hospitals— - Order of implementation of EHR functionalities (what order did they use and how well did it go) - Balancing the needs/preferences of the various members of the health care team (including patients and their caregivers) through all stages of implementation (system selection, fit with workflow, etc.) - Level of buy-in achieved from various health care team members, how it was achieved, and lessons learned - Transition strategy from paper and isolated electronic systems, speed of transition, and how grantees handled productivity stress during implementation - Types of training used and how well it went
Qualitative Information On Health Information Exchange (HIE)	Because the field of HIE has been rapidly changing, updating information from the HIE grantees is critical before disseminating any information on this key topic for AHRQ and others.	Specific issues related to health IT projects were not covered	Implementation Projects Involving HIE—Supplemental specifics to help identify pathways to successful implementation—parallel to the items above for rural hospital EHRs but applied in the HIE context
Rural Experience with Health IT The announcement about the THQIT grant opportunity emphasized rural organizations. AHRQ needs to be able to understand the contribution made by the THQIT grants to the knowledge of how to make health IT work in rural areas.	There was no significant difference in success reported by rural versus nonrural grantees; the grantee literature did not shed light on whether and how being rural affected the experience with health IT or the pathways to success.	Implementation—Survey asks grantees to self-identify as major rural focus, moderate rural focus, or no rural focus for more reliable analysis of rural versus other grantees To facilitate rural/other analysis, four factors believed to often be different for rural partnerships are included on the list of potential strengths and weaknesses of implementation: geographic distance between partners; sufficiency of personnel with needed knowledge, skills, and abilities; infrastructure foundation for health IT; and business case for the health IT and availability of funds. The impact of these factors on the project can also be assessed through this question	Implementation (and Planning, if Subsequently Implemented Health IT)—Discussion with grantees of how the following factors may have influenced the project; for rural-focused projects, ask if there were any other factors associated with being rural that influenced how they implemented health IT or how successful they were: - Geographic distance between partners - Availability of personnel with the right knowledge, skills, and abilities - Access to needed capital - Health IT infrastructure [Survey information will be used to probe] Value—Whether any of the above factors influenced the value study’s methods or results
Effects on AHRQ Priority Populations AHRQ needs to be able to describe the extent to which the THQIT grant projects focused on and are believed to have positively influenced quality of care for the AHRQ priority populations, which are central to its mission.	Most of the grantee literature did not discuss effects on AHRQ priority populations.	Implementation—The survey will identify projects that focus on AHRQ priority populations For these projects, what is the perceived effect on quality of care for each priority population [Note to AHRQ: we recommend also adding this to the value survey]	Implementation (and Value, if Added)—Interviews will follow up on survey information to discuss the effects on AHRQ priority populations, obtain examples that will bring the survey information “to life” If adverse effects are discovered (see below), probe on whether they affect particular population subgroups more than others
Quality Measures	Literature review identified a small number of grantees with good or suggestive evidence on quality outcomes and analyzed the information provided.	Value—Continuation of outcomes measurement after the project period: which types of outcomes were measured during and after the project period, and whether most recent measurement was improved, unchanged, or worsened	Implementation, Value—Lessons learned about measuring results and getting others to use the results
Other Outcomes, Benefits, Drawbacks	Quantitative and qualitative outcomes reported by grantees were categorized broadly. Information was not reported consistently enough to draw conclusions about the prevalence of various types of benefits.	Value - Whether the value study prompted health IT implementation or increases in use of health IT (and among what types of organizations) Extent to which value study led to pursuit of additional health IT or research pursuits Implementation—Types of benefits experienced currently (list) Text summary of benefits other than those previously reported in final grant report or other publications Planning—Extent to which organizational goals for planning were accomplished and an implementation plan was developed Whether planning grantees implemented the planned health IT Benefits of planning (list + other/specify) Any long-term difficulties resulting from the planning process	All—Main successes and disappointments of the project Implementation (and Planning, if Health IT Subsequently Implemented)—Discussion of benefits seen from the project for organizations, individual clinicians, patients Discussion of any adverse effects that were discovered Implementation, Value - Whether the original business case for health IT was the business case at the end of the project; how the grantee would describe the business case for health IT Value—Insights about whether there was synergy between the health IT implementation effort and the value study, and how that synergy could be enhanced to improve outcomes or documentation of outcomes
Sustainability/Expansion of Implemented Health IT		Implementation—Continuation or expansion of use of health IT implemented during the grant period; reasons for discontinuation and for sustained/increased use, where applicable Adequacy of technical support, post-implementation	Implementation—Discussion of survey responses to understand the reasons for sustained, increased, or decreased use of health IT Financial issues in sustaining and expanding health IT; how and by whom are financial costs of sustaining or expanding health IT being borne; what has been learned about addressing financing issues; what types of indirect and direct costs are incurred.