Supporting Statement B -- System Redesign for Value in Safety Net Hospitals and Delivery Systems

SUPPORTING STATEMENT

Part B

System Redesign for Value in Safety Net Hospitals and Delivery Systems

Version: March 2012

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent Universe and Sampling Methods 3

2. Information Collection Procedures 5

3. Methods to Maximize Response Rates 6

4. Tests of Procedures 7

5. Statistical Consultants 7

B. Collections of Information Employing Statistical Methods

Definitions of Key Terms:

• Safety Net Hospitals/Health Systems: Hospitals and health systems which provide a significant portion of their services to vulnerable, uninsured and Medicare patients.

• System Redesign: Aligned and synergistic quality improvement efforts across a hospital or health system leading to multidimensional changes in the management or delivery of care or strategic alignment of system changes with an organization’s business strategy.

• Resources: Learning materials and environments developed to support, advance, and facilitate quality improvement efforts (e.g.: tools, guides, webinars, learning collaboratives, training programs). Note: For the purposes of this project, the term resources should not be interpreted to imply financial support for routine staffing or operations of Safety Net systems.

1. Respondent Universe and Sampling Methods

The purpose of this project is to understand available and existing resources for Safety-Net (SN) hospitals working on system redesign and quality improvement initiatives and to identify opportunities for further development of educational and training resources to support the work of SN hospitals. In order to achieve this aim, this project will consist of two phases. The first will be a scan of existing knowledge through available data sources. The second will be a series of case studies of 8 SN hospitals involving the primary data collection for which OMB clearance is being sought. The purpose of this data collection is not to make population estimates from a representative sample of SN hospitals, but to provide illustrative examples of the ways SN hospitals have used available resources to support their efforts in system redesign. Therefore, we are using a purposive sample. The cases will be selected to ensure diversity on the following characteristics:

Case study sites will be selected from a pool of SN systems that 1) are currently or recently engaged in significant system redesign, preferably strategic redesign, and 2) reflect variety on five additional criteria:

1. Different levels of system performance success in terms of quality, based on available quality performance measures, and, where data are available, financial stability as indicated, for example, by positive operating margins.

2. Different levels of success of system redesign efforts, including some that were not successful.

3. Different foci of system redesign, such as one or more of the following:

   1. Redesigning organizational structures and processes to reflect new organizational strategies to improve patient care, financial viability, and competitive position;

   2. Restructuring clinical practices and operations to enhance coordination across the continuum of care to increase efficiency while strengthening quality of care;

   3. Redesigning the built environment to support improved patient care with more efficient processes;

   4. Improving administrative operations and financial management to improve efficiency and increase revenue;

   5. Using information technology (IT) to improve system-wide quality, safety, and efficiency;

   6. Improving performance and outcome measurement for quality improvement and accountability;

   7. Building knowledge and structures to improve, retain, and spread system redesign skills.

4. Variety in external system redesign supports, such as one or more of the following:

   1. Training in targeted improvement methods (e.g., Lean, system re-engineering, Six Sigma, rapid cycle improvement);

   2. Participation in collaboratives/learning sessions (e.g., Institute for Healthcare Improvement, Premier);

   3. Participation in funded grants/programs (e.g., Robert Wood Johnson Foundation, National Association of Public Health Fellowship Program, Healthy Communities Access Project);

   4. Work with consultants/suppliers/academic affiliates that provide expertise and technical assistance.

5. Range of safety-net hospital types (e.g., ownership/governance, size/complexity/academic affiliation, rural/urban, geography, and potentially, financial stability).

The case study sites will be identified in two groups. For Group 1, we have pre-selected a sample of three SN hospitals, with which the research team has worked with in the past: Boston Medical Center (Boston, MA), Montefiore Medical Center (MMC), and Cambridge Health Alliance (CHA) for two reasons. First, as SN systems that are engaged in redesign and show a commitment and receptivity to change and improvement, they are natural candidates for inclusion in this study. They meet all the criteria above and vary in size, complexity, foci of system redesign, and variety of resources used to facilitate redesign. Therefore, they offer diverse redesign experiences and will offer different lessons from their redesign experiences and provide a valuable first round of case studies. Second, they are accessible. Previous experience shows that obtaining permission to conduct interviews and site visits, particularly at hospitals undergoing major system change, can be a time consuming process. Starting our research at three hospitals with which we are familiar and have worked successfully in the past will facilitate project start-up. Hospital leadership at each of the three sites has already agreed to participate.

Group 2 will be selected from a pool of candidates identified through our systematic review of the academic and grey literature and conversations with industry experts and our AHRQ Task Order Officer. Group 2 is intended to broaden the case-study inquiry by selecting cases to capture variation on combinations of the criteria listed at the beginning of the section that were not represented in Group 1.

The final criterion for the study will be willingness to participate. We are confident, based on the expected size of the candidate pool and past experience in conducting similar studies, that we will have no difficulty securing a final sample of 5 SN hospitals that meet all criteria.

With the assistance of key contact people at each participating hospital, relevant hospital staff will be identified using purposive sampling. These individuals will include: hospital leadership, nursing and physician leadership, quality improvement leadership, or middle managers and front line staff involved in system redesign efforts, Approximately 15 – 20 people at each hospital will be identified and asked to participate in in-person interviews, producing a total sample of 160 respondents across 8 sites. In sum, the respondent universe includes any frontline clinical providers, hospital administrators and leaders, at all participating hospitals, who have been involved in system redesign activities. We expect a 95% response rate.

In addition to individual in-person interviews, we will ask each participating site to compile some publically available documents such as annual reports and quality metrics. The total sample for this portion of the data collection is 8 (one person from each participating hospital) and we expect a response rate of 100%.

2. Information Collection Procedures

Data collection and analysis will follow the same methods for both groups of case study sites.

Qualitative methods (individual and small group interviews and document review) will be used to generate rich descriptive examples to illustrate the resources available and used by safety net hospitals in system redesign.

In-person interviews: Interviews will be conducted on site during case study site visits. Site visits will take place after OMB approval is received and will take place over 2 days. This will be necessary to conduct in-depth interviews with up to 20 people at each hospital or health system. Informants will range from senior medical center leadership and clinical managers to staff working on quality improvement initiatives at the unit level. Each interview will last no more than 60 minutes.

We have developed an interview guide to ensure consistent data collection and data management across site visits (see Attachment A). Boston University’s IRB will approve all protocols, recruitment, and interview procedures before any contact is made with potential participants or any data is collected.

The interview guide has been developed based on a review of the academic and grey literature and on discussions among the project team, including the AHRQ Project Officer, and a panel of industry experts. The focus will be broad, and the questions will be open-ended with the goal of obtaining the perspectives of multiple stakeholders. The interview guide includes questions that need to be covered, as well as probes designed to ensure that all critical topics are addressed. As exact roles of participants or informants at each site may vary across sites, the questions we ask each interviewee may differ.

Interviewees will be identified by a primary contact person at each site based on criteria identified by the study team. The BU study team will work directly with the point of contact to identify a relevant set of interviewees at each case study site.

The ability to tailor initial questions and probes to individual informants is a strength of qualitative data collection. Open-ended questions and variability of probes will allow the interviewer to explore responses more fully and capture evolving themes as the interview progresses, which more structured interview formats do not allow. As noted, data analysis will be guided by a structured analytic tool used by the study team on similar projects to facilitate qualitative data analysis. The conceptual framework for the research will be used to identify key constructs. The analytic framework will be further refined as the project team completes the review of relevant literature. There are two potential sources of bias in this form of data collection. First, if the language and sequence of the questions are associated with the responses, our conclusions may vary in an unknown way. Second, since there are no rigid categories, aggregating responses requires interpretation by the analyst and it is possible that there will be some analyst bias. To offset these threats to validity, we will triangulate responses among interviewers (and where possible across interviews). Additionally when possible and appropriate, we will compare interview responses to information obtained from internal or published documents or external expert knowledge of the site under investigation.

Following each site visit, the study team will jointly fill out the full matrix with narrative evidence, where possible, of how each construct appears in that site. The analytic tool will provide the basis for detailed site profiles from which to develop a rich understanding of the dynamics of each case and the basis for cross-site comparisons to identify common patterns or contrasts. No estimation procedures will be employed in this project, as the universe of potential respondents is small. There are no plans to impute missing data in the analysis.

Document Review: Each site will be asked to produce documents relevant to its redesign efforts and use of external resources. This may include, but is not limited to strategic plans, redesign team charters, scorecards, and other performance monitors and materials from consultants. The burden of collection of these documents should not exceed 2 hours at each site.

These documents will be reviewed by both members of the site visit team, and together with the completed analytic framework for each site, will contribute to the development of the detailed site profile.

3. Methods to Maximize Response Rates

Given the nature of the case study model, our sample will be limited by those sites which are willing to participate. We will recruit organizations that express an interest in participating. Participating organizations will be those who are activity involved in system redesign or have worked on it the recent past. The availability, or lack thereof, of resources to support the work they are doing, or have done, will undoubtedly be of interest to them so we suspect that there will be a high level of organizational support for participation. Individual staff members at participating organizations will be identified by the site-specific point of contact based on their participation in system redesign either at the strategic or operational level and again, we suspect given the nature of the project and the minimal burden of participation, there will be a high response rate. These conditions should also facilitate the provision of honest and valid data from study participants.

No person-specific reminders will be used, a reminder email will be sent to non-responders to encourage participation. A lack of response after the reminder email will be interpreted as a refusal to participate.

4. Tests of Procedures

The interview guide used in this study has been adapted from one used successfully by this research team in their evaluation of improvement capability grants in the VA Healthcare system. The improvement capability grant evaluation project considers a specific external resource used for system redesign as oppose to system redesign more broadly, so some of the questions have been adapted to account for this difference. Additionally, the BU project team has worked with the AHRQ Task Order Office to adapt the interview guide to capture system redesign resources and tools more broadly. It has also been reviewed by the Expert and Stakeholder Panel.

During data collection, items which are difficult to answer, unclear or present substantial response burden, or are otherwise problematic, will be noted by the interviewer with a description of concerns. Such items will be modified to deal with these concerns and will subsequently be reviewed.

5. Statistical Consultants

This project does not employ statistical methods. The following persons were consulted regarding the qualitative methods used in this project.

Name (Affiliation)	Telephone Number	Email
Michael Harrison (AHRQ)	301-427-1434	Michael.harrison@ahrq.hhs.gov
Carol VanDeusen Lukas	857-364-5685	Carol.vandeusenlukas@va.gov
Sally Holmes	857-364-2615	Sally.Holmes@va.gov
Martin Charns	857-364-4945	Martin.Charns@va.gov
Gouri Gupte	617-414-1426	gourig@bu.edu

7