Supporting Statement Part B -- Reducing waste and Inefficiency through Process Redesign Lean Implementation_06-03-09_clean

SUPPORTING STATEMENT

Part B

Reducing Waste and Inefficiency through Process Redesign: Lean/TPS Implementation

Version: June 3, 2009

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 3

3. Methods to Maximize Response Rates 7

4. Tests of Procedures 7

5. Statistical Consultants 8

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The purpose of this study is to provide an illustrative and informative picture of Lean/Toyota Production System (TPS) implementation in health care. This study is not intended to make population estimates from a representative sample of providers who use Lean/TPS methods. Therefore, we have a purposive sample. The cases will ensure diversity on the following characteristics:

• Location: large or small community; urban ↔ rural

• Type and size of hospital and/or clinic: academic and tertiary medical center ↔ community medical center

• Type of Lean/TPS work flow process: linear ↔ non-linear (circular)

• Experience with Lean/TPS: early adopter ↔ mid-range--recent or new adopter

• Analytic perspective and methods: retrospective ↔ prospective

The use of a purposive sampling strategy to provide descriptive information has three limitations with respect to representativeness and causation. First, we have not sampled with known probability from a complete universe of a well-defined population. Thus, we are unable to make inferences to the entire universe of hospitals that use Lean/TPS and we are unable to estimate the statistical reliability of our findings. Second, although we will not make inferences to this population, and will state this in our reports, some readers may choose to do so. In that case, to the extent that our findings differ from those that would have resulted from a probability sample of the complete universe, our conclusions might be biased. Third, because this is an observational study without an experimental or quasi-experimental design, we are unable to exclude alternative explanations for our findings with much certainty. Nevertheless, our approach is appropriate for the descriptive purpose of our study.

Through our acquisition of documents and qualitative interviews, we will obtain information about the internal studies that our participating sites have used to assess the value of their Lean/TPS projects. We expect that some of these internal studies will have used quasi-experimental designs, such as pre-post comparisons, to attribute their findings to the Lean/TPS project. In such cases, we will judge the validity and reliability of those finding when describing them in our reports.

2. Information Collection Procedures

Qualitative methods (the interviews via phone and in-person, document review, and digital diaries described in section A1 will be used to generate the rich description and examples required to develop the lessons and business case that focus on the implementation of Lean/TPS in diverse markets, organizations, and departments.

For the retrospective case studies, only document review and in-person interviews will be conducted. In contrast, the prospective case studies will use all 4 methods so that we can assess the implementation process in real time, before memory fades and perspectives change.

In Exhibit 4, we outline the data collection strategies for each type of case study.

Exhibit 4. Summary of Qualitative Methods

Type of study	Method
Retrospective	Documentation In-person interviews during site visit
Prospective	Documentation Digital Diaries Telephone interviews In-person interviews during site visits

Digital Diary. For prospective cases, the participating establishment will assign an onsite quality specialist or coordinator who will work closely with each department and Lean/TPS project. This onsite person will keep a “digital diary,” using a diary entry guide and a hand-held digital voice recorder to describe key aspects of the implementation process. The onsite coordinator will dictate his or her answers to these questions into the digital recorder and send the MP3 files to AIR researchers via e-mail. The onsite coordinator will make diary entries at least weekly, and possibly more frequently depending on the project stage and how the project is proceeding. Up to two individuals may complete the digital diaries depending on their knowledge and participation in onsite coordination of the Lean/TPS projects.

Telephone Interviews. For each prospective case, we will conduct telephone interviews with approximately 3 persons per department. The types of interviewees will depend on the specific department and projects. For example, in studies of radiology department projects, interviewees could include radiologists, radiology technicians, or nurses. For ambulatory practices, interviewees could include physicians, referring physicians, nurses, desk attendants, technologists, or technicians. For inpatient projects, the interviewees could include hospitalists, surgeons, and anesthesiologists if the effort involves surgery, transport personnel, nurses, and nurse managers. We may also add as interviewees one to two managers having oversight responsibility for the Lean/TPS projects. An email will be sent to all interviewees prior to the appointment and before data collection takes place (see Attachment D).

The telephone interviews will last approximately 30 minutes. Potential topics include current stage of implementation of redesign efforts, elements that are working well or are problematic, activities conducted to address problems, and the impact of redesign efforts and implementation on interviewees, their practice, and their patients.

In-Person Interviews. These interviews will be conducted during site visits. The first site visit for both retrospective and prospective studies will take place as soon as possible given OMB approval. Retrospective case studies will be conducted over 5 five days. This is necessary to collect all possible documents and metrics and to conduct all in-depth interviews at one time. The team will conduct in-person interviews with approximately 10 persons from each department (see discussion of “telephone interviews” for the types of individuals to be interviewed) and up to 5 other key system staff from the institution’s system design team, IT/IS, and senior management, including financial managers. Three departmental staff will be selected to be interviewed in-person, as well as over the telephone. These in-person interviews will last about 60 minutes, depending on the person’s role in the individual projects and redesign efforts.

Before scheduling the interviews during each site visit, an email will sent to all potential interviewees in order to schedule the interviews during the site visit at the most convenient time (See Attachment D).

Approximately 8 months into the prospective studies, the same teams will conduct a second 3-day site visit. Again, the team will conduct interviews with approximately 10 people from the departments and up to 5 other key system staff from the institution’s design team, IT/IS, and senior management. These interviews will last approximately 60 minutes.

We have developed a case study site visit topic guide to ensure consistent data collection and data management across site visit teams (see Attachment B). Procedures for the telephone interviews will be similar. The guide includes common topics, any site-specific topics, interview guides, and data management procedures. AIR’s IRB will approve all protocols, recruitment, and interview procedures before any contacts are made or any data are collected.

Interview guides have been developed based on the literature scan and on discussions among the project team, including the AHRQ Project Officer. Initially, the focus will be broad and interview questions will be open-ended, with the goal of obtaining the perspectives of multiple stakeholders. The guides include questions that need to be covered, as well as probes designed to ensure that all critical topics are addressed. As the participants or informants may differ among sites, the initial and follow-up questions that we ask each interviewee may differ. .Though the interviews will be tailored for the various informants, the average burden on each respondent will not exceed 60 minutes (1 hour)..

The freedom to tailor questions and probes to each respondent is the hallmark of the qualitative interviewing methodology. Because the interviewer does not adhere inflexibly to the written questions—by asking every question, using the exact written language, and in the exact sequence—the study findings are limited by two potential sources of bias. First, if the language and sequence of the questions are associated with the responses, our conclusions might also vary in an unknown way. Second, because there are no rigid categories, aggregating responses requires interpretation by the analyst and it is possible that one analyst’s interpretation may differ from another’s.

Nevertheless, the qualitative interviewing method offers some benefits that offset this potential bias. Because the interviewer is free to probe and discuss each response in depth, it is more likely that the responses will capture the nuances of the phenomenon being studied than will the responses to highly structured questions that cannot be adapted based on previous responses. Furthermore, the interviewers in qualitative studies are much more experienced with the topic than the interviewers who administer highly structured quantitative surveys. Thus, we often learn more from semi-structured qualitative interviews than from highly structured quantitative interviews. From the perspective of total error, in some instances qualitative data can be more accurate than quantitative data.

All qualitative data will be managed and analyzed using Atlas.ti, a qualitative data analysis software program. Atlas.ti can accommodate diverse types of qualitative data, including observation or interview notes “written-up” in Microsoft Word, electronic copies of reports (originals or scanned) or electronic report summaries, and digital diary (MP3) files.

We will define and apply a coding system for the notes from the in-depth interviews and digital diaries. This coding system will be applied using the qualitative software to facilitate the retrieval, summary, and synthesis of data. The data are coded by assigning labels to the units, clustering the codes into categories and hypothesizing about relationships among the categories. This process facilitates the identification of patterns and themes that explain the outcomes in different cases. We will compare the data obtained for each case study and will identify key similarities and differences by retrospective or prospective case, location, department, and redesign effort. We will employ a variety of qualitative techniques to draw conclusions from the data (e.g., noting patterns and themes, plausibility, relationships between variables, and finding intervening variables).

3. Methods to Maximize Response Rates

Because this is a purposive rather than a probability sample, we will recruit only those organizations that initially express an interest in participating. These organizations are actively involved with the implementation of Lean/TPS in health care settings. Information about the efficacy of their implementations and about the identification of barriers to success and about strategies to overcome these barriers will be invaluable for them. Accordingly, we anticipate strong organizational support and endorsement of this research. This support, combined with guarantees of confidentiality, should serve to encourage staff participation and should also to facilitate the provision of honest and valid data from study participants.

In addition, we are using digital diaries to reduce respondent burden. This should further enhance cooperation and response rates.

4. Tests of Procedures

Prior to data collection, we will submit the topic guides to 2-3 selected individuals from our participating organizations for review. These individuals will be employees of the case study organizations and will be familiar with Lean/TPS implementation, but will not be the key informants interviewed during data collection for the case studies. We will ask these individuals if there are any confusing questions or if different language should be employed. We will also ask if any key areas are missing from the topic guides. Our topic guides are currently based on guides used in previous redesign projects.

During data collection, items which are difficult to answer, unclear, present substantial response burden, or are otherwise problematic are noted by the interviewer with a description of the concerns. The items are modified to deal with these concerns and subsequently re-reviewed.

5. Statistical Consultants

This project does not employ statistical methods. The following persons were consulted regarding the qualitative methods used in this project.

Name (Affiliation)	Telephone Number	Email
Michael Harrison (AHRQ)	301-427-1434	Michael.Harrison@ahrq.hhs.gov
Dina Moss (AHRQ)		Dina.Moss@ahrq.hhs.gov
Kristin Carman (AIR)	202-403-5090	kcarman@air.org
Steven Garfinkel (AIR)	919-918-2306	sgarfinkel@air.org
Kelly Devers (VCU)	804-828-5509	kjdevers@vcu.edu
Lauren Smeeding (AIR)	202-403-5067	lsmeeding@air.org

7