ATTACHMENT B--Logic Model of Post-TeamSTEPPS Master Training Patient Safety Activities and Outcomes by Participant Role

Logic Model of Participant Usage and Perceived Outcomes

As a preliminary step in the refinement of our evaluation strategy, AIR developed a logic model, presented in Exhibit 1 that specifies potential indicators of participant usage in the form of post-training activities. In addition, AIR identified potential outcomes resulting from post-training activities associated with the National Implementation program. This model is founded on the assumption that participants may have different roles and responsibilities with regard to patient safety and, therefore, different uses for the TeamSTEPPS tools and strategies or training material. Each participant may have served in one or more of the roles depicted in this model.

Exhibit 1 – Logic Model of Post-Training Activities and Outcomes by Participant Type

Role of Participant		Potential Activities Post-TeamSTEPPS Master Training		Potential Outcomes
Implementers		Implement TeamSTEPPS to improve teamwork and communication in your organization Use TeamSTEPPS tools and strategies to evaluate hospital investigations of reported events Use TeamSTEPPS tools and strategies to improve organizational patient safety outcomes		Improved (more effective) implementation of patient safety interventions Improved patient safety Improved patient safety culture Improved communications/interactions between hospitals, State health depts, and QIOs on patient safety issues
Trainers / Facilitators		Disseminate TeamSTEPPS information, tools, and strategies to internal staff and to staff at non-participating hospitals, State health dept, &/or QIO Train others on TeamSTEPPS concepts, tools, and strategies Facilitate use or implementation of TeamSTEPPS in hospitals/State health depts/QIOs/etc. by serving as a coach or consultant to implementers		Increased awareness and understanding of TeamSTEPPS within &/or across organizations Improved communications between hospitals, State health depts, and QIOs on patient safety issues

For example:

• Implementers will typically be represented by hospital participants who are more likely to be implementing TeamSTEPPS tools and strategies on a daily basis and will be more likely to affect specific work processes being conducted within an organization. As a result, hospital participants are likely to have a focused and specific impact within that organization only; and

• Trainers and facilitators are more likely to be comprised of QIO personnel who would be expected to have both an in-depth and broad impact assuming that they have supported the training, implementation and use of TeamSTEPPS while assisting a particular organization in their patient safety activities, as well as to provide general patient safety guidance to a large number of organizations.

Several moderating factors also may have an effect on the impact of an initiative like TeamSTEPPS, such as (1) organizational and environmental constraints/facilitators (e.g., leadership buy-in, funding and resource support) and (2) fit of the material to the organization (e.g., ease of transfer).

AIR will work closely with AHRQ to ensure that the assumptions underlying this model are accurate prior to developing any data collection instruments for this evaluation. Our goal for the training evaluation will be to demonstrate the impact of the TeamSTEPPS National Implementation program on training participant reactions, learning, post-training behavior, and program outcomes.

Kirkpatrick’s Model of Training Evaluation

Given the focus of our evaluation, our research design is founded on Kirkpatrick’s model of evaluation. Kirkpatrick presented four sequential levels of training evaluation; each level is important and has an impact on the next level (Kirkpatrick, 1994; Kirkpatrick & Kirkpatrick, 2006).

Level 1, the most basic level, measures participant reactions to the training. Results usually illustrate how much the participants “liked” the training (i.e., affective reactions). Among its many uses, Level 1 data are typically useful for modifying course content and delivery.

Level 2 evaluates the extent to which “learning” has occurred as a result of training. More specifically, it can measure changes in trainee attitudes, improvements in knowledge, or increases in skill as a result of the training program. Data at this level of evaluation are often collected with post-training assessments and can be used to revise the course content; provide feedback to the participants on their learning; and build credibility (e.g., demonstrate that learning is taking place, the course is making a difference (McCain, 2005; Phillips, 1991)).

Level 3 measures the extent to which behavior (i.e., job performance) has changed as a result of the training. In other words, is there transfer to the job and work environment? Level 3 evaluation measures often include items pertaining to “use” and “environment” (McCain, 2005). Kirkpatrick stresses the importance of a favorable work climate for successful “application” of training on-the-job.

Finally, Level 4 evaluation measures the benefits or impact of the training on the organization. “Results” might include, for example, improved patient safety, improved processes and/or interventions, and improved communications, to name a few.

A secondary model for assessing the impact of large-scale initiatives like TeamSTEPPS is the the CIPP, or Context, Input, Process, Product approach, developed by Stufflebeam (1983). The primary objective of this model is to assess the impact of any educational initiative by asking a series of critical questions about the context for training, the inputs such trainee characteristics, the process used to deliver training, and the outcomes of the training for all parties involved. Typical questions asked might include the following:

 

Context

• Is there a need for the course?

• Is the course relevant to job needs?

Inputs

• What is the motivation of students?

• What is the students’ existing knowledge ?

 Process

• Is knowledge only transferred to students, or do they use and apply it?

• Are there any problems which students face in using/applying the knowledge and skills?

• Is the teaching and learning process continuously evaluated?

• Is teaching and learning affected by practical/institutional problems?

 

Product

• What are the students’ KSA levels after the course?

• How do students use what they have learned?

These two models can be used to create an integrated approach for evaluating the National Implementation of TeamSTEPPS program’s impact and spread. For example, we can develop a model that examines the context and inputs for TeamSTEPPS training and then examines the core Level 3 transfer of training behaviors and Level 4 outcomes of training participation. This integrated model for evaluation would focus on developing an understanding of why TeamSTEPPS master training was needed and what the training participants knew about teamwork prior to training, as well as how they applied the lessons learned during training to their organizations or those that they support and what the eventual outcomes of their TeamSTEPPS participation were.

1

American Institutes for Research^®