TS Online Supporting Statement Part B-FINAL

SUPPORTING STATEMENT

Part B

TeamSTEPPS 2.0 Online Master Trainer Course

Version: March 10, 2014

Agency for Healthcare Research and Quality

Table of Contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent Universe and Sampling Methods 3

2. Information Collection Procedures 3

3. Methods to Maximize Response Rates 4

4. Tests of Procedures 5

5. Statistical Consultants 5

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The TeamSTEPPS 2.0 Online Post-Training Survey will be distributed to all individuals who have participated in the TeamSTEPPS 2.0 Online Master Trainer program. It is anticipated that 3,000 participants will be trained during 2014–2016. No sampling strategy will be employed, as the Agency for Healthcare Research and Quality (AHRQ) seeks to obtain feedback from everyone who has participated in the program (i.e., a census survey). A 75 percent response rate or higher is anticipated based on the positive relationship established between AHRQ and the training participants.

Some of the limitations of this study, which will also be noted in the final report, include the following:

• Retrospective analysis of post-training experiences. This is not an empirical study; therefore, neither causality nor generalizability can be established;

• Survey nonresponse, which can limit AHRQ’s understanding of the concerns and issues participants encountered when trying to apply TeamSTEPPS tools and concepts in their home organizations or organizations that they support;

• Lack of access to contact information for nonrespondents; and

• Inability to survey participants who have changed jobs since TeamSTEPPS 2.0 Online Master Training. The survey will be administered six months after training.

Despite the limitations in this study, the data collected via the survey will provide valuable information that will help refine the TeamSTEPPS 2.0 Online Master Training program. By understanding the factors that facilitate or inhibit the use of tools or the spread of knowledge, AHRQ will better understand the needs of preparing individuals to implement TeamSTEPPS in health care settings and will be better prepared to address their future concerns, issues, and needs for improving patient safety.

2. Information Collection Procedures

The TeamSTEPPS 2.0 Online Post-Training Survey will be administered to all individuals who complete the TeamSTEPPS 2.0 Online Master Trainer program using the Web-based questionnaire included in Attachment A. The TeamSTEPPS 2.0 Online Post-Training Survey will be sent to all training participants via email. All correspondence to training participants is presented in Attachments C–F. These correspondences contain a hyperlink to enable easy access to the online questionnaire.

The TeamSTEPPS 2.0 Online Post-Training Survey will be administered six months after the completion of training. The survey will be accessible to training participants 24 hours a day, for a total of 30 days. On entrance into the questionnaire, respondents will view an introduction page that explains the questionnaire objectives and stresses the importance of participation. Respondents will be able to easily respond to the questionnaire items by clicking on precoded options for closed-ended items and typing in boxes for any open-ended items.

Completed responses to the questionnaires will be backed up daily onto Reingold’s dedicated data collection server and will also be printed and stored in a locked cabinet. Data responses will be checked visually by researchers and analysts on a regular basis to assure that data are entered appropriately into the database.

Following data collection, training participant questionnaire responses will be compiled and assessed formally for data quality to produce a finalized database for statistical analyses. Incomplete response data poses a substantial threat to confident interpretation of the study results. The general approach to handling incomplete response data is to salvage as much data as possible using multiple techniques for examining patterns of missing data. Given the scope of the training participant questionnaire, Reingold will review questionnaire items with a substantial proportion of omitted responses. The precise cutoff percentage is typically chosen once the distribution of missing data has been established. It will be determined whether responses are missing in a manner that relates to other observable values. If data are determined to be missing in a manner that affects the interpretability of the responses, descriptive statistics and point estimates of relations among variables may be adjusted to account for missing data. This may be accomplished by using multiple imputation and full information maximum likelihood estimation techniques.

3. Methods to Maximize Response Rates

To effectively bolster the response rate, time-staggered notices of the opportunity to participate in the questionnaire will be sent via email. If at any point an invited respondent refuses to participate, he or she will not be contacted again through any means.

During the data collection period, invited respondents who have yet to respond to the training participant questionnaire will be contacted via email reminding them of the opportunity to participate and the importance of their feedback regarding the training program. The re-contact notice sent via email will provide the hyperlink to access the questionnaire, the estimated time (in minutes) it will take to respond, the impending deadline for submission of their responses, and additional information regarding privacy of responses and confidentiality of personal information. The re-contact notice is provided in Attachment D.

It is important to note that the most common way to ensure an acceptable response rate is to provide introductory information through advance notice from a known and respected source—in this case, AHRQ. For the purposes of this study, the AHRQ Project Officer will distribute an email to all invited respondents to provide advance notice of the study and the importance of participation (see Attachment C).

4. Tests of Procedures

A prior version of the TeamSTEPPS 2.0 Online Post-Training Survey was piloted. The pilot test involved: (1) a series of two to three cognitive laboratory interviews with AHRQ staff members and (2) a field test of the survey with a subsample of no more than nine potential respondents. The results were used to refine the questionnaire prior to field testing.

5. Statistical Consultants

IMPAQ International, LLC will serve as the primary consultants for statistical aspects of the design and analysis of the Web questionnaire data. Dr. David P. Baker, Chief Research Officer at IMPAQ, is the primary point of contact for statistical design and analyses. He can be reached at dbaker@impaqint.com or 443-539-9773.