SUPPORTING STATEMENT
Part A
“Implementation of TeamSTEPPS in Primary Care Settings (ITS-PC)”
Version: April 12, 2017
Agency of Healthcare Research and Quality
Table of Contents
1. Circumstances That Make the Collection of Information Necessary 3
2. Purpose and Use of Information 5
3. Use of Improved Information Technology 5
4. Efforts to Identify Duplication 6
5. Involvement of Small Entities 6
6. Consequences if Information Collected Less Frequently 6
8. Federal Register Notice and Outside Consultations 6
9. Payments/Gifts to Respondents 7
10. Assurance of Confidentiality 7
11. Questions of a Sensitive Nature 7
12. Estimates of Annualized Burden Hours and Costs 8
13. Estimates of Annualized Respondent Capital and Maintenance Costs 9
14. Estimates of Annualized Cost to the Government 9
16. Time Schedule, Publication and Analysis Plans 10
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, the Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. Research that develops and presents scientific evidence regarding all aspects of health care;
2. The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. Initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
As part of its effort to fulfill its mission goals, AHRQ, in collaboration with the Department of Defense’s (DoD) Tricare Management Activity (TMA), developed TeamSTEPPS® (aka, Team Strategies and Tools for Enhancing Performance and Patient Safety) to provide an evidence-based suite of tools and strategies for training teamwork-based patient safety to health care professionals. TeamSTEPPS includes multiple toolkits which are all tied to or are variants of the core curriculum. In addition to the core curriculum, TeamSTEPPS resources have been developed for primary care, rapid response systems, long-term care, and patients with limited English proficiency.
The main objective of the TeamSTEPPS program is to improve patient safety by training health care staff in various teamwork, communication, and patient safety concepts, tools, and techniques and ultimately helping to build national capacity for supporting teamwork-based patient safety efforts in health care organizations. Since 2007, AHRQ’s National Implementation Program has produced (and continues to produce) Master Trainers who have stimulated the use and adoption of TeamSTEPPS in health care delivery systems. These individuals were trained using the TeamSTEPPS core curriculum at regional training centers across the U.S. AHRQ has also provided technical assistance and consultation on implementing TeamSTEPPS and has developed various channels of learning (e.g., user networks, various educational venues) for continued support and the improvement of teamwork in health care. Since the inception of the National Implementation Program, AHRQ has trained more than 8,000 participants to serve as TeamSTEPPS Master Trainers.
Given the success of the National Implementation Program, AHRQ launched an effort to provide TeamSTEPPS training to primary care health professionals using the TeamSTEPPS in Primary Care version of the curriculum, which is now referred to as “TeamSTEPPS for Office-Based Care,” to align with updated terminology used in the field. Most of the participants in the current National Implementation Program’s training come from hospital settings, because the TeamSTEPPS core curriculum is most aligned with that context. Under this new initiative, primary care practice facilitators will be trained through online instruction. Upon completion of the course, these individuals will be Master Trainers who will (a) train the staff at primary care practices, and (b) implement or support the implementation of TeamSTEPPS tools and strategies in primary care practices.
As part of this initiative, AHRQ seeks to conduct an evaluation of the TeamSTEPPS for Office-Based Care training program. This evaluation seeks to understand the effectiveness of the TeamSTEPPS for Office-Based Care training and how trained practice facilitators implement TeamSTEPPS in primary care practices.
This research has the following goals:
1) Conduct a formative assessment of the TeamSTEPPS for Office-Based Care training program to determine what revisions and improvement should be made to the training and how it is delivered, and
2) Identify how trained participants use and implement the TeamSTEPPS tools and resources in primary care settings.
To achieve the goals of this project, AHRQ will train primary care practice facilitators using the TeamSTEPPS for Office-Based Care training curriculum. Primary care practice facilitators may voluntarily sign up for this free, AHRQ sponsored training. Training will be delivered through online instruction. Training will cover the core TeamSTEPPS tools and strategies that can be implemented in primary care, as well as coaching, organizational change, and implementation science. Practice facilitators, who complete the training, will be surveyed six months post-training.
The TeamSTEPPS for Office-Based Care Post-Training Survey is an online instrument that will be administered to all primary care practice facilitators who complete the TeamSTEPPS for Office-Based Care training. The survey will be administered six months after participants complete training. The TeamSTEPPS for Office-Based Care Post-Training Survey can be found in Attachment A.
This study is being conducted by AHRQ through its contractor, the Health Research & Educational Trust (HRET) and HRET’s subcontractor, IMPAQ International (IMPAQ), pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).
This is a continuation of data collection for the purpose of conducting an evaluation of the TeamSTEPPS for Office-Based Care training program. The evaluation is formative in nature as AHRQ seeks information to improve the delivery of the online training.
To conduct the evaluation, the TeamSTEPPS for Office-Based Care Post-Training Survey will be administered to all individuals who complete the TeamSTEPPS for Office-Based Care training six months after training. The survey assesses the degree to which participants felt prepared by the training and what they did to implement TeamSTEPPS in primary care practices. Specifically, participants will be asked about their reasons for participating in the program; the degree to which they feel the training prepared them to train others in and use TeamSTEPPS in the primary care setting; what tools they have implemented in primary care practices; and resulting changes they have observed in the delivery of care.
The TeamSTEPPS for Office-Based Care Post-Training Survey will be administered via the Web to participants.
In order to reduce respondent burden, the training participant questionnaire will be administered via the Web. Participant information acquired by HRET and its partner Reingold, Inc. (Reingold) when participants enroll in the TeamSTEPPS for Office-Based Care training program will be used to develop the distribution lists. Each potential respondent will receive up to five e-mail communications to encourage participation (i.e., an advance notice of the questionnaire, an initial invitation to complete the questionnaire, and three follow-up e-mails to remind respondents to complete the questionnaire).
Using an online system for data collection, rather than administering a paper-based questionnaire, makes completing and submitting the questionnaire less time-consuming for respondents. Any skip patterns included in the questionnaire (i.e., questions that are appropriate only for a subset of the respondents) will be automatically programmed into the Web-based form of the questionnaire, thereby eliminating any confusion during questionnaire completion. In addition, the contractors can also ensure that important items are not inadvertently skipped or ignored by setting software requirements to ensure proper completion of questionnaires based on specific respondent selections.
AHRQ is aware of two other evaluations of the TeamSTEPPS suite of training programs, each requiring an Office of Management and Budget (OMB) clearance package. Each project evaluates a different TeamSTEPPS training program and collects data from distinct populations of individuals who participate in training. These evaluations are described below.
Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program. This effort provides in-person training to participants using the TeamSTEPPS 2.0 core curriculum. AHRQ is administering a version of the post-training survey to these participants. The survey used for this effort has obtained initial and renewed OMB clearance (#0935-0170).
TeamSTEPPS 2.0 Online Master Training. This effort provides online training to participants using a Web-based version of the TeamSTEPPS 2.0 core curriculum. AHRQ proposes to continue to conduct a formative evaluation of the TeamSTEPPS 2.0 Online Master Trainer program using a version of the post-training survey tailored to this group of training participants. The survey obtained OMB approval (#0935-0224) and will be undergoing review for OMB renewal.
Besides the two evaluation efforts listed above, AHRQ proposes to continue to collect similar data on the TeamSTEPPS for Office-Based Care training program. This effort (as described in this renewal submission) will provide information on the activities in which participants of this training program engage post-training, as well as the corresponding outcomes. The survey approved for the Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program effort has been slightly modified to align with this program’s target participants and curriculum.
The information collected may involve small entities, as primary care practice facilitators may work for very large or small primary care practices. Individuals from small practices will be required to complete the same number of items as those from larger practices. For this study, only items that provide critical information for answering the study questions will be included.
This is a one-time collection to answer specific questions about the TeamSTEPPS for Office-Based Care Training program.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on May 5, 2017 on page 21230 for 60 days (see Attachment C). No public comments were received.
AHRQ consulted with the American Board of Internal Medicine to determine if any secondary data sources were available to answer the questions for this study. None were identified.
No remuneration of respondents is planned.
Individuals will be assured of the confidentiality of their replies is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify any individual respondent will not be disclosed outside of the Department of Health and Human Services unless the respondent has consented to that disclosure.
Information that can directly identify the respondent, such as a name and/or social security number, will not be collected. Only basic demographic information related to the individual’s organization and position will be collected for the purpose of describing the respondents. Participation will be voluntary and participants will be informed that their responses will be confidential. The following statement of confidentiality will appear on the initial screen of the Web-based survey and in email correspondence:
“This survey is authorized under 42 U.S.C. 299a. The confidentiality of your survey responses is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed outside of the Department of Health and Human Services without your consent. If needed, AHRQ may contact you in the future to request additional information about your experiences with TeamSTEPPS.”
HRET’s subcontractor, IMPAQ, will collect all information. Response data will be stored on password-protected secure servers at IMPAQ. Only members of the IMPAQ project team will have access to the password-protected secure server. Only aggregated data will be included in reports submitted to AHRQ. A data file containing individual-level responses to the survey will be delivered to the AHRQ Task Order Officer at the end of the data collection effort.
Questionnaire items do not require respondents to provide information of a sensitive nature as defined by OMB and DHHS or to provide information such as social security numbers or Medicare/Medicaid numbers. The TeamSTEPPS for Office-Based Care Post-Training Survey includes an introduction that addresses aspects of informed consent such as a description of the research objectives, a discussion of the importance of their input and experiences, details concerning how the data will be used, and confidentiality of personal information. The introduction will be positioned at the beginning of the questionnaire. Continuation to complete the questionnaire will indicate the respondent’s consent.
Although written consent will not be required, all respondents will be informed about the nature of the study and that their participation is voluntary. There are no known consequences of participation, and all confidentiality procedures will be described.
Exhibit 1 shows the estimated annualized burden hours for the respondent’s time to participate in the study. The TeamSTEPPS for Office-Based Care Post-Training Survey will be completed by approximately 600 individuals per year. We estimate that each respondent will require 20 minutes to complete the survey. The total annualized burden is estimated to be 200 hours.
Exhibit 2 shows the estimated annualized cost burden based on the respondents’ time to participate in the study. The total cost burden is estimated to be $24,944.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
TeamSTEPPS for Office-Based Care Post-Training Survey |
600 |
1 |
20/60 |
200 |
Total |
600 |
NA |
NA |
200 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
TeamSTEPPS for Office-Based Care Post-training Survey |
600 |
200 |
$96.54 |
$19,308 |
Total |
600 |
200 |
$96.54 |
$19,308 |
*Based on the mean hourly wage for Family and General Practitioners (29-1062) presented in the National Compensation Survey: Occupational Wages in the United States, May 2016, U.S. Department of Labor, Bureau of Labor Statistics (https://www.bls.gov/oes/current/oes_nat.htm).
There are no direct costs to respondents other than their time to participate in the study.
The total contractor cost to the government to conduct the one-time questionnaire, as well as to analyze and present all results is estimated to be $40,143. As shown in Exhibit 3a, this amount includes costs for collecting the data ($24,148), analyzing the data ($11,687), and reporting the findings ($4,301).
Exhibit 3a. Estimated Annualized Cost
Cost Component |
Annualized Cost |
Data Collection Activities |
$24,148 |
Data Processing and Analysis |
$11,687 |
Publication of Results |
$4,301 |
Total |
$40,143 |
A Health Communications Specialist (the Project Officer) will be responsible for project management and oversight. This will include oversight of the data collection approach and review of interim and final reports of summarized results. The estimated cost to the Federal Government for these activities is provided in Exhibit 3b. The average annual salary for the GS-13, Step 4 level is $104,275. Federal salary information based on the 2017 OPM Pay Schedule for the Washington/DC area is available on the OPM website at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2017/DCB.pdf.
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Average Annual Salary |
Estimated Hours Per Year |
Annual Cost |
Total Cost* |
Data Collection Oversight |
Health Communications Specialist (GS-13, Step 4) |
$104,275 |
6 |
$300.78 |
$676.76 |
Review of Interim and Final Results |
Health Communications Specialist (GS-13, Step 4) |
$104,275 |
4 |
$200.52 |
$451.17 |
Total |
$501.30 |
$1,127,93 |
|||
*Total cost based on the 27-month period of performance.
The estimated total annualized cost for this activity is $40,644. This cost includes annual contractor costs ($40,143) and Federal personnel costs ($501).
This is a continuation of data collection for the purpose of conducting an evaluation of the TeamSTEPPS for Office-Based Care training program. The prior hour burden was based on a pilot version of the training program. Based on its success, a larger number of training opportunities are being made available. Therefore, a larger number of trainees will respond to the survey.
The time schedule for the data collection, data analysis, and final report preparation is presented in Exhibit 4.
Exhibit 5. Timeframe for data collection, analysis, and preparation of final report
Data Collection and Analysis |
Timeframes |
Administer training participant questionnaire |
Immediately upon OMB approval |
Analyze data |
60 days from end of data collection |
Prepare final report |
90 days from end of data analysis |
IMPAQ will analyze the survey data to identify trends in usage of the TeamSTEPPS for Office-Based Care curriculum, as well as the perceived impact of the program on organizational outcomes. To that end, a three-phase analysis is proposed. That analysis includes (1) ensuring the quality of the data collected, (2) conducting descriptive analyses, and (3) conducting comparisons of specific Master Trainer types and cohorts. These phases of analyses are described below.
To ensure maximum integrity of the results, several data screening and checking procedures will be conducted (Tabachnick & Fidell, 1996). Specifically, data quality checks will be performed by searching for deviant response ranges, anomalous response patterns, excessive missing data, extreme outliers, and highly skewed or irregular distributions. From these analyses, faulty data or data of poor measurement quality will be flagged, corrected, and/or eliminated. For example, excessive missing data is an indicator of poor data quality. Respondents who fail to respond to more than 10 percent of the protocol questions on the Web survey will be identified; their pattern of responses will be reviewed more carefully to determine if, for example, the respondent’s data should be eliminated from the analyses. Obviously any strategies that result in the elimination of data would first be discussed with AHRQ representatives and then fully documented in the final report.
For the descriptive analyses, HRET will employ the following approach: (1) compute a number of descriptive statistics for each variable measured by the survey; (2) develop early warning data protocols (specific statistical analyses to indicate significant variability, low response rates, or error in the data); (3) conduct item analyses; and (4) conduct comprehensive group and subgroup analyses.
HRET will calculate frequency distributions, means, and standard deviations for each closed-ended item included in the survey and combinations of related items that focus on a particular variable or issue. In addition, these statistics will be calculated for each subgroup represented in the sample (e.g., year of training attendance) and conduct analyses to identify subgroup differences. Frequency distributions will show the percentage of people who responded to each response option for each item included in the protocol. Means and standard deviations will be used to examine the relative importance of different items and item combinations that measure specific issues associated with each survey. Finally, standard deviations will be used to examine the level of agreement among respondents regarding issues that are identified as important.
A few items included will be open-ended in nature as a means of following up on closed-ended items to obtain richer detail on unique activities being conducted post-training. Individual responses to the open-ended items will be recorded and compiled; these responses will be examined for any themes or patterns of interest. If appropriate, codes will be defined based on the themes identified and the open-ended responses would be coded into closed-ended categories, which would then be tabulated.
The data gathered from the survey will allow us to determine differences between roles that Master Trainers may fulfill. For example, comparisons of each TeamSTEPPS tool and strategy will be conducted based on the ratings of usefulness by averaging the usefulness ratings of each tool across all participants and then conduct t-tests to assess the magnitude of any subgroup differences. In addition to conducting comparisons of tool usefulness, comparisons of tool usage and perceived impact will also be conducted, and variations in these characteristics will be analyzed by training participant type. To identify the magnitude of any differences by training participant type, t-tests will be conducted. For example, the analyses will identify how useful a specific tool is for direct users/implementers as opposed to those Master Trainers whose roles are primarily to support others in the implementation or training of TeamSTEPPS.
AHRQ does not seek this exemption.
Attachment A -- TeamSTEPPS for Office-Based Care Post-Training Survey
Attachment B – Email Notifications
Attachment C -- Federal Register Notice
| File Type | application/msword |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| Last Modified By | SYSTEM |
| File Modified | 2017-07-27 |
| File Created | 2017-07-27 |