Supporting Statement Part B

SUPPORTING STATEMENT

Part B

AHRQ Safety Program for Improving Surgical Care and Recovery

Version: 8/10/2020

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 6

3. Methods to Maximize Response Rates 8

4. Tests of Procedures 9

5. Statistical Consultants 9

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The purposes of the surveys and assessments addressed in Statement A are to provide feedback to participating hospitals to help with their improvement efforts and to compare results before and after project implementation to evaluate the impact of the overall quality improvement effort. Also, they will help the national project team tailor the technical assistance to the needs of the hospitals. Since the goals of this project are for quality improvement in participating hospitals and not to produce national estimates, purposive sampling of hospitals will be done.

Surveys will be administered pre and post intervention in a variety of participating hospitals that vary by type and size and geographic location within the U.S. Data will be collected from 38 hospitals participating in Cohort 3B, as well as any hospital that are recruited to participate in Cohort 4. The types of surgery will vary by cohort and are anticipated to include colorectal, orthopedic, gynecologic, and emergency general surgery. All hospitals in the US are eligible to participate in the quality improvement program. JHU and American College of Surgeons will continue to recruit hospitals from their extensive contact list that includes representatives from all US hospitals.

The project aims to improve patient outcomes. As noted in Supporting Statement A, clinical outcomes will be available from data extracts from an existing ongoing American College of Surgeons registry. Hospitals included in the analysis must have started data collection in the first two months of their cohort and entered cases through at least month eleven and/or twelve of their cohort. Hospitals will not be excluded if cases were not entered each month through the end of their cohort.

Safety Culture Survey. For the next and final phase of this project, the respondents will be perioperative staff members at hospitals participating in the project. The number of staff surveyed will vary based on hospital size but will average about 50 staff per hospital. Hospitals will be asked to form a list of all perioperative staff who should be included in the survey, and based on experience with cohort 1, approximately 10% of the 200 hospitals, or 20 hospitals, will elect to participate in this aspect of the project. The pre-implementation safety culture survey will be administered at the beginning of cohort 4. Table 1 displays the estimated number of surveys to be administered in the remainder of the project.

Table 1. Hospital staff surveyed pre-implementation for Cohort 4

Surgical Service Line	Number of hospitals	Expected responses per hospital	Number of staff surveyed pre-implementation
Emergency General Surgery And Others (Cohort 4)	20	50	1,000

Estimated sample size	10,000
Estimated response rate	10%
Estimated total responses	1,000

Patient Experience Survey. For this project, the patient frame is patients who have undergone specific surgical procedures including colorectal surgery, hip and knee replacement, and gynecology surgery. Using data from previous cohorts, we anticipate that 3,268 patients will be surveyed over a 3-month post-implementation data collection period. Table 2 displays the estimated number of surveys to be administered for Cohort 3B.

Table 2. Patients surveyed post-implementation for Cohort 3B

Surgical Service Line (Cohort)	Number of hospitals	Number of patients surveyed post-implementation	Expected responses
Cohort 3B	38	3,268	980

Total sample size	3268
Estimated response rate	30%
Total responses	980

Readiness and Implementation Assessments. A pre-and post-assessment will be administered with the hospital project leads (e.g. one physician, one nurse). Assuming an average of 2 staff being part of each pre- and post- semi structured assessment per hospital, table 3 displays the estimated number of surveys to be administered.

Table 3. Hospital staff surveyed pre and post implementation for Cohort 4

Surgical Service Line (Cohort 4)	Number of hospitals	Number of staff surveyed pre-implementation (readiness assessment)	Number of staff surveyed post-implementation (implementation assessment)	Number of staff surveyed pre and post
Emergency General Surgery and Others (Cohort 4)	190	380	380	760

Total sample size	760
Estimated response rate	90%
Total responses	684

Site visits. Six site visits will be conducted during this period. No site visits will be completed during the first cohort. Assuming an average of 3 staff being a part of each site visit, about 18 staff would be part of the site visit that will take 4 hours to complete.

Table 4. Hospital staff surveyed in site visits for Cohort 4

Site visits	# of hospitals chosen for site visit	# of staff in site visit per hospital	# of staff in site visit per cohort
Cohort 4	6	3	18

Total sample size	18
Estimated response rate	100%
Total responses	18

2. Information Collection Procedures

Important note: Since this project is a demonstration/implementation project a probability sample is not practical and will not be conducted for any of our surveys or assessments.

Safety culture survey. To successfully measure perioperative patient safety culture in this project, we will use the Ambulatory Surgery Center Survey on Patient Safety Culture (ASC-SOPS), adapted for the inpatient surgery setting to collect safety culture measures from participating hospitals. The ASC-SOPS provides an efficient way to assess aspects of patient safety culture most pertinent in surgical services, offers more detailed mapping of perioperative provider and staff roles, and more clearly defines the parameters of the unit under investigation.

The recommendation to prepare for and participate in the safety culture survey will be included in the enrollment procedures. More information about the survey and how to participate will be discussed as open dialogue during the readiness assessment semi-structured interview, and participating hospitals will be identified. Following the interview, project leads from each participating hospital will receive an email from the national team describing the minimal steps for participating in the safety culture survey. Project leads will be given ideas and recommendations on how to communicate this survey to their staff to encourage participation.

The following are the safety culture composites categories that the Ambulatory Surgery Center Survey on Patient Safety Culture are intended to measure:

• Communication About Patient Information 

• Communication Openness 

• Staffing, Work Pressure, and Pace 

• Teamwork 

• Staff Training 

• Organizational Learning – Continuous Improvement 

• Response to Mistakes 

• Management Support for Patient Safety 

• Near-Miss Documentation 

• Overall Patient Safety Rating 

• Communication in the Surgery/Procedure Room 

Hospital staff that will be a part of the program will be asked to anonymously complete the adapted ASC-SOPS at the beginning of cohort 4. The hospital’s project team will receive their survey results and then debrief their staff on their safety culture and identify opportunities for further improvement. Participating hospitals will promote awareness of the survey among their staff, , encourage staff to complete the survey and provide staff time to do so, and organize their local debrief of the reports of their hospital’s results. The national project team will analyze the data and send a report to the participating hospitals. A hospital that has less than six survey responders, will not be included in the reports or sent reports to debrief.

Patient experience survey data collection will include these steps:

• Mailing the surveys to the individuals sampled.

• Receiving completed surveys by mail. Returned surveys will be accepted throughout the data collection period.

• Mailing thank you/reminder postcards to nonrespondents. A thank you/reminder postcard will be mailed to all sampled individuals about 7-10 calendar days after the initial survey is mailed.

• Mailing reminder survey packet to nonrespondents. A second survey will be mailed to sample members who did not return the first survey. This second survey will be mailed to nonrespondents about 3-4 weeks after the first survey is mailed.

Readiness and Implementation assessment. The readiness assessment is a part of every hospital’s enrollment into each cohort. The implementation assessment is a part of every hospital’s sign out of the program at the end of each cohort. For the readiness assessment, the national team will send an email encouraging the hospitals to sign up for their 1-hour phone call with a member from the national team. For the implementation assessment, the national project team will send an email encouraging the hospitals to complete a survey through the Qualtrics platform. Hospitals that do not complete the survey will be sent a friendly reminder to complete the survey.

The Readiness assessment will be conducted as a semi-structured qualitative interview with hospital project leads (e.g. one physician, one nurse). The readiness assessment will help identify which, if any, technical components of the enhanced surgical care and recovery intervention already exist at the hospital, project management and resources, clinician engagement, leadership engagement and potential barriers and facilitators to implementation. The implementation assessment will evaluate what elements of the enhanced recovery practices have been adopted, resources invested, team participation, major barriers (e.g., medications, equipment, trained personnel), and leadership participation. These assessments will help identify training needs of hospitals and inform the national team’s approach. In addition, the results will inform the national team’s understanding of local adaptations of the intervention and the degree to which intervention fidelity impacts changes in outcomes.

Site visits. Virtual semi-structured site visits will be conducted at a subset of participating hospitals. Findings will help inform the national project implementation strategy. Information from these visits will be critical in understanding if and how team and/or leadership issues may affect implementation of enhanced surgical care and recovery practices, including how this may differ across surgical service lines. Interviews will help uncover misalignments in role clarity, needed time and resources, best practices, and potential enablers of and barriers to enhanced surgical care and recovery implementation. Site visits will be conducted at six hospitals for cohort 4, and each will be approximately 4 hours long. The types of hospital personnel anticipated to be involved in part or all of the site visit include senior leadership, perioperative leadership, and patient safety and quality staff. Participating hospitals will receive a structured debriefing and brief summary report at the conclusion of the visit.

3. Methods to Maximize Response Rates

As described above, surveys and assessments will be administered pre-and post-intervention depending of the survey or assessment. Response rates are discussed above in statement B, section 1. We have the following processes in place to mitigate loss of participation and increase response rate for all surveys and assessments:

Safety culture survey. We are not offering cash remuneration for respondents; however, the national team will provide the hospitals’ project leads with training and tools to promote awareness of the survey among their staff, encourage staff to complete the survey and provide staff time to do so. The national team will also encourage the project leads to remind staff to check their spam folder for the email to complete the survey. Finally, for participating hospitals the national team will send the hospital’s project leads updates on their response rates to encourage their staff to complete the survey.

Patient experience survey. We are not offering cash remuneration for respondents; however, the national team will send a second survey packet to those sample patients who do not respond to the first survey.

Readiness and Implementation assessment. We are not offering cash remuneration for respondents; however, each assessment is a part of every hospital’s enrollment into and sign out of each cohort (readiness assessment is a part of enrollment, while implementation assessment is a part of sign out). For the readiness assessment, the national team will send an email encouraging the hospitals to sign up for their 1-hour phone call with a member from the national team. The national team will follow up with a friendly reminder to those hospitals who do not sign up. Our availability will be flexible and we will offer additional times throughout the data collection period to ensure that as many hospitals as possible can participate in these assessment. For the implementation assessment, the project team will send an email encouraging the hospitals to complete a survey through the Qualtrics platform. The project team will follow up with a friendly reminder to those hospitals who do not complete the survey. The surveys have been designed to only ask questions relevant to the service line the hospital is implementing to minimize the burden for hospitals.

Site visits. We are not offering cash remuneration for respondents, but conducting the site visits virtually, and for a half-day instead of a full day, should reduce the burden for participating hospitals. Findings will help inform the national project implementation strategy as well as the hospital’s implementation strategy. National team program experts will gather information from these visits on if and how team and/or leadership issues may affect implementation of enhanced surgical care and recovery practices, including how this may differ across surgical service lines. Interviews will help uncover misalignments in role clarity, needed time and resources, best practices, and potential enablers of and barriers to program implementation. Additionally, national team program experts will share insights, best practices and strategies with participating hospitals to they too can benefit from the visit. Finally, participating hospitals will receive a structured debriefing and brief summary report at the end of the visit.

4. Tests of Procedures

The procedures for this specific project have not been subjected to testing, however, Johns Hopkins Armstrong Institute for Patient Safety and Quality, American College of Surgeons, and Westat have conducted many similar projects and are using well-established research methods with this project.

5. Statistical Consultants

Mark Cohen Ph.D.

American College of Surgeons

633 North St Clair

Chicago, IL 60611-3211

mcohen@facs.org

Yea-Jean Hsu, PhD

Armstrong Institute for Patient Safety and Quality

Yhsu9@jhu.edu

Naomi Yount, PhD

Westat

1600 Research Blvd.

Rockville, MD 20878

naomiyount@westat.com