Ahrq_nhccc_ssb_02_10_2022

As part of AHRQ’s National Nursing Home COVID-19 Coordinating Center, the Agency is conducting an assessment of whether and how the AHRQ National Nursing Home COVID-19 Action Network (henceforth referred to as ‘the Network’) activities covered by the Provider Relief Funds (PRF) aided the nursing homes’ efforts to mitigate the challenges posed by the COVID-19 pandemic. The assessment will draw upon primary data collection in the form of a survey of nursing homes and semi-structured key informant interviews, as well as secondary data, to assess motivations for participation and nonparticipation in the Network, the reach and scale of the Network, participants’ perceptions on whether the disseminated knowledge is actionable, and the progress toward achieving the goals of the Network. The proposed primary data collection efforts described below are part of this assessment.

B1. Respondent Universe and Sampling Methods

Recruitment Methods, Respondent Selection, and Sample Size

The assessment will include primary data collection, consisting of a survey of all nursing homes eligible for the PRF and key informant interviews with nursing home staff, by AHRQ’s Coordinating Center contractor NORC at the University of Chicago (NORC). NORC will conduct analyses of the survey results and the key informant interviews, as well as analyses of the data submitted by the nursing homes as part of their participation in the Network.

Primary Data Collection by Coordinating Center Contractor

Survey. NORC will conduct a survey of the target population of Provider Relief Fund (PRF) eligible nursing homes (N=15,372), attempting to obtain completed interviews from all participating nursing homes and a 50% representative sample of eligible nonparticipating nursing homes (see Exhibit 1). This population includes approximately 9,058 nursing homes that participated in the Network, as well as 6,314 eligible nursing homes that did not participate in Network activities. Because the target population is relatively small, and due to expected levels of non-response as is common in survey research, all eligible participants and a 50 percent sample of eligible nonparticipants will be targeted by the survey to ensure that the response sample size is adequate to analyze the outcomes of priority subpopulations, such as nursing home participants that serve vulnerable populations. About 1,500 nursing homes that are being targeted by other federal data collection efforts will also be excluded from the survey frame to minimize respondent burden. After such exclusions, the resulting survey frame will include a total of 13,872 eligible nursing homes.

If all eligible nursing home participants are targeted by the survey, an anticipated sample yield of 20% would result in completed interviews from roughly 1,662 nursing home participants from the original 8,308 eligible nursing home participants targeted by the survey. Even with such a sample size the margin of error for the estimate of a proportion assuming a modest design effect of 1.25 (resulting from weighting adjustments) would be 2.5 points using a 95% confidence level. Sampling only a portion of the eligible nursing homes would yield estimates with less precision and may not satisfy research objectives for precise estimates of characteristics in the population.

Exhibit 1. Target population and survey sample

	Survey of Participants	Survey of Nonparticipants
Target population of nursing homes	9,058	6,314
Eligible* nursing homes targeted by the surveys	8,308	5,564
Survey sample**	8,308	2,782
Response sample (Assuming 20% response rate)	1,662	556
Expected Margin of Error (Assuming a design effect of 1.25; 95% confidence level)	2.5%	4.4%

Note: *The following are the eligibility criteria for the survey: Met eligibility criteria to receive Provider Relief Funds; and Nursing home was not in the survey sample of the CMS nursing home survey. ** The sample for the survey of participants will include all eligible nursing home participants. The survey of nonparticipants will include a 50% representative sample of eligible nonparticipants.

Importantly, assessing the implementation experience and outcomes for participating nursing homes that serve vulnerable populations is a key objective of the assessment. Key subgroups include nursing homes located in areas with high COVID-19 positivity rates; small, independently run facilities; and facilities with a disproportionate share of residents with high needs (i.e., 3+ chronic conditions and a functional limitation). This proposed cross-classification of the nursing homes will result in analysis cells with approximately 2,000 eligible nursing homes and 400 eligible completes each. For each of these key nursing home participant subgroups of interest, the expected margin of error for a proportion is 5.0 percent (1.25 design effect and 95% confidence). Surveying a representative sample of nursing home participants even as large as a 50% sample will result in a margin or effort as large as 7.5% (1.25 design effect and 95% confidence).

Administrative data sources, such as the Centers for Medicare & Medicaid Services (CMS) Provider of Services (POS) file, monthly PRF eligibility data, Network participation data, Minimum Data Set (MDS) nursing home assessments, and Certification and Survey Provider Enhanced Reports (CASPER) data will be used to identify and stratify the survey’s target populations.

To identify the proper contacts at all participating and non-participating nursing homes, initial efforts will focus on gathering contact information for leadership staff within each nursing home, including name, email address, mailing address, and telephone number. To identify Network participants, AHRQ and NORC will work closely with the University of New Mexico (UNM) ECHO Institute to compile existing contact information. All PII will be transferred between the three stakeholders (the Coordination Committee, the Network, and AHRQ) using AHRQ’s secure file transfer site. In the event that UNM has contact information for a sizable number of non-participating nursing homes, AHRQ and NORC will create an additional contact list using that information. If not, NORC and AHRQ may work with a contractor like IQVIA One Key to acquire a more extensive list of email addresses for non-participating nursing homes.

The survey will target the following job titles within each nursing home to respond on behalf of their facility (only one response will be collected per facility, but due to non-response it may be necessary to reach out to multiple potential respondents within each facility):

Executive Director/Administrator
Medical Director
Director of Nursing/Nursing Supervisor
Department Head
Unit Manager/Charge Nurse
Assistant Director/Assistant Manager
Minimum Data Set (MDS) Coordinator/ Resident Nurse Assessment Coordinator (RNAC)

Key Informant Interviews. The assessment will aim to conduct semi-structured qualitative interviews with 32 nursing homes (up to 96 participants) across a variety of nursing homes involved in the Network. From the universe of nursing homes, nursing homes will be selected based on a combination of contextual and structural factors. The contextual factors include 1) location (metropolitan or nonmetropolitan) and 2) COVID-19 incidence (high COVID-19 positive rates in the county or low COVID-19 positivity rates in the county). The structural factors include 1) nursing home ownership (not-for-profit or for-profit) and 2) vulnerability of residents (high proportion of vulnerable residents or low proportion of vulnerable residents). Exhibit 2 displays the 16 possible combinations for selection of nursing homes. Two nursing homes will be selected from each combination below, resulting in 32 nursing homes.

Exhibit 2. Combinations of contextual and structural factors for nursing home selection

Combination	Location	COVID-19 Positivity Rate in County	Ownership	Proportion of vulnerable residents
1	Metropolitan	Low	Not-for-profit	Low
2	Metropolitan	Low	Not-for-profit	High
3	Metropolitan	Low	For-profit	Low
4	Metropolitan	Low	For-profit	High
5	Metropolitan	High	Not-for-profit	Low
6	Metropolitan	High	Not-for-profit	High
7	Metropolitan	High	For-profit	Low
8	Metropolitan	High	For-profit	High
9	Nonmetropolitan	Low	Not-for-profit	Low
10	Nonmetropolitan	Low	Not-for-profit	High
11	Nonmetropolitan	Low	For-profit	Low
12	Nonmetropolitan	Low	For-profit	High
13	Nonmetropolitan	High	Not-for-profit	Low
14	Nonmetropolitan	High	Not-for-profit	High
15	Nonmetropolitan	High	For-profit	Low
16	Nonmetropolitan	High	For-profit	High

When selecting these 32 nursing homes, the study will ensure that there is variability in the level of engagement in the Network training sessions, as well as on outcomes of interest, such as the participating nursing homes’ capacity to obtain protective equipment; robustness of the testing protocols; mitigating staffing challenges; and reducing the incidence of COVID-19 among residents and staff. Representation across nursing home size and location (i.e., Northeast, Midwest, South, and West of the U.S) will also be taken into consideration when selecting the sample.

Participants in the key informant interviews will be selected to represent key roles within the nursing homes participating in the Network, namely: leadership (e.g., Administrators, Executive Directors, Directors of Health Services, Directors of Nursing, Assistant Directors of Nursing, Directors of Staff Development, Medical Directors, and Quality Assurance and Performance Improvement (QAPI) Specialists) and staff (e.g., Registered Nurses (RNs), Licensed Practical Nurses or Licensed Vocational Nurses (LPNs/LVNs), Certified Nursing Assistants (CNAs), Infection Preventionists, Social Workers, Activities Directors, Human Resources, Dietary Managers, and Housekeeping Supervisors). Following the selection of the 32 nursing homes, AHRQ will aim to recruit 1nursing home leader and 1-2 nursing home staff, for a total of up to three respondents from each nursing home to participate in key informant interviews. In total, up to 96 participants may be recruited to complete an interview (Exhibit 3). During the outreach process, the scheduler will work with the nursing home via email and through phone calls to identify the appropriate staff who attended the training or who made key decisions around participating in the program.

Exhibit 3. Potential Qualitative Semi-Structured Interview Participants

Respondent Type	Key Roles Represented	Respondents
Leadership	Administrators, Executive Directors, Directors of Health Services, Directors of Nursing, Assistant Directors of Nursing, Directors of Staff Development, Medical Directors, and Quality Assurance and Performance Improvement (QAPI) Specialists	32
Staff	RNs, LPNs/LVNs, CNAs, Infection Preventionists, Social Workers, Activities Directors, Human Resources, Dietary Managers, and Housekeeping Supervisors	64
Total		96

Secondary Data Collected By the Network

Data on the Network’s training centers, cohorts, and sessions, and topics covered
Nursing home eligibility, level of participation, and duration of participation
Session attendees and their feedback on the sessions

Response Rates

Survey.

Completion rates for the nursing home survey, based on previous national surveys of nursing homes, are anticipated to range between 20-25 percent.¹

Key Informant Interviews

We anticipate a 33 percent positive response to invitations for semi-structured, key informant interviews with nursing home leadership and staff. We will utilize contact information from the Network participation data to reach initial contacts at the nursing homes and use a snowball sampling approach to ensure representation from various types of the nursing home employees (i.e., leadership and staff) that participated in the training sessions.

B2. Procedures for Collection of Information

This project consists of two primary data collection activities by the Coordinating Center contractor and will contribute substantively to the assessment.

Survey. AHRQ will conduct the survey utilizing a sequential mixed mode approach, with the primary mode of response being a web survey for the nursing homes that participated in the Network (Participant Survey) and both web and telephone for the nursing homes that did not participant in the Network (Non-Participant Survey). For the Participant Survey, eligible nursing homes will receive a web survey invitation and if needed, the telephone follow-up option. To help ensure that the survey’s response rate thresholds are met, the Network’s Learning Collaborative Newsletters may be used to help publicize the survey.

Outreach to nursing home staff with the survey request and prompts to complete the survey will be sent via mail, telephone, and/or email, depending on the availability of valid contact information. The survey request will include a hyperlink (for email requests) or URL (for mail requests) and unique PIN to access the web questionnaire. A team of specially-trained NORC telephone interviewers will reach out to any respondents that have only a telephone number available. This team will also use telephone prompting to reach non-responding facilities, prompt them to respond (providing the URL and PIN if needed), and if requested by the respondent, conduct the interview via telephone. For the Non-Participant Survey, telephone interviewers will attempt to conduct the interview via telephone if the respondent is available and willing to do so.

Key Informant Interviews. Key informant interviews will be conducted using a virtual web platform. The interviews will cover domains such as motivation for participation; facilitators and barriers to engagement; the facility’s experience of participation in the Network training and activities; gaps in knowledge, skills, and resources; and impacts on patient safety procedures, infection control protocols, and quality improvement activities. Attachment C includes the interview recruitment materials and thank you e-mail template, and Attachment D includes the information sheets for all informant types. Attachments E includes the interview protocol.

Evaluation Design

Data Analysis. For the survey responses, the assessment will employ post-stratification weighting procedures, such as raking to produce survey weights. The analysis will utilize appropriate survey analysis specifications and the survey weights to produce estimates that are representative of the target population and subpopulations. Analyses will include survey-weighted univariate and bivariate analysis to assess implementation and employ appropriate statistical tests to assess difference in responses across subpopulations.

The semi-structured key informant interviews will be systematically coded for themes using Dedoose qualitative data analysis software. To perform analysis, the team will develop a codebook of existing (from the interview guide) and emergent (from responses) themes. To establish strong inter-rater reliability and test the reliability of the codebook, the analysis team will seek to achieve a kappa of >.60.ⁱ Coded data will be used to develop narratives that answer the assessment questions. Analysis of findings within codes will reveal similarities and differences in the perspectives of key informants, as well as the range of opinions and experiences on a given topic. Analysis of the relationship between codes or among a combination of codes will examine the interrelationship between themes or concepts.

Comparative case analysis will also be used to help understand differences across the nursing homes participating in the Network, and to supplement the quantitative findings on variability in outcomes.

B3. Methods to Maximize Response Rates

The data collection is part of the assessment of the Network’s activities covered by the Provider Relief Funds. As a part of participation in the Network, cohorts of nursing homes engage with local training centers. To maximize the response rate for the key informant interviews, the assessment team will work with nursing home industry associations, the University of New Mexico Project ECHO – the entity overseeing the implementation of the Action Network – as well as the training facilitators in the outreach and will clearly present the goals of the key informant interviews (i.e., understanding the challenges and successful strategies with the intent to disseminate them).

For the survey, the proposed sequential mixed mode approach involves a web survey with targeted mail, email and telephone prompting as well as the option to complete the survey over telephone. Using multiple modes of approach (mail, email, and phone), a well-designed and easy-to-navigate web survey experience, and the option to complete via phone will maximize response rates for the target population and subpopulations by minimizing respondent burden. Additionally, to help increase response rates, respondents will be sent prompts via mail and email, as will receive telephone prompts with the option to complete via telephone.

B4. Tests of Procedures or Methods to Be Undertaken

The survey instruments and the key informant interview guide were developed by drawing on the Coordinating Center contractor’s past experience conducting surveys and qualitative data collection and analysis. Input from a technical expert panel of nationally recognized experts in nursing home quality and patient safety informed the development of the survey instrument. Response categories for select question items were based on information in the Nursing Home Survey of Patient Safety Culture (NHSOPS).

B5. Statistical Consultants

NORC at the University of Chicago will serve as the primary consultant for statistical aspects of the design and analysis of the assessment data. See Exhibit 3 for a list of statistical consultants.

Exhibit 4. List of Statistical Consultants

Name	Title and Institution
Edward Mulrow	Senior Vice President, Statistics and Methodology, NORC at the University of Chicago
Michael Yang	Principal Statistician, Statistics and Methodology, NORC at the University of Chicago
Evan Herring-Nathan	Statistician III, Statistics and Methodology, NORC at the University of Chicago

The survey data and key informant interview data will be collected and analyzed by NORC at the University of Chicago. The secondary data will be collected by nursing homes and shared with NORC at the University of Chicago by the Network.

1 Recent surveys of long-term care providers, such as the National Study of Long-Term Care Providers conducted by the Centers for Disease Control achieved a response rate of 30 percent. These surveys, as well as NORC’s recent experience conducting surveys of health care providers since the onset of the COVID-19 pandemic informed this estimation.

i Mchugh ML. Interrater reliability: the kappa statistic. Biochemia Medica. 2012:276-282. doi:10.11613/bm.2012.031

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	NORC
File Modified	0000-00-00
File Created	2022-03-02