Supporting Statement Part B -- Pretest of an Assisted Living Consensus Instrument_11-22-2011

SUPPORTING STATEMENT

Part B

Pre-test of an Assisted Living Consensus Instrument

Version: November 10, 2011

Agency for Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 1

1. Respondent universe and sampling methods 1

2. Information Collection Procedures 2

3. Methods to Maximize Response Rates 5

4. Tests of Procedures 6

5. Statistical Consultants 7

B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The target population for this pretest is all assisted living facilities (ALFs) in the following eight states in the United States: California, Florida, Iowa, Maryland, New Jersey, North Dakota, Ohio and Oregon. These states, identified by national experts, represent the variability of assisted living (AL) regulatory environments in the United States. There are approximately 14,900 ALFs in these eight states. We plan to select a sample of 300 ALFs across these states. This sample will be selected using a two-phase sampling design.

For the purposes of this data collection, an ALF is defined as:

“Assisted living” refers to residential long-term care options that are licensed, certified, or registered by states as assisted living or other residential care names, such as board and care. They combine housing and supportive services, which include at a minimum, assistance with activities of daily living and/or health care (such as help with medication administration). Assisted living settings have on-site staff available to meet both scheduled and unscheduled needs for assistance 24 hours per day, seven days per week. They also offer dining (two or more meals per day) and a variety of supportive services related to social and wellness activities. They care for individuals with a range of functional needs including dementia, and may provide a dedicated wing/area with additional security and cueing devices among other special services for those individuals. Assisted living rooms/apartments may be offered in freestanding communities or in a separate wing or building in a long-term care campus that provides other types of care. In this project, assisted living does not include residential long-term care options that are licensed, certified, or registered by states as nursing homes, or to exclusively serve persons with intellectual and developmental disabilities, mental illness (which is different than dementia), or substance use disorders. ALFs with fewer than 6 beds are not included.”

This definition of an ALF is the same as that used by NCHS in their recent National Survey of Residential Care Facilities (NSRCF)(with the exception that the NCRCF includes facilities with 4 or 5 beds and this pre-test does not).

State licensure lists will be used as the sampling frame for the selection of ALFs in Phase I of our two-phased sampling approach. (Similar sampling frame construction methods as were used in the NSRCF will be used for this pre-test; due to data confidentiality limitations the NSRCF frame can’t be used for this effort.) These state lists come in various formats and provide only basic information such as ALF name, address, telephone number and number of beds. In some states, more information may be available (e.g., ALFs that accept Medicaid) than in other states; we therefore plan to obtain representation for these domains in as many of the eight states as possible. Table 1 shows the information available for each state from the individual licensure lists.

Table 1: Domains of Interest Available from State Licensure Lists

State	Name, Address, Telephone	Bed Capacity	Profit Status	Medicaid Status	Dementia Care	CCRC affiliation
California	X	X				X
Florida	X	X	X
Iowa	X	X			X
Maryland	X	X	X		X	X
New Jersey	X	X		X
North Dakota	X	X
Ohio	X	X		X	X
Oregon	X	X		X

2. Information Collection Procedures

The objective is to test the efficacy of a mailed paper survey (which will be conducted on the second phase sample of 300), to collect data from ALFs using a carefully designed survey instrument. Therefore, it is important to ensure that subgroups that might differ with regard to the ability to respond to the survey questionnaire are represented in the sample. Sampling error (precision) of the overall estimates and domains is less of a concern than ensuring that the sample is representative of the total population of ALFs in the eight states. It is also important that the sample attempt to represent ALFs that belong to some important domains and subgroups in the overall population. These domains/subgroups include: profit status (for-profit/Other), Medicaid (accept Medicaid/Other), dementia neighborhood (presence of dementia neighborhood/Other), affiliated with another level of care (part of a Continuing Care Retirement Center (CCRC)/Other). As the information available for ALFs in different states may not include full domain information, we propose to select a larger first phase sample, and then select a subsample from strata created based on information from the first phase sample. Keeping in view the total budget for the survey and the number of domains that need representation, we plan to select a sample of 600 ALFs in the first phase sample, and seek more information about the domains of interest from publically available sources (such as provider web sites when available). With information about the 600, we will then select 300 ALFs for the second phase sample.

The first phase and second phase samples are allocated to each of the eight states in proportion to the square root of the total number of ALF beds in each state, with a requirement that at least 20 ALFs are selected in each of the eight states. This allocation takes into account the total number of ALF beds available in each state rather than just the total number of ALFs. The square root allocation is to ensure moderate sample sizes in states where the number of ALFs is small compared to other states. Table 2 displays the distribution of ALFs by State, and the planned sample size in each phase of sampling.

Table 2: Distribution of Facilities by State

State	Number of Facilities	Number of Facilities in the First Phase Sample	Number of Facilities in the Second Phase Sample
California	7,633	146	73
Florida	2,168	100	50
Iowa	385	56	28
Maryland	1,168	52	26
New Jersey	319	60	30
North Dakota	112	40	20
Ohio	886	78	39
Oregon	2,273	68	34
Total	14,894	600	300

Independent samples will be selected in each state. For the selection of the first-phase sample in each state, the population of ALFs will be stratified by bed size. Three size strata will be created: 6-16 beds, 17-60 beds, and >60 beds. The total sample in each state will be allocated to each size stratum in proportion to either the total number of ALFs in each size stratum or the total number of beds in each size stratum in the state. The final decision about size strata (by total ALFs or total beds) will be made after examining the distribution of facilities and beds by size group. A systematic sample of ALFs will be selected in each stratum. This will form the first phase sample, from which additional publically available domain information will be sought.

The information collected from the first phase sample will be used to create further strata within each state and size stratum, for selecting the second phase sample. The stratification for selection of the second phase could be different in each state, based on the available information. For example, the strata created in a state with ALFs that accept Medicaid could be size by Medicaid (yes or no). This means that instead of three size strata in this state, there would be six strata for the selection of the first phase sample. Sub sampling will be done from each stratum, providing representation to both size and Medicaid (yes/no) groups. If a stratum has only one ALF in the first phase sample, then it will be selected with certainty.

Every effort will be made to obtain a 65 percent response rate from the second phase sample of 300, resulting in 191 completed surveys (.65 x (300 ALFs – 5 ALFs found to be out-of-business during the telephone verification survey) = 191 completes). Based on this sample size, the margin of error for estimated population percentages of around 50 percent will be plus or minus 7.7 percentage points at the 95 percent confidence level, assuming a simple random sample. Since the population of ALFs will be stratified by state and size groups, and the sample will be selected within each stratum, we expect the precision to be much higher than stated above.

Sampling Weights

Each responding facility will be assigned a sampling weight. This weight is necessary since the sampling rates in different strata could be different. This combines a base weight and an adjustment for nonresponse. The base weight is the inverse of the probability of selection of a facility within a stratum. The overall weight which is the product of the base weight and a nonresponse adjustment factor will be used for estimating population proportions and ratios. The estimated proportions and ratio relate to the population of facilities in the 8 states selected for the survey.

Non-Response Bias Analysis

Non-response bias is a function of the non-response rate and the difference between the responding and non-responding ALFs on the variables of interest. This is typically represented by the following formula:

where

Bias(y_r) = the nonresponse bias of the unadjusted respondent mean;

y_r = the unadjusted mean of the respondents in a sample of the target population;

Y_r = the mean of the respondents in the target population;

Y_m = the mean of the non-respondents in the target population;

M = the number of non-respondents in the target population; and

N = the total number in the target population

We will assess potential bias by first examining the response rates by state and facility size. We will also look at the results of the sample for detecting any differences in the domain characteristics. For example, whether facilities in one domain seem to have more difficulty completing the questionnaire than facilities in other domains will provide information on bias due to an imbalance in the distribution by domains. This information coupled with the response rates will give an indication of bias in the survey results. We will also compare the frame characteristics of respondents and non-respondents to assess any bias in the survey results. We also plan to compare the characteristics of those who are early respondents and those responding later after reminders. This is to identify whether late responders are more similar to non-responders (and using the formula above to assess the nonresponse bias).

Data Processing

Data from survey respondents will be received, entered into a project database, and cleaned by the survey contractor. For this project, Abt Associates and its subcontractor Abt-SRBI will utilize double data entry, and entry with verification. Double data entry consists of the data being keyed twice and compared. Discrepancies are flagged and a third party determines which entry was correct, based upon the hard copy questionnaire, and makes any necessary changes. In double data entry, an estimated error rate of 5/100 keystrokes is routine.

Entry with verification requires two entries of the data as well. However, during the second entry, the computer system will emit a beep to alert the operator that a discrepancy has been found with the first entry. The second operator will then reassess their entry and when necessary refer any problems to the quality assurance supervisor. In entry and verification, an estimated error rate within 1/100 keystrokes is routine.

Upon completion of all data entry, the data file will be loaded into an SPSS program to generate frequencies and checks for inconsistencies. The Data Receipt file will then be matched against the Data Entry file to ensure that all questionnaires that were receipted were also entered. Frequencies on all variables will be reviewed to determine that all questions are accounted for in the data, that the data lines up according to the database layout, and that there are no erroneous values in the database.

3. Methods to Maximize Response Rates

Project staff will verify the name of the ALF’s Executive Director/Administrator via a telephone call to the 300 second phase sample ALFs prior to the first mailing of the Assisted Living Provider Information Tool for Consumer Education. The mailing address for the facility will also be verified to minimize the chance of “return to sender” due to an incorrect mailing address or out of business status. A 95 percent response rate is expected for the telephone verification (285 responding); the verification is also expected to identify 5 additional facilities as out-of-business (based upon pre-test results of the National Survey of Residential Care Facilities).

Each sampled ALF will be mailed the survey materials as follows:

1. Pre-notification of mail survey data collection via a letter to the Executive Director/Administrator of all sampled second phase ALFs (minus the 5 ALFs found to be out-of-business during the telephone verification)

2. The first mailing of the questionnaire to 295 (300 facilities in the second phase sample minus 5 facilities found to be out-of-business during the telephone verification)) sampled ALFs will contain:

• Cover Letter on white stationary with ALC Banner in color;

• Survey instrument (16 sided booklet, saddle-stitched) with survey ID (black and white, keypunch format);

• ALC one-page information brochure on white paper;

• Postage-paid 9”x12” business reply envelope to Abt SRBI Cambridge, Massachusetts; and

• 10”x13” outgoing HHS AHRQ government envelope with return address for, c/o Abt SRBI Cambridge, Massachusetts.

3. A reminder post card will be mailed approximately one week after the initial mailing of the survey packet.

4. If there is no response (completed survey booklet, note of undeliverable, etc.) to the first mailing, an additional copy of the questionnaire with a revised cover letter will be mailed four weeks after the initial mailing.

5. A second reminder post card will be mailed approximately one week after the second mailing of the survey packet.

6. If there is still no response after the second mailing (completed survey booklet, note of undeliverable, etc.), a third (and final) copy of the questionnaire and cover letter will be mailed four weeks after the second mailing.

7. A third and final reminder post card will be mailed approximately one week after the third mailing of the survey packet, and this card will indicate the last date to receive completed surveys.

8. The data collection period will end four weeks after the third mailing of the survey packets.

4. Tests of Procedures

A small scale test of the data collection instrument was conducted with nine ALFs via cognitive interviewing. The pre-test was conducted in two waves, with four interviews in the first wave, followed by five interviews in the second wave. In between the waves, the data collection instrument was revised based upon feedback from the pre-test participants. The data collection instrument was also revised one final time after the second wave of cognitive interviewing. The burden estimate of 25 minutes for the Assisted Living Provider Information Tool for Consumer Education is based on these interviews.

The procedures used in the pre-test were not subjected to pre-testing. However, well established survey methodology procedures will be used for the pre-test and AHRQ and its’ contractor have ample experience using these methods.

5. Statistical Consultants

Kadaba P. Srinath, Ph.D.

Abt Associates Inc.

4550 Montgomery Avenue

Bethesda, MD 20814

Sheryl Zimmerman, Ph.D.

University of North Carolina, Chapel Hill

725 Martin Luther King Jr. Blvd., Campus Box 7590

Chapel Hill, NC 27599-7590

D.E.B. Potter, M.S.

Senior Survey Statistician

Division of Statistical Research and Methods

Center for Finance, Access and Cost Trends

Agency for Healthcare Research and Quality

William A. Carroll, MA

Survey Statistician

Division of Survey Operations

Center for Finance, Access and Cost Trends

Agency for Healthcare Research and Quality

i