Supporting Statement Part B -- Consumer Choice Study 12 21 09

SUPPORTING STATEMENT

Part B

Study of Factors Influencing Consumer Choices Among Health Plans and Clinicians

Version: December 18, 2009June 3, 2009

Agency forof Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 3

3. Methods to Maximize Response Rates 4

4. Tests of Procedures 4 5. Statistical Consultants 5

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

Respondents for this experimental study are drawn from the Knowledge Networks Internet panel.  Knowledge Network's Internet panel consists of 50,000 adult panel members who are recruited by random-digit dialing (RDD) or by using address-based sampling.  Typical panel members receive 3-4 invitations per month to participate in research projects.  

Although the Knowledge Networks Panel is constructed to include those who do not otherwise have internet access (by providing them with free access in return for their participation on the panel) and those who are not computer-savvy (by providing them internet access through a web-TV appliance), we will limit the experimental sample to those who access the internet via computer, since these approximate the type of individuals who are likely to seek information about health plans or doctors over the internet. Excluding respondents using web-TV appliances reduces the sample by approximately 11 percent, though a substantial number of older and less-educated individual (the subgroups least likely to be computer literate) remain in the eligible panel.

We do not intend to generate nationally or locally representative results or precise estimates of population parameters from this study. The sample used is best understood as a convenience sample, rather than a probability sample. The Knowledge Networks panel is large and variegated enough to produce samples with a reasonable degree of diversity in key demographic characteristics.  Furthermore, no legitimate weights can be constructed from non-probability samples such as the one used here.  Hence, we will not in any publications emerging from this work construe this sample or the results generated from this sample as nationally or locally representative.  The strength of the experimental study lies in its internal validity, on which meaningful estimates of differences across the experimental exposures (complexity of choice sets) can be produced and generalized.

Participants in this experiment will be recruited through the Knowledge Networks (KN) national online panel of consumers. KN is a consumer information company that conducts on-line surveys with a panel of consumers. The company began as a venture-funded privately-owned firm in 1998. It maintains offices in CA, NY, NJ, and OH. KN has approximately 225 employees. Its Government and Academic Research has conducted many studies with a variety of researchers and their teams in the fields of health policy and services, epidemiology, environmental protection, political science, sociology, and social psychology. Researchers in these and other fields have conducted text-based and multimedia surveys using the web-enabled KN Panel.

Approximately 1000 participants will be recruited for the clinician choice experiment and 1200 participants for the health plan choice experiment. The participants will be randomly selected from the KN Panel. The Panel, recruited randomly through Random Digit Dialing, represents the broad diversity and key demographic dimensions of the U.S. population. The web-enabled panel tracks closely to the U.S. population on age, race, Hispanic ethnicity, geographical region, employment status, and other demographic elements. The differences that do exist are small and are corrected statistically in survey data (i.e., by non-response adjustments).

Adults aged 25 years or older will be recruited to ensure that participants are able to make health decisions on their own. The study sample will be balanced with respect to gender and will include a mix of educational levels and racial/ethnic groups. All study subjects will be English-language proficient. Individuals employed in the health care sector will be excluded, since they are likely to use other sources of information from their work affiliations rather than rely on information provided in report cards.

2. Information Collection Procedures

For both the health plan and clinician choice experiments, study subjects will be randomly assigned to one of several arms (see Part A Section 1 for a description of the arms) that vary according to the type and complexity of performance information and the size of the choice set (number of plans or doctors) included in the Web report. Participants will complete the experiment through a secure online connection from their homes. Data will be derived from pre and post-test questionnaires and from server logs that record the web pages visited and viewing times.

3. Methods to Maximize Response Rates

The response rate is estimated at about 75% based on results obtained from the past projects conducted by KN. Procedures for maximizing response rates include:

• Field period of 3 to 4 weeks

• Use of the Federal agency or University/College name in the email invitation

• Email reminders

• Telephone reminder calls to non-responders

 

The initial analysis of response rate of 75% or better indicates that this response rate, in the combination with the size of the population selected for each experiment (described the section titled Respondent Universe and Sampling Methods), will provide sufficient power to test for the experimental differences.

4. Tests of Procedures

To achieve the aims of the experiment, the following four analyses will be performed:

1. McFadden’s (1974) conditional logit model will be used to model choice of doctors or plans that are of higher-quality, as measured by CAHPS scores. Conditional logistic regression models multicategorical choice as a function of the characteristics of the choices themselves. It also allows for examination of the influence of characteristics that do not vary within a given set of choices for a single experimental participant (such as characteristics of the person making the choice or of the choice set) through interactions with characteristics of the choices.

In this experiment, each participant chooses among either 12 or 24 doctors (or 5 or 10 health plans) whose characteristics depend on the experimental condition. The choices vary according to CAHPS scores (high or low), and in some experimental conditions they also vary according to HEDIS (The Healthcare Effectiveness Data and Information Set ) scores (high or low) and modal affect of anecdotes (strongly positive/negative, weakly positive/negative, or mixed). The presence or absence of HEDIS information, the availability of complaint counts (for health plan choices) and the presence or absence of anecdotes each changes the characteristics of the choices, so these factors will be incorporated by using parallel conditional logit models for choice sets that do or do not include each of these types of information.

In certain experimental conditions, the quality of choices can also be measured by HEDIS scores or modal affect of anecdotal information. Within these conditions, conditional logit can also be used to examine these outcomes.

2. The effects of demographic variables, such as age, gender, and education, and of other individual difference variables, such as perceived health status and decision-making style, will be examined by testing for interactions with choice characteristics.

3. Analysis of variance will be used to examine the effects of type and amount of information presented on how the respondents use the web site, including which pages they visit and how much time they spend viewing them.

4. Responses to post-experimental questionnaires will be analyzed using descriptive summaries and analysis of variance to provide insight into the strategies participants employ in using the information presented to choose a doctor or health plan and how they view this information. .

5. Statistical Consultants

Statistical expertise in analyzing the results of the experiment will be available from Marc Elliott, PhD, of RAND

5