Form #2 Form #2 Experimental Website

List of Tables

Table 1 Steps of the Physician Choice Experiment 2

Table 2 ARMS of the Physician Choice Experiment 3

List of Figures

Figure 1 Tutorial Page 1 5

Figure 2 Tutorial Page 2 6

Figure 3 Tutorial Page 3 7

Figure 4 Tutorial Page 4 8

Figure 5 Start Page of the Web Site 9

Figure 6 Data Selection Screen ARM #4 10

Figure 7 Performance Overview Screen 11

Figure 8 Performance Overview ARM #2 12

Figure 9 Service Quality Tab 13

Figure 10 Treatment Quality Tab 14

Figure 11 Patient Reviews Tab 15

Figure 12 Physician Choice Confirmation 16

Appendix A: Data Selection Screens for all Six ARMS of the Experiment 18

Physician Choice Experiment(s)

Introduction

The Physician Choice Experiment consists of a series of steps in which respondents are transferred from the Knowledge Networks Web site to the SelectMD Web site and then back to Knowledge Networks. Table 1 summarizes the steps of the experiment.

Table 1 Steps of the Physician Choice Experiment

Step 1	Respondent views the Knowledge Networks introduction to the experiment
	
Step 2	Respondent completes the pre-test questionnaire on Knowledge Networks’ Web site
	
Step 3	Respondent is transferred to SelectMD Tutorial and views the 4-page tutorial
	
Step 4	Respondent uses SelectMD Web site to “select” a physician
	Start screen displayed; then respondent is assigned/sent to one of six ARMS 
	ARM 1	ARM 2	ARM 3	ARM 4	ARM 5	ARM 6
	Control	Experimental
	
Step 5	Respondent is transferred back to the Knowledge Networks Web site
	
Step 6	Respondent completes the post-test questionnaire on the Knowledge Networks Web site
	
Step 7	Respondent views conclusion and thank you on the Knowledge Networks Web site

The Physician Choice Experiment consists of six ARMS: one of which is a control ARM and five of which are experimental ARMS. (See Table 2.) The ARMS differ based on the types of information displayed to the respondents and on the number of physicians listed on the SelectMD Web site.

As many as four types of information will be shown depending upon the ARM involved. The information types are as follows:

• Performance Overview: Summary of the information available in the ARM. (Shown to all, but with varying content.)

• Service Quality: Survey data using CAHPS items and fabricated CAHPS data.

• Treatment Quality: Fabricated HEDIS-like data.

• Patient Reviews: Mimics volunteered patient comments about doctors.

ARMS numbered 1-4 present different sets of information to respondents. ARM 5 presents the same information as ARM 3 but offers a larger choice set – the number of doctors listed doubles from 12 (ARM 3) to 24 (ARM 5). ARM 6 presents the same information as ARM 4, but offers the same larger choice set of 24 physicians.

The ARMS are described in detail in Table 2.

Table 2 ARMS of the Physician Choice Experiment

ARM Numbers	Tabs				Choice Set Size (Number of Physicians)
	Performance Overview	Service Quality (CAHPS scores)	Treatment Quality (HEDIS scores)	Patient Reviews (Volunteered comments)
Control
1					12
Experimental
2					12
3					12
4					12
Experimental, Larger Choice Set
5					24
6					24

About This Document

This document shows example screenshots for one of the ARMS of the experiment. The selected example is ARM #4 because it presents the largest amount of data. (ARM 4 is highlighted in Table 2.) All of the screenshots come from Steps 3 and 4 of the experiment.

To see the SelectMD Web site, please go to: http://www.selectmd.org/site/intro/.

User name: wowza

Password: cahps15

The experiment opens with an introduction from Knowledge Networks followed by a Pre-test Questionnaire. Upon completion of the Pre-test Questionnaire, respondents are provided with an introduction to the SelectMD tutorial pages and transferred to the SelectMD Web site. The respondents view the tutorial pages and then enter the main SelectMD site.

Tutorial Page 1

(Phase 3)

Figure 1 is the first tutorial screen respondents will see. It is intended to explain what they will find on the SelectMD Web site and to familiarize them with the labels the site will use. When respondents are finished with this page, they will click “Continue” at the lower right hand side of the page.

Figure 1 Tutorial Page 1

Tutorial Page 2

(Phase 3)

Figure 2 is the second screen of the tutorial. It is intended to introduce respondents to the various things they can do while they are on the SelectMD Web site. When respondents are finished with this page, they will click “Continue” at the lower right hand side of the page.

Figure 2 Tutorial Page 2

Tutorial Page 3

(Phase 3)

Figure 3 shows a screenshot of the third page of the tutorial. It introduces the respondent to the sorting and filtering functions available on the site. When respondents are finished with this page, they will click “Continue” at the lower right hand side of the page.

Figure 3 Tutorial Page 3

Tutorial Page 4

(Phase 3)

Figure 4 shows the fourth and final screen in the tutorial. It introduces the respondent to how they will “choose” a doctor. When respondents are finished with this page, they will click “Continue” at the lower right hand side of the page. This will bring the respondent to the Start page of the SelectMD Web site.

Figure 4 Tutorial Page 4

Select MD Start Screen

(Phase 4)

Figure 5 is the Start screen for the SelectMD web site. All respondents will see this screen. All respondents will choose a type of doctor, enter their zip code, and select the number of miles they are willing to travel. This screen is intended to increase the feeling of reality on the site. However, the data and options subsequently presented do not vary with the choices respondents make on this screen.

When respondents have finished making choices, they hit the submit button in the lower right hand corner of the screen.

Figure 5 Start Page of the Web Site

Data Selection Screens

(Phase 4)

Figure 6 shows what happens after respondents hit the “submit” button on the previous screen. The respondent is taken to one of six Data Selection screens, depending upon the ARM of the experiment in which they are participating. The screenshot shown below displays the Data Selection screen respondents will see if they are in ARM #4, the most complex ARM of the experiment. All the Data Selection Screens are shown in Appendix A.

Tabs displayed on the Data Selection screen for ARM #4 are:

• Performance Overview

• Service Quality (CAHPS scores)

• Treatment Quality (HEDIS scores)

• Patient Reviews (Patient comments)

Respondents will select among the tabs. The next few pages will show what is displayed when the respondent selects each tab. Respondents may select any of the tabs in any order and can display only one tab at a time. They must select at least one tab in order to choose a doctor.

Figure 6 Data Selection Screen ARM #4

Performance Overview Screen

(Phase 4)

Figure 7 is a screenshot of what respondents see when they select the Performance Overview Tab. The screen is scrollable and includes a total of 12 Physicians for ARMS 1-4. The physicians are initially listed in alphabetical order, but they can be sorted, highlighted, or filtered by the respondents. For ARMS 5 and 6 the list will include 24 physicians. All physician names were created with a random name generator.

For ARM #4, the Performance Overview provides a summary of the measures of Service Quality (CAHPS), Treatment Quality (HEDIS) and Patient Reviews (Patient Comments) for 12 physicians. (See Figure 7.)

Figure 7 Performance Overview Screen

This page is scrollable

Only a portion of the page is shown here.

For ARMS with less data, the Performance Overview page will include summaries of the data that is included that ARM. For example, the Performance Overview for ARM #2 will include summaries of the Service Quality and Treatment Quality data, but no Patient Reviews information will appear. (See Figure 8.)

This page is scrollable. Doctors initially appear in alphabetical order, but can be sorted, highlighted, or filtered by respondents.

Figure 8 Performance Overview ARM #2

This page is scrollable

Only a portion of the page is shown here.

Service Quality Screen

(Phase 4)

Figure 9 displays a screenshot of what the respondents will see when the Service Quality tab is chosen. The Service Quality tab presents CAHPS data attributed to individual doctors. The data shown for individual physicians will not vary across ARMS; however, more data is presented for ARMS 5 and 6 because these ARMS list 24 physicians. ARMS 1-4 will list 12 physicians. The page is scrollable. Physicians are initially listed in alphabetical order, but can be sorted, highlighted or filtered by the respondent.

Figure 9 Service Quality Tab

This page is scrollable

Only a portion of the page is shown here.

Treatment Quality Screen

(Phase 4)

Figure 10 displays a screenshot of what the respondents will see when the Treatment Quality tab is chosen. The data shown on this screen is intended to mimic HEDIS data. Data for individual physicians will not vary across ARMS. However, for ARMS 1-4 the number of Physicians listed on the page is 12 and for ARMS 5 and 6 the number of Physicians listed is 24. The page is scrollable. Physicians are initially listed in alphabetical order, but can be sorted, highlighted or filtered by the respondent.

Figure 10 Treatment Quality Tab

This page is scrollable

Only a portion of the page is shown here.

Patient Reviews Screen

(Phase 4)

Figure 11 displays a screenshot of what the respondents will see when the Patient Reviews tab is chosen. The data shown for individual physicians is the same for all ARMS in which it is displayed. However, for ARMS 1-4 the number of physicians listed on the page is 12 and for ARMS 5 and 6 the number of physicians listed is 24. The page is scrollable. Physicians are initially listed in alphabetical order, but can be sorted, highlighted or filtered by the respondent.

The Patient Reviews are comments volunteered by patients. They are drawn from two sources. One source is actual patient comments found on physician rating web sites. These comments were modified and masked so that real names or identifying information is removed. The other source of comments is a set of fabricated comments created by the researchers using words and phrases with known emotional valence.

Figure 11 Patient Reviews Tab

This page is scrollable

Only a portion of the page is shown here.

Doctor Confirmation Screen

(Phase 4)

Figure 12 shows a screenshot of the screen that is displayed once a respondent has chosen a doctor. The respondent is asked to confirm their choice. If respondents confirm their choices, they are returned to the Knowledge Networks system to fill out the Post-Test questionnaire. If respondents do not confirm their choices, they are returned to viewing doctors.

Respondents must choose a doctor before they can go on the Post Test questionnaire. However, they may choose a doctor at any point in the experiment after they land on a tab. Respondents do not have to view all of tabs before selecting a doctor. Indeed, they may pick any tab, review the doctors, and make their choice at that point or they may look at other tabs if they wish.

Figure 12 Physician Choice Confirmation

Appendix A: Data Selection Screens for all Six ARMS of the Experiment

Arm #1 Tabs Displayed:

• Performance Overview

• Service Quality (CAHPS scores)

ARM Number	Title	Description
1	Control Arm	CAHPS Scores Only [One Drill-Down Page/Tab, 12 Physicians Listed]

Arm #2 Tabs Displayed:

• Performance Overview

• Service Quality (CAHPS scores)

• Treatment Quality (HEDIS scores)

ARM Number	Title	Description
2	Experimental Arm 1	CAHPS Plus HEDIS Scores [Two Drill-Down Pages/Tabs, 12 Physicians Listed]

Arm #3 Tabs Displayed

• Performance Overview

• Service Quality (CAHPS scores)

• Patient Reviews (Patient Comments)

ARM Number	Title	Description
3	Experimental Arm 2	CAHPS Plus Comments [Two Drill-Down Pages/Tabs, 12 Physicians Listed]

Arm #4 Tabs Displayed:

• Performance Overview

• Service Quality (CAHPS scores)

• Treatment Quality (HEDIS scores)

• Patient Reviews (Patient Comments)

ARM Number	Title	Description
4	Experimental Arm 3	CAHPS Plus HEDIS Scores Plus Comments [Three Drill-Down Pages/Tabs, 12 Physicians Listed]

Arm #5 Tabs Displayed:

• Performance Overview

• Service Quality (CAHPS scores)

• Patient Reviews (Patient Comments)

• Larger Choice Set

ARM Number	Title	Description
5	Experimental Arm 4	CAHPS Plus Comments; Larger Choice Set [Two Drill-Down Pages/Tabs, 24 Physicians Listed]

Arm #6 Tabs Displayed:

• Performance Overview

• Service Quality (CAHPS scores)

• Treatment Quality (HEDIS scores)

• Patient Reviews (Patient Comments)

• Larger Choice Set

ARM Number	Title	Description
6	Experimental Arm 5	CAHPS Plus HEDIS Scores Plus Comments; Larger Choice Set [Three Drill-Down Pages/Tabs, 24 Physicians Listed]

23 of 23