0990-0335_Attachment G_PilotTestReport

Attachment G

Pilot Test Report

Attachment G

PILOT TEST REPORT

The pilot study was conducted in May of 2008. A total of nine respondents completed the General Population questionnaire (see Attachment E-1), and nine tested the Physician Population questionnaire (see Attachment E-2). The pilot study was conducted to test the appropriateness and accuracy of the questions and response options included in the questionnaires and their corresponding protocols, and to accurately estimate respondent burden. All respondents were asked to provide feedback on various components of the questionnaire including: the introduction, the clarity of the question and response option wording, the ability to recall the requested information, and the length of the instrument.

Sample Selection

Pilot study respondents who completed the General Population questionnaire were chosen using unclaimed random digit dial numbers. Respondents who completed the Physician questionnaire (i.e., practicing physicians) were selected randomly from the AMA Masterfile.

Results

Overall, the questionnaires and their corresponding protocols were well received by respondents. None of the respondents found the questionnaire items “too sensitive” and there were no items that respondents refused to answer. However, some respondents noted that they were averse to being asked questions that dealt with race so early on in the interview. They suggested that a few less sensitive questions be introduced at the beginning of the questionnaire to help respondents acclimate themselves to the survey and ultimately help them transition to the race-focused questions.

In addition, respondents understood and were satisfied with the number of forced response options available to them. This was evident in the extremely low levels of missing data collected, and the low number of alpha responses (open-ended) or “Other,” responses which only appeared in one instance. Nevertheless, several respondents noted that certain items need a “mark all that apply” option for cases where more than one response applies. Also, a couple of respondents stated that at times they had trouble remembering all the response options for specific items, given that the items employ five different types of scales.¹

Finally, respondent burden was computed by measuring the elapsed time required to complete all instrument components for all responders. Exhibit 1 shows that the average time required to complete the questionnaire was 15 minutes for the general population, and 18 minutes for the physician population.

Exhibit 1: Average Elapsed Time to Complete the Questionnaire, by Population

Population	Elapsed Time to Complete Questionnaire (in minutes)									Average*
	1	2	3	4	5	6	7	8	9
General	15.6	17.3	11.3	16.8	18.4	12.3	17.3	11.9	17.2	15
Physician	16.0	16.2	16.4	18.4	19.0	19.1	18.8	20.6	21.1	18

*Average elapsed time rounded to the nearest whole number.

Although the average elapsed time to complete the instrument was three minutes more for the physicians than the general population, this difference is not attributed to the length of the instrument. The interviewers noted that physicians elaborated on their answers by providing the interviewers with an unsolicited rationale to their responses. Thus, it was this behavior that contributed to the three additional minutes to complete the interview and not to the number of questionnaire items. In addition, both the physician and general population questionnaires 80 percent of respondents indicated that the length of the interview was “just right.” The other twenty percent noted that the interview was a bit long. This finding validates the conclusion that the instruments do not place an overwhelming burden on respondents. Given that the physician questionnaire will be completed via pencil and paper and will not be administered via a phone interview, the project team determined that there is no need to make a modification to the instrument and protocol since the respondents will not have an opportunity to verbalize a rationale for their responses.

Recommended Modifications

1. Add a few “less sensitive,” non-race focused questions to the beginning of both questionnaires. Rather than developing new items that are not directly related to the analysis (which would add to respondent burden), the project team selected four existing items that focus on general knowledge of health disparities which were included toward the end of the instruments and moved them to the front. For the physician instrument, a couple of items were added that queries respondents about the type of practice and the percentage of minority patients they typically treat in one year.

2. Include a script that asks the respondents if they would like the response options repeated before answering. A script has been included in the general population protocol that reminds respondents that response options frequently change from item to item. An additional script was added to the protocol that asks respondents if they would like the response options repeated before they answer.

3. Add a response option “select all that apply,” to two items. For two items a new response option was included which allows respondents to mark multiple responses.

1 Scales include: 1) likelihood of experiencing an event (more likely, just as likely, less likely); 2) comparison of well-being (better off, worse off, just as well off); 3) assessment of quality (same quality, higher quality, lower quality); 4) perceived problem (major problem, minor problem, not a problem at all); and 5) level of health (excellent, very good, fair, poor).

3