AttachmentF_Pretest_Memorandum

ATTACHMENT F
PRETEST MEMORANDUM

MEMORANDUM

600 Maryland Ave., SW, Suite 550
Washington, DC 20024-2512
Telephone (202) 484-9220
Fax (202) 863-1763
www.mathematica-mpr.com

TO:

Sylvia Trent-Adams, HRSA/HAB

FROM:

Julie Ingels, Mathematica Policy Research

SUBJECT:

HIV Clinician Workforce Study: Clinician and Practice Surveys:
Pretest Memorandum with Recommendations

DATE:

10/21/2011

A. OVERVIEW OF HIV CLINICIAN WORKFORCE STUDY PRETEST
The Department of Health and Human Services (DHHS) is embarking on a 24-month
quantitative HIV clinician workforce study to provide the Health Resources and Services
Administration (HRSA) and other federal and state agencies with national and regional estimates
of the number of primary care clinicians providing medical care to people living with HIV or
AIDS in the United States, as well as projections of the magnitude of the shortage or surplus of
HIV-related primary care clinicians through 2015. Among other components of the study are two
national surveys: one of HIV clinicians (physicians, nurse practitioners, and physician assistants)
and one of HIV practices (clinical practices or facilities within which a sample of these clinicians
provide care). The primary purpose of the surveys is to collect information that will be used to
develop HIV-specific input parameters for a model that will forecast the surplus or shortage of
clinicians by 2015. We expect to complete 3,500 surveys with clinicians and 350 surveys with
office managers or administrators responding in behalf of the practices.
We include the pre-tested versions of the two surveys in Appendix A-1 (clinician
questionnaire) and Appendix A-2 (practice questionnaire). Unlike household surveys where the
respondents may have experience with, for example, a given insurance type, but are otherwise
simply sampled citizens, the respondents to the clinician survey pretests are experts in HIV or
AIDS clinical topics addressed by the questionnaire. Respondents to the practice survey pretests
are experts in the management of HIV or AIDS practices. This fact led us to treat the pretest as
more of a structured discussion with the experts than as a means of identifying troublesome
questions, concepts, or skip errors as would normally be the case during a pretest.
1.

Pretest Purposes

The purpose of the pretest was to determine from our expert respondents’ perspectives
whether (1) the questionnaires’ structures and questions adequately address the goals of the
surveys by providing information that would enable DHHS to forecast the shortage or surplus of
clinicians, (2) the question language was clear and understandable, (3) the section and question
flows was smooth, and (4) the response categories made sense.
2.

Pretest Sample

To identify an adequate sample for both clinicians and practice administrators, we consulted
with HRSA. The project officer emailed an invitation letter to 7 clinicians and 3 practice
An Affirmative Action/Equal Opportunity Employer

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
3
administrators, some of whom had previously participated in the HIV clinician workforce expert
consultant meeting for this project. Mathematica followed up by telephone recruiting potential
pretest respondents. We found we needed to network to identify additional practice
administrators for the pretest.
3.

Conducting the Pretests

After recruiting the pretest respondents, we emailed the appropriate surveys to the
consenting respondents. They completed the surveys using the paper version of the instrument,
and either emailed them back to us before the telephone interview or read us their responses
during the pretest interview. The pretest was conducted by telephone by the project director, Dr.
Boyd Gilman, and the survey director, Ms. Julie Ingels. (Dr. Gilman conducted all 10 pretests
and Ms. Ingels assisted in 7.) We took notes and audio-recorded all interviews and referenced the
audios when writing up the notes. We completed 7 clinician pretests (with the HRSA-identified
clinicians) and 3 practice pretests (one with a HRSA-identified practice administrator, one with a
HRSA-identified practice administrator’s designee, and one with a practice administrator
identified through networking). Because our pretest respondents were experts in the subject
matter, we discussed all of their responses and comments in an effort to arrive at suggestions for
wording and other substantive changes.
After completing the pretests and revising the instruments to reflect the participants’
comments, we sent the revised questionnaires to two pretest participants for a second review.
Both pretest participants responded that the surveys were greatly improved, easier to understand
and answer, appropriate to clinic practices, and relevant to the survey goals.
4.

Pretest Timing

On average the clinician survey took between 15 to 20 minutes to complete and the practice
survey took between 20 and 30 minutes to complete. One of the practice respondents asked two
other staff members to provide her information about recruitment and retention numbers and
salary ranges.
5.

Changes to HIV Workforce Study Survey Instruments

The final survey questionnaires, after pretesting and quality review, are included as
Attachments B-1 (final clinician instrument) and B-3 (final practice instrument) to the Part B
supporting statement. In the following section, we discuss the changes made to the clinician and
practice questionnaires following pretesting of both instruments. We first present the pretest
respondents’ queries and/or suggestions and, then, in italicized bold, our recommended changes.
In addition, we made general changes throughout the instruments to improve the language in
response to the comments from our internal quality reviewer following the pretest.

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
4
B. PRETEST COMMENTS ON AND REVISIONS TO CLINICIAN SURVEY
1.

Introduction
(1) Several pretest respondents reported that they did not read the introductory section
and immediately proceeded to answer the survey questions. One respondent
remarked that she would have read the introduction if the section had been titled
“Instructions.” She recommended that the introductory section provide clearer
information on the privacy of the information; acknowledge that the respondent’s
identity, clinics, and data will be kept private; and explain that the responses would
be presented in aggregate form only. We renamed the introduction “Instructions,
Please Read.” Because the introductory language was already stated in the
advance materials, we felt this would not result in loss of important background
information. We therefore omitted the background information and changed the
section to be instructional. We also included detailed information on privacy.

2.

Eligibility Screener
(1) The pretest eligibility screener listed a number of health professions at S2; pretest
respondents advised that “the care provided by geriatric and pediatric health
professionals is fundamentally different from the care provided to adults with HIV or
AIDS” and recommended we drop pediatric and geriatric specialties. We decided to
omit these specialties from the survey frame.
(2) The pretest respondents also felt that responses to primary area of specialty would
differ for physicians and nonphysician clinicians, and suggested we separate the
response categories. We included specialty areas for nurse practitioners and
physicians assistants and included skip instructions to move respondents to the
correct set of responses.

3.

Section A: Background
(1) Pretest respondents pointed out that the highest professional health care degree
earned might be, for instance, a Ph.D., rather than a clinical degree. As this study is
concerned with clinical practice, we changed questions 1 and 2 to refer to “highest
clinical degree.”
(2) Another respondent recommended more specificity in Q4. He felt we should ask
about the “initial” decision to pursue a career providing care to HIV or AIDS
patients. After discussion with other pretest respondents, we agreed to add the word
“initial” before “decision” in Question 4 to refer to the initial decision to practice
HIV or AIDS medicine.
(3) Several respondents mentioned that one reason why they began practicing HIV
medicine was that no one else in their community was doing it. They recommended

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
5
adding this as a new response category to Question 4. As most pretest respondents
agreed this was an important response, we added: “No one else in my practice or
community was willing to treat patients with HIV or AIDS” as a new response
category.
4.

Section B: Hours Spent in Patient Care
(1) Respondents felt that it would be cognitively easier to answer questions 5 (total
hours in patient care) and 6 (hours spent in HIV or AIDS patient care) if they were
combined into a single question. We combined Q5 and Q6 to help the respondents
think about the relationship between total hours and hours spent in HIV or AIDS
care.
(2) Pretest participants also felt that respondents would need more guidance about the
difference between clinical versus non-clinical care (this became a recurrent theme in
both surveys). We crafted definitions of clinical and non-clinical care to ensure all
respondents were answering these questions using the same definitions. This
definition was used for both the clinician and practice surveys.
(3) The pretest respondents felt that question 7 was cognitively challenging because it
asked clinicians to divide their total hours in care into nine activities that might occur
over multiple practice sites. We worked with the pretest respondents to collapse the
nine activities into four broader categories that reflected the way the pretest
respondents reported thinking about their time.

5.

Section C: Patient Load
(1) This section was challenging for respondents because they felt that the definition of
“total patient load” was unclear. Did it include only the panel of patients they
managed or did it include any patient who was assigned to their open appointment
slots? To clarify the context of the patient load questions, we started Section C with
a new question (new Q7) about whether the clinician independently managed
his/her own panel of patients or saw all patients assigned to him or her.
(2) Pretest respondents felt that the questions in section 3 were needed a reference
period. To ensure consistency in answers, we assigned a time frame to all
questions: “During the past 12 months...”
(3) Several respondents felt we should increase the number of digits in the patient
number response categories from 3 to 4. We reformatted responses to accept 4
digits.
(4) One pretest respondent was concerned that we did not adequately identify what we
intended by “serious mental illness.” After discussion with other pretest
respondents, we added the clarification “(including affective disorders such as
depression and anxiety).”

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
6
6.

Section D: Patient Environment (renamed Practice Settings for clarity)
(1) Pretest respondents felt that the list of settings where clinicians practice (old Q16)
included settings that were unlikely to appear as a primary practice site for our
survey frame, such as inpatient and emergency departments. We decided to delete
“inpatient settings” and “emergency departments” to streamline the list of practice
settings that had to be read and considered and to exclude unlikely primary
practice site choices for our survey frame (new Q15).
(2) At old Q18, the pretest respondents argued that we should try to learn what type of
Ryan White funding was received by the clinician or at his/her practice. We decided
to add an additional question if the respondent answered YES to the clinic
receiving Ryan White funding. We now ask (new Q17a) what type of Ryan White
funding was received and provide a “Don’t Know” response to accommodate
clinicians who might not be aware of the specific type of Ryan White funding their
clinic receives.
(3) Old Q19 (“Is your primary practice part of an integrated health care system
composed of multiple provider organizations that share resources and offer a
comprehensive continuum of care?”) was confusing to several pretest respondents
because certain phrases are not consistently used, especially the reference to
“integrated health system” and “multiple provider organizations.” Because these
terms were unclear to some pretest respondents, they arrived at divergent
interpretations. We reworded question 19 (new Q18) to employ more descriptive
language: “Is your primary practice an independent, freestanding clinic or
practice or is it part of a larger integrated health care organization composed of
multiple units that share administrative and clinical resources?” Pretest
respondents agreed the new wording was clearer and less open to multiple
interpretations.
(4) Several respondents recommended that we add a question about length of patient
encounters and wait times. We felt this recommendation would add useful
information to the clinician study, but that it fit better into Section D under
practice setting. As a result, we added these questions in Section D as new Q19 and
new Q20.

7.

Section E: Practice Management
(1) It was suggested that we did not need the lead-in questions to the electronic medical
record and scheduling procedures questions. We agreed that the lead-in questions to
Q21a (old Q20) and Q22a (old Q22) were unnecessary and dropped them. Instead,
we decided to ask the electronic medical record and scheduling questions directly
(new questions are now renumbered Q21 and Q22).

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
7
(2) One pretest respondent felt we should clarify that the focus of old Q20a.2 was
specifically helping the clinician adhere to recommended clinical guidelines. To
clarify that the focus is on clinician adherence, we changed new Q21.2 to read:
“To increase clinician’s adherence to clinical guidelines…”
(3) The term “share management” in old Q22 was confusing and needed clarification.
To clarify “share management,” we reworded Q22 to read more descriptively: “At
your primary practice, do you provide HIV-related care to patients for whom you
are not the primary care physician? That is, the patient’s primary care physician
addresses non-HIV related issues, while you address the HIV-related issues?”
(4) The term “delegate” (old Q23) has regulatory/legal meaning and should be clarified.
In addition, delegating to staff with less training was perceived as possible offensive
to nonphysician clinicians. We decided to retain the word ‘delegate,’ but changed
the question wording to be more direct and descriptive of what delegation entails
and also to avert the awkwardness of referring to nonphysician clinicians as less
highly trained. We changed the question to read: “Does your primary practice
devote time and resources to delegating clinical tasks for the care of patients with
HIV or AIDS to nurse practitioners, physician assistants, registered nurses, or
others?”
(5) At Q24, the respondents asked that peer counseling be added to the list of services
designed to promote long-term self-management. Since peer counseling seems to be
a major type of service to promote long-term self-management, we added (new
Q25), “peer counseling” to the list of services designed to promote long-term selfmanagement.
8.

Section F: Future Plans
(1) A skip pattern needed clarification and we clarified it with the addition of an
instruction.
(2) One respondent noted that the phrase “retiring from the health profession” does not
necessarily mean retiring from clinical work. One could leave clinical practice to join
a pharmaceutical firm or the government and still be in the health profession. Thus,
we changed “retiring from the health profession” to the more precise and clearer
phrase “retiring from clinical practice.”

9.

Section G: Your Perception about Workforce Capacity
(1) The major topic of discussion in Section G involved question 29 (new Q32) asking
the clinician to rate 15 strategies for effectiveness in meeting an increase in demand
for HIV clinicians. The response list was variously found to be not relevant, too long,
and including categories that do not add much insight. After careful discussion with

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
8
the pretest respondents, we reduced the list of strategies from 15 to 8, by
eliminating those that were deemed least relevant or appropriate for this study.
(2) The pretest respondents also remarked that it might be useful to add a question about
what would encourage young clinicians to adopt a career in HIV treatment. As this
question would fit well into the inquiry, added new question (Q33) about factors
would have the greatest influence on the decisions of clinicians to pursue a career
providing medical care to patients with HIV or AIDS today.
10. H: Your Demographic Characteristics
(1) One respondent commented that some clinicians might not want to provide their
salary information. We added a “refused” option to the salary question.
11. Section I: Your Contact Information
(1) No questions and no changes.
C. PRETEST COMMENTS ON AND REVISIONS TO PRACTICE SURVEY
The practice survey will be completed by respondents who can answer for an entire practice
rather than as an individual clinician. We expect the respondent to be the practice administrator,
office manager, or medical director, or a combination of these positions. We encourage the
respondent who is addressed on the survey envelope to seek help from other practice experts, if
necessary.
1.

Introduction
(1) Several pretest respondents reported that they did not read the introductory section
and immediately proceeded to answer the survey questions. One respondent
remarked that she would have read the introduction if the section had been titled
“Instructions.” She recommended that the introductory section provide clearer
information on the privacy of the information; acknowledge that the respondent’s
identity, clinics, and data will be kept private; and explain that the responses would
be presented in aggregate form only. We renamed the introduction “Instructions,
Please Read.” Because the introductory language was already stated in the
advance materials, we felt this would not result in loss of important background
information. We therefore omitted the background information and changed the
section to be instructional. We also included detailed information on privacy.

2.

Eligibility Screener
(1) Pretest respondents found the practice screener confusing. After careful
consideration and discussion with the internal quality reviewer, we decided that

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
9
the confusing screener did not provide a viable means of excluding ineligible
practices and deleted it in favor of moving directly into the survey. Instead, we will
conduct a telephone screening with the sample member to confirm eligibility for
the survey.
3.

Section A: Practice Management
(1) There were no substantive comments on the practice management section. We
decided to delete the practice management section because it duplicates the
practice management section of the clinician questionnaire. Because we a drawing
our practice sample from the clinician sample, we should receive at least one
clinician survey for each sampled practice. Thus, we do not need to repeat
questions on the practice questionnaire that already appear on the clinician
questionnaire.

4.

Section B: Practice Staffing (Renamed Section A in the Final Questionnaire)
(1) Some pretest respondents felt that full-time equivalent (FTE) might be a difficult
concept to grasp for some respondents. In addition, several respondents said that the
formatting of the practice staffing questions was confusing. Some practices, as a
matter of policy, do not staff nurse practitioners or physician assistants. This made it
difficult to answer staffing questions without the response being ambiguous: do they
never staff these individuals or do they simply have none on staff right now? There
was no easy way to format the questions without the formatting becoming
cumbersome for a paper questionnaire. Because more than one person may be
needed to answer the survey questions, we decided to encourage the practice
respondent to reach out to other staff to help complete the survey. To address the
formatting problem, we added a new question (Q1) asking the respondent to
identify the kinds of health care professionals used by the practice to provide
patient care. This information will help us interpret answers to later questions
where we ask the respondent to indicate the FTEs or numbers of patients seen by
each clinician type and then ask for an indicator of ‘did not use.’
(2) Next, there was concern that certain questions did not adequately instruct the
respondent in how to distinguish between direct and indirect patient care activities.
We added a paragraph at the beginning of the section that instructs the respondent
in how to think about this distinction so that all respondents would answer using
the same definitions.

3.

Section C: Recruitment and Retention
(1) An issue discussed earlier reappeared for the recruitment and retention questions in
this section (new Q8a and new Q9). Respondents noted that we need to better
distinguish between, for example, “this practice does not use nurse practitioners to

MEMO TO: Sylvia Trent-Adams, HRSA/HAB
FROM:
Julie Ingels, Mathematica Policy Research
DATE:
10/21/2011
PAGE:
10
deliver care versus this practice does use nurse practitioners but is not recruiting any
nurse practitioners at the moment.” The addition of the new Q1 will help us
determine whether practices never use a certain type of clinician or is simply not
recruiting them at this time.
(2) Pretest respondents felt that certain questions needed a reference period. We added
“During the past 12 months…” to new Q6, Q8, Q9, and Q10.
4.

Section D: Perceptions of Workforce Capacity
(1) No questions and no changes.

5.

Section E: Patient Characteristics
(1) Pretest respondents noted that, for practices receiving Ryan White HIV/AIDS
Program funding, these responses can be taken from the most recent Ryan White
HIV/AIDS Program data reports (RDRs) and, thus, should be easy to answer for
some our sample. However, the RDRs ask for number of patients rather than
percentages of patients. We decided to ask the questions both ways (numbers and
percentages) and let the respondent decide whether to answer in percentages or
numbers (new Q22 – Q31).
(2) We add the question on relative value units to better measure severity of patient
caseload.

If you have any questions about any of these revisions, please direct them to Boyd Gilman,
project director, by telephone at (617) 301-8974 or via email bgilman@mathematica-mpr.com or
Julie Ingels, survey director, by telephone at (202) 554-7535 or via email at
jingles@mathematica-mpr.com.