passback 1

Health Resources and Services Administration: Responses to OMB Question on Chiropractic and Pharmacy Loan Repayment Program Surveys

Question 1. Confidentiality

Does HRSA have the statutory authority to provide assurances of confidentiality? If so, please cite it. Otherwise, the word “confidential” should not be used anywhere in the study (e.g. on the cover letters, on informed consent forms, in the supporting statement, etc.). Alternate words like “privacy” or “data safeguarding” can be used instead.

HRSA does not have the statutory authority to provide assurances of confidentiality as agencies such as NCHS do under Section 308(d) of the Public Health Service (PHS) Act and this project does not fall under 45 CFR 46; therefore, a Certificate of Confidentiality under Section 301(d) cannot be issued. As a result, language assuring confidentiality will be revised to remove use of the word “confidential” and replaced with wording to indicate that HRSA will not receive personal identifiable information and that all responses will be protected and kept private.

Question 2. Cover Letters

The letters state “Answers from all responding entities will be tabulated and published in aggregate form. However, individual responses may be used on occasion for illustrative purposes. In these instances, the individual clinic will not be noted.” Even if the names of clinics or clinicians will not be disclosed, will it be possible for the public to infer from these individual responses which clinics/clinicians provided these responses? Either way, this should be spelled out in the cover letter. If there will be no way to infer who provided individual responses, this will aid to improve assurance and increase response rates. If there will be a way to infer the identity of the respondents, even if the individual responses are anonymised, this should be disclosed.

No, individual responses used for illustrative purposes will not be identifiable. One criterion for selecting a response will be the inability to infer which clinics/clinicians provided the response. Revised letters that state this point more clearly are attached.

Question 3. Instruments/Letters

a. There are several items that are referenced in the supporting statements which OMB will need to see before it can consider this ICR complete. Can HRSA please send them (e.g., the written summary of the project, FAQs that will be sent to clinic administrators, etc.).

The written summary of the project and the answers to frequently asked questions are attached.

b. The PRA blurb, the OMB control number, and the expiration date should be printed on all questionnaires (and available for interviews should the respondent wan to know this information).

The Public Burden Statement, OMB control number, and expiration date will all be clearly displayed on all instruments. This information will also be available to any respondents during interviews. Each survey will have its own Public Burden statement to accurately display the estimated burden per respondent for that particular survey. The general format for the Public Burden Statement will read:

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless is displays a currently valid OMB control number. The OMB control number for this project is 0915-xxxx. Public reporting burden for this collection of information is estimated to average xx hours per respondent, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: HRSA Reports Clearance Officer, 5600 Fishers Lane, Room 10-33, Rockville, MD. 20857.

The OMB control number and expiration date will be displayed as:

OMB No. 0915-xxxx

Expiration Date:

c. Are the survey instruments for non-demo sites the same as for the demo-sites except that the section called “demonstration sites” will not be included in the survey for comparison sites?

Yes, the questionnaires are identical, with the exception of the section called “Demonstration Sites,” which is not included in the questionnaire of comparison sites.

Question 4. Data Collection

a. Is there some reason why a baseline study was not conducted at the outset of this program?

Conducting a baseline study was not feasible. HRSA was not able to award the evaluation contract until after the demonstration started. In addition, the National Health Service Corps (NHSC) loan repayment program only gives loan repayment awards to clinicians who either have a job offer or are currently employed by clinics located in underserved areas. Clinicians typically begin serving their NHSC obligation immediately after, or within a month or two of, receiving their loan repayment award. This short time period makes it very difficult to conduct a baseline study. Nevertheless, the evaluation will use secondary data from HRSA’s Uniform Data System (UDS) to conduct a pre/post analysis of the number of clients and encounters served per physician at the demonstration and comparison sites.

b. Have separate ICRs been submitted for the IDS and the 6 in-depth case studies referenced on page 3 of supporting statement A?

The UDS is the Uniform Data System utilized by HRSA’s Bureau of Primary Health Care for collecting information from funded health centers to ensure compliance with legislative mandates and to report on program accomplishments. The UDS has been in use for a number of years under OMB control number 0915-0193, and is currently approved through May 31, 2007. An extension request for additional years will be forthcoming for this activity, as it is one of the major data systems for the agency.

The supplemental information to be obtained from the unstructured consultations with 6 in depth case studies was not submitted as an ICR, as the number of respondents did not meet the threshold required to fall under the purview of the PRA. Under the definitions in the Paperwork Reduction Act of 1995 as codified in 44 U.S.C. 3502(3)(A)(i), the “term collection of information” means obtaining “answers to identical questions posed to, or identical reporting or recordkeeping requirements imposed on, ten or more persons, other than agencies, instrumentalities, or employees of the United States.” The in depth case studies did not involve identical questions, instead our questions varied by the type of demonstration clinic (chiropractor or pharmacist) and person interviewed (clinic administrator, medical director, physician assistant, and chiropractor or pharmacist). In addition, we used semi-structured protocols that allowed us to tailor questions to the site and to the respondent’s level of knowledge or experience. As a result of this variation, identical questions were asked of no more than 6 respondents, which is less than the minimum 10 required to fall under the purview of the PRA, and an ICR was not submitted.

c. Has HRSA considered audio-taping the interviews rather than relying on note takers? This might improve reliability of the data collection and also enable the interviewer to focus more on what the respondent is saying and allow the interviewer to asking probing questions more effectively.

We have elected not to audio-tape interviews because of concerns about the security of information and a desire to ensure the highest response rate possible. Through our past experience interviewing providers by telephone we have found that some providers are reluctant to participate if the interview is audio-taped, particularly if they anticipate providing negative input about a program. Mathematica Policy Research, Inc. (MPR), the contractor that will conduct these interviews, has been very successful at using this approach for these types of interviews. The interviewer can focus effectively on the interview, while the note taker is recording provider responses, and the provider is more likely to provide candid responses.

d. Is there some reason why all 30 demo sites will not be included in the user surveys? The only selection criterion listed is clinic size. Does that mean the sites that aren’t selected serve a small patient caseload?

When designing the clinic user surveys we considered several factors including the number of demonstration sites, number of completed questionnaires per clinic, and length of the survey field period. A key consideration was how each factor was likely to impact clinic burden and their willingness to participate. To minimize burden on the clinics, we decided to minimize the number of questionnaires per clinic and to keep the field period as short as possible. We anticipate fielding the clinic user survey for two to three weeks at each clinic. The challenge of obtaining 30 or 40 completed questionnaires of adult clinic users within two to three weeks will increase as the clinic size decreases. Smaller clinics serve fewer patients and the community health centers where NHSC clinicians serve typically treat large numbers of children. In addition, the survey of clinic users will be more burdensome on smaller clinics because their administrative staff typically have more responsibilities. Thus, we determined that the smallest demonstration clinics, where the field period would have to be longer and the burden would be greater, would not be included in the user survey.

We do not anticipate this selection criterion will a large effect on the results. The purpose of the clinic user survey is to provide information to answer the first research questions set forth in the legislation. The laws specifies that the evaluation study “the manner in which the demonstration project…has affected access to primary care services, patient satisfaction, quality of care, and health care services provided for traditionally underserved populations…” How the introduction of a chiropractor or pharmacist serving an NHSC loan obligation to a community health center or clinic in an underserved area affects these outcomes is not expected to be greatly different in small clinics. Nevertheless, our discussion of the results will clearly state that the results may not apply to the smallest community health centers or clinics.

e. Relatedly, because of the relatively small sample sizes for chiropractic sites, it will take at least an 18 percent difference because a difference is detected as significant. That is quite big. Wouldn’t HRSA want to know about the differences of a smaller magnitude?

Yes, HRSA would like to know about differences of a smaller magnitude, but the sample size reflects several considerations including burden and effective use of project resources. Our estimates of statistical power for the design were reviewed by a statistician at Mathematica Policy Research, Inc., Frank Potter, Ph.D. He found that the power estimates are conservative and the 18 percentage points is based on an alpha error of 1 percent. If we use the more traditional 5 percent alpha error, the minimum detectable difference for estimates of 50 percent (which represents the maximum level of variance) is approximately 9 to 11 percentage points. For other percentages, the detectable difference is even smaller. In addition, as we learn more about the actual estimates during the data analysis, the statistical power may be improved.

f. How will HRSA adjust for non-response bias?

We anticipate that despite our efforts to achieve high response rates to the clinic user survey, some clinic users who are asked to complete a questionnaire will refuse to participate and some who complete a questionnaire will not provide answers to all questions. These two types of refusals will bias the estimates if the survey and item response rates differ across different classes of patients (for example, if men are more likely to refuse the survey than women). To reduce the potential for this bias, we propose to use a post-stratification approach to compensate for differential nonresponse. At a minimum, we anticipate obtaining from the clinics some basic summary counts or proportion of patients seen during the sampling time frame by demographic factors (such as age, gender, and race). We will use this summary information for post-stratification of the counts of respondents to reduce the potential for nonresponse bias. In post stratification, the count of respondents is aligned with whatever demographic summary totals are available for the sampling period. For example, if proportionately more women than men complete the questionnaire, but the same number of women and men visited the clinic during the field period, then we would weight each male respondent upwards by the inverse of the proportion of male patients responding to the survey.

g. How are demonstration sites picked? For example, if they are picked precisely because they are unique in their needs and patient profile (e.g., they represent clinics that are in particularly underserved areas with high patient need with illnesses of a high severity), how will that affect the comparability of results between demo sites and comparison clinics?

The demonstration sites represent where the chiropractors and pharmacists decided to accept an employment offer. Some of these clinicians were employed at the clinics for a year or more before applying for an NHSC loan repayment award. The other clinicians had at least a job offer from the clinics when they applied for the demonstration. As a result, the demonstration clinics reflect the mutual selection of providers and clinics.

The comparison clinics will be selected from clinics that employ physicians, physician assistants, and nurse practitioners serving NHSC loan repayment obligations. These clinics also reflect the mutual selection of providers and clinics. When selecting comparison clinics we will match demonstration and comparison clinics on key characteristics such as clinic size, demographics of the client population, and Health Professional Shortage Area (HPSA) score. This HPSA score reflects a measure of the need for primary care providers in the area or population served by the clinic.

h. Is there a way to avoid duplicate responses for the clinic user survey? (e.g. if a patient comes twice to the clinic during the study period)?

The short field period of two to three weeks minimizes the likelihood that a clinic user will complete the questionnaire more than once. Nevertheless, to ensure we do not receive multiple questionnaires from a clinic user, patients will be instructed to mark the blank questionnaire as completed during an earlier visit to the clinic, place the marked questionnaire in the envelope provided, and return it to the reception desk before leaving the clinic. Our training of clinic staff distributing the questionnaires will also include these instructions.

Question 5. Analysis

a. What kinds of quality checks are available under the Questar software/program?

Questar is a company that provides high-speed optical scanning services. Questar processes all image scan questionnaires through an intelligent character recognition system from Captiva Software, one of the oldest and largest forms processing software companies in the United States.

Questar employs several quality checks and scanning completed questionnaires. They use a two-stage process for checking all programs for scanning, editing, and data output. During the first stage they conduct standard validation checks, such as using sample questionnaries to verify item coding accuracy, omitted and multiple responses, and conditional edit flags. They also manually inspect data output at this point for correct formatting and coding. The second stage is a beta test. Questar uses dummy forms for this test to check that program documentation is correct and all possible combinations and permutations of the input data are captured properly. A dump of the data created from the processing of the dummy forms is then hand-checked, character by character, against each scanned sheet to ensure the program is operating properly and that the editing programs are identifying errors.

In addition to these tests, they careful calibrate their scanners each day using test forms with dark and light marks from different writing instruments and a sample of completed questionnaires that contain extremely light marks. These tests ensure that the lightest possible respondent marks are captured without creating false reads from over enhancement of unmarked response areas.

Once they are satisfied that the programs and equipment are operating properly, each completed questionnaire is passed through a scanner. At that time an image of each page is displayed and checked for quality. During the scan each completed questionnaire is given a unique identifying number and the data are written directly to an electronic database. The processing number is also printed on the physical document by the scanner. Scanned questionnaires will be returned to us, along with the electronic database so that we can verify the accuracy of the scanning process.

b. What is the analytical framework HRSA will be using to analyze the interview data? The use of qualitative data software is good, but even the best software cannot analyze the data for you.

In addition to the three research questions set forth in the legislation, we use a conceptual framework to guide the development of our interview protocols and we will use this framework to guide our analysis of interview data. The conceptual framework in Figure 1 depicts the mechanism by which a chiropractor or pharmacist is likely to complement and enhance care provided by physicians and other non-physician primary care providers. The addition of a chiropractor or pharmacists to an underserved area is expected to increase the health system’s capacity to provide care, either to existing patients or new patients or both. In addition, these health professionals may introduce new programs—such as a drug therapy management program for individuals with a specific chronic condition—or new clinical services—such as spinal manipulation. By making more services available to more individuals, these enhancements to the health care system are expected to improve access to care, patient satisfaction, the quality of care, and health service utilization patterns.

FIGURE 1: CONCEPTUAL FRAMEWORK

Increases the capacity to provide care to existing patient base

A pharmacist or chiropractor recruited to serve an underserved area

Improvements in

Access to care

Patient satisfaction

Quality of care

Health service utilization

Introduces new capacity to provide care to a new group of patients

Introduces a new clinical service or program

c. How will the use of different sampling techniques at each clinic affect the analysis of pooled data for the user surveys?

We do not expect the differences in sampling technique at each clinic to present any serious problems in the analysis of pooled data. The sites can be treated in the statistical analysis as strata, analogous to blocks in an experimental design. The site-level estimates will reflect the sampling process used at the specific site, primarily accounting for differences in the survey period and sampling rates during the survey period. The site-level estimates can be combined as in a stratified random sample.

Question 6. Report to Congress

Since this is a study involving a relatively small number of study sites, the reporting of findings should be tempered appropriately.

We agree, the presentation of findings must acknowledge that the results are not necessarily generalizable to all community health centers in underserved areas that employ an NHSC chiropractor or pharmacist. Because the small number of demonstration sites makes it difficult to detect demonstration effects we will integrate several different analyses whenever possible and build a body of evidence to answer each research question. First, our discussion of results will clearly note this issue, particularly if we find there are no differences between demonstration and comparison sites. Second, to the extent possible, we will present any available benchmark data to provide context for understanding the results. For example, the Agency for Healthcare Research and Quality (AHRQ) has sponsored research on the treatment of back pain and there is a body of independent research on health services for back pain treatment and patient satisfaction. Third, the results of the analysis of survey data will be discussed in the context of what we learned from the surveys of medical directors and the chiropractors and pharmacists.

We designed the clinic user survey and the comparison group design to help answer the first research question set forth in the law, how the demonstration “affected access to primary care services, patient satisfaction, quality of care, and health care services…” Qualitative information and research methods will be used to answer the other two research questions about the affect on the health professional shortage area designation and the feasibility of permanent inclusion of chiropractors and pharmacists in the NHSC loan repayment program. Table 1 illustrates how we will build a body of evidence to support the findings from the survey of clinic users. For example, we will estimate regression-based differences in the probability of receiving care at the clinic for the treatment of back pain that control for patient and clinic characteristics and other observable factors. These results will then be further analyzed in the context of the information about how the chiropractors contributed to the expansion of services and how they were integrated within the clinics.

TABLE 1

Illustrative Example of Building a Body of Evidence

	Demonstration Sites	Comparison Sites	Statistical Significance of Difference
Regression-based analysis
Probability of receiving care for treatment of back pain at the clinic	X	X	p-value
Contributions made by NHSC chiropractors, as reported by medical directors
Able to serve additional patients	Percent	Not applicable	Not applicable
Able to serve a new population not served before	Percent	Not applicable	Not applicable
Able to offer a broader array of services	Percent	Not applicable	Not applicable
Added a new clinic service	Percent	Not applicable	Not applicable
Expanded an existing clinic service	Percent	Not applicable	Not applicable
Implemented a new community-oriented health care program	Percent	Not applicable	Not applicable
How chiropractors are integrated at sites, as reported in Survey of NHSC Chiropractors and in-depth case studies
NHSC Chiropractor:
Co-management of patients
Participation in interdisciplinary care teams

In addition to combining the qualitative and quantitative analyses, during the study design phase we formed an expert panel to review and provide input on the study’s approach to answering the research questions set forth by legislation. This panel met twice to discuss the demonstration, the research questions, data collection instruments, and analysis plans. Once the data collection phase ends and preliminary data analyses are completed, the expert panel will be convened again to review the results and discuss their presentation. The purpose of this intensive review prior to reporting the results to Congress is intended to ensure findings are presented appropriately. The members of this panel are listed below.

Betty Chewning, Ph.D.

Associate Professor

School of Pharmacy

University of Wisconsin-Madison

Judith Cooksey, M.D., M.P.H.

Associate Professor

Department of Epidemiology and Preventive Medicine

University of Maryland-Baltimore

Gary Gaumer, Ph.D.

Assistant Professor

Health Administration

Simmons College

Christine Goertz, D.C., Ph.D.

Director of Clinic Research

Samueli Institute

Eric Hurwitz, D.C.

Assistant Professor in Residence

Department of Epidemiology

UCLA School of Public Health

George McClelland, D.C.

Chairman of the Board

American Chiropractic Association

Freda Mitchem

Director

Systems Development and Policy Administration

National Association of Community Health Centers, Inc.

Perry Pugno, M.D.

Director, Division of Medical Education

American Academy of Family Physicians

Jon Schommer, Ph.D.

Associate Professor

Department of Pharmaceutical Care and Health Systems

University of Minnesota

Steve Wilhide

Former Executive Director

National Rural Health Association

Question 7. Pretest

When will the pretests occur? If there are going to be less than 9 respondents, does that mean there will be 4 or 5 respondents for each test clinic?

Pretests of the surveys of NHSC chiropractors and pharmacists have been completed. We found that these interviews last approximately 30 to 35 minutes as planned and that respondents had no difficulties understanding the questions.

The surveys of clinic users and medical directors have considerable start-up costs, even for a pre-test, and we determined that the best use of project resources is to conduct pretests immediately before the field period begins. This approach allows us to test our survey procedures, make adjustments to these procedures based on the results, and then ramp up our survey operations without any significant delays. This approach is feasible because the majority of our survey questions are taken from national household surveys and are well tested.¹

We plan to conduct a pretest with 4 or 5 patients at one demonstration clinic that hired a chiropractor, and another pretest with 4 or 5 patients at a demonstration clinic that hired a pharmacist. We will then conduct a pretest with 4 or 5 medical directors at demonstration clinics.

1 Many of the questions in the clinic user survey are from the Consumer Assessment of Healthcare Providers and Systems (CAHPS) questionnaires. We also drew some questions from the National Health Interview Survey (NHIS) and Medical Expenditure Panel Survey (MEPS).

10