REVISED SSA_0925-0589 OMB Passbacks 6 27 14

Supporting Statement Part A

Request for Generic Clearance:

QUESTIONNAIRE COGNITIVE

INTERVIEWING AND PRETESTING (NCI)

(OMB #0925-0589, Expiry 4/30/2014)

This is a revision of a currently approved submission.

Changes are indicated in yellow highlights.

Contact Information:

Gordon Willis, PhD.

Cognitive Psychologist

Applied Research Program

Division of Cancer Control and Population Sciences

National Cancer Institute

9609 Medical Center Drive, Rm 3E358,

Bethesda, MD 20892-9762

Phone: 240-276-6788

Fax: 240-276-7906

willis@mail.nih.gov

REVISED: June 27, 2014

Table of Contents

Table of Contents ii

List of Attachments iii

A. JUSTIFICATION 1

A.1. Circumstances Making the Collection of Information Necessary 1

A.2. Purpose and Use of Information Collection 3

A.2.1. Development and Testing of Specific Survey Questionnaires 3

A.2.2. Research on the Cognitive Aspects of Survey Methodology….. 7

A.2.3. Research on Human-Computer Interfaces/ Usability 8 A.2.4. Pilot Household Interviewing 9

A.2.5. Formative Research involving self-report 11

A.3. Use of Improved Information Technology and Burden Reduction 13

A.4. Efforts to Identify Duplication and Use of Similar Information 13

A.5. Involvement of Small Businesses and Other Small Entities 15

A.6. Consequences of Collecting the Information Less Frequently 15

A.7. Special Circumstances Relating to 5 CFR 1320.5 15

A.8. Comments in Response to the Federal Register Notice and Efforts to

Consult Outside Agencies 15

A.9. Explanation of Any Payment of Gift to Respondents 16

A.10. Assurances of Confidentiality Provided to Respondents 18

A.11. Justification of Sensitive Questions 21

A.12. Estimates of Annualized Burden Hours and Costs 21

A.13. Estimates of Total Annual Cost Burden to Respondents and

Record keepers 24

A.14. Annualized Costs to the Federal Government 24

A.15. Explanation for Program Changes or Adjustments 25

A.16. Plans for Tabulation and Publication and Project Time Schedule 26

A.17. Expiration Date Display Exemption 27

A.18. Exceptions to Certification 27

LIST OF ATTACHMENTS FOR SUPPORTING STATEMENTS

ATTACHMENT 1: Background and History of Cognitive Interviewing

ATTACHMENT 2: List of Approved and Currently Pending Generic Sub-studies from 2011-2014

ATTACHMENT 3: Description of cognitive testing: Willis, G.B. (2005). Cognitive Interviewing. In S.J. Best & B. Radcliff, Polling America: An Encyclopedia of Public Opinion, pp. 92-98. Greenwood Press: Westport, CT.

ATTACHMENT 4: Consultants both Within and Outside of the National Cancer Institute

ATTACHMENT 5: Letter from NIH Privacy Act Officer

ATTACHMENT 6: Sub-study #1 - Reliability of Computer Adaptive Tests (CAT) Study for the NIH-SSA Collaboration to Improve Disability Determination

Sub-study #1_Mini-Supporting Statement A

Sub-study #1_Mini-Supporting Statement B

All Behavioral Health Items

All Physical Function Items

CAT Instrument Screenshot

IRB Review

VR12 Legacy Instrument

Consent

This is a request for OMB to approve the revision of the generic collection titled, “Questionnaire Cognitive Interviewing and Pretesting” for an additional three years of data collection. For many surveys and self-report-based data collection efforts, it is advantageous to the government if development follows a pretesting sequence equivalent to that used at National Center for Health Statistics or the Census Bureau. For example, the Health Information National Trends Survey (HINTS: OMB No. 0925-0538) has undergone multiple cycles of cognitive testing to refine both the questionnaire, and supporting materials such as advance letters and brochures. The types of activities covered by this Generic request include: 1) Survey material development and pretesting based on cognitive interviewing methodology and use of focus groups , 2) Research on the cognitive aspects of survey methodology, 3) Research on computer-user interface design for computer-assisted instruments, also known as Usability Testing, 4) Pilot Household interviews are pilot tests (either personal, telephone, or Web-based) conducted with respondents using professional field interviewers; and 5) Formative research that depends on the use of interviewing techniques to develop products such as research priorities or expert consensus on best practices. Formative research has been increasingly used to develop new data collection instruments using psychometric procedures, including Computerized Adaptive Testing (CAT). Test-retest reliability testing can also be used as a type of formative research in the development of questionnaires, software applications that depend on self-report, and other measurement instruments.

A. JUSTIFICATION

A.1. Circumstances Making the Collection of Information Necessary

Data collection for this project is authorized under Section 410 (42 USC 285) and 412 (42 USC 285a-1) of the Public Health Service Act. The National Cancer Institute/Division of Cancer Control and Population Sciences (NCI/DCCPS) is requesting a three year clearance with terms similar to that previously granted under OMB No. 0925-0589. DCCPS staff will submit individual or bundled sub-studies under this generic. Sub-studies will include a description of the methods, participants, instruments, and incentives, to be reviewed by OMB on a case-by-case basis.

Cognitive research techniques - now commonly referred to as cognitive interviewing, developed subsequent to a seminar conducted on the Cognitive Aspects of Survey Methodology (CASM) in 1983-1984. A major conclusion of a second CASM seminar (CASM II, held in 1997) was that cognitive testing of survey questionnaires has become a standard practice in the Federal government, as well as in private and academic survey research organizations. This work has proven to be effective for enhancing the quality of Federal survey data for over twenty years (Attachment 1). In recent years, the science of survey development, evaluation, and pretesting has come to include a range of activities, including cognitive interviewing, focus groups, and usability testing. The term intensive interviews will be used as a general term to refer to thee activities. A further pretesting approach is based on observational, field-based pilot household interviewing. In particular, the use of the Pilot Household Interview, often supplemented by behavior coding, and normally conducted subsequent to cognitive testing, was introduced by NCHS researchers in the 1990s. Pilot Household Interviews have been supported first under OMB No. 0920-0222, and then under the two cycles of NCI Generic Pretesting Clearance (OMB No. 0925-0589).

DCCPS/NCI modeled its initial Generic clearance request after that used by NCHS and approved by OMB (No. 0920-0222), but tailored to NCI activities. The original submission of “Questionnaire Cognitive Interviewing and Pretesting” was approved in May, 2008, for three years (0925-0589). For a second cycle of activities, an Extension was requested and approved by OMB on 4/19/2011, expiring 4/30/2014. Attachment 2 contains a summary of projects that have been conducted under this clearance during the three-year cycle to expire on 4/30/2014, or are currently pending a draft or review. The current submission constitutes a request to approve this revision, with (a) a decrease in requested burden hours, and (b) a change in the methodology to include formative research involving self-report that applies interviewing techniques similar to those used for intensive interviews.

A.2. Purpose and Use of Information

The purpose and use of collecting this information fall into five categories—the first three of which involve cognitive/intensive interviews, and the fourth relies on pilot testing with behavior coding:

2.1 Development and testing of specific survey questionnaires

2.2 Research on the cognitive aspects of survey methodology

2.3 Research on human-computer interfaces/usability

2.4 Pilot household interviewing

2.5 Formative research involving self-report

A.2.1. Purpose and Use of: Development and Testing of Specific Survey Questionnaires

Cognitive interviewing techniques focus on the use of both think-aloud and on verbal probing. Generally, a volunteer participant is asked to think aloud as he/she answers the questions, and the specially-trained interviewer probes the participant for additional information. The interviews are generally semi-structured; the interviewer uses draft survey questions as a guide, but probes as needed to determine the participant's interpretation of the questions and the recall, and decision processes used to arrive at his/her answers. This method uncovers ambiguities in question wording, participant strategies for dealing with vague questions, or questions that ask for information that is not readily available (see Attachment 3 for more detailed information).

A variant of this approach is retrospective cognitive interviewing (or debriefing), in which the interviewer first administers the entire draft questionnaire, and then reviews the questions and responses with the participant, probing for reactions to the questions. While less information is gained about the recall techniques used by participants, there is also less deviation from the natural flow of an interview. In some cases, interviews are audiotaped or videotaped (assuming the subject provides appropriate consent), so that the interviewer can concentrate on probing the responses and can analyze content of the collected information later.

Occasionally, focus groups, typically of 5-12 individuals, are used to discuss general concepts that survey questions will focus on. Individual interviews are generally preferable to focus groups for evaluating specific questions because respondents usually respond to surveys individually, and the group dynamic associated with a focus group format can have a strong influence on interpretations and responses¹. However, focus groups can sometimes assist questionnaire designers in understanding the relevant background circumstances of various groups of people, and this information can be used to craft questions that better match respondent experiences².

This data collection primarily uses cognitive interviewing methodology to identify and correct questionnaire flaws, e.g., questions which are vague or ambiguous, cannot be answered readily or accurately by the participant, or otherwise contribute to the non-sampling errors of the survey. Attachment 3 contains a short description that outlines the contributions of the cognitive interviewing methodology to the questionnaire development process, the methods used at various stages of the process, and the strengths and limitations of this methodology. The methods used will vary depending on the stage of development of the various data collection instruments to be studied. When questions have been used successfully in earlier surveys, testing will evaluate whether the questions function appropriately in the new context. In cases where there is evidence that previously developed questions were not reliable or valid, more extensive evaluation will be conducted. The most extensive questionnaire development activities will be applied to untested draft questions and undeveloped lists of data objectives.

Although we cannot anticipate all of our pretesting activities over the next several years, especially because plans for future surveys depend on budget and establishment of priorities, our planning currently anticipates the following specific survey projects:

1. The Health Information National Trends Survey (HINTS) (OMB No. 0925-0538). NCI’s HINTS survey has been conducted in 2003, 2005, and 2007-8 primarily by telephone. For the planned fourth cycle of HINTS, administration is primary be through mail. Further, starting in 2012, the survey has adopted a continuous administration model in which a fairly static core will be used for each annual cycle, with supplementary question sets that are modified for each annual cycle. HINTS is meant to provide dynamic information concerning issues of current interest in the field of health communication. This generic clearance plans to conduct pre-testing of HINTS mailed materials and new questions.

b) Tobacco Use Supplement to the Current Population Survey (TUS-CPS) (OMB No. 0925-0368). The National Cancer Institute periodically sponsors the administration of a large-scale population-based tobacco survey within the CPS, which is itself conducted by the Census Bureau for BLS. For the TUS, NCI varies the topics of emphasis as new data collection needs arise (for example, in 2003 NCI developed the TUS Special Cessation Supplement to track tobacco quitting behaviors; the 2006-7 TUS-CPS provided population surveillance data on tobacco use; and the 2010-2011 cycle represented a combination of 2003 and 2006-7 versions). The next cycle of the TUS-CPS is planned for 2014-2015, and we anticipate conducting pretesting under the current clearance request. Because new or modified questions developed for the CPS require careful pretesting, we will also conduct cognitive interviewing of draft forms of new questions that are developed, both to determine the functioning of these items, and to ensure that they function appropriately in the context of existing items.

c) Health-Related Quality of Life/Quality of Care Assessment

Especially given the advent of the Patient Protection and Affordable Care Act (ACA), a major challenge to NCI and to epidemiologists and researchers is the development of self-report items for use by patients, and members of the general public, relating to Health-Related Quality of Life (HR-QOL). More specifically to NCI, it is vital to assess self-assessed quality of cancer care from medical providers (i.e., Patient-Reported Outcomes, or PROs). Development of research priorities, and of survey item banks that include items that are reliable, and that do not place significant burden on respondents, has presented a consistent challenge to practitioners and researchers interested in the development of items and scales that represent these concepts^3. Cognitive testing, and pretesting in general, is useful in the development of these items, especially in conjunction with quantitative methods that involve psychometrics and statistical analysis. As such, NCI staff is actively engaged in the development of item banks of questions focused on Quality of Life/Care, to be used across a range of future investigations and surveys.

We anticipate conducting focus groups, cognitive testing, and perhaps pilot testing activities, that focus specifically on the qualitative aspects of these items, and that in particular assess whether they are clear and interpreted similarly across individuals, across patient groups, and across racial/ethnic/cultural groups. The results of these pretesting activities may not be targeted toward a specific survey, but rather toward the establishment of scales that are appropriate for incorporation into future studies. These efforts should greatly facilitate the development of new surveys, as much of the requisite evaluative work will have already been conducted.

d) Other questionnaire testing and development: In addition to the specific questionnaire testing and development activities listed above, we anticipate that NCI staff will perform testing of other questionnaires that require development over a short time-frame. Because the requests may arrive with little advance notice, we cannot presently specify the nature of these questionnaires.

The interviews for questionnaire development activities (a) through (e) above will usually be conducted using procedures described in Attachment 3. Interviews are normally conducted at NCI facilities (e.g. at the NCI Usability Laboratory) or in contractor offices (such as the Westat cognitive laboratory). If we are unable to obtain adequate numbers of individuals from particular population subgroups (e.g., the elderly, or those who have specific health problems), we will attempt to make arrangements with organizations such as centers for the elderly, or service organizations for persons with specific health conditions, in order to interview participants at outside locations. Usually, cognitive interviews will be conducted in the mode intended for the survey (face-to-face, telephone, self-administered, or web-based). For a telephone interview, we will either make arrangements to call the participant at home, or to conduct the interview in our laboratory, but calling the participant from another room for questionnaire administration, followed by face-to-face debriefing.

A.2.2. Purpose and Use of: Research on the Cognitive Aspects of Survey Methodology

The second major purpose of data collection is to conduct several cognitive research projects:

1. Cross-cultural research. NCI endeavors to conduct basic studies of how best to measure increasingly important factors associated with the cross-cultural aspects of survey response, such as measurement of respondent acculturation. Such questions are key to understanding language and cultural issues that impact access to care, and health in general. NCI staff intend to conduct cognitive testing of these questions, or of newly developed alternative approaches, of Hispanics who are of varying levels of acculturation to U.S. society, to determine whether the questions (in English, and in Spanish) are both understandable, and obtain the types of information intend. Further, we anticipate the development of acculturation questions appropriate as well to Asian or other respondents, and plan to be prepared to develop and evaluate appropriate measures.

b) General methodological research: DCCPS/NCI staff constantly evaluate and refine cognitive interviewing methods, especially in order to respond to changes such as the conversion from telephone-based interviewer administration, to paper-based self-administration, associated with the widespread adoption of Address-Based Sampling (ABS). Further, NCI staff regularly conduct applied research on questionnaire design issues, such as the optimal wording for measures of complex concepts related to cancer risk factors and related issues (e.g., physical activity; diet; tobacco use; cancer screening). For the next cycle of pretesting under this generic clearance, DCCPS staff plan to continue research on methods evaluation and general questionnaire design research. We envision that over the next three years, NCI staff and contactors will work collaboratively with survey researchers from Universities and other federal agencies to define and examine several research areas, including, but not limited to: 1) differences between face-to-face and telephone-based cognitive interviewing, 2) effectiveness of different approaches to cognitive interviewing, such as concurrent and retrospective probing, and 3) social, cultural and linguistic factors in the question response process.

Procedures for each of these studies will be similar to those applied in the usual testing of survey questions. For example, different versions of a survey question will be developed, and the variants then administered to separate groups of participants in order to study the cognitive processes that account for the differences in responses obtained across different versions. These studies will be conducted either by NCI personnel or by contractors who are trained in cognitive interviewing techniques. The results of these studies will be applied to our specific questionnaire development activities in order to improve the methods that we use to conduct questionnaire testing, and to guide questionnaire design in general.

A.2.3. Purpose and use of: Research on Human-Computer Interfaces/Usability

The third major purpose of this data collection is to conduct research on computer-user interface designs for computer-assisted and Web-based instruments, which is often referred to as usability testing. This research examines how survey questions, instructions, and supplemental information are presented on computer instruments, especially over the Internet, and investigates how the presentation affects the ability of users to effectively utilize these instruments. Authors of computer-assisted instruments make numerous design decisions: how to position the survey question on a computer screen; how to display interviewer instructions to respondents; the maximum amount of information that can be effectively presented on one screen; how supplemental information such as “help screens” should be accessed; whether to use different colors for different types of information presented on the screen; and so on.

Sometimes issues arise in computer-assisted and Internet based survey instruments, involving the human-interface design, ease of use, comprehension, privacy, quality of on-line help and efficiency of screen organization⁴. For questionnaires that involve Web-administration, we will rely on usability testing techniques that are very similar to those used for cognitive interviews, but that involve a more technologically-intensive environment (e.g., administration via laptop or desktop computer).

Research has shown that these decisions can have a significant effect on the time required to administer survey questions, the accuracy of question-reading, the accuracy of data entry, and the full exploitation of resources available to help the user complete his or her task⁵.

Usability testing has many obvious similarities to questionnaire-based cognitive research (described in Section A.2.1), as it focuses on the ability of individuals to understand and process information in order to accurately complete survey data collection. It is also somewhat divergent in the sense that dynamic visual information is of greatly increased importance. In particular, it also focuses more heavily on matters of formatting and presentation of information than does traditional cognitive testing.

It is anticipated that this generic clearance will again be actively used by DCCPS/NCI staff who want to test the usability of their web pages or possibly, the usability of software or mobile device applications (e.g., Tablets and Smartphones). In 2010, three of the five approved sub-studies for this generic clearance involved usability testing that was funded through NIH Set-Aside funds, and for the second (most recent) clearance cycle, usability testing was conducted for one internet-based data collection development project (ASA-24) and a DCCPS/ARP website. It anticipated that there will be continued interest and possibly an increase in the use of usability testing at NCI.

A.2.4. Purpose and Use of: Pilot Household Interviewing

The activities described above – cognitive interviews, focus groups, and usability studies – can together be terms as intensive interviewing methods. The fourth major purpose of data collection differs from these, as it instead applies to unobtrusive field-based questionnaire evaluation techniques, with respect to future surveys conducted either within the household or over the telephone. Although the cognitive interviewing methods described above are effective for identifying problems that are missed by traditional field pretests, they are limited because they do not administer questions under actual field-based interviewing conditions. Further pilot tests conducted within selected households, and that may include up to 200 households, depending on the size of the survey to be fielded later, are a vital complement to cognitive interviews. Survey methodologists may conduct small-scale pilot household interviewing at various points in questionnaire development – not for purposes of field data collection and computation of survey estimates -- but rather as a vital step in the questionnaire development sequence. Also, as time and resources allow, researches apply behavior coding to record the behaviors of both interviewers and survey participants in such interviews to allow for systematic analysis⁶. These activities have been used successfully to develop the questionnaires used in previous Federal questionnaires, such as the NCHS National Health Interview Survey (NHIS) Supplements, and the NCI-sponsored Tobacco Use Supplement to the Current Population Survey (OMB No. 0925-0368). NCI therefore proposes to make use of similar activities in the development of future cancer-related surveys.

Generally, pilot interviews for face-to-face surveys are conducted in the participant's household; pilot interviews for telephone surveys are conducted over the telephone; and those for mail-based self-administration are sent via the mail. Professional field interviewers (often, contactor staff interviewers who are enlisted for the tested survey) normally conduct the interviews. A subset of these interviews is usually observed by a survey professional (a Federal staff member or member of the contract staff). As the interviewer conducts the pilot household interview, the observer compiles notes regarding respondent misunderstandings or difficulties in answering, or questions that interviewers have difficulty administering, which help to identify potential question revisions. This practice allows testing of types of individuals who do not ordinarily volunteer for cognitive interviews, and who may be more typical of the usual survey participant; it also provides information collected under field conditions, and is collected early enough to be useful for questionnaire design decisions.

The tested questionnaires may be pilot-tested either individually or in combination, depending on developmental status of the instruments, the appropriateness of combining them, and their overall length. It is envisioned that for any single pilot test, four or five professional field interviewers will conduct a total of approximately 100-200 pilot household interviews. There are three components to the proposed form of testing: a) a limited number of interviews on a draft version of the questionnaires that are conducted using household participants, by NCI and other staff trained in observational techniques; b) inclusion in the questionnaires of two different versions of particular questions, to gather information relevant to determining which version functions better in the field environment; and c) to make such determinations, the systematic coding of interviewer-respondent interactions.

Although no pilot testing of this type was conducted under the first two three-year cycles of this clearance, for the next cycle, we anticipate conducting such tests as part of the development of the Health Information National Trends Survey (HINTS), described above. Details of this testing will be submitted as separate sub-studies under this request.

A.2.5. Purpose and Use of: Formative Research involving self-report

NCI-sponsored research projects often involve the development of products or approaches that are guided by the collection of self-reported information, sometimes by experts or individuals who are highly skilled or have a particular knowledge in a certain field. For example, the development of materials and approaches to a smoking-cessation program may incorporate input from experts that is obtained through systematic collection, synthesis, and re-collection of information from the same individuals. A frequently used specific methodology is the Delphi Method (developed by Rand Corporation) in which respondents answer questionnaires in two or more cycles, and subsequent to each cycle, a facilitator produces a summary of input from the previous round. The respondents then revise their earlier responses in light of the replies of other members of the panel. Other examples of formative research are the use of interviews concerning long-past events (e.g., exposure of previous U.S. servicemen to radiation due to atomic weapons tests) to estimate lifetime radiation doses; or interviews of professionals to develop a list of research priorities in the area of assessment of patient-reported outcomes. In each of these cases, small numbers of interviews are conducted (generally no more than 30), but using systematic procedures that are consistent with those methods used in the development of survey questionnaires, such as cognitive testing. For this reason, NCI proposes the inclusion of such formative research projects within the current clearance request.

A final type of formative research that has been increasingly used to develop new data collection instruments involves the conduct of psychometric testing, especially in conjunction with cognitive and usability testing (e.g., mixed-methods research). Researchers have begun to develop applications such as Computerized Adaptive Testing (CAT), which relies on knowledge of the ‘strength’ with which items measure a specific concept to selectively administer these in a way that reduces respondent burden. For example, in assessing physical functioning, the system can be designed to administer a more challenging item (e.g. about whether one can run a mile) only if ‘easier’ items (e.g., walking a quarter mile) are answered affirmatively. Because these applications tend to involve logic that is more sophisticated than that included in the usual system of skip patterns and sequencing instructions included in questionnaires, they require pretesting that ascertains whether respondents provide coherent sets of responses that satisfy the assumptions underlying the CAT models. Such testing can be conducted using qualitative techniques, such as cognitive testing, that examine the underlying reasons respondents provide for their patterns of responding. Similarly, psychometric approaches such as the use of test-retest reliability examine whether, over a limited time period for which the ‘true score’ of a survey item is not expected to change (usually 1-3 weeks), respondents provide matching answers for two independent administrations of the same items. If responses are not consistent, this constitutes evidence of measurement error in the overall statistical estimates obtained, and call for a re-examination of item interpretation and recall demands (See Proposed Sub-study #1 included with this request – Attachment 6).

Overall, the five major activities outlined above have well-demonstrated practical utility. As a result of pretesting, questionnaires may produce clearer materials, and therefore less response error, than would occur in the absence of this testing. Thus, users of NCI data, in both Federal agencies and in the general health research community, will be less likely to be misled by erroneous statistical results. This assertion is supported by almost thirty years of experience in using these techniques, and has been supported by findings presented at many statistical and research related conferences, and published in scientific journals such as Public Opinion Quarterly and Applied Cognitive Psychology. The practical utility of Pilot Household Interviewing has also been supported in findings reported at an annual meeting of the American Statistical Association and the American Association for Public Opinion Research. Further evaluation of the efficacy of these methods will be ongoing.

A.3. Use of Improved Information Technology and Burden Reduction

Pretesting will be conducted using most recent modes of survey data collection, including CAPI/CASI, touch-tone data entry (TDE), the Internet, mobile devices (e.g., Tablets and Smartphones or other modes applied to Federal data collections.

A Privacy Impact Assessment (PIA) is not needed because there is no information technology (IT) system associated with this information collection. Should this change, or should an individual generic sub-study have an IT system, then they will pursue a PIA at that time.

A.4. Efforts to Identify Duplication and Use of Similar Information

NCI staff work closely with staffs of other Federal agencies that conduct pretesting activities, including (a) the QDRL at NCHS, (b) The Census Bureau’s Center for Survey Methods Research), and (c) the BLS Collection Procedures Research Laboratory. Further, participation of key NCI staff in the Interagency Response Error Group (IREG), which meets quarterly to discuss questionnaire development and pretesting among Federal statistical and research agencies, ensures that pretesting is conducted in a manner that is coordinated across agencies. NCI staff will avoid the conduct of pretesting activities that are duplicative of those of the other agencies. In most cases this is true simply because the various agencies evaluate different survey questionnaires (generally those they develop and administer). However, where surveys do overlap between Agencies (such as the National Health Interview Survey, where responsibility for Cancer-related Modules is shared between NCI and NCHS), NCI staff will collaborate regularly with the other Agency to produce a pretesting plan that is optimal for purposes of timely and efficient production of results, in a way that minimizes respondent burden. This could involve sharing of pretesting responsibilities, but in a coordinated, non-duplicative manner. In some cases parallel testing of the same questions could be conducted across Agencies, for purposes of comparison of pretest results. In particular, as cognitive techniques have been applied, there has been a paucity of research concerning the reliability of obtained results⁷; parallel testing between agencies provides an important methodological bridge to provide an answer to this persistent question.

Overall, NCI questionnaire design researchers also maintain very close contact with other experts in the field of questionnaire development in the academic survey community, in the health sciences field, at the Census Bureau, BLS, NCHS, the General Accounting Office, the National Science Foundation, and the Energy Information Administration. From these contacts, it is clear that no other projects that duplicate the current proposal are now underway.

A.5. Involvement of Small Businesses and Other Small Entities

It is possible that representatives of small businesses will be interviewed as part of testing involving medical offices and other establishments, for provider/physician surveys. For these interviews, the organization/office will be approached in the same manner as the individuals we normally recruit; we will ask the organization to identify the appropriate staff members with whom to conduct the cognitive interview.

A.6. Consequences of Collecting the Information Less Frequently

Most projects involve one-time data collection activities; however, there are a few projects in which pre- and post- tests (or test-retest – See Attachment 6) are necessary to conduct to confirm validity and/or accuracy of the instrument. In such cases, a justification will be provided to explain the rationale for conducting information collection more than once. There are no legal obstacles to reducing the burden.

A.7. Special Circumstances Relating to 5 CFR 1320.5

There are no special circumstances.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies

A 60-Day Federal Register notice for this collection was published on January 3, 2014, (Vol. 79, p. 402) and allowed 60 days for public comment. No public comments were received.

Other agencies and individuals: Some of the topics selected for NCI surveys may be developed in conjunction with other agencies: For example, the Tobacco Use Supplement to the Current Population Survey TUS-CPS) has in the past been developed in conjunction with the Office of Smoking and Health within the Centers for Disease Control and Prevention. These agencies may be involved in development of draft questionnaires. Further, NCI staff maintain ongoing connections with staffs of NCHS and of the Census Bureau, concerning the development and pretesting of the NHIS, the TUS-CPS, and other joint efforts.

Researchers who have special interest and expertise in the research areas explored will be contacted as necessary (see Attachment 4). Consultation with representatives of those from whom data will be collected will take place in the form of interviews with volunteers to determine the feasibility of collecting the needed data, the most promising approach for data collection, and general attitudes about the participants which might influence data collection.

A.9. Explanation of Any Payment or Gift to Respondents

For intensive forms of interviews (that is, cognitive interviews, focus groups, and usability tests), participants generally receive an incentive, for several reasons:

 Eligibility criteria for participants are usually specific. Some of these criteria are determined by the subject matter of the survey (e.g., questions may be only relevant to people with certain health conditions). The more specific the subject matter, the more difficult it is to recruit eligible participants.

 Intensive forms of interviews require an unusual level of mental effort, as participants are asked to explain their mental processes as they hear the question, discuss its meaning and point out any ambiguities, and evaluate the acceptability of response options that are provided.

 Participants are usually asked to travel to a cognitive laboratory or other testing location, which involves transportation and parking expenses. Many participants incur additional expenses due to leaving their jobs during business hours, making arrangements for child care, etc.).

For a standard interviewing project, in which one-hour intensive interviews are conducted at NCI or contractor offices and eligibility requirements are of average complexity, participants will receive up to $50.00. The incentive may be reduced to an amount no lower than $30.00 if the interview is of shorter duration, or does not require the participant to travel to NCI. Higher incentives may be requested for particularly difficult recruitments.

Focus groups or individual interviews of highly-compensated professionals, such as physicians, are vital for development of provider surveys, but normally require incentive amounts of up to $75 for an hour-long interview. Some interviews involve complex topics or questionnaire materials that require more time (up to 40 additional minutes, or 1.67 total hours); or that require up to 40 minutes (.67) hours of record retrieval or review of survey-related materials. Incentives were previously used as part of the development of three NCI provider/physician surveys (where n<10): The National Survey of Primary Care Physicians’ Cancer Screening Recommendations and Practices (OMB No. 0925-0562); The National Survey of Energy Balance-Related Care Among Primary Care Physicians (OMB No. 0925-0583); and The Survey of Physician Attitudes Regarding the Care of Cancer Survivors (OMB No. 0925-0595). Additionally, a sub-study that was recently approved under this generic and is titled, “Cognitive Interview for Multidisciplinary Care (MDC) Survey (OMB No. 0925-0589-07), was approved for $150 to recruit physicians to participate in both an hour long cognitive interview and complete a 30 minute survey.

It is important to offer an incentive sufficient to attract the full range of needed participant types for intensive interviewing projects. Inadequate participant recruitment limits the effectiveness of the questionnaire evaluation. In addition, we face competition from other laboratories (public and private) in a highly saturated research area. Sometimes our advertisements are adjacent to ads offering participants substantially higher incentives for the same commitment. Requests and justification for incentives will be included in each individual collection submission.

For activities that are meant to resemble the usual household interview – in particular, Pilot Household Interviewing -- participants will not receive remuneration, given that the methods are meant to replicate usual field conditions, for which survey respondents are normally not provided remuneration. Further, Pilot interviews are conducted in respondent households, so no travel is required, and also generally take far less than the one hour required for more intensive pretesting activities such as the cognitive interview.

A.10. Assurances of Confidentiality Provided to Respondents

Information collected under this clearance will include personally identifiable information (PII) in the form of names and contact information. Names and contact information will be used only for purpose of subject recruitment for pretest interviews (e.g., focus groups, cognitive interviews, usability tests); will not be associated with substantive data collected during interviews; and will be destroyed immediately after the interview. The NIH Privacy Act Officer has reviewed the work scope of this proposal to determine whether the Privacy Act is applicable to this data collection. Additionally, the NIH Privacy Act Officer will be asked to review the protocols of each sub-study under this generic clearance to ensure that NCI adheres to privacy requirements (see Attachment 5). Individual sub-studies will also submit, as part of their sub-study memo, a consent form and a plan for ensuring that identifiers are not retained as part of the research.

Activities covered under this clearance are generally considered to be Exempt from IRB review at NIH. NCI Staff for each sub-study, as well as submitting a specific request for Privacy Act review and an assessment of Privacy Impact Assessment, will submit a request for Exemption to the NIH Office of Human Subjects Research (OHSR). If OHSR determines that the data collection involves non-exempt activities, and should be reviewed by the NCI Special Studies Institutional Review Board (SSIRB), then NCI staff will develop appropriate materials, and will not contact human subjects for that project until the SSIRB has approved the research Protocol. If a contractor is involved in human subjects research activities, that contractor’s IRB will also review that testing project.

For Pilot interviews, whether of household and telephone participants, standard operating procedures regarding informed consent specific to the survey being tested will be slightly modified to reflect participation in the testing of survey questions, rather than participation in the actual survey to be field-administered. Again, no PII will be maintained by the sub-studies carried out under this clearance.

Plans for assuring confidentiality, and for safeguarding of collected information, will be specified by each sub-study submitted under this clearance. In general, the key NCI staff or contractor project director is responsible for safeguarding schedules, consent documents, audiotapes and videotapes, questionnaires, and cash incentives to participants.

Upon completion of a cognitive interview, the interviewer is responsible for the questionnaire, any notes written on other pieces of paper, and if created, the interview recording. The interviewer is instructed to lock all materials in his/her work area until all analysis is completed. Recordings are labeled by participant identifier number, date, time, and project title. No other identifying information is labeled on the recording. Once analysis is completed, interviewers are responsible for returning questionnaires and recordings to the project coordinator, who stores the materials in a locked location. No participant names or other identifying information is included in any reports, publications, or presentations of interview results.

Sometimes interviewers must travel to establishments or individuals’ homes in order to conduct interviews⁸. It is the interviewer’s responsibility to take necessary steps to ensure privacy, confidentiality, and safeguarding of materials. Generally interviews will be conducted in private rooms with a closed door. If no private room is available, the participant can select a private area and the interviewer will judge whether the area is sufficient for ensuing privacy. If the interviewer determines that the area is not private and/or soundproof enough, and no alternative area can be provided, the interview is postponed. For those surveys conducted in the participant’s home, the interviewer requests in advance that the participant arrange for privacy. However, interview location within the home is the choice of the participant.

As for other interviewing formats, focus group confidentiality and informed consent procedures will be specified for each sub-study. In focus group settings, participants are interviewed together and can hear each other’s comments, statements, and questions. Participants are told in their initial telephone screening interview that they will be participating in a discussion group with other volunteers. Before the group discussion begins, participants sign a consent form which is tailored to specify that they will be participating in a focus group. Generally, the interviewer (usually referred to as a Moderator when conducting a focus group) will instruct the group that the information discussed will be kept secure, to the extent of the law by NCI staff. Participants are asked to respect the privacy of the other participants and not to reveal to others what was discussed by the group.

When contractors are employed to collect data as part of NCI projects, they are contractually bound by NCI confidentiality provisions, and must submit documentation concerning ­their safeguarding practices to NCI prior to data collection. For any data collection activity, the contractor’s Institutional Review Board will review the data collection plan, and will complete the review and approval process before contact with human subjects is made.

A.11. Justification of Sensitive Questions

There will be no personally identifiable information retained for the generic sub-studies under this clearance. Additionally, the questionnaires currently proposed for study generally do not contain questions that are highly sensitive in nature. There are exceptions, however, and item sensitivity cannot always be predicted (note that one purpose of pretesting is to assess level of sensitivity). Therefore, a major purpose of cognitive and other pretesting of such questions is to determine means for fashioning them – and explanations for their administration-- in such a way that sensitivity is minimized, and responses are valid.

A.12. Estimates of Annualized Burden Hours and Costs

A. Hour Burden Estimates

The average annual participant burden is estimated to be 1200 hours, or a total of 3,600 hours over a three-year approval period (Table A.12-1). This is same burden estimate that was requested during the initial 2011 submission cycle for this project; a change request was approved in the interim which increased the total burden hours to 6,000, which is the current total hours available through the April, 2014. Estimates are based mainly on the practice of conducting one-hour interviews with participants. The estimates cover the time that each participant will spend communicating with the individual serving as the initial point of contact, in answering screening questions and survey questions and, in some cases, being debriefed following the interviews concerning their thoughts about the tested items. In rare cases, the burden may be more than one hour (although not more than 1.5 hours). Because the time per response is expected to vary, we will select the final sample size for each project in such a way that the total burden hours do not exceed the estimate listed above. For focus groups, the usual amount of time required is 90 minutes (1.5 hours) with instructions and ancillary paperwork processes taking an additional 15-25 minutes.

For all intensive interviewing activities (cognitive, focus group, or usability) conducted at NCI or contractor offices, the time required to travel to the location of the interview is not included in the current burden estimates, because distances and modes of transportation are unknown. Retrieval of records by participants is usually not required, although it is possible that validation of data at some point may require participants to check records, probably those kept at home or at physician offices. All estimates are based on conferring with NCI staff who coordinate or lead the relevant questionnaire development activities, and on previous small-scale pretesting activities (involving samples of less than nine) that have been conducted under NCI auspices.

Table A.12-1 Estimated Burden Hours Over Three-Year Approval Period

Type of Respondents	Number of Respondents	Number of Responses per Respondent	Average Burden per Response (in hours)	Total Burden Hours Over 3 Years
Physicians, Scientists and similar Respondents	1,200	1	75/60	1,500
Experts in their Field (e.g., smoking cessation)	600	1	75/60	750
Administrators/ Managers	600	1	75/60	750
General Public	1,200	1	30/60	600
Total	3,600			3,600

B. Annualized Costs to Respondents Over the Three-Year Approval Period

Each sub-study will identify the most appropriate respondents to complete the cognitive interviewing, survey development, usability testing and Pilot household surveys. As a result, the respondents for each category may include: the general public, physicians and a hospital administrators, scientists, and experts in their field. As a result, it makes estimating the respondent cost difficult. The table below is our best estimate of the respondent costs. The hourly wage rate for physicians, scientists and similar respondents is the mean hourly wage rate for health professionals. This includes the hourly rates of the experts in their fields who are also scientists. These estimates are based on the following data from the Bureau of Labor Statistics: the physician and scientist wage rate was obtained from http://www.bls.gov/oes/current/oes_nat.htm#29-0000 occupation code 29-1069; the wage rate for administrators /managers was obtained from http://www.bls.gov/oes/current/oes_nat.htm#11-0000 with the occupation title Operation Specialist Managers occupation code 11-3000; and the general public rate was obtained from the http://www.bls.gov/oes/2013/may/oes_nat.htm#00-0000 occupation title “All occupations” occupation code 00-0000.

Table A.12-2 Respondent Costs Over Three-Year Approval Period

Type of Respondents	Number of Respondents	Total Annual Burden Hours	Hourly Wage Rate*	Total Annual Cost
Physicians, Scientists, and similar Respondents	1,200	1,500	$90	$135,000.00
Experts in their Field (e.g., smoking cessation)	600	750	$90	$67,500
Administrators/Managers	600	750	$55	$41,250.00
General Public	1,200	600	$22	$13,200.00
Total	3,600	3,600		$256,950

A.13. Estimates of Total Annual Cost Burden to Respondents and Record keepers

There are no direct costs to record keepers or respondents other than their time to participate.

A.14. Annualized Costs to the Federal Government

The cost to the government consists mainly of respondent incentive costs (Table 14.1), and the salaries of Federal and contract staff who will: (1) recruit, schedule, and interview volunteer participants, and (2) assist in the analysis of the results and recommend changes in questionnaire wording. The Federal employees grade/step are listed and are based on the 2014 Salary Table for the Washington-DC Area. The total annual cost of the Federal employees is estimated to be $143,089. The annualized project costs are estimated to be $409,089 (Table A.14-2), which amounts to $1,281,267 over the three-year information collection period.

Table A.14-1 Annualized Incentive Costs to the Federal Government

Type of Respondents	Number of Respondents	Incentive	Total Annual Cost
Physicians, Scientists, and similar Respondents	1,200	$75	$90,000
Other Respondents	2,400	$30	$72,000
Total	3,600		$162,000

Table A.14-2 Annualized Costs to the Federal Government

Annual costs for NCI staff to plan, conduct, and analyze the outcomes of the questionnaire development activities:	Managerial (Grade 12/1 -$75,621)	0.50 FTE	$37,811
	Professional (Grade 14/1- $106,263)	0.50 FTE	$53,132
	Support (Grade 9/1-$52,146)	1.00 FTE	$52,146
Payment, under contract, for assistance with pretesting activities/research			$100,000
Travel costs (mainly local travel):			$1,000
Materials for conducting household Interviews			$1,000
Recruitment materials: (flyers, newspaper advertisements):			$2,000
Respondent Incentives			$162,000
TOTAL			$409,089

Travel costs: Most data will be collected in NCI or contractor office space. However, it will be more efficient in certain instances to hold interviews with individuals at other locations (homes, health centers, elderly centers), which will involve minor travel costs. Further, household interviews will require limited numbers of in-person interviews in participant households. Household interviews will be done locally, in order to limit travel costs, unless there is a compelling reason to do otherwise (for example, if participants critical to the study can be interviewed only at a distant location).

A.15. Explanation for Program Changes or Adjustments

This is a request for a revision of OMB#: 0925-0589, which is a considered a program change as a result of action by the Agency. A change request was approved in February 2013 to increase the burden from 3,600 to 6,000. However, this request was made in anticipation of a study that has since been canceled. Therefore we are decreasing the anticipated burden level in the current request to 3,600 hours, to match the amount originally received through the 2011 submission request. The methodology remains essentially the same as was proposed in the previous generic clearance, except to include formative research involving self-report which uses interviewing techniques to develop products such as research priorities, or expert consensus on best practices.

A.16. Plans for Tabulation and Publication and Project Time Schedule

This clear­ance request is for questionnaire development activities to be conducted prior to field administration, and for developmental work that will guide future questionnaire design. The majority of intensive interviewing investigations will be analyzed qualitatively. The survey designers and lab staff serve as interviewers, and use detailed notes and transcriptions from the in-depth cognitive interviews to conduct analyses. The results of these investigations will be used primarily to develop reliable survey instruments and methods. For the Pilot Household Interviewing activities, qualitative and quantitative analysis will be performed on samples of observational data from household interviews, in order to determine where additional problems occur. In particular, Behavior Coding, which involves the systematic tabulation of several categories of interviewer behavior (e.g., erroneous reading of question) and respondent behavior (e.g., request for repeat or clarification; providing an uncodeable response) will be used to assess problems with survey questions. Because NCI is using state-of-the-art questionnaire development techniques, NCI staff and collaborating contract staff will produce methodological papers which may include descriptions of response problems, recall strategies used, and quantitative analysis of frequency counts of several classes of problems that are uncovered through the cognitive interview and observation coding techniques.

The time schedule of activities will in most cases be linked to the development cycle of surveys to be supported by the pretesting activities described (e.g., the HINTS or TUS-CPS). All activities will be conducted over the period CY 2014 – 2017.

A.17. Expiration Date Display Exemption

All surveys and interview guides will display the OMB number and expiration date.

A.18. Exceptions to Certification

No exceptions to the Certification for Paperwork Reduction Act submissions are requested.

1 Fowler, F.J. Jr., (1995), Improving Survey Questions: Design and Evaluation, Applied Social Research Methods Series Volume 38, Sage, Thousand Oaks, CA.

2 Krueger, R. A. (1994). Focus groups: A practical guide for applied research, Sage, Thousand Oaks, CA.

3 Lipscomb, J., Gotay, C., & Snyder, C., 2005, Outcomes Assessment in Cancer, Cambridge.

4 Couper, M., 1999. The Application of Cognitive Science to Computer Assisted Interviewing, in Sirken, M., Hermann, D., Schechter, S., Schwarz, N., Tanur, J., and Tourangeau, R. (eds.), Cognition and Survey Research, Wiley, New York, pp. 277–300.

5 Couper, M., 1999, The Application of Cognitive Science to Computer Assisted Interviewing, in Sirken, M., Hermann, D., Schechter, S., Schwarz, N., Tanur, J., and Tourangeau, R. (eds.), Cognition and Survey Research, New York, Wiley, 277–300.

6 Fowler, F. J., & Cannell, C. F. (1996), Using behavioral coding to identify problems with survey questions. In N. Schwarz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research, San Francisco, Jossey-Bass, pp. 15-36.

7 Beatty & Willis, The Practice of Cognitive Interviewing, Public Opinion Quarterly, 2007.

8 Off-site interviews fall into two categories. First, on rare occasions, participants who are recruited for lab interviews request that the interview be conducted in their home. Second, we occasionally conduct establishment studies where a visit to the business location is pertinent to data collection.