Open Call OMB SSA submission

Mini Supporting Statement A for

Open Call Usability Testing

Generic Clearance for: Formative Research, Pretesting, and Customer Satisfaction of NCI’s Office of Communications & Public Liaison”

OMB No. 0925-0046-16

Expiry Date 04/8/2016

January 29, 2021

Silvia Inéz Salazar, MPA, Health IT Audience Research Laboratory Manager Analytics and Audience Research Branch Office of Communications & Public Liaison National Cancer Institute (NCI)/NIH 9609 Medical Center Drive, Suite 2E212 Rockville, MD 20850-9760 (240) 276-6631 salazarsi@mail.nih.gov

Dr. Anita Ousley Project Officer & Acting Branch Chief Analytics and Audience Research Branch Office of Communications & Public Liaison National Cancer Institute (NCI)/NIH 9609 Medical Center Drive, Suite 2E-306 Rockville, MD 20850-9760 (240) 276-6629 ousleya@mail.nih.gov

List of Attachments

Attachment Number 1: Lay Audiences Standard Screener (English)

Attachment Number 2: Healthcare Professionals Standard Screener (English)

Attachment Number 3: Lay Audiences Standard Screener (Spanish)

Attachment Number 4: Healthcare Professionals Standard Screener (Spanish)

Attachment Number 5: Moderator’s Guide (English)

Attachment Number 6: Moderator’s Guide (Spanish)

Attachment Number 7: Informed Consent Form (English)

Attachment Number 8: Informed Consent Form (Spanish)

Attachment Number 9: IRB Approval

Mini Supporting Statement A

A.1 Circumstances Making the Collection of Information Necessary

The National Cancer Institute (NCI), established under the National Cancer Act of 1937, is the Federal Government's principal agency for research on cancer cause, prevention, detection, diagnosis, treatment, and rehabilitation, and for the dissemination of information for the control of cancer. Current authorization for NCI's education and information dissemination activities is contained in Section 410 of the Public Health Service Act (42 USC § 285a-2).

NCI’s Office of Communications & Public Liaison (OCPL) is the main office within the NCI that provides communications expertise within the Institute and between NCI and a variety of organizations and audiences, including Congress, other executive agencies, state and local governments, scientific and medical communities and institutions, voluntary groups, the press, the general public, and cancer patients. The OCPL, which supports NCI’s priorities through activities that span NCI programs, specializes in the design, implementation, and evaluation of education programs over the entire cancer continuum, including prevention, screening, diagnosis, treatment, survivorship, and palliative care. The office also manages NCI initiatives that address specific challenges in cancer research and treatment. Health care providers, professional societies, patient groups, federal agencies, and the public are audiences for OCPL’s educational programs and products/materials.

Within OCPL, the AARB ensures that the National Cancer Institute’s (NCI) communication and education resources are appropriate, useful, and effective. AARB uses scientific methods to: identify the cancer-related needs of diverse audiences; inform the design and development of NCI resources; monitor audience trends; and, assess the impact of resources and activities. The research conducted in AARB helps ensure that NCI communication and education resources are appropriate, useful, and effective.

This NCI office is requesting approval of this sub-study under the “Formative Research, Pretesting, and Customer Satisfaction of NCI’s Office of Communications & Public Liaison” generic clearance (OMB No. 0925-0046-16, Expiry Date 04/8/2016).

Per, the scope of the “Formative Research, Pretesting, and Customer Satisfaction of NCI’s Office of Communications & Public Liaison” generic clearance (OMB No. 0925-0046-16, Expiry Date 04/8/2016), section A.1.:

“The National Cancer Institute’s (NCI) Office of Communications & Public Liaison (OCPL), Analytics and Audience Research Branch (AARB), conducts various types of research to identify and learn about target audiences.”

The proposed information collection is aligned with the scope of the generic by working to ensure that the materials produced by NCI can be understood and are well-received by intended audiences (Formative Research/Pretesting), and meet the satisfaction of NCI’s audiences

(Customer Satisfaction). Formative research/pretesting helps ensure that messages have the potential to be received, understood, and accepted by those for whom they are intended; while the customer satisfaction research helps NCI ensure the relevance, utility, and appropriateness of the many educational programs and products that the Office produces.

A.2 Purpose and Use of the Information Collection

NCI’s Office of Analytics and Audience Research (AARB) plans on conducting ongoing user-centered design research to better understand the needs of users of NCI’s flagship Web sites, Cancer.gov and Cancer.gov/espanol. Prior research with populations affected by cancer has refined NCI’s understanding of audience segments, their differing needs, and suggested improvements to the content, structure and presentation of standard Cancer.gov pages.

NCI AARB plans to schedule bimonthly Open Call usability testing starting in FY 2015 to inform research efforts for NCI’s digital communications products, as well as the Cancer.gov and Cancer.gov Español websites. These regularly scheduled usability sessions will provide an opportunity for any NCI web portal managers, designers, content managers, application designers, communications practitioners, researchers, or digital communications strategists to better understand the needs of the users. To reflect the diversity of the various types of populations that visit Cancer.gov in English and Spanish, participants will include a mix of lay audiences and healthcare professionals. Potential respondents will be recruited via telephone calls made by a participant recruiting vendor. The screening questions will be administered via telephone by the vendor. The purpose of the questions is to identify the type of audience each participant belongs to (Attachments 1-4). It will take approximately fifteen minutes for respondents to answer the screening questions via telephone. Once participants are selected and invited to participate, they will receive a one page consent document before the session begins (Attachments 7-8). Participants will review the consent document. If the participant agrees, they will sign the consent. After the consenting process has been completed, participants will begin the scheduled fifty-five minute session that will entail answering questions electronically on a desktop computer or mobile device (Attachments 5-6). With a regular schedule of sessions, usability testing can be easily included for testing the Cancer.gov and Cancer.gov/espanol websites, without creating a special project or schedule. This will also facilitate sharing information across various NCI divisions and lead to a greater understanding of appropriate and effective content, structure, and presentation of information on cancer.gov pages.

The formative research process is used to determine whether or not a draft message or message concept is effective in reaching and communicating with its audience. Pretesting involves presentation of draft messages designed to convey specific information to a sample of the audience for whom the materials are intended. Information collected to determine the level of customer satisfaction with products helps NCI identify strategies for improving the accessibility of materials/programs, their user-friendliness, and their relevance to the needs of cancer patients and their families, health educators and interventionists, cancer advocates, cancer information specialists, and health care professionals. Research explores various domains shown in the table below.

Factors assessed	Research questions
Attention	Do the messages attract and/or hold the audience's attention?
Comprehension	Are the messages and language clearly understood, and does the main theme of the message resonate with the audience?
Personal relevance and Self-efficacy	Do members of the target audience perceive the message as personally relevant, and do they see themselves as capable of acting on the message provided?
Believability	Is the message and/or its source perceived as credible?
Acceptability	Is there any­thing in the message that may be offensive or unacceptable to the target audience, and is the piece culturally sensitive?
Accessibility	Will members of the target audience be able to find the information or message with relative ease, considering factors such as the availability of products in multiple formats (e.g., print materials, videos, online documents, CDs, audio tapes) and in a variety of settings (e.g., clinics, hospitals, doctors’ offices, community organizations, libraries)?
Usability	How likely is the respondent to use the information in the format provided, and is the content provided in a logical, organized and user-friendly way? And, do the topics addressed by NCI and OCPL products meet the needs, concerns and interests of its customers?
Behavioral intent	Do respondents think they will take action as a result of seeing/hearing the message?
Readability and clarity	Can the reader understand publications and educational materials, both in terms of the needs of low-literacy audiences and with respect to plain language principles and design?
Availability	Are NCI and OCPL products present for low-English fluency audiences?
Cultural appropriateness	Are foreign-language translations or adaptations of products accurate and available for customers?

Respondents' input and reactions to each of these areas provide insight into how the audiences for these messages may react, how the messages should be formulated or revised to communicate most effectively­, and the usefulness, relevance and appropriateness of these products to its customers.

Systematic formative research and pretesting has been widely adopted by health education program planners as an integral step in the development and targeted dissemination of messages and materials. Through this research, NCI is able to:

• Understand characteristics of the target audience—its attitudes, beliefs, and behaviors—and use these in the development of effective communications tools;

• Design messages and select formats that have the greatest potential to influence the target audience’s attitudes and behavior in a favorable way;

• Determine the best promotion and distribution channels to reach the target audience with appropriate messages;

• Expend limited program resource dollars wisely and effectively; and

Results of past pretesting and customer satisfaction efforts have been instrumental in helping NCI carry out its legislative mandate.

A.3 Use of Information Technology to Reduce Burden

As computer technology has continued to improve and become more widespread, opportunities to pretest online messages with Internet users have increased. Improved technology in the collection and processing of data has the potential to reduce the time burden for respondents and data collectors. For example, respondents can access and respond to data collection requests at a time and place that is convenient to them, eliminating the need to travel for in-person or group interviews. Wherever possible, NCI will make use of Web- or computer-based data collection methods. Transmission of data collection instruments and responses by electronic mail or facsimile will be utilized as appropriate (for example, with intermediary audiences such as NCI’s Cancer Information Service or Public Affairs Network). NCI anticipates that of the majority of data will be collected electronically.

Possible information technologies to assess customer satisfaction include:

• Technology now enables the conduct of focus groups, which traditionally take place in-person with a moderator facilitating a discussion regarding a product, issue, or program with a small group of individuals in a designated physical location. Depending on factors such as geographic distribution and schedules of NCI customers and on the nature of the products and services under investigation, focus groups, when appropriate, may be implemented using a variety of technology-based formats.

• Internet or online conferencing is especially useful for discussions with specific individuals or international participants. This format functions as a sort of “chat room” in which a moderator intercepts and distributes e-mail transmissions from participants who have logged onto a specially designated Web site.

• Teleconferencing uses telephone technology to facilitate an exchange among participants located in multiple geographic locations. Participants dial into a specially designated phone number or “bridge line” that is moderated by a focus group facilitator.

A.4 Efforts to Identify Duplication

No similar collection of information exists.

A.5 Impact on Small Businesses or Other Small Entities

N/A

A.6 Consequences of Collecting the Information Less Frequently

This is a one-time information collection.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This survey will be implemented in a manner that fully complies with 5 C.F.R. 1320.5.

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

N/A

A.9 Explanation of Any Payment of Gift to Respondents

There is no remuneration.

A.10 Assurance of Confidentiality Provided to Respondents

No personally identifiable information will be collected. All data will be kept private to the extent allowed within the law.

Information provided by respondents will be kept confidential and private, except as otherwise required by law. This will be communicated to respondents by means of introductory letters, explanatory texts on the cover pages of questionnaires, scripts read prior to focus groups or telephone interviews, and consent forms (Attachment 7 and 8). Respondents will also be advised of the following: the nature of the activity; the purpose and use of the data collected; NCI sponsorship; and the fact that participation is voluntary at all times. Because responses are voluntary, respondents will be assured that there will be no penalties if they decide not to respond, either to the information collection as a whole or to any particular questions.

As a further guarantee of confidentiality, all presentation of data in reports will be in aggregate form, with no links to individuals preserved. Reports will be used only for research purposes and for the development of communication messages and educational materials. Only NCI staff and contractor personnel conducting the information collection will have access to individual-level survey, interview, or focus group data. All project/contractor staff conducting the information collection will sign a confidentiality agreement, and all electronic and hard-copy data will be maintained securely throughout the information collection and data processing phases. While under review, electronic data will be stored in locked files on secured computers; hard-copy data will be maintained in secure building facilities in locked filing cabinets. Before any data are released for public use data sets, any identifying information will be stripped from each respondent’s record and the identifying information will be destroyed.

A PIA is not necessary per the NCI Privacy coordinator.

A.11 Justification for Sensitive Questions

No sensitive questions will be asked. Furthermore, the proposed project is not applicable to human subjects.

A.12.1 Estimated Annualized Burden Hours

The estimated annualized burden hours are 172 hours for a total of 492 respondents. The number of respondents to be included during each round of Open Call testing will vary from 9 to 16, depending on the target audience and the nature of the message/content/website being assessed. The burden per respondent will be 75 minutes (1.25 hours) which includes 15 minutes for the screener that will determine which participants are eligible. Once respondents are selected to participate, it will take 5 minutes to complete the consenting process and 55 minutes to conduct the usability testing session in either Spanish or English. Each year, a total of 6 Open Call Usability studies will be conducted. Table A.12-1 below provides annualized estimations of a distribution of total number of respondents that could include up to 492 per year that could vary according to the content being tested during each round of Open Call. Even if the actual participant distribution for Open Call testing varies from the estimate below, the total burden will not be exceeded.

A.12-1 Estimated Annualized Burden Hours

Type of Respondent	Form Name	Number of Respondents	Number of Responses per Respondent	Average Burden Per Response (in hours)	Total Annual Burden Hour
Lay Audience	Screener	150	1	15/60	38
	Informed Consent	48	1	5/60	4
	Usability testing	48	1	55/60	44
Healthcare Professional	Screener	150	1	15/60	38
	Informed Consent	48	1	5/60	4
	Usability Testing	48	1	55/60	44
Totals		492			172

A.12-2 ANNUALIZED COST TO RESPONDENTS

The annual cost to respondents is $ 5,095.50. This was calculated using the Bureau of Labor Statistics most recent wage rates for the occupation title “All Occupations” code 00-0000 (http://www.bls.gov/oes/current/oes_nat.htm#00-0000) for the lay audience and “healthcare practitioners and technical occupations”, code 29-0000 (http://www.bls.gov/oes/current/oes290000.htm) for the healthcare professionals.

A.12-2 Annualized Cost to the Respondents

Type of Respondent	Number of Respondents	Average Burden Per Response (in hours)	Hourly Wage Rate*	Respondent Cost
Lay Audience	246	86	$22.71	$ 1,953.06
Healthcare Professionals	246	86	$36.54	$ 3,142.44
TOTAL	492			$ 5,095.50

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers

There are no capital or start-up costs to the data collection efforts requested; nor are there any costs associated with operation, maintenance or purchase of services.

A.14 Annualized Cost to the Federal Government

The estimated annualized cost to the federal government is approximately $204,198. Table A.14-1 shows the breakdown of estimated costs, with figures that include the costs of study design, facility rental (e.g., for focus groups), travel, data collec­tion, analysis, and report/publication writing. A mileage reimbursement will be provided to respondents who travel to and from the location where the research will take place. The reimbursement will be in accordance with the Federal mileage rate of and the overall amount will not exceed $100 per respondent. This estimate also includes federal personnel costs for the Project Officer and Senior Analyst that will be monitoring the contractor which is projected to be about 500 hours of effort a year. The contractor will be responsible for analyzing data, presenting findings and issuing recommendations. Given a total NCI personnel average cost of $60.00 per hour, $30,000 would be spent annually on Government staff salaries.

Table A.14-1 Annualized Costs to the Federal Government

Staff	Grade/Step	Salary	% of Effort	Fringe (if applicable)	Total Cost to Gov’t
Federal Oversight
Project Officer	12/7	$91,657	40%		$36,663
Senior Analyst	12/7	$91,657	20%		$18,335
Contractor Cost					$100,000
Travel
Travel Reimbursement					$49,200
TOTAL					$204,198

A.15 Explanation for Program Changes or Adjustments

N/A

A.16 Plans for Tabulation and Publication and Project Time Schedule

Research staff will review the material to be tested; discuss the objectives with the individuals responsible for development; determine the analytic questions to be addressed; and then prepare the research procedures, instruments, and data analysis plan. The analysis conducted for each Open Call usability study will be determined by the objectives of the research, the messages being tested, and the audience for the messages. Specifics of each analysis cannot be determined until the messages to be tested are prepared. Research techniques may include qualitative analysis and customer satisfaction analysis. Due to the low sample size, statistical analysis will not be appropriate; therefore we will only be conducting descriptive statistics. Analysis may be conducted using cross-tabulation procedures, with categorical variables; or between-group procedures, with continuous variables.

While the primary purpose of all studies is to provide information to the developers of the messages/materials/services for the purpose of improving them, NCI shares information internally and also makes results available to a variety of health program planners at Government agencies, voluntary organizations, health professional organizations, user experience research professionals, and medical institutions. Information collected will be compiled and presented in reports and briefings for staff from OCPL and other divisions within NCI and NIH. Reports will include information regarding respondent demographics, basic descriptive data with OCPL products and services, comparisons across demographic and customer subgroups, recommendations for improving programs and products, and analyses of longitudinal changes. In addition, NCI may also publish results in journals and present the findings of its research at meetings of professional associations, for example, the American Public Health Association and the Society for Public Health Education. Formative research conducted by OCPL is also sometimes summarized in news-related publications such as the NIH Record.

The specific messages and materials/service that will be evaluated and the timing of these studies are not known at this time. The Open Call research period will take place every two months (bimonthly). The number of respondents required will vary for each round of testing and will range from 9 to 16 participants. Conducting Open Call testing will require approximately 12 weeks from initial design to preparation of the report of pretest findings. A schedule for a typical Open Call testing round is shown below:

A.16-1 Project and Publication Timeline

Activity Time Schedule

Initial review of research questions 1-2 weeks after OMB approval

Write data collection instrument 3-4 weeks after OMB approval

Preparation of design and plan 4-5 weeks after OMB approval

Review of design 6-7 weeks after OMB approval

Collection of data 7-8 weeks after OMB approval

Analysis of data 9-10 weeks after OMB approval

Write report of findings 11-12 weeks after OMB approval

Develop manuscript (when seeking publication) 4-6 months after OMB approval

Submit for publication 7-8 months after OMB approval

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

We are not requesting an exemption to the display of the OMB Expiration date.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

Open Call Usability testing will comply with the requirements in 5 CFR 1320.9.

12