NIH2007 Supporting Statement

"Evaluation of User Satisfaction with NIH Internet Sites"

Supporting Statement

A. JUSTIFICATION

1. Circumstances Making the Collection of Information Necessary

This is a request, made by the National Institutes of Health (NIH) Office of the Director (OD) Office of Communications & Public Liaison (OCPL) Online Information Branch (OLIB), for the Office of Management and Budget (OMB) to approve, under the Paperwork Reduction Act of 1995, the request for a three-year generic clearance granted to the NIH to conduct user satisfaction research around its Internet sites associated with the www.nih.gov domain. The original generic clearance was received in April 2001 [OMB # 0925-0486; Exp. Date 04/30/2004]. It was renewed in May, 2004 [OMB # 0925-0486; Exp. Date 05/31/2007]. Data collections performed under this clearance are made according to Executive Order 12862 which directs federal agencies that provide significant services directly to the public to survey customers to determine the kind and quality of services they want and their level of satisfaction with existing services.

In accordance with Executive Branch and Congressional mandates to provide information dissemination, and under its mission, NIH continues to expand the availability of vital health information over the Internet. The NIH Internet sites comprise a vast collection of Web pages created and maintained by its component organizations. Collectively, these pages receive nearly four million visits per month (Attachment B, pp 2-3). All 27 of the NIH's Institutes and Centers (ICs) have built Web sites to communicate with their respective audiences (Attachment B, pp 4-30). The many NIH Offices (e.g., Office of Human Resources, Office of Communications and Public Liaison) have also put up Web sites that describe their activities (Attachment B, pp 31-32). When an Institute, Center or Office develops a communications strategy for launching a new program or initiative, a Web site is always included.

The audience for each site varies. The National Cancer Institute (NCI), for example, maintains nearly 3 million web pages providing information geared toward patients on types of cancer, treatments, clinical trials, and cancer statistics (Attachment B, p.4). In addition, the Web site provides information on funding opportunities and training programs for researchers. Information for policy-makers also exists on the site in the form of strategic plans, budgets, and legislative affairs documents. The NCI Web site attracts cancer patients, family members of cancer patients, oncologists, researchers, librarians, physicians, nurses, and patient advocates.

In contrast, the NIH News in Health (NIHNiH) Web site consists of less than 100 pages (Attachment B, p.33), and provides practical health news featuring timely health topics. It features a variety of easy-to-use health tips and directs readers to resources where they can find more information. Yet the NIHNiH’s objective is the same as the NCI's in terms of disseminating information in a way that satisfies their customers.

When the majority of NIH Internet sites were created, Webmasters designed the Web pages according to the organization of available material, taking stock of their existing information resources and simply creating electronic versions to be viewed across the Internet. In June 2000, NIH Webmasters were polled to find out if customer service feedback had been incorporated into their initial designs. Only nine out of 31 NIH Webmasters who responded reported that their Web site was built based on data from focus groups, results of customer inquiries, working groups, or other public feedback mechanisms (Attachment B, p. 34-41). All 31 of those sites had been redesigned prior to polling, and only one was able to incorporate user feedback from an online survey. This demonstrated a clear need to measure user satisfaction with NIH Internet sites, in order for the NIH to comply with Executive Order 12862. On the strength of this information, and given the assurance that the NIH OD OCPL Online Information Branch would strictly monitor the use of the clearance vehicle, a generic clearance was granted in April of 2001 for the purpose of evaluating user satisfaction with NIH Internet sites. It was subsequently renewed in May of 2004.

Since the generic clearance vehicle has been available, 18 separate NIH components have conducted 44 user satisfaction surveys to collect data about their Web-related activities (Attachment B, pp 42-43). The results have provided guidance for improving Web-based information dissemination services at NIH. This clearance renewal request aims to allow NIH to continue to evaluate user satisfaction with the content and usefulness of web-based delivery methods. Authorization to collect these data is given under Section 301 of the Public Health Service Act (42 USC 241).

2. Purpose and Use of the Information Collection

The purpose of the data collection activities performed under this clearance is to ensure that intended audiences find that the content of NIH Internet sites is easy to access, clear, informative, and useful. Specifically, the research examines whether the information is presented in an appropriate technological format and whether it meets the needs of users of these Internet sites. The research also provides a means by which to classify visitors to the NIH Internet sites, to better understand how to serve them.

The results are used internally, enabling NIH components to evaluate the success of Web-based communication methods in response to fulfilling legislative mandates to disseminate information. Reports from customer satisfaction surveys are included in Appendix B to illustrate the value of these data collections (Appendix B, pp 62-130).

3. Use of Improved Information Technology and Burden Reduction

Automated information technology will be used to collect and process information in order to reduce the burden on the public. Survey instruments similar to those conducted in the past six years under OMB# 0925-0486 (Attachment A, pp 1-30) will be used to collect data over the Internet.

4. Efforts to Identify Duplication and Use of Similar Information

As a large-scale information delivery system, the Internet has matured and stabilized over the past three years. Web standards have emerged with regard to the use of search engines, site maps, and content management technologies. The Web development industry’s recognized usability experts promote the application of these and other standards through their books and Web sites. The Department of Health and Human Services also provides research-based guidelines on Web design and usability issues on its Usability.gov Web site (Attachment B, pp 44-45). The Webcontent.gov Web site, managed by the Web Managers Advisory Council, an inter-agency group of about 40 web managers from every cabinet-level agency and many independent agencies, also promotes usability principles and techniques (Attachment B, p.46).

Guidelines like these can help NIH components improve the presentation of their content. However, these guidelines cannot tell Webmasters if their content is useful, desirable, or organized in a way that is satisfactory to their site visitors. Nor can a measure of satisfaction with one Web site be generalized to another. It is vital that each NIH component be able to survey its own audience about its Web-related activities to learn the specific needs of their particular customers.

ACSI (American Customer Satisfaction Index) Surveys have emerged as an option for evaluation of Web sites. Even though ACSI provides an indication of overall trends, a more flexible survey instrument is often needed for collecting data about smaller Web sites and specific issues. Additionally, it is more expensive to implement the ACSI than a typical survey under this generic vehicle.

Online survey instruments submitted by different components of a single Institute are compared across target audiences to ensure that data collections are not duplicated.

5. Impact on Small Businesses or Other Small Entities

The burden on small businesses or other small entities is reduced both by the voluntary nature of the study and the short length of time required to respond to the survey instruments.

6. Consequences of Collecting the Information Less Frequently

This research will be conducted over the next three years and survey results will be administered, analyzed, and interpreted as needed on an on-going basis by individual ICs and Offices. Additional improvements and subsequent surveys can be offered on an as-needed basis to continue to refine the content and navigation of the NIH Internet sites.

The frequency with which users will complete the online surveys will depend on the number of different ICs and Offices they visit. Participation will be voluntary. The survey instrument will be presented to the user only if the user clicks on a link to the form itself.

7. Special Circumstances Relating to the Guidelines of 5CFR 1320.5

The request fully complies with regulation 5CFR 1320.5.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A. As required by 5CFR 1320.8(d), a notice of this proposed data collection appeared in the Federal Register, Vol. 72, No. 13, pg. 2700 on January 22, 2007. There were no public comments.

B. Outside consultation was conducted with the following usability professionals who provided insight on designing online surveys to minimize burden on respondents:

• Richard L. Horst, Ph.D., President, UserWorks, (301) 431-0500

• Kath Straub, Ph.D., Chief Scientist, Human Factors International, (410) 327-1012

9. Explanation of Any Payment or Gift to Respondents

There are no payments to respondents.

10. Assurance of Confidentiality Provided to Respondents

The Privacy Act is not applicable. Demographic information is used for analytical purposes only. Names are not recorded on the questionnaires, nor is personal identifying data maintained in the database. While respondents can be given the option of providing their e-mail addresses for the purpose of receiving a response to a particular inquiry, these addresses are not be used for purposes of linking particular respondents to particular survey responses. The addresses are not retained after the reply is sent to the respondent.

Cookies will not be employed as part of any online survey mechanism. A Cookie is a small file that a Web site transfers to a user’s hard disk to allow the Web server to record specific information about the user’s session while they are visiting the Web site. As the user will be voluntarily supplying requested information, there is no need to surreptitiously collect other details “behind the scenes.” A copy of the NIH Web Privacy Notice that explains Cookie use on its Internet sites has been included in Attachment B (Attachment B, p.47).

11. Justification for Sensitive Questions

No questions of a sensitive nature are included in the survey.

12. Estimates of Hour Burden Including Annualized Hourly Costs

1. Estimated Annualized Burden Hours:

Participants will require no more than 5 minutes to complete each survey. Actual time required will vary based on participant reading speed and level. The annual burden table below shows total annual burden to participants.

	Survey Area	Number of Respondents	Frequency of Response per Respondent	Avg. Burden Per Response (hrs.)	Burden Hours	.
	NIH Organization-wide	4000			334
	Overall user satisfaction	2000	1	0.1002	200
	Specific indicators:
	Top-level and entry pages	1000	1	0.0668	67
	Tools and initiatives	1000	1	0.0668	67
	Individual Institute/Office	100000			8350
	Overall user satisfaction	50000	1	0.1002	5010
	Specific indicators:
	Top-level and entry pages	25000	1	0.0668	1670
	Tools and initiatives	25000	1	0.0668	1670
	Total	104000		0.084	8684	.

On average, the NIH Internet sites receive nearly two million unique visitors per month (Attachment B, pp 2-3). The sample size provides adequate indication of the strengths and weaknesses of the NIH Internet sites.

2. Estimated Annualized Burden Costs:

An average hourly salary of approximately $15.00 is assumed for all respondents, based on the Department of Labor (DOL) civilian labor force average wage guide. With a maximum annual respondent burden of 8,684 hours, the overall annual cost of respondents’ time for the proposed surveys is estimated to be a maximum $130,260 (8,684 hrs x $15.00). There will be no direct costs to the respondents other than their time to participate in each study.

Total Respondent Hours	Hourly pay rate	Total Respondent Burden
8,684	$15.00	$130,260

13. Estimate of Other Total Annual Cost Burden to Respondent or Recordkeepers

There will be no capital, operating, or maintenance costs to the respondent.

14. Annualized Cost to the Federal Government

The approximate annualized cost to the government for these data collections is estimated at $160,200, which includes costs for clearance administration, survey design, application development, data collection, data analysis, and reporting.

Assuming an annual salary of $52,000 for government personnel, Federal FTE hours involved in survey activities are averaged at 20 hours per survey for a total of $500. Contractor costs for survey activities are averaged at $15,000 per survey. These averages are based on costs for surveys conducted in the three years that the generic clearance has been in effect.

Due to NIH’s growing awareness of the importance of customer satisfaction data to the design of Web sites, and the visibility of this clearance vehicle, costs for an average of 10 surveys are projected for each of the next three years.

Activity	Cost per annum
Clearance Administration
Usability specialist – 10% FTE @$52,000/yr	$5,200
Customer Satisfaction Surveys
10 surveys/yr @ $15,500/survey	$155,000
Total	$160,200

15. Changes in Hour Burden

There is no change.

16. Plans for Tabulation, Publication, and Project Time Schedule

1. Time Schedule

This research has been conducted over the past three years, and will continue as soon as clearance renewal is received. Surveys will be administered, analyzed, and interpreted as needed on an on-going basis by individual ICs and Offices over the next three years. Follow-up surveys can be conducted on an as-needed basis to continue to refine the content and navigation of NIH Internet sites.

When this generic clearance was first received, the Online Information Branch (OLIB) assumed responsibility for its operation and compliance. The OLIB fields questions related to survey instruments, processes clearance requests, tracks burden hours, and maintains published procedures for using the clearance vehicle (Attachment B, pp 48-61).

The OLIB submits the individual requests to OMB in a batch, once a week, after reviewing them for potential conflict with OMB guidelines. Once the forms have been cleared by OMB, OLIB notifies the NIH component that data collection may begin.

Data is collected continuously until the target number of respondents is reached. The form is then removed from the Web site. Data is analyzed, a report is prepared, and improvements to the Web site are made. Follow-up surveys are conducted at a time deemed appropriate by the individual NIH component.

2. Publication

Study results will be used internally by NIH components to improve the usefulness of their Internet sites. Reports, articles, and presentations may be developed as appropriate to share findings among NIH components.

3. Analysis Plan

Given the specified objectives, analysis will consist of descriptive statistics, which will allow NIH components to understand the perceived strengths and weaknesses of their Web sites. The OLIB encourages NIH components to formulate survey questions using checkboxes and radio buttons, thereby easing respondent burden while providing quantitative data. Open-ended questions are also utilized. Responses can be summarized and assessed as a whole, or coded and categorized for value in potential improvements. A sample summary of survey question analysis is included in Appendix B (Appendix B, pp 103-106).

Each individual NIH component will bear the responsibility of interpreting survey results and making changes to their Web sites as appropriate.

17. Reason(s) Display of OMB Expiration Date is Inappropriate

No exemption is requested.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

This data collection has been designed in accordance with the requirements specified in Item 19 of the OMB 83-I. No exceptions to certification are requested.

B. STATISTICAL METHODS

There are no formal statistical methods employed in this study. The emphasis of the study is on general audience response to the proposed materials. The limited nature of the data will support only descriptive and summary statistics. As such, statistical power is not germane to this project.

12