Mini SSA

This is a request to conduct voluntary customer satisfaction surveys of the National Institutes of Health’s (NIH’s) Enhancing Peer Review Initiative. These surveys will help fulfill the requirements of:

Executive Order 12862, “Setting Customer Service Standards,” which directs Agencies to continually reform their management practices and operations to provide service to the public that matches or exceeds the best service available in the private sector; and
The March 3, 1998 White House Memorandum, “Conducting Conversations with America to Further Improve Customer Service,” which directs Agencies to determine the kind and quality of service their customers want as well as their level of satisfaction with existing services.

A.1.2 Background

The peer review system is a cornerstone of NIH and has been adopted internationally as the best guarantor of scientific independence. However, the increasing breadth, complexity, and interdisciplinary nature of modern research have created many challenges for this system.^¹ The NIH recognizes that as the scientific and public health landscape continues to evolve, it is critical that the processes used to support science are fair, efficient, and effective. In June 2007, therefore, the NIH Director established working groups to examine peer review at NIH as part of a broad Enhancing Peer Review initiative. Specific implementation milestones were articulated and a process of assessment and continuous improvement was also established. The proposed surveys will inform the process of assessment and continuous improvement. Additional information on the Enhancing Peer Review Initiative can be found at http://enhancing-peer-review.nih.gov.

The peer review surveys are designed to partially assess the extent to which NIH upholds the following core values of peer review: to 1) provide scientific expertise on the initial review panel that is suitable for evaluating the potential impact of the proposed work; 2) evaluate the applications submitted to the NIH using only the established, published review criteria as the basis for scientific and technical merit; 3) manage any circumstance that might introduce apparent or real conflict of interest, bias, or predisposition into the review process by any participant in the process, to avoid inappropriate influence in the review process; 4) apply equivalent review processes to all applications received for NIH review. Other core values, including the 5) protection of confidentiality, 6) preservation of research integrity, and 7) efficiency of the review process, are more appropriately monitored internally through other evaluation processes. More information about the core values of peer review can be found at: http://grants.nih.gov/grants/peerreview22713webv2.pdf.

The NIH is committed to a quality control and improvement process for peer review. It is crucial to get ongoing satisfaction information from constituents to inform this improvement.

A.2 Purpose and Use of the Information Collection

Three surveys are planned -- a Reviewer Survey, an Applicant Survey and an Advisory Council Survey. The primary objective of these surveys is to assess these stakeholders’ experiences with the peer review process. The findings from the surveys will provide an important source of information for developing recommendations to further refine the enhanced peer review process. The information collected in these surveys is needed by NIH to obtain customer input about their satisfaction with recent changes in the peer review process and anticipated changes to address the reproducibility of research funded by NIH. The surveys will form one component of a variety of sources of information that NIH relies on for meeting the need for timely review of peer review. They will assess the procedural changes to the peer review system, particularly with respect to the R-series funding mechanisms (i.e., R01, R03, and R21). The surveys are intended to garner specific information about stakeholders’ most recent experience with peer review process.

The survey will focus on respondents’ experience with a new biographical sketch format that has been introduced in phases over 2014-2015, as well as a new resubmission policy that was introduced in May 2014, and a new table of scoring anchors that was developed to encourage reviewers to spread their scores across the entire 1-9 score range available to them. There also is a series of questions about a new initiative to improve the rigor and reproducibility of NIH funded research.

Other questions will assess the ongoing experience of the community of NIH peer reviewers to changes to the peer review system that were made in January 2009 (e.g., enhanced review criteria, templates for structured critiques, scoring of individual review criteria, use of a 9-point scoring scale, and clustering of New Investigator/ Early Stage Investigator applications for review).

The Applicant Survey will ask respondents to report on their most recent application experience. Questions will also focus on the shortened applications and the usefulness of the 9-point rating scale, scoring of individual review criteria and overall impact/priority, and other key elements.

Advisory Council members will be questioned about whether the information contained in the artifacts of the review process, overall impact scores and summary statements are sufficiently detailed to permit them to execute their roles as Advisory Council members.

A cross-sectional design will be used in implementing the surveys. The present set of surveys is the third of three planned surveys, and will be administered approximately three years after the second surveys were administered. This approach will provide “snapshots” of reviewers’, applicants’ and Advisory Council members’ perceptions of and experience with the NIH peer review process.

A.3 Use of Information Technology and Burden Reduction

The mode of data collection for these surveys was carefully considered with respondent burden in mind. It was determined that automated information technology will be used to collect and process the information. The surveys will be conducted online. Invitations to participate will be sent to the selected sample members via email and later by mail, if needed. A Privacy Impact Assessment has been completed for the databases used in this submission.

A.4 Efforts to Identify Duplication and Use of Similar Information

Collected information will be limited to that which is needed to assess customer satisfaction. Some of the data we are seeking is available through NIH data systems where administrative information relating to research grants and contracts is stored. For applicants, for example, this includes administrative data on individual grant applications (e.g., date of submission, type of application, and application status). For reviewers and Advisory Council members, NIH maintains data on the number, dates, type of review activities, and other topics.

As part of the preparations for these surveys, NIH consulted staff members involved in the development of peer review enhancements, staff members involved in administering NIH grants programs, Scientific Review Officers (SROs), and with external survey experts for input on the factors that should be included in the survey analysis. Based on these consultations, NIH determined that some of the data elements included in the NIH databases are essential for achieving the aims of this survey. However, OMB Generic Clearance 0925-0627 does not allow these data to be linked to the customer satisfaction survey responses. The proposed survey instruments minimize the duplication to the maximum extent possible. Only essential demographic data are requested.

A.5 Impact on Small Business or Other Small Entities

No small businesses or other small entities will be impacted by this information collection.

A.6 Consequences of Collecting the Information Less Frequently

Individual applicants and reviewers will be asked to complete the survey only once in FY2015. For prior and subsequent years, unique samples of applicants and reviewers are involved.

Absent this survey frequency, changes to the peer review system would not be adapted to meet customer needs based on customer satisfaction, because satisfaction with the system would not be ascertained.

A.7 Special Circumstances Relating to the Guidelines of 5 C.F.R. 1320.5

This data collection fully complies with 5 C.F.R. 1320.5.

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

Not applicable.

A.9 Explanation of Any Payment of Gift to Respondents

No payment or gift will be offered to survey participants.

A.10 Assurance of Confidentiality Provided to Respondents

The NIH Privacy Act Officer has reviewed this OMB request and determined that the Privacy Act is applicable (Attachment 4).

Concern for privacy and protection of respondents’ rights will play a central part in the implementation of the surveys. All survey procedures will be consistent with OMB Generic Clearance 0925-0627, which assures “Generally the responses to the questionnaire surveys will be entirely anonymous and will have no identifiers to link them to individual respondents”. Strict procedures will be followed for protecting the anonymity of information gathered from the participants. Participation will be fully voluntary, and non participation will have no impact on eligibility for or receipt of future funding.

All computer-based systems employed by RTI will comply with the Privacy Act of 1974, as well as the Federal Information Security Management Act of 2003 (FISMA), which imposes rigorous data security requirements for all Federal agency information systems (internal or contractor-developed).

The system security features include the following:

RTI has developed our Information Systems using the Minimum Security Requirements identified in FIPS-200 for the FIPS-199 system security category. The security controls in place correspond to the security requirements those defined in NIST SP 800-53, including control enhancements.
RTI requires that all our information system users have a User ID assigned by RTI ITS and that they use a strong Password to access all computer systems. Remote access is accomplished using two-factor authentication.
The Website will operate on our certified and accredited Internet-accessible Public Network, which has received an Authority to Operate in accordance with NIST SP 800-37 (Guide for the Security Certification and Accreditation of Federal Information Systems).
Web content delivery is on FIPS 140-2 compliant hardware.
RTI maintains a fully certified and accredited Private Network to support both corporate and project initiatives. RTI maintains several fully switched and routed Ethernet-based local area networks (LANs) in support of our networks. RTI wide area networks (WANs) employ technologies including site-to-site VPN, metro Ethernet, MPLS, VSAT, Voice over Internet Protocol (VoIP), and WAN acceleration appliances. Access from the Internet is available to authorized staff only and is controlled by RTI's Internet firewalls. Remote access to RTI's data networks is provided through the use of client-computer-installed VPN software, a clientless SSL/VPN portal, and direct dial-in connections. The use of RSA SecurID two-factor authentication for remote access is supported.

Safeguarding procedures that we will implement include:

The safeguarding protections offered to survey participants are described in the informed consent language in the introduction to the survey instruments. Respondents will be informed their participation is voluntary and that no consequences will be associated with not responding or with responding. Individuals contacted in the course of these surveys will be assured of the privacy of their replies under 42 USC 1306, 20 CFR 401 and 422, 5 USC 552 (Freedom of Information Act), 5 USC 552a (Privacy Act of 1974), Privacy Act System of Records Notice: 09-25-036, and OMB Circular No.A-130.
All data will be analyzed and reported in an aggregate form that does not personally identify any applicants or reviewers.
An independent contractor, RTI International (RTI), will collect and collate the surveys electronically. RTI will also be responsible for initial analysis and reporting of the data. The data sets that will be transferred back to NIH staff will be fully de-identified. RTI has the required security clearances in order to assure privacy and protection of the data.
RTI’s Institutional Review Board (IRB) has determined that these surveys are Not Human Subjects research (IRB ID Number 12444). In addition, all study staff members will receive Human Subjects Protection Awareness training. This training will promote awareness of the human subjects’ protection offered by the survey design, ethical issues and concerns, and regulations and assurances by which the survey is governed.
Access to data will be restricted to project staff members on an as needed basis.

RTI will observe high standards of information technology (IT) security to protect the privacy, integrity and availability of all computer-based systems and the data they contain. RTI IT security policies and procedures are designed to protect information systems and data from a wide range of risks and will educate their staff to be aware of their responsibilities for ensuring information security and to comply with these policies. RTI also participates with agencies to ensure that their policies conform to agency information security requirements and applicable laws and regulations as required by contract. RTI has System Security Plans for its infrastructures in which it documents how they secure their systems using administrative, technical, and physical controls.

All computer-based systems employed by RTI will comply with the Privacy Act of 1974. The system security features will include:

User ID and Password authentication required to access all computer systems
The Website will operate on a certified and accredited Internet-accessible Standard Security Infrastructure which has received an Authority to Operate in accordance with NIST special publication 800-37 (Guide for the Security Certification and Accreditation of Federal Information Systems).
Web content delivery will be on FIPS 140-2 compliant hardware.
Fully switched and routed Ethernet-based local area networks (LANs) support both corporate and project initiatives. RTI wide area networks (WANs) employ technologies which include site-to-site VPN, Metro Ethernet, MPLS, VSAT, Voice over IP (VoIP), and WAN Acceleration appliances.
Access from the Internet is available to authorized staff only and is controlled by RTI’s Internet firewalls. Remote access to RTI’s data networks is provided through the use of client-computer-installed VPN software, a clientless SSL/VPN portal, and direct dial-in connections. The use of RSA SecurID two-factor authentication for remote access is supported.

A.11 Justification for Sensitive Questions

The NIH is committed to providing high-quality service to its customers. Given the diversity of its constituents, it is important for NIH to collect survey data from a wide range of customers. Hence, the Applicant Survey and Reviewer Survey contain questions regarding respondents’ race, ethnicity, gender, and age. All three surveys contain questions about work-related information (type of employer organization, job title, education). This information will allow NIH to analyze the survey data by subgroups and support NIH’s long-standing efforts to strengthen the diversity of the membership of its applicants and reviewers.

Respondents may skip any or all of the questions concerning race, ethnicity, gender, age and work-related information in the surveys. Those who choose to provide these demographic data will do so voluntarily. The surveys will not collect any personally identifiable information. Thus, any demographic information gathered by the surveys will not be linked to individual respondents.

A.12 Estimates of Hour Burden Including Annualized Hourly Costs

The total number of respondents from whom information will be collected is 4,176. These participants are university faculty and other members of the NIH research community. The total sample size is expected to be 150 Advisory Council members, 1,685 applicants, 1,104 reviewers, and 1,237 who are both an applicant and a reviewer. It is estimated that the Advisory Council Survey will each take an average of 15 minutes to complete. The Applicant Survey and Reviewer Survey are estimated to take an average of 30 minutes to complete. The annual hour burden is, therefore, estimated to be $96,198 hours for approximately 4,176 respondents (Table A.12-1).

Table A.12 – 1 Estimates of Annual Hours Burden

Types of Respondents	Number of Respondents	Frequency of Response	Average Response Time	Annual Hour Burden
FORM: Reviewer/Applicant Questionnaire
Adult Science Professionals – Applicant Only	1,685	1	30/60	843
Adult Science Professionals – Reviewer Only	1,104	1	30/60	552
Adult Science Professionals – Both Applicant and Reviewer	1,237	1	30/60	619
Form : Advisory Council Questionnaire	150	1	15/60	38
Total	4,176			2052

Estimated costs to the respondents consist entirely of their time. Costs for time were estimated using a rate of $46.00 per hour, the national 75%ile hourly wage for adult life science professionals reported in the May 2014 National Occupational Employment and Wage Estimates from the Bureau of Labor Statistics. A rate of $95.00 per hour for Advisory Council members is based upon the 2011 NIH salary cap for senior investigators, $199,700 per year. The estimated annual cost burden for respondents for the first year for which the generic clearance is requested is (Table A.12-2).

Table A.12 – 2 Annualized Cost to Respondents (Based on Expected 60% Response)

Types of Respondents	Number of Respondents	Frequency of Response	Average Time per Respondent	Hourly Wage Rate	Respondent Cost
Form: Reviewer/Applicant Questionnaire
Adult Science Professionals – Applicant Only	1,685	1	30/60	$46	$38,755
Adult Science Professionals – Reviewer Only	1,104	1	30/60	$46	$25,392
Adult Science Professionals – Both Applicant and Reviewer	1,237	1	30/60	$46	$28,451
Form: Advisory Council Members	150	1	15/60	$96	$3600
Total	4,176				$96,198

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record Keepers

We do not require any additional record keeping.

A.14 Annualized Cost to the Federal Government

For the first year, the approximate annualized cost to the government for this data collection effort is approximately $568,058 (Table A.14-1). Total government personnel costs will be $39,000/year, taking into account benefits. This figure assumes a GS-15 annual salary of $156,000 for an NIH professional to manage the projects. Salaries are based on the January 2015 General Schedule for the Washington, DC metropolitan area (http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2015/DCB.pdf). Details are provided in the table below. Estimated annual federal personnel costs for the second year are $26,000, which adjusts for a reduced level of effort for federal staff and shorter project duration in the second year.

Contractor support will be required to carry out the data collection efforts. It is estimated that the first year of this effort will cost approximately $318,550. Survey efforts in subsequent years will cost an estimated $159,275 for the second year. The cost reduction in the second years is because survey programming will have mostly been finalized in the first year and the project duration extends over only 6 months in the second year. The NIH anticipates undertaking no more than one project in 18 months.

Mailing costs for paper surveys total $25,233 assuming a USPS priority shipping flat rate of $6.40 per letter, printing and preparation charges of 3.00 per package, and a 40% rate of follow-up mailings (worst case scenario).

Table A.14-1. Annualized Costs

Activity	Cost Year 1	Cost Year 2
Administration of the Clearance
NIH staff (1 GS-15) – 20% FTE @ $156,000/yr in Oct 2014 – May 2016	$39,000	$26,000
Contract Support for Data Collection
3 surveys, 18-month project period	318,550	159,275
Courier Cost for Paper Surveys
2,684 surveys (6,711*40%) x $9.40	$25,233	0

Total	$382,783	$185,275

A.15 Explanation for Program Changes or Adjustments

Adjustments were made to the annualized costs estimated for the previous round of surveys based on changes in process and efficiencies realized as a result of experience with the previous surveys.

A.16 Plans for Tabulation and Publication and Project Time Schedule

A.16.1 Plans for Tabulation

The analysis plan is designed to examine the degree to which survey responses differ across key analysis groups or combinations of those groups. Key analysis groups are defined by combining the following information to form groups of interest, such as race and ethnicity.

Comparisons across key groups will focus on topics such as experience with the peer review process, satisfaction ratings about the peer review process, as well as the format of grant applications.

Analyses will focus mainly on descriptive information including two-way tables to compare groups of interest.

Data collected for this study will be aggregated. No results will be reported that identify respondents by name or another identifier that allows respondent’s identity to be disclosed. Specific procedures for analyzing the data are described in the following paragraphs.

Descriptive Information

Analysis will begin with a description of the applicants, peer reviewers and Advisory Council members who responded to the peer review surveys. The three surveys are provided in Attachments 1, 2 and 3, respectively. One analysis table will be created with the demographic variables collected in the questionnaires. No gender, race, or ethnicity information will be collected from Advisory Council Members. The data will be presented in two tables. One table will contain two columns, one column for applicant questionnaire data and one column for peer reviewer questionnaire data. The second table will contain the Advisory Council questionnaire data.

Data will be presented in tabular format with frequencies and percents for categorical variables; means, minimum and maximum values will be displayed for continuous variables.

Table A.16-1 is an illustration of the table that will be compiled during analysis for the descriptive and demographic related questions shown above. This table shell shows only a subset of the variables and the actual table produced from survey responses will contain additional rows. For categorical variables, the cells of the table will contain the frequency count of the responses as well as their respective percentages (based on non-missing data) and, for continuous variables, the cells of the table will contain the mean, responding sample size, and minimum and maximum values. The overall numbers of respondents will be given in the column headers.

Table A.16-1: Demographic Information – Sample Table Shell

Demographic Question	Applicant Questionnaire	Reviewer Questionnaire
Demographic Question	N =	N =
Ethnicity
Hispanic	n (%)	n (%)
Non-Hispanic	n (%)	n (%)
Type of Employer Organization
Institution of Higher Education	n (%)	n (%)
Hospital/Medical Center	n (%)	n (%)

Assessing Unit and Item Non-response

After an overall descriptive summary of the sample respondents, a Unit and Item non-response analysis will be carried out. While sampling weights will be adjusted for unit non-response within sampling strata, if the response rate within sampling strata is low (i.e., less than 50%), then the sample respondents may not be representative of the relevant target population. In order to assess whether or not unit response rates are low, response rates will be tabulated for each race and ethnicity group within the three selected samples (Applicant only, Reviewer only, and individuals who are both Applicant and Reviewer).

Even when unit response rates are high, item nonresponse amongst respondents may reduce the degree to which inferences about such an item is trusted. Since there are a variety of analyses that may be carried out using the peer review surveys’ responses, one could calculate item nonresponse for a variety of analytical subgroups. We will tabulate item response rates, separately for the Applicant, Reviewer and Advisory Council questionnaires, overall and within some key analytical subgroups, where applicable (e.g., race and ethnicity).

Analysis of Survey Responses

Survey responses to various questions will be analyzed by comparing survey responses between the key groups described in the first section. Categorical responses will be analyzed by cross-tabulating weighted responses across given groups (such as race or ethnicity). Statistical differences will be assessed by performing sample survey appropriate Chi-square tests of proportions to test for independence of survey responses across the groups. Continuous responses will be analyzed by reporting weighted means across given domains. Statistical differences will be assessed by performing sample survey appropriate t-tests to test for differences in mean response across the domains. Two-way tables will be created for all satisfaction/opinion questions in order to compare the groups of interest. All categorical variables will contain the frequency counts of the responses as well as their respective percentage of non-missing data. All continuous variables will be displayed with means along with the number of non-missing responses, minimum and maximum values.

Tables A.16-2 and A.16-3 are examples of tables to display the results of the analysis.

Table A.16-2. Experience of Applicants – Sample Table

Question	Applicant Questionnaire
Question	N =
Application assigned numerical impact/priority score	n (%)
Application received NOA -- funded	n (%)

Table A.16-3. Experience of Peer Reviewer – Sample Table

Question	Reviewer Questionnaire
Question	N =
Capacity as a NIH reviewer
Regular (appointed)	n (%)
Ad hoc (temporary)	n (%)
Both regular and ad hoc	n (%)
Reviewer for Components of NIH
Center for Scientific Review	n (%)
One or more NIH Institutes/Centers (ICs)	n (%)
Both CSR and ICs	n (%)

A.16.2 Plans for Publication

A written report with accompanying charts will be provided to NIH management for internal use. There are no plans to publish the results of these surveys.

A.16.3 Project Time Schedule

The project time schedule is provided in Table A.16-4. OMB clearance is being requested for one year.

Table A.16-4. Project Time Schedule

Activity	Time Schedule
Launch survey website and email invitations	Applicant/Reviewer: August 22, 2015 Advisory Council: November 1, 2015
Conduct data collection	Applicant Reviewer: August 22 to September 30, 2015 Advisory Council: November 1 to December 15, 2015
Create analysis file and analyze data	Applicant Reviewer: October 1 to November 30, 2015 Advisory Council: December 16 to February 15, 2016
Document findings	February 16 to April 30, 2016

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

We are not requesting an exemption to the display of the OMB Expiration date.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

These surveys will comply with the requirements in 5 CFR 1320.9.

1 NIH (2008) 2007-2008 Peer Review Self-Study. Final Draft.(URL: http://enhancing-peer-review.nih.gov/meetings/NIHPeerReviewReportFINALDRAFT.pdf)

File Type	application/msword
File Title	NLM Reading Room example - 04/03/2009
Subject	NLM Reading Room example - 04/03/2009
Author	OD/USER
Last Modified By	Roberts, Luci (NIH/OD)
File Modified	2015-07-02
File Created	2015-07-02