Revised Attachment 2 - Reviewer Survey Instrument 0925-0474

OMB No: 0925-0474 Expiration Date: 9/30/2011

Enhancing Peer Review Initiative

Survey of Peer Reviewers

Sponsored by:

National Institutes of Health

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number of this information collection is 0925-0474. The time required to complete this information collection is estimated to average 15 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974 Attn: PRA (0925-0474). If you have comments or concerns regarding the status of your individual submission of this form, write directly to: RTI International, 3040 Cornwallis Road, PO Box 12194, Research Triangle Park, NC 27709-2194. 1-800-334-8561 Attn: RTI Project # 0212255)

Introduction

This survey of NIH peer reviewers is to help examine NIH’s Enhancing Peer Review Initiative (http://enhancing-peer-review.nih.gov/). The objectives of the initiative are to engage the best reviewers, improve the quality and transparency of peer review, and ensure balanced and fair reviews. This is the first annual “point in time” survey to gather reviewers’ opinions about the peer review process. This information will be useful in assessing the changes introduced by the Enhancing Peer Review Initiative and may be used to further improve the peer review process. 

You have been randomly selected to participate in this survey from a pool of individuals who served as peer reviewers for NIH at least once from May 2008 through September 2009. We are interested in the opinions of reviewers with different levels of peer review experience. Even if you have limited experience reviewing grant applications, your opinions are very important to us.

The survey should take approximately 15 minutes to complete. You can stop at any point and continue at another time. There are no right or wrong answers, so please give the answer that best describes your opinion.  While we would like you to answer all the questions in this survey, you may skip any questions that you do not wish to answer.

Your participation is entirely voluntary.  If you choose to complete the survey, your responses will remain anonymous. Your responses will not be linked to your name and will not be made known to NIH staff or grant applicants. They will not be used to assess the performance of individual NIH Institutes, Centers, or Scientific Review Groups. Aggregate responses will be used to guide NIH management in refining enhancements to the peer review process.

Your participation is greatly appreciated.

SECTION A: YOUR EXPERIENCES AS A PEER REVIEWER

A1. In what capacity have you ever served as a NIH peer reviewer?

Select all that apply

Regular reviewer or “appointed” member of a chartered scientific review group (study section)

• A reviewer who agrees to serve a fixed duration (typically 4-6 years); may also be called a “charter” or “permanent” reviewer

Ad hoc or “temporary” reviewer

• An ad hoc member of a scientific review group (study section) or Special Emphasis Panel (SEP)

A2. For which component(s) of NIH have you ever been a peer reviewer?

Select all that apply

Center for Scientific Review (CSR)

One or more NIH Institutes/Centers (ICs) (e.g., NCI, NIAID)

IF A1=1 (Regular) CONTINUE ELSE GO TO A5

A3. Are you currently serving as a regular reviewer on a chartered scientific review group (study section) for NIH?

O Yes

O No

A4. As a regular reviewer, how many full terms (typically 4-6 years each) have you completed for NIH?

O 0 terms

O 1 term

O 2 terms

O 3 terms

O 4 or more terms

IF A1=2 (Ad hoc) CONTINUE ELSE GO TO A6

A5. As an ad hoc reviewer, in how many review meetings did you serve for NIH from May 2008 to September 2009?

O 0

O 1

O 2

O 3 or more

A6. In which calendar years have you served as a peer reviewer for NIH, including all peer review service as a regular reviewer and as an ad hoc reviewer?

Select all that apply

2009 1995-1999

2008 1990-1994

2007 1985-1989

2006 1980-1984

2005 1975-1979

2000-2004 Before 1975

A7. Now, please think about the time period May 2009 through September 2009.

During this time, did you serve as a regular and/or an ad hoc reviewer for NIH?

O Yes

O No

IF A7=1 CONTINUE ELSE GO TO SECTION B

A8. Since May 2009, what type(s) of grant applications have you reviewed?

Select all that apply

Grant applications related to the American Recovery and Reinvestment Act (ARRA) of 2009, such as the Challenge grants program and Grand Opportunities “GO” grants program

Regular (non-ARRA) grant applications

SECTION B: REVIEW PROCESS AND PROCEDURES

IF A7=1 DISPLAY SECTIONS B AND C WITH PEER REVIEW CHANGES ELSE DISPLAY SECTIONS B AND C WITH NO PEER REVIEW CHANGES (ALTERNATE QUESTIONS ARE PROVIDED AT THE END OF THIS QUESTIONNAIRE)

In answering the questions in this section, please consider your experience with the enhanced peer review procedures implemented in 2009, regardless of whether they were applied in reviewing regular or ARRA grant applications.

Please refer to this table when answering the following questions.

Based on your most recent review experience using the new scoring procedures, please indicate if you agree or disagree with the following statements.

B1a. The 1-9 rating scale had sufficient range for me to communicate meaningful differences in the quality of the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B1b. The 1-9 rating scale allowed me to communicate strengths and weaknesses for each review criterion.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B1c. The descriptors for the 1-9 rating scale (exceptional to poor) helped me to determine the criterion scores for the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B1d. The descriptors for the 1-9 rating scale (exceptional to poor) helped me to determine the overall impact/priority scores for the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

Based on your most recent review experience using the new scoring procedures, please indicate if you agree or disagree with the following statements.

B1e. Additional guidance on strengths and weaknesses for each score assisted me in determining the criterion scores for the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B1f. Additional guidance on strengths and weaknesses for each score assisted me in determining the overall impact/priority scores for the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

Based on your most recent review experience using the structured critique templates (an example is shown here), please indicate if you agree or disagree with the following statements.

B2a. The structured critique templates allowed me to fully describe my evaluations of the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B2b. The bulleted format in the structured critique templates was adequate for capturing the strengths and weaknesses of the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B2c. The structured critique templates helped me complete my critiques efficiently.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

Based on your most recent review experience involving not discussed (ND) applications, please indicate if you agree or disagree with the following statements.

B3a. The structured critiques helped the reviewers to decide whether or not to discuss an application.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B3b. The bulleted format in the structured critiques helped me communicate to the applicants why their applications were not discussed.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B3c. The criterion scores helped me communicate to the applicants why their applications were not discussed.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

IF A8=1 and A8≠2 ONLY DISPLAY B4a-B4d THEN GO TO B5, ELSE RECEIVE ALL B4

Based on your most recent review experience, please indicate if you agree or disagree with the following statements.

B4a. My scientific expertise was necessary and appropriately used in the review process.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4b. The other review group members seemed to be experts in their fields.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4c. The format and duration of the discussion was sufficient for reviewers not assigned to evaluate an application to be able to cast well-informed votes.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4d. An appropriate amount of time was spent discussing the potential impact of the applicants’ research.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4e. Clustering applications from New and Early Stage Investigators resulted in a more consistent review of those applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4f. Clustering clinical applications (those involving human subjects) resulted in a more consistent review of those applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B5. How well did the reviewer orientation and any training materials you received prepare you to review applications using the new procedures, such as the 1-9 scoring scale, scoring of individual review criteria, and structured critique templates?

O Very well

O Somewhat well

O Not well at all

SECTION C: YOUR OPINIONS ABOUT THE NIH PEER REVIEW PROCESS SINCE 2009

When answering the questions in this section, please think of the peer review process at NIH after enhancements were made in 2009, the one under which your most recent peer review service occurred (regardless of whether it was related to regular or ARRA grant applications).

C1. Overall, which peer review system do you prefer – the new system (with enhancements) or the old system (without enhancements)?

O New system

O Old system

O No preference between the new or the old systems

O Not applicable – no experience with the new system

C2. How fair is the peer review process at NIH after enhancements were made in 2009?

O Very fair

O Somewhat fair

O Neither fair nor unfair

O Somewhat unfair

O Very unfair

C3. How satisfied are you with the peer review process at NIH after enhancements were made in 2009?

O Very satisfied

O Somewhat satisfied

O Neither satisfied nor dissatisfied

O Somewhat dissatisfied

O Very dissatisfied

SECTION D: PEER REVIEW SERVICE

IF A1=2 CONTINUE ELSE GO TO D2

D1. Below are three aspects of the peer review process that may affect individuals’ willingness to serve as ad hoc or “temporary” reviewers, such as ad hoc members of a standing review group or reviewers for a Special Emphasis Panel (SEP).

Please rate the importance to which each of the following affects your willingness to serve in the future as an ad hoc reviewer.

	Very important 	Somewhat important 	Not at all important 	Not sure/ NA 
D1a. The time commitment required to prepare for the meeting (read, assess, and critique applications)	O	O	O	O
D1b. The time commitment required to attend review meetings/discussions	O	O	O	O
D1c. The time commitment required to travel in order to attend meetings	O	O	O	O

IF A1=1 CONTINUE ELSE GO TO SECTION E

D2. Below are four aspects of the peer review process that may affect individuals’ willingness to serve as regular reviewers or “appointed” members of a chartered scientific review group (study section).

Please rate the importance to which each of the following affects your willingness to serve in the future as a regular reviewer.

	Very important 	Somewhat important 	Not at all important 	Not sure/ NA 
D2a. The requirement for a multi-year commitment	O	O	O	O
D2b. The time commitment required to prepare for the meeting (read, assess, and critique applications)	O	O	O	O
D2c. The time commitment required to attend review meetings/discussions	O	O	O	O
D2d. The time commitment required to travel in order to attend meetings	O	O	O	O

SECTION E:                         BACKGROUND

As a reminder, the information you provide in this survey will remain anonymous.  No individual respondents will be identified, and all responses will be summarized and reported in aggregate form.  

E1.  What type of organization do you work for? 

Select all that apply

  University

  Research Foundation

  Private Sector/For-profit Organization

  Hospital/Medical Center

  Federal, State, or Local Government Agency

  Other Non-profit Organization

  Other (specify):  ______________________________

E2.  What is your job title or position?

O  Professor

O  Associate Professor

O  Assistant Professor

O  Adjunct Professor

O  Senior Scientist

O  Other (specify):  ______________________________

E3.  Have you ever submitted a research grant application to NIH as a Principal Investigator (PI) for a single-PI or multiple-PI grant?

O  Yes

O  No

THE QUESTION WORDING “PRINCIPAL INVESTIGATOR” WILL INCLUDE A HYPERLINK AND IF CLICKED THE FOLLOWING DEFINITION WILL APPEAR ON SCREEN:

NIH Definition of a Principal Investigator:  The individual(s) judged by the applicant organization to have the appropriate level of authority and responsibility to direct the project or program supported by the grant.  The applicant organization may designate multiple individuals as PDs/PIs [Program Directors/Principal Investigators] who share the authority and responsibility for leading and directing the project, intellectually and logistically.  Each PD/PI is responsible and accountable to the applicant organization, or, as appropriate, to a collaborating organization, for the proper conduct of the project or program including the submission of all required reports.  The presence of more than one identified PD/PI on an application or award diminishes neither the responsibility nor the accountability of any individual PD/PI.

IF E3=1 CONTINUE ELSE GO TO E9

E4.  When did you submit your first research grant application to NIH as a PI for a single-PI or multiple-PI grant? 

O  2008-2009                          O  1996-1998

O  2005-2007                          O  1993-1995

O  2002-2004                          O  1990-1992

O  1999-2001                          O  Prior to 1990

E5.  As a PI, have you ever received a R01, R03, or R21 grant from NIH? (Please include single-PI grants and multiple-PI grants.)

 O  Yes

 O  No

IF E5=Yes CONTINUE ELSE GO TO E7

E6.  In total, how many years of NIH funding have you received as a PI on R01, R03, and R21 grants? (Please include single-PI grants and multiple-PI grants.)

Enter total number of years here  __________

E7.  In which of the following fiscal years did you receive any type of NIH funding as a PI?  (Please include single-PI grants and multiple-PI grants.) 

Examples of NIH funding include research grants (R series), program project/center grants (P series), career development awards (K series), research training and fellowships (T and F series), and SBIR/STTR grants/contracts.

 

Select all that apply

 O FY 2009 (October 2008 – September 2009)

 O  FY 2008 (October 2007 – September 2008)

 O FY 2007 (October 2006 – September 2007)

 

 O  Did not receive NIH funding for the fiscal years listed

E8. Did you submit any applications as a PI in response to NIH funding opportunity announcements related to the American Recovery and Reinvestment Act (ARRA) of 2009, such as the Challenge grants program and the Grand Opportunities “GO” grants program?

 O  Yes

 O  No

E9. Please indicate the degree(s) you have. 

Select all that apply

  Ph.D. or other research doctorate

  M.D.

  D.D.S.

  D.V.M. or V.M.D.

  Other (specify):  _______________

E10. What is your age?

O Under 35                  O  46-50                     O  61-65

O  35-40                      O  51-55                      O  66-70

O  41-45                      O  56-60                      O  Over 70

E11.  What is your gender?

O  Female

O  Male

E12. What is your ethnicity?

O  Hispanic or Latino

O  Not Hispanic or Latino

E13. What is your race?

Select all that apply

O  American Indian or Alaska Native

O  Asian

O  Black or African American

O  Native Hawaiian or Other Pacific Islander

O  White

Thank you very much for completing the survey!

For more information about the peer review changes that have been implemented at NIH, please visit http://enhancing-peer-review.nih.gov/faqs.html.

If you have any ideas for improving the peer review process at NIH, please enter your suggestions at [insert URL].

ALTERNATE VERSION OF SECTIONS B AND C FOR RESONDENTS WHO HAVE NOT EXPERIENCED ENHANCED PEER REVIEW

SECTION B: REVIEW PROCESS AND PROCEDURES

In answering the questions in this section, please consider your experience with the peer review procedures used by NIH prior to 2009, the one under which your most recent peer review service occurred.

Based on your most recent review experience using the previous 41-point scale ranging from 1.0 to 5.0, please indicate if you agree or disagree with the following statements.

B1a. The 1.0 - 5.0 scale had sufficient range for me to communicate meaningful differences in the quality of the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B1b. The 1.0 - 5.0 scale allowed me to communicate the strengths and weaknesses of each application.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

Based on your most recent review experience using the previous narrative critique format, please indicate if you agree or disagree with the following statements.

B2a. The narrative critique format allowed me to fully describe my evaluations of the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B2b. The narrative critique format was adequate for capturing the strengths and weaknesses of the applications.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B2c. The narrative critique format helped me complete my critiques efficiently.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

Based on your most recent review experience involving unscored applications, please indicate if you agree or disagree with the following statements.

B3a. The narrative critique format helped the reviewers to decide whether or not to discuss an application at the review meeting.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B3b. The narrative critique format helped me communicate to the applicants why their applications were not scored.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

Based on your most recent review experience, please indicate if you agree or disagree with the following statements.

B4a. My scientific expertise was necessary and appropriately used in the review process.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4b. The other review group members seemed to be experts in their fields.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4c. The format and duration of the discussion was sufficient for reviewers not assigned to evaluate an application to be able to cast well-informed votes.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B4d. An appropriate amount of time was spent discussing the potential impact of the applicants’ research.

O Strongly Agree

O Agree

O Neither Agree nor Disagree

O Disagree

O Strongly Disagree

O Not Applicable

B5. How well did the reviewer orientation and any training materials you received prepare you to review applications using the previous peer review procedures, such as the 41-point scale ranging from 1.0 to 5.0, review criteria, and narrative critique format?

O Very well

O Somewhat well

O Not well at all

SECTION C: Your Opinions about the NIH Peer Review Process Prior to 2009

When answering the questions in this section, please think back to the peer review process at NIH prior to 2009, the one under which your most recent peer review service occurred.

C1. Overall, which peer review system do you prefer – the new system (with enhancements) or the old system (without enhancements)?

O New system

O Old system

O No preference between the new or the old systems

O Not applicable – no experience with the new system

C2. How fair was the peer review process at NIH prior to 2009?

O Very fair

O Somewhat fair

O Neither fair nor unfair

O Somewhat unfair

O Very unfair

C3. How satisfied were you with the peer review process at NIH prior to 2009?

O Very satisfied

O Somewhat satisfied

O Neither satisfied nor dissatisfied

O Somewhat dissatisfied

O Very dissatisfied

18

REVIEWER SURVEY (Version 8-18-09)