OMB# 0925-xxxx
Exp: xx/xx/xx
Last Name, First Name:
Title:
Date:
STPI staff:
STATEMENT OF INFORMED CONSENT
The Science and Technology Policy Institute (STPI), a federally funded research and development center based in Washington, DC, has been requested by the National Institutes of Health (NIH) to evaluate the process by which recipients of the NIH Director’s New Innovator Awards (NIA) were chosen. The primary objectives of the evaluation are to:
assess the NIA award selection process;
examine whether the program was implemented as planned; and
determine if the process was conducted in accordance with the overall mission of the NIA program.
We are employing various data collection techniques to answer these questions; however, we believe that some of the most valuable information will come from those who were involved directly in the evaluation process. These informal interviews are one mechanism that will provide important information concerning the overall NIA process, and will hopefully highlight aspects of the award process that may need to be revised or improved for future rounds of awards. We anticipate conducting approximately 92 such interviews.
Public reporting burden for this collection of information is estimated to average 30 minutes per interview. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: NIH Project Clearance Officer, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA 0925-xxxx.
Please note that:
Your responses will be kept strictly confidential: If you choose to participate, respondent confidentiality will be protected to the extent provided by law, and STPI will report only aggregate information concerning overall impressions of the process to the NIH.
Your participation is entirely voluntary: You are under no obligation to interview with us, but we strongly encourage you to do so. A successful evaluation of the NIA awards process depends on a high response rate to gather as much information and as many perspectives as possible. There are no consequences or risks for participating. Refusal to participate will involve no penalty or loss of benefits to which you are otherwise entitled, and you may discontinue the interview at any time without penalty or loss of benefits to which you are otherwise entitled.
Whom to contact for additional information: For additional information about the study you may contact Bhavya Lal, STPI project director (blal@ida.org). If you have any questions that you would like to address to the NIH Office of the Director, please contact G. Stephane Philogene, Ph.D., the Office of the Director’s Program Officer responsible for this evaluation (e-mail: PhilogeS@OD.NIH.GOV).
If the NIA Extramural Reviewer only has limited time, ask questions 8-10.
[A] Introductory Questions
During your training, were the terms adequately defined and review criteria made explicit? Did NIH explain what was meant by “highly innovative,” “exceptionally creative,” and “potentially high impact?”
[B] Review Criteria
2. Reviewers were instructed to use 3 primary criteria to evaluate individuals (i) scientific problem to be addressed; (ii) innovativeness of the research, and (iii) investigator qualifications). Can you please give an example of how you applied each of the criteria?
(Probing question) Which of the criteria was most important to you in your assessment of the application packages?
(Probing question) How did you decide if an application was responsive/competitive? In other words, what specific markers or characteristics were you looking for in a competitive applicant?
(Probing question) How do you define “innovativeness”? How did you apply it? (if they mention “creative,” “high risk,” “transformative,” ask them to elaborate on how they define and apply those terms)
(Probing question) Do you believe these criteria are adequate to identify an "innovator?" If not, what other criteria might be used in the future years?
3. Aside from the essay, how did the information provided to you in the application package help you to evaluate an individual's "innovativeness?"
a. (Probing question) What role did existing or past grant support play in your review? What role did the CV of the applicant play in your review?
b. (Probing question) approximately what percentage of the applications you reviewed contain preliminary data? How did the presence or absence of preliminary data affect your review?
c. (Probing question) What additional information would have helped you make a better decision?
4. Would blinding the applications affect the review process? How?
5. Which of the 10 research areas (http://grants.nih.gov/grants/guide/rfa-files/RFA-RM-08-014.html) does your research fall under?
1. behavioral and social sciences
2. clinical and translational research
3. instrumentation and engineering
4. molecular biology
5. cellular biology
6. chemical biology
7. pathogenesis
8. epidemiology
9. physiological and integrative systems
10. quantitative and computational biology
How did you evaluate applications outside of your area of expertise? Were there instances where you were not comfortable with reviewing an application because of the subject area?
6. How often did you look individuals up online or consult with others during your review?
[C] Scoring System
Was the scale (1-5) used to rate a package the best way to provide feedback to NIA? If not, what would have been a better way?
(Probing question) How difficult was it to choose the top 4 applications during the process?
(Probing question) (07 evaluators only) Was the process used in 2007 to give applicants feedback (e.g. standardized comments) the best way to give feedback to applicants? Would you recommend that NIA use this method in the future? If not, why not?
(Probing question) Were you concerned about the transparency of the selection process? If so, why?
[D] General Characterization
(Depending on the evaluator’s time, these are the most important questions to ask.)
One of the stated goals of the Innovator Award Program is to “support exceptionally creative new investigators who propose highly innovative approaches that have the potential to produce an unusually high impact […] but not the required preliminary data to fare well in the traditional peer review system.”
(Probing question) Based on your review of applications and any follow up you have had on the program (e.g. you attended the NIA Symposium in September), do you believe that NIA accomplished this goal? What are your thoughts on the awardees - are they "innovators"?
Do you have a sense of how did the nominations that you reviewed compare with those applying through other NIH mechanisms, e.g. R01, R21s?
In your opinion, did NIH truly capture researchers and/or ideas that wouldn't otherwise be in the NIH system?
To the best of your knowledge, to what extent is NIA adding value to the NIH portfolio? (If they’ve been an evaluator for NDPA) How do proposals from NIA differ from NDPA proposals?
(Probing question) How would you characterize the applications you reviewed? How did the awarded and unawarded compare? Were the ideas outside the realm of convention (conceptually or technically risky)?
(Probing question) Were the ideas multidisciplinary?
(Probing question) Were the ideas outside the realm of the investigator's experience?
(Probing question) Did the proposals you reviewed meet your expectations, given the goals of the NIA program?
(Probing question) Have you participated as an evaluator of other research programs? How does NIA differ?
(Probing question) Have you participated in other traditional NIH study sections? How did this process differ? In your opinion, did the NIA process allow you to choose applications that might not be funded under a traditional study section? In what way is your review of high-risk proposals different than traditional proposals?
(Probing question) Have you participated as an evaluator in other high-risk research programs (HHMI, NDPA, TR01, etc)? How does the NIA program differ from those?
(Repeat Evaluators Only) In your opinion, how did this year’s review evaluation of NIA applications compare to that of previous years in terms of clarity, consistency, etc? How did the applicants and proposals compare?
13. Do you have any final feedback on the FY 200X NIA process? Do you think the selection process design or implementation should change? Do you have any other recommendations for how the program could be improved?
14. Given your experiences this year, would you consider being involved in the program next year? (If they say no) What must change for you to participate again?
| File Type | application/msword |
| Author | sshipp |
| Last Modified By | sshipp |
| File Modified | 2009-01-29 |
| File Created | 2009-01-29 |