Supporting Statement A

6

Supporting Statement for the

Paperwork Reduction Act Submission

Part A: Justification

National Institutes of Health

A Process Evaluation of the NIH Director’s

Pioneer Award (NDPA) Program

This request seeks a reinstatement of a currently approved collection related to the NIH Director’s Pioneer Award (NDPA) Program Process Evaluation. The data collection consists of one survey to be completed by a sample of participants (applicants and awardees). The information gathered from these surveys will document the NDPA program operations and will be used as a guide to the program officers in their future strategic and management decisions.

A.1 Circumstances Making the Collection of Information Necessary

The NDPA Program was initiated in 2003 by the Office of the Director (OD) at the National Institutes of Health (NIH) to identify, encourage, and support people and ideas geared toward high-risk/high-impact research. The program awards are up to $500,000 in direct costs each year, for five years. The objective is to encourage pioneering scientists to cross traditional disciplinary lines, bring forward novel conceptual frameworks as well as methodologies in order to pursue breakthroughs in biological and medical sciences. For program description and requirements, the RFA is appended in Attachment 1.

NDPA has evaluated three competitions (FY2004 - FY2006), and the OD wishes to extend its OMB clearance to complete the evaluation for the remaining two years of the pilot program (FY2007 and FY 2008).. The primary objectives of the study are to: (1) assess the NDPA award selection process; (2) determine if the program was implemented as planned; and (3) determine if the process was conducted in accordance with the overall mission of the NDPA program.

As reflected in U.S. Code: Title 42 USC §6686, Congress established the Science and Technology Policy Institute (STPI or the Institute) to assist the government in formulating and evaluating federal policy and programs by providing objective, high quality analytic support. Consistent with its legislative authority, STPI is tasked by NIH to evaluate the NDPA candidate selection process in an annual basis through survey collection and targeted interviews.

A.2 Purpose and Use of the Information

The purpose of the survey is twofold: to assess the NDPA program's selection process and to collect information that could be used by the NDPA program officers to guide strategic and management decisions. Samples of the survey are appended.

Specifically, information will be collected for three processes:

• Application process: activities related to submitting an application package, including: the time provided to complete a package; the form used (Standard Form 424); and components of the application, including the personal essay.

• External review process: activities related to reviewing and scoring applications, including the application of criteria.

• Interview process: activities related to the interviews of leading candidates. These activities include: preparation time; duration of the interview; and the opportunity to present ideas during the interview, and panel deliberations.

Once information from the survey has been gathered and analyzed, a report will be prepared that documents the nature and extent of NDPA activities and outputs and overall program efficiency and effectiveness. The findings of the evaluation will be presented to the OD management and will be used by them to determine how the program and its administration may be altered and improved in future years.

A.3 Use of Information Technology and Burden Reduction

The proposed surveys use Internet technology in order to minimize burden on respondents. All respondents will be sent a link to a Web-based survey that can be completed and submitted on-line. Non-respondents will be followed-up by email and/or telephone. We expect that some follow-up may be necessary with respondents, especially those who made it to the latter phases of the review process, to clarify open-ended responses that are submitted. In the process of the survey design significant effort was made to ensure easy access on-line. In addition, the majority of the survey questions are closed-ended (e.g. ‘yes/no’) to reduce burden on respondents. Finally, all information that is available from other sources (e.g. participant names and affiliations) will be preloaded, so that respondents will need only to verify it.

A.4 Efforts to Identify Duplication and Use of Similar Information

The proposed survey does not gather data that duplicates other efforts. Because a process evaluation of the NDPA program has never been undertaken, information similar to that which we propose to collect is currently unavailable.

The NDPA staff has examined carefully the data that are available on individual NDPA applicants in order to assure that no information is replicated under the present evaluation. While NDPA applicants submitted information on their most significant accomplishments, creativity, motivation, and communication skills, this does not contain answers to questions posed in the survey (some examples are listed below). Factual information from the application process will be preloaded into the survey, as discussed in section A.3.

Only direct contact with program participants can yield information such as:

• Where applicants have received previous funding (e.g. have they received grants from other institutions geared toward high-risk research such as Howard Hughes Medical Institute and the Department of Defense’s Defense Advanced Research Projects Agency);

• How participants heard of the NDPA program;

• If the criteria used to evaluate grants is perceived as useful in identifying “pioneers;”

• Factors that make a given application successful

• Level of satisfaction of participants, both applicants and reviewers.

A.5 Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this study. All survey respondents are biomedical researchers who have participated in the NDPA program.

A.6 Consequences of Collecting the Information Less Frequently

The surveys will only be conducted one time; there will be no periodicity. The NDPA program is a pilot program, and if the information is not collected, NIH will be unable to document the outputs of the NDPA program and the OD will not be able to identify concerns and improve the program.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This project fully complies with all guidelines of 5 CFR 1320.5.

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency

A copy of the 60-day Federal Register notice (Vol. 72, No. 151, Tuesday August, 7 2007, pp. 44165 – 44166) can be found as an attachment. The announcement received no public comment or suggestions regarding the study.

The Science and Technology Policy Institute had several in-person meetings and conference calls with OD project officers, who provided important background information and made suggestions on the evaluation design.

A.9 Explanation of Any Payment or Gift to Respondents

Respondents will not be paid for participating in this survey and will not receive any gifts in return for participation. Participation is completely voluntary.

A.10 Assurance of Confidentiality Provided to Respondents

The subjects will be informed that their responses to the questionnaire are to be reviewed by the Science and Technology Policy Institute for purposes of analysis and reporting. Given the nature of the study, assurance of confidentiality is provided to respondents. Some materials may be disseminated in aggregate to the public by NDPA staff in order to inform the research community of the results of the study, although no quotes or specific answers will be attributed to individual respondents. NIH’s System of Records and privacy procedures under the Privacy Act are included in the Privacy Act System of Records Notice (Federal Register Vol. 67 No. 187, September 26, 2002, pp/ 60741-60794,), and attached as Attachment 4.

The survey will contain the following general assurance of confidentiality:

“…If you choose to participate, your responses to the survey will be kept strictly confidential. Your responses will be submitted to STPI, where the information will be kept secure. Information reported to NIH will be limited to aggregate results and overall impressions, and every attempt will be made to mask the identity of individual respondents. The NIH or any other outside parties will not have access to raw survey data or to the identity of individual respondents at any time....”

In order to ensure data security, all employees of the Science and Technology Policy Institute, are required to adhere to strict standards and sign a non-disclosure agreement as a condition of employment (for details, please consult Attachment 5). The Science and Technology Policy Institute has extensive experience collecting information and maintaining its confidentiality and security. All data files on multi-user systems will be under the control of the database manager, with access limited to project staff on a “need to know” basis only. Finally, respondent access to the web-based survey will be password-protected.

­

A.11 Justification for Sensitive Questions

There are no questions of a sensitive nature in the surveys. In order to understand the background characteristics of applicants, the questionnaire does ask where a given applicant has received previous funding. This information will be used analytically to determine if the NDPA program is attracting individuals with “pioneer” backgrounds, i.e. has received funding from other programs or institutions that are known to support high-risk/high-impact research. Respondents may not know or may choose not to provide the information that they feel is privileged, such as previous funding sources. As discussed in the previous section, the Science and Technology Policy Institute will hold individual data strictly confidential, and any public reporting of the data will be in aggregate form that will not allow for the identification of individuals.

A.12 Estimates of Hour Burden Including Annualized Hourly Costs

Every effort will be made to minimize the burden on the respondents. The instruments have been reduced to the minimum possible length (e.g., through the use of largely closed-ended question formats) and should take, on average, 15 minutes to complete (.25 hours). In addition, each person’s name, address, institution, etc. will be entered on the survey form in advance based on the NDPA program records, and space will be available for respondents to make changes where any of this information is incorrect. Respondents will be given up to 4 weeks to reply and will only have to fill out one questionnaire, which will be Web-based. No copies will have to be made or sent. The survey will be conducted only once, not repeatedly. Respondents do not need to retain or consult records for purposes of this survey.

Further reductions in the instrument length would jeopardize the power of the survey to accurately assess the program. Because program participants come from many institutions and backgrounds, reducing the respondent population would have a negative impact of our ability to identify true differences between important subpopulations: for example, researchers from the social sciences versus natural sciences, or males versus females.

Based on estimated response rates from the last few years and hour burden we anticipate the following total burden:

Annualized Estimate of Hour Burden
Type of Respondents	Number of Respondents	Frequency of Response	Average Time for Response (hr)	Total Hour Burden*
Applicants	600	1	.25	150
Extramural evaluators	110	1	.25	27.50
Total	710	1	.25	177.50

Total Burden = N Respondents*Response Frequency*minutes to complete/60

Thus, the expected burden level for this study is 177.50 hours.

An hourly earning rate for participants was estimated using the U.S. Department of Labor, Bureau of Labor Statistics National Compensation Survey (http://www.bls.gov/ncs/ocs/sp/ncbl0658.pdf).

NDPA program officers provided a list of research areas of applicants. Based on this list, relevant professions listed in the NCS (e.g. engineers, mathematical and computer scientists, natural scientists, physicians, and college and university professors or relevant disciplines) were averaged. The average hourly earnings of this group is $39.17. Given that nominators and nominees are likely leading biomedical scientists, we assumed an addition 40% in hourly earnings, for an estimated hourly wage of $55.00. With an average of 15 minutes of time required to complete the questionnaires (see Section A.12 above), the annual cost for the applicants for a Pioneer Award would equal approximately $9,662.50.

Annualized Cost to Respondents
Type of Respondents	Number of Respondents	Response Frequency	Approx. Hourly Wage Rate	Total Respondent Cost**
Applicants	600	1	$55.00	$8,250
Extramural evaluators	110	1	$55.00	$1,512.50
Total	710	1	$55.00	$9,662.50

**Total Respondent Cost = N Respondents* Response Frequency*minutes to complete/60* hourly rate.

A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers

There are no capital, maintenance or operating costs to respondents.

A.14 Annualized Cost to the Federal Government

A.14.1 Annualized Cost to Contractor

The cost to maintain and implement the survey, including contractor’s fixed fee will be $15,000. This does not include analyses of collected data or preparation of reports.

A.14.1 Estimate of Other Total Annual Cost Burden to Contractor
Survey maintenance and software licenses	$5,000
Website design and support	$5,000
Record keeping and follow up	$10,000
Total	$20,000

A.14.2 Annualized Cost to the Federal Government

Annualized Cost to the Federal Government is composed, in part, of an aggregate estimate from Items A.12 and the information above, as this is a one-time survey that will require less than one year to complete. In addition, there are costs of the NDPA Project Officer, NIH OMB Clearance Officer, other NDPA professional staff, and support staff time. Based upon a discussion with the Project Officer, we have estimated that approximately a quarter of a year’s time is required in association with the conduct of this study. With an average salary of $80,000, this adds $20,000 in NIH costs. Thus, total cost is $49,662.50 (Table A.14.2).

A.14.2 Total Cost Burden of Information Collection
Annualized Cost to Respondents	$9,662.50
Other Annual Cost to Contractor (from A.14.1)	$20,000
NIH/NDPA Staff Time	$20,000
Total	$49,662.50

A.15 Explanation for Program Changes or Adjustments

The NDPA Process Evaluation is a reinstatement of a previously approved collection. There are no program changes or adjustments. Based on the results of three previous annual collections and ongoing trends, we estimate that the number of survey respondents will continue to be in the range of 500 to 700 with an estimated average of about 600 applicants.

A.16 Plans for Tabulation and Publication and Project Time Schedule

The NDPA Process Evaluation began in September 2004, and will end in August 2009. The NDPA Project Officer has reviewed and approved the survey instrument prior to its implementation. The survey will take place in fall of each year. The evaluation contractors are required to deliver a draft final report on the evaluation by August, 2009.

A.16 Estimated Annual Project Time Schedule
Activity	Time Schedule
Survey implementation	1-2 months after OMB approval (November-December, 2007)
Data analyses	4-5 months after OMB approval (February-March 2007)
Report writing, dissemination	8-9 months after OMB approval (July-August 2007)

A.17 Reasons Why Display of OMB Expiration Date is Inappropriate

No exceptions are sought; the OMB Expiration Date will be displayed on the survey instruments.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are sought from the Paperwork Reduction Act or from form 83-I.

A.19 Certification of Paperwork Reduction Act Submissions (OMB 83-I)