Supporting Statement A for
Process Evaluation of the NIH Roadmap Interdisciplinary Research Work Group Initiatives, NIDCR
March 23, 2010
Sue Hamann, Ph.D.
Science Evaluation Officer
Office of Science Policy and Analysis
National Institute of Dental and Craniofacial Research (NIDCR)
National Institutes of Health
31 Center Drive, MSC 2190
Building 31, Room 5B55
Bethesda, MD 20892-2190
Phone: 301-594-4849
Fax: 301-496-9988
sue.hamann@nih.gov
Table of contents
A.1 Circumstances Making the Collection of Information Necessary 1
A.2 Purpose and Use of the Information Collection 2
A.3 Use of Information Technology and Burden Reduction 2
A.4 Efforts to Identify Duplication and Use of Similar Information 3
A.5 Impact on Small Businesses or Other Small Entities 3
A.6 Consequences of Collecting the Information Less Frequently 3
A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 3
A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 4
A.9 Explanation of Any Payment or Gift to Respondents 4
A.10 Assurance of Confidentiality Provided to Respondents 4
A.11 Justification for Sensitive Questions 5
A.12 Estimates of Hour Burden Including Annualized Hourly Costs 5
A.13 Estimate of Other Total Annual Cost Burden to Respondents or Record-keepers 6
A.14 Annualized Cost to the Federal Government 6
A.15 Explanation for Program Changes or Adjustments 6
A.16 Plans for Tabulation and Publication and Project Time Schedule 6
A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 7
A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 7
List of Attachments:
Attachment 1. Investigator Survey
Attachment 2. Trainee Survey
Attachment 3. Introductory email
Attachment 4. 30 Day Federal Register Notice
The National Institute of Dental and Craniofacial Research (NIDCR), requests that the Office of Management and Budget (OMB) approve, under the Paperwork Reduction Act of 1995, clearance for NIDCR to conduct data collection efforts for the evaluation of the Interdisciplinary Research Work Group (IDRWG) initiatives. The authority to collect this information is under National Institute of Dental and Craniofacial Research (NIDCR) – 42 USC 285h.
The Interdisciplinary Research Work Group (IDRWG) was established as part of the National Institute of Health’s Roadmap for Medical Research, Research Teams of the Future theme to develop solutions to perceived barriers to interdisciplinary research inherent in the existing structures and processes at the NIH and within academic institutions. The IDRWG, which included representatives from most Institutes and Centers at the NIH, launched several initiatives, each designed to address a specific limitation in the current process of funding allocation and credit sharing. The barriers to interdisciplinary research identified by the IWRG included training of researchers in a single discipline, the mission-oriented (“silo”) organization of the NIH Institutes and Centers, and inequalities in credit sharing at host institutions among researchers submitting collaborative grant proposals. Correspondingly, the IDRWG launched initiatives that had the following aims: formally educate scientists in several diverse disciplines; fund collaborative projects; fund research that promotes the integration of disciplines; and introduce changes to the grant application process to allow more than one Principal Investigator on individual research awards.
The IDRWG took a multi-pronged approach to funding programs that would promote training, discovery, innovation and application grounded in interdisciplinary research and collaborations through the launch of a variety of programs including:
Interdisciplinary Health Research Training: Behavior, Environment and Biology
Short Programs for Interdisciplinary Research Training
Curriculum Development Award in Interdisciplinary Research
Training for a New Interdisciplinary Workforce
Interdisciplinary Research Consortia
Facilitating Interdisciplinary Research via Methodological and Technological Innovation in the Behavioral and Social Sciences
Supplemental grants (Supplements for Methodological Innovations in Behavioral and Social Sciences; Administrative Supplements in Behavioral, Social and Biological Sciences)
These initiatives represent a coherent trans-NIH effort to address specific needs in the current system of scientific research and education, by formally educating scientists in several diverse disciplines, funding collaborative projects, funding research that promotes the integration of disciplines, and introducing changes to the grant application process to facilitate credit sharing between research teams.
The important role that interdisciplinary research is playing in improving human health makes the monitoring and evaluation of these initiatives an important priority for the NIH. The evaluation of these initiatives will help allow the NIH to assess the extent to which these trans-NIH efforts were implemented as planned, to understand at project outcomes, and to help develop future initiatives that have similar goals.
Because of the importance of the IDRWG initiatives to the NIH, a feasibility study was conducted for the process evaluation of these the IDRWG initiatives. The current process evaluation, which is the subject of this application, developed from this feasibility study.
The purpose of this study is to collect data that support the Process Evaluation of the IDRWG initiatives. The proposed data collection is one element of a larger Process Evaluation, which would include other data collection activities not subject to OMB clearance (for example, analyses of program documents). This is the second phase of the IDRWG portfolio evaluation; the first phase, a Feasibility Study, was completed by Abt Associates Inc. The evaluation plan for this Process Evaluation relies heavily on data collected during the Feasibility Study and the findings of that study. The evaluation will be used to determine whether the initiatives have been, and are being, conducted as planned, whether the expected outputs are being produced, and how the activities and processes associated with the initiatives can be improved.
Data collected through activities for which this clearance is requested are not available through other sources. For example, a survey of trainees who participated in programs developed by the IDRWG and PIs of these grants has never been conducted.
Information collected during the evaluation will be used in multiple ways. First, as the NIH Institutes and Centers consider establishing similar programs or participating in similar trans-NIH efforts, the results of this study will help NIH staff determine whether the IDRWG programs are appropriate for their Institute/Center and what changes to the current structures and procedures might be necessary. Second, the results of the evaluation may be used as NIH staff are making decisions on whether to continue and/or to modify the IDRWG programs. Third, it will be used to focus any future impact study and provide context for the interpretation of study results. And finally, any methodological tools and approaches developed for this Process Evaluation could be applied to other evaluation studies involving interdisciplinary research and training.
A Privacy Impact Assessment (PIA) is currently being conducted.
The feasibility study that led to the proposed evaluation design identified the information that could be obtained through extant data sources. Only data not available through other sources will be collected. Steps to reduce the burden on respondents will be taken.
Investigator and student surveys. To reduce burden on respondents the surveys will be implemented in a web-based format. The instrument will be designed to include skip patterns, so that respondents are presented only with the questions that are relevant to their specific situation. Furthermore, to the extent possible, the questions will be in a multiple-choice format, with the choices carefully selected to be applicable to most respondents; space for an open-ended response will be provided to capture more unique answers. Finally, information available from other sources will be pre-loaded so that respondents will need only to verify it (see Attachment 1 for the investigator survey and Attachment 2 for the student survey instrument). The software used for the survey allows respondent status—completed the survey, in process of completing the survey, has not logged in—to be reported in real time. This feature would be used to send reminder emails to non-respondents only.
As part of the feasibility study a thorough assessment of available data sources was conducted to investigate what information is contained within the extant sources and what new data would need to be collected. The alternatives were carefully explored through interviews, literature review, analyses of extant data (including the internal NIH IMPAC II database), and by observing the Interdisciplinary Research Group and grantee meetings.
This comprehensive review revealed that several evaluation areas—related to the development of grants announcements, the application, review and selection processes, program management and grants oversight, grants implementation, and early outcomes—will require primary data collection.
No small businesses will be involved in this study.
Respondents will be contacted only once to collect information for this study.
The IDRWG engaged in a series of grant awards and other non-funded activities to make it easier for scientists to conduct interdisciplinary research. If the data of the proposed evaluation are not collected, the NIH will be unable to document whether these efforts were successfully implemented and whether they have made steps toward the original goals.
The project will fully comply with the guidelines of 5 CFR 1320.5. No special circumstances apply to this data collection.
Comments on this data collection effort were solicited in the Federal Register on January 5, 2010 (p. 382). A copy of the notice is included as Attachment 4. During the first comment period prior to submission to OMB, one comment was received from Jean Public. She requested additional information about the data collection. The draft Supporting Statement and study instruments were sent in response. No additional comments were received, and therefore, NIH is proceeding with seeking approval from OMB.
Consultation on the study design was conducted by a firm contracted by the National Institute on Drug Abuse to prepare an evaluation plan and conduct the feasibility study and by the National Institute of Dental and Craniofacial Research to conduct a process evaluation. The evaluation plan was developed in consultation with the IDRWG members. The proposed design is grounded in extensive background research, involving in-depth interviews regarding design issues, with individuals who included staff at NIH, grantees, and evaluators of similar programs. Further, two external advisors provided input during the feasibility study, and an advisory group has been formed and has provided input into the design of this process evaluation. Finally, the proposed data collection instruments were pilot tested with respondents drawn from the target populations, including grantees and trainees. Respondents were asked to comment on the clarity and content of the questions. The duration of their time spent was recorded to help with an accurate estimation of time burden.
No payments or gift will be provided to respondents.
The contractor conducting the study will be required to adhere to the following procedures:
Access to the electronic files shall be controlled by user ID and by group membership. All paper files (such as handwritten interview notes) shall be stored in locked cabinets. All electronic and paper files shall be destroyed two years after the end of the contract.
Names and other identifiable information shall be redacted in all primary data (interview notes, survey results) and replaced with identifier numbers. A separate file shall be created that links interviewee names to the identifier numbers.
All data shall be reported in aggregate and will not contain any identifying information (such as respondent’s name or affiliation).
Respondents will be provided with the following statement of confidentiality: “The information that you provide will be kept confidential, and will not be disclosed to anyone but the researchers conducting the study, except as otherwise required by law.” Respondents will also be told that participation in the study is voluntary and that there will be no consequences to non-participation.
In the introductory contact (Attachment 3), as well as prior to any data collection (see introductory text in Attachments 1 and 2), individuals will be advised of the purpose and use of the data collection, and the fact that participation is voluntary.
Data collection instruments will not include any sensitive questions.
The requested burden for this evaluation is 250 hours for 450 respondents, which represents the universe of PIs, additional investigators, and graduate students across the IDRWG initiatives to be evaluated. Estimates for the hour burden are based on the pilot test of instruments conducted during the feasibility phase of the evaluation.
A.12–1. Estimates of Hour Burden
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time per Response |
Annual Hour Burden |
IDRWG grantee (PI) |
50 |
2 |
.50 |
50 |
Grant investigators |
100 |
1 |
.50 |
50 |
Trainee |
300 |
1 |
.50 |
150 |
Totals |
450 |
|
|
250 |
A.12-2. Annualized Cost To Respondents
Type of Respondents |
Number of Respondents |
Frequency of Response |
Average Time per Respondents |
Hourly Wage Rate |
Respondent Cost |
IDRWG grantee (PI) |
50 |
2 |
.50 |
40.00 |
2,000.00 |
Grant investigators |
100 |
1 |
.50 |
40.00 |
2,000.00 |
Trainee |
300 |
1 |
.50 |
23.00 |
3,450.00 |
Totals |
450 |
|
|
|
7,450.00 |
There will be no capital, operating, or maintenance costs to the respondents. Record-keeping will be conducted by a contractor (costs detailed in A.14).
NIH staff will be involved in the supervision of the evaluation, including the Project Officer, Evaluation Advisory Committee, and OMB Clearance Officer. The estimate of the total NIH staff time is three months. With an average salary of $80,000, this adds $20,000 in NIH staff costs. Contractor costs are estimated at $126,693. Thus, the total cost to the Federal Government is estimated at $253,387
A.14-1. Annualized Cost to the Federal Government
Annualized Cost |
Amount |
NIH evaluation oversight |
$20,000 |
Contractor information collection fees |
$126,693 |
Total |
$146,693 |
This is a new collection of information.
The data collection will be conducted by an outside contractor firm, Abt Associates, that will work with the Project Officer. There are no plans for complex analytical techniques.
To provide the NIH understanding of the implementation of the IDRWG initiatives, the contractor will prepare a report for the NIH that describes the study and findings. The report will characterize the research and educational activities supported by the initiatives. Descriptive analysis of extant program data that provides information about the activities offered by the consortia and other grantees will be used to frame the findings of the study. Analyses will include descriptive analyses, using the measures of central tendency and frequency distributions that describe students’ participation in center activities, their other educational experiences, and their education and career paths. Correlational analyses will be used to look at the relationship between participation in center activities and subsequent education and career paths.
A.16-1 Project Time Schedule
Activity |
Schedule |
Develop web-based survey instrument |
1-2 months after OMB approval |
Recruit survey respondents |
3-6 months after OMB approval |
Implement web-based survey |
7-9 months after OMB approval |
Analyze data |
10-15 months after OMB approval |
Report findings |
16-24 months after OMB approval |
The data collection instruments will display the expiration date.
No exceptions are sought.
| File Type | application/msword |
| File Title | Supporting Statement 'A' Preparation - 05/01/2008 |
| Subject | Supporting Statement 'A' Preparation - 05/01/2008 |
| Author | OD/USER |
| Last Modified By | MartinezA1 |
| File Modified | 2010-03-23 |
| File Created | 2010-03-23 |