Supporting Statement A_NHGRI Short Course_final

Supporting Statement A For:

Surveys to Support an Evaluation of the National Human Genome Research Institute (NHGRI) Summer Workshop in Genomics (Short Course)

Updated December 14, 2015

Carla L. Easter

Education and Community Involvement Branch

National Human Genome Research Institute

Building 31, Room B1B55

Bethesda, MD 20892

Telephone: 301-594-1364

Fax: 301-480-5008

E-mail: easterc@mail.nih.gov

Table of Contents

ABSTRACT 1

A. Justification 1

A.1 Circumstances Making the Collection of Information Necessary 1

A.2 Purpose and Use of the Information Collection 3

A.3 Use of Improved Information Technology and Burden Reduction 14

A.4 Efforts to Identify Duplication and Use of Similar Information 14

A.5 Impact on Small Businesses or Other Small Entities 14

A.6 Consequences of Collecting the Information Less Frequently 14

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 15

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 15

A.9 Explanation of Any Payment or Gift to Respondents 15

A.10 ASSURANCE OF CONFIDENTIALITY PROVIDED TO

RESPONDENTS……………………………………………………………………………………………..15

A.11 Justification for Sensitive Questions 16

A.12 Estimates of Annualized Burden Hours and Costs 16

A.13 Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers 18

A.14 Annualized Cost to the Federal Government 18

A.15 Explanation for Program Changes or Adjustments 19

A.16 Plans for Tabulation and Publication and Project Time Schedule 19

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate 20

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions 20

List of Attachments

Attachment 1: NHGRI Short Course Survey

Attachment 2: Background and Rationale for Short Course

Attachment 3: IRB Exemption Notice

Attachment 4: Invitation Letter to Short Course Survey for Students and Faculty

Attachment 5: Applicability of the Privacy Act

ABSTRACT

The National Human Genome Research Institute’s (NHGRI) Education and Community Involvement Branch (ECIB) has been administering the NHGRI Summer Workshop in Genomics (referred to as “Short Course”) since 2003, as a way to accelerate the dissemination of genetic and genomic information to science faculty, especially those at minority serving institutions. The Short Course is an intensive multi-day educational workshop designed to update biology instructors, as well as other instructors and researchers in related disciplines, on genomic science.

NHGRI has collected course evaluations from active participants immediately upon course completion, and used that data to inform presenters for the next year. However, there is an absence of information about if, and how the new knowledge and skills gained by participants has been applied since participating in the program, and whether or not the course is meeting program goals. Therefore, NHGRI will electronically survey past program participants (from 2004-2012) to explore key areas including curriculum updates and integration; disseminating genomics beyond the classroom, participants’ educational and career pathways, and the extent of continued engagement with NIH and NHGRI. The evaluation will be used to inform program funding decisions, future program design, and needed data collection and measurement improvements for future cohorts.

A. Justification

A.1 Circumstances Making the Collection of Information Necessary

The National Human Genome Research Institute (NHGRI), part of the National Institutes of Health (NIH), supports the development of resources and technology that will accelerate genome research and its application to human health. Within NHGRI, the Education and Community Involvement Branch (ECIB) has been administering the NHGRI Summer Workshop in Genomics (aka, the Short Course) since 2003, as a way to accelerate the dissemination of genetic and genomic information to science faculty, especially those at minority serving institutions.

The Short Course is an intensive multi-day educational workshop designed to update biology instructors, as well as other instructors and researchers in related disciplines, on genomic science. The course focuses on the continuing effort to find the genetic basis of various diseases and disorders, and current topics on the ethical, legal and social implications of genomics. The course targets college and university faculty seeking to update their curriculum or develop new courses related to genetics. Preference is given to applicants from racial and ethnic groups underrepresented in health related sciences; institutions that predominantly train students with disabilities; or disadvantaged backgrounds including certain rural and inner-city environments. Students from those same circumstances have been included in the program since 2004, but student selection has evolved. Initially, faculty attendees selected students for attendance, and then in 2012, NHGRI began actively recruiting, screening and accepting graduate students independent of faculty attendees, as a way of preparing and attracting future scientists and health care professionals to a genomics workforce. Additional background is provided in Attachment 2.

The Short Course is a highly visible program within NHGRI and is considered by NHGRI leadership as an important investment by the Institute. The Short Course annual budget is approximately $80,000 and represents a total investment of almost one million dollars since its inception. Each year, more than 15 NIH faculty are recruited (from NHGRI and other ICs) to serve as instructors and presenters for the Short Course; 1 to 2 NHGRI staff are responsible for coordinating the logistics and planning throughout the year. As genomic discoveries continue to accelerate, and the need for an informed and educated workforce and public becomes more relevant to health care and decision-making, NHGRI wants evidence of Short Course performance results to inform future course offerings.

NHGRI has assessed the course each year by gathering feedback from participants immediately upon course completion, and using that data to inform the next year, but a multi-year mixed-methods evaluation focused on longer-term outcomes has not been conducted. There is an absence of information about if, and how, the new knowledge and skills gained by participants have been disseminated over the years, and whether or not the course is meeting program goals. The collection of information activities set forth herein will be conducted under the authorities granted in the Public Health Service Act, Title 42 USC 285s.

A.2 Purpose and Use of the Information Collection

NHGRI completed a feasibility study in 2013, which has provided the framework for the current evaluation design. The feasibility study (see appendix in supplemental evaluation plan) involved interviews with five key stakeholders, analysis of annual participant evaluations, and a review of annual project documents and recruitment materials. As a result of the feasibility study, program goals (short, medium and long-term goals) and objectives were formally articulated with development of a logic model, evaluation questions, and potential data collection techniques; all components were assessed for efficiency and effectiveness and the recommendation was electronic survey of program participants from 2004-2012.

In addition to the feasibility study, each program year, administrative staff gathered feedback from participants directly following their participation in the Short Course. The evaluation forms typically offered a three-point rating scale and space for open-ended comments in response to questions, which focused on the quality of: course content; presenters; workshop activities; and supports. Feedback was used to evolve the program in successive years, and although annual process evaluations, applicant data and anecdotal evidence all suggest the Short Course is a valuable training opportunity, popular among educators and highly recommended by past-participants, formal evidence is unavailable.

The proposed web-based survey is intended to provide additional descriptive data to help determine if program participants report evidence of supporting interim indicators of long-term program goals following the Short Course:

1. Prepare the next generation of genomics professionals for an era of genomic medicine

2. Train and diversify the pipeline of genome professionals in alignment with the NIH and US Department of Health and Human Services diversity and inclusion efforts

The web-based survey of past program participants (faculty and students) from 2004 to 2012 will assess the following specific medium-term outcomes and contextual factors (see logic model in supplemental evaluation plan attached).

Medium Term Outcomes:

• New knowledge integrated into existing teaching materials

• Updated curriculum at faculty institutions

• Dissemination of genomics beyond the classroom (professional related activities)

• Continued participation in NIH-related activities

• Student participant’s pursuit of a career related to genomics (broadly defined)

Contextual Factors:

• Resources and barriers associated with curriculum integration

The survey protocol (Attachment 1) obtains different perspectives on the Short Course from students and faculty by using skip logic. The table below outlines which questions are for faculty respondents, student respondents, and questions for both respondents.

Respondent Type	Survey Question
Both	Are you still with [PIPED INSTITUTION NAMED]?
Both	Did the Short Course influence your research?
Both	Do/did you have a disadvantaged background, which can be defined either as coming from a family with an annual income below established low-income thresholds, or coming from an educational environment such as that found in certain rural or inner-city environments that have demonstrably and directly inhibited you from obtaining the knowledge, skills, and abilities necessary to develop and participate in a research career?
Both	Does your current position involve teaching genetic or genomic material?
Both	Have you completed the credential or degree(s)?
Both	Please describe how the Short Course influenced your decision to pursue new career or educational options.
Both	Please describe how your experience at the Short Course influenced your research.
Both	Please enter any other comments you would like to share with us about the Short Course and your experiences since attending the program. If you have no additional comments, please hit the next button.
Both	Please identify any credentials or degrees that you have received or pursued since attending the Short Course. (Select all that apply)
Both	Please indicate the ways you have been involved with Short Course attendees, NIH or NHGRI since attending the program. (Select all that apply)
Both	Please list your current institution or organizational affiliation.
Both	Select your role during your participation in the Short Course.
Both	Since the Short Course, have you been involved with Short Course attendees, NIH or NHGRI?
Both	To what degree did your experience at the Short Course influence your decision to engage in the following activities? Select "Not Applicable" if you did not engage in the activity listed.
Both	To what degree did your experience at the Short Course influence your decision to pursue new career or educational options?
Both	What discipline(s) is your credential or degree associated with?
Both	What is your ethnicity?
Both	What is your sex?
Both	What race do you consider yourself? (Select all that apply)
Both	What year were you born?
Both	Which of the following best describes your primary field of work?
Faculty	Beyond curriculum integration, in what other ways did you disseminate information from the Short Course to students? If you did not disseminate information beyond curriculum integration, please write “None.”
Faculty	Did you hold a teaching appointment at the time of your attendance to the Short Course?
Faculty	Has your Position or Occupation Title changed from [PIPE IN SELECTED CHOICE] since the time of your attendance in the Short Course?
Faculty	How many year(s) have you been engaged in teaching? (Numbers only)
Faculty	How much time was required to update your curriculum and teaching materials after participating in the Short Course?
Faculty	I was able to update my curriculum and teaching materials in the following content areas: (Select all that apply)
Faculty	I was able to update my genetics or genomics curriculum and teaching materials in the following ways: (Select all that apply)
Faculty	Out of 100 percent, what best represented your teaching/research ratio at the time?
Faculty	Out of 100 percent, what best represents your current teaching/research ratio?
Faculty	Please rate how much you think the following institutional factors influenced your ability to transfer your knowledge to students following participation in the Short Course. Select "Not Applicable" if the factor was not relevant to your situation.
Faculty	Please rate how much you think the following student-related factors influenced your efforts to transfer your knowledge to students. Select "Not Applicable" if the factor was not relevant to your situation.
Faculty	Please select one of the three options below in response to the following statement: I was able to update my curriculum as a result of my participation in the Short Course.
Faculty	Please specify whether this is a tenure or non-tenure track position.
Faculty	To what degree did your experience at the Short Course influence your teaching to research ratio?
Faculty	Was this a tenure or non-tenure track position?
Faculty	What is your current primary Position or Occupation Title?
Faculty	What was your primary Position or Occupation Title at the time of your attendance to the Short Course?
Student	Out of 100 percent, what best represents your current teaching/research ratio?
Student	Please specify the type of instructor or professor position.
Student	Please specify trainee type.
Student	Please specify whether this is a tenure or non-tenure track position.
Student	Select the content areas you teach: (Select all that apply)
Student	What degree program were you enrolled in when you attended the Short Course?
Student	What is your current primary Position or Occupation Title?
Student	What type of degree program are you enrolled in?

Although this evaluation is not designed or intended to provide causal inferences, the descriptive data from the survey will provide valuable insight on the pathway of program participants following the Short Course and whether these participants report behaviors determined to be indicators of desired medium- and long-term outcomes. Thresholds for success have not been established. This descriptive data will provide insight on the presence or absence of achieving specific outcomes and can inform the development of program thresholds or outcome targets in the future by establishing a baseline. The table below provides examples of each outcome, how it will be measured from the survey, and examples of use of information from evaluation findings.

Outcome Area	Survey Question	Example of Use
Continued participation in NIH-related activities (Medium-Term)	Please indicate the ways you have been involved with Short Course attendees, NIH or NHGRI since attending the program. (Select all that apply)	If findings reveal participants have continued interactions with presenters but not other Short Course participants, NHGRI could develop more strategies for promoting participant networking during, and after, the Short Course concludes.
	Since the Short Course, have you been involved with Short Course attendees, NIH or NHGRI?
Dissemination of genomics beyond the classroom (professional related activities) (Medium-Term)	Beyond curriculum integration, in what other ways did you disseminate information from the Short Course to students? If you did not disseminate information beyond curriculum integration, please write “None.”	If findings indicate participants have not disseminated genomics knowledge beyond the classroom in other professional related activities, NHGRI may incorporate more time in the Short Course curriculum to discussing best practices for this extended dissemination.
	Did the Short Course influence your research?
	Please describe how your experience at the Short Course influenced your research.
	To what degree did your experience at the Short Course influence your decision to engage in the following activities? Select "Not Applicable" if you did not engage in the activity listed.
Increased genomics knowledge by students at faculty’s institution (Medium-Term)	Please rate how much you think the following student-related factors influenced your efforts to transfer your knowledge to students. Select "Not Applicable" if the factor was not relevant to your situation.	Depending on the relevant student factors, NHGRI could develop materials to help mitigate potential barriers to knowledge transfer to students.
New knowledge integrated into existing teaching materials (Medium-Term)	I was able to update my curriculum and teaching materials in the following content areas: (Select all that apply)	This finding will inform NHGRI as to the major content areas where curriculum changes occurred and identify potential gaps for future courses.
	I was able to update my genetics or genomics curriculum and teaching materials in the following ways: (Select all that apply)
Pursue a career related to genomics (broadly defined) (Medium-Term)	Does your current position involve teaching genetic or genomic material?	These questions will provide additional insight into whether program participants continue to pursue careers related to genomics. These questions will also inform the long-term outcome of “training and diversifying the pipeline of genomics professionals” by increasing understanding of the potential pathways (e.g., focus on teaching versus research). NHGRI may also use the findings to target specific types of faculty (e.g., non-tenured faculty with an interest in teaching genomics) depending on identified gaps.
	Out of 100 percent, what best represents your current teaching/research ratio?
	Please describe how the Short Course influenced your decision to pursue new career or educational options.
	Please specify the type of instructor or professor position.
	Please specify trainee type.
	Please specify whether this is a tenure or non-tenure track position.
	Select the content areas you teach: (Select all that apply)
	To what degree did your experience at the Short Course influence your decision to pursue new career or educational options?
	To what degree did your experience at the Short Course influence your teaching to research ratio?
	What is your current primary Position or Occupation Title?
	Which of the following best describes your primary field of work?
Pursuit of coursework/learning opportunities related to genomics (Medium-Term)	Have you completed the credential or degree(s)?	Findings would provide additional insight into potential educational-related outcomes of participation in the Short Course and whether participants continue to pursue genomics and genetics-specific fields.
	Out of 100 percent, what best represents your current teaching/research ratio?
	Please identify any credentials or degrees that you have received or pursued since attending the Short Course. (Select all that apply)
	What discipline(s) is your credential or degree associated with?
	What type of degree program are you enrolled in?
Updated curriculum at faculty institutions (Medium-Term)	How much time was required to update your curriculum and teaching materials after participating in the Short Course?	Findings may inform strategies to accelerate updating curriculum, or more broadly, expectations on time required to achieve this medium-term outcome.
	Please select one of the three options below in response to the following statement: I was able to update my curriculum as a result of my participation in the Short Course.
Train and diversify the pipeline of genome professionals (Long-Term)	Has your Position or Occupation Title changed from [PIPE IN SELECTED CHOICE] since the time of your attendance in the Short Course?	These questions will provide insight into the career pathway for program participants (both faculty and students). Coupled with demographic data on program participants, findings can provide insight into the pipeline of genome professionals who completed the Short Course.
	Please specify whether this is a tenure or non-tenure track position.
	What is your current primary Position or Occupation Title?

The majority of the examples in the table provide insight into how NHGRI may use the descriptive findings from the survey. Looking critically at NHGRI’s effort to provide cutting edge information to science faculty at minority serving institutions will inform other ICs and trans-NIH groups interested in connecting to, and training, faculty at minority-serving institutions. Specifically, by exploring the degree and type of curriculum integration (i.e., “I was able to update my genetics or genomics curriculum and teaching materials in the following ways: (Select all that apply),” the barriers experienced when trying to achieve curriculum integration (i.e., “Please rate how much you think the following institutional factors influenced your ability to transfer your knowledge to students following participation in the Short Course”), and the diffusion of knowledge (i.e., “Beyond curriculum integration, in what other ways did you disseminate information from the Short Course to students?”), the evaluation will shed light on the program model’s potential to enhance education of undergraduate students at minority-serving institutions.

The findings for the medium-term outcomes also serve as interim indicators for long term outcomes. For example, if faculty and student participants report a positive program influence on their decision to pursue new career or educational options, this will provide one piece of evidence the program is meeting the long-term goal to train and diversify the pipeline of genome professionals. If evidence shows the program is not meeting its goals, the current NHGRI Director would like to make programmatic changes to meet the institute’s mission and strategic plan.

NHGRI leadership will share evaluation findings with the NHGRI Director and other key decision-makers to help guide the scope of the program, and branch level-chief will use the results to inform the type and scope of program activity. If effective, the Short Course may stand as a model program for other ICs to follow as an effective means for disseminating cutting-edge information and for increasing diversity in the scientific workforce.

A.3 Use of Improved Information Technology and Burden Reduction

The use of web-based surveys will reduce the burden to respondents. A database will be developed that contains the names of the respondents, contact information, and the dates of all email and telephone contacts. All quantitative data will be standardized and compiled into a searchable database. If deemed necessary and worthwhile by the evaluation lead, more sophisticated data management and quantitative analysis software tools (e.g. SPSS software) may be employed.

In addition, a Privacy Impact Assessment (PIA) is being conducted.

A.4 Efforts to Identify Duplication and Use of Similar Information

This is the first evaluation focused on longer-term outcomes conducted since the Short Course’s inception in 2003. The Short Course is unique to NIH.

A.5 Impact on Small Businesses or Other Small Entities

No small businesses will be involved in this study.

A.6 Consequences of Collecting the Information Less Frequently

This is a one-time collection.

A.7 Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This study complies fully with the guidelines of 5 CFR 1320.5. No exceptions to the guidelines are required.

A.8 Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

A 60-day Federal Register Notice (Volume 80, page 13845) was published on March 17, 2015 and allowed 60 days for public comment and review. There were no comments.

A.9 Explanation of Any Payment or Gift to Respondents

No payment or gift will be made to the respondents.

A.10 Assurance of Confidentiality Provided to Respondents

Respondents will be provided with information about the voluntary survey and relevant evaluation contact information on the Start Page prior to giving consent and continuing with the Web-based survey. Individuals completing this survey will be at least 18 years of age. The survey platform, Survey Analytics, meets all FIPS security and Section 508 Accessibility requirements. Respondents will have the option to skip any question they would prefer not to answer and to quit the survey at any time. Unless express permission is provided by the respondent, all data will be de-identified and reported in the aggregate.

To protect the security of respondents’ information, all project files will be password protected and access to the files will be limited to authorized project staff. All contractors are required to sign an agreement developed by eRA that says that they agree to protect grant data. In addition, project members will be notified in writing prior to the survey data collection process their responsibilities for protecting the information collected. Survey information will be stored on a secure server protected with a Secure Sockets Layer (SSL) certificate and 128-bit encryption, the strongest online data encryption protection available. The tracking database with individual contact information will be stored separately from the data. Contractors need access to e-mail and phone information for grantees to faciliate data collection and follow-up. This includes distributing the survey via e-mail and following up via phone and e-mail to ensure a sufficient response rate. The database will contain IDs only.

The tracking database that links IDs to individual information will be destroyed at the end of the project. This includes removing variables that may identify respondents if viewed in conjunction with other variables. Project reports will not identify individuals who completed the survey. No names or personal identifying information will be used in any published reports of this study unless given express permission from the respondent. Survey reports will present all findings in aggregate so individual responses cannot be identified.

The NIH Privacy Act Officer has reviewed this application and has determined that the Privacy Act is applicable (Attachment 5). Additionally, the Office of Human Subjects Research Protection (OHSRP) has reviewed this project and deemed it does not apply (Attachment 3).

A.11 Justification for Sensitive Questions

The questions being asked do not require responses that will include privacy information.

A.12 Estimates of Annualized Burden Hours and Costs

Web-based surveys will be issued to approximately 300 respondents. This includes all 299 participants, out of 314 total between 2004 and 2012 (186 faculty and 111 students), where the evaluation team has initial contact information. These surveys should take approximately 15 to 30 minutes for each respondent. For burden estimate purposes, we have used the conservative end of the range at 30 minutes. Based on this, it is estimated that the total and annualized burden will be 150 hours over a year information collection request (Table A.12-1).

Table A.12-1 Estimate of Annual Burden Hours

Form Name	Type of Respondents	Number of Respondents	Number of Responses per Respondent	Average Burden per Response (in hours)	Total Annual Burden (in hours)
Short Course Survey	Students and Faculty	300	1	30/60	150
Totals		300			150

Faculty respondents are comprised primarily of scientists and health professionals; therefore, a mean hourly wage was calculated taking the average of three occupations: $42.98 Medical Scientists (occupation code 19-1040); $37.32 Biological Scientists (occupation code 19-1020); and $44.87 Health Diagnosing and Treating Practitioners (occupation code 29-1000). The May 2013 National Occupational Employment and Wage Estimates - United States (http://www.bls.gov/oes/current/oes_nat.htm#19-0000) was used for the above general categories, which were averaged and resulted in $41.72 per hour. Student respondents are at the graduate student level. An hourly wage was calculated taking the average monthly stipend levels for graduate students at the Office of Intramural Research at NIH and dividing this value by 160 (20 8-hour days a month). This resulted in $17.20 per hour (http://oma1.od.nih.gov/Manualchapters/person/2300-320-7/Appendices/Student14.PDF). The total and annualized cost to respondents is estimated to be $4,909.40 annually (Table A.12-2).

Table A.12-2 Annualized Cost to Respondents

Type of Respondents	Number of Respondents	Total Annual Burden Hours	Hourly Wage Rate*	Respondent Cost
Short Course Students	110	55	$17.20	$946.00
Short Course Faculty	190	95	$41.72	$3,963.40
Total				$4,909.40

A.13 Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no direct costs to respondents other than their time to participate in the study.

A.14 Annualized Cost to the Federal Government

The total and annualized cost to the Federal Government is approximately $124,442.00 for this information collection. Non-federal personnel costs include the salary and benefits of a project director (contractor), research associates, and a survey statistician for approximately 4 to 5 months. Federal personnel costs include several hours of participation from 2 program officers and an NIH privacy liaison over 4 to 5 months. Materials costs include hosting charges for Survey Analytics. The total costs are in Table A.14-1.

A.14-1 Estimate of Total Cost-Government

Personnel Costs	Grade/Step	Annual Rate	% of time over 12 Months	Total Cost
Paperwork Reduction Act Liaison	13/8	$112,000	0.5 %	$560.00
Program Officer 1	Title 42	$200,000	0.5 %	$1,000.00
Program Officer 2	15/3	$134,662	5.0 %	$6,733.00
Sub-Total Federal Personnel				$8,293.00
Total Contractor Costs (Ripple Effect Communications Inc.)				$116,149.00
TOTAL				$124,442.00

A.15 Explanation for Program Changes or Adjustments

This is a new information collection.

A.16 Plans for Tabulation and Publication and Project Time Schedule

Data Analysis

Data quality control and quality assurance procedures will be developed and implemented by senior evaluation professionals and applied to all collected data. This will include procedures to ensure accuracy and consistency in data entry, data manipulation, and calculation. Internal validity will be checked as necessary for analysis. Descriptive and summary statistics will be calculated. If warranted, data from multiple sources may be cross-tabulated to address the study questions. Qualitative data from the survey will be coded and analyzed using standard qualitative methods.

Plans for Publication

Publication is not a main goal of this survey, but the team may investigate publishing the findings within the context of the larger evaluation, in a peer reviewed journal.

Project Time Schedule

It is estimated that it will take an experienced evaluation team 12 months to complete the evaluation, with the information collection component taking approximately 3 months. Provided below is an anticipated timeline of major tasks.

Table A.16-1 Project Time Schedule

Major Tasks	Months after OMB approval
	Month 1-3	Month 4-6	Months 7-9	Months 10-12
Identify Respondent Groups
Contact Respondents
Administer surveys
Summarize results and deliver final report

A.17 Reason(s) Display of OMB Expiration Date is Inappropriate

No exemption from the display of the OMB Expiration Date is being requested.

A.18 Exceptions to Certification for Paperwork Reduction Act Submissions

There are no exceptions to certification being requested.

5