NHGRI Short Course Evaluation Plan

NHGRI Short Course Evaluation Plan_OMB Supplemental_11.27.15 v2.pdf

Surveys to Support an Evaluation of the National Human Genome Research Institute (NHGRI) Summer Workshop in Genomics (Short Course)

NHGRI Short Course Evaluation Plan

OMB: 0925-0732

Document [pdf]

Download: pdf | pdf

EVALUATION PLAN
NHGRI Summer Workshop in Genomics

Ripple Effect Communications, Inc.

Contents
Background ................................................................................................................................................... 1
Feasibility Study ........................................................................................................................................ 1
Evaluation Logic Model ................................................................................................................................. 1
Inputs ........................................................................................................................................................ 2
Outputs ..................................................................................................................................................... 3
Outcomes .................................................................................................................................................. 3
Data Collection Process................................................................................................................................. 3
Focus Group .............................................................................................................................................. 4
Surveys ...................................................................................................................................................... 4
Sampling.................................................................................................................................................... 4
Plan for Data Security, Confidentiality, and Quality ................................................................................. 5
Data Analysis Plan ......................................................................................................................................... 5
Conceptual Framework and Analysis Questions....................................................................................... 5
Challenges and Limitations ..................................................................................................................... 14
Data Analysis Strategies .......................................................................................................................... 14
Appendix A: Short Course Feasibility Report ................................................................................................ 1
Appendix B: Focus Group Questions ............................................................................................................ 1
Appendix C: Survey Questions ...................................................................................................................... 1

NHGRI Short Course Evaluation Plan
Background
The National Human Genome Research Institute (NHGRI), part of the National Institutes of Health (NIH),
supports the development of resources and technology that will accelerate genome research and its
application to human health. The NHGRI Education and Community Involvement Branch (ECIB) initiates,
develops, implements, and evaluates education and community involvement programs designed to
engage a broad range of the public in understanding genomics, its translation to health, and its
importance to society.

Feasibility Study
In February 2014, Ripple Effect completed a Feasibility Study for NHGRI (see Appendix A) to design
evaluation strategies to measure several NHGRI training programs and assess the value of those
programs to the NHGRI and NIH Mission. As a result, Ripple Effect produced a feasibility report, which
included several evaluation options and recommended approaches.
NHGRI staff reviewed the Feasibility Study and discussed the various options. As a result NHGRI decided
to move forward with a structured evaluation of their Short Course program, which has involved the
training of academic and nursing faculty, as well as students since the program began as described
below:


Summer Workshop in Genomics (Faculty) – 2004-2012



Genome Scholars Program (Students-Mentees of Faculty) – 2008-2012



Advances in Genomics Research Summer Program (Nurse Educators) – 2012



Advances in Genomics Research Summer Program (Students) – 2012

This document summarizes the evaluation plan, including the logic model, data collection plan, and
planned analyses.

Evaluation Logic Model
Ripple Effect synthesized and organized the background information to describe the program and to
articulate the overall goal of the program evaluation (Figure 1). The logic model elements are described
below.

Figure 1. Evaluation Logic Model

Inputs
The Short Course is an intensive multi-day educational workshop designed to update biology instructors,
as well as other instructors and researchers in related disciplines, on genomic science. The course
focuses on the continuing effort to find the genetic basis of various diseases and disorders, and current
topics on the ethical, legal, and social implications of genomics. The course targets college and university
faculty seeking to update their curricula or develop new courses related to genetics and genomics.
Preference is given to applicants from racial and ethnic groups that are underrepresented in the healthrelated sciences; from institutions that predominantly train students with disabilities; and from
institutions that serve students from disadvantaged backgrounds including certain rural and inner-city
environments. Students from those same circumstances have been included in the program since 2004,
but student selection has evolved. Initially, faculty attendees selected students for attendance; then, in
2012, NHGRI began actively recruiting, screening, and accepting graduate students independent of
faculty attendees, as a way of preparing and attracting future scientists and health-care professionals to
a genomics workforce.
Applicants must complete an online application describing their research and teaching goals. In addition,
applicants are asked to submit a current biography, a curriculum vitae, and a supporting letter from
their department head (or equivalent). Women and minorities are encouraged to apply. Participants are
then selected by an NHGRI committee based on their likelihood to update their courses and to include
the broadest range of institutions supporting underrepresented students possible that meet program
goals. NHGRI pays for room and board; participant’s institutions are responsible for paying participant
travel costs for faculty, both to and from the NIH in Bethesda, Maryland.
The Short Course is a highly visible program within NHGRI and is considered by NHGRI leadership as an
important investment by the Institute. The annual budget of the Short Course is approximately $80,000
and represents a total investment of almost one million dollars since its inception. Each year, more than
15 faculty are recruited from NHGRI and other ICs to serve as instructors and presenters for the Short
2

Course, and 1-2 NHGRI staff are responsible for coordinating the logistics and planning throughout the
year.

Outputs
Short Course activities include lectures, lab tours, one-on-one meetings with NHGRI staff, and other
networking opportunities. Both faculty and student Short Course participants follow a similar schedule,
but at times the faculty and students are split into separate groups and receive targeted content. For
faculty participants, the program provides content for the specific purpose of curriculum integration;
when possible, materials are provided in electronic format to facilitate ease of integration. For student
participants, the program functions to attract students to careers in genomic science, medicine, allied
health, or any profession related to the application or use of genetic and genomic knowledge. Through
activities such as lab tours, one-on-one meetings, and planned and spontaneous networking
opportunities (e.g., post-lecture small group meetings, shared meals, tours of local attractions), the
program creates and fosters a community of genomics professionals committed to contributing
knowledge and expertise toward an era of genomic medicine.

Outcomes
There is an absence of information about whether and how the new knowledge and skills gained by
participants has been disseminated over the years, and whether or not the course is meeting program
goals. The evaluation will focus on two main outcomes: the degree of genetic and genomic curriculum
integration at institutes represented by the faculty participants, and the current career paths of former
student participants, including their perceptions about the influence of the summer workshops on their
career choices. The overarching outcomes are categorized in to short-, medium-, and long-term goals at
the faculty and/or institutional and student levels; these outcomes will help inform the evaluation
research questions.
The short terms outcomes focus on increasing knowledge of research, implications, and educational
strategies for genomics faculty. At the student level, the primary, anticipated short-term outcome is an
overall increase in general knowledge of genomics. The medium term outcomes emphasize
implementation and behavior change associated with the increased knowledge of the Short Course. At
the faculty level, outcomes include integrating new material in to the classroom, as well as continued
participation in professional activities and learning opportunities. These outcomes at the faculty level
also imply that a broader change across the participating institutions will see updated curriculums and
increased genomics knowledge by its students as a result of faculty participation. Student-level
outcomes are similar and include a future pursuit of learning opportunities related to genomics and
continued participation in NIH-related activities. The specified long-term outcomes reflect the program’s
goals of preparing the next generation of genomics professions and training and diversifying the pipeline
of genome professionals.

Data Collection Process
Between 2004 and 2012, there were a total of 314 participants between 2004 and 2012, with 187
faculty and 127 students. Participants include faculty, students, and nursing participants. This evaluation
utilizes a mixed-methods design to assess the process and outcome evaluation questions of interest.
Drawing from web-based surveys and focus groups, the study team will retrospectively examine key
areas of interest.

Focus Group
The purpose of the focus group is to seek detailed information from a sample of past Short Course
participants that could be used to inform further data collection and to respond to concerns identified
by the focus group. Responses from this data collection effort will help to inform a wide range of
questions and indicators, including post-program integration and engagement with NIH and/or NHGRI,
the educational and career paths of Short Course participants, and the influence of the program on the
careers of faculty attendees.
The evaluation study team will develop focus group questions based on the program logic model and
feedback from program stakeholders. Questions will be designed to provide supplemental information
to the evaluation’s web-based survey (planned for administration in late 2015). See Appendix B for a
preliminary draft of the focus group questions. The focus group will be 90-minutes in length and cover
four general areas of questions:


Perceptions of the program



Institutional involvement



Diversity in the genome professional pipeline



Recommendations for improvement

The evaluation’s project director will facilitate the focus group and another team member will take
detailed notes. Evaluation team members will then analyze the notes to summarize participants’
experiences. Findings will be presented in relation to program goals, articulated by the Short Course
evaluation logic model as short-, medium- and long-term outcomes.

Surveys
The evaluation team will administer a web-based survey to all past program participants (n=314)1 to
provide detailed information and insight about medium and long-term outcomes of the Short Course.
The survey instrument focuses on collecting information on integration of Short Course knowledge into
teaching materials; dissemination of genomics beyond the classroom; pursuit of careers related to
genomics; and continued participation in NIH-related activities.
The web-based survey underwent several iterations of development to ensure questions are clear,
concise, and aligned with key evaluation questions of interest. In addition, pilot tests were conducted to
assess conceptual clarity, time burden, make needed changes to the web-based interface and skip logic,
and to ensure survey data would be useful for planned analyses.

Sampling
Intended respondents of the web-based surveys are faculty and students of the Short Course between
2004 and 2012. The sampling method for the survey is a population-based sample of all program
participants. There will be no cohort-specific (or year-specific) analyses. There are a maximum of 314
participants between 2004 and 2012, with 187 faculty and 127 students.

This number is likely to be significantly smaller due to unavailable current contact information for all participants.

Table 1. Short Course Participants by Year

Year
2004
2005
2006
2007
2008
2009
2010
2011
2012
Total

Number of Faculty
16
18
18
20
14
18
18
36
29
187

Number of Students
16
18
18
13
16
17
17
0
12
127

Total
32
36
36
33
30
35
35
36
41
314

Contact information (e.g., email address, phone number) is only available for 299 program participants
based on a list provided to the evaluation team by NHGRI. The evaluation searched for updated contact
information in the summer of 2015 and could not find current contact information for 77 potential
respondents. While some of the previous contact information may be accurate, the likely pool of survey
respondents is closer to 225. The anticipated response rate is approximately 50% given some
respondents completed the program a decade ago.
To select participants for the focus group, the study team will narrow the list of potential participants to
those who participated between 2008 and 2012 (a five-year period). The team will utilize a Doodle poll
to assess availability for a 90-minute focus group on a range of dates and times. No more than 9
participants will be issued a final invite for the virtual focus group.

Plan for Data Security, Confidentiality, and Quality
Ripple Effect will program the survey instrument into an electronic format utilizing the appropriate
software. For this evaluation, Ripple Effect will utilize a web based tool called Qualtrics that has met all
the HHS requirements for security, privacy, accessibility and has an established terms of service
agreement with HHS. The web-based version of the survey may include different versions or skip
patterns for different audiences.
Ripple Effect will work with the appropriate forms and offices at NIH to obtain OMB and IRB clearances.
IRB clearance, or a granted waiver, is required when human participants are involved in the study. OMB
clearance is required when the number of participants is expected to be more than nine.
Ripple Effect will work with NHGRI to select an appropriate sample to pilot test the survey to ensure
questions are clear and elicit the type of responses most useful for program evaluation. The survey will
be adjusted as necessary to address feedback.

Data Analysis Plan
Conceptual Framework and Analysis Questions
Using the conceptual framework and logic model, Ripple Effect identified the primary evaluation
questions. The questions focus on key medium- and long-term outcomes of the project, as seen in Table
2.
5

Table 2. Primary Evaluation Questions

Outcome Area
Engagement with NIH/NHGRI

New Knowledge Applied

Continued Genomic Discovery

Institutional Barriers

Evaluation Question
Do faculty or students who participated in the program
continue to engage in NIH- and/or NHGRI-specific activities or
programs?
Do faculty apply knowledge by integrating genomic content
into individual teaching materials, creating new educational
resources, updating curricula at the institutional level, and/or
developing new courses related to genomic science?
Does the program spur curiosity in genomic science and lead
participants toward further discovery, learning, research,
and/or a career in genomics research or a related discipline?
Are there institutional limitations or barriers that prevent
program attendees from taking the next step (e.g., curriculum
integration, access to further information, pursuit of career)?

Table 3 presents each survey question and its linkage to elements of the logic model. This table is part of
our SharePoint database and can be filtered and sorted as needed. This full list provides a mapping for
our planned descriptive analyses.
Table 3. Linkages Between Survey Questions and Logic Model

Section

Question

Historical

Select your role during
your participation in
the Short Course.
Did you hold a teaching
appointment at the
time of your
attendance to the Short
Course?
Out of 100 percent,
what best represented
your teaching/research
ratio at the time?

Historical

What degree program
were you enrolled in
when you attended the
Short Course?

Historical

Response options
Faculty attendee (1)
Student attendee (2)
Nursing faculty attendee (3)

Yes (1)
No (2)
Teaching (1)
Research (2)
Other, please specify: (3)
Bachelor of Science (BS) (1)
Bachelor of Arts (BA) (2)
Master of Science (MS) (3)
Master of Arts (MA) (4)
Master of Public Health (MPH) (5)
Doctor of Philosophy (PhD) (6)
Doctor of Medicine (MD) (7)
Dual degree (MD & PhD) (8)
Registered Nurse (RN) (9)
Other, please specify: (10)

Respondent
Type

Logic Model
Link

Logic
Model
Area

Both

Diverse
representation

Participants

Faculty

Teaching
genetics related
coursework

Participants

Faculty

Teaching
genetics related
coursework

Participants

Student

Students/
Mentees of
faculty

Participants

Section

Historical

Historical
Current

Current

Question

What was your primary
Position or Occupation
Title at the time of your
attendance to the Short
Course?
Was this a tenure or
non-tenure track
position?
Are you still with [PIPED
INSTITUTION NAMED]?
Please list your current
institution or
organizational
affiliation.
Has your Position or
Occupation Title
changed from [PIPE IN
SELECTED CHOICE]
since the time of your
attendance in the Short
Course?

Response options
Researcher (1)
Adjunct Instructor/Professor (2)
Assistant Professor (3)
Associate Professor (4)
Professor (5)
Distinguished and/or Endowed
Professor and/or Emeritus (6)
Other, please specify: (7)
Tenure track (1)
Non-tenure track (2)
Yes (1)
No (2)

Open text

Current

What is your current
primary Position or
Occupation Title?

Yes (1)
No (2)
Researcher (1)
Adjunct Instructor/Professor (2)
Assistant Professor (3)
Associate Professor (4)
Professor (5)
Distinguished and/or Endowed
Professor and/or Emeritus (6)
Administrator (7)
Other, please specify: (8)

Current

Please specify whether
this is a tenure or nontenure track position.

Tenure track (1)
Non-tenure track (2)

Current

What is your current
primary Position or
Occupation Title?

What type of degree
program are you
enrolled in?

Student (1)
Trainee (2)
Clinician (3)
Educator (K-12) (4)
Researcher (5)
Administrator (6)
Instructor or Professor (7)
Other, please specify: (8)
Master of Science (MS) (1)
Master of Arts (MA) (2)
Master of Public Health (MPH) (3)
Doctor of Philosophy (PhD) (4)
Doctor of Medicine (MD) (5)
Dual degree (MD & PhD) (6)
Registered Nurse (RN) (7)
Other, please specify: (8)

Respondent
Type

Logic Model
Link

Logic
Model
Area

Both

Teaching
genetics related
coursework
Teaching
genetics related
coursework
Type of
Institution

Both

Type of
Institution

Participants

Faculty

Train and
diversify the
pipeline of
genome
professionals

6. LongTerm
Outcomes

Faculty

Participants

Participants
Participants

Faculty

Train and
diversify the
pipeline of
genome
professionals
Train and
diversify the
pipeline of
genome
professionals

Student

Pursue a career
related to
genomics

5. MediumTerm
Outcomes

Student

Pursuit of
coursework/
learning
opportunities
related to
genomics

5. MediumTerm
Outcomes

Faculty

6. LongTerm
Outcomes

Section

Current

Question

Please specify trainee
type.

Please specify the type
of instructor or
professor position.
Please specify whether
this is a tenure or nontenure track position.
Does your current
position involve
teaching genetic or
genomic material?

Current

Select the content
areas you teach: (Select
all that apply)
Out of 100 percent,
what best represents
your current
teaching/research
ratio?

Current

Which of the following
best describes your
primary field of work?

Current

Response options
Predoctoral (1)
Postdoctoral (2)
Clinical (3)
Other, please specify: (4)
Researcher (1)
Adjunct Instructor/Professor (2)
Assistant Professor (3)
Associate Professor (4)
Professor (5)
Distinguished and/or Endowed
Professor and/or Emeritus (6)
Administrator (7)
Other, please specify: (8)
Tenure track (1)
Non-tenure track (2)

Yes (1)
No (2)
Biology of genetics and genomics (1)
Biology of human genetics (2)
Molecular/cellular genetics (3)
Genetics and genomics of common
diseases (4)
Genetic and genomic technology (5)
Pharmacogenetics and genomics (6)
Ethical, legal and social implications of
genetics/genomics (7)
Resources for genetic/genomic
education (8)
Career resources (e.g., grant writing)
(9)
Other, please specify: (10)

Teaching (1)
Research (2)
Other, please specify: (3)
Academia (1)
Government (2)
Clinical (3)
Clinical/Government (4)
Industry (5)
Non-profit (6)
Other, please specify: (7)

Logic Model
Link

Logic
Model
Area

Pursue a career
related to
genomics

5. MediumTerm
Outcomes

Student

Pursue a career
related to
genomics
Pursue a career
related to
genomics

5. MediumTerm
Outcomes
5. MediumTerm
Outcomes

Both

Pursue a career
related to
genomics

5. MediumTerm
Outcomes

Student

Pursue a career
related to
genomics

5. MediumTerm
Outcomes

Student

Pursue a career
related to
genomics

5. MediumTerm
Outcomes

Both

Pursue a career
related to
genomics

5. MediumTerm
Outcomes

Respondent
Type

Student

Section

Question

Response options

Current

Please identify any
credentials or degrees
that you have received
or pursued since
attending the Short
Course. (Select all that
apply)

None received or pursued (1)
Bachelor of Science (BS) (2)
Bachelor of Arts (BA) (3)
Master of Science (MS) (4)
Master of Arts (MA) (5)
Master of Public Health (MPH) (6)
Doctor of Philosophy (PhD) (7)
Doctor of Medicine (MD) (8)
Dual degree (MD & PhD) (9)
Registered Nurse (RN) (10)
Other, please specify: (11)

Current

Have you completed
the credential or
degree(s)?

Yes (1)
No (2)
Other, please specify: (3)

Current

What discipline(s) is
your credential or
degree associated
with?

Short Course
Program

Out of 100 percent,
what best represents
your current
teaching/research
ratio?
To what degree did
your experience at the
Short Course
influence your teaching
to research ratio?
How many year(s) have
you been engaged in
teaching? (Numbers
only)
Please select one of the
three options below in
response to the
following statement: I
was able to update my
curriculum as a result of
my participation in the
Short Course.

Respondent
Type

Logic Model
Link

Logic
Model
Area

Faculty

Pursuit of
coursework/
learning
opportunities
related to
genomics
Pursuit of
coursework/
learning
opportunities
related to
genomics
Pursuit of
coursework/
learning
opportunities
related to
genomics
Pursuit of
coursework/
learning
opportunities
related to
genomics

No influence (1)
Some influence (2)
Moderate influence (3)
Great influence (4)

Faculty

Pursue a career
related to
genomics

5. MediumTerm
Outcomes

Open text (numbers only)

Faculty

Diverse
representation

Participants

Faculty

Updated
curriculum at
faculty
institutions

5. MediumTerm
Outcomes

Open text

Both

Teaching (1)
Research (2)
Other, please specify: (3)

Yes, I made substantial changes (1)
Yes, I made some changes (2)
Yes, I made minor changes (3)
No, I did not make any changes (4)

5. MediumTerm
Outcomes

Section

Short Course
Program

Question

I was able to update my
genetics or genomics
curriculum and
teaching materials in
the following ways:
(Select all that apply)

I was able to update my
curriculum and
teaching materials in
the following content
areas: (Select all that
apply)
How much time was
required to update your
curriculum and
teaching materials after
participating in the
Short Course?
Beyond curriculum
integration, in what
other ways did you
disseminate
information from the
Short Course to
students? If you did not
disseminate
information beyond
curriculum integration,
please write “None.”

Response options
Informal/unplanned integration (e.g.,
anecdotal narratives, spontaneous
examples) (1)
Added content to my lecture materials
(2)
Added assignments to my courses (3)
Removed other content to make way
for new information (4)
Developed new teaching objectives for
my existing courses (5)
Created new lab experiments (6)
Developed a new course (7)
Shared teaching material with other
faculty (8)
Other, please specify: (9)
Biology of genetics and genomics (1)
Biology of human genetics (2)
Molecular/cellular genetics (3)
Genetics and genomics of common
diseases (4)
Genetic and genomic technology (5)
Pharmacogenetics and genomics (6)
Ethical, legal and social implications of
genetics/genomics (7)
Resources for genetic/genomic
education (8)
Career resources (e.g., grant writing)
(9)
Other, please specify: (10)
Less than 3 months (1)
Between 3 and 6 months (2)
Between 6 months and 1 year (3)
Between 1 and 2 years (4)
Between 2 and 5 years (5)
More than 5 years (6)

Open text

Respondent
Type

Logic Model
Link

Logic
Model
Area

Faculty

New knowledge
integrated into
existing
teaching
materials

5. MediumTerm
Outcomes

Faculty

New knowledge
integrated into
existing
teaching
materials

5. MediumTerm
Outcomes

Faculty

Updated
curriculum at
faculty
institutions

5. MediumTerm
Outcomes

Faculty

Dissemination
of genomics
beyond the
classroom
(professional
related
activities)

5. MediumTerm
Outcomes

Section

Question

Response options

Short Course
Program

Please rate how much
you think the following
institutional factors
influenced your ability
to transfer your
knowledge to students
following participation
in the Short Course.
Select "Not Applicable"
if the factor was not
relevant to your
situation.
Please rate how much
you think the following
student-related factors
influenced your efforts
to transfer your
knowledge to students.
Select "Not Applicable"
if the factor was not
relevant to your
situation.

Time and space to accommodate or
integrate new information into
existing curriculum (1)
Relevant course(s) not available (2)
Process for changing curriculum (3)
Support of colleagues (4)
Support by supervisor or leadership
(5)
Institutional awareness about the
importance of genetic and genomic
content (6)
Curriculum committee support (7)
Incentive (financial or
recognition/praise) (8)
Institutional funding or resources (9)
Ability to secure external funding or
resources (10)
Protected time to develop content
(11)
Student preparedness (e.g., students’
prerequisite knowledge) (1)
Student perception around
applicability of information to their
career (2)
Student awareness about the
importance of genetic and genomic
content (3)
Students’ interest (4)
Students’ self-confidence (5)

Short Course
Program

Did the Short Course
influence your
research?

Yes (1)
No (2)

Short Course
Program

Please describe how
your experience at the
Short Course influenced
your research.

Short Course
Program

Respondent
Type

Faculty

Both

Open text

Both

Logic Model
Link

Institutional
Factors (e.g.
Financial
Resources,
Culture Change,
Diversity)

Increased
genomics
knowledge by
students at
faculty’s
institution
Dissemination
of genomics
beyond the
classroom
(professional
related
activities)
Dissemination
of genomics
beyond the
classroom
(professional
related
activities)

Logic
Model
Area

Environment

5. MediumTerm
Outcomes

Section

Question

Short Course
Program

To what degree did
your experience at the
Short Course influence
your decision to engage
in the following
activities? Select "Not
Applicable" if you did
not engage in the
activity listed.
To what degree did
your experience at the
Short Course influence
your decision to pursue
new career or
educational options?
Please describe how
the Short Course
influenced your
decision to pursue new
career or educational
options.
Since the Short Course,
have you been involved
with Short Course
attendees, NIH or
NHGRI?

Short Course
Program

Please indicate the
ways you have been
involved with Short
Course attendees, NIH
or NHGRI since
attending the program.
(Select all that apply)

Short Course
Program

Response options
Presentations at scientific meeting (1)
Attendance at scientific meeting (2)
Involvement in professional
organizations (3)
Took additional classes on
genetics/genomics (4)
Attended seminars/events on
genetics/genomics (5)
Informal genetics/genomics chats (6)
Engaged a mentor in the
genetics/genomics field (7)
Became involved in a research group
related to genetics/genomics (8)
Conducted research in
genetics/genomics (9)
Attended professional meetings
related to genetics/genomics (10)
Presented material at various events
related to genetics/genomics (11)
Published findings related to
genetics/genomics (12)
No influence (1)
Some influence (2)
Moderate influence (3)
Great influence (4)
Not applicable – I did not pursue new
career or educational options (5)

Open text

Yes (1)
No (2)
Communications with fellow
attendees (1)
Additional contact with
presenter(s) (2)
Additional contact with NIH or NHGRI
staff (3)
Pursued other training or educational
opportunities at NIH (4)
Pursued other training opportunities
at NHGRI (5)
Used NHGRI online resources (6)
Involved in writing NIH grantapplication (7)
Awarded NIH grant (8)
Joined NIH Listserv(s) or other forms

Respondent
Type

Logic Model
Link

Logic
Model
Area

Both

Dissemination
of genomics
beyond the
classroom
(professional
related
activities)

5. MediumTerm
Outcomes

Both

Pursue a career
related to
genomics
(broadly
defined)

5. MediumTerm
Outcomes

Both

Pursue a career
related to
genomics
(broadly
defined)

5. MediumTerm
Outcomes

Both

Continued
participation in
NIH-related
activities

5. MediumTerm
Outcomes

Both

Continued
participation in
NIH-related
activities

5. MediumTerm
Outcomes

Section

Question

Response options

Respondent
Type

Logic Model
Link

Logic
Model
Area

of communication (9)
Other, please specify: (10)

Background
Information
Background
Information
Background
Information

Background
Information

What year were you
born?
What is your sex?
What is your ethnicity?

What race do you
consider yourself?
(Select all that apply)
Do/did you have a
disadvantaged
background, which can
be defined either as
coming from a family
with an annual income
below established lowincome thresholds, or
coming from an
educational
environment such as
that found in certain
rural or inner-city
environments that have
demonstrably and
directly inhibited you
from obtaining the
knowledge, skills, and
abilities necessary to
develop and participate
in a research career?

Drop down menu

Both

Male (1)Female (2)Do not wish to
provide (3)
Hispanic or Latino (1)Not Hispanic or
Latino (2)Do not wish to answer (3)

Diverse
representation

Participants

Both

Diverse
representation
Diverse
representation

White or Caucasian (1)
Black or African American (2)
Asian (3)
American Indian/Alaska Native (4)
Native Hawaiian or other Pacific
Islander (5)
Do not wish to answer (6)

Both

Diverse
representation

Participants

Yes (1)
No (2)
Do not wish to answer (3)

Both

Diverse
representation

Participants

Both

Participants
Participants

Section

Question

Short Course
Program

Please enter any other
comments you would
like to share with us
about the Short Course
and your experiences
since attending the
program. If you have no
additional comments,
please hit the next
button.

Response options

Open text

Respondent
Type

Both

Logic Model
Link

Logic
Model
Area

Varies based on
response

Varies based
on response

Challenges and Limitations
This evaluation presented certain challenges that affected both its design (and will affect the
interpretation of findings). First, because the Short Course is a compressed multi-day workshop, it is
difficult to attribute any outcomes to the course itself beyond short term knowledge gain. Second, the
evaluation is collecting retrospective data, thus introducing the potential for recall bias or simply
forgetting details that may be integral to some of the survey and/or focus group questions. Finally, the
Short Course is very selective with its participants, thus limiting the size of the sample pool. Coupled
with the retroactive nature of the evaluation (up to 10 years for the survey and 7 years for the focus
group), it may be difficult to reach past program participants, potentially impacting the survey response
rate.

Data Analysis Strategies
Data quality control and quality assurance procedures will be developed and implemented by senior
evaluation professionals and applied to all collected data. For quantitative data, internal validity will be
checked as necessary for analysis (e.g., consistency of responses within a case). Descriptive and
summary statistics will be calculated from survey responses to inform the evaluation questions. If
warranted and appropriate, data may be cross-tabulated to determine if medium- and long-term
outcomes differ between the student and faculty participants.
Analysts will review and analyze the qualitative data by question (“Beyond curriculum integration, in
what other ways did you disseminate information from the Short Course to students?”; “Please describe
how your experience at the Short Course influenced your research.”; and “Please describe how the
Short Course influenced your decision to pursue new career or educational options.”). The evaluation
team will develop and apply a coding scheme to identify themes within the questions. These findings
will be incorporated to provide additional contextual information for descriptive quantitative findings.
After the survey and focus group data are analyzed, Ripple Effect will draw conclusions and
recommendations about the outcomes of the NHGRI Short Course. The results will be delivered in the
form of a narrative report.

Appendix A: Short Course Feasibility Report

Summary of Recommendations
Ripple Effect Communications, Inc. was selected by the National Human Genome Research Institute
(NHGRI) to conduct a feasibility study of the NHGRI Summer Workshop in Genomics Program and to
design evaluation strategies that will assess its value to NIH and NHGRI. This report outlines our findings
and includes the following recommendations:









The techniques for evaluation described in this document can be used for all NHGRI sponsored
“short courses” described in this document; however, the specific questions would need to be
tailored based on the content and target audience of the short course.
The evaluation is designed to retrospectively gather data from participants who attended the
Short Course from 2003 to 2012 (approximately 300 faculty and 150 student participants). This
includes nursing, faculty and student participants (Genome Scholars and the Advances in
Genomics Summer Research Program participants).
If the NHGRI goal is to evaluate the success of the Short Course program to-date, then we
recommend NHGRI implement the first technique: Electronic Survey of Short Course
Participants. This approach will provide the most detailed information about the degree of
curriculum integration at the institutions of participants, and about the supports or barriers they
experienced at the institutional level.
NHGRI may not know if the Focus Group with Selected Participants is useful or necessary until
feedback from the survey is received.
If funds are limited, then NHGRI may want to consider investing in the final two techniques
(Document Review of Related Archival Data, Request for Information on Key Issues) which
provide useful information, but may not fulfill the desire to evaluate program success.
NHGRI should consider building in a long-term expectation for program participants to provide
follow-up information for future short courses.

A-1

Background
Project Scope
The National Human Genome Research Institute (NHGRI), which is part of the National Institutes of
Health (NIH), supports the development of resources and technology that will accelerate genome
research and its application to human health. The Education and Community Involvement Branch (ECIB)
of the NHGRI initiates, develops, implements, and evaluates education and community involvement
programs intended to engage a broad range of the public in understanding genomics and its translation
to health and its importance to society.
NHGRI engaged Ripple Effect to conduct a feasibility study to design evaluation strategies that will
measure several programs and assess the value of those programs to the NHGRI and NIH Mission.
Specifically, Ripple Effect was engaged to answer the following specific questions:






Does NHGRI have sufficient resources (e.g., human capital, funding, archival data) to pursue
process or outcome evaluations?
Is the estimated cost of a process or outcome evaluation reasonable given the cost of the
program?
What clearance requirements might be necessary to conduct the evaluation?
What data collection efforts are needed to evaluate the program?
What data are currently available that can be used to evaluate the program and determine
baseline data?

Feasibility Activities
Ripple Effect completed the following activities:
 Review and gather background data;
 Assess program goals, inputs and outputs;
 Define evaluation questions and identify data sources; and
 Recommend feasible approaches to evaluation.
Review and gather background data. Background data was provided through interviews with NIH
stakeholders (Appendix A and B), and background materials (Appendix C) were provided by NIH staff.
This process was iterative and collaborative: interviews shed light on relevant background materials, and
the background materials identified areas for clarification from additional interviews. In addition, Ripple
Effect reviewed other related studies that may serve as models for this evaluation.
Assess program goals, inputs and outputs. Ripple Effect synthesized and organized the background
information to describe the program and to articulate the overall goal of the program evaluation
(Appendix H). The program description includes: program history; program goals; logic model including
inputs, outputs and outcomes; program activities and the programs alignment with NHGRI and NIH
mission. The high level description of the evaluation included defining the overall purpose of the
evaluation, how the evaluation results would be used, and the evidence gaps that currently exist.
Define evaluation questions and identify data sources. Ripple Effect focused on proposing outcome
evaluation strategies to assess the value of each program, and process evaluation strategies to improve

A-2

the efficiency and effectiveness of program implementation. For each program, Ripple Effect defined
the following (Appendix E):



Program Goals: Specific goals that align with program purpose and NIH/NHGRI mission.
Evaluation Objectives: The objective describes the purpose of the evaluation and aligns with the
program goals.
 Evaluation Questions: Key questions that will be answered by the evaluation that help
determine evidence of the program’s contribution to long-term outcomes. Questions are often
dependent upon the uses of the evaluation.
 Evaluation Indicators: Criteria to be met in order for a program to achieve its intended results.
 Data Collection Methods and Sources: Identify appropriate data sources available for each
indicator including feasibility, clearances and sampling for each method.
Develop and recommend feasible approaches to evaluation. In collaboration with NHGRI, we
developed several data collection approaches, which were evaluated for feasibility, usefulness and
alignment with program priorities. This prioritization process resulted in a short list of possible
techniques for each program. For each technique, Ripple Effect outlined in more detail the design of the
evaluation and level of effort. In addition, this report outlines the advantages and limitations of each
technique. Finally, this report includes recommendations to NHGRI, based on our knowledge of NIH and
our experience on advantageous evaluation approaches.
NHGRI will be able to use the information gleaned from this feasibility study to develop and implement
the appropriate evaluation technique to meet NHGRI needs.

A-3

Program Overview
Program Summary
Program Description
The NHGRI Summer Workshop in Genomics (Short Course) is offered by the National Human Genome
Research Institute (NHGRI), at the National Institutes of Health (NIH). This intensive, four-day course is
designed to update biology instructors, as well as other instructors and researchers in related
disciplines, on genomic science. The course focuses on the continuing effort to find the genetic basis of
various diseases and disorders, and current topics on the ethical, legal and social implications of
genomics. This course is especially intended for college and university faculty seeking to update their
curriculum or to develop new courses related to genetics.
This course is designed to update instructors who train students from:
 Racial and ethnic groups underrepresented in health related sciences;
 Institutions that predominantly train students with disabilities; or
 Disadvantaged backgrounds including certain rural and inner-city environments.
Applicants must complete an online application describing their research and teaching goals. In addition,
applicants are asked to submit a current biography, curriculum vitae, and supporting letter from their
department head (or equivalent). Women and minorities are encouraged to apply.
Participants are then selected by an NHGRI committee based on their likelihood to update their courses
and to include the broadest range of institutions supporting underrepresented students possible that
meet program goals.
NHGRI pays for room and board; participant’s institutions are responsible for paying participant travel
costs for faculty, both to and from the NIH in Bethesda, Maryland.
Genome Scholars Program. In 2008, NHGRI expanded the program and invited successful faculty
applicants to select one promising student from their institution to attend the Genome Scholars
Program. The Genome Scholars Program parallels the Short Course, and offers a close-up view of
careers in genetic research while providing an enhanced mentoring experience. Genome Scholar
applicants must have a minimum 3.0 GPA, be currently enrolled at the sponsor's school in a sciencerelated major, and successfully complete a formal application before being enrolled as a Genome
Scholar. Each Short Course faculty participant is limited to one Genome Scholar.
Advances in Genomics Research Summer Program. In 2012, NHGRI developed a separate track targeted
to nurse educators. This track was an intensive, five-day course for nursing faculty at colleges and
universities with substantial under-represented minority, rural and/or disadvantaged student
enrollment. Nursing faculty attended seminars presented by leading NHGRI research investigators and
educators. Seminars provided updates on the latest advances in genomics research, focus on the most
current understanding of the genetic/genomic basis of disease, examine the ethical, legal and social
implications of genomics research, and provide potential strategies for nursing education.
Advances in Genomics Research Summer Program. In 2012, NHGRI hosted another short course titled,
“Advances in Genomics Research Summer Program” that is an intensive, five-day course for students at
colleges and universities with substantial under-represented minority, rural and/or disadvantaged
A-4

student enrollment. The NHGRI Advances in Genomics Research Summer Program was targeted to Ph.D.
and M.D.-Ph.D. students enrolled in programs with areas of concentration in Molecular Biology,
Computational Biology and/or Genetics. This intensive five-day course included seminars presented by
leading NHGRI research investigators, focusing on the latest advances in basic genomics research, the
genetic basis of disease, and the ethical, legal and social implications of genomics research. Students
presented their research at a poster session and engaged with NIH investigators, postdoctoral trainees
and career counselors. This was a unique opportunity for students interested in pursuing careers in
genomics research to gain information, career advice and visibility in the research-rich context of the
NIH.

Approximate Size
The Short Course is a highly visible program within NHGRI and is considered by NHGRI leadership as an
important investment by the Institute. The Short Course annual budget is approximately $80,000 and
represents a total investment of almost one million dollars since its inception. Each year, more than 15
faculty are recruited from NHGRI and other ICs to serve as instructors and presenters for the Short
Course, and 1-2 NHGRI staff are responsible for coordinating the logistics and planning throughout the
year.

Organizational Location and Establishment
The NHGRI Education and Community Involvement Branch (ECIB) within the Division of Policy,
Communications and Education at NHGRI offers the Summer Research Workshop in Genomics (Short
Course) as a targeted education and outreach program. The course was initially sponsored by the NHGRI
Division of Intramural Research and later moved to ECIB for leadership. The Short Course has been
offered and supported by the Institute for more than ten years. Due to the lengthy process for gaining
travel approvals that recently was instituted NIH-wide, this program was not offered in 2013; however,
NHGRI plans to offer it in 2014 and future years.

Stakeholders
The NHGRI Summer Workshop in Genomics includes the following key stakeholders:





NIH Short Course Faculty: Each year, more than 15 faculty are recruited from NHGRI and other
NIH ICs to serve as instructors and presenters for the Short Course
Faculty Program Participants: College and university faculty seeking to update their curriculum
or to develop new courses related to genetics and train students from: racial and ethnic groups
underrepresented in health related sciences; institutions that predominantly train students with
disabilities; or disadvantaged backgrounds including certain rural and inner-city environments.
Student Program Participants: Mentees or students at the same universities of selected faculty
program participants interested in pursuing a genetics related career and selected by the
successful faculty applicants. These students participated in the genome scholars program.
Nursing Program Participants: College and university nursing faculty focusing on the specific
needs of nurse education who are seeking to update their curriculum or to develop new courses
related to genetics and train students from the underrepresented groups described of the
faculty program participants.

Goals

A-5

This intensive, six-day course is designed to update post-secondary faculty and researchers in related
disciplines on the latest research trends and topics in genomic science. The program aims to accomplish
the following goals:




Expand NIH and NHGRI’s professional network to reach out to diverse communities, and to
create new partnership opportunities.
Prepare the next generation of genomics professionals for an era of genomic medicine.
Train and diversify the pipeline of genome professionals in alignment with the NIH and US
Department of Health and Human Services diversity efforts.

Support of NHGRI and NIH missions
The Short Course was created as a way to efficiently and effectively accelerate the dissemination of
genetic and genomic information to science faculty, especially those faculty at minority serving
institutions. As such, the Short Course meets aspects of both the NHGRI mission by “supporting the
development of resources (faculty) that will accelerate genome research and its application to human
health” and the NIH mission by “developing, maintaining, and renewing human resources that will
ensure the Nation's capability to prevent disease.” This program fulfills the NHGRI Strategic Plan for
education and training, which outlines the importance of preparing the next generation of genomics
researchers in many disciplines and expanding the diversity of the genomics workforce.
Specifically, the program goals align with the following component of the NHGRI strategic plan:





Building healthcare providers’ genomic competencies.
o Prepare the next generation of genomics professionals for an era of genomic medicine.
Conducting public outreach.
o Expand NIH and NHGRI’s professional network to reach out to diverse communities, and
to create new partnership opportunities
Preparing the next generation of genomics researchers.
o Train and diversify the pipeline of genome professionals in alignment with the NIH and
US Department of Health and Human Services diversity and inclusion efforts.

A-6

LOGIC MODEL/CONCEPTUAL FRAMEWORK

Program Functions and Activities
Short Course activities include lectures, lab tours, one-on-one meetings with NHGRI staff, and other
networking opportunities. Initially, the program was only open to faculty participants; however, in 2008,
the program was expanded to include undergraduate and graduate students from the same institution
as the faculty participants. In 2012, the program was further expanded to include nursing faculty, and
address their unique nursing education needs as related to genomics. In addition, in 2012, another
related 5-day Short Course, the Advances in Genomics Research Summer Program, was created to
specifically target Ph.D. and M.D.-Ph.D. students enrolled in programs with concentrations in Molecular
Biology, Computational Biology and/or Genetics. Both faculty and student Short Course participants
follow a similar schedule, but at times the faculty and students are split into separate groups and receive
targeted content. For faculty participants, the program functions to provide content for the specific
purpose of curriculum integration; when possible, materials are provided in electronic format to
facilitate the ease of integration. For student participants, the program functions to attract students to
careers in genomic science, medicine, allied health, or any profession related to the application or use of
genetic and genomic knowledge. Through activities such as lab tours, one-on-one meetings, planned
and spontaneous networking opportunities (e.g., post-lecture small group meetings, shared meals, tours
of local attractions), the program creates and fosters a community of genomics professionals committed
to contributing knowledge and expertise toward an era of genomic medicine.

Evaluation Design
Evaluation Objectives
Purpose
Ideally, the evaluation will retrospectively gather data from participants who have attended the Short
Course from 2003 to 2012 (approximately 300 faculty and 150 student participants). This will include the
nursing participants, faculty and students, including the Advances in Genomics Summer Research
Program. The evaluation will focus on two main outcomes: the degree of genetic and genomic
curriculum integration at institutes represented by the faculty participants, and the current career paths
of former student participants, including their perceptions about the influence of the summer
workshops on their career choices.

Evidence Gaps
Each year, administrative staff have gathered feedback from participants directly following their
participation in the Short Course. The evaluation forms typically offered a three-point rating scale and
space for open-ended comments in response to questions, which focused on the quality of: course
content; presenters; workshop activities; and supports. Feedback was used to evolve the program in
successive years, but knowledge about the longer-term outcomes is unknown.

Use of Results
Findings. NHGRI will use the findings of this evaluation to make appropriate changes to the program
design, which could include modifications to recruitment strategies, the application process, course
programming, format, length and follow-up. For example, if it is found that the participation in the
program was more useful for certain groups of participants, the application process could be adjusted to
recruit the that group of participants. Alternatively, or in addition, the program agenda could be
adjusted to better serve those participants who did not find the Short Course as useful.
This program is more than 10 years old and sufficient time has passed to evaluate the long term impact
of the program. During his time as NHGRI Director, the current NIH Director, Dr. Francis Collins, had
significant interest in the course with regard to its potential to enhance genomics education within the
undergraduate curriculum and increase the diversity of trainees within NIH funded training programs; if
the program is found to be successful, it could be suggested as a model to many ICs. If the program is
not found to be successful, or only parts of the program are deemed successful, the current NHGRI
Director, Dr. Eric Green, would likely change the program so that it would better meet the desired
outcomes. Furthermore, since one of the program goals is to enhance diversity of the workforce, the
results of this evaluation would provide input into current trans-NIH discussions on this subject.
According to the ACD Working Group on Diversity of the Biomedical Workforce, “NIH needs to be more
attentive to collecting the data on an ongoing basis to better inform next steps and future actions that
are required to address” diversity. If this program is able to demonstrate success at increasing diversity,
it would be a significant finding against the backdrop that “despite longstanding efforts from the NIH to
increase the number of scientists from underrepresented groups, diversity in biomedicine still falls short
of mirroring that of the U.S. population.”2

Evaluation Questions
2

http://acd.od.nih.gov/dbr.htm

A-1

Below is an outline of the evaluation questions that are relevant to the techniques included in this
report. The full set of evaluation questions is available in Appendix F.
Goal: Expand NIH and NHGRI’s professional network to reach out to diverse communities, and to create
new partnership opportunities
Evaluation Type Evaluation Question
Indicator(s)
Outcome
Engagement with NIH/NHGRI
(03) Degree to which former presenters
Do faculty or students who participated have expanded their professional
in the program continue to engage in
network due to participation
NIH or NHGRI specific activities or
(04) Degree to which former program
programs?
participants continue(d) to interact with
NIH/NHGRI
(05) Post-program integration/
engagement with NIH/NHGRI
Goal: Prepare the next generation of genomics professionals for an era of genomic medicine.
Type
Evaluation Question
Indicator(s)
Outcome
New knowledge/skills applied
(08) Length of time after program that
Do faculty apply knowledge and skills
curriculum change was achieved
by integrating genomic content into
(09) Type and quantity of genomic
individual teaching materials, creating
curriculum updates
new educational resources, updating
curriculum at the institutional level, or
developing new courses related to
genomic science?
Goal: Train and diversify the pipeline of genome professionals in alignment with the NIH and US
Department of Health and Human Services diversity and inclusion efforts.
Type
Evaluation Question
Indicator(s)
Outcome
Continued genomic discovery
(06) Educational and career paths of
Does the program spur curiosity in
participants
genomic science and lead participants
(11) Faculty attendee perceptions of the
toward further discovery, learning,
program influence on their career
research or a career in genomics
(12) Student attendee perceptions of
research or a related discipline?
the program influence on their career
(13) Degree to which participants
pursued activities that demonstrate
interest in genomics
Outcome
Institutional barriers
(14) Perception of institutional
Are there institutional limitations or
limitations or barriers that interfere with
barriers that prevent program
achieving program outcomes
attendees from taking the next step
(e.g., curriculum integration, access to
further information, pursuit of career)?

Feasibility and Evaluation Techniques
Ripple Effect compiled and assessed a comprehensive matrix of all evaluation questions, the information
required to answer each question, all potential data sources and methods for collecting data, and the
scope of resources required (see Appendix D and E). The most feasible data sources and methods were
A-2

selected, and grouped into four overarching data collection techniques. Each technique addresses one
or more of the evaluation questions and indicators shown in the table above.
The selected data collection techniques are presented below:
 Electronic Survey of Short Course Participants
 Focus Group with Selected Participants
 Document Review of Related Archival Data
 Request for Information on Key Issues
Electronic Survey of Short Course Participants
Survey all program participants from 2004-2012 to assess genomic
Overview:
curriculum integration, institutional limitations for achieving program
goals, program influence on continued participant engagement with
NIH/NHGRI, and the career/educational paths of program participants.
Data Source(s):

Data Source
Short Course Faculty Participants
Short Course Student Participants

Data Collection Method(s):

Survey: Electronic

Sampling:

None

Data Analysis Method(s):

Quantitative Analysis

Frequency:

One-Time Survey

Clearance Requirements:

Institutional Review Board
Paperwork Reduction Act of 1995

Estimated Budget

Medium

Estimated Level of Effort

1000-2000 hours

Estimated Timeline

8-12 months (depends on clearances)

Category
New Data
New Data

Qty.
N=186
N=111

Indicator(s):
This technique has the capacity to inform the following indicators:
(05) Post-program integration/engagement with NIH/NHGRI
(06) Educational and career paths of participants
(08) Length of time after program that curriculum change was achieved
(09) Type and quantity of genomic curriculum updates
(11) Faculty attendee perceptions of the program influence on their career
(12) Student attendee perceptions of the program influence on their career
(14) Perception of institutional limitations or barriers that interfere with achieving program
outcomes
Data Sources and Collection Strategies:
Data Sources

A-3

Faculty Participants. Contact information, including first and last name, email address, phone number
and name of home institution, is available for the 186 faculty participants who have attended the Short
Course since 2004. Contact will be attempted with each individual.
Student Participants. Contact information, including first and last name, email address, phone number
and name of home institution, is available for the 95 student participants who have attended the Short
Course since 2005. In 2004, 16 student participants attended, but their email and phone numbers are
unavailable. Contact will be attempted with all 111 student participants.
Sampling

Over the course of nine years (2004-2012), a total of 297 individuals participated in the program.
Because this technique involves making contact with all 297 program participants, and no comparative
sub-groups are necessary, there is no need to conduct sampling.
Data Collection Instrument(s)

Electronic Survey. Defining effective survey questions is among the most important elements of survey
research. The survey should consider: the goals of the survey (indicators defined above); information
desired from each respondent; and respondent burden. Questions will consider best practices in survey
design to elicit proper feedback and encourage participation. For example, questions should consider
the concepts they intend to measure, if all respondents would interpret the question in the same way,
and if all respondents would be willing to answer.
The nature of each evaluation question and type of information required will inform the type of
question created to collect responses. For example, if the goal is to understand the strength of a
participant’s point of view (e.g., how influential the Short Course was for integrating genomics into their
curriculum), Likert scales could be developed that would shed light on the magnitude of program
influence. However, if the goal is to capture an illustration of a student participant’s career path then
categorical choices may be developed. Where appropriate, opportunities for short open-ended
narratives will be allowed; one method for optimizing response rate it to be sure the respondent can
offer constructive criticism3.
The purpose of the survey is to collect as much information as possible from as many program
participants as possible with minimal burden. The survey should take no longer 20 minutes to complete.
A small group of NHGRI program staff will be selected to pilot the survey and collection process.
Data Collection Process

Step 1. Contact Participants. Electronic delivery of an introductory letter to all program participants will
provide initial contact. The letter will explain the intent of the evaluation, the data collection instrument
and process, and timeline for completing the data collection process (including a date for delivery of the
electronic survey). To encourage responses, the letter will offer online links to new resources for
teaching genomics (to illicit program loyalty), provide an option to receive a hard-copy of the survey
with a postage-paid return envelope, and offer an honorarium or financial incentive to participate.

Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment
& Evaluation in Higher Education, 33(3), 301–314. DOI: 10.1080/02602930701293231

A-4

It should be expected that a number of email addresses will fail due to inaccurate contact information.
All email failure notices will be captured, and the following steps will be taken to locate the individual:



Attempt contact via telephone
Search the World Wide Web, PubMed, or Lexis Nexus database using first and last name of the
participant, in conjunction with the name of the institution reported at time of program
attendance
o Follow up using available information (e.g., new email address, phone number, LinkedIn
profile, etc.)
 Review eRA Commons profiles for a matching name and history.
 Send a letter to the institution reported at time of program attendance to identify contact
information.
Once contact is made, we will provide the former participant with the introductory letter and update
their contact information in the database. If the individual cannot be located using any of the three
steps above, s/he will be categorized as “Unable to Locate”.
Step 2. Send the Survey. A hard copy of the survey will be mailed to individuals upon request (names
will be gathered and surveys sent once per week until complete); otherwise, the URL for the survey will
be emailed to participants one week after the introductory letter was delivered. This communication will
restate pertinent information and remind the participant of the resources available to them and the
honorarium eligible to them when complete the survey. The email will assure former participants their
feedback is anonymous and valued, and will be used to make program funding and design decisions. The
letter will conclude by thanking them for time and candidness.
Step 3. Reminders and Deadline Extension. After one week, a reminder note will be sent to all former
participants who have not completed the survey. After two weeks, another note will be sent to those
who have not completed the survey, and at that point the deadline will be extended one week.
Depending on the response rate (e.g., if lower than 20%), evaluation staff may decide to send hard
copies of the survey, or continue with another reminder.
The data sources and collection strategies presented here include techniques documented by
researchers who have achieved what is considered a high response rate to online surveys4. Response
rates have been declining in recent years (since the 1950s), and although there is no official gold
standard for response rates in evaluation research, anywhere between 50-60% is deemed
adequate/average5. This group of individuals is assumed to be highly motivated and committed to the
scientific endeavor; provided a high rate of successful contact with former program participants, a
response rate of at least 50% is reasonable.
Design Advantages, Limitations and Challenges:
Advantages

Nulty, D. D. (2008). The adequacy of response rates to online and paper surveys: What can be done? Assessment
& Evaluation in Higher Education, 33(3), 301–314. DOI: 10.1080/02602930701293231
5
Baruch, Y. (1999). Response rate in academic studies-A comparative analysis. Human Relations, 52(4), 421-438.
DOI: 10.1177/001872679905200401

A-5







Faculty participants can provide the most detailed information about the degree of curriculum
integration at their institution, and about the supports or barriers they experienced at the
institutional level.
Only Faculty participants can provide insider information about their experiences with the
program and the lasting effects it may be having on them professionally (e.g., knowledge gains,
capacity for teaching genomics, professional activities).
Only Faculty participants can provide information about the overall value of the program.
Program leaders can opt to prioritize and select the number of questions deemed appropriate
for available resources.
Study results are suited for publication.

Limitations and Challenges







This technique is limited to participant perspectives (subject to response bias). Including other
perspectives, such as other faculty at the participants’ institution, or gathering data from more
objective sources (e.g., NIH Listservs or the eRA Commons Profile), could provide a more
comprehensive understanding of program outcomes.
The information age facilitates ease for locating and contacting an individual (especially
professionals), but the level of effort required to update contact information is unknown.
It can be challenging to get a sufficiently high response rate, especially from individuals who
participated over 5 years ago.
It may be difficult to attribute successful outcomes specifically to this program.

A-6

Focus Groups with Selected Participants
Conduct focus groups with a sample of past participants who
Overview:
responded to the survey to investigate in more detail their survey
feedback.
Data Source(s):

Data Source
External stakeholders: Former
Program Participants

Data Collection Method(s):

Focus group: Virtual

Sampling:

Maximum Variation Sampling

Data Analysis Method(s):

Qualitative Analysis

Frequency:

One-Time Focus Groups

Clearance Requirements:

Institutional Review Board

Estimated Budget

Low to Medium6

Estimated Level of Effort

500-1000 hours

Estimated Timeline

3 months

Category
New Data

Qty.
N=9 or Less

Former Program Participants. This approach collects new data in focus groups from previous program
participants based on their feedback to the survey.
Sampling

Number of Focus Groups. Focus groups will allow NHGRI to derive meaning from the abbreviated
responses received from the survey and ask additional probing questions of survey respondents. Based
on results from the survey (technique one), NHGRI will identify which areas they’d like to understand in
greater depth, and then plan to conduct more than one focus group, based on those areas.
Conceptually, some areas of questioning may be appropriate to stand alone in a singular focus group
(e.g., institutional barriers), while others areas may be complementary and should be grouped (e.g.,

Cost is impacted by number of focus groups and travel costs if focus groups are in person (vs. virtual).

A-7

career path and influence of the program on career path). To address OMB clearance requirements,
each focus group must be significantly different.
Method. Depending on the areas of interest that arise from the survey results, either criterion or
maximum variation sampling should be used to create samples. Both are purposeful sampling methods,
but they differ by the number of selection criteria7. Criterion sampling is used when one important and
overarching criteria is identified, and maximum variation sampling is used when a number of criteria are
identified from within one group (e.g. traditionally underrepresented institutions). For example, if
NHGRI wants to develop a deeper understanding of institutional barriers, then criterion sampling would
be used first, and all survey respondents who identified institutional barriers as a problem would
become a part of that sample. The maximum variation sampling may be more appropriate to include
individuals on both sides of a particular survey response (e.g. including vs. not including Genome
Scholars in the program) would become part of the sample. In some cases, a mixture of both methods
makes sense; still with the institutional barriers example, once criterion sampling results in a group of
those who identified barriers, then the maximum variation sampling method could be used to ensure
representation on other criteria (e.g., faculty or student rank, size of the institution, geographic location,
and racial composition). In the end, the questions will dictate the required sampling method, and those
are undetermined at this time.
Number of Participants. Regardless of sampling method, when creating samples for each focus group,
the total size for each focus group should include 12 individuals. However, to avoid the need for OMB
clearance, no more than nine participants will be invited to each focus group. Selecting 12 individuals
allows space for up to four participants to decline participation; however, because the focus groups are
virtual, acceptance to participate is expected to be high.
Data Collection Instrument(s)

A discussion guide will be developed as a tool to walk participants through the focus group session and
keep the facilitator on track. The guide will include explanation and request for consent, information
about the process and purpose of the focus group, study questions and probes, a potential exercise, and
space to collect individual narrative responses.
A sign-in sheet will be used to collect demographic information including name, position, name of
institution, and contact information (e.g., phone, email address and mailing address). If appropriate, we
may also collect race and ethnicity for student participants (Genome Scholars).
Data Collection Process

Step 1. Contact Participants. Once identified, 9 participants will be contacted via email inviting them to
participate. The e-mail letter will outline the intent of the overall project and data collection effort,
explain the data collection instruments and process, and request participation in the Focus Group. This
invitation will include a date and time so that participants can save the date.
Step 2. Determine Participation. Follow-up contact with potential participants will be made one week
after sending an email to determine commitment to participate. Return to step one to contact the backup participants for those who decline to participate.

Patton, M. Q. (1990). Qualitative evaluation and research methods. Thousand Oaks, CA: Sage.

A-8

Step 3. Send Materials and Reminder. An email will be sent to participants at least three days prior to
the focus group to confirm their attendance. The email will include virtual focus group login instructions,
the consent form, sign-up sheet and discussion guide. Request will be made for the signed consent to be
returned prior to the meeting (via email).
Step 4. Conduct the Focus Group. The length of time required for the focus group will depend on the
number of questions selected by NHGRI. The Focus Group should last no longer than two hours. The
focus group will be recorded electronically for further analysis and participants will be made aware of
this prior to initiating the focus group.
Step 5. Thank you. Each participant will receive a thank you e-mail and receive financial honorarium if
deemed appropriate.
Following the data collection effort, the feedback will be analyzed and summarized in a focus group
report that identifies key themes and makes strategic recommendations for next steps.
Design Advantages, Limitations and Challenges:
Advantages







This technique as a follow-up to the first survey technique limits the number of narrative
questions needed in the original survey, which keeps the survey burden lower.
This technique provides important perspectives and new or more detailed insights on selected
topic areas.
When little is known about a phenomenon, a focus group is an effective method for gaining
some detailed information that could potentially inform further data collection, or respond to
current trends/concerns.
Focus groups are ideal for encouraging discussion, especially when participants can benefit from
hearing perspectives of others and build on that discussion (e.g., action research8).

Limitations and Challenges





Sampling may require more of an investment than NHGRI is willing to make at this time.
The virtual focus group method is less personal and is not as likely to encourage participation.
Two hours is a large investment for a virtual meeting; consider offering intellectual incentives.

Rossman, G. B. & Rallis, S. F. (2003). Learning in the field: An introduction to qualitative research. Thousand Oaks:
Sage.

A-9

Document Review of Related Archival Data
To determine post-program engagement with NIH/NHGRI, conduct
Overview:
targeted searches of NIH/NHGRI databases.
Data Source(s):

Data Source
Databases: Outreach databases
and Listservs
Databases: IMPACII

Category
Archival

Qty.
unknown

Archival

N=297

Data Collection Method(s):

Document Review

Sampling:

None

Data Analysis Method(s):

Descriptive Statistics

Frequency:

One-time

Clearance Requirements:

None; however, any analysis and publication of this data will require
Institutional Review Board (IRB) clearance

Estimated Budget

Low

Estimated Level of Effort

500 hours

Estimated Timeline

2 months

Indicator(s):
This technique has the capacity to inform the following indicator:
(04) Degree to which former program participants continue(d) to interact with NIH/NHGRI
Data Sources and Collection Strategies:
Data Sources

Outreach databases and Listservs. All NIH outreach databases and the central NIH Listserv database
should be scanned for subscription by former short course participants based on name and e-mail
address.
IMPACII. NIH maintains records in IMPACII of all applicants and awardees for NIH grants. IMPACII could
be searched to determine the frequency of all 297 participants in applying for and receiving NIH awards.
Sampling

This technique involves reviewing existing data for all past program participants; therefore, sampling is
not relevant.
Data Collection Instrument(s)

We recommend that NHGRI create an internal database to compile and analyze information.
Data Collection Process

Step 1. Confirm data availability and access. Contact NIH CIT about the ability to query the entire
Listserv database for participants. Utilize the QVR system to access the appropriate information on each
participant. Gather NHGRI specific communication database lists.

A-10

Step 2. Determine the desired data to collect. It is assumed that NHGRI is searching to match names or
e-mail addresses of past participants against the other databases to determine the degree to which
these individuals are participating in NIH activities. The data collected from each database will help to
not only identify the number of individuals still connected to NIH in some way, it will also characterize
the type and amount of involvement.
Step 3. Develop data structure. Create database to collect and analyze data.
Step 4. Collect data. Review the data sources and populate the data structure with database
information.
Design Advantages, Limitations and Challenges:
Advantages



Assuming access to the data, this technique can be completed quickly.

Limitations and Challenges






Results are informational and should not be considered indicators of program success or failure;
there are several ways to stay engaged with NIH and NHGRI beyond electronic means.
Former participants may have changed their names, e-mail addresses or institutional location,
making it difficult to match.
Engagement with NIH or NHGRI may have continued without funding success.
Former short course participants may be working on an NIH grant without being named on the
grant itself (especially if in a postdoc position). Although recent records in IMPACII include such
information as part of the digitized “All Personnel” report, older records may only contain
scanned copies (images); therefore, the text contained within the report is not searchable
because it is an image.

A-11

Request for Information on Key Issues
Create a Request for Information to gather feedback about the
Overview:
program goals and related evaluation issues/questions.
Data Source(s):

Data Source
Scientific Community At-Large

Category
New Data

Data Collection Method(s):

Request for Information

Sampling:

None

Data Analysis Method(s):

Descriptive Statistics; Qualitative Analysis

Frequency:

One-time RFI

Clearance Requirements:

None

Estimated Budget

Low-Medium9

Estimated Level of Effort

500-1500 hours

Estimated Timeline

4 months

Qty.
unknown

Indicator(s):
This technique has the capacity to inform the following indicator:
(14) Perception of institutional limitations or barriers that interfere with achieving program
outcomes
Data Sources and Collection Strategies:
Data Sources

Scientific Community At-Large. This technique uses a public comment period to solicit feedback from
the scientific community at-large. It is possible to not only get feedback from past and future
participants, it is also likely to get feedback from other stakeholders in the process such as Institutional
leaders.
Sampling

This approach utilizes an announcement that is broadly announced through the NIH Guide for Grants
and Contracts and the Federal Register. Therefore, it is open to anyone who wishes to respond;
sampling does not apply to this technique.
Data Collection Instrument(s)

“Public comment” describes the process of soliciting feedback from the public or membership on official
reports, regulations, guidelines or programs such as this short course. In today’s electronic world, public
comments can be received via Facebook, blogs, agency web sites and other social media outlets. The
management and analysis process systematically consolidates and synthesizes all public comments to
provide organizations with information to drive the decision making process. NHGRI must carefully
design their approach to public consultation so they ask the right questions and maximize their effort.
The public comment can be utilized by NHGRI for the following:

Dependent upon the breadth of the RFI and the number of responses

A-12



Obtain New Ideas. Public comment periods can be used to get new ideas from stakeholders. For
example, NHGRI could get focus feedback on the topic areas that future participants are
interested in learning about.
Business Process Feedback. Public comment periods may focus on business processes. These
RFIs often focus on implementation of changes and the implications it may have for individuals
and organizations affected by a change in process. For example, NHGRI could ask about barriers
that exist to implement curriculum changes at institutions and how that relates to the size and
type of institution.

Data Collection Process

Public Comment Management and Analysis is a five step process to facilitate the solicitation, receipt and
analysis of public comments.


Step 1. Planning and Release. This phase includes drafting the notice for publication in the NIH
Guide for Grants and Contracts and/or the Federal Register. It also includes:
o Determining the collection instrument (web based form or e-mail responses)
o Considering the Paperwork Reduction Act
o Preparing the analysis plan
o Developing the questions



Step 2. Response Collection. This phase includes:
o
o
o



Step 3. Data Organization. This phase includes:
o
o
o
o



Creating and hosting the analysis database
Cleaning the data for consistency
Counting the responses
Creating dashboard reports

Step 4. Data Analysis. This phase includes:
o
o
o
o
o



Publishing the request in the NIH Guide and/or Federal Register
Creating a database to analyze responses
Hosting the database/web site for response collection

Developing a coding schema for analyzing the narrative responses
Parsing the qualitative responses into comments
Assessing inter-coder reliability
Determining the themes from the data
Creating initial reports from the responses

Step 5. Report Summaries. This phase includes:
o
o
o
o

Reporting the quantitative and qualitative results
Developing summaries and recommendations
Generating and formatting graphs
Creating reports (text and/or slides)
A-13

Design Advantages, Limitations and Challenges:
Advantages





This approach is cost effective and can provide useful information to inform future program
design
This approach can gather feedback from a wider audience including institutional leadership, and
students.
This approach focuses on developing program approaches that reflect current needs.

Limitations and Challenges




This approach will not provide information to evaluate the success of the current short course
program.
Without addressing topics that the scientific public consider interesting or controversial,
response rates can sometimes be skewed (limited perspective) or limited in number.

A-14

Conclusions and Recommendations
Feasibility
As described in the Center for Disease Control (CDC) Program Evaluation Framework,10 the feasibility of
an evaluation should consider the following:
 Purpose of the Evaluation
 Use of the Evaluation Results
 Stage of Program Development
 Intensity or Scope of the Program
 Resource Considerations
Purpose of the Evaluation. As outlined in the evaluation design section of this proposal, NHGRI would
like to focus on two main outcomes: 1) the degree of genetic and genomic curriculum integration at
institutes represented by the faculty participants, and 2) the current career paths of former student
participants, including their perceptions about the influence of the summer workshops on their career
choices.
Use of the Evaluation Results and Stage of Program Development. Enough time has passed since
program implementation to properly evaluate the program’s impact on intermediate-term outcomes.
This information will be used to evaluate what aspects of program design should be changed for future
implementations.
Intensity or Scope of the Program and Resource Considerations. The overall investment in the program
is significant if all the years of implementation are considered. Each year, NHGRI serves a small number
of participants; however the overall program as described is an important investment by NHGRI
leadership.

Recommendations
Technique Selection
If it is NHGRI’s goal to evaluate the success of the Short Course program to-date, then we recommend
NHGRI implement the first technique: Electronic Survey of Short Course Participants. This approach will
provide the most detailed information about the degree of curriculum integration at the institutions of
participants, and about the supports or barriers they experienced at the institutional level. Only
participants can provide insider information about their experiences with the program and the lasting
effects it may be having on them professionally (e.g., knowledge gains, capacity for teaching genomics,
professional activities).
NHGRI may not know if the Focus Group with Selected Participants is useful or necessary until they
receive the feedback from the survey. NHGRI can consider using the focus group technique in different
ways as described in the alternatives below.

http://www.cdc.gov/eval/framework/index.htm

A-15

If funds are limited, then NHGRI may want to consider investing in the final two techniques (Document
Review of Related Archival Data, Request for Information on Key Issues) which provide useful
information, but may not fulfill the desire to evaluate program success.

Alternatives and Considerations.
Although we recommend any combination of the three techniques detailed in this report, we would like
to point to alternative considerations.
Alternative: Conduct Focus Groups Before the Survey
Technique: Focus groups with Selected Participants
Data Collection Instrument(s): Focus Group
Details. Oftentimes focus groups are used as a means to identify relevant survey questions. Once
themes are identified by focus group, surveys can help provide information about prevalence. NHGRI
could chose to conduct the focus groups before the survey, which would provide NHGRI with the
capacity to ask more informed and targeted survey questions and develop potentially more meaningful
categorical response options for the survey.
Alternative: Focus Group with institutional Leaders
Technique: Focus groups with Selected Participants
Data Collection Instrument(s): Focus Group
Details. The focus groups could comprise of institutional leaders from the selected institutions,
specifically those individuals who recommended the faculty to participate in the program. This audience
could offer unique insights about the barriers that exist for faculty who are trying to update curriculum
at their institution.
Alternative: Focus Group Without the Survey
Technique: Focus groups with Selected Participants
Data Collection Instrument(s): Focus Group
Details. Though the focus group technique presented in this document is targeted to occur after the
survey, this technique could be considered as a separate technique entirely and be used as a lower cost
method, with less clearances, to gather data from past participants. However, sampling the appropriate
9 participants may be more challenging without the data from the survey.
Future Program Development
NHGRI should consider building evaluation into program design, which may translate into building
participant follow-up into the program design. As part of acceptance into the program, NHGRI should
seek program participant approval to provide follow-up information. This approach could reduce the
clearances required and increase the participation rates in long term follow-up studies.

A-16

Appendix B: Focus Group Questions
Perceptions of the Program






How would you say attendance at the Short Course influenced the way you felt or feel about
your capacity to teach genetics and genomics?
Would you say it had a long-term effect?
Describe aspects of the training program you felt were the most useful?
Which aspects of the program were not useful for you?
After you completed the Short Course, and when you got back to your institution, what was is
like to try to integrate what you learned at the Short Course into your teaching materials? What
kinds of personal challenges did you experience?

Institutional Involvement




In what ways did your institution help you incorporate new knowledge and skills into your
teaching materials?
What kinds of institutional challenges have you experienced while disseminating and integrating
knowledge from the Short Course into your teaching materials?
What challenges do you believe are unique to your institution and which ones are systemic (e.g.,
typical across all higher education institutions)?

Diversity in the Genome Professional Pipeline


One of the goals of the program is to diversify the pipeline of genome professionals. In what
ways do you think the Short Course is contributing to this effort?

Recommendations for Program Improvement




Consider other genomic-related professional development activities you’ve attended over the
years (e.g., other trainings either online or in person), how does the Short Course compare to
those? What is unique about it? What are its strengths and weaknesses?
Suppose you were going to put together a Short Course to prepare the next generation of
genomic professionals for genomic medicine, how would your course look different than the
short course you attended? Consider the structure and the curriculum and how the program is
funded.

B-1

Appendix C: Survey Questions
Participant Information
Historical Information
This first set of questions focuses on verifying or gathering information as it stood at the time of your attendance to
the Short Course; we will ask for updated information next.
Select your role during your participation in the Short Course (All Attendees)
1. Faculty Attendee
2. Student Attendee
3. Nursing Faculty Attendee
[Skip logic: If Student Attendee] What degree program were you enrolled in when you attended the Short Course?
1. Bachelor of Science (BS)
2. Bachelor of Arts (BA)
3. Master of Science (MS)
4. Master of Arts (MA)
5. Master of Public Health (MPH)
6. Doctor of Philosophy (PhD)
7. Doctor of Medicine (MD)
8. Dual degree (MD & PhD)
9. Registered Nurse (RN)
10. Other, please specify: ____________
[Skip logic: If Faculty or Nurse Faculty Attendee] What was your primary Position or Occupation Title at the time
of your attendance to the Short Course?
1. Researcher
2. Adjunct Instructor/Professor
3. Assistant Professor
4. Associate Professor
5. Professor
6. Distinguished and/or Endowed Professor and/or Emeritus
7. Other, please specify: ____________
[Skip logic: If Faculty or Nurse Faculty Attendee] Was this a tenure or non-tenure track position?
1. Tenure track
2. Non-Tenure track
Did you hold a teaching appointment at the time of your attendance to the Short Course? (Faculty/Nursing Faculty
Only)
1. Yes

C-1

2. No
[Skip logic: If Yes] Out of 100 percent, what best represented your teaching/research ratio at the time?
 Teaching __________


Research __________



Other, please specify __________

Current Information
This section will capture change or updates to your professional or academic profile since your attendance to the
Short Course.

Are you still with _____________________________________(prepopulate with Institutional name)? (All
Attendees)
1. Yes
2. No
[Skip logic: If No] Please list your current institution or organizational affiliation.

Has your Position or Occupation Title changed from INSERT PIPED ANSWER since the time of your attendance in
the Short Course? (Faculty/Nursing Faculty Only)
1. Yes
2. No
[Skip logic: If Yes] What is your current primary Position or Occupation Title?
 Researcher
 Adjunct Instructor/Professor
 Assistant Professor
 Associate Professor
 Professor
 Distinguished and/or Endowed Professor and/or Emeritus
 Administrator
 Other, please specify __________
[Skip logic: If Yes] Please specify whether this is a tenure or non-tenure track position
 Tenure-track
 Non-Tenure-track
What is your current primary Position or Occupation Title? (Student Attendee Only)
1.
Student
2.
Trainee
3.
Clinician

C-2

4.
5.
6.

7.
8.

Educator (K-12)
Researcher
Administrator
Instructor or Professor
Other, please specify __________
o

[Skip Logic: If Student] What type of degree program are you enrolled in?
 MS


MPH



PhD



MD/PhD



Other, please specify __________

[Skip logic: If Trainee] Please specify trainee type:
 Predoctoral
 Postdoctoral
 Clinical
 Other, please specify __________

[Skip Logic: If Instructor/Professor] Please specify the type of instructor or professor position.
 Researcher
 Adjunct Instructor/Professor
 Assistant Professor
 Associate Professor
 Professor
 Distinguished and/or Endowed Professor and/or Emeritus
 Administrator
 Other, please specify __________
o [Skip Logic: If Instructor/Professor] Please specify whether this is a tenure or non-tenure track
position.
 Tenure-track
 Non Tenure-track
Does your current position involve teaching genetics or genomics? (Student Attendee Only)
 Yes
 No
[Skip Logic: If Yes) Select the content areas you teach: (Select all that apply).
1.
Biology of genetics and genomics
2.
Biology of human genetics
3.
Molecular/cellular genetics

C-3

4.
5.
6.
7.
8.
9.
10.

Genetics and genomics of common diseases
Genetic and genomic technology
Pharmacogenetics and genomics
Ethical, legal and social implications of genetics/genomics
Resources for genetic/genomic education
Career resources (e.g., grant writing)
Other, please specify: __________

[Skip Logic: If Yes) Out of 100 percent, what best represents your current teaching/research ratio?
(Student Attendee Only)




Teaching __________
Research __________
Other __________

Which of the following best describes your primary field of work? (All Attendees)
1. Academia
2. Government
3. Clinical
4. Clinical/Government
5. Industry
6. Non-profit
7. Other, please specify: __________
Please identify any additional credentials or degrees you have received or pursued since the Short Course. (Select all
that apply). (All Attendees)
1. None received or pursued
2. BS/BA
3. MS/MA
4. MPH
5. PhD
6. MD/PhD
7. MD
8. RN
9. Other, please specify: _____________
[Skip logic: If received or pursuing] Have you completed the credential or degree(s)?
 Yes
 No
 Other, please specify: ______________
[Skip logic: If received or pursuing] What discipline(s) is your credential or degree associated with?

The Short Course Program
The next set of questions focuses on your professional activities since the Short Course.

Teaching Appointment
C-4

Out of 100 percent, what best represents your current teaching/research ratio? (Faculty/Nursing Faculty Only)
1. Teaching __________
2. Research __________
3. Other __________
[Skip logic: If Yes] To what degree did your experience at the Short Course influence your teaching to research
ratio? (Faculty/Nursing Faculty Only)
1. No influence
2.

Some influence

Moderate influence

Great influence

How many year(s) have you been engaged in teaching? (Faculty/Nursing Faculty Only)

Integration of Short Course Knowledge
Please select one of the three options below in response to the following statement: I was able to update my
curriculum as a result of my participation in the Short Course. (Faculty/Nursing Faculty Only)
Yes, I made some changes
Yes, I made minor changes
No, I did not make any changes
[Skip logic: If Yes or variant of Yes] I was able to update my genetics or genomics curriculum and teaching
materials in the following ways: (Select all that apply)
1. Informal/unplanned integration (e.g., anecdotal narratives, spontaneous examples)
2. Added content to my lecture materials
3. Added assignments to my courses
4. Removed other content to make way for new information
5. Developed new teaching objectives for my existing courses
6. Created new lab experiments
7. Developed a new course
8. Shared teaching material with other faculty
9. Other, please specify: __________
[Skip logic: If Yes or variant of Yes] I was able to update my curriculum and teaching materials in the following
content areas: (Select all that apply)
10. Biology of genetics and genomics
11. Biology of human genetics
12. Molecular/cellular genetics
13. Genetics and genomics of common diseases
14. Genetic and genomic technology
15. Pharmacogenetics and genomics
16. Ethical, legal and social implications of genetics/genomics
17. Resources for genetic/genomic education
18. Career resources (e.g., grant writing)
19. Other, please specify: __________
[Skip logic: If Yes or variant of Yes] How much time was required to update your curriculum and teaching materials

C-5

after participating in the Short Course?
1. Less than 3 months
2. Between 3 and 6 months
3. Between 6 months and 1 year
4. Between 1 and 2 years
5. Between 2 and 5 years
6. More than 5 years
Beyond curriculum integration, in what other ways did you disseminate information from the Short Course to
students? If you did not disseminate information beyond curriculum integration, please write “None.”
(Faculty/Nursing Faculty Only)

Time and space to accommodate or integrate new
information into existing curriculum
Relevant course(s) not available
Process for changing curriculum
Support of colleagues
Support by supervisor or leadership
Institutional awareness about the importance of
genetic and genomic content
Curriculum committee support
Incentive (financial or recognition/praise)
Institutional funding or resources
Ability to secure external funding or resources
Protected time to develop content

No Influence

Minor
Influence

Moderate
Influence

Great
Influence

Not
Applicable

❏
❏
❏
❏
❏
❏
❏
❏
❏
❏
❏

Please rate how much you think the following student-related factors influenced your efforts to transfer your
knowledge to students. (Faculty/Nursing Faculty Only)

Student preparedness (e.g., students’ prerequisite
knowledge)
Student perception around applicability of
information to their career

No Influence

Some
influence

Moderate
influence

Great
influence

Not
applicable

❏
❏

C-6

❏
❏
❏

Student awareness about the importance of genetic
and genomic content
Students’ interest
Students’ self-confidence

❏
❏
❏

Did the Short Course influence your research? (All Attendees)
1. Yes
2.

[Skip logic: If Yes] Please describe how your experience at the Short Course influenced your research.

Presentations at scientific meeting
Attendance at scientific meeting
Involvement in professional organizations
Took additional classes on genetics/genomics
Attended seminars/events on genetics/genomics
Informal genetics/genomics chats
Engaged a mentor in the genetics/genomics
field
Became involved in a research group related to
genetics/genomics
Conducted research in genetics/genomics
Attended professional meetings related to
genetics/genomics
Presented material at various events related to
genetics/genomics
Published findings related to genetics/genomics

No
Influence

Some
influence

Moderate
influence

Great
influence

Not applicable

❏
❏
❏
❏
❏
❏
❏
❏
❏
❏
❏
❏

C-7

❏
❏
❏

To what degree did your experience at the Short Course influence your decision to pursue new career or educational
options? (All Attendees)
1. No influence
2. Some influence
3. Moderate influence
4. Great influence
5.
Not applicable – I did not pursue new career or educational options
[Skip logic: If Yes or variant of Yes] Please describe how the Short Course influenced your decision to pursue new
career or educational options.

Since the Short Course, have you been involved with Short Course attendees, NIH or NHGRI? (All Attendees)
 Yes
 No

[Skip logic: If Yes] Please indicate the ways you have been involved with Short Course attendees, NIH or NHGRI
since attending the program. (Select all that apply)
1. Communications with fellow attendees
2. Additional contact with presenter(s)
3. Additional contact with NIH or NHGRI staff
4. Pursued other training or educational opportunities at NIH
5. Pursued other training opportunities at NHGRI
6. Used NHGRI online resources
7. Involved in writing NIH grant-application
8. Awarded NIH grant
9. Joined NIH Listserv(s) or other forms of communication
10. Other, please specify: _____________

Background Information
This section is focused on demographic information.

What year were you born? (All Attendees)
1999
9.

1991

17. 1983

1998

10. 1990

18. 1982

1997

11. 1989

19. 1981

1996

12. 1988

20. 1980

1995

13. 1987

21. 1979

1994

14. 1986

22. 1978

1993

15. 1985

23. 1977

1992

16. 1984

24. 1976

C-8

25. 1975

58. 1942

91. 1909

26. 1974

59. 1941

92. 1908

27. 1973

60. 1940

93. 1907

28. 1972

61. 1939

94. 1906

29. 1971

62. 1938

95. 1905

30. 1970

63. 1937

96. 1904

31. 1969

64. 1936

97. 1903

32. 1968

65. 1935

98. 1902

33. 1967

66. 1934

99. 1901

34. 1966

67. 1933

100. Do not wish to provide

35. 1965

68. 1932

36. 1964

69. 1931

37. 1963

70. 1930

38. 1962

71. 1929

39. 1961

72. 1928

40. 1960

73. 1927

41. 1959

74. 1926

42. 1958

75. 1925

43. 1957

76. 1924

44. 1956

77. 1923

45. 1955

78. 1922

46. 1954

79. 1921

47. 1953

80. 1920

48. 1952

81. 1919

49. 1951

82. 1918

50. 1950

83. 1917

51. 1949

84. 1916

52. 1948

85. 1915

53. 1947

86. 1914

54. 1946

87. 1913

55. 1945

88. 1912

56. 1944

89. 1911

57. 1943

90. 1910

C-9

What is your sex? (All Attendees)
1. Male
2.

Female

Do not wish to provide

What race do you consider yourself? (Select all that apply) (All Attendees)
1. White or Caucasian
2.

Black or African American

Asian

American Indian/Alaska Native

Native Hawaiian or other Pacific Islander

Do not wish to answer

Do you consider yourself to be Hispanic/Latino? (All Attendees)
1. Yes
2.

Do not wish to provide

Do/did you have a disadvantaged background, which can be defined either as coming from a family with an annual
income below established low-income thresholds, or coming from an educational environment such as that found in
certain rural or inner-city environments that have demonstrably and directly inhibited you from obtaining the
knowledge, skills, and abilities necessary to develop and participate in a research career? (All Attendees)
1. Yes
2.

Do not wish to answer

Please enter any other comments you would like to share with us about the Short Course and your experiences since
attending the program. If you have no additional comments, please hit the next button. (All Attendees)

This is the end of the survey, thank you for your time, we appreciate your input.

C-10

File Type	application/pdf
File Title	Evaluation Plan
Subject	NHGRI Summer Workshop in Genomics
Author	Ripple Effect Communications, Inc.
File Modified	2015-12-14
File Created	2015-12-14