Statement A Attachment 4 IRB exemption

Statement A, Attachment 4

OHSRP # 12731
REQUEST FORM: OHSRP DETERMINATION FORSURVEYS, INTERVIEW PROCEDURES,
PROGRAM EVALUATION, EDUCATIONAL TESTING AND RESEARCH
Date of Request:

12-4-14

Requestor's name: Kyla Taylor

Email: taylorkw@niehs.nih.gov

Role: _Administrative support _Investigator

_lL_Other, explain: Contributor

Name of NIH Senior Investigator:
Senior investigators:
Kris Thayer, Ph.D., NIEHS/DNTP/OHAT
email: thayer@niehs.n ih.gov
phone: 919-541-5021
building/room: Keystone 2150

Mary Wolfe, Ph .D., NIEHS/DNTP/OLPR
email: wolfe@niehs.nih.gov
phone: 919 .541.7539
building/room: Keystone 2130

Is the NIH Sen ior Investigator an NIH employee(FTE)? _X__ Yes
Senior Investigator Signature:

.

.

Supervisor Signature:

No

_..,/~
/C~~.I.d
.: :~-F:=._ _ T.
: . . . .&j
t. .~~----------­
(Signature of Investigator who will conduct research)

~
<-·,/..__~

.

./
v-

-~
/1"

/

-/~

; ......._ '-"-

=>-

=

(Signature of official for IC, e.g., Lab/Branch Chief)
Name of NIH investigator conducting research if not the NIH Senior Investigator: (i.e,

junior investigator, contractor investigator, fellow, student)

Please provide t he name and e-mail of any others who should receive a copy of the
OHSRP determination: Kyla Taylor; taylorkw@niehs.nih.gov
1. What role will the NIH investigator(s) have in this research project? {check all that

apply)

__x_ Conduct research activity
__x_ Ana lyze samples/data only
_ Consultant/advisor to collaborator(s)
__x_ Author on publication(s)/manuscript(s) pertaining to this research
_ Other, please d e s c r i b e : - - - - - - - - - - - - - - - - -- - --

Page 1 of3

REQUEST FORM: OHSRP DETERMINATION FORSURVEYS, INTERVIEW PROCEDURES,
PROGRAM EVALUATION, EDUCATIONAL TESTING AND RESEARCH

2. Title: NTP Research Project: Updating Level of Concern Categories

{Provide a short title to distinguish this activity from other projects that you may have)
3. Describe in lay terms the research activity that will be performed:

One hundred experts, about 20 from five sectors (federal government, state government,
industry, academia, and non-government organizations) will be recruited, including
toxicologists, epidemiologists, and experts in risk assessment.
Expetis will be instructed to sort through 30 "cards", cards are theoretical hazard
identification and exposure level scenarios, into as many categories as they wish. The
categories will represent their level of concern (LoC) based on the given scenario with
the extreme categories representing "high" and "negligible" LoC scenarios. Each expert
will be trained using a web-based instructional script with visual instructions on sorting
the cards and documenting the results and will work independently. Each expert using a
unique password will access the case-study cards on a password-protected Website.
Using a web-based tool, experts will sort the cards into as many LoC categories as they
wish. Experts will be given 2 weeks to complete the task (trial 1). The outcomes from
individual experts will not be shared with the other experts.
Experts will repeat the exercise a second time (trial 2), approximately 3-5 weeks
following the initial session, to evaluate consistency of response within individuals. For
trial 2, the 30 cards will be reordered randomly. Following completion of the card sorting
for trial 2, experts will be asked to provide descriptive labels for the LoC categories they
identify.

4. Proposed start date March, 2015

Proposed completion date September, 2018

5. Specify the nature of the data: (select all that apply)
_Interview procedure
25._ Survey
_Educational Testing
Educational Research
_Research on pub lic benefit or service programs
25._ Other, describe: Risk communication
6. What kind of human data (e.g., private information, responses to questionnaires,
test results, recordings) will be collected in your research?
Responses to survey questions
7. Will human data be? (select all that apply)
Collected
Yes -X NoReceived
Yes
No

Page 2 of3

REQUEST FORM: OHSRP DETERMINATION FORSURVEYS, INTERVIEW PROCEDURES,
PROGRAM EVALUATION, EDUCATIONAL TESTING AND RESEARCH
Sent

Yes

No

8. If receiving or sending, list the collaborating investigator(s):
Name
lnstitution/IC
Address/e-mail

FWA number*

9. Where are the subjects of this research activity located? (Provide a general

description or complete the institutional information below)
This research will be web based
Instit ution:
Address:

NA

--~~-----------

Contact Name: ---------------------

------------------------------- Phone: ---------------------

10. Will NIH investigator(s) have direct contact or intervention with the subjects of
the study? (For example, by interviewing, surveying or recording the subjects?)
Yes X No
If yes, what is the age range of subjects involved in the research?
_Chi ldren aged< 18 years
X
_Adults
aged 2: 18 years
11. Who will collect the data or information?
(a)_ NIH Investigator
(b ) _
non-NIH Collaborator
(c) __K_ NIH Contractor
(d )_ Other, specify
If b or c, will an Honest Broker or data use agreement be used? Yes --x_ No_
X

If yes, complete and attach the Honest Broker Assurance or data-use agreement to
this submission; e-mail ohsr nih ddir@od.nih.qov to request a form.
12. Select the best description that applies to the human data or information:
__lL Data or information will not contain any identifiable information, nor can it be
linked to individual subjects by you or your collaborators.
_ Data or information will be recorded in such a manner that subjects can be
identified directly or through identifiers linked to the subjects
13. Per NIH guidance, are all conflicts of interest by NIH employees (sender or
receiver), if any, resolved? _X_Yes ____ No**

Page 3 of3

REQUEST FORM: OHSRP DETERMINATION FORSURVEYS, INTERVIEW PROCEDURES,
PROGRAM EVALUATION, EDUCATIONAL TESTIN G AND RESEARCH

*A Federalwide Assurance (FWA) is issued by the U.S. Department of Health and Human
Services (DHHS)/ Office of Human Research Protections (OHRP) to institutions which
receive Federal funds/support to conduct human subjects research. To search for the
FWA# for domestic or international institutions go to
h ttp:/1ohrp. cit. nih. qov/search/fwasearch. aspx ?styp=bsc
**If the answer is "No", note that OHSRP will be unable to make a determination and
research may not proceed until all conflicts are resolved. For more information, see the
October 2011, A Guide to Preventing Fin ancial and Non -Financial Conflict of Interest in
Human Subjects Research at NIH. For assistance review the list of Ethics Coordinators
and find the contact for your /C: http://eth ics.od.nih.gov/coord. pdf

Page 4 of3

Eiserman, Julie (NIH/OD) [C]
From:
Sent:
To:
Cc:
Subject:

Taylor, Kyla (NIH/NIEHS) [E]
Thursday, January 08, 2015 5:04 PM
Eiserman, Julie (NIH/OD) [C]
Thayer, Kristina (NIH/NIEHS) [E]; Wolfe, Mary (NIH/NIEHS) [E]
RE: Follow Up re: Req fo Determination OHSRP 12731

Hi Julie,
Thank you for your email. In answer to your questions:
Yes, all respondents will be adults over 18.
You are correct, we are not using an honest broker.
We believe we are doing research defined as “a systematic investigation, including research development, testing 

and evaluation, designed to develop or contribute to generalizable knowledge.”
With the data we plan to update NTP's Level of concern framework (as part of implementing systematic reviews) 
to enhance transparency in describing how conclusions are reached, and identify strategies for improving the LoC 
framework as a risk communication tool. 
We will get you the documents with the language that will be administered to the participants soon.
Thanks,
Kyla
From: Eiserman, Julie (NIH/OD) [C]
Sent: Thursday, January 08, 2015 2:45 PM
To: Taylor, Kyla (NIH/NIEHS) [E]
Subject: RE: Follow Up re: Req fo Determination OHSRP 12731

Hello Kyla, 

I apologize for the delay.  We were overloaded with determinations before the break and then I was on vacation. I am 
reviewing your request for determination and have some questions. 

You didn’t answer the age question.  Can you confirm all respondents will be adults over 18?  The honest broker 
question is still checked as “yes”, but you aren’t using one, correct?  

I am still struggling with whether this is human subjects research or falls under the category of quality improvement or 
assessment.  You mentioned that “The purpose of the research activity is to improve risk communication language used 
by the National Toxicology Program” which sounds like QI to me.  It makes a difference in what type of determination I 
give you in our records (not human subjects research or exempt human subjects research).  In other words, would you 
say that you are ‘collecting data regarding the implementation of a practice for clinical, practical, or administrative 
purposes’? Or would you say you are doing research which is defined as “a systematic investigation, including research 
development, testing and evaluation, designed to develop or contribute to generalizable knowledge.” Can you explain 
what you plan to do with the data?  

1

The form instructions (on the cover page) ask that you ‘please attach the survey, questionnaire, interview script or test 
to the completed form together with the consent language that will be administered before the subject participates in 
the activity’.  If you have those documents, could you please send them? 

Julie M. Eiserman, MA, CCRP [C] 
Health Science Policy Analyst 
Office of Human Subjects Research Protections 
10 Center Drive, Bldg. 10, Suite 2C146 
Bethesda, MD  20892‐1154 
Office Phone: 301‐402‐3444 
Fax: 301‐402‐3443 
OHSRP website: https://federation.nih.gov/ohsr/nih/index.php (NIH login required) 
Public site: http://ohsr.od.nih.gov/ 

From: OHSR (NIH/DDIR)
Sent: Thursday, January 08, 2015 1:44 PM
To: Taylor, Kyla (NIH/NIEHS) [E]
Subject: RE: Req fo Determination Rec'd_OHSRP 12731 

Hi Kyla, 
Thanks for your email. This submission should be sent out to you by Friday afternoon. Thanks for your patience through 
the holiday season delays. 
Chris 

From: Taylor, Kyla (NIH/NIEHS) [E]
Sent: Thursday, January 08, 2015 11:47 AM
To: OHSR (NIH/DDIR); Thayer, Kristina (NIH/NIEHS) [E]
Subject: RE: Req fo Determination Rec'd_OHSRP 12731 

Hello Chris,  

Could you please update us on the status of our Request for Determination: OHSRP #12731? 

Thanks, 
Kyla 
From: OHSR (NIH/DDIR)
Sent: Wednesday, December 17, 2014 11:34 AM
To: Thayer, Kristina (NIH/NIEHS) [E]
Cc: Taylor, Kyla (NIH/NIEHS) [E]
Subject: Req fo Determination Rec'd_OHSRP 12731 

Good morning Dr. Thayer, 

This email is to verify that OHSR has received your Request for Determination and it is currently being processed as 
OHSRP #12731. Please use this number in any future correspondence regarding this study.   

Protocol Title: NTP Research Project: Updating Level of Concern Categories 

Thank you. 
Sincerely, 
Chris Brentin 
OHSRP ‐ National Institutes of Health 
2

Bldg 10, Suite 2C146                                                                                                                           
Bethesda, MD 20892 
Office Telephone: 301‐402‐3444 
Office Fax: 301‐402‐3443 

The NIH is committed to maintaining the highest standards for the protection of human
subjects.  

Please consider the environment before printing this e-mail 

3

OHSR (NIH/DDIR)
From:
Sent:
To:
Cc:
Subject:
Attachments:

Taylor, Kyla (NIH/NIEHS) [E]
Thursday, December 11, 2014 11:09 AM
OHSR (NIH/DDIR)
Thayer, Kristina (NIH/NIEHS) [E]
RE: Honest broker form request
OHSR Request Form 12-11-14.pdf

Julie, 

I have made the changes you suggested and attached the document with signatures. Are you the person I should send 
this to? Please let me know if you need anything else. 

Thanks for your help! 
Kyla 

From: OHSR (NIH/DDIR)
Sent: Tuesday, December 09, 2014 5:59 PM
To: Taylor, Kyla (NIH/NIEHS) [E]
Cc: Thayer, Kristina (NIH/NIEHS) [E]
Subject: RE: Honest broker form request

Hi Kyla, 

I would make the following changes. 

5. I would also check “survey”  

10. How will subjects be recruited? I am guessing that someone will need to contact them to request that they 
participate and someone on the team will know who is being contacted?  If so, I would change your answer here to 
“yes”. 

11. You can answer ‘no’ to the honest broker question. 

Once you have made these changes and gotten the signatures, I think you are good to go. 

Julie M. Eiserman, MA, CCRP [C] 
Health Science Policy Analyst 
Office of Human Subjects Research Protections 
10 Center Drive, Bldg. 10, Suite 2C146 
Bethesda, MD  20892‐1154 
Office Phone: 301‐402‐3444 
Fax: 301‐402‐3443 
OHSRP website: https://federation.nih.gov/ohsr/nih/index.php (NIH login required) 
Public site: http://ohsr.od.nih.gov/ 

From: Taylor, Kyla (NIH/NIEHS) [E]
Sent: Tuesday, December 09, 2014 3:40 PM
To: OHSR (NIH/DDIR)
1

Cc: Thayer, Kristina (NIH/NIEHS) [E]
Subject: RE: Honest broker form request

Julia, 

The purpose of the research activity is to improve risk communication language used by the National Toxicology 
Program. Participants will be non‐scientists and experts in toxicology, epidemiology, and risk assessment. Experts would 
be required to have a Masters of Science, PhD, MD, or equivalent degree and at least 3 years’ documented employment 
working in public health or a risk assessment‐related activity, such as developing hazard identification conclusions based 
on human and/or animal data or conducting qualitative or quantitative characterizations of risk.   

Data collection will be web based. Participants will be asked to complete an exercise where they read through different 
theoretical hazard identification and exposure scenarios and categorize the scenarios based on their level of concern. 
They will provide descriptive labels for the categories they create. We are not asking for any personal information. 

I’ve attached the form as I’ve filled it out so far. Hopefully that will help. I’ve also CC’d my supervisor in case she wants 
to weigh in on the information I’ve provided. 

Thanks, 
Kyla 

From: OHSR (NIH/DDIR)
Sent: Tuesday, December 09, 2014 3:09 PM
To: Taylor, Kyla (NIH/NIEHS) [E]
Subject: RE: Honest broker form request

I am sorry.  I don’t feel like I have enough information to help you and I think our form (which is being revised) is 
confusing.  My questions to you are these: 

What is the purpose of the research activity? In other words what do plan to do with the data? For example if it is for 
quality improvement purposes vs. to publish research data you may not need a determination. 

Can you tell me more about who the research subjects be?  What kinds of data will be collected? How the data be 
collected?  Live, survey, telephone….  If the NIH contractor is a member of the research team then they can’t be an 
honest broker.   

It isn’t necessarily a problem for you to have access to the identities of survey participants.  It depends on what type of 
information you are collecting.  I think it makes more sense for you to submit the form and a copy of the survey 
questions to our main email address and then I can review and we can talk about it more. 

Julie M. Eiserman, MA, CCRP [C] 
Health Science Policy Analyst 
Office of Human Subjects Research Protections 
10 Center Drive, Bldg. 10, Suite 2C146 
Bethesda, MD  20892‐1154 
Office Phone: 301‐402‐3444 
Fax: 301‐402‐3443 
OHSRP website: https://federation.nih.gov/ohsr/nih/index.php (NIH login required) 
Public site: http://ohsr.od.nih.gov/ 

2

From: Taylor, Kyla (NIH/NIEHS) [E]
Sent: Monday, December 08, 2014 10:47 AM
To: OHSR (NIH/DDIR)
Subject: RE: Honest broker form request

Hi Julia, 

I am filling out an OHSRP request form called “OHSRP DETERMINATION FOR SURVEYS, INTERVIEW PROCEDURES, 
PROGRAM EVALUATION, EDUCATION TESTING AND RESEARCH”. We are submitting this to see whether or not we need 
to go through IRB. Our study will have participants, who are all experts in risk assessment, answering questions about 
hypothetical scenarios. It seems that the main issue is whether or not we will have access to identifying information on 
the participants. We don’t need to have access to this information for our study and the request form asks: 

“Who will collect the data or information? If b or c, will an Honest Broker or data use agreement be used?” I’ve answered 
“c” which is an NIH contractor. 
“If yes, complete and attach the Honest Broker Assurance or data‐use agreement to this submission; e‐mail 
ohsr_nih_ddir@od.nih.gov to request a form.” 

I assumed we would need an honest broker to show that we do not need access to identifying information. Is that 
correct? 

Thanks, 
Kyla 
From: OHSR (NIH/DDIR)
Sent: Monday, December 08, 2014 10:38 AM
To: Taylor, Kyla (NIH/NIEHS) [E]
Subject: RE: Honest broker form request

I want to make sure that you understand that these items are not requirements necessarily.  It depends on what you are 
trying to do.   
A data use form is not a requirement when you submit a determination request, rather it is something that you would 
get from tech transfer, esp. if you are sending data out to another site.   

The honest broker assurance form can be found here: https://federation.nih.gov/ohsr/nih/formtmp.php 

An honest broker agreement is only needed in certain circumstances.  Can you clarify what you are trying to do? 

Julie M. Eiserman, MA, CCRP [C] 
Health Science Policy Analyst 
Office of Human Subjects Research Protections 
10 Center Drive, Bldg. 10, Suite 2C146 
Bethesda, MD  20892‐1154 
Office Phone: 301‐402‐3444 
Fax: 301‐402‐3443 
OHSRP website: https://federation.nih.gov/ohsr/nih/index.php (NIH login required) 
Public site: http://ohsr.od.nih.gov/ 

From: Taylor, Kyla (NIH/NIEHS) [E]
Sent: Monday, December 08, 2014 9:13 AM
To: OHSR (NIH/DDIR)
Subject: Honest broker form request

Hello, 
3

I’d like to request an Honest Broker Assurance and data‐use agreement form to submit with my application to OHSRP. 
Thanks, 
Kyla 

Kyla Taylor, MS 
Health Scientist 
Office of Health Assessment and Translation 
National Toxicology Program 
National Institute of Environmental Health Sciences 

4

OHSR (NIH/DDIR)
From:
Sent:
To:
Cc:
Subject:

OHSR (NIH/DDIR)
Wednesday, December 17, 2014 11:34 AM
Thayer, Kristina (NIH/NIEHS) [E]
Taylor, Kyla (NIH/NIEHS) [E]
Req fo Determination Rec'd_OHSRP 12731

Good morning Dr. Thayer, 

This email is to verify that OHSR has received your Request for Determination and it is currently being processed as 
OHSRP #12731. Please use this number in any future correspondence regarding this study.   

Protocol Title: NTP Research Project: Updating Level of Concern Categories 

Thank you. 
Sincerely, 
Chris Brentin 
OHSRP ‐ National Institutes of Health 
Bldg 10, Suite 2C146                                                                                                                           
Bethesda, MD 20892 
Office Telephone: 301‐402‐3444 
Office Fax: 301‐402‐3443 

The NIH is committed to maintaining the highest standards for the protection of human
subjects.  

Please consider the environment before printing this e-mail 

1

Updating Level of Concern
Categories
Kris Thayer, Ph.D., OHAT
Mary Wolfe, Ph.D., OLPR
PRC Meeting, October 31, 2014

Level of Concern (LoC) Project
• Project leaders
– Kris Thayer, OHAT
– Mary Wolfe, OLPR

• Contributors
– Kyla Taylor, OHAT
– Shepherd Schurman, CRU
– Grace Kissling, Biostatistics Branch
– Mike Shelby, retired NIEHS
– David Budescu, Fordham University (technical advisor)
– Thomas Wallsten, University of Maryland (technical advisor)
– Barbara Forsyth, private consultant

Outline
• Background on current LoC framework
• Project to revise LoC framework
• Specific Aim 1 of project
• Next steps: Specific Aims 2 and 3

Level of Concern (LoC)
• System used as a means for NTP to communicate its
interpretation of the potential for the chemical under
evaluation to adversely affect human reproductive
health or children’s development.
• NTP’s conclusions are expressed as “level of concern”
• 5 categories plus 1 category for
“insufficient data”

How are LoC conclusions reached?
Integrates evidence for toxicity + extent of human exposure

+

Separately determined
for human and animal
evidence

Extent of human
exposure and other
factors

=

LoC Conclusions
• LoC conclusions developed for 20 substances including
acrylamide, amphetamines, BPA, bromopropanes, ethylene
glycol, fluoxetine, hydroxyurea, methanol, methylphenidate,
phthalates, propylene glycol, soy infant formula, and styrene
– There is serious concern that certain intensive medical treatments of
male infants may result in di-(2-ethylhexyl) phthalate exposure levels
that adversely affect development of the male reproductive tract.
– There is some concern for effects on the brain, behavior, and prostate
gland in fetuses, infants, and children at current human exposures to
bisphenol A.
– There is negligible concern for adverse developmental and reproductive
effects from acrylamide exposure to the general population.
– There are insufficient data to draw conclusions regarding possible
reproductive effects of methylphenidate in humans.

Why does NTP need to update its LoC categories?
• Confusion over what the different LoC categories mean (e.g.,
“some concern” vs “minimal concern”)
– Need to clarify what LoC categories mean and how NTP reaches a
conclusion

• Is 5 “concern” categories the appropriate number?
• What are the best modalities to use (color, text, numbers, etc.) to
communicate LoC?
• Can we address level of confidence / uncertainty in
communicating LoC conclusion?
• OHAT Approach for systematic review has a new process and
categories to describe hazard conclusions (Rooney et al. 2014)
– Need to address how new hazard conclusions fit within LoC
framework
Rooney et al. Systematic review and evidence integration for literature-based environmental
health assessments. Environ. Health Perspect. 2014 Jul;122(7):711-8. doi:
10.1289/ehp.1307972. Epub 2014 Apr 18. PMID: 24755067.

How are LoC conclusions reached?
Integrates evidence for toxicity + extent of human exposure

+

Separately determined
for human and animal
evidence

Extent of human
exposure and other
factors

=

New OHAT Approach for hazard identification?
Human and animal evidence are integrated with consideration
of “other relevant data”
• Five categories:
– Known to be a Hazard
– Presumed to be a
Hazard
– Suspected to be a
Hazard
– Not Classifiable as a
Hazard
– Not Identified to be
Hazard (high confidence of
no health effect)

What LoC categories should NTP use?
LoC Categories
1998- Present

Known to be a Hazard
Presumed to be a Hazard
Suspected to be a Hazard
Not Classifiable as a Hazard
Not Identified to be a hazard

+

Extent of human
exposure and
other factors

=

New LoC
Categories

LoC Project to Update LoC Categories
Goals
• Address issues with current LoC system
• Integrate new OHAT Approach for reaching hazard
identification conclusions with LoC framework
• Improve transparency of LoC conclusions
• Improve use of LoC as a communication tool

Objectives
• Reassess the number and labels used for LoC categories
and identify visual and/or web-based strategies to enhance
the communicability of LoC conclusions
• Determine whether the revised LoC framework is an
improvement over the current framework in terms of
transparency and use as a communication tool

LoC Project to Update LoC Categories

Specific Aim 1:

Determine the number of and descriptors for LoC
categories

Phase 1:

Engage technical experts to determine the number
of and descriptors for LoC categories

Phase 2:

Pilot test revised LoC categories with technical
experts

Specific Aim 1
LoC Case-Study Scenarios: LoC “Cards”

Hazard identification label: known,
presumed, or suspected
• Level of evidence for animal studies (high, moderate, low, no) & basis
• Level of evidence from human studies (high, moderate, low, no) & basis

Extent of human exposure and other
factors
• Level of exposure for human population group(s)
• Other factors (e.g., pharmacokinetics)

What is LoC for identified population
group?
Obtain feedback on LoC cards from DNTP advisory group (5-6)

Specific Aim 1
Sample LoC Card
Hazard ID label: Suspected hazard for humans
Level of evidence for animals

Moderate
Inhalation exposure of male rats to ≥500
ppm of chemical Y caused reduced testes
weight…

Level of evidence for humans

Low
Some female workers reported alterations in
menstrual cycle length.

Exposure to human population: 0.01 – 0.035 ppm in worker breathing zones
Other factors:

Chemical “positive” for estrogen receptor
agonist activity in HTS assays

What is your LoC for female workers of childbearing age?

Specific Aim 1
Technical Experts
• Recruit experts in toxicology, epidemiology, or risk
assessment through targeted advertisements
• Inclusion criteria for experts
– MS, PhD, MD or equivalent and at least 3 years’ documented
experience working in public health or on a risk assessmentrelated activity
– 5 sectors: federal and state government, academia, industry,
and nongovernment organizations
– Informed consent

Specific Aim 1
Phase 1: Engage technical experts to determine
optimal number of and descriptors for LoC categories

Trial 1

• Train 100 experts to use web-based sorting tool
• Randomly order 30 cards in web-based tool
• Experts independently sort cards into ≤15 LoC categories

Trial 2

• Randomly order same 30 cards in web-based tool
• Experts repeat card sorting into ≤15 LoC categories, 3-5 weeks after 1st
session
• Experts provide descriptive labels for their LoC categories

Focus
Group

• 10-15 experts from 5 sectors from Phase 1
• Experts provide input on features of LoC scenarios (e.g., margin of
exposure, subpopulation, confidence in level of evidence for health
effect) that best predict category

Phase 1
Collect data and conduct analyses to
• Determine most commonly selected number of LoC categories
(mean, median, mode)
• Determine reliability of experts’ card placements in trials 1 and 2
• Determine consistency in LoC categorization among technical
experts from different sectors
• Determine if certain features of LoC scenarios best predict the
category

Outcomes from Phase 1
• Revised LoC system with X categories
• Descriptors for each category

Specific Aim 1
Phase 2: Pilot test revised LoC categories with
technical experts

Experts

• Recruit ≤100 experts (60% new, 40% repeats)
• Repeat experts are from Phase 1, trial 2 who (1) chose X categories, (2)
chose X-1 categories, or (3) X+1 categories
• Train new experts to use web-based sorting tool

Trial 2

• Randomly order 30 cards in web-based tool
• Experts place cards into current 5-level LoC categories and revised X-level
LoC categories in cross-over design; experts randomly assigned
• Experts rate their confidence in card placements into revised and new LoC
frameworks on 1-7 scale (1 “not confident” to 7 “highly confident”)

Focus
Group

• Focus group from Phase 1
• Experts provide input on features of LoC scenarios that best predict
category for revised LoC framework

Phase 2
Collect data and conduct analyses to
• Compare experts’ categorization within current and revised LoC
frameworks (both number of categories and descriptors)
• Determine consistency in LoC categorization among technical experts
from different sectors
• Determine if certain features of LoC scenarios best predict the category
• Determine consistency in ratings of experts’ confidence for
categorization between two LoC frameworks (current and revised)

Outcomes from Phase 2
• Understanding of whether revised LoC framework is better than current
LoC system
• An updated LoC framework for further development as a communication
tool and for broader NTP stakeholder and public input

LoC Project to Update LoC Categories
Specific Aim 1:

Determine the number of and descriptors for LoC
categories.

Phase 1:

Engage technical experts to determine the number
of and descriptors for LoC categories

Phase 2

Pilot test revised LoC categories with technical
experts

Updated LoC categories and descriptors

Specific Aim 2:

Develop a variety of visual schemes using
multiple modalities (e.g., text, color, numbers,
interactive web graphics, etc.) to communicate
LoC conclusions.

Specific Aim 3:

Utilize focus groups of NTP stakeholders and
the general public to obtain feedback on the
communication tools to refine them.

Questions?

Hazard Identification Categories
Hazard Categories
1998- 2014

New Hazard
Categories

Known Hazard
Presumed Hazard
Suspected Hazard
Not Classifiable

Specific Aim 1

Phase 1
100 technical experts from five
sectors

Phase 2
≤100 technical experts from five
sectors
(60% new, 40% Phase 1)

Determine optimal number of
LoC categories

Pilot test revised LoC categories
vs current LoC categories

Determine consistency in LoC
categorization among technical
experts from different sectors

Determine consistency in LoC
categorization among technical
experts from different sectors

Identify the features of LoC
scenarios (“cards”) that best
predict the category

Identify the features of LoC
scenarios (“cards”) that best
predict the category

LoC Categories
(1998 – current)

New
LoC Categories

NTP’s Current Level of Concern (LoC) Scale
• 5 categories plus 1
category for
“insufficient data”