Form 1 Data/Sci Moderator Guide

NCI Data Science Usability Sessions – Moderator Guide

OMB No.: 0925-0776

Expiration Date: 04/30/2023

Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch,6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA(0925- 0766). Do not return the completed form to this address.

Session Information

Date
Time
Moderator
Participant info
Additional info

Introduction

Welcome & Purpose

• Hi. I'm [Name]. I am a user researcher with NCI. How are you today?

• Thank you so much for joining us this [morning/afternoon].

• During this session, we hope to learn more about your work in cancer data science. We’ll start with questions to help us better get to know you, your role, the work you do with data science, your information needs to accomplish that work, and how you want/expect to access content and resources.

• In the latter half of the exercise, we will visit NCI’s data science website, datascience.cancer.gov, to learn more about your preferences for finding and viewing meaningful content that supports your work.

• Your feedback is critical to helping us to create data science content and resources for you and the cancer research community.

Your Role

• Your job for our session today is to let us know what you like, don't like, or anything that you find to be confusing.

• There are no right or wrong answers. Please let me know if you have questions, comments, or areas of confusion during our session.

• Remember, we are testing/evaluating our site, NOT YOU! Your thought process is important so try to think out loud as much as possible – tell me what you see, what you think things will do, or if something doesn't react the way you would expect.

• Also, don’t worry about giving negative feedback. We just want to ensure that we are moving in the right direction and your feedback helps us to make our site better.

• If you do not want to answer a question, you do not have to. Please just say so, and we will move on. And, of course, you can stop at any time.

• Before we start, I want to share more information about how your responses will be used and confirm we may continue.

Consent

Can you confirm that you were able to send in your signed consent form?

• If yes, “do you consent to continue with this session today?”

• If no, read consent out loud and receive verbal consent.

Project Title	NCI Data Science Web User Research
What is the purpose of this study?	This research study is being conducted by the National Cancer Institute. In this study, we want to learn more about our target audience’s (mid-late cancer researchers, mid-late data scientists, early career investigators) digital profile: who they are; what data science content they read; how they find information, and what information they find useful.
How many people are participating?	A maximum of 9 people will be in this part of the study.
What will happen during this study?	If you agree to be in this study, you will be asked about your role, the work you do, how you use data science content, and questions related to datascience.cancer.gov (its content, its applicability/use/value to you in your role). The total study will last for about 60 minutes.
What are the potential risks and discomforts of participating in this study?	This study does not include any medical procedures and has minimal risk. Visiting websites for extended periods of time may cause eye strain.
What are the potential benefits of participating in this study?	By sharing your perspective, you will help guide us to offering future data science content to help you.
How will my information be protected and kept confidential?	During the session, your identity will only available to the team members directly responsible for running and taking notes during your interview. You will not be asked to share your screen. You will not be asked to share any sensitive information about your work. You may decline to respond to any additional question you do not wish to answer. After the session, your responses will be anonymized, and your contact information will be removed from our records.
How will I be compensated for my time?	No compensation
Do I have the right to withdraw from the study?	Your participation in this research is completely voluntary. You may stop participating at any time.
To whom can I ask questions?	You may contact CBIIT Communications.

Moderator's Role

• I will take notes as we go, so you may see me occasionally look down. I am here to ask you questions and observe your reactions and comments as you interact with our site.

• We will also have a few of my teammates listening to this session. They will be helping me take notes.

• Additionally, we would like to ask for your permission to record this session for note-taking purposes. We’ll be capturing your voice and what you are doing on the screen. These recordings will not be seen by anyone except the people working on this project. Your name will be kept confidential. Do you give your permission for the recording to begin?

• [START SESSION RECORDING & TRANSCRIPTION]

Background Questions/Identifying Personas (15 mins)

To start, I want to learn more about you and your work in data science.

1. Tell me a bit about your role and experience and how you use data science in your daily work.

2. What data techniques do you use? Do you use them regularly?

3. How do you define cancer data science?

4. What are some of the pain points you experience in your work?

5. What are the most important cancer data science resources you need to support the work you do?

6. What are your go-to places/sources to get information for your work?

   1. What do you like about these resources?

   2. What do you find useful about these resources?

   3. What is your format preference for getting/accessing information? Ex: email, websites,

   4. social, podcasts, videos

      1. If they don’t mention social sites: ask, do you use any social media sites like Twitter to find information?

7. What is a data science gap in your work? What can’t you do/find and need/want to?

Website Navigation Journey (20-25 minutes)

For the next part of our session, we will explore the resources on datascience.cancer.gov. I am going to share my screen and give you control of my mouse. For now, please do not click on anything, but feel free to scroll and move around the landing page.

Site Navigation Task:

• Without hovering over or clicking, is there any information missing that you would want to see on this page (homepage)?

• Let’s scroll up to the top. Without hovering over or clicking the menu items, tell me what kind of information you would expect to find under Data Sharing?

  • How helpful would this information be in your work?

  • Go ahead and hover over. Is this the content you expected to see?

  • If familiar with the site:

    • Do you use this content?

    • Do you share/refer this content?

    • Is there anything missing?

    • How would you organize this content?

  • If not familiar with the site:

    • What are your first impressions with this content?

    • How would you use this content for your work?

    • Is there anything missing?

    • How would you organize this content?

• Without hovering over or clicking, tell what kind of information you would expect to find under Data Commons?

• How helpful would this information be in your work?

• Go ahead and hover over. Is this the content you expected to see?

• If familiar with the site:

  • Do these page names resonate with you? Do you know what you’d find on these pages?

  • Would you expect to find data sets here?

  • Do you use this content (data/tools)?

  • Is there anything missing?

• If not familiar with the site:

  • What are your first impressions with this content?

  • What information would you expect to find here?

  • Would you expect to find data sets here?

  • Do you use/look for data sets?

  • Do you use/look for tools?

  • Is there anything missing?

• Without hovering over or clicking, tell me what kind of content you would expect to find under Collaborations?

  • How helpful would this information be in your work?

  • Go ahead and hover over. Is this the content you expected to see in Collaborations?

    • Do these menu choices give you an idea of the type of content you will find?

    • What information/details about a collaboration would you expect to find on these pages?

    • Are you interested in data science initiatives at NCI?

      • If yes, what would you want to know about?

    • Are you interested in learning about the people who are involved in data science at NCI? If yes, what do you want to know and why?

• Without hovering over or clicking, tell me what kind of content you would expect to find under Resources?

  • How helpful would this information be in your work?

  • Go ahead and hover over. Is this the content you expected to see?

    • Do you use these resources?

    • Are there any resources missing that you would expect here?

Do they mention training? If not, ask: would you be interested in training resources? What would you expect from training resources?

OPTIONAL (If time permits)

• Click on NCI Data Catalog. Is this what you expected?

  • Is this layout easy to read?

  • Do you know what to do with this content? Would you use it in your work?

  • Is there content missing from this page?

• Without hovering over or clicking, tell me what kind of content you would expect to find under Funding?

  • Do you ever look for funding opportunities to support your work?

    • If yes, how do you typically search for opportunities?

  • Click on Funding. Is this what you expected?

    • Are there other ways you want to search for funding opportunities?

    • Are you interested in how much NCI spends on data science activities?

    • Are you interested in seeing all the NCI/NIH data science-related grants?

      • Do you care which group at NCI is supporting the work?

      • Would you prefer to research funding opportunities on a specific DOC site or collectively on a data science hub?

    • Would you refer this content to others?

• Without hovering over or clicking, what kind of content you would expect to find under About?

  • Go ahead and hover over. Is this the type of content you expected to see?

  • Are you interested in knowing which of the many groups across NCI are responsible for the various data science initiatives and tools?

    • If yes: What, if anything, would you want to know about those groups?

    • If no: Next questoin

  • Are you interested in who is doing the work, where they are doing the work, or what work are they doing?

• Let’s go back to the homepage. If we could make a custom homepage for you, what categories of information would you readily want available? Would you be interested in a homepage that lets you select an option to navigate to a page of resources just for the work you do as a [cancer researcher OR data scientist]?

• Would you want a one-stop site for all the NCI data science tools and resources you need?

Content Tactic (15 minutes)

• For this next set of questions, we want to share one of our blogs. Can you find the Blog section on the site?

  • Remember to share your thoughts as you click through; comment on anything that stands out.

  • Go ahead and click on any of the blog posts that you are interested in.

• [Ask content and format related questions depending on responses]

  • Do you read data science related blog posts?

  • What blog topics would you want to read?

  • How would you like to receive/be notified about new blog posts?

  • What are your initial thoughts about the layout of this blog article?

  • How about the length of the blog posts?

  • Do you/would use search for blog posts by categories?

  • Would you share this blog? If so, how?

  • If there is an image, is it helpful?

  • If no image, would you expect some?

• We are interested in developing a chatbot to help visitors like you quickly find or answer questions related to cancer data science.

  • If I could make a personal chatbot to assist you, what kind of questions would you want it to answer?

  • Do you have preferences for how that chatbot should respond?

  • Are there chatbots you use that you find helpful?

  • Are there things about a chatbot that deter you from using it?

  • Would you use this chatbot when searching information?

OPTIONAL (If user identifies as early career investigators)

• As someone early in their career, we want to create content that is meaningful to your career journey.

  • What resources do you use to get information and answers to your questions? (ex: forums/collaboration sites)

  • Where do you go for information about career opportunities, people (i.e., mentors)?

  • (We ask about training above; not sure if we need/want to add more training questions in this section)

    • Are you looking for data science training opportunities?

    • What training formats appeal to you?

    • What training topics are you interested in?

Now, I’m going to take back control of my screen to share our last activity with you.

Post Study Questions (5 minutes)

The following questions will be about your experience with datascience.cancer.gov in our session today. Your candid feedback will help us to improve our site experience.

Rate your agreement with the following statements:

The datascience.cancer.gov website is an authoritative source on cancer data science content.

1	2	3	4	5	6	7
Strongly Disagree	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Agree	Strongly Agree

The content and resources of datascience.cancer.gov meet my needs.

1	2	3	4	5	6	7
Strongly Disagree	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Agree	Strongly Agree

I would refer the website to my colleagues and professional network.

1	2	3	4	5	6	7
Strongly Disagree	Disagree	Somewhat Disagree	Neither Agree nor Disagree	Somewhat Agree	Agree	Strongly Agree

Select three words from the list shown here that you would use to describe datascience.cancer.gov. Are there any other words that you would use to describe your experience that are not on this list?

Approachable	Professional	High-quality
Useful	Dated	Familiar
Comprehensive	Frustrating	Meaningful
Boring	Overwhelming	Confusing
Trustworthy	Friendly	Innovative
Hard to understand	Intimidating	Accessible
Complex

• Is there any additional functionality or information that you would want/expect to find on datascience.cancer.gov?

  • If something has been mentioned that does not exist on the website, what would you call it?

  • Where would you look for it on the site?

• Do you have any other suggestions or recommendations for us to improve datascience.cancer.gov?

• Now that you have interacted with this site, I am going to circle back to an earlier question, who do you think the audience is for datascience.cancer.gov? Is this site for you?

End Session

Thank you so much for your time and feedback today.

[END RECORDING]

0