Form 1 Moderator Guide

OMB Control Number: 0925-0648 

Exp. Date: -06/30/2024

National Institutes of Health (NIH) Office of Management (OM) Public Website Design Contract – UX Testing Moderator Guide Template

Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a current valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0648). Do not return the completed form to this address. _________________________

Section 1: Introduction (5 minutes)

Thank you for participating in this interview today. My name is _________ and I am a researcher with Fors Marsh Group, a private research company. In this interview, I will show you a website for the National Institutes of Health (NIH) Office of Management (OM) and ask you to explore the website as you normally would. Then we will discuss your feedback, thoughts, and overall experience with using the websites.

Your comments and feedback will help us to improve the designs for these sites. I did not create any of the websites or materials you’ll be reviewing, so I invite you to openly share your thoughts and ideas with me. I’m interested in what you like and what you don’t like about the websites. Any trouble you may run into reflect issues with the design of the websites, not your skills or abilities. Please keep in mind this is not a test of you or your knowledge. You are helping us to see how we can improve the websites. The entire session will last about 45 minutes. Do you have any questions so far?

Before we begin, I’ll just cover a few things:

• Everything you share today will be kept secure to the extent permitted law, so your name and any identifying information will not be used in any of our reports.

• Your participation is voluntary, and you have the right to end the interview at any time. If I ask any questions that you do not wish to answer, you do not have to answer them.

• We are making a video recording of the computer screen during this session, but your name and personal information will not be put together with the recordings or your responses. Other project staff may listen to the recording at a later date.

• I am interested in your thoughts and reactions as we proceed. While you are looking through the websites, I would like you to think aloud. Please describe aloud the steps you would take and what you’re thinking about as you are reviewing the pages.

• There are no right or wrong answers, and your comments and opinions will only be used together with the feedback that we get from other people.

• You might notice there are some other people on the line. They are simply here to help me by taking notes so that I can focus on our conversation.

• Do you have any questions before we get started?

Alright, we are ready to begin. Do you consent to me beginning the recording?

[Begin Recording]

Section 2: Exploratory (5 minutes)

Before we pull up the webpages we will be looking at today, I’d like to ask a few questions about your experience with the site.

1. Tell me a little bit about your role related to NIH Office of Management.

   1. What do you do with or for them?

   2. How long have you been in that role?

2. Have you used the [name of website] website before?

[If yes]

3. What have you used it for?

4. What pages do you typically visit?

5. How often do you use it?

6. What has your experience been like using it in the past?

   1. What obstacles, if any, have you run into while using the website?

   2. What did you like about the website?

[If no]

7. What, if anything, do you know about the purpose of this website?

8. Under what circumstances would you visit this website? What makes you say that?

9. What information would you expect to find on this website?

Section 3: Initial impressions (5 minutes)

Now I’ll have you pull up this [name of website] website. I’ll send the link in the chat. Take a minute to look over this page and let me know when you’re finished.

10. What, if anything, catches your eye on this page?

11. How well does this website match your original expectations of what you’d find here, if at all?

12. [IF YES ABOVE] What do you think is the purpose of this website?

13. What do you think you can do on this website?

14. Who would you say is the intended audience for this website?

Section 4: Natural Use (10 minutes)

Next I want to learn more about how you typically use this website.

15. Before clicking anywhere, where would you want to click first?

1. [PROBE]: What would make you want to click there first?

Next, I’ll have you show me how you [might typically use the website/use this website for the first time]. As you show me how you would use the website, I would like to ask you to think out loud. This will help me follow along and understand why you’re doing what you’re doing. Go ahead and click on [wherever they just referenced] to show me what you would [normally] do.

16. What else, if anything, [do you/would you want to] do on this website? Go ahead and show me that.

17. What makes you go there?

As needed:

• What made you click there?

• What information are you seeking?

• To what extent was this information what you expected to find here?

• What are you thinking about right now?

Section 5: IDEA Compliance (10 minutes)

Now I’d like to talk about some specific aspects of the website.

Consistent

18. What thoughts, if any, do you have about the look and feel of this website? (E.g., font, imagery, colors, navigation)

19. Thinking across the website, how consistent do you find the look and feel?

1. [PROBE]: What makes you say that?

Authoritative

20. How confident do you feel about the accuracy of the information on this website?

21. What on the website makes you trust or distrust the accuracy of the information?

22. Are there other websites you might use to look for this information instead? What makes you say that?

Design, UX

23. How easy or difficult is it for you to find the information you need on this website? What makes you say that?

24. How easy or difficult is it for you to navigate this website? What makes you say that?

Search

25. Under what circumstances, if any, do you use the search bar on this website?

26. What makes you use the search bar rather than the navigation within the website?

Mobile-friendly

27. Have you ever tried to access this website on your smartphone?

1. [IF YES], What was that experience like?

   1. Were you able to access all of the pages and content you needed?

Section 6: Debrief (5 minutes)

Now that you have finished reviewing the website, I’d like to talk about your overall experience.

28. Tell me about your overall experience with exploring this website today.

1. If you had to pick one word to describe it, what word would you choose?

29. What improvements, if any, can be made to make this website more usable for you?

    1. How would these changes make this website more useful for you?

    2. If implemented, how would these changes impact your usage of this website, if at all?

    3. How would these changes impact your work and/or work-related responsibilities?

30. If you had questions while using of the website, what would you do?

31. Was there anything that you expected to see, or expected to have access to, but didn’t?

Section 7: Conclusion & SUS (5 minutes)

Those are all the questions I have for you today, but I do have one final task I’d like to ask of you. I’d like for you to please complete this short 10-question survey. Your participant ID will be [provide ID]. Please click on the link in the chat box to complete the Usability Scale. [Participant will click on a link that takes them to a Google form to fill out. LINK] For each statement, choose the numbers that most appropriately reflect your impressions about using this website. 

Before we conclude, is there anything you would like to share that you have not had the chance to?

I think that I would like to use the [fill] website frequently: 

Strongly  Disagree	Disagree	Neutral	Agree	Strongly  Agree
	2		4	5

I found the [fill] website unnecessarily complex: 

Strongly  Disagree	Disagree	Neutral	Agree	Strongly  Agree
1			4	5

I thought the [fill] website was easy to use: 

Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
1	2	3	4	5

I think that I would need the support of a technical person to be able to use the [fill] website: 

Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
1	2	3	4	5

I found the various functions in the [fill] website were well integrated with each other: 

Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
1	2	3	4	5

I thought there was too much inconsistency in the [fill] website: 

Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
1	2	3	4	5

I would imagine that most people would learn to use the [fill] website very quickly: 

Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
1	2	3	4	5

I found the [fill] website very awkward to use: 

Strongly Disagree	Disagree	Neutral	Agree	Strongly Agree
1	2	3	4	5

I felt very confident using the [fill] website: 

Strongly  Disagree	Disagree	Neutral	Agree	Strongly  Agree
	2	3	4	5

I needed to learn a lot of things before I could get going with the [fill] website: 

Strongly  Disagree	Disagree	Neutral	Agree	Strongly  Agree
1	2	3	4	5

Do you have any questions at all for me?

Thank you very much for participating in this interview. I appreciate your time and your feedback has been extremely helpful.

ARLINGTON, VA | (571) 858-3800 | FORSMARSHGROUP.COM

1