Round-3 Protocol

Appendix O3: Round 3 Protocol

Note: Email P a reminder one day before session, along with the Consent Form and all study web links.

Thank you for agreeing to participate in this study. My name is XX and I am a member of the study team and I’ll be working with you today. The purpose of this study is to test the usability of an online survey. This is a national survey of food that Americans get. The survey will help us better understand what kind of food Americans get and how they get it, and will help make policies for all Americans to get affordable and healthy food. The survey is to be conducted by the Census Bureau on behalf of U.S. Department of Agriculture.

In today’s session, we will evaluate how easy or difficult it is to use the food survey. Your experiences will help us improve the food survey before it is used by a larger group of people. Today, you will first watch three short videos that teach you how to use the survey to report foods and drinks you get. Then, we have several made-up scenarios where you are asked to pretend that you got some food items from a store and need to report the in the food survey. If you already went food shopping today, we are not asking about that shopping trip. Instead we are asking you to report the food items in the made-up scenarios. After you are done with these scenarios, we will ask you a few questions about your experience with the food survey when completing these tasks. Upon completion of today’s session, you will receive $100 for your participation. Do you have any questions before we move on?

<Answer participants’ questions if any>

Great. Now, I would like you to read this Consent Form. It explains the purpose of this study and your rights as a participant. It also informs you that we would like to take an audio-and-video recording of our meetings during the study. The purpose for the recording is to get an accurate record of your performance and feedback so that we can best improve the web instrument. Only those of us who are involved in this study will have access to the recording and it will be strictly used for research purposes. Your name will not be associated with the recording or any of the other data collected during the study period.

<Consent Form Link: TBD >

<Wait until the participant completes reading the Consent Form>

Do you have any questions about the Consent Form?

< Answer participants’ questions if any>

Great, now I would like you to sign the Consent Form if you agree with it.

<Wait until the Consent Forms are sighed>

Thank you. Now I would like you to complete this demographic questionnaire.

<DemoQ Link: TBD >

<Wait until the participant completes the demographic questionnaire>

Thank you.

Before we start the study, I’d like to teach you a method called “Think Aloud.”

I want you to complete the tasks as if you are alone by yourself, but with one major difference. I would like you to “think aloud” as you are performing the tasks: Speak out aloud everything that you are thinking and feeling as you go about working with the online survey. I am interested in the process you go through in your mind when you are using the online survey. Let me show you how I would think aloud: For example, if I were to think aloud while I’m grocery shopping, it might sound like this: “I start my shopping in the produce section because that’s the first section I encounter. I check my list to see what I need. After I found the vegetable I want to buy, I move to the deli counter because that’s next to the produce section.”

Let’s do a practice: Please think aloud as you answer this question: How many windows are there in your home?

<Practice Think Aloud>

[HINT: Probe if they fall silent. If they just give a number ask them to do it again but this time so that you can understand how they came up with the number. Probes should be kept to a minimum – trying to not interfere with the timing data, so used sparingly: Keep talking…; Um-hum?; What are you thinking?; Tell me more…]

<Complete Think Aloud practice>

Great. That’s what I want you to do throughout this session. I will remind you to think aloud if you get quiet.

< P joins MS Teams>

I’d like you to join the Teams so that you can watch the training videos.

<Assist the P to join the Teams>

<Put the link to the first video (Video 4) in the chat >

<Link to Video 4: TBD >

Now I’d like you to watch three short videos and learn how to use the survey.

Please tap the link in the chat and watch the first video.

<Wait until P completes watching the first video, then put the link to the 2^nd video (Video 5) in the chat>

<Link to Video 5: TBD >

Thank you. Now please tap the 2nd link in the chat and watch the 2nd video.

<Wait until P completes watching the 2nd video, then put the link to the 3rd video (Video 6) in the chat>

<Link to Video 6: TBD >

Thank you. Now please tap the 3rd link in the chat and watch the 3rd video.

<Wait until P completes watching the 3rd video, then put the link to the Training Quiz in the chat>

<Link to the Training Quiz: TBD >

Thank you. Now I’d like you to tap the latest link in the chat and take a quiz on what you have learnt from the videos. Please keep thinking aloud while you are taking the quiz.

<Prompt the P to think aloud during the quiz if P falls into silent>

<Wait until P completes the quiz, set up smartphone screen sharing, if P doesn’t have MS Teams installed, help install it >

In order to observe how you use the survey instrument, I’d like you to share your phone screen on the MS Teams.

<Assist the P to set up screensharing if needed>

<Wait until seeing participant’s smartphone screen >

Now we will start recording our session.

<Start screen recording>

<Put the link to the survey instrument in the chat>

<Link to the survey instrument: TBD >

Great. Now we are ready to start the food reporting tasks. Just to remind you that, there are made-up scenarios designed to get your feedback on the online food survey, but not grocery shopping trips in your real life. If you run into any difficulties as you work with the survey, please don’t blame yourself. Any difficulties are the result of survey itself, not your skills or abilities. Please open the chat and tap the link I just put there. Let me know when you see the survey instrument on the screen.

<Wait until the P has the survey instrument on screen>

<Start the 1^st FAH use case>

Thank you. Now I’d like you to perform a food reporting task. Please read aloud the task instructions.

<Hand P the 1^st FAH Use Case (supermarket)>

<After P completes reading aloud the use case, put the grocery bag on the table>

Here are the groceries you just got, I’d like you to report them now. Keep Thinking Aloud.

<Observe P’s performance >

<Wait until the P completes the task>

<Start the 2nd FAH use case>

Thank you. Now I’d like you to do another task. Please read aloud the task instructions.

<Hand P the 2nd FAH Use Case (drug store purchase)>

<After P completes reading aloud the use case, put the shopping bag on the table>

Here are the things you purchased at [STORE]. I’d like you to report them now. Keep Thinking Aloud.

<Observe P’s performance >

<Wait until the P completes the task>

<Start FAFH use case>

Thank you. Now let’s do another task. Please read aloud the task instructions.

<Hand P the FAFH Use Case >

<Email P the receipt while P is reading the use case>

<After P completes reading the use case, show P the food items PPT on a laptop or on paper>

Here are the foods you remember ordered, and you have also received an electronic receipt in your email. The receipt also has some information you need to report. I’d like you to report what you ordered now. Keep Thinking Aloud.

<Observe P’s performance of FAFH Use Case >

<Wait until the P completes FAH Use Case>

<Once P completes all tasks, put the link to the satisfaction questionnaire in the chat>

Great job. Now I’d like you to tap the link I’ve just put in the Chat and answer that survey as well.

< Link Satisfaction: TBD >

<Wait until the P completes the satisfaction questionnaire>

Thank you. Now I’d like to ask you a few questions about your experience with completing these tasks:

<Conduct an ad-hoc debriefing on all tasks based on observation, also probe on extreme responses to the satisfaction questionnaire>

<Put the link to the debriefing questionnaire in the chat>

< Link debriefing questionnaire: TBD >

Here is the last questionnaire I’d like you to complete. Please tap the link in the chat and start to answer the questions. Please Think-Aloud while you are complete the questionnaire.

<Wait until P completes the debriefing questionnaire>

Thank you. You have completed this session.

<Stop screen recording>

<Put the link to the incentive voucher in the Chat>

< Voucher Link: TBD >

Now, I’d like you to sign the incentive voucher. Please open the Chat and tap the link to the incentive voucher, fill in all the information, and sign.

<Wait for the incentive voucher to be signed, verify all information being correct and complete – Full name and full address, then give the incentive cash to PR in person>

Thank you again for your participation!

TA Notes:

Timing on Individual Cases

After 15 minutes for the grocery FAH use case; After 4 minutes for the drug-store FAH use case; After 8 minutes for the FAFH use case – Ask Participant:

Do you feel like you are close to being done reporting for this scenario?

If they say that they are far from being done, you can move them on, say:

Due to time we will move on to the next scenario.

If they say they are close – allow them to work for two additional minutes.

After 18 minutes for the grocery FAH use case; After 5 minutes for the drug-store FAH use case; After 10 minutes for the FAFH use case;– if participant is not close to an answer, say:

Thank you for your work on this scenario. Due to time constraints, I would like to move us onto the next scenario.

7