NAEP Grade 4 Writing Computer-Based Assessment Usability Study, &The Assessment and Teaching of 21st Century Skills (ATC21S) Project Phase 2.2 Cognitive Interviews

Appendix C

Phase
2.2
Cognitive
Lab
“Think
Aloud”
Protocol


Cognitive
Lab
Observation
Notes


ATC21S Cog Lab Think-Aloud Protocol

Page 1 of 5

ATC21S Cognitive Lab “Think Aloud” Protocol
PREPARATION
Checklist:
❏
❏
❏
❏
❏
❏

Parent consent letter signed.
Think aloud record form with unique student ID.
Computer with touchpad and optional mouse.
Voice and screen recording software.
Internet access.
Two different colored pens.

Remember:
•

Sit behind the student but not in their personal space.

•

If the student is silent for more than 5 seconds, say “Keep talking.”

•

If the student is talking too quietly, say “Please speak louder.”

•

If the student is not reading the questions aloud, ask him/her to do so: “Please read the
questions out loud.”

•

Be sure that the student is actually typing her/his responses, not just talking: “Please enter
your response.” If the student is having trouble entering responses and talking
simultaneously, have the student talk first, then enter her/his responses.

•

If the student asks you what to do because s/he does not understand a question, tell her/him:
“Do whatever you normally do when you take a test.” You should not help them solve the
problem.

•

Be attentive with body language by head nodding and occasional, “um hm’s” in response to
students.

•

DO NOT tell the student if s/he got an answer right or wrong.

•

DO NOT tell the student if s/he he did well/poorly on the activity.

•

DO NOT show bias for certain questions or item formats (e.g., Do not say anything like,
“This is not a very good problem.” OR “Problems like these don’t test many 21st century
skills.”)

ATC21S Cog Lab Think-Aloud Protocol

Page 2 of 5

INTRODUCTION – Read this to students
“We asked for your help today because we want to know what you think about when you work
on work on an assessment of information and communication technology skills. In order to do
this I am going to ask you to THINK ALOUD as you work on the different problems. What I
mean by think aloud is that I want you to tell me EVERYTHING you are thinking from the time
you first see the screen until you click on the NEXT button. I would like you to talk aloud
CONSTANTLY from the time I present the assessment until you finish. I don’t want you to try
to plan out what you say or to try to explain to me what you are saying. Just act as if you are
alone in the room speaking to yourself. It is most important that you keep talking. If you are
silent for any long period of time I will ask you to talk.
I will sit over here, but I cannot help you during the assessment. If you stop talking at any point,
I’ll remind you to ‘Keep talking.’ If your voice becomes too quiet for the microphone, I’ll ask
you to ‘Please speak louder.’ If you feel uncomfortable or don’t wish to continue, you can stop
the assessment at any time.
Do you understand what I want you to do?
First I want you to practice talking out loud. I want you to say everything you are thinking as you
play an online game. The online game that I want you to play is similar to the kind of experience
you will have in the assessment. We will use the game to practice thinking out loud. Remember
to say out loud everything that you might say to yourself silently and to talk CONSTANTLY as
you play the game. You can start now.”
[Link to game for the think aloud practice:
http://pbskids.org/dragonflytv/games/game_planetary.html
“Your goal is to launch needed parts to the target. Change the angle, amount of force and
launch.”
After a couple of minutes playing the game and talking out loud, give the student feedback. Tell
the student if he needs to speak more. You may need to model thinking aloud in order to help the
student understand what to do.

ATC21S Cog Lab Think-Aloud Protocol

Page 3 of 5

THINK ALOUD INTERVIEW
(Appendix C Observation Notes will be used in conjunction with this part of the protocol)
[Turn on voice and screen recorder.]
Prompts:
“Please keep talking.”
“Please speak louder.”
Do not ask the student any other questions or prompt the student in any other way.
What to do:
1. Fill out the think aloud record form. Listen carefully to what the student is saying.
a. ICT skills targets
i. Indicate Yes or No, Did you see evidence of student using specified ICT
skill?
ii. Briefly summarize evidence that led to your decision.
b. Usability issues
i. Indicate yes or no, Did you see evidence that the student had difficulty
using the technology or interface?
ii. Briefly summarize the usability issues encountered.
c. Screen Specific questions
i. Indicate Yes or No whether specific behavior for screen was witnessed?
(e.g., Did student make tags in the concept map for Webspiration?)
d. Comments
i. Summarize any issues, student comments or behaviors not mentioned
above. Did students do something unexpected? Express delight or
frustration?
ii. Keep a record of when you did not understand what the student was doing
or on screens where the student seemed to struggle. When the student has
completed the scenario, you will have the opportunity to ask the student
about these things.

ATC21S Cog Lab Think-Aloud Protocol

Page 4 of 5

FOLLOW-UP INTERVIEW
[Use different colored pen to record student responses to follow-up interview questions.]
Interview Questions
Log in to the assessment site.
Review of Problematic Questions/Issues
Tell the student we are going to look at a few of the screens from the module so that we can
improve the ICT Literacy assessment. Use your notes to choose the screens where you did not
understand what the student was doing or the screens where the student seemed to struggle.
•

Some prompts: What do you think this task is asking you to do? How could we make the
instructions more clear? How could we make the wording more clear?

Overall Impressions – last page of coding documents
1. Have you ever taken an online test like this before? Yes/No
o Was it similar to this assessment? How were the questions different?
2. On a scale of 1 to 4, where 1 is very bad and 4 very good, how would you rate this
assessment? Indicate 1-4
o What factors contributed to your score?
3. Did you feel like you knew what you were supposed to do to complete all of the tasks?
Yes/No
o Please explain your response
4. Do you think this assessment allows you to show you skills in using information and
communications technology? Yes/No
o Explain why or why not.
WRAP-UP
[Stop the audio recorder.]
At the completion of the interview, thank the student for her/his participation: “Thank you very
much, _____________. That’s it for today.”

ATC21S Cog Lab Think-Aloud Protocol

Page 5 of 5

APPENDIX C - Observation Notes for:
• Webspiration
• Second Language Chat

ATC21S Appendix C

Webspiration

Task ID: 42
Tasks:
☐ Open to first screen
☐ Locate menu and navigation buttons to previous page and next page

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 2

Webspiration

Task ID: 165
Tasks:
☐ Find link to read poem text

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 3

Webspiration

Task ID: 132
Tasks:
☐ Navigate to video collection (view Podcast for help)
☐ Find Poem video
☐ Note what speaker remembers in box provided
Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 4

Webspiration

Task ID: 45
Tasks:
☐ Use screen tools to organize concept map
Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 5

Webspiration

Task ID: 46
Tasks:
☐ Type 2 questions in boxes provided

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 6

Webspiration

Tasks:
☐ Rank 5 questions
*Questions stated differentiated by age

Task ID: 166

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 7

Webspiration

Task ID: 48
Tasks:
☐ Evaluate poem using dropdown menus

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 8

Webspiration

Task ID: 167
Tasks:
☐ Note login and password
☐ Go to Webspiration Graffiti Wall
*Graffiti wall urls, logins, and passwords differentiated by team
Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 9

Webspiration

Task ID: 133

Tasks:
☐ Upload Graffiti Wall following instructions

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 10

Webspiration

Task ID: 53
Tasks:
☐ Use screen tools to update poem organizer
Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 11

Webspiration

Task ID: 168
Tasks:
☐ Sort card statements by category
*Card text is differentiated by age
Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 12

Webspiration

Task ID: 43
Tasks:
☐ Choose a poem from favoritepoem.org
☐ Make a 3-minute movie using Screentoaster (follow Podcast help)
☐ Submit Screentoaster link
Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 13

Webspiration

Task ID: 56

Tasks:
☐ Evaluate self in collaborative role
☐ Check preferences
Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 14

Webspiration

Task ID: 58
Tasks:
☐ Evaluate the collaboration experience
☐ Give suggestions for improvement

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 15

Webspiration

Task ID: 57
Tasks:
☐ Evaluate benefit/no benefit from collaborating on the poem

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 16

Webspiration

Task ID: 134
Tasks:
N/A

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 17

Second Language Chat

Task ID: 107
Tasks:
☐ Open screen.
☐ Locate menus and navigation buttons

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 18

Second Language Chat

Task ID: 162
Tasks:
☐ Pick a topic

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 19

Second Language Chat

Task ID: 163

Tasks:
☐ Follow 3 consecutive instructions to create a chatroom
*Native language version

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 20

Second Language Chat

Task ID: 129
Tasks:
☐ Access and use Google spreadsheets
☐ Paste the chatroom web address created in Chatzy
☐ Copy and paste secret code where indicated
*Native language version

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 21

Second Language Chat

Tasks:
☐ Facilitate chat
☐ Log partner’s response on checklist
☐ Save document on webpage to desktop

Task ID: 157

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 22

Second Language Chat

Task ID: 117
Tasks:
☐ Rate Partner

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 23

Second Language Chat

Task ID: 128
Tasks:
☐ Close screen

Skills Applied:
ICT Consumer

Y☐

N☐

ICT Producer

Y☐

N☐

ICT Social Capital

Y☐

N☐

ICT Intellectual Capital

Y☐

N☐

Teach Aid:

Y☐

N☐

Usability Issues:

Y☐

N☐

Describe Issues:

Comments:

ATC21S 24