NAEP Touch-Screen Tablets Usability Study 2013

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 75 minutes including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 400 Maryland Avenue, S.W., Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573]. OMB No. 1850-0803

Demographic Information and Welcome Script

The Fulcrum IT interviewer will complete the demographic information, below, prior to starting the user testing session:

_{Ref #}	_Information	_{Observer Notes}
100	Date
101	Name of Facilitator
102	Name of Observer
103	Participant ID
104	School Name

Welcome Script

Text in italics is suggested content with which the facilitator should be thoroughly familiar in advance.

The facilitator should project a warm and reassuring manner toward the students and use conversational language to develop a friendly rapport. While the facilitator is talking, the Paperwork Burden Statement will be displayed on the screen of the tablet for the participant.

Hello, my name is [Name of Facilitator] and I work for Fulcrum IT, which is conducting a study for the National Center for Education Statistics (NCES) within the U.S. Department of Education. Thank you for helping us with our study today.

Today you are going to be looking at some assessment items made for touch-screen tablets. This is part of a study to see whether tablets work well for students in [insert grade] grade if they were to take one of our tests. In some of these items I will ask you to simply try out the controls on the touch screen to see how easy they are to operate. In other items you will see a short story or text and some questions about it, and you’ll use the controls to move around in the text as you read it and answer the questions, so we’ll be able to see how easy the controls are to operate during a task. Sometimes you will watch or read an explanation on how to do one of the items, and sometimes there will be no instructions at all, so we can see if students can figure something out without instructions.

But you are not here to take a test today. You are helping us to try out the system used to give different tests. We’ll be looking to see how well you can find your way through the system and operate the different controls, as well as your thoughts about how the system looks or anything you like or don’t like. We won’t be telling anyone how you personally did, but we will use your opinions and results without your name, along with the ideas and answers of other students, to make changes and improve the system for [insert grade]graders all across the country. It’s okay if you don’t know how to do any parts of the task. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the controls in the task.

While you are doing the tasks, we will be recording everything that happens on the touch screen. The recording will capture all of your responses and movements on the touch screen.

Do you have any questions so far?

I’ll be asking you questions and will be taking notes on what you say and do. We will go through a few different test items, and I will ask you to test out some parts of the system on the tablet. Again, we are not testing how well you do; you will be helping us to test the system.

Your participation is voluntary and if at any time you decide you do not want to continue, you may stop. Let’s begin.

Computer and Tablet Familiarity Survey

The Computer Familiarity Survey will be given to student participants at the beginning of the user testing session, after the welcome. The facilitator will place the survey in front of the student and read the questions and answer options aloud. Clarifying language will be used to help guide the students to an accurate response. Some examples of this type of language are “Well, do you use a tablet every day?”, and “How about the last time you used a computer? When was that?”

Please select the answer(s) that best describe(s) your experience with computers, tablets and mobile phones, either in or out of school:

About how much time do you spend using a computer (desktop or laptop) in a typical week?

	None
	Less than 1 hour
	1-5 hours
	More than 5 hours

About how much time do you spend using a tablet (like an iPad or Android tablet) in a typical week?

	None
	Less than 1 hour
	1-5 hours
	More than 5 hours

Now I’d like to find out how often you use a mobile phone (also known as a cell phone) for different activities.

About how many calls do you make per week on a mobile phone?	None	1-2 times	3-10 times	>10 times
About how many texts do you send per week on a mobile phone?	None	1-2 times	3-10 times	>10 times
About how many emails do you send per week on a mobile phone?	None	1-2 times	3-10 times	>10 times
About how much time do you spend per week surfing the internet or using other apps on a mobile phone?	None	<1 hour	1-5 hours	More than 5 hours

User Testing Scenarios

Description

The facilitator will set up items on the tablet between each task. Some tasks will be completed using the NAEP system, while others will be completed using prototypes that require switching from the NAEP system to a Web browser.

Participants will be given specific tasks to complete using the tablet. They will be instructed to narrate their actions, explaining what they are doing and why they are doing it while completing the tasks. If participants fail to provide sufficient narration of their actions, they will be prompted by the facilitator using questions such as, “What are you doing now?” and “Can you tell me why you did that?”

If a participant performs an action that did not have the desired effect, the facilitator may say, “Hmm, that didn’t work. What else could you try?” in order to get the multiple attempts required for data collection.

If participants are either unable to complete a particular task, or make a series of three errors while attempting to perform a task element, they will be told how to perform that task element, and will be directed to continue completing the task following successful interaction with the correct control or information element. The tasks described in this section are examples of the types of tasks that students will be performing, and do not constitute a comprehensive list. For each user testing session, the actual tasks, as well as their order of presentation, will be determined by the nature of the interactions being studied, as well as the counter-balancing needed for accurate interpretation of the results.

Below is a sample screen displaying the control elements available to the students.

Instructions

“We are going to take a look at some different types of test items that could be used for students throughout the country. We need to find out how easy it is to use this tablet for taking different tests. You are not taking a test today. You are just helping us learn if there will be any problems for students who do take different tests in the future.

I’m going to be asking you to perform some tasks on the tablet, and I will be writing down everything you do as you try to complete the task. There are no right or wrong answers today.

After each task, I will ask you to tell me how easy it was to perform that task. I will ask you if that task was very difficult, very easy, or somewhere in the middle.”

The following are descriptions of components of the user testing scenarios:

Task instructions –– The task instructions (shown in italic text below) let the participant know what they should try to accomplish. For example, in the first sample task below, the participant is asked to get the tablet ready to take the test. These instructions are often intentionally vague to see how intuitive the process is for the student.

Task step –– A task step (also frequently called a “task element” and shown in the left column of the table below) is a specific action that is required for completion of a task. Students will not be told what task steps are required for completion of a task. However, as mentioned earlier, prompts may be used to facilitate task completion if the participant is unlikely to do so without them, but the need for prompts is part of the data we will be collecting, so they will be provided only at certain points for each task step.

Control mechanism –– The control mechanism (shown in the right column of the table below) is a part of the interface on the tablet that the user interacts with to complete a task step. Examples of control mechanisms are the NEXT button, the volume control (both hardware and software) and the highlighter tool. The control mechanism, like the task step, is not described or mentioned to the participants. It is included here as an indicator of what is being tested. The task completion data, collected at the control mechanism level, will provide the high level of detail needed to provide sufficient feedback to item developers to maximize the usability of their items for actual NAEP assessments.

A paper containing the Ease of Use scale [section 4] will be placed on the table so both the student and facilitator can refer to it at any time. The Ease of Use scale will be administered at the completion of each of the tasks (time permitting).

The tasks in the scenarios below represent a sample of the types of tasks that participants will be asked to complete using the tablet. Each user testing session will comprise a different series of tasks and instructions depending on the availability and development stage of different assessment mechanisms in the NAEP system.

1Sample Task 1 – Launch the System

[Participant is presented with a tablet that is open, but asleep.]

The first thing we are going to do is get set up. Show me how you would get the tablet ready to take a test.

Task Step	Control Mechanism
Wake up tablet	Hardware button
Launch NAEP	Click “Begin” button

[Facilitator administers the Ease of Use scale.]

2Sample Task 2 – Adjust System Hardware and Settings

[Participant is presented with an adjustment screen displaying instructions for adjusting volume, tablet tilt, and color scheme.]

OK, we can now see the instructions for getting the earbuds and screen adjusted.

I’d like you to read and follow the directions on the screen.

Task Step	Control Mechanism
Read text	None
Go to next item	NEXT button
Adjust angle of tablet	Hardware: Stand
Insert earbuds	Hardware: Earbuds
Adjust volume	Toolbar volume control
Change color scheme	Toolbar Color Scheme button

OK, we can now see the instructions for the first test item.

[Facilitator administers the Ease of Use scale.]

3Sample Task 3 – Video Item

[Participant is presented with a screen with instructions for the first assessment item.]

Now what I’d like you to do is read the instructions and complete the first item.

Task Step	Control Mechanism
Read text	None
Go to item	NEXT button
Read item instructions	None
Play video	Video PLAY button

[Student will be interrupted after watching the video for administration of the Ease of Use scale.]

4Sample Tasks 4-7 – Word Processing Tools

[Participant is presented with a screen with the first item. There is a video mechanism on the left and a word- processing response window on the right.]

I’d like you to type a couple of sentences in the answer box. They can be about you, or the video, or anything.

Task Step	Control Mechanism
Activate text box	Text box
Type	Hardware: keyboard

[Facilitator administers the Ease of Use scale.]

Now I’m going to ask you to do some things to your answer. The first thing I want you to do is select your first word, copy it, and paste it to the end of the sentence.

Task Step	Control Mechanism
Select word	Finger/Selection gesture
Choose “Copy” from menu	Popup menu
Set cursor at end of sentence	Finger/text box
Tap/hold position with finger	Finger/text box
Choose “Paste” from menu	Popup menu

[Facilitator administers the Ease of Use scale.]

Now I’d like you to make your last sentence bold.

Task Step	Control Mechanism
Select word in sentence	Finger/select gesture
Extend selection to whole sentence	SELECT handles
Make text bold	Activate BOLD button

[Facilitator administers the Ease of Use scale.]

Let’s see if your last word is spelled correctly. How would you check the spelling?

Task Step	Control Mechanism
Select word	Finger/select gesture
Trigger spell-check	SPELLCHECK button
Select best option	List of Options
Trigger spelling change	“Change” button
Close Window	“Finish Check” button

[Facilitator administers the Ease of Use scale.]

5Sample Tasks 8-9 – Toolbar Controls

If these directions were too small for you to read, how would you make them bigger?

Task Step	Control Mechanism
Zoom text	Zoom control

[Facilitator administers the Ease of Use scale.]

I have a couple of questions for you. Are you almost done with this test? How can you tell? How much time do you have left?

Task Step	Control Mechanism
Read progress indicator	Progress indicator
Read time	Timer text

[Facilitator administers the Ease of Use scale.]

6Sample Tasks 10-11 – Support Tools, Scrolling

[Participant is presented with an item with a long series of pictures that require scrolling to view.]

I need you to look at all the pictures for this item. Once you’ve looked at them all, I’d like you to answer some more questions.

Can you use a calculator in this section?
If you get stuck, is there any way to get help on this screen?

Task Step	Control Mechanism
Scroll content area	Finger/scroll gesture
Check for calculator availability	Calculator tool
Open HELP screen	HELP button

[Facilitator administers the Ease of Use scale.]

[Participant is presented with a multiple-choice question.]

In looking at this item, I happen to know that answer C is wrong. Let’s cross that one out so we don’t have to think about it while deciding on the right answer. Now, of the three remaining answers, go ahead and select the one you think is correct. [PAUSE] If you don’t know, select option A.

Task Step	Control Mechanism
Eliminate choice	Eliminate choice mechanism
Select response option	Radio button

[Facilitator administers the Ease of Use scale.]

7Sample Tasks 12-13 – Item Control Mechanisms

[Participant is presented with a screen with a column of black and red boxes in the middle of the screen.]

For this next item, I’d like you to move all the black boxes to the left side and all the red boxes to the right side. Let’s go to the next item.

Task Step	Control Mechanism
Drag and drop interactive objects	Finger/drag gesture

[Facilitator administers the Ease of Use scale.]

[Participant is presented with an item requiring multiple “check box” responses.]

Here, I’d like you to just read the instructions and answer the item as if you were taking the test. Now, let’s move to the next item. [Participant moves to next tab.] Actually, let’s go back to make sure that we chose the answer we really wanted. What are those things on the right side of the screen? What does it mean when they look like this? [Facilitator points to “item visited” tab.]

Task Step	Control Mechanism
Check multiple boxes	Checkboxes
Navigate to next tabbed item	Tab
Navigate to previous tabbed item	Tab
Label visited tab	Visited Tab

[Facilitator administers the Ease of Use scale.]

8Sample Tasks 14-16 – Reading Scenarios

For this section, each participant will be completing up to three reading items (abbreviated reading passages and associated cognitive questions) that have not been broken down into tasks. The reading passages can be navigated in a number of different ways during the course of answering the questions, and the purpose of this portion of the user testing is to gather information on the preferred method for navigating the passage and answering the questions. The students will be asked to answer the cognitive questions, because without the intent to answer an item correctly, many of the navigation and control tasks might be skipped or they might not be used in a representative way.

Each reading passage will be shortened so that only the text required for task completion will need to be read by the participants. Items will contain only as many cognitive questions as needed to prompt the range of tasks being studied. The three different reading items contain different formatting and presentation elements and give the student access to different types of navigational controls. For example, some passages will be presented on a series of digital pages, while others will be presented using continuous scroll. Examples of cognitive question tasks include following a link or a textual reference to a specific portion of the accompanying reading passage. Because these items are not broken down into tasks, ease-of-use ratings will not be collected for these items. After completion of each item, students will be asked to provide feedback on item-specific features. After completion of all three activities, students will be guided through the Reading Items Student Feedback (section 3.9) which probes preferences among the different mechanisms presented across the three reading items. The following script will be used to introduce the students to this portion of the user testing.

Our next three items are for a reading assessment. I’m not going to guide you through each item, because I want to see how you would do it. Sometimes I will ask you questions as you work through or right after you read a passage. After we’ve completed all three reading items, I will ask you for your opinion on how the passages and questions were presented.

Now, I’d like you to read the instructions and answer the questions as if you were taking the test. It’s okay if you don’t know how to do any parts of the task. My goal is to learn how you use the system to complete the task, so just try to do whatever seems natural to you.

9 Reading Items Student Feedback

The following are examples of the types of reading-related questions asked of students. These will be administered verbally once the student has completed all three reading items, and the spoken answers will be recorded. The specific questions asked will be determined by the tasks the student previously viewed, since they are designed to target specific aspects of the formatting and interactive functionality. To keep within the time constraints of the session, not all questions will be asked of all participants. For some questions the facilitator will mime the action or point to a sample screen layout on paper, to help the student understand and recall the issue that is being probed.

Now that you have completed all three of the reading items, I’d like to get your opinion about what parts of them were easy or hard to use, and which you liked the best.

To move through some texts, you had to turn each page, like this [mime page-turning action]. To move through other texts, you scrolled up and down, like this [mime scrolling action] and there were no separate pages to turn. Did you like one way of moving through the texts more than the other? Why?
- [Add the following follow-up question if student has not indicated why one way easier already]: Was one way of moving through the texts easier for you than the other? Why?
When you tapped the button to show the questions, in some tasks the question appeared and took up half of the screen and the passage took up the other half [point to example screen layout on paper]. In other tasks the text stayed where it was and the questions were shown on top of the text [point to example screen layout on paper]. Did you like one of these more than the other? Why?
- [If student has not indicated why one way is easier already]: Did you find one of these easier to use than the other? Why?
In some tasks, you had to answer questions on more than one passage [point to example]. What problems did you have, if any, moving between the two passages and finding what you needed in both?
When a question referred to a specific place in a text, sometimes you could tap on a link that took you to that place in the text [point to example]. Sometimes the question gave the page number where you would need to go if you wanted to find that place in the text [point to example]. Did you like one of these more than the other? Why?
- [If student has not already indicated why one method is easier]: Do you think one of these is easier to use than the other? Why?
- Did having a link to tap make you more likely to go back and look at the text?
- When you didn’t have a link you could tap, and you had to go back and find the reference yourself, did it make a difference if you had page numbers you could tap, versus not having any page numbers and having to turn the pages to find the right place in the text?
In general, did you use the navigation buttons and tabs, like page numbers and passage numbers, to move around the texts, or did you tend to turn the pages or scroll up and down to get where you wanted to go?
In some tasks, you had a link to a picture. Sometimes the link took you to the place in the passage where the picture was [point to example]. Sometimes the link made the picture pop up for you [point to example]. Did you like one kind of link more than the other? Why?
- [If student has not already indicated why one method was easier]: Did you find one kind of link easier to use than the other? Why?
- Do you think having a link was better than having to find the picture yourself? Why?

10Sample Tasks 17-19 – Math Tools

[Participant is presented with a math item.]

For this item, I would like you to use a calculator to answer this question.

Task Step	Control Mechanism
Open calculator	Calculator tool
Enter equation	Touch screen number pad
Get calculated result	ENTER key
Activate response field	Click in response field
Enter calculated result in response field	Hardware or touch screen number pad

[Facilitator administers the Ease of Use scale.]

[Participant is presented with another math item with space for capturing handwriting.]

For this item, I would like you to use this stylus [hands stylus to student] in this space here [points to on-screen writing area] to answer the question.

Task Step	Control Mechanism
Adjust tablet	Screen and keyboard
Write answer	Stylus

[Facilitator administers the Ease of Use scale.]

How would you type in this equation?

Task Step	Control Mechanism
Launch equation editor	Equation editor toggle
Activate equation entry field	Touch screen
Enter equation	On-screen symbol pad

[Facilitator administers the Ease of Use scale.]

11Summary Questions

That’s all we need to do with the test. Now I would like to hear your opinions about everything we did today.

Overall, how easy was it to work on this tablet? We are not asking how hard the questions were to answer, but how hard or easy it was to move around and do things required for the test on this tablet. [Facilitator administers the Ease of Use scale]
What is the one thing you liked most about using the tablet?
Can you tell me a way we could make our assessments on tablets better?
Do you have any other comments for us about taking an assessment on a tablet?

Thank you for helping us to improve our test.

Ease of Use Rating Survey/ Comments

The following Likert scale will be administered at the conclusion of tasks (time permitting) during the user testing.

Very Difficult	Somewhat Difficult	Can’t Decide	Somewhat Easy	Very Easy
1	2	3	4	5

Initially, the facilitator will use the following script to prompt a response for the Ease of Use scale.

I’d like to ask you about what you just did. How easy was it for you to figure out what to do? Was it very easy, very difficult, or somewhere in the middle?

If the participant answers to the effect of “in the middle,” the facilitator will then ask, “Was it somewhat easy, somewhat difficult, or is it hard to decide?”

These prompts will be used initially and less frequently as needed by participants, but may be dropped for some students as they become accustomed to responding to the Ease of Use scale.

After a rating has been given for the task, the facilitator will then ask, “Do you have any comments about this task? Do you have any suggestions to help us make it better?”

Responses will be written down verbatim by the facilitator.

System Usability Scale

Each participant will be asked to take the System Usability Scale (SUS) after completion of the user testing. The SUS consists of 10 survey questions and provides a norm-referenced comparison among a wide range of user interface designs. Though some of the questions are not relevant to this particular study, and the wording is not necessarily geared toward the ages of our participants, it is important that this survey be administered in exactly the way it was standardized in order to accurately interpret the norm-referenced scores. The survey will be administered by the facilitator who will read the prompts and the response options to the participant, as needed.

I think that I would like to use this system frequently.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I found the system unnecessarily complex.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I think the system is easy to use.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I think that I would need the support of a technical person to be able to use this system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I found the various functions in this system were well integrated.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I think there is too much inconsistency in this system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I would imagine that most people would learn to use this system very quickly.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I found the system very cumbersome to use.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I felt very confident using the system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐
I needed to learn a lot of things before I could get going with this system.
Strongly Agree 1	2	3	4	Strongly Disagree 5
☐	☐	☐	☐	☐

File Type	application/msword
File Title	OMB Apprioval App, Vol II
Author	joconnell
Last Modified By	Kashka Kubzdela
File Modified	2013-10-31
File Created	2013-10-30

NAEP Touch-Screen Tablets Usability Study 2013

NCES Cognitive, Pilot, and Field Test Studies System

NAEP Tablet Usability Study 2013 Volume II