KASA Technical Report

Knowledge and Skills Appropriate
(KaSA) Study
Cognitive Laboratory Report:
Proposed Questionnaire Items

July 2010

Knowledge and Skills Appropriate (KaSA) Study
Background Questionnaire Item Cognitive Laboratory
Report and Recommendations
This report details the administration of three Knowledge and Skills Appropriate (KaSA)
questionnaire items to four eighth-grade students and five fourth-grade students in a cognitive
laboratory environment. Students were given ten NAEP cognitive mathematics items1 and then
asked to respond to three questionnaire items using a cognitive lab protocol developed for this
purpose. The report presents information about the cognitive lab participants. The format of the
cognitive laboratory is described next. Then student responses to the items that were
administered in the cognitive labs are discussed. Finally, recommendations for potential
revisions to items are presented.

Participant Information
Grade 4
The five fourth-grade students who participated in the KaSA cognitive laboratories attend
Franklin School in Trenton, New Jersey. Approximately 37 percent of the students are Black and
69 percent are Hispanic, with the remainder identified as White, Asian, and American
Indian/Alaskan Native, in that order. Franklin School is a Title I School, and approximately 69
percent of the students receive free or reduced-price lunches. The school administrators/teachers
selected the students to participate in the study. In total there were two females and three males.
Grade 8
Three of the four eighth-grade students who participated in the KaSA cognitive laboratories
attend Grace A. Dunn Middle School in Trenton, New Jersey. Approximately 33 percent of the
students are Black and 58 percent are Hispanic, with the remainder identified as White, Asian,
and American Indian/Alaskan Native, in that order. Grace A. Dunn Middle School is a Title I
School, and approximately 55 percent of the students receive free or reduced-price lunches. The
school administrators/teachers selected the students to participate in the study. In total three
males participated. On the day of the interviews at the school, one of the selected students failed
to provide a signed parental permission form and could not participate. A female, eighth-grade
student who had volunteered to participate in another study was recruited to participate in the
KaSA cognitive lab, and the interview was conducted at ETS. The female student attends a
private Roman Catholic all-girls school.

1

The cognitive mathematics items were administered as part of the cognitive item tryout.

2

Cognitive Laboratory Process
Each participant met with an interviewer, who conducted the cognitive lab, and observer. Before
the cognitive lab began, students participated in a tryout of ten NAEP cognitive Mathematics
items.
After completing the cognitive questions, the students were told that they would be asked to
“think aloud” as they answered the last few questions. The interviewer then explained the
cognitive lab procedure to participants, and students were informed that the lab would be
recorded.
The interviewer instructed students in the “think aloud” method. The interviewer first explained
the process the student would follow and then answered a sample question by thinking aloud as
he/she responded to the sample question. Students were then given the sample question and
asked to practice the think aloud process as they answered the sample question.
After explaining the protocol and practicing the think aloud process, students were asked to
begin by reading the first question aloud and talking through their answers. After the student
answered each question, the interviewer asked follow-up questions as appropriate to further
elucidate the student‟s cognitive processes as she/he answered the questions. The questions are
located in the Appendix, and item revisions address the students‟ reactions to the questions.
Each administration of the cognitive lab lasted between 45 and 60 minutes, but no record of the
exact length of each interview was kept. Scheduling allowed for one hour for each interview, but
no student required more time than was allotted.
Because of concerns regarding reading comprehension, fourth-grade students were explicitly
asked the meaning of certain key words, while eighth-grade students were not. The form used for
this protocol is included in the Appendix. The same study protocol was used for both grades 4
and 8, however, some follow-up questions were used for only grade 4.
The remainder of this report will discuss the results for the fourth- and eighth-grade levels,
respectively, and provide recommendations for the wording of specific items.

3

Proposed Questionnaire Items
Grade 4 Results
Item 1 (VE390240)
Four of the five students chose option A “I answered all of the questions on this test” in response
to the first item: “Which statement best explains why you did not answer some of the questions
on this test.” The other student selected C “I did not learn the material asked about in some of the
questions on the test.” With that in mind, it should be noted that all five students answered all of
the questions on the test. The same student that selected option C also indicated (on the second
question and follow-up) that some cognitive questions covered topics not discussed in class.
During the follow-up questions for item 1, fourth-grade students were asked to define three
specific words that had been of concern during the item development process. These words were
“explain,” “material,” and “statement.” All students that were asked (one student was not asked)
were able to easily define the words “explain” and “statement” but did not give a contextappropriate definition for the word “material.” Instead of providing a definition related to
information taught in class, students referenced physical objects that could be used in the
execution of a mathematics problem, such as pencils, paper, calculators, and erasers.
Based on this pattern of responding, it is recommended that the wording of response option C be
revised from “I did not learn the material asked about in some of the questions on the test” to “I
did not learn how to solve the problems asked about in some of the questions on the test.” This
rewording avoids the context-sensitive definition of the word “materials” and clarifies the
response appropriately.
Item 2 (VE390261)
Item 2 also contains the same context-sensitive word “materials.” While it appears that students
were able to garner the meaning of the word from the context of the item, for the sake of clarity,
a revision to the stem is recommended. The original wording, “How many of the questions on
this test were about materials that you did not learn in school?” should be changed to “How
many of the questions on this test asked you to do something that you did not learn how to do in
school?” This modification should make the item easier for fourth-grade students to answer
without having to rely on context clues.
In terms of responses, two students selected option B “Only a few,” while three students selected
option A “None.” Follow-up questions indicate that the students understood the item and the
differences in response frequencies presented. One incident of note, however, is that two of the
students at various points in the interviews indicated that long multiplication is not their typical
method of solving mathematics problems, using instead the Lattice Multiplication framework. In

4

response to this question, one student indicated that he/she had expected to see the lattice
framework and considered long-multiplication to be material not covered in class.
Item 3 (VE390265)
Item 3 did not present any challenge for students to comprehend. Four students responded that
they had known about the test prior to the day it was given. The fifth student responded that
he/she had not but then indicated in follow-up that he/she had known about the test.
Additionally, in responding to the follow-up questions, none of the students indicated that they
had any information regarding the content of the test. Given that parental consent was required
for participation, all students had been told something about an upcoming test. Given that the
students did not show any difficulty in answering the question, no changes to the item are
recommended.
There is no evidence that the item order represents any particular difficulty for the students, so it
is recommended to keep the current order.
Grade 8 Results
Item 1 (VE390240)
In response to the initial question, all eighth-grade students were able to respond clearly and did
not demonstrate any trouble understanding the item. One student selected option A, two students
selection option C, and one student selected option E. While the eighth-grade responses do not
indicate that revisions are needed, a modification is recommended based on the fourth-grade
responses so that ultimately comparisons can be made between the fourth- and eighth-grade
questionnaires.
Item 2 (VE390261)
In response to item 2, two students selected option A, and two students selected option B
(indicated either “None” or “Only a few” of the questions were about new material). None of the
students, however, showed any difficulty in understanding or answering the question. However,
as with item 1, an item revision is recommended so that comparisons can be made between the
fourth- and eighth-grade questionnaires.
Item 3 (VE390265)
All eighth-grade students answered item 3 without difficulty and did not express any confusion
in regard to the item. It is recommended that the item remain unchanged from the version that
was used in the cognitive lab.
Concluding Summary
It should be noted that unless a response indicates that a student misunderstood the item, it is not
of particular note for the purposes of this study which responses students chose to any particular
5

item. Given the parameters of the study, it is more noteworthy if a student had difficulty
answering a question.
While fourth-grade students showed some difficulty in answering or understanding the first two
items administered in this study, eighth-grade students did not. However, for the purposes of
cross-grade comparison alone, it is recommended that the fourth-grade changes be applied
uniformly. The importance of this relies entirely on the deemed value of cross-grade
comparisons.

6

Appendix
KaSA Items Presented in the Cognitive Laboratory
1. VE390240
Which statement best explains why you did not answer some of the questions on this test?
A. I answered all of the questions on the test.
B. I did not have enough time to answer all of the questions on the test.
C. I did not learn the material asked about in some of the questions on the test.
D. I am not used to taking math tests where I have to choose an answer.
E. I am not used to taking math tests where I have to explain my answer.
F. I do not know how to use a calculator.
2. VE390261
How many of the questions on this test were about materials that you did not learn in school?
A. None
B. Only a few
C. About half
D. More than half
3. VE390265
Did you know about this test before today?
A. Yes
B. No

7

KaSA Cognitive Laboratory Protocol
1. VE390240
Which statement best explains why you did not answer some of the questions on this test?
A. I answered all of the questions on the test.
B. I did not have enough time to answer all of the questions on the test.
C. I did not learn the material asked about in some of the questions on the test.
D. I am not used to taking math tests where I have to choose an answer.
E. I am not used to taking math tests where I have to explain my answer.
F. I do not know how to use a calculator
Follow-up questions for item 1 (grades 4 and 8)
Was there anything confusing about this question?
Was there anything confusing about the answer choices?
Follow-up questions for item 1 (grade 4 only)
Please tell me what the following words mean: explain, material, statement.
Were there any words in the question that you did not understand?
2. VE390261
How many of the questions on this test were about materials that you did not learn in school?
A. None
B. Only a few
C. About half
D. More than half
Follow-up questions for item 2 (grades 4 and 8)
How many do you think is a few?
Why did you choose _______?
How many questions asked about material you did not learn in school?

8

3. VE390265
Did you know about this test before today?
A. Yes
B. No
Follow-up questions for item 3 (grades 4 and 8)
If the student chooses yes:
Where did you hear about this test?
What did you know about this test?
Who told you about this test?
How did you find out about this test?

9

Revised KaSA Items Based on Cognitive Laboratory
1. VE390240
Which statement best explains why you did not answer some of the questions on this test?
A. I answered all of the questions on the test.
B. I did not have enough time to answer all of the questions on the test.
C. I did not learn how to solve the problems asked about in some of the questions on the test.
D. I am not used to taking math tests where I have to choose an answer.
E. I am not used to taking math tests where I have to explain my answer.
F. I do not know how to use a calculator.
2. VE390261
How many of the questions on this test asked you to do something that you did not learn how to
do in school?
A. None
B. Only a few
C. About half
D. More than half
3. VE390265
Did you know about this test before today?
A. Yes
B. No

10

Part II
Two of the three KaSA items were significantly revised after the first round of cognitive
laboratories. Cognitive labs were then conducted on the two new items in May 2010. The
protocols and items follow.

11

DVR INSTRUCTIONS
Instructions on using the digital voice recorder (DVR)
Tips for the recorder
Saving the recordings onto your laptop
Set up DVR before students arrive.
Instructions for Using the DVR

The digital voice recorder and external microphone are very simple to use. To record a session
make sure that the microphone switch is in the “On” position and make sure the plug from the
microphone is securely plugged into the “MIC” jack on the recorder. You should turn the
microphone off when it’s not in use.
To make a recording, simply press the REC button. You will notice that the timer should be
counting up and the little button should light up red. Every time you press REC you store one file
in whatever folder is open. Pressing STOP will end that file and the light will turn off. Pressing
REC again will create another file in the same folder.
There are 4 folders, A, B, C, D. The screen will have one of these letters displayed to indicate
which folder is open. You can move from folder to folder by pressing the FOLDER/INDEX
button.
There are 4 different recording quality settings, XHQ, HQ, SP, and LP. We will use the HQ
setting and the recorder should already be in that mode. The recording mode can be seen on
the top left hand corner of the display. If it’s not in the HQ mode, you can change the mode by
pressing and holding down on the DISP/MENU button for 1 second. Press the Left (<<) and the
Right (>>) arrow keys until HQ is flashing. Press the down button (-) and make sure the
microphone is set to HI, not low. Then press the STOP button to exit the menu screen.
It’s easy to replay a recording. As long as you don’t move around folders, just hit the PLAY
button. You should be able to hear the recording even if the microphone is plugged into the
recorder. The light will be green when a wav file is playing.
Between sessions you may put the recorder on “HOLD” it serves as the ON/OFF button. To put
the recorder on hold push the side “HOLD” button up.

12

KaSA Questionnaire Cognitive Laboratory Session Script
Be sure, that the written parental consent form has been received from the student’s
parent(s) before proceeding.
I. KaSA PROTOCOL FOR ADMINISTERING MATH QUESTIONS
The following script may be used as a guide to ensure that all the relevant information that needs
to be conveyed to students is covered. It does not have to be read verbatim. However, try to
follow it closely.
I very much appreciate your agreeing to help us with this important project. We are
making a test that will be used with [4th/8th/12th] graders like you and asking [4th/8th/12th]
graders some questions about the test. Before we finish making the questions, we want to find out
how students like you will answer the questions. This will help us make the questions better and
more fair. When the questions are finished, they will be given to thousands of students across the
entire country. That’s why it is important that you are helping us today.
Here is what we are going to do. I will ask you to answer 8 sample mathematics
questions from the test that we are making. You will have 10 minutes to answer as many of the
math questions as you can. If you don’t finish the test, that is OK.
I would like you to try your best on each mathematics question, and to answer it just the way
you would if you were taking a real test. You should know, however, that this is not a real test -it does not have anything to do with your school grades or scores on tests that you take in school.
In fact, no one will know that it was you who took this test. Even though it does not really matter
how well you do on this test, I would like you to answer the questions as if it were a real test.
Here is a calculator you may use. Please practice using the calculator. Do you have any
questions about how to use the calculator?
Do you have any questions before we begin?
[Give student the test booklet.]
Here is the test booklet with 8 mathematics questions. You will have 10 minutes to
complete the test. It will be okay if you do not finish all of the questions before time is up. But, I
would like you to try and answer as many as you can. Please circle your answer, or write your
answer in the space below each question. You may use your calculator on any question. You may
begin.

13

Check your watch and write down the time the student starts the test. When 10 minutes are up,
please say: Your time is up. Please give me your booklet. Note how many questions the student
answered.

II. KaSA COGNITIVE LABORATORY PROTOCOL

Introduction and Practice
The following script is provided as a guide to explaining the KaSA study, the purpose of the
questionnaire, and the reason for carrying out cognitive interviews focused on the
questionnaire items. The sections in italics need not be read aloud to the student, but the
format should be carefully followed to ensure that all information is given and that the
format is the same for each student.
Explain that the interview will be audio-taped in case the interviewer needs to review the
student responses after the interview.
Assure the student that his/her responses will be kept confidential.
The interviewer should explain and demonstrate the think aloud technique. As a model for
the student, it will be helpful to use pencil and paper to take notes during this demonstration.
The student should then be provided with pencil and paper and given the opportunity to
practice thinking aloud. Allow as much practice time as necessary to make the student
comfortable in using the think aloud technique. The Introduction and Practice section of the
protocol does not need to be audio-taped.
Now I am going to ask you two more questions about the mathematics questions you just
answered. We are in the process of trying to improve these questionnaires that you are going to
respond to today by using something called cognitive laboratory techniques. During cognitive
laboratories, you will answer the questions and describe what you are thinking as you figure out
their answers. By studying these think aloud comments, we hope to be able to make the questions
easier to understand and answer.
Use this script, or something similar, to explain the purpose of the interview, the reason for
audio-taping it, and to obtain the student’s consent to be interviewed:
I have some questions that students like you in many different schools will be asked to answer
about themselves, their schools, and their families. The questions have no “right or wrong”
answers, but some may not be easy to understand and answer. I would like your help in trying to
make these questions easier to understand and answer. I’ll be taking notes on what you tell me,
but I would also like to use a tape recorder so I can check back and make sure I know exactly
what you said. No one else will listen to this tape and I won’t use your name or the name of your
school.

14

Before continuing, ask the student if he/she has any questions. After answering questions
and giving further explanation, continue:
Now let’s talk about how you will help me try out the questionnaire. The main thing that I need
you to do is think aloud as you answer each question so that I will know what you are thinking.
In other words, I would like you to say aloud everything you say to yourself silently when you are
thinking. This is not what people usually do, so I'll give you an example. Suppose there was a
question that asked: “How many different kinds of fruit have you eaten today and yesterday?”
I would probably start by saying: I think I’ll use paper and pencil to help me figure out my
answer. (Select a piece of paper and a pencil.) Now let me see. Today, I had a banana for
breakfast. That's one. (Record the number.) What kind of fruit did I eat yesterday? Well I had
orange juice for breakfast. That was juice, but it comes from a fruit, so I guess I’ll count that.
(Record the number.) I had another banana but the question says how many different kinds so I
won’t count that again. I had some carrot sticks for lunch, but carrots are vegetables not fruit, so
I won’t count that either. But I had an apple for a snack yesterday afternoon and apples are fruit
so I’ll count that. (Record the number.) And I had watermelon for dinner. (Record the
number.) So my answer is four different kinds of fruit.
Now give the student opportunity to practice thinking aloud.
OK, how about you? How many different kinds of fruit have you eaten today and yesterday?
Remember to tell me what you're thinking. Here’s a pencil and a piece of paper you can use to
help you figure out your answer.
Give the student pencil and paper and have him/her answer the same question. Alternate
questions (which may also be used for additional practice) are: How many different kinds
of vegetables have you eaten in the past three days? How many windows are in your house?
How many television programs have you watched in the past three days? Model a response
to these questions in a manner similar to the original question. When the student seems
comfortable with the think aloud process, proceed to the questionnaire.
Now proceed to The Questionnaire. Before turning on the tape recorder, remind the student that
you will be taping the remainder of the interview. Each question and the question responses,
including the sample questions, should be read aloud by the student, and the student should then
be directed to answer the question and to think aloud while doing so. If the student has
difficulty reading the questions, they may be read to him/her. Monitor and record the
student‟s think aloud comments, paying attention to how the question is interpreted, information
retrieved and synthesized, and a response produced. If the student is not volunteering comments,
prompts may be used such as: Could you tell me what you are thinking now? What's going on in
your mind now? What else? Do not, however, attempt in any way to help the student answer the
question. If the student asks for help in choosing an answer, you might say: Just choose the
answer you think is best. Remember, there are no right or wrong answers. The probes following
15

each question may be asked to obtain more information. You may formulate and ask other
probes as well, but remember that the purpose of the interview is not to lead the student to the
“correct” answer but to determine how the student has interpreted the question and chosen a
response.

16

KASA

1. Did you answer all of the math questions?
A. Yes
B. No
If you answered “Yes” to question 1, you are finished.

2. Why did you leave one or more of the math questions blank? Fill in one or more ovals.
A. I did not have enough time.
B. I do not know how to use a calculator.
C. I am not used to writing my answers.
D. The questions were too hard.
For all students, ask:
Was there anything confusing about question 1?
Were there any words in question 1 that you did not understand?

Follow up for those students who answered “Yes” to item 1:
Note whether the student actually answered all of the math items. If the student answered
“Yes” but did not answer all of the items, inquire why the student responded as he/she
did.
Note whether he/she reads the sentence “If you answered „Yes‟ to question 1, you are
finished,” i.e., does the student put down paper and pencil or indicate that he/she is done,
or does the student go on to read item 2. If so, ask why the student skipped the sentence
and read item 2. At this point, do not prompt the student to answer item 2 if he/she
answered question 1 as “Yes.”
Now ask the student to pretend he/she answered “No” and ask the student to explain in
his/her own words what each of the response options (A-D) in item 2 mean.
17

Follow up for those students who answered “No” to item 1:
Note whether the student actually left some of the questions blank. If the student
answered all of the questions, inquire why the student answered “No.”
If the student answered “No” to question 1, ask the student to indicate what he/she would
do next.
Ask the student if the arrow helped him/her know to go to question 2 and skip the
sentence about “If you answered “Yes” to question 1, you are finished.”
Ask the student why he/she selected the options he/she did.
Ask the student what he/she thinks the other options mean.

18

Now give the student these questions. Ask the student to read item 1 aloud and answer the
question. Note whether he/she follows the directions.
1. Did you answer all of the math questions?
A. Yes→You have finished. Thank you for your time.
B. No→Please answer the next question.

2. Why did you leave one or more of the math questions blank? Fill in one or more ovals.
A. I did not have enough time.
B. I do not know how to use a calculator.
C. I am not used to writing my answers.
D. The questions were too hard.

Ask the student which version of the items he/she thinks is easier to understand.

Ask the student why.

Be sure to record the race/ethnicity and sex of the student.

19

Participant Information and Results
Grade 4
Cognitive laboratories were conducted at two elementary schools: Martin Luther King
Elementary School in Trenton, NJ and Barnard Elementary School in Washington, D.C.
Martin Luther King Elementary School was opened in March 2010. Students at the school match
the following demographics: 94.44 percent Black, 4.76 percent Hispanic, and 0.79 percent
White. Additionally, 51.59 percent of the students are male, and 48.41 percent are female.
Approximately 58 percent of the students at this school are eligible for the free lunch program.
This is a Title I school with a Title I school-wide program. Although the school currently houses
only pre-K through fifth-grade students, plans are that the school will eventually house pre-K
through eighth grade students.
Both students interviewed at the school answered all math questions. Both students were African
American males. Both students were given a calculator with instructions for use, and while both
utilized the calculator on the exam, they also declined to spend any real time practicing with it,
stating that they were already familiar with the instrument. One student indicated that the ETSprovided calculator was identical to the one he used in class.
Both students were presented with the two questionnaire items. One student indicated that he
found the two-page formulation simpler, while the other preferred the single page formulation
(with the arrow). Both students accurately responded that they had answered all the questions on
the math test. One student read the print in bold (on the form with the arrow) and then said, “So
I‟m finished” and put down his pencil. The other student did not read the questions out loud, but
responded to audio prompts. His movements indicated that he had proceeded to the second
question, although he did not make any responses to the question. When asked why he had not
stopped upon reading that he was finished, he indicated that he had been curious about the
second question.
Both students indicated in their own words that they understood the responses to the second part
by giving correct and complete rewordings. Both students also indicated that they understood all
the words in the questions and that there was nothing confusing about the questions.
Barnard Elementary school enrolls 381 students, of whom 193 are male and 188 are female.
There are 3 Asian/Pacific students, 215 African American students, 158 Hispanic students, and 5
White students. This is a Provision II school, and all students receive free or reduced lunches. It
is also a Title I school with a school-wide Title I program.
Three students were interviewed: two males and one female. Two of the students were African
American, and the third was Hispanic. Two students were able to complete the test in the allotted
time, while the third student (a male) was only able to answer six of the eight questions.
All three students indicated that they preferred the second version of the questionnaire, i.e., the
version that has the questions on separate pages and the instructions regarding what to do
immediately after the response options. When reading the questions on version 1 (which uses an
20

arrow to direct students to the next question), student #3 read the bold print and started to read
the second question, but halted quickly. Student #1 read the bold part and kept reading the
second question. When asked why the student continued on, the student responded, "I didn't
understand the bold part." Student #2 skipped the bold part and went right to the second
question.
When asked to explain the meaning of the response options in question #2, all students were able
to describe in their own words what the options meant, although for option C one student
ommitted the word “not,” and described it this way, "don't bubble in answers, you write them,"
indicating that the student may not have a complete comprehension of the option. Another
student described it this way, "Means you're used to circling or filling." The final student
described it as "I don't write answers all the time," using a self-referent frame of orientation. The
other options did not present any special challenges for the students.
Based on these students‟ responses and those from the Martin Luther King Elementary School on
Wednesday, four students have preferred the version without the arrow to the next question
(version 2), with only one student indicating that the first setup was preferred. Note, however,
that all students in the Thursday lab preferred the format that they saw second. This is why it is
recommended to reverse the order of presentation for the final group of students.
Finally, from the Barnard group, when one student was asked why they preferred version 2, they
indicated that the bold print immediately following the response option made it more clear what
they were to do. Another student gave a similar answer, while the third student indicated only
that the bold text separating items might confuse a student.

Grade 8
MacFarland Middle School in Washington, D.C. is comprised of 210 students in grades 5
through 8, with 109 male students and 101 female students. Student ethnicity is as follows: 1
Asian student, 106 African American students, 103 Hispanic students, no American Indian
students, and no White students. One hundred seventy-four students are on free or reduced lunch
programs, and 52 students are in a special education program.
Three students were interviewed. Two students (one male, one female) were Hispanic, and the
other student (male) was African American. Two students were unable to complete the math
questions in the allotted five minutes, while the other student was able to put down an answer for
all the questions. That student did state, however, that their answer for the final question was a
guess.
In contrast to the fourth-grade students interviewed in previous labs, these students saw the
version of the questionnaire that does not use an arrow to direct student action first, and then the
version with the arrow. Two students stated that they preferred the non-arrow version and
indicated that the arrow could be confusing. One of these students also indicated that the
immediate directions made the non-arrow version less confusing. The third student stated that
they preferred the version with the arrow. However, this was after realizing that they had misread
the non-arrow version (seen first) and answered it at variance with their performance (this
21

student responded that he/she had answered all the questions when he/she had not). This student
indicated that the non-arrow format was confusing.
Finally, two students indicated that they had not answered all the questions because they ran out
of time, while the student who had guessed for the final question stated that he/she had not
answered all (although he/she did technically provide an answer to all questions) because the
questions were too difficult.
Based on both the fourth- and eighth-grade cognitive laboratories, the majority of students
preferred the non-arrow version of the questionnaire with directions immediately after each
response in question 1 over the version that used an arrow to direct action.

22