MST Report

Multi-Stage Testing (MST) Study
Cognitive Laboratory Report:
Proposed Questionnaire Items

April 26, 2010

Multi‐Stage Testing (MST) Study Background Questionnaire 
Item Cognitive Laboratory Report and Recommendations 

This report details the administration of three Multi-Stage Testing (MST) Study items to eighthgrade students in a cognitive laboratory environment with a mock computer interface. The report
presents information about the cognitive lab participants. Then the format of the cognitive
laboratory is described. Student responses to the items that were administered in the cognitive
labs are discussed next. Finally, recommendations for potential revisions to items are presented.
Participant Information
For a number of reasons, including the limited timeframe in which to conduct the cognitive labs,
the difficulty in recruiting schools due to spring break, other standardized testing, etc., it was
decided to recruit the students by asking ETS employees if they had eighth-grade students who
would be interested in participating. Nine eighth-grade students were recruited to participate in
the study and given a $20 gift card as incentive to participate. Parents brought the students to
ETS for the cognitive lab. Students met with an interviewer and observer to respond to the three
items and follow-up questions. Items were administered via computer so that the student could
respond similarly to how they would respond in the actual study questionnaire.
Information about the schools the students attend was collected via the Internet using several
sources, including the CCD public school data (2007-2008 school year) from the National Center
for Education Statistics website and the schools’ websites.
 Of the nine students who participated in this study, eight students attend six public schools,
and one student attends a private Roman Catholic school.
 Five of the schools are located in Mercer County, one in Hunterdon County, and one in
Middlesex County, which are all located in central New Jersey.
 All of the schools are classified as suburban, except one, which is listed as rural.
 All of the schools are co-educational.
 The public schools include grades 6–8 or 7–8 only, but the private school includes
kindergarten through grade 8.  
 Most of the students in the schools are identified as White, non-Hispanic. In one school
Hispanics make up approximately 15 percent of the total population; in another 8 percent of
the students are identified as Asian/Pacific Islander; in another school 13 percent are
identified as Black, 10 percent as Hispanic, and 3 percent as Asian/Pacific Islander, for a
total of 26 percent non-White; in another school 44 percent are identified as Asian/Pacific
Islander and 44 percent as White; and in another school just over 50 percent of the students
are minority students, with 38 percent identified as Asian, 8 percent as Black, and 4 percent
as Hispanic.

2



Of the five public schools, two are classified as Title I Schools. Sixteen percent of students at
one school qualify for free or reduced-price lunches, and 11 percent of students qualify at
another school. Fewer than 10 percent of students qualify for free or reduced-price lunch at
the remaining schools.

Cognitive Laboratory Process
The three questions posed to the participants were all multiple choice or yes/no format. The
questions are located in the Appendix and address the students’ reactions to computer-based
testing.
When students arrived to the cognitive lab, they were seated in front of a laptop presenting the
computer interface. The interviewer then explained the cognitive lab procedure to participants
(after verifying that the student had parental permission to participate in the form of a signed
consent form), and students were informed that the lab would be recorded.
Students were instructed in the “think aloud” method. The interviewer first explained the process
the student would follow, and then the interviewer answered a sample question by thinking aloud
as he responded to the sample question. The student was then given the sample question and
asked to practice the think aloud process as he/she answered the sample question.
After explaining the protocol and practicing the think aloud process, students were asked to
begin by reading the first question aloud and talking through his/her answers. After the student
answered each question, the interviewer asked follow-up questions as appropriate to further
elucidate the student’s cognitive processes as she/he answered the questions. After answering the
second MST item (not the second follow-up question), students were told to imagine that the
third question referred to the last math test that they took in school and to answer the question as
if it were following that test. The questions used in the protocol as follow-up to the actual MST
questions are listed in the Appendix.
Proposed Questionnaire Items
Item 1 (VE355201)
The first item asked if the student had taken a test on the computer before; this item elicited the
most questions from students. Three students asked if this question referred to practice tests or
just graded tests. Seven students indicated that they had taken practice tests on the computer, and
four students indicated they had taken real tests or otherwise been engaged in computer-based
data collection (online student council voting). Multiple students explicitly mentioned an online
practice center called Study Island (available at http://www.studyisland.com/).

3

Based on participant responses, ETS recommends a change to the original stem of the item.
Since practice tests are to be included, this will be made explicit in the item. (See page 8 in the
Appendix to reference the revised version of this item.) This clarification will help verify that
students are providing the desired information. Several of the students interviewed indicated that
they had never taken a high-stakes test on the computer but had used the computer to practice
through one or more of the available study-aid oriented websites available. Response options to
the item would remain the same.
Item 2 (VE355201)
Students responded to item 2 (VE355216) without any questions or confusion. One student said
he/she preferred taking the test on the computer because he/she could “see the answers, there is
no erasing, and that makes it easier.” One student indicated that he/she preferred taking the test
on paper, as it allows him/her to take notes and use it as scratch paper. Seven students selected
the response option “It doesn’t matter,” but when asked if there was anything that would incline
them to prefer one or the other, they were able to articulate a variety of reasons. These reasons
included the perception that some computer tests make research resources available (making
computer tests preferable), while other times computer tests are timed (which the student said
made the test anxiety-inducing). Another student indicated that sometimes computer tests are
harder to understand but still said the test format did not matter. Another student indicated that if
scratch paper was available, then the computer would be preferable. It is recommended to keep
the current item wording.
Item 3 (VE355176)
Students responded to item 3 (VE355176) without difficulty, even though the students had to
imagine that they were answering the question in regard to a test taken in the past, rather than a
test they had just finished. Six students selected “Agree,” and three selected “Disagree.” Reasons
for selecting a response typically centered on how students performed on the test or whether they
studied for the test. Students were not asked about test performance, although some volunteered
that information. None of the students displayed any hesitation in answering the question. It is
recommended to keep the current item wording.

4

Appendix

MST Items Presented in the Cognitive Laboratory
1. VE355201
Have you ever taken a test on the computer?
A. Yes
B. No
2. VE355216
Do you prefer taking a test on computer or on paper?
A. Computer
B. Paper
C. It does not matter.
[Interviewer] I would like you to think about the last mathematics test you took in school.
Now I am going to show you a question. Please answer this question based on the test you
are thinking about.1
3. VE355176
Please indicate how much you disagree or agree with the following statement: This test was easy.
A. Strongly disagree
B. Disagree
C. Agree
D. Strongly agree

1

This text is not part of the actual question but was clearly read to the students prior to them seeing the final item.

5

Cognitive Laboratory Protocol
1. VE355201
Have you ever taken a test on the computer?
A. Yes
B. No
Follow-up questions for item 1
If the student is unable to respond to this question or hesitates when responding ask why they are
having difficulty. Drill down.
What type of tests have you taken on a computer?

2. VE355216
Do you prefer taking a test on computer or on paper?
A. Computer
B. Paper
C. It does not matter.
Follow-up questions for item 2
Please ask the student why he/she chose a specific answer.
Why do they have this preference?
Is there anything that would make you want to change your answer?

6

[Interviewer] I would like you to think about the last mathematics test you took in school.
Now I am going to show you a question. Please answer this question based on the test you
are thinking about.
3. VE355176*
Please indicate how much you disagree or agree with the following statement: This test was easy.
A. Strongly disagree
B. Disagree
C. Agree
D. Strongly agree
Follow-up questions for item 3
Why do you think this test was easy?
OR
Why do you think this test was not easy?
OR
If the student is unable to decide ask the student why he/she cannot decide.

7

Revised MST Items Based on Cognitive Laboratory

1. VE355201
Have you ever taken a test on the computer? The test may have been a classroom test, a standardized test,
a practice test or quiz, or any other type of test.
A. Yes
B. No

2. VE355216
Do you prefer taking a test on computer or on paper?
A. Computer
B. Paper
C. It does not matter.

3. VE355176
Please indicate how much you disagree or agree with the following statement: This test was easy.
A. Strongly disagree
B. Disagree
C. Agree
D. Strongly agree

8