Download:
pdf |
pdfNAEP Writing Background Question
Cognitive Lab Report
Prior to pilot testing new or revised writing background questions in 2010 with a large, national
sample of students, teachers and school administrators, ETS conducted a structured cognitive lab
study to investigate:
Accessibility: are respondents able to understand the intent of the questions and
provide a response;
Readability: is the vocabulary used appropriate for the intended respondents,
particularly fourth-grade students; and
Usability: for computer-based student questions at grades 8 and 12, are students
comfortable with the single and/or multiple-question presentation.
Data from the cognitive lab will primarily inform subsequent revisions to new writing
background questions prior to the 2010 national pilot as well as provide the program with
preliminary information on presentation of computer-based questions. Early identification of
problematic question, prior to administration to a large number of respondents, will increase the
quality of the questions by reducing or eliminating confusing or intrusive language. Sample size
limitations do not allow in-depth examination of question performance or relationships among
questions. Data from the 2010 pilot will allow for a more thorough examination of the questions.
Appendix A contains the student, teacher and school background questions included in the
cognitive lab study.
New writing background questions were investigated in cognitive lab studies in December 2008,
and the selected and/or revised questions will be included in a national pilot test in 2010. ETS
has proposed a mixture of new and previously-administered (or trend) questions for the 2011
writing assessment. Given the change in administration mode for the grades 8 and 12 student
background questions (from paper-and-pencil to computer-based), all grades 8 and 12 student
questions will be classified as new, even if the wording of the questions are identical to
previously-administered questions. Trend background questions proposed for the 2011 writing
assessment were not included in the cognitive labs.
Table 1 presents the number of participants and questions1 included the December 2008
cognitive labs. Cells have different sample sizes reflecting our attempt to maximize information
1
The number of questions included in the cognitive lab only represent the number of proposed new questions; the
number of proposed trend questions are not reflected.
1
while staying within the upper limit of 9 participants per question (which can be tested without
clearance from OMB). Smaller samples of teachers and school administrators reflect our
assumption that feedback from adults on questions would not vary as much as feedback from
students. ETS’s experience with cognitive labs with adults for various research projects,
including those conducted for the National Indian Education Study (NIES), supports the use of
smaller teacher and school administrator samples. For students, there are six new question for 4th
grade, and 24 new questions or subparts of matrix questions2 that are shared across 8th and 12th
grades. At grades 8 and 12, all the proposed student writing questions will be included in the
cognitive labs due to the change in the administration mode; at grade 4, only the subset of
proposed questions that are new for the 2011 administration are included. For teachers, the same
ten new questions are proposed for grades 4 and 8; and for schools, the same six new questions
are proposed for grades 4, 8 and 12.
Table 1: Number of Cognitive Lab Participants and Questions
Respondent
Student
Teacher
School
Grade Level
# of
Questions
# of
Participants
4
8
12
4
8
4
8
12
6
24
24
10
**
6
††
††
5
4
4
3
3
2
2
2
** same teacher questions for grades 4 and 8
†† same school questions for grades 4, 8 and 12
2
For matrix questions composed of a single stem and multiple subparts, each subpart is considered a separate
question.
2
Cognitive Lab Procedures
Students
The cognitive labs for students (grades 4, 8 and 12) were conducted at the ETS usability lab.
Students were recruited via an internal ETS newsletter to ETS employees, which is the typical
procedure for recruiting cognitive lab participants, and has proven successful previously. ETS
attempted to balance males and females, and to include minority group participants in the
samples. The cognitive lab lasted no more than an hour and a signed parent permission form was
secured for each participant. Students were compensated ($25 gift card) for participating in the
study. The 4th grade students responded to the new writing background questions in paper-andpencil format since this is the administration format for the 2010 pilot and 2011 operational
writing assessments. The 8th and 12th grade students responded to the new writing background
questions on a computer delivered via PowerPoint. This procedure has worked well in previous
cognitive labs where the goal is to simulate a computer-based survey. At grades 8 and 12, we
also investigated two presentation formats for matrix questions. The first presentation format
displayed matrix questions with the set leader and one sub-item displayed at a time on each slide.
The second presentation format displayed the matrix questions in a manner similar to how matrix
questions appear in paper-and-pencil questionnaires, with several sub-items with their
corresponding set leader on each slide. Comparing these two formats enabled us to evaluate the
display preference for students as well as any particular problems students might have with either
of the formats. (Additional information on presentation formats will be gathered following the
selection of the delivery platform and hardware configuration as part of a larger system tryout.)
The 8th and 12th grade students completed the entire questionnaire section3 first, without
interruption, allowing ETS to determine the length of time students actually take to respond to all
of the proposed questions. The students then were asked to read the questions again and “think
aloud” as they responded to the questions. Prompts, prepared in advance, aided interviewers in
collecting as much relevant information as possible from each student. Directions for the
computer-delivered 8th and 12th grade student questionnaires also were new. Students were asked
about the clarity of the directions. Directions for other groups (4th grade students; all teachers; all
school administrators) did not change and are standardized across NAEP assessments; thus, these
directions were not specifically included in the “think-aloud” cognitive labs for those groups.
Both paper-and-pencil and computer-based formats were matched as closely as possible to the
current NAEP questionnaire formats. Fourth grade students did not complete the entire
questionnaire section (i.e., Section 4) first, as the 8th and 12th grade students did, because most of
the 4th grade questions are trend, and only seven of the 21 proposed questions were classified as
new and included in the cognitive lab.
3
All questions proposed for the grades 8 and 12 student writing background question section (i.e., Section 4) are
considered new, due to change in administration mode, and included in the cognitive lab.
3
Teachers and School Principals
The data collection with the teacher and school principal samples were conducted at the schools
where the teachers and principals were employed. Teachers and principals were compensated
$50 for their time. They responded to the new writing background questions in a paper-andpencil format. In the NAEP 2011 writing assessment, although teachers and school principals
will be able to choose to answer questionnaires using a computer or paper and pencil, typically
90% of them have answer using paper and pencil. For each question, they followed instructions
for the think-aloud protocol.
Think-Aloud Protocol
The main goal of the think-aloud protocol was to address issues of question accessibility,
readability, and sensitivity with the new questions and new directions. The think-aloud protocols
entailed asking participants to articulate their thought processes while reading directions and
answering questions. Interviewers were trained to maximize a respondent’s verbal expression
without biasing responses and to take notes in the session. One training exercise entailed
interviewers watching a video of a think-aloud procedure to familiarized themselves with how to
ask questions of cognitive lab participants. Additionally, think-aloud protocols were audio taped
if reference to individual sessions are needed. (See Figure 1 for the think-aloud protocol
introduction and template.)
4
Figure 1: Think-Aloud Protocol Introduction and Template
Hello! My name is (YOUR NAME). Thank you so much for your help. Today you will be
participating in a special study to understand how people take survey items. The goal of this
research is to make sure the questions we are asking in the survey are the most appropriate
questions. I have some questions that students like you, in many different schools, will be asked
about themselves, their schools, and their families. The questions have no “right or wrong”
answers, but some may not be easy to understand and answer. I would like your help in trying to
make these questions easier to understand and answer. I’ll be taking notes on what you tell me.
Now let’s talk about how you will help me try out the questionnaire. The main thing that I need
you to do is think aloud as you answer each question so that I will know what you are thinking.
In other words, I would like you to say aloud everything you say silently to yourself when you are
thinking. This is not what people usually do, so I'll give you an example. Please read and answer
the item below.
The choices for some questions will be displayed on your screen as shown below. Now read the
example, think out loud your answer, and fill in the oval beside your answer.
How often do you watch television?
A
Never or hardly ever
B
Once or twice a month
C
Once or twice a week
D
Almost every day
While the participant is responding, the interview will prompt the participant with probes,
such as the following;
Probes
Did you understand the question?
Did you understand the choices of answers (e.g., Never or hardly ever, Once or twice a week)?
How many times did you have to read the choices for an answer?
Did you understand the activities being asked about (e.g., Watch television)?
How many times did you have to read the question before you were able to answer?
If a student in your class asked you what this questions means, what would you say?
5
Data Analysis and Results
The members of the cognitive lab interview team were all ETS researchers with previous
experience in conducting cognitive labs using a think-aloud protocol. Some members of the
team participated in the cognitive labs for the NIES. Data analysis was qualitative in nature and
the cognitive lab team met to discuss the think-aloud protocols and decide which questions need
to be revised. The following Results section presents summary statements for questions flagged
as potentially confusing or problematic. Some, but not all, of the flagged questions were revised
and revisions are included in Appendix A for NCES review.
A total of 46 new writing background questions were included in the cognitive lab study. Table
2 summarizes the number of flagged questions by grade level and respondent. None of the six
new school questions were flagged as problematic based on data collected in the study. Also,
none of the six new grade 4 student questions were flagged.
Table 2: Number of Flagged Questions
Respondent
Student
Teacher
School
Grade Level
4
8 & 12
4&8
4, 8 & 12
New Questions
Total
Flagged
7
0
25
5
19
2
6
0
6
Student – Grades 8 and 12
Timing. All 8th grade students took less than 10 minutes to answer the 24 questions. The
12th grade students took well under 10 minutes.
Computer-based Matix Questions. One 8th grader and two 12th graders preferred having
the multiple-stem format for matrix questions because they didn’t appear to stretch out
forever. Three 8th graders and two 12th graders preferred to have the single-stem format for
matrix questions because they were less confusing. We recommend testing the two matrix
formats with a larger sample in the computer system tryouts.
Computer-based Directions. Students had no trouble understanding the directions used for
the computer based questions. Figure 2 is a screen shot of the Power Point presentation for
grades 8 and 12 students.
Figure 2. Example Grades 8 and 12 Student Matrix Question: Multiple Sub-items per Screen.
7
Flagged Questions. Based on the comments of one or more student, the following questions
were flagged for further review by ETS research and item development staff. Some, but not
all, flagged questions were revised to address problematic wording.
o Overall. There was variability in opinions about what the questions mean by
“write.” One student thought it might mean using your hand to write with a pen
or pencil and asks if she should include time spent typing (she asked this
regarding Question 11). Some students included the time they spend on
homework, even when that homework involves reading something and then
writing a response. Some included the time it takes to figure out a response to a
question and write it. Some said of math class, “We don’t write, we just use
numbers.” For example, is solving a math problem writing? Based on these
results we recommend that the Writing Standing Committee and the Background
Variable Standing Committee be consulted to define “writing” and to suggest the
best way to incorporate a definition into the student background questionnaire.
o Question 3a. One student did not know what log meant. Recommend deleting
the word “log” from the stem.
o Question 3g. Students did not know what business material was. Recommend
deleting “business material” and adding “job application”.
o Question 5b. Some students were unclear about what “cut and paste” meant.
Also it may imply plagiarism. Recommend deleting “cut and paste”.
o Question 10. Students wanted to know if this question includes note taking?
Recommend adding “note taking” to the stem.
o Question 12. The examples in this question lead students to eliminate e-mails
and IMing. Recommend adding “e-mailing” to the stem.
Teacher – Grades 4 and 8
Flagged Questions. Based on the comments of one or more teacher, the following questions
were flagged for further review by ETS research and item development staff. The flagged
questions were revised to address problematic wording.
o Question 1c. Teachers were unsure what prewriting exercises are. Recommend
adding examples of prewriting activities to the stem.
o Question 3. Teachers thought “informal” should be taken out since all
professional development activities are formal. Additionally, teachers did not
interpret this item to reflect outside writing activities. Recommend revising the
stem to read “During the past two years, have you written outside of school for
personal enjoyment or enrichment?”
8
Appendix A
Proposed New Writing Background Questions
Tested in Cognitive Lab & Proposed Revisions
9
Grade 4 Student
VE032882
1. In a week, about how much do you write for reading homework?
A
None
B
A few words
C
A few sentences
D
1 page
E
More than 1 page
VE032905
2. In a week, about how much do you write for science homework?
A
None
B
A few words
C
A few sentences
D
1 page
E
More than 1 page
10
VE032909
3. In a week, about how much do you write for social studies homework?
A
None
B
A few words
C
A few sentences
D
1 page
E
More than 1 page
VE032912
4. In a week, about how much do you write for math homework?
A
None
B
A few words
C
A few sentences
D
1 page
E
More than 1 page
11
VE036679
5. In a day, about how much time do you spend writing during classes at school?
A
None
B
15 to 30 minutes
C
1 hour
D
More than 1 hour
VE032916
6. In a day, about how much time do you spend writing for school homework?
A
None
B
15 to 30 minutes
C
1 hour
D
More than 1 hour
12
VE032918
7. In a day, about how much time do you spend writing on your own and not for school—for
example, writing stories or keeping a journal at home?
A
None
B
15 to 30 minutes
C
1 hour
D
More than 1 hour
13
Grades 8 and 12 Student
VE035611
1. Please indicate how much you disagree or agree with the following statements about reading and
writing. Fill in one oval on each line.
a. Writing things like stories or letters is one of
my favorite activities.
b. Writing helps me express my ideas.
Strongly
disagree
A
Disagree
Agree
C
Strongly
agree
D
B
VE035613
A
B
C
D
VE035628
VE035633
2. How often do you do each of the following? Fill in one oval on each line.
a. Write in a private diary or journal on your
own time
b. Write stories or poems for fun on your own
time
Never or
hardly
ever
A
Once or
twice a
month
B
Once or
twice a
week
C
Almost
every day
D
VE035640
A
B
C
D
VE035644
14
VE035647
3. For school this year, how often do you write each of the following? Fill in one oval on each line.
a. Thoughts or observations in a log or
journal
Never or
hardly
ever
A
A few
times a
year
B
Once or
twice a
week
C
At least
once a
week
D
VE035648
A
B
C
D
VE036272
A
B
C
D
VE036275
A
B
C
D
VE036277
A
B
C
D
VE036280
A
B
C
D
VE036282
A
B
C
D
VE036283
REVISION
a. Thoughts or observations in a journal
b. A simple summary of something you
have read
c. A report about something you have
studied or researched
d. An essay in which you analyze or
interpret something
e. A letter or essay in which you try to
convince or persuade others to believe or
do something
f. A story, such as a story about a personal
or imagined experience
g. Business material, such as a résumé or a
letter to a company
REVISION
g. Job application, a résumé or a letter to a
company
VE036287
4. When you write a paper or report for school this year, how often do you do each of the
following? Fill in one oval on each line.
a. Brainstorm with other students to decide what to
write about
b. Organize your paper before you write—for example,
make an outline or draw a chart
c. Make changes to your paper to fix mistakes and
improve it
d. Work with other students in pairs or small groups to
discuss and improve your paper
Never or
hardly
ever
A
Sometimes
Almost
always
B
C
VE036296
A
B
C
VE036297
A
B
C
VE036298
A
B
C
VE036313
15
VE036314
5. When you write a paper or report for school this year, how often do you do each of the
following? Fill in one oval on each line.
a. Use a computer from the beginning to write the
paper or report—for example, use a computer to
write the first draft
b. Use a computer to make changes to the paper or
report—for example, by means of spell checking
or cut and paste
Never or
hardly
ever
A
Sometimes
Almost
always
B
C
VE036315
A
B
C
VE036316
A
B
C
VE036318
REVISION
b. Use a computer to make changes to the paper or
report—for example, by means of spell checking
c. Look for information on the Internet to include in
the paper or report
VE036713
6. In a week, about how much do you write for English/language arts homework?
A
None
B
Several words
C
A few sentences
D
A couple of paragraphs
E
1 to 3 pages
F
4 to 5 pages
G
More than 5 pages
NOTE: Based on reviews from NCES following the cognitive lab study, the number of
options for this item were reduced from seven to five.
16
VE036714
7. In a week, about how much do you write for science homework?
A
None
B
Several words
C
A few sentences
D
A couple of paragraphs
E
1 to 3 pages
F
4 to 5 pages
G
More than 5 pages
NOTE: Based on reviews from NCES following the cognitive lab study, the number of
options for this item were reduced from seven to five.
VE036715
8. In a week, about how much do you write for social studies homework—for example, geography,
government, or history?
A
None
B
Several words
C
A few sentences
D
A couple of paragraphs
E
1 to 3 pages
F
4 to 5 pages
G
More than 5 pages
NOTE: Based on reviews from NCES following the cognitive lab study, the number of
options for this item were reduced from seven to five.
17
VE036716
9. In a week, about how much do you write for math homework?
A
None
B
Several words
C
A few sentences
D
A couple of paragraphs
E
1 to 3 pages
F
4 to 5 pages
G
More than 5 pages
NOTE: Based on reviews from NCES following the cognitive lab study, the number of
options for this item were reduced from seven to five.
VE036755
10. In a day, about how much time do you spend writing during classes at school?
A
None
B
15 to 30 minutes
C
1 hour
D
2 to 3 hours
E
More than 3 hours
REVISION
10. In a day, about how much time do you spend writing, including taking notes, during classes at
school?
18
VE036760
11. In a day, about how much time do you spend writing for school homework?
A
None
B
15 to 30 minutes
C
1 hour
D
2 to 3 hours
E
More than 3 hours
VE036761
12. In a day, about how much time do you spend writing on your own and not for school—for
example, writing stories or keeping a journal at home?
A
None
B
15 to 30 minutes
C
1 hour
D
2 to 3 hours
E
More than 3 hours
REVISION
12. In a day, about how much time do you spend writing on your own and not for school—for
example, writing stories, e-mailing, or keeping a journal at home?
19
Grades 4 and 8 Teacher
VB429595
1. How often do you ask your students to do the following when you ask them to write about
something? Fill in one oval on each line.
a. Think about ways to improve their
writing by revising and editing
b. Collaborate with other students—
for example, by reading each
other’s drafts or critiquing each
other’s work
c. Use a computer for completion of
the prewriting exercises for a
paper
Never
A
Rarely
B
Sometimes
C
Often
D
VE032854
A
B
C
D
VE032878
A
B
C
D
VE032879
A
B
C
D
VE036831
A
B
C
D
VE036840
A
B
C
D
VE036849
A
B
C
D
VE032881
A
B
C
D
VE036856
A
A
B
B
C
C
D
D
VE036860
A
B
C
D
VE036893
REVISION
Use a computer for completion of
prewriting (e.g., outlines, graphic
organizers, etc.) for a paper
d. Use a computer for the first draft
of a paper
e. Use a computer for all drafting
and revising
f. Use a computer to complete their
writing
g. Use a computer for spelling
check, dictionaries, and thesaurus
h. Use the Internet to get information
for their writing
i. Assess their own writing
j. Assess their own writing by using
rubrics provided by you or other
teachers
k. Assess the writing of other
students by using rubrics provided
by you or other teachers
20
VE036861
Educational Testing Service
Rosedale Road
Princeton, NJ 08541
VE032896
2. To what extent do you consider each of the following when you assess student writing? Fill in
one oval on each line.
a. Variety in sentence construction
b. Word choice
c. Development of ideas
d. Creativity of ideas
e. Organization of ideas
f. Mechanics and conventions, such
as spelling, grammar, and
punctuation
Length of a paper
Not at
all
A
A
A
A
A
A
A small
extent
B
B
B
B
B
B
A moderate
extent
C
C
C
C
C
C
A large
extent
D
D
D
D
D
D
VE032898
VE036894
VE036895
VE032910
VE032900
VE032906
A
B
C
D
VE032903
VE032921
3. During the past two years, did you participate in any informal professional development
activities related to the teaching of language arts, such as writing novels or short stories?
A
Yes
B No
REVISION
3. During the past two years, have you written outside of school for personal enjoyment or
enrichment?
A
Yes
B
No
www.ets.org
Educational Testing Service
Rosedale Road
Princeton, NJ 08541
Grades 4, 89 and 12 School
1. To what extent are the following activities/programs offered in your school? Fill in one oval on
each line.
a. Schoolwide writing initiative,
such as specific guidelines or
objectives for school writing
instruction
b. Literacy or writing coaching for
students
c. Professional development in
writing or teaching writing—for
language arts faculty only
d. Professional development in
writing or teaching writing—for
all teachers
Not at
all
A
A small
extent
B
A moderate
extent
C
A large
extent
D
VE032874
A
B
C
D
VE032877
A
B
C
D
VE032894
A
B
C
D
VE032897
E032907
2. To what extent is writing taught at your school in content areas other than language arts—for
example, in social studies or science classes?
A
Not at all
B
A small extent
C
A moderate extent
D
A large extent
VE032914
3. To what extent does the NAEP Writing Assessment influence the writing curriculum at your
school?
A
Not at all
B
A small extent
C
A moderate extent
D
A large extent
www.ets.org
Educational Testing Service
Rosedale Road
Princeton, NJ 08541
www.ets.org
| File Type | application/pdf |
| File Title | Start Text here |
| Author | jwert |
| File Modified | 2010-07-28 |
| File Created | 2010-07-28 |