Appendix C
NAEP Science Cognitive Interview Reports
and
List of New Questions
1. Science survey question cognitive interview study results (January 2013)
2. List of new/revised questions for Wave 1 materials
NAEP Item Development (ID)
2015 Science Survey Questionnaires
2012–2013 Cognitive Interview Study
Grades 4, 8, and 12
Students, Teachers, & School Administrators
Deliverable in response to ID Task 3.1.3
Submitted: January 23, 2013
Listening.
Learning.
Leading.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
Table of Contents
Overview ......................................................................................................................................... 2
Project Objectives ........................................................................................................................... 3
Study Design and Implementation ................................................................................................. 4
Data Collection and Analytic Methods ........................................................................................... 6
ETS Proposed Recommendations ................................................................................................... 8
Student ................................................................................................................................... 9
Teacher ................................................................................................................................ 58
School Administrator ........................................................................................................... 72
Cognitive Laboratories Report and Findings (Eureka Facts) ......................................................... 86
Appendices.................................................................................................................................. 183
1
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
Overview
As required by the National Assessment Governing Board (Governing Board), the National Center for
Education Statistics (NCES) of the U.S. Department of Education will conduct nationwide assessments at
grades 4, 8, and 12 for the National Assessment of Educational Progress (NAEP). In addition to assessing
subject-area achievement, NAEP collects questionnaire data to provide context for the reporting and
interpretation of assessment results. Questionnaire data comes from one of three respondent types:
students, teachers, and school administrators. NAEP questionnaires serve to fulfill reporting
requirements of federal legislation and to provide a context for reporting student performance.
This document presents findings and ETS recommendations based on results from a recent cognitive
interview study of the 2015 Science survey questionnaire items. The report presents project objectives,
study design and implementation, research design, cognitive interview findings, and recommendations
for new or revised items. The report concludes with an appendix, including interview protocols, scripts,
and forms.
As part of the item development process, items undergo several forms of pretesting including expert
panel reviews and cognitive interviews (or cog labs). Cognitive interviews are a form of in-depth, oneon-one pretesting used to assess whether items are being understood as intended and to identify any
problems in respondent comprehension. A goal of cog labs is to identify any problems that warrant
adding, revising, or dropping of items prior to the pilot assessment that will be administered in 2014.
These survey items will be included in all science administrations: paper-and pencil, Hands-on Tasks
(HOTs), and Interactive Computer Tasks (ICTs).
Note that proposed item revisions in the section “ETS Proposed Item Recommendations” are
recommendations for changes to the versions that were administered during the cognitive laboratories.
Deleted text is indicated by strikethrough and new text is highlighted yellow.
2
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
Project Objectives
This cognitive interview study investigates the cognitive processes that respondents use to answer
survey questions. The ability of the respondent to comprehend the questions and provide valid
responses is helpful in identifying the problems and limitations associated with the questions prior to
the 2014 pilot. Early identification of such concerns, prior to administration to a large number of
respondents, will increase the quality of the information collected from the questionnaires by reducing
potentially confusing language and/or improving response categories.
In cognitive interviews, an interviewer uses a structured protocol in a one-on-one interview using two
methods: think-aloud interviewing and verbal probing techniques. With think-aloud interviewing,
respondents are explicitly instructed to “think aloud” (i.e., describe what they are thinking) as they
determine their answers to the survey questions. The interviewer reads each question to the
respondent and then records the cognitive processes that the respondent uses in arriving at an answer
to the question. With verbal probing techniques, the interviewer asks probing questions, as necessary,
to clarify points that are not evident from the “think aloud” process. These probes might include, for
example, asking the respondents to rephrase the question in their own words or assess whether the
response categories are relevant. The specific protocols used in this study are presented later in this
document.
Cognitive interview studies are largely observational. The data collected are predominantly qualitative
verbal reports given in response to probes, think-aloud tasks, and volunteered comments. The objective
is to identify and correct problems of ambiguity or misunderstanding, or other difficulties respondents
have answering questions. The result should be questionnaires that are easier to understand and
therefore less burdensome for respondents while also yielding more accurate information.
3
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
Study Design and Implementation
Recruitment process
EurekaFacts, a subcontractor to ETS, recruited student, teacher, and school administrator participants
for this study within the greater Washington, DC and Baltimore, MD metropolitan areas through email
and telephone outreach. Using an approved screener script via telephone, trained EurekaFacts
interviewers screened parents of student candidates who responded to the outreach.
During the telephone screening, interviewers confirmed the interview date, time, and location with
parents. Student interviews took place at the EurekaFacts cognitive interview laboratory in Rockville,
MD. Legal guardians of the participating students completed consent forms in advance of the cognitive
interviews (see Appendix for recruitment scripts, consent forms, etc.) Students age 18 or older signed
their own consent forms. Teacher and school administrator interviews were conducted at the
respondents’ school or via telephone.
A monetary incentive aimed at ensuring participation and motivation of the participants was provided.
Each participating student earned a $25 Visa gift card in compensation for time and effort. In addition, a
Visa gift card of $25 for the students’ parent(s) was provided to remunerate parents for their time and
to help offset the travel/transportation costs of bringing the participating student to and from the
EurekaFacts cognitive laboratory site. Teachers and school administrators were provided with a $40 Visa
gift card.
Study Sample and Population
Existing research and practice have failed to offer a methodological or practical consensus regarding the
minimum or optimal sample size necessary to provide valid results for background question
development.1 Nonetheless, a sample size of five to fifteen individuals has become the standard. Several
researchers have confirmed the standard of five as the minimum number of participants per subgroup
for analysis (i.e., cell) for the purposes of exploratory cognitive interviewing for question development.2
Other researchers have indicated that although a sample size of five per cell will likely identify major
problems with a question, more is better, up to approximately fifteen per cell.3
Interview Protocols and Processes
Participants were first welcomed, introduced to the interviewer and the observer, and told they were
there to help answer questions about how people answer survey questions. Participants were reassured
that their participation was voluntary and that their answers may be used only for research purposes
1
Almond, P. J., Cameto, R., Johnstone, C. J., Laitusis, C., Lazarus, S., Nagle, K., Parker, C. E., Roach, A. T., & Sato, E.
(2009). White paper: Cognitive interview methods in reading test design and development for alternate
assessments based on modified academic achievement standards (AA-MAS). Dover, NH: Measured Progress and
Menlo Park, CA: SRI International.
2
Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think-aloud method: A practical guide to
modeling cognitive processes. San Diego, CA: Academic Press.
3
Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.
4
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
and may not be disclosed, or used, in identifiable form for any other purpose except as required by law
[Education Sciences Reform Act of 2002, 20 U.S.C §9573].
Interviewers explained the cognitive interviewing process, conducted a practice question, and then
participants answered questions verbally.
Interviewers used several different cognitive interviewing techniques, including general think-aloud and
question-specific probes, observation, and debriefing questions.
The interview protocols were developed to facilitate capturing respondents’ thought processes while
answering the questionnaire items. The interviewers encouraged participants to speak aloud all their
thoughts while answering the questions and asked follow-up probes after each item. The probes were
designed to help participants elaborate their responses.
If a participant did not appear to fully think aloud their response, interviewers used “continuers” to
encourage the participant to be more descriptive and verbalize their thoughts without “putting words in
their mouth.” Interviewers were instructed to remain as objective and unbiased as possible, but to offer
a verbal “nudge,” such as:
What are you thinking now?
Any other thoughts?
Tell me how you came to pick that answer.
If a participant exhibited difficulty reading or understanding an item, interviewers used probes similar to
the examples below:
Can you tell me in your own words what that question was asking? [comprehension and
interpretation probe]
What does the word [term] mean to you as it is used in this question? [comprehension
and interpretation probe]
How much would you say you know about [topic]? [confidence probe]
How sure are you of your answer? [confidence probe]
How did you come up with that answer? [recall strategy and bias probe]
Was this easy or hard to answer?” [comprehension and recall probe]
How easy or difficult is it to remember [topic]? [recall probe]
Did you find the list of answer choices easy or hard? [response probe]
How easy or hard was it to choose an answer from that list of choices? [response probe]
The question uses the word [term]. Does that sound OK to you, or would you choose
something different? [sensitivity probe]
In general, how do you feel about this question? [sensitivity probe]
The generic probes were designed to understand why the question as written was not clear, and
interviewers were instructed to reword the question in a way that the participant would understand
based on shared interviewer feedback or previous interview experiences. Any instances of reworded
questions that were tested have been documented in the “Findings” section of this report.
In addition to generic probes, interviewer protocols included item-specific probes. Not all items had an
item-specific probe. Item-specific probes were provided only if the researchers deemed it important to
5
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
learn more about a cognitive process or issue that might not be raised naturally from the think-aloud
process or generic probing.
The interview protocols can be found in the Appendix.
Data Collection and Analytic Methods
The primary objective of the analysis was to identify items with problems and understand the nature of
those problems. In the course of cognitive interviewing, some type of response difficulty will likely occur
for almost every question. Our analytical approach is designed to identify the items that posed
difficulties to respondents in ways that (1) appeared to reduce the validity of the response, (2) could be
predicted to occur at a statistically significant rate in a larger population, and (3) exhibited a logical basis
for why the problem occurred.
Interview teams recorded their notes into a Microsoft Excel data entry spreadsheet. The spreadsheet is
based on a template developed by ETS researcher Zhitong Yang. The spreadsheet is designed to
facilitate the collection and analysis of cognitive interview notes based on Roger Tourangeau’s fourstage task-analytical cognitive model, as elaborated and adapted by Gordon Willis,4 described below.
There are four primary questions regarding respondents’ survey response processes that are explored
through cognitive interviewing:
1. Comprehension: What does the respondent believe the question to be asking?
2. Memory: How does the respondent recall the information needed to answer the question?
3. Decision: How does the respondent approach (e.g., effort, sentiment) answering the question
accurately and truthfully?
4. Response: Does the respondent’s answer match the given or planned response categories?
To collect information to identify problems with respondents’ survey response processes, we asked
interviewers to explicitly code the occurrence of six different threats to obtaining valid and reliable
survey responses.
1. Did the respondent skip over, misread, or reread any word(s)?
2. Did the respondent exhibit difficulty reading an item?
3. Did the respondent exhibit difficulty understanding the question?
4. Did the respondent exhibit difficulty understanding the meaning of particular words or
concepts?
5. Did the respondent exhibit difficulty remembering the question or answer choices?
6. Did the respondent exhibit difficulty or discomfort answering the question?
In addition to the explicit coding to identify problems, interviewer data entry sheets also include
descriptions of identified problems, summarizations of respondent verbal reports, and respondent
responses to item-specific probes.
For the purposes of this report, the analysts focused on providing and presenting evidence to better
assist in the item-level adjudication process. Consequently, most of the findings reported in this report
4
Willis, G. (2005). Cognitive Interviewing: A Tool for Improving Questionnaire Design. Thousand Oaks, CA: Sage.
6
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
are centered on potential problems with items and discussion of the revisions to items that could best
correct for the identified problems.
A four-stage process was used to identify problems with items. In the first stage, explicit codes related
to the six aforementioned threats to obtaining valid and reliable survey responses were evaluated to
highlight problematic items. In the second stage, the descriptions of identified threats were then
reviewed to determine the nature of the problem. In the third stage, respondents’ verbal reports and
respondents’ responses to item-specific probes were then reviewed to locate logical bases for the
problems. In the fourth stage, analysts and item developers met to discuss the findings to identify
potential approaches to item revision. These data collection and analytical methods are the basis for the
results presented in the following findings section.
7
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
ETS Proposed Item Recommendations
8
Student Questionnair e–Gr ade 4
[Grade 4 Science–Student #1]
New (Revised based on VC519362)
In this school year, how often have you done activities or projects to learn about
electricity (for example, circuits and energy)?
A.
B.
C.
D.
Never
Rarely
Sometimes
Often
Cognitive lab results:
1. There was much variation in how students understood the phrase “circuits and energy,” and two of
the ten students reported not knowing what the phrase “circuits and energy” meant. They reported
that “batteries and light” is clearer to them.
2. Four students reported that they would have responded to the item differently if the phrase
“batteries and light” was used in place of “circuits and energy.”
Rationale:
1. Cognitive lab results show that grade 4 students’ understanding of the phrase “circuits and energy”
varies, and some students do not know what the phrase meant.
2. Cognitive lab results also show that grade 4 students shared a common understanding of the
phrase “batteries and light” and correctly linked the example to learning about electricity.
Recommendation:
1. ETS recommends using the trend phrase “batteries and light” instead of revised phrase “circuits
and energy.”
9
[Grade 4 Science–Student #2]
VC315219
In this school year, how often have you done activities or projects to learn about
chemicals (for example, mixing sugar or salt in water)?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. When asked how they would respond to the item if they had done activities with chemicals only at
certain times during the school year, the methods used by grade 4 students to calculate the rate
vary. Some provided the average rates; some others provided a frequency based on the particular
period of time when they had the activities. Other grade 4 students did not understand the probe.
2. Compared with the revised version of this item (using “Never,” “Rarely,” “Sometimes,” and
“Often” as options), more than half of students (6 of 10) preferred the response options in this
item, and three stated they liked these response options because they are more concrete.
Rationale:
1. Cognitive lab shows that grade 4 students recall the information based on two major methods:
frequency within the particular period of time when they have the activities, and averaging the
frequency across the whole school year. Reporting frequency within a particular period is more
common among grade 4 students.
2. Grade 4 students’ interpretation of the revised response options (“Never,” “Rarely,” “Sometimes,”
and “Often”) vary and are vague.
Recommendation:
1. Keep the trend item in the 2014 pilot administration.
10
[Grade 4 Science–Student #3]
New (Revised based on VC315219)
In this school year, how often have you done activities or projects to learn about
chemicals (for example, mixing sugar or salt in water)?
A.
B.
C.
D.
Never
Rarely
Sometimes
Often
Cognitive lab results:
1. Grade 4 students’ interpretation of the revised response options (“Never,” “Rarely,” “Sometimes,”
and “Often”) vary and are vague.
2. When asked how they would respond to the item if they had done activities with chemicals only at
certain times during the school year, most students chose the more frequent response choice,
implying that they tend to report the frequency based on the particular period time when they had
the activities.
Rationale:
1. Grade 4 students’ interpretation of the revised response options (“Never,” “Rarely,” “Sometimes,”
and “Often”) vary and are vague.
2. Fewer students preferred these response options in this item.
Recommendation:
1. Drop the revised version of the item in the 2014 pilot administration.
11
[Grade 4 Science–Student #4]
New (Revised based on VC315222, VC315227, and VC315229)
In this school year, how often have you done science activities using scientific tools?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. All students displayed a good understanding of the item.
2. Most students provided examples of “scientific tools.” The “magnifying glass” was the most cited
tool; others included “pH testing strip” and “dropper.”
3. When being questioned which of the three versions of the answer choices made the most sense,
the opinions were divided. Three students said the answer choices in this item made more sense to
them because it provide more details; two students preferred response options “Never,” “Rarely,”
“Sometimes,” and “Often” stating they are easier; five students reported that the response options
“No,” “Yes, a little,” and “Yes, a lot” made more sense because they are simple and fit their
spontaneously generated answers.
4. When asked how they would respond if they had only used tools at certain times of the year,
students typically used two major methods to recalculate the frequency of use 1) recall the
frequency within the particular period, and, 2) average the frequency across the whole school year.
5. The cognitive lab results show that most students answered the question based on the frequency
within the particular period of time when they have science classes.
Rationale:
1. Since all students displayed a good understanding of the item, the revised item stem works well.
2. Most students’ spontaneous answers were based on the frequency of scientific tool use within the
particular period of time when they had the science activities.
3. Students’ are divided among which of the three sets of response options they prefer. Therefore, the
current response options should be used to maintain trend.
Recommendation:
1. Replace current items VC315222, VC315227, and VC315229 with this item in the 2014 pilot
administration.
12
[Grade 4 Science–Student #5]
New (Revised based on VC315222, VC315227, and VC315229)
In this school year, how often have you done science activities using scientific tools?
A.
B.
C.
D.
Never
Rarely
Sometimes
Often
Cognitive lab results:
1. All students displayed a good understanding of the item.
2. When asked to define “rarely,” “sometimes,” and “often,” students either used subjective, vague
terms, or referred to a frequency similar to the trend response options (e.g., “once very few
weeks”) to explain.
Rationale:
1. Although a few students preferred the response options in this item, when asked to define the
options, they either used vague terms or used a frequency similar to the trend options.
Recommendation:
1. Do not use this item in the 2014 pilot administration.
13
[Grade 4 Science–Student #6]
New (Revised based on VC315222, VC315227, and VC315229)
Have you used scientific tools in school this year?
A. No
B. Yes, a little
C. Yes, a lot
Cognitive lab results:
1. All students displayed a good understanding of the item.
2. When asked to define “Yes, a little,” and “Yes, a lot,” students either used vague terms, or referred
to the frequency response similar to the trend response options (e.g., “once very few weeks”) to
explain.
3. Half of the students (5 of 10) preferred the response options in this item because they are easier.
Rationale:
1. Although half of the students (5 of 10) preferred the response options in this item, when being
asked to define the options, they either used vague terms or used the frequency similar to the trend
options to provide explanation.
Recommendation:
1. Do not use this item in the 2014 pilot administration.
14
[Grade 4 Science–Student #7]
VC315239
In this school year, how often have you read a book or magazine about science?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. All students displayed a good understanding of the item.
2. Out of 4 students who stated that they read books about science that are not for school work, only
one said that he/she included it in selecting the choices.
3. Of the five students who responded that they had read a book or magazine about science in this
school year, all indicated that they read most often in the classroom.
Rationale:
1. The cognitive lab results show that grade 4 students read books or a magazine about science most
often in school.
2. There is no need to specify whether grade 4 students read books or magazines that are for school
or not for school.
Recommendation:
1. Keep the trend item in the 2014 pilot administration.
15
[Grade 4 Science–Student #8]
VC315259
In this school year, how often have you presented what you learned about science to
your class?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. Two students showed hesitation in pronouncing the word “presented.”
2. All students displayed a good understanding of the item.
3. When asked if “to show what you have learned” was different from “to present what you have
learned,” most students (6 of 10) said that these were the same. Only one student said that they
differ, and one student did not understand what “show what you have learned,” meant.
4. Six of the ten students indicated that they have presented something to their class. Of these six,
four presented information that they learned, and the other two presented a project that they
worked on.
Rationale:
1. The cognitive lab results show that grade 4 students understand the word “presented,” and
interpreted the word similar to the word “showed.”
Recommendation:
1. Keep the trend item in the 2014 pilot administration.
16
[Grade 4 Science–Student #9]
VC315288
In this school year, how often have you talked about measurements or results from your
science activities or projects?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. When asked to define “talked about,” the majority of students (8 of 10) answered in a similar
manner, stating that it meant to “say something” or “talk to someone.” Only one student did not
know how to respond.
2. When asked to define the term “measurements or results,” the students showed good
understanding of these terms, and explained by referring to results, the end-effect, or product of
experiments, and solutions to problems. Two students used examples to help illustrate what they
were trying to say.
3. Seven of the ten students stated that they “never or hardly ever” talked about measurements or
results. The remaining three reported that they talked about measurements or results with a
teacher, classmates, or the whole class.
Rationale:
1. The cognitive lab results show that grade 4 students understand the phrase “talked about.”
Recommendation:
1. Keep the trend item in the 2014 pilot administration.
17
[Grade 4 Science–Student #10]
VC315265
In this school year, how often have you written a report on your science activities or
projects?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. Interviewer notes report no major problems with comprehension in this item. When asked to
paraphrase the item, the students displayed a good understanding of the item.
2. All students but one defined “report” in a similar manner as writing or sharing information from
measurements or science projects with other people.
3. Half of the students (5 of 10) responded that they had “never or hardly ever” written a report this
school year; two responded “once every few weeks,” and three said, “two or three times a week.”
Rationale:
1. The cognitive lab results show that grade 4 students understand the word “report.”
2. The revised version of the item (item 11) produced response that is more thorough.
Recommendation:
1. Replace the item with the revised version of the item (item 11) in the 2014 pilot administration.
18
[Grade 4 Science–Student #11]
New (Revised based on VC315265)
In this school year, how often have you been asked to write about your science activities
or projects (such as reports, science journals, or lab write-ups)?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. Most students (9 of 10) displayed a good understanding of the item.
2. All students had a similar understanding of the phrase “science journal,” describing it as
something you write in to keep track of information you have for science topics.
3. Four of the ten students would call the written work they do for a science activity or project a
“science journal” (these same four students were unable to define a “lab write-up”). Two students
would call the work a “report,” while two students would call it something else (a “writing piece”
and “results”), and one was unsure what to call it. Only one student said he/she would call the
work a “lab write-up,” but did not know why.
4. The descriptions that students provided for the phrase “lab write-up” were less consistent.
Rationale:
1. The cognitive lab results show that, for grade 4 students, this item captured more comprehensive
information than the trend item in which only the frequency of report writing was asked. Although
the interviewer notes for the previous item (item ten) show that all but one student had a good
understanding of the term “report,” the more generic formulation of item eleven (“asked to write
about your science activities or projects”) produced a different answer pattern among students.
This suggests that the item ten did not capture all possible information from students on this topic.
2. To further clarify the intent of the item, ETS suggests adding common examples (i.e., “such as
reports, science journals, or lab write-ups”) to the item stem.
Recommendation:
1. Replace the trend item with this revised version in the 2014 pilot administration.
19
[Grade 4 Science–Student #12]
VC315266
In this school year, how often have you used your school library resources for science
(such as books, magazines, computers, and audio-video materials)?
A.
B.
C.
D.
E.
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Cognitive lab results:
1. Two students could not define the phrase “library resources.” Of the remaining eight students, all
gave books as an example of a library resource. Three also gave the example of computers, and
two others gave the example of magazines.
2. Only after being probed, did students acknowledge the internet as being a library resource. After
being asked, six students said that internet was a library resource, while three said that it was not.
When probed, two students also acknowledged computers as library resources; another student
was unsure, because computers in his/her school were not for fourth graders.
3. Nine students were asked whether they thought of their school library, the public library, or both
in selecting the options. Of the nine, seven thought only of the school library, one thought of both,
and one answered that they “do not go to the library.”
4. When explicitly asked about individual resources, three students included encyclopedias, and three
included videos, as library resources.
Rationale:
1. The cognitive lab results show that a majority of students thought only of books as library
resources.
2. To capture all the other library resources used by student, ETS suggests adding common examples
to the item stem (i.e., “such as books, magazines, computers, and audio-video materials”).
Recommendation:
1. Revise and keep the item in the 2014 pilot administration.
20
[Grade 4 Science–Student #13]
VC315410
How often do you do science activities that are not for schoolwork?
A.
B.
C.
D.
Never or hardly ever
Sometimes
Often
Always or almost always
Cognitive lab results:
1. Overall, the students understood the phrase “science activities” in a similar manner: eight of the
ten students described it as activities or projects that one does, that are related to a science topic.
Only two did not explicitly refer to a science topic.
2. Interviewer notes demonstrated a good and consistent understanding of the phrase “not for
schoolwork” among the students: five students said it meant activities done at home, and three
said it meant being “out of school.”
3. When asked to provide examples of activities they had done that were not for schoolwork, two
students gave technology/engineering examples (e.g., fixing a TV), two gave natural science
examples (e.g., studying bugs), and two gave chemical examples (e.g., mixing substances
together). Two students who answered “never or hardly ever” could not remember any examples
of activities they had done that were not for schoolwork.
Rationale:
1. The cognitive lab results show that a majority of students have a good understanding of the item as
intended.
Recommendation:
1. Keep the trend item in the 2014 pilot administration.
21
[Grade 4 Science–Student #14]
New
In this school year, have you participated in a science club, a science fair, or a science
competition?
A. Yes
B. No
Cognitive lab results:
1. When asked to define a “science fair,” six students described it as related to making experiments;
of these six, two also stated that “science fair” involved showing their work to other people. One
student defined it as showing something, but did not mention experiments.
2. Two students defined science fairs as having a competitive element; one said first place gets a
prize, and the other said science fair was synonymous with science competition.
3. Two students did not understand what a “science fair” was; they knew only that it had something
to do with science.
4. When asked to provide an example of a science fair, there were few prevailing trends. Six students
stated that they had not participated in one; two referred to “event nights” at their schools where
science activities took place, but they did not present their own experiments.
Rationale:
1. Cognitive lab results show that grade 4 students do not differentiate between the three scientific
activities (i.e., “science fair,” “science competition,” and “science club”).
2. ETS suggests combining the three scientific activities in one question for grade 4 questionnaire.
Recommendation:
1. Revise the item combining the three scientific activities and add the item to the 2014 pilot
administration.
22
[Grade 4 Science–Student #15]
New
In this school year, have you participated in a science club?
A. Yes
B. No
Cognitive lab results:
1. When asked to define a “science club,” all students provided responses indicating that science
clubs had to do with people meeting together to learn more about science or do science activities.
Additionally, four students explicitly said they were after-school activities.
2. When asked to provide an example of a science club, four students provided concise examples of
specific clubs, three could only give examples of activities they had done in the clubs, and one
student had never participated in a club. Two students could not recall any examples of science
clubs.
Rationale:
1. Cognitive lab results show that grade 4 students do not differentiate between the three scientific
activities (i.e., “science fair,” “science competition,” and “science club”).
2. Suggest combining the three scientific activities in one question for grade 4 questionnaire.
Recommendation:
1. Revise the item combining the three scientific activities and add the item to the 2014 pilot
administration.
23
[Grade 4 Science–Student #16]
New
In this school year, have you participated in a science competition?
A. Yes
B. No
Cognitive lab results:
1. When asked to define a “science competition,” five students gave responses indicating that this
activity involves “prizes” or “first place, second place, etc.” Six students said it involved either
doing your own project/doing science, or that it was like a science fair.
2. When asked to provide an example of a science competition, three students either gave “science
fair” as their example, or gave their same example from item fourteen (which asked for an
example of a science fair). One student said fairs and competitions were similar, but not the same.
3. Students were asked whether science fairs, science clubs, and science competitions were different
things. Over half (6 of 10) said that they were all different things; three said they were all the
same; one said that fairs and competitions were the same, but clubs were different. All three of the
students who indicated that they were all the same gave the reason that they all dealt with science.
Rationale:
1. Cognitive lab results show that some grade 4 students do not differentiate between the three
scientific activities (i.e., “science fair,” “science competition,” and “science club”).
2. Suggest combining the three scientific activities in one question for grade 4 questionnaire.
Recommendation:
1. Revise the item combining the three scientific activities and add the item to the 2014 pilot
administration.
24
[Grade 4 Science–Student #17]
New
In this school year, have you visited a science museum?
A. Yes
B. No
Cognitive lab results:
1. The cognitive results show that there were some variations in the definitions of “science museum”
among grade 4 students. However, most (7 of 10) indicated that it involved scientific activities,
lessons, and/or technology, or gave appropriate examples. Two students were unsure of the
meaning, and one student seemed to include non-scientific museums in their definition.
2. Nine students were asked what specific museums they considered science and non-science
museums. Two students could not remember any science museums; two students could not
remember any non-science museums.
3. When students were asked what museums they thought of as science museums, examples
included: The Museum of Natural History, the Space Museum, and an animal museum called
Amazing Animals.
4. When students were asked what museums they thought of as non-science museums, examples
included: The National History Museum, the Wax Museum, and the Native American Museum.
Rationale:
1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science
museum” with some variations.
2. Since the intent of the item is to measure students’ “out of school” science activities, ETS suggests
not using the “not on a school trip” version of this item.
Recommendation:
1. Do not use this item in the 2014 pilot administration.
25
[Grade 4 Science–Student #18]
New
In this school year, have you visited a science museum on a school trip?
A. Yes
B. No
Cognitive lab results:
1. When asked to define “on a school trip,” three students stated it was a trip involving their class
and/or grade; three students mentioned riding on a school bus; three students said it was a field
trip; two students simply re-worded the phrase (e.g., “a trip for school”).
2. Of the five students who provided examples, two referenced museums, and two could only discuss
the activities they did at the museums, (one simply referenced a field trip to the Chesapeake Bay).
The referenced museums were The Animal Museum and the Kid’s Museum in Baltimore Harbor.
3. Only one student had been to a science museum on a school trip this year.
Rationale:
1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science
museum” with some variations.
2. Since the intention of the item is to measure students’ “out of school” science activities, ETS
suggests omitting the “not on a school trip” version.
Recommendation:
1. Do not use this item in the 2014 pilot administration.
26
[Grade 4 Science–Student #19]
New
In this school year, have you visited a science museum that was not on a school trip?
A. Yes
B. No
Cognitive lab results:
1. Seven of the ten students displayed good understanding of the item.
2. When asked to define “not on a school trip,” seven of the ten students stated it was a trip that did
not include school individuals or classmates, while five students described it as involving family
and/or friends.
3. Only one student indicated that they had visited a science museum this year that was not on a
school trip they went once.
4. Five students provided the following examples of trips that were not for school: The Air and
Space Museum, the Natural History Museum, and the American Indian Museum.
Rationale:
1. Cognitive lab results show that grade 4 students have a basic understanding of the term “science
museum” with some variations.
2. Most of the students (7 of 10) displayed good understanding of the phrase “not on a school trip” as
the phrase intended.
3. Since the intention of the item is to measure students’ “out of school” science activities, ETS
suggests using this version of the item.
Recommendation:
1. Add this item to the 2014 pilot administration.
27
Student Questionnair e–Gr ade 8
[Grade 8 Science–Student #1]
VC304986
In your science class this year, have you done hands-on activities or projects with any of
the following? Fill in one oval on each line.
a. Living things (for example,
plants, animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving sugar or
salt in water)
d. Rocks or minerals (for
example, identifying types)
e. Magnifying glass or
microscope (for looking at
small things)
f. Thermometer or barometer (for
making measurements)
g. Simple machines (for example,
pulleys and levers)
Yes
No
A
B
A
B
A
B
A
B
VC305007
A
B
VC305008
A
B
VC305009
A
B
VC305012
VC304988
Revised sub-item
based on VC304989
VC304991
Cognitive lab results:
1. In regards to the meaning of the phrase “circuits and energy,” students’ responses varied greatly.
While two students understood a circuit as something consisting of wires, another two understood
it as something that electricity moves through. Two students understood energy to be synonymous
with the term “power.” Four students provided a limited and vague definition for the phrase,
showing a partial understanding that “circuits and energy” referred to electricity. Two of the
students were unable to provide a definition.
2. Of the four students who answered “yes” to sub-item b, three recalled that the activity they
thought of when formulating their answer was related to using wires to light a light bulb. The
remaining student recalled learning about electricity, but could not provide examples of specific
activities related to this topic.
3. When asked how they would respond to sub-item b if the example had been “batteries and light”
instead of “circuits and energy,” half (5 of 10) responded that they would have changed their
answers. Four of these students would have changed their answers from “no” to “yes,” while the
fifth would have changed from “yes” to “no.”
28
Rationale:
1. There was significant variation in how students understood the phrase “circuits and energy.” In
terms of their recollections of activities relating to electricity, students predominantly recalled
activities where they had used batteries, wires, and/or light bulbs. In addition, probing revealed
that if sub-item b included the example of “batteries and light,” instead of “circuits and energy,”
twice as many students would have answered “yes” to this sub-item. Thus, “batteries and light”
appears to be the most relevant examples for grade 8 students.
2. Although grade 8 students comprehend the item, there could be better distinction between
response options as shown in the following two items.
Recommendation:
1. For sub-item b, ETS recommends using the trend version examples “batteries and light” for the
grade 8 student questionnaire in the 2014 pilot administration.
2. ETS recommends using the revised version of the item as shown in the student item #3 (with
response options “Never,” “Rarely,” “Sometimes,” and “Often”) in the 2014 pilot administration.
29
[Grade 8 Science–Student #2]
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
a. Living things (for
example, plants,
animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving
sugar or salt in water)
d. Rocks or minerals (for
example, identifying
types)
e. Magnifying glass or
microscope (for looking
at small things)
f. Thermometer or
barometer (for making
measurements)
g. Simple machines (for
example, pulleys and
levers)
Never or
hardly ever
Once every
few weeks
About once
a week
Two or
three times
a week
Every day
or almost
every day
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
VC305007
A
B
C
D
E
VC305008
A
B
C
D
E
VC305009
A
B
C
D
E
VC305012
VC304988
Revised subitem based on
VC304989
VC304991
Cognitive lab results:
1. All students displayed a good comprehension of the item.
2. When asked how they would respond if they had only done activities or projects at certain times
during the school year (e.g., doing an activity every day for three weeks, but not since then),
students provided a variety of responses.
3. Cognitive lab results show that grade 8 students typically use two methods to respond to the
hypothetical question if they had only done activities or projects at certain times during the school
year: recall an experience within a particular period, or averaging the frequency across the entire
school year.
4. Four of the 10 students expressed that the response options are hard to follow and do not align
well with their science course schedule.
Rationale:
1. Although grade 8 students comprehend the item, the response options are difficult to follow.
Recommendation:
1. ETS recommends not using this item in the 2014 pilot administration.
30
[Grade 8 Science–Student #3]
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
a. Living things (for example,
plants, animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving sugar or
salt in water)
d. Rocks or minerals (for
example, identifying types)
e. Magnifying glass or
microscope (for looking at
small things)
f. Thermometer or barometer (for
making measurements)
g. Simple machines (for example,
pulleys and levers)
Never
Rarely
Sometimes
Often
A
B
C
D
A
B
C
D
A
B
C
D
A
B
C
D
VC305007
A
B
C
D
VC305008
A
B
C
D
VC305009
A
B
C
D
VC305012
VC304988
Revised sub-item
based on VC304989
VC304991
Cognitive lab results:
1. Six students defined the response options using more subjective and vague terminology, with
“rarely” meaning “not much,” “sometimes” meaning “once in a while,” and “often” meaning “a
lot.” Of these six students, four defined “rarely” in terms of comparisons to the other categories
(“not often,” or “less than sometimes”). The definitions for “sometimes” and “often” varied
among these six students, but their definitions showed that they understood the concept in all
cases.
2. Four students defined the response options according to more objective descriptions. Among these
respondents, the term “rarely” was defined by all four students as either “once,” or “once a
month”; the term “sometimes” had varying definitions; the term “often” was defined by all four
students as once or multiple times a week.
3. When asked which version of the response options they preferred, nine of the ten students said
they preferred the response options used in this item. Many implied that answering the item with
the dichotomous response options used in item one was easy and these options were too broad to
assess their internally generated answers. Moreover, the students explained that it was difficult to
answer the item with frequencies, as used in item two, because the response options were too
specific.
31
Rationale:
1. Verbal reports and answers to probing questions show that the students strongly preferred the
response option scale used in item three, as compared to items one and two. It appears that the
students could more easily fit their internally generated answers into the response options provided
in item three; in addition, these response options placed a lower overall cognitive burden upon
students.
Recommendation:
1. ETS recommends replacing the trend item VC304986 with this revised version of the matrix item
in the 2014 pilot administration.
2. For sub-item b, ETS recommends using the trend wording of the sub-item (using examples
“batteries and light”) in the 2014 pilot administration.
32
[Grade 8 Science–Student #4]
VC305292
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
a. Read a science
textbook, in class or at home
b. Read a book or
magazine about science topics
c. Use the Internet to learn about
science topics
d. Watch a movie, video, or DVD
about science topics
Never or
hardly
ever
Once
every
few weeks
About
once
a week
Two or
three
times
a week
Every day
or
almost
every day
A
B
C
D
E
VC546510
A
B
C
D
E
Revised subitem based on
VC305295
A
B
C
D
E
Revised subitem based on
VC720562
A
B
C
D
E
Revised subitem based on
VC305307
Cognitive lab results:
1. When asked to rephrase the item in their own words, the students displayed a good understanding
of the item or instructions.
2. When asked what the phrase “science topics” meant, five students implied that this meant subjects
studied in science class, two students gave discrete examples of specific science topics, and two
answered that this phrase meant “anything” that has to do with science.
3. When asked if they would respond differently if the item used the term “science” instead of
“science topics,” seven said they would answer differently. The most common reason given for
this change of response among students (4 of 7) was that the term “science” was broader than
“science topics,” and would allow them to consider more types of media as related to “science” in
their responses.
Rationale:
1. The item is classified under the issue “Organization of Instruction” and sub-issue “Instructional
Strategies.” Adding the word “topics” to the relevant sub-items will help clarify the intent of the
item.
Recommendation:
1. ETS recommends using the revised version of the three sub-items in the 2014 pilot administration.
33
[Grade 8 Science–Student #5]
VC720622
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
a. Identify questions that can be
addressed through science
experiments
b. Design a science experiment
c. Talk about measurements you
took for your science project or
activity
d. Talk about the results of your
science project or activity
e. Watch your teacher do a
science experiment or activity
f. Make graphs or charts of the
results from your science
project or activity
g. Write a report on your science
project or activity (such as
reports, science journals, or lab
write-ups)
h. Write about your science
activities or projects (such as
reports, science journals, or lab
write-ups)
h. Write a lab write-up on your
science project or activity
i. Write a science journal
Never or
hardly
ever
Once
every
few weeks
About
once
a week
Two or
three
times
a week
Every day
or
almost
every day
A
B
C
D
E
VC720623
A
B
C
D
E
VC720634
A
B
C
D
E
VC720635
A
B
C
D
E
VC720636
A
B
C
D
E
VC720638
A
B
C
D
E
VC720639
A
B
C
D
E
VC720641
A
B
C
D
E
New
A
B
C
D
E
New
A
B
C
D
E
New
Cognitive lab results:
1. When asked, half of the students (5 of 10) indicated that the phrase “talk about” in sub-items c and
d implied summarizing what they did or learned in class. The other five students defined the
phrase as generally talking with people (students or teachers) in their class.
2. When asked what is meant by the term “lab write-up,” half of the students (5 of 10) described it as
something akin to a report, describing a science lab that they had completed. Three students
described it as another type of class assignment unrelated to a science lab. Two students were
unable to provide a definition.
3. In regards to the meaning of the term “science journal,” responses varied greatly, and included the
following: where they write their “warm-up” exercise (2 students); where to write notes (2
students); where to write about experiments (2 students); something that is used to describe what
happened in class (2 students); where one can write about science in general (2 students).
34
Rationale:
1. Interview notes revealed considerable variation in how students understood the phrases “lab writeup” and “science journal.” Considerable overlap between the two phrases exists in some students’
interpretations. To avoid confusion and loss of data, ETS suggests revising the two sub-items by
using the more generic phrase to describe the activities, i.e., “write about your science activities or
projects (such as reports, science journals, or lab write-ups).”
Recommendation:
1. ETS recommends using the revised sub-item h in the 2014 pilot administration.
35
[Grade 8 Science–Student #6]
VC315266
In this school year, how often have you used your school library resources for science
(such as books, magazines, computers, and audio-video materials)?
A. Never or hardly ever
B. Once every few weeks
C. About once a week
D. Two or three times a week
E. Every day or almost every day
Cognitive lab results:
1. When probed, seven students stated that they were thinking about their school’s library when
answering this item and three stated that they were thinking of both their school’s library and a
public library.
2. In regards to what students thought of as library resources, the top library resource identified by
students was books (9 of 10). Other resources that students mentioned, unaided, were computers
(4 students), internet (2 students), encyclopedias (2 students), dictionaries (2 students), videos (1
student), and magazines (1 student).
3. When asked directly about each resource, eight students said they thought of computers as
resources while answering the item, seven students thought of encyclopedias, five thought of the
internet, five of magazines, and four thought of videos.
Rationale:
1. Not all students’ spontaneous responses included library resources other than books. ETS suggests
adding specific examples at the end of the item to be consistent with the item used in grade 4 and
grade 12 questionnaires.
2. Since the item is under issue “Availability and Use of Instructional Recourses” and sub-issue
“Products,” ETS suggests clarifying that the question as asking about “school library” uses.
Recommendation:
1. Revise and keep the item in the 2014 pilot administration.
36
[Grade 8 Science–Student #7]
VC305330
Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill
in one oval on each line.
a. I do science-related activities that are not for
schoolwork.
b. I like science.
c. Science is one of my favorite subjects.
d. I take science only because I have to.
e. I take science only because it will help me in
the future.
Strongly
disagree
Disagree
Agree
Strongly
agree
A
B
C
D
A
A
A
B
B
B
C
C
C
D
D
D
A
B
C
D
VC305348
VC305350
VC305351
VC305352
VC305353
Cognitive lab results:
1. Interviewer notes do not indicate any consistent problems in students’ understanding of the item or
instructions.
2. When asked what was meant by the phrase “science-related activities” in sub-item a, the students
either provided general, broad descriptions related to activities that had “anything to do with
science,” or offered some examples such as measurements, experiments, and research.
3. When asked what was meant by the phrase “not for schoolwork,” five students emphasized in their
responses that these activities were done outside of school, or were not assigned schoolwork. Four
students emphasized in their explanations that these activities were done based on personal
interests.
Rationale:
1. Cognitive lab results show that all students understood and interpreted the item as intended.
Recommendation:
1. ETS recommends keeping the trend item in the 2014 pilot administration.
37
[Grade 8 Science–Student #8]
VC304978
In your science class this year, which of the following topics have been covered? Fill in one oval on each
line.
a.
b.
c.
d.
Life science (for example, biology, the human body, or ecology)
Physical science (for example, energy, physics, or chemistry)
Earth and space science (for example, geology or astronomy)
Engineering and technology (for example, designing solutions to
problems)
Yes
A
A
A
No
B
B
B
A
B
VC304982
VC304983
VC304984
VC304985
Cognitive lab results:
1. When asked what was meant by the term “covered” in the item stem, seven students provided a
general, broad definition for the term (“gone over,” “what we learned”). In their definitions, three
of the students included a term that described the extent of the learning (“we learned about it
completely,” “we know it thoroughly”).
2. Students that answered “yes” to any of the sub-items were asked to describe how the material was
“covered” in their class. A majority of the students (8 of 9) answered this by providing the names
of the topics that were covered. All of the topics mentioned by students fit within the existing subitem’s category. One student described the types of assignments and materials that were used to
learn each topic.
3. When probed, seven students said that they would prefer a larger scale that asked to what extent
the topics were covered, as opposed to “yes” and “no” response options. The main reason for this
preference, according to five of these seven students, was the ability to more accurately assess the
extent to which these topics were covered. Of the three students who preferred the “yes/no”
format, two had previously included a term that described the extent of learning in their
description of the term “covered.”
Rationale:
1. Cognitive lab results show that all students displayed a good understanding of the item and the
word “covered.”
Recommendation:
1. ETS recommends keeping the trend item in the 2014 pilot administration.
38
[Grade 8 Science–Student #9]
New
In this school year, have you participated in any of the following activities? Fill in one
oval on each line.
a. Science fair
b. Science club
c. Science competition
Yes
A
A
A
No
B
B
B
Cognitive lab results:
1. When asked what was meant by the term “science fair” in sub-item a, all ten students provided
explanations which indicated that science fairs had to do with “experiments” or “projects.” Seven
of the ten explanations also indicated that these projects were shown or talked about to others.
a. Eight students were able to provide an example of a science fair in which they or a
classmate had participated.
2. When asked what was meant by the term “science club” in sub-item b, students had varying
answers. Five students indicated that these clubs occurred after school; one student said that they
occur annually, and the other four students did not discuss when clubs occurred. Four students
included in their definition that the purpose of the club was to do science activities, and three
noted that the purpose of the club was to discuss or talk about science.
a. Five students were able to provide an example of a science club in which they or a
classmate had participated.
3. When asked what was meant by the term “science competition” in sub-item c, nine students
provided a definition that described it as similar to a science fair, where one makes a project,
invention, or experiment and competes against others. One student mentioned that people also
compete by answering questions. One student was unable to define the term.
a. Only one student was able to provide an actual example of a science competition, which
they described as a “science club competition, where they had to build things to compete in
certain challenges.” One student answered that a geography bee was a science competition.
The other eight students were unable to retrieve a cogent example.
4. All ten students answered “no” to sub-items a and c, and nine answered “no” to sub-item b.
Rationale:
1. Most students comprehend the item as intended.
2. Although almost all students indicated that they have not participated in any of the activities, ETS
suggests that the item could be still used to measure the growth of out-of-school science activities.
Recommendation:
1. ETS recommends adding the item to the 2014 pilot administration.
39
[Grade 8 Science–Student #10]
New
In this school year, have you visited a science museum?
A. Yes
B. No
Cognitive lab results:
1. All ten students interpreted the item as asking about trips to museums that involve science.
2. When asked what was meant by the term “science museum,” a majority of students (6 of 10) gave
general and vague definitions for the term: “a museum that has to do with science of any kind” and
“a museum about science.” The other four students provided examples of what might be found in
a science museum (“a collection of experiments” and “rocks and dinosaurs”).
3. When asked what they think of as science museums, eight students were able to provide examples
of actual science museums, including Air and Space, Smithsonian, and Port Discovery. Only one
student provided an example that was not an actual science museum.
Rationale:
1. Cognitive lab results show that grade 8 students comprehend the item as intended.
2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school
trip” and “not on a school trip,” as shown in the following two items, ETS suggests using the “not
on a school trip” version of the item to measure students’ out-of-school science-related activities.
Recommendation:
1. ETS recommends not using the item in the 2014 pilot administration.
40
[Grade 8 Science–Student #11]
New
In this school year, have you visited a science museum on a school trip?
A. Yes
B. No
Cognitive lab results:
1. The students interpreted the item as asking if they have visited a museum on a school trip, or
under the supervision of the school.
2. When asked what was meant by the phrase “on a school trip,” all ten students either found the
term synonymous with the term “field trip,” or described traveling to a location with people from
their school.
3. When asked if they had been on a school trip to a science museum, six students described a
situation where they had been on a school trip. Of those six, three provided descriptions of the
specific trip, including traveling to a science museum with classmates, the class, or the teacher on
a bus.
Rationale:
1. Cognitive lab results show that grade 8 students understand the item as intended.
2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school
trip” and “not on a school trip” (as compared to the following item), ETS suggests using the “not
on a school trip” version of the item to measure students’ out-of-school science-related activities.
Recommendation:
1. ETS recommends not using the item in the 2014 pilot administration.
41
[Grade 8 Science–Student #12]
New
In this school year, have you visited a science museum that was not on a school trip?
A. Yes
B. No
Cognitive lab results:
1. When asked what was meant by the phrase “not on a school trip,” seven of the students stated that
it meant visiting the museum without others from their school, and three said it meant visiting in
the presence of family or friends.
2. Six students described a time when they visited a science museum that was not on a school trip,
and four students said they could not remember a visit to a science museum that was not on a
school trip.
Rationale:
1. Cognitive lab results show that grade 8 students understand the item as intended.
2. Since grade 8 students can clearly differentiate between a trip to science museum “on a school
trip” and “not on a school trip," ETS suggests using the “not on a school trip” version of the item
to measure students’ out-of-school science-related activities.
Recommendation:
1. ETS recommends adding this item to the 2014 pilot administration.
42
Student Questionnair e–Gr ade 12
[Grade 12 Science–Student #1]
VC304986
In your science class this year, have you done hands-on activities or projects with any of
the following? Fill in one oval on each line.
a. Living things (for example,
plants, animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving sugar or
salt in water)
d. Rocks or minerals (for
example, identifying types)
e. Magnifying glass or
microscope (for looking at
small things)
f. Thermometer or barometer (for
making measurements)
g. Simple machines (for example,
pulleys and levers)
Yes
No
A
B
A
B
A
B
A
B
VC305007
A
B
VC305008
A
B
VC305009
A
B
VC305012
VC304988
Revised sub-item
based on VC304989
VC304991
Cognitive lab results:
1. When the students were asked what they thought was meant by the term “circuits and energy,”
half the students (5 of 10) attempted to explain the concept of how circuits and energy are related
(“circuits carry electrical energy from one place to another”). The other half of students named
tangible components of circuits such as wires, batteries, and light bulbs.
2. Of the three students that answered “yes” to sub-item b, two recalled activities in class related to
electricity or circuits. The other described a chemistry lab where reactions were measured as a
function of heat applied by an electric heater.
3. When probed, eight of the ten students would not change their answer if they would change their
response to sub-item b if instead of “circuits and energy” the example was “batteries and light.”
Five students noted they believed that asking the item this way would have changed its intent.
Three of these five thought that “light” did not fit with the concept of electricity.
Rationale:
1. Cognitive lab results show that, for grade 12 students, the phrase “circuits and energy” appears to
be a better example because this phrase includes “batteries and light,” in its overall meaning.
Recommendation:
1. For sub-item b, ETS recommends using the revised examples “circuits and energy” for the grade
12 student questionnaire instead of using “batteries and light” in the 2014 pilot administration.
2. ETS recommends using the revised version of the item as shown in the student item #3 (with
response options “Never,” Rarely,” Sometimes,” and “Often”) in the 2014 pilot administration.
43
[Grade 12 Science–Student #2]
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
a. Living things (for
example, plants,
animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving
sugar or salt in water)
d. Rocks or minerals (for
example, identifying
types)
e. Magnifying glass or
microscope (for looking
at small things)
f. Thermometer or
barometer (for making
measurements)
g. Simple machines (for
example, pulleys and
levers)
Never or
hardly ever
Once every
few weeks
About once
a week
Two or
three times
a week
Every day
or almost
every day
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
VC305007
A
B
C
D
E
VC305008
A
B
C
D
E
VC305009
A
B
C
D
E
VC305012
VC304988
Revised subitem based on
VC304989
VC304991
Cognitive lab results:
1. All students displayed a good comprehension of the item.
2. When asked how they would respond if they had only done activities or projects at certain times
during the school year (e.g., if they had done an activity every day for three weeks, but they have
not done the activity since then), students typically use two methods to respond: recall experience
within a particular period of time, and averaging the frequency across the whole school year.
a. Of the five students who would answer on the lower end of the scale, three explained their
choices by describing a process similar to averaging the frequency of the activity over the
course of a school year.
b. The other two students explained that the response option “about once a week” or higher
would be a poor way of describing this situation.
c. One student chose “every day or almost every day” explaining that they would consider the
period of time that they were working on that project rather than the entire year.
44
d. The remaining three students did not specify the response option they would choose and
explained that their answer would depend on how often they studied that topic or how well
they would remember it.
Rationale:
1. Cognitive lab results show that grade 12 students used two methods to respond to the item: recall
experiences within a particular period of time and averaging the frequency across the whole school
year.
2. Two students explicitly stated that the response option “about once a week” or higher would be a
poor way of describing the situation.
Recommendation:
1. ETS recommends not using this item in the 2014 pilot administration.
45
[Grade 12 Science–Student #3]
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
a. Living things (for example,
plants, animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving sugar or
salt in water)
d. Rocks or minerals (for
example, identifying types)
e. Magnifying glass or
microscope (for looking at
small things)
f. Thermometer or barometer (for
making measurements)
g. Simple machines (for example,
pulleys and levers)
Never
Rarely
Sometimes
Often
A
B
C
D
A
B
C
D
A
B
C
D
A
B
C
D
VC305007
A
B
C
D
VC305008
A
B
C
D
VC305009
A
B
C
D
VC305012
VC304988
Revised sub-item
based on VC304989
VC304991
Cognitive lab results:
1. The students were asked to define what the terms “rarely,” “sometimes,” and “often” meant to
them. Five of the students provided objective, quantifiable answers, and five students provided
subjective answers.
a. For the term “rarely,” three of the five who answered objectively indicated that the term
means “once,” and two described it as “once every few weeks.” Of the five who answered
subjectively, three compared the term to the other response categories (“between never and
sometimes”).
b. For the term “sometimes,” three of the five who answered objectively indicated that it
meant more than once during the week, and two described it as more than once in a month.
Of the five that answered subjectively, three compared it to the other response categories
(between “rarely” and “often”).
c. For the term “often,” three of the five who answered objectively indicated that it meant
every day, and two described it as once or more in a week. Of those who answered
subjectively, all used a term to describe a large amount of use as “very frequently.”
2. When asked how they would respond if they had only done activities or projects at certain times
during the school year, six of the ten students provided a clear indication of which of the response
options they would choose.
a. All six students provided answers at the higher end of the scale by choosing either
“sometimes” (4 of 6) or “often” (2 of 6).
46
b. Two of these students explained that they based their answers on the amount of time that
they were working on that project rather than the entire school year. Two explained that if
they had spent three weeks on a project they would have learned a lot, and one stated that
“rarely” would not be applicable in the situation described in the probe.
3. When asked which of the first three item formats they preferred, six students preferred item three,
two preferred item two, one preferred item one and one said that all are easy to understand.
a. Of the six students that preferred item three, five stated they believed these response
options better matched their answers compared to the other items.
b. The two students that preferred item two stated that they liked the specificity of the
response options.
Rationale:
1. Verbal reports and answers to probing questions show that the students strongly preferred the
response option scale used in item three, as compared to items one and two. It appears that the
students could more easily fit their internally generated answers into the response options provided
in item three; in addition, these response options placed a lower overall cognitive burden upon
students.
Recommendation:
1. ETS recommends replacing the trend item VC304986 with this revised version of the matrix item
in the 2014 pilot administration.
2. For sub-item b, ETS recommends using the revised wording of the sub-item (using examples
“circuits and energy”) in the 2014 pilot administration.
47
[Grade 12 Science–Student #4]
VC305292
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
a. Read a science
textbook, in class or at home
b. Read a book or
magazine about science topics
c. Use the Internet to learn about
science topics
d. Watch a movie, video, or DVD
about science topics
Never or
hardly
ever
Once
every
few weeks
About
once
a week
Two or
three
times
a week
Every day
or
almost
every day
A
B
C
D
E
VC546510
A
B
C
D
E
Revised subitem based on
VC305295
A
B
C
D
E
Revised subitem based on
VC720562
A
B
C
D
E
Revised subitem based on
VC305307
Cognitive lab results:
1. When asked what the phrase “science topics” meant, six students explained that it means academic
subjects such as physics, chemistry, and biology. Four students answered that this phrase means
anything that has to do with science.
2. Four of the five students who indicated they read books or magazines that were not for
schoolwork did not include these books or magazines when answering sub-item b. The reason
provided for not including these books or magazines was that they were not being read for their
science classes.
3. When asked if they would respond differently if the item used the term “science” instead of
“science topics,” four of the ten students responded that they would answer the item differently.
One student explained that he/she would have included activities done at home in addition to those
completed in class. The explanations of the other two students suggest that the phrase “science
topics” was interpreted as referring to more narrow and strictly defined science subjects, while the
term ”science” would include a wider range of subjects.
Rationale:
1. The item is classified under issue “Organization of Instruction” and sub-issue “Instructional
Strategies.” Adding the word “topics” to the relevant sub-items, helps further clarify the intent of
the item.
Recommendation:
1. ETS recommends using the revised version of the three sub-items in the 2014 pilot administration.
48
[Grade 12 Science–Student #5]
VC720622
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
a. Identify questions that can be
addressed through science
experiments
b. Design a science experiment
c. Talk about measurements you
took for your science project or
activity
d. Talk about the results of your
science project or activity
e. Watch your teacher do a
science experiment or activity
f. Make graphs or charts of the
results from your science
project or activity
g. Write a report on your science
project or activity (such as
reports, science journals, or lab
write-ups)
h. Write a lab write-up on your
science project or activity
i. Write a science journal
Never or
hardly
ever
Once
every
few weeks
About
once
a week
Two or
three
times
a week
Every day
or
almost
every day
A
B
C
D
E
VC720623
A
B
C
D
E
VC720634
A
B
C
D
E
VC720635
A
B
C
D
E
VC720636
A
B
C
D
E
VC720638
A
B
C
D
E
VC720639
A
B
C
D
E
VC720641
A
B
C
D
E
New
A
B
C
D
E
New
Cognitive lab results:
1. When asked what the phrase “talk about” meant in sub-item c and d, all ten students, described it
as “discussing” with the entire class, classmates, or teachers.
2. When probed, most students (7 of 10) provided similar definitions of the term “lab write-up” in
sub-item h.
a. These seven students provided explanations that were centered on the idea that it was a
written piece that summarized a science lab. However, three of these students expressed
some confusion about the meaning of the term “lab write-up.”
b. Two students believed that the term “lab write-up” meant answering questions.
c. One student was unable provide a definition.
3. When initially answering, three students expressed some confusion about the meaning of the term
“science journal” in sub-item i. When asked directly what the term meant, the students had
varying responses. While providing a definition, six of the students expressed that they were
unsure what the term meant; of these six, two were unable to guess a definition. Two students
believed the term could mean a long science report. Two other students believed a science journal
might be where students keep their notes for science. Two students thought the term might mean a
short daily class or homework assignment. One student thought a science journal might be where
49
short brief constructed response items were answered. Another student thought that it might be
used to provide a summary of an experiment for publication.
Rationale:
1. Interview notes revealed considerable variation in how students understood the phrases “lab writeup” and “science journal.” Considerable overlap between the two phrases exists in some students’
interpretations. To avoid confusion and loss of data, ETS suggests revising the two sub-items by
using the more generic phrase to describe the activities: “write about your science activities or
projects (such as reports, science journals, or lab write-ups).”
Recommendation:
1. ETS recommends using the revised sub-item in the 2014 pilot administration.
50
[Grade 12 Science–Student #6]
VC315266
In this school year, how often have you used your school library resources for science
(such as books, magazines, computers, and audio-video materials)?
A. Never or hardly ever
B. Once every few weeks
C. About once a week
D. Two or three times a week
E. Every day or almost every day
Cognitive lab results:
1. When answering the probe, “did you think about your school’s library, a public library, or both,”
there was an even split among students.
a. Five students thought only of the school library resources when answering.
b. Five students thought of both the school and public libraries and their resources.
2. When asked to provide examples of library resources that they have used for science class, all
students were able to provide at least one example. Some of these included:
a. Biographies, websites, online journals, internet databases, magazines, autobiographies,
encyclopedia Britannica, and textbooks.
3. A majority of students (9 of 10) considered computers and the internet as library resources.
a. While thinking about his answer, one student was not sure whether the probe was asking
about the use of computers in general, or solely as a means to access the internet.
4. A majority of students (9 of 10) did not think of videos as a library resource. One of these students
responded to the probe by indicating that while not mentioning videos as a library resource
originally, he had used the library computers to watch online videos.
Rationale:
1. Not all students’ spontaneous responses included library resources other than books. ETS suggests
adding specific examples at the end of the item to be consistent with the item used in grade 4 and
grade 8 questionnaires.
2. Since the item is under issue “Availability and Use of Instructional Recourses” and sub-issue
“Products,” ETS suggests clarifying the question as asking about “school library” use.
Recommendation:
1. Revise and keep the item in the 2014 pilot administration.
51
[Grade 12 Science–Student #7]
VC305330
Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill
in one oval on each line.
a. I do science-related activities that are not for
schoolwork.
b. I like science.
c. Science is one of my favorite subjects.
d. I take science only because I have to.
e. I take science only because it will help me in
the future.
f. When I graduate from high school, I would
like to have a job related to science.
Strongly
disagree
Disagree
Agree
Strongly
agree
A
B
C
D
A
A
A
B
B
B
C
C
C
D
D
D
A
B
C
D
VC305353
A
B
C
D
VC720596
VC305348
VC305350
VC305351
VC305352
Cognitive lab results:
1. Overall, students did not exhibit consistent comprehension problems. Most students (7 of 10)
stated that the item asked if they agreed or disagreed with these statements about science. Three
students stated that the item is asking about their opinions and feelings about science.
2. When probed what the phrase “science-related activities” meant to them, a majority of students (9
of 10) reported that it meant doing anything related to science such as watching a movie or
conducting an experiment.
3. When asked what the phrase “not for schoolwork” meant to them, a majority of students (9 of 10)
stated that it was something that was not required by the school (something extracurricular).
4. When asked to provide an example of “science-related activities that you or your classmates have
done or might do that are not for schoolwork,” more than half of the students (6 of 10) were able
to provide an example, such as:
a. Using a telescope, working at NIH, reading Stephen Hawking books, nature treks, and
science fairs.
b. More than half of the students (6 of 10) stated that these activities were not for schoolwork
because a teacher did not instruct them to perform these activities, they would not be
graded, and the activities would not be discussed in class.
Rationale:
1. Cognitive lab results show that all students understood and interpreted the item as intended.
Recommendation:
1. ETS recommends keeping the trend item in the 2014 pilot administration.
52
[Grade 12 Science–Student #8]
New
In this school year, have you participated in any of the following activities? Fill in one
oval on each line.
a. Science fair
b. Science club
c. Science competition
Yes
A
A
A
No
B
B
B
Cognitive lab results:
1. A majority of students (7 of 10) indicated that the item asked whether they had done the listed
activities during the current school year. Three students thought it referred to all extracurricular
activities.
2. A majority of students (8 of 10) stated that they did not participate in any of the listed activities.
Two students had participated in the listed activities; one had participated in a science club and a
science competition. The other student had participated in a science fair. Those who reported
participating in the listed activities also specified that they participated in the activities during the
current school year.
3. When asked what the phrase “science fair” means to them, most students (7 of 10) explained that
it refers to creating a research project and presenting it to other people.
a. Two students did not differentiate the terms “science fair” and “science competition.”
b. One student had never participated in a science fair but had attended one.
c. Over half of students (6 of 10) were able to provide an example of a science fair, but only
two students mentioned science fairs that were taking place this year. Four students
provided examples of science fairs ranging chronologically from elementary school to their
first year of high school. Four students were unable to provide specific examples of science
fairs.
4. When asked what the phrase “science club” meant to them, students had varied ideas about
whether the club was a school sponsored, extracurricular, or an activity separate from the school
altogether. All students stated that a science club was a club where students meet and discuss
science-related topics. When asked to provide an example of a science club that they or their
classmates have participated in or might participate in, students provided the following examples:
a. Future Doctors and Scientists, Science Bowl club, Medical Masterminds, turtle club,
biology, physics, or chemistry clubs, and an environmental club.
b. Three students were unable to provide an example, or stated that a science club was the
same as a science fair or competition.
5. Most students (7 of 10) stated that a science competition includes conducting an experiment and
being judged against other students, explaining that it was different from a fair. Two students
considered a science competition to be the same as a science fair or club. When asked to provide
an example of a science competition that they or their classmates have participated in or might
participate in, some examples included:
a. Science bowl, Siemens competition, NIH competition, and county competitions.
53
Rationale:
1. Most students comprehend the item as intended.
2. Although almost all students indicated that they have not participated in any of the activities, ETS
suggests that item could be still used to measure the growth of out-of-school science activities.
Recommendation:
1. ETS recommends adding the item to the 2014 pilot administration.
54
[Grade 12 Science–Student #9]
New
In this school year, have you visited a science museum?
A. Yes
B. No
Cognitive lab results:
1. Overall, students did not exhibit any consistent comprehension problems.
2. A majority of students (9 of 10) stated that they were being asked if they had been to a museum
that focuses on science. Only two students indicated that the visit had to take place during the
current school year.
3. When asked what the phrase “science museum” meant to them, all students stated that it is a
museum that focuses on any field of science.
4. When asked to provide examples of science museums, all students responded and listed:
a. The National Air and Space Museum, the Natural History Museum, and the Cryptologic
Museum.
5. When asked to provide examples of museums that are not for science, all students responded and
listed:
a. The National Gallery of Art, Spy Museum, Native American Museum, and Holocaust
Museum.
6. When asked to provide an example of a science museum that they or their classmates have visited
or might visit, all students responded and provided these examples:
a. The Baltimore Science Museum, the Natural History Museum, the National Air and Space
Museum, and the National Aquarium. One student described “Bodies: the Exhibition,” but
was unable to name it.
7. Of the four students who were asked who they went to the science museum with this school year,
two went with their class, one went with friends, and one went with family.
Rationale:
1. Cognitive lab results show that grade 12 students comprehend the item as intended.
2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip”
as shown in the following two items, ETS suggests using the “not on a school trip” version of the
item to measure students’ out-of-school science-related activities.
Recommendation:
1. ETS recommends not using the item in the 2014 pilot administration.
55
[Grade 12 Science–Student #10]
New
In this school year, have you visited a science museum on a school trip?
A. Yes
B. No
Cognitive lab results:
1. Overall, students did not exhibit any consistent comprehension problems.
2. All students understood that the item was asking about field trips for science class.
3. All students stated that “on a school trip” means a trip that is sponsored by the school with school
supervision. Most students (8 of 10) were able to report an example of a school trip. They
included:
a. The Medical Museum in Walter Reed, the Natural History Museum, the National
Aquarium, and the Air and Space Museum.
4. None of the students had visited a museum with their school during the current school year.
Rationale:
1. Cognitive lab results show that grade 12 students comprehend the item as intended.
2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip”
as compared to the following item, ETS suggests using the “not on a school trip” version of the
item to measure students’ out-of-school science-related activities.
Recommendation:
1. ETS recommends not using the item in the 2014 pilot administration.
56
[Grade 12 Science–Student #11]
New
In this school year, have you visited a science museum that was not on a school trip?
A. Yes
B. No
Cognitive lab results:
1. Overall, students did not exhibit any consistent comprehension problems.
2. A majority of students (9 of 10) were able to identify that the item was asking if they have visited
a science museum on a trip that was not sponsored by the school.
3. All students stated that “not on a school trip” meant that the visit was not sponsored by the school,
and done on the students own time.
4. More than half (6 of 10) of students gave examples of museum trips with their families. Some of
the museums mentioned were:
a. The National Air and Space Museum, and the Smithsonian.
5. A majority of students (9 of 10) reported that they have not visited a museum this year.
Rationale:
1. Cognitive lab results show that grade 12 students comprehend the item as intended.
2. Since grade students can clearly differentiate between “on a school trip” and “not on a school trip”
as compared to the previous item, ETS suggests using the “not on a school trip” version of the
item to measure students’ out-of-school science-related activities.
Recommendation:
1. ETS recommends adding this item to the 2014 pilot administration.
57
Teacher Questionnair e–Gr ade 4
[Grade 4 Science–Teacher #1]
New (Revised based onVC305014)
To what extent do you use each of the following student groupings for science instruction in your
classroom? Fill in one oval on each line.
a. Groupings based on students’ interest
in science/science-related topics
b. Groupings based on students’ learning
preferences or styles
c. Groupings based on students’
readiness needs or ability levels
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
New
A
B
C
D
New
A
B
C
D
New
Cognitive lab results:
1. The teachers shared a common understanding of the item as asking about how students were
placed in smaller groups to work during class and how this related to students’ learning.
2. All teachers interpreted the phrase “students’ learning preferences” in sub-item b to refer to
students’ learning style.
3. Teachers interpreted “students’ readiness needs” in sub-item b to mean students’ ability levels
related to reading or content comprehension, though one teacher noted some confusion with the
term “students’ readiness needs,” calling it “weird wording,” and wondered if this referred to how
English learners are grouped with other students.
4. Compared to the original version of the item (i.e., “Do you create groups within this class for
science instruction on the basis of ability,” solely with “yes/no” options), all five teachers
responded that they preferred the way the item was currently asked, noting that this would allow
them to better explain how they teach.
Rationale:
1. The cognitive lab report shows that all five teachers interpreted the items as intended, and
compared to the trend version of the item, they preferred the cognitive lab tested version.
2. To address the confusion regarding the wording “learning preferences” and “readiness needs,” and
further clarify the intention of the two sub-items, ETS recommends revising the two phrases to
“learning preferences or styles” and “readiness or ability levels” respectively.
Recommendation:
1. Replace the trend version of the item (VC305014) with the cognitive lab tested and further revised
version.
58
[Grade 4 Science–Teacher #2]
VC970876
To what extent do you emphasize each of the following objectives in teaching science to your fourthgrade class? Fill in one oval on each line.
a. Increase students’ interest in
science
b. Increase awareness of the
importance of science in daily life
c. Learn about applications of science
to environmental issues
d. Teach scientific facts and principles
e. Teach scientific methods
f. Equip students with the knowledge
and skills needed for studying
science in upper grade levels
g. Develop systematic observation
skills
h. Develop inquiry skills
i. Develop skills in lab techniques
j. Develop problem-solving (design)
skills
k. Develop scientific writing skills
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
VC970917
A
B
C
D
VC970928
A
B
C
D
VC970930
A
A
B
B
C
C
D
D
A
B
C
D
Revised sub-item
based on
VC970922
A
B
C
D
VC970929
A
A
B
B
C
C
D
D
A
B
C
D
VC970925
A
B
C
D
VC970931
VC970919
VC970920
VC970923
VC970926
Cognitive lab results:
1. Some phrases, such as “inquiry skills,” “develop skills in lab techniques,” and “scientific writing
skills,” were difficult to comprehend.
2. One teacher expressed that sub-item f “developing problem-solving (design) skills” is asking two
different questions at the same time.
3. A majority of teachers (4 of 5) interpreted the phrase “upper grade levels” in sub-item d as
intended, including any grade above fourth grade and middle school.
4. All teachers referred to school, county, or science standards and curricula to explain what
knowledge and skills that student would need for studying science in upper grade levels.
5. One teacher felt that the phrase “knowledge and skills” was necessary for clarification.
59
Rationale:
1. The difficulties that teachers have with understanding some phrases might result from not
grouping the sub-items together categorically. If these sub-items were grouped together in a
logical order, it would potentially clarify the issue.
2. Indeed, sub-item f “develop problem-solving (design) skills” is confusing in that it implies that
problem-solving skills equate experimental design skills. Removing the word “design” and the
parenthesis would clarify the issue. Since experimental design skills should have been included in
“inquiry skills,” there is no need to develop a sub-item to address it.
Recommendation:
1. Re-order the sub-items (as shown above) and remove the word “design” and the parenthesis from
sub-item f “develop problem-solving (design) skills.”
60
[Grade 4 Science–Teacher #3]
VC970932
How much of the following instructional materials and other resources does your school system provide
you with to teach science to your fourth-grade class? Fill in one oval on each line.
a. Science textbooks
b. Science magazines and books
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
e. Space to conduct science labs
f. Computers for students’ use in class
g. Computer labs
h. Computers for teachers’ use
i. Computerized science labs for classroom use
j. Audiovisual materials
k. Science kits
l. Scientific measurement instruments (e.g.,
telescopes, microscopes, thermometers, or
weighing scales)
None
Little
Some
A lot
A
A
B
B
C
C
D
D
A
B
C
D
A
A
A
A
A
A
A
A
B
B
B
B
B
B
B
B
C
C
C
C
C
C
C
C
D
D
D
D
D
D
D
D
A
B
C
D
VC970953
VC970954
VC970955
VC970956
VC970957
VC970958
VC970959
VC970960
VC970961
VC970962
VC970963
VC970964
Cognitive lab results:
1. One teacher found sub-item j and the term “audiovisual” to be vague.
2. There are two occasions where teachers appeared confused, likely due to forgetting what the item
stem was originally asking.
3. Teachers differentiated between the “school” and the “school system.” They interpreted “school
system” as including the school district and the county school office.
4. One teacher found it difficult to choose between options B (little) and C (some) for sub-items c
(supplies or equipment for science demonstrations) and d (supplies or equipment for science labs).
Rationale:
1. Cognitive lab results reveal that teachers have a consistent understanding on the revised version of
the item (see next item) and preferred the response options in the revised version of the item.
Recommendation:
1. Replace the item with the revised version of the item (Teacher, #4).
61
[Grade 4 Science–Teacher #4]
New (Revised based on VC970932)
To what extent does your school system (including your school, county office, and school district)
provide the following to you? Fill in one oval on each line.
Not at all
Small
extent
Moderate
extent
Large
extent
A
A
B
B
C
C
D
D
A
B
C
D
A
A
A
A
A
B
B
B
B
B
C
C
C
C
C
D
D
D
D
D
A
B
C
D
A
A
B
B
C
C
D
D
A
B
C
D
a. Science textbooks
b. Science magazines and books
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
e. Space to conduct science labs
f. Computers for students’ use in class
g. Computer labs
h. Computers for teachers’ use
i. Computerized science labs for classroom
use
j. Audiovisual materials
k. Science kits
l. Scientific measurement instruments (e.g.,
telescopes, microscopes, thermometers, or
weighing scales)
VC970953
VC970954
VC970955
VC970956
VC970957
VC970958
VC970959
VC970960
VC970961
VC970962
VC970963
VC970964
Cognitive lab results:
1. Teachers differentiated between “school” and “school system” and the role of each entity in
providing materials. Teachers would respond to the sub-items differently between this item (the
revised version) and previous item (the original version).
2. All teachers indicated that the options in this item (i.e., “not at all,” “small extent,” “moderate
extent,” “large extent”) appropriate and sufficient for describing the resources available, and had a
similar interpretation of each option.
3. Most teachers implied that they preferred this item (the revised version) compared to the previous
item (the original item).
Rationale:
1. Cognitive lab results show that this item (the revised version) performed better than the previous
item (the original item).
2. To address the potential of losing information regarding resources provided by the county and
school districts caused by revising the item stem from “school system” to “school,” ETS
recommends revising the item stem from “school” to “school system (including your school,
county office, and school district).”
Recommendation:
1. Replace the original item (VC970932) with this item (the revised version).
62
[Grade 4 Science–Teacher #5]
VC767811
When you teach science to your fourth-grade class, do you do any of the following? Fill in one oval on
each line.
a. Use a different set of methods
in teaching some students
b. Supplement the regular course
curriculum with additional
material for some students
c. Vary the pace of instruction for
some students
d. Have some students engage in
different classroom activities
e. Set different achievement
standards for some students
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
VC767814
A
B
C
D
VC767820
A
B
C
D
Revised sub-tem based on
VC767821
A
B
C
D
VC767823
A
B
C
D
VC767824
Cognitive lab results:
1. All teachers understood the item was asking about different instructional methods they use with
their students.
2. All teachers interpreted the phrase “vary the pace of instruction for some students” to refer to
whether students could learn at a faster or slower pace based on how quickly they understood the
material or lesson.
3. All teachers stated that they would have answered sub-item c “vary the pace of instruction for
some students” the same way if it had instead read “pace my teaching differently for some
students.” One teacher, however, said she would prefer the question asked in the alternate
wording.
Rationale:
1. Cognitive lab results show that teachers consider the alternate wording of the sub-item c as asking
the same question, slightly preferring the alternate wording to (i.e., the original wording “pace my
teaching differently for some students.”
2. ETS recommends replacing the original wording for sub-item c with this one (the revised
wording) to be consistent with grade 8 questionnaire.
Recommendation:
1. Replace the original wording of the sub-item c with this one (the revised wording).
63
[Grade 4 Science–Teacher #6]
New
To what extent do you provide students with the following learning experiences? Fill in one oval on each
line.
A
A
A
B
B
B
C
C
C
Large
Extent
8 or
more
times
per year
D
D
D
A
B
C
D
Small
Extent
Not at all
1–3 times
per year
a.
b.
c.
d.
Science fairs
Science clubs
Science competitions
Trips to science museums
(including traditional museums
and other similar sites)
Moderate
Extent
4–7 times
per year
New
New
New
New
Cognitive lab results:
1. Two teachers interpreted the item as asking what outside opportunities teachers are providing,
while others indicated that it asked what the teachers or the school provided.
2. All teachers answered the questions in regards to activities or opportunities that the school
organized, rather than what the teacher personally provided.
3. Teachers made a distinction between “science fair” and “science competition.”
4. All teachers felt the response options provided were appropriate, although one teacher commented
that it would be better to ask teachers to quantify the frequency of events per school year. When
asked the follow-up probe of whether it would better to use frequencies such as certain times a
week, a semester, or a school year, a majority of teachers (4 of 5) said this would be better.
Rationale:
1. To address a majority of teachers’ (4 of 5) preference on frequency options, and to be consistent
with the school questionnaires, ETS recommends revising the options to “1–3 time per year,” “4–7
times per year,” and “8 or more times per year.”
2. ETS recommends revising the wording of sub-item c to be consistent with school questionnaire.
Recommendation:
1. Add this item to the 2014 pilot questionnaire.
64
Teacher Questionnair e–Gr ade 8
[Grade 8 Science–Teacher #1]
New (Revised based onVC305014)
To what extent do you use each of the following student groupings for science instruction in your
classroom? Fill in one oval on each line.
a. Groupings based on students’ interest
in science/science-related topics
b. Groupings based on students’ learning
preferences or styles
c. Groupings based on students’
readiness needs or ability levels
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
New
A
B
C
D
New
A
B
C
D
New
Cognitive lab results:
1. All teachers interpreted the term “student groupings” to mean separating the students into work
groups to collaborate on assignments.
2. Three of the five teachers interpreted the phrase “students’ learning preferences” as referring to
learning style, while the remaining two teachers interpreted the phrase as referring generally to
interests and preferences.
3. Three of the five teachers expressed confusion regarding the phrase ‘students’ readiness needs” in
sub-item c. One teacher noted that "students' readiness needs" could mean “the desire and
willingness to learn.”
4. Four of the five teachers stated that they use all three grouping methods in their teaching day.
5. All teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,”
and “large extent,” are appropriate for describing the extent of instructional grouping strategies.
Rationale:
1. The cognitive lab report shows that all five teachers interpreted the items as intended. Three
teachers further indicated that it is necessary to ask about various grouping methods.
2. To address the confusion regarding the wording “learning preferences” and “readiness needs,” and
further clarify the intention of the two sub-items, ETS recommends revising the two phrases to
“learning preferences or styles” and “readiness or ability levels” respectively.
Recommendation:
1. Replace the trend version of the item (VC305014) with this cognitive lab tested and further
revised version.
65
[Grade 8 Science–Teacher #2]
VC976013
To what extent do you emphasize each of the following objectives in teaching science to your eighthgrade class? Fill in one oval on each line.
a. Increase students’ interest in
science
b. Increase awareness of the
importance of science in daily life
c. Learn about applications of science
to environmental issues
d. Teach scientific facts and principles
e. Teach scientific methods
f. Equip students with the knowledge
and skills needed for studying
science in upper grade levels
g. Develop systematic observation
skills
h. Develop inquiry skills
i. Develop skills in lab techniques
j. Develop problem-solving (design)
skills
k. Develop scientific writing skills
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
VC976015
A
B
C
D
VC976023
A
B
C
D
VC970926
A
A
B
B
C
C
D
D
A
B
C
D
Revised sub-item
based on VC976019
A
B
C
D
VC97625
A
A
B
B
C
C
D
D
A
B
C
D
VC976021
A
B
C
D
VC970927
VC976017
VC976018
VC976020
VC976022
Cognitive lab results:
1. A majority teachers (4 of 5) stated that the item asks about how much they focus on different areas
and topics when teacher science.
2. All teachers interpreted the phrase “upper grade levels” as high school, and one of them included
honors courses, AP courses, and college.
3. All teachers explained the phrase “knowledge and skills needed for study science in upper grade
levels” as the principles and techniques that students need to be successful in high school science.
4. A majority of teachers (4 of 5) referred to state curriculum standards to determine what knowledge
and skills will be needed beyond 8th grade.
5. Most teachers (3 of 5) reported that their answer would change slightly but remain similar if the
item had been phrased “prepare students for further study in science."
Rationale:
1. The cognitive lab report on this matrix item did not reveal any difficulty among teachers with the
wording of the item.
2. Since issues have been identified in grade 4 version of the item and recommended revision has
been made, ETS recommends revising the item to be consistent with the grade 4 version.
66
Recommendation:
1. Re-order the sub-items (as shown above) and remove the word “design” and the parenthesis from
sub-item f “develop problem-solving (design) skills.”
67
[Grade 8 Science–Teacher #3]
VC976028
How much of the following instructional materials and other resources does your school system provide
you with to teach science to your eighth-grade class? Fill in one oval on each line.
a. Science textbooks
b. Science magazines and books
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
e. Space to conduct science labs
f. Computers for students’ use in class
g. Computer labs
h. Computers for teachers’ use
i. Computerized science labs for classroom use
j. Audiovisual materials
k. Science kits
l. Scientific measurement instruments (e.g.,
telescopes, microscopes, thermometers, or
weighing scales)
None
Little
Some
A lot
A
A
B
B
C
C
D
D
A
B
C
D
A
A
A
A
A
A
A
A
B
B
B
B
B
B
B
B
C
C
C
C
C
C
C
C
D
D
D
D
D
D
D
D
A
B
C
D
VC976031
VC976032
VC976034
VC976035
VC976036
VC976037
VC976039
VC976040
VC976041
VC976042
VC976043
VC976045
Cognitive lab results:
1. The teachers displayed inconsistent interpretations of the item. Two interpreted the item as asking
about the resources they have available; three interpreted the item as asking about what resources
were provided by their school system, including county or school district.
Rationale:
1. Cognitive lab results reveal that grade 8 teachers do not have a preference between the current
revised versions of the item stem.
2. A majority of teachers (4 of 5) indicated that the response options in the revised version of the
item (i.e., “not at all,” “small,” “moderate extent,” “large extent”) are appropriate and sufficient
for describing the extent to which resources are provided by the school.
Recommendation:
1. Replace the item with the revised version of the item to be consistent with grade 4 teacher
questionnaire.
68
[Grade 8 Science–Teacher #4]
New (Revised based on VC976028)
To what extent does your school system (including your school, county office, and school district)
provide the following to you? Fill in one oval on each line.
a. Science textbooks
b. Science magazines and books
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
e. Space to conduct science labs
f. Computers for students’ use in class
g. Computer labs
h. Computers for teachers’ use
i. Computerized science labs for classroom use
j. Audiovisual materials
k. Science kits
l. Scientific measurement instruments (e.g.,
telescopes, microscopes, thermometers, or
weighing scales)
Not at all
Small
extent
Moderate
extent
Large
extent
A
A
B
B
C
C
D
D
A
B
C
D
A
A
A
A
A
A
A
A
B
B
B
B
B
B
B
B
C
C
C
C
C
C
C
C
D
D
D
D
D
D
D
D
A
B
C
D
VC976031
VC976032
VC976034
VC976035
VC976036
VC976037
VC976039
VC976040
VC976041
VC976042
VC976043
VC976045
Cognitive lab results:
1. Teachers differentiated between “school” and “school system” and the role of each entity in
providing materials.
2. Three teachers reported that the way this item is phrased (the revised version) and the previous
item (the original version) would not make a difference in how they would answer any of the subitems.
3. A majority of teachers (4 of 5) stated that the options in this item (i.e., “not at all,” “small extent,”
“moderate extent,” “large extent”) are appropriate and sufficient for describing the extent to which
resources are provided by the school.
4. All teachers shared a consistent understanding of the options and implied that they prefer these
options compared to the ones in the previous item (the original item).
Rationale:
1. Cognitive lab results show that this item (the revised version) performed better than the previous
item (the original item).
2. To address the potential loss of information regarding resources provided by county and school
district caused by revising item from “school system” to “school,” ETS recommends revising the
item stem from “school” to “school system (including your school, county office, and school
district).”
Recommendation:
1. Replace the original item with this item (the revised version).
69
[Grade 8 Science–Teacher #5]
VC976085
When you teach science to your eighth-grade class, do you do any of the following? Fill in one oval on
each line.
a. Use a different set of methods in
teaching some students
b. Supplement the regular course
curriculum with additional
material for some students
c. Vary the pace of instruction for
some students
d. Have some students engage in
different classroom activities
e. Set different achievement
standards for some students
Not at all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
VC976086
A
B
C
D
VC976088
A
B
C
D
Revised sub-item
based on VC976091
A
B
C
D
VC976092
A
B
C
D
VC976094
Cognitive lab results:
1. A majority of teachers (4 of 5) stated that the question is asking about different instructional
methods, and how they plan and change lesson presentation methods to accommodate students
with varying levels of understanding.
2. A majority of teachers (4 of 5) stated that the phrase “vary the pace of instruction for some
students” meant providing students with different amounts of time to complete tasks, based on
their different performance levels.
3. Three teachers stated that they would have answered sub-item c “vary the pace of instruction for
some students” the same way if it had instead said “pace my teaching differently for some
students.” Two stated that their answers would be different because they interpreted the first
version to be more student-specific, instead of referring to their overall teaching approach.
4. All teachers stated that they prefer the current phrasing: “vary the pace of instruction for some
students.”
Rationale:
1. Two teachers stated that their answer on sub-item c would be different if it had instead asking
“pace my teaching differently for some students.”
2. All teachers stated that they prefer the current phrasing, “vary the pace of instruction for some
students.”
Recommendation:
1. Replace the original wording of the sub-item c with this one (the revised wording).
70
[Grade 8 Science–Teacher #6]
New
To what extent do you provide students with the following learning experiences? Fill in one oval on each
line.
A
A
A
B
B
B
C
C
C
Large
Extent
8 or
more
times
per year
D
D
D
A
B
C
D
Small
Extent
Not at all
1–3 times
per year
a.
b.
c.
d.
Science fairs
Science clubs
Science competitions
Trips to science museums
(including traditional museums
and other similar sites)
Moderate
Extent
4–7 times
per year
New
New
New
New
Cognitive lab results:
1. A majority of teachers (4 of 5) stated that the item asks about science-related activities outside of
the classroom.
2. Three teachers made a distinction between “science fair” and “science competition,” while the
other two did not.
3. Three teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,”
and “large extent,” were sufficient to describe the extent of the learning experience provided to the
students.
4. One thought the choices were sufficient but felt that the term “science museum” was too limiting
and suggested to add science field trips.
5. In regards to whether frequencies, such as certain times a week/a semester/a school year, would be
better than the response options provided for this item, opinions were divided. While two teachers
rejected the idea that frequencies would be more relevant response options, two noted that they
would be preferred. One respondent specifically suggested the following options: per quarter, per
semester, or per year.
Rationale:
1. To address some teachers’ preference on frequency options and to be consistent with grade 4
questionnaires, ETS recommends revising the options to “1–3 times per year,” “4–7 times per
year,” and “8 or more times per year.”
2. ETS recommends revising the wording of sub-item c to be consistent with school questionnaire.
Recommendation:
1. Add this item to the 2014 pilot questionnaire.
71
School, Grade 4:
[Grade 4 Science–School #1]
New (Revised based on VE013369)
Is there a science specialist or coach available (full- or part-time) to fourth-grade
teachers at your school?
A. Yes, available full-time at my school.
B. Yes, available part-time at my school.
C. No
Cognitive lab results:
1. School administrators were not unified in their understanding of what the role and
responsibilities would be for a “science specialist or coach.”
2. Although a majority of administrators (4 of 5) indicated that they did not have a part-time
or full-time science specialist or coach in their school, they emphasized that the
responsibilities they associated with the role were taken on by classroom teachers and/or
other types of specialists (e.g., instructional support teacher).
3. All administrators felt that the sub-items in the following matrix item covered the
majority of the responsibilities they would consider part of the role of a science specialist
or coach.
Rationale:
1. ETS posits that the intention of the item should be whether or not there are personnel
available in school performing the function of a “science specialist or coach,” not
whether the particular title of “science specialist or coach” is available in the school.
Since all administrators felt that the sub-items in the following matrix item (item 2)
covered the majority of the responsibilities of science personnel, the content of the subitems provide a better context for administrators to comprehend the question than
providing an abstract definition of “science specialist or coach.” Therefore, ETS
recommends collecting the information through revising the following item in which the
majority of the responsibilities of a “science specialist or coach” have been explained in
more detail.
2. It appears to be difficult for some school administrators to choose between the options of
“full-time” and “part-time” since the role may be filled by school, district, or county staff;
or by someone who is not a school employee. Therefore, in the revised version of the
following item, the availability of a “science specialist or coach” will not differentiate
between “part-time” and “full-time.”
Recommendation:
1. Remove item from the 2014 pilot administration.
72
[Grade 4 Science–School #2]
New (Revised stem based on VE013372)
To what extent are any of the following a responsibility of the science specialist or coach, or other
instructional support personnel, available to fourth-grade science instruction teachers at your school?
Fill in one oval on each line.
Not available
at my school
a. Provide technical
assistance/support to
individual teachers about
science content or the
teaching of science
b. Conduct professional
development for groups of
teachers about science
content or the teaching of
science
c. Provide science instruction
to students on various topics
d. Interpret and incorporate
Translate science standards
and current research into
classroom practice
e. Provide science enrichment
to some student groups
Not
at all
Small
extent
Moderate
extent
Large
extent
A
B
C
C
E
VE013373
A
B
C
C
E
VE013382
A
B
C
C
E
VE013383
A
B
C
C
E
Revised
sub-item
based on
VE013384
A
B
C
C
E
VE013385
Cognitive lab results:
1. Although a majority of administrators (4 of 5) indicated that they did not have a part-time
or full-time science specialist or coach in their school, they emphasized that the
responsibilities they associated with the role were taken on by classroom teachers and/or
other types of specialists (e.g., instructional support teacher).
2. All administrators felt that the sub-items covered the majority of the responsibilities they
would consider part of the role of a science specialist or coach.
73
Rationale:
1. ETS posits that the intention of the item should be whether or not there are personnel
available for fourth-grade science instruction functioning as a “science specialist or
coach”, not whether the particular title of “science specialist or coach” exists in school.
ETS recommends adding “other instructional support personnel” to the item stem. Since
all administrators felt that the sub-items covered the majority of the responsibility of a
“science specialist or coach” these sub-items would provide administrators better context
to comprehend the question than providing abstract definitions of “science specialist or
coach.” Therefore, ETS recommends combining the matrix item with the previous skip
pattern item and adding “Not available at my school” to the option list to capture the
information originally collected through the previous skip pattern item (VE013369).
2. For sub-item b, the word “translate” is vague (used in an analogous sense). ETS
recommends revising the word “translate” to the phrase “interpret and incorporate” to
clearly reflect the kind of service and support that a “science specialist or coach”
provides.
Recommendation:
1. Replace the trend item (VE013372) with this revised version of the item in the 2014 pilot
administration.
74
[Grade 4 Science–School #3]
New
In the last school year, to what extent did does your school provide students with the following
opportunities? Fill in one oval on each line.
a.
b.
c.
d.
Not at all
Small
extent
(1–3
times)
Moderate
extent (4–
7 times)
A
A
A
B
B
B
C
C
C
Large
extent
(8 times
or
more)
D
D
D
A
B
C
D
Science fairs
Science clubs
Science competitions
Trips to science museums
(including traditional museums
and other similar sites)
Cognitive lab results:
1. The definition for “science fair” and “science competition” differed among
administrators.
2. A majority of administrators (4 of 5) preferred the response options of “large extent,”
“moderate extent,” and “small extent,” versus frequencies (i.e., certain times a
week/semester/school years).
Rationale:
1. The Science Subject-Area Standing Committee (06/19/2012) has suggested developing
more items asking about students’ science activities outside of school. The Assessment
Development Committee (ADC) (07/05/2012) specifically suggested that out-of-school
activities could include science fairs, national competitions, robotics competitions, etc.
To address these suggestions, ETS developed relevant items for all student, teacher, and
school administrator questionnaires and tested these items in cognitive laboratories.
2. Since cognitive lab results revealed that school administrators differentiate between the
terms “science fair” and “science competition,” ETS recommends keeping them as two
separate sub-items.
3. To be consistent with grade 12 questionnaire, the phrase “in the last school year” and
frequency ranges have been added to the item stem and response options to further
quantify the answers.
Recommendation:
1. Add to the 2014 pilot administration
75
New
New
New
New
School, Grade 8:
[Grade 8 Science–School #1]
New (Revised based on VE013964)
Is there a science specialist or coach available (full- or part-time) to eighth-grade
teachers at your school?
A. Yes, available full-time at my school.
B. Yes, available part-time at my school.
C. No
Cognitive lab results:
1. Three of the five administrators indicated that they did not have a part-time or full-time
science specialist or coach in their school, and also emphasized the responsibilities they
associated with the role taken on by classroom teachers and/or other types of specialists
such as “instructional lead teacher,” “instructional coach,” or “mentor teacher.”
2. Three of the five administrators agreed that the sub-items in the following matrix item
covered all main responsibilities of a science specialist or coach.
Rationale:
1. ETS posits that the intention of the item should be whether there are personnel within
school performing the function of a “science specialist or coach”, not whether personnel
exists within the school with the specific title of “science specialist or coach”. Since most
administrators felt that the sub-items in the following matrix item covered the majority of
the responsibility of the personnel, the content of the sub-items would provide better
context for administrators to comprehend the question than providing abstract definitions
of “science specialist or coach”. Therefore, ETS recommends collecting the information
through revising the following item in which the majority of the responsibilities of a
“science specialist or coach” have been provided in detail.
2. It appears to be difficult for some school administrators to choose between “full-time”
and “part-time” even though there are personnel available fulfilling the role of a “science
specialist or coach”, because not all personnel who function as a “science specialist or
coach” are from the school. They could be also from the school district or the county (not
a school employee), as evidenced in the grade 12 cognitive lab results. Therefore, in the
revised version of the following item, the availability of a “science specialist or coach”
will not be differentiated between “part-time” and “full-time”.
Recommendation:
1. Remove the item from the 2014 pilot administration
76
[Grade 8 Science–School #2]
New (Revised stem based on VE013966)
To what extent are any of the following a responsibility of the science specialist or coach, or other
instructional support personnel available to eighth-grade science instruction teachers at your school?
Fill in one oval on each line.
Not available
at my school
a. Provide technical
assistance/support to
individual teachers about
science content or the
teaching of science
b. Conduct professional
development for groups of
teachers about science
content or the teaching of
science
c. Provide science instruction
to students on various topics
d. Interpret and incorporate
Translate science standards
and current research into
classroom practice
e. Provide science enrichment
to some student groups
Not
at all
Small
extent
Moderate
extent
Large
extent
A
B
C
C
E
VE013967
A
B
C
C
E
VE013968
A
B
C
C
E
VE013969
A
B
C
C
E
A
B
C
C
E
Cognitive lab results:
1. The use of the word “translate” in sub-item “d” is confusing to one administrator.
2. Although two administrators indicated that they did not have a part-time or full-time
science specialist or coach in their school, they were able to answer the question based on
other available personnel performing the function of a “science specialist or coach.”
3. Most (3 of 5) administrators felt that the sub-items covered the majority of the
responsibilities they would consider part of the role of a science specialist or coach.
4. Two of the five administrators agreed that the four answer choices are sufficient for
describing the extent of a science specialist or coach’s responsibilities in their school.
77
Revised
sub-item
based on
VE013970
VE013971
Rationale:
1. ETS posits that the intention of the item should be whether there are personnel available
for fourth-grade science instruction functioning as a “science specialist or coach”, not
whether the particular title of “science specialist or coach” exists in school. ETS
recommends adding “other instructional support personnel” to the item stem. Since all
administrators felt that the sub-items covered the majority of the responsibility of a
“science specialist or coach”, these sub-items would provide administrators better context
to comprehend the question than providing abstract definitions of “science specialist or
coach”. Therefore, ETS recommends combining the matrix item with the previous skip
pattern item and adding “Not available at my school” to the option list to capture the
information originally collected through the previous skip pattern item (VE013964).
2. For sub-item b, the word “translate” is vague (used in an analogous sense). ETS
recommends revising the word “translate” to the phrase “interpret and incorporate” to
clearly reflect the kind of service that a “science specialist or coach” provides.
Recommendation:
1. Replace the trend item (VE013966) with this revised version of the item in the 2014 pilot
administration.
78
[Grade 8 Science–School #3]
New
In the last school year, to what extent did does your school provide students with the following
opportunities? Fill in one oval on each line.
a.
b.
c.
d.
Not at all
Small
extent
(1–3
times)
Moderate
extent (4–
7 times)
A
A
A
B
B
B
C
C
C
Large
extent
(8 times
or
more)
D
D
D
A
B
C
D
Science fairs
Science clubs
Science competitions
Trips to science museums
(including traditional museums
and other similar sites)
Cognitive lab results:
1. All administrators made a clear distinction between “science fair” and “science
competition.”
2. A majority of administrators (4 of 5) agreed that the response options of “not at all,”
“large extent,” “moderate extent,” and “small extent,” are sufficient for describing the
extent of the opportunities their school provides.
Rationale:
1. Standing Committee (06/19/2012) has suggested developing more items asking about
students’ science activities outside of school. ADC (07/05/2012) specifically suggested
out-of-school activities could include science fairs, national competitions, robotics
competitions, etc. To address these suggestions, ETS developed relevant items for all
student, teacher, and school administrator questionnaires and tested these items in
cognitive lab.
2. Cognitive lab results reveals that school administrators differentiate “science fair” and
“science competition”, ETS recommends keeping them as two separate sub-items.
3. To be consistent with grade 12 questionnaire, phrase “in the last school year” and range
of the option “small extent,” “moderate extent,” and “large extent” haven been added to
the item stem and options to further quantify the answers.
Recommendation:
1. Add to the 2014 pilot administration.
79
New
New
New
New
School, Grade 12:
[Grade 12 Science–School #1]
New (Revised based on VE015623)
Is there a science specialist or coach available (full- or part-time) to twelfth-grade
teachers at your school?
A. Yes, available full-time at my school.
B. Yes, available part-time at my school.
C. No
Cognitive lab results:
1. Two of the five administrators stated that although they did have an individual fulfilling
the role of a “science specialist or coach,” it is difficult to determine whether or not they
are full- or part-time. One administrator further explained that none of the response
options fit his school’s situation because the resource teacher is responsible for all high
schools in the county and the 12th grade teachers in his school rarely ask for support from
this individual.
2. A variety different terms were often used in schools for describing individuals with the
responsibilities of a “science specialist or coach,” such as “resource teacher,” “science
department chairperson,” and “science coordinator.”
3. Overall, administrators felt that the response option did not fit their preferences.
4. All administrators felt the sub-items in the following matrix item covered the majority of
responsibilities they would consider as parts of the role of “science specialist or coach.”
Rationale:
1. ETS posits that the intention of the item should be whether there is personnel available in
school performing the function of “science specialist or coach”, not whether the
particular title of “science specialist or coach” available in school. Since most
administrators felt that the sub-items in the following matrix item covered the majority of
the responsibility of the personnel, the content of the sub-items would provide better
context for administrators to comprehend the question than providing abstract definitions
of “science specialist or coach”. Therefore, ETS recommends collecting the information
through revising the following item in which the majority of the responsibilities of a
“science specialist or coach” have been provided in detail.
2. It appears to be difficult for some school administrators to choose between “full-time”
and “part-time” even though there are personnel available fulfilling the role of a “science
specialist or coach”, because not all personnel who function as a “science specialist or
coach” are from the school. They could be also from the school district or the county (not
a school employee), as evidenced in the grade 12 cognitive lab results. Therefore, in the
80
revised version of the following item, the availability of a “science specialist or coach”
will not be differentiated between “part-time” and “full-time”.
Recommendation:
1. Remove the item from the 2014 pilot administration
81
[Grade 12 Science–School #2]
New (Revised stem based on VE015624)
To what extent are any of the following a responsibility of the science specialist or coach, or other
instructional support personnel available to twelfth-grade science instruction teachers at your school?
Fill in one oval on each line.
Not available
at my school
a. Provide technical
assistance/support to
individual teachers about
science content or the
teaching of science
b. Conduct professional
development for groups of
teachers about science
content or the teaching of
science
c. Provide science instruction
to students on various topics
d. Interpret and incorporate
Translate science standards
and current research into
classroom practice
e. Provide science enrichment
to some student groups
Not
at all
Small
extent
Moderate
extent
Large
extent
A
B
C
C
E
VE015625
A
B
C
C
E
VE015626
A
B
C
C
E
VE015627
A
B
C
C
E
A
B
C
C
E
Cognitive lab results:
1. Overall, the administrators share a common understanding of the term “science specialist
or coach.”
2. All administrators felt that the sub-items covered the majority of the responsibilities they
would consider part of the role of a “science specialist or coach.”
Rationale for recommendation:
1. ETS posits that the intention of the item should be whether there are personnel available
for fourth-grade science instruction functioning as a “science specialist or coach”, not
whether the particular title of “science specialist or coach” exists in school. ETS
recommends adding “other instructional support personnel” to the item stem. Since all
82
Revised
sub-items
based on
VE015628
VE015629
administrators felt that the sub-items covered the majority of the responsibility of a
“science specialist or coach”, these sub-items would provide administrators better context
to comprehend the question than providing abstract definitions of “science specialist or
coach”. Therefore, ETS recommends combining the matrix item with the previous skip
pattern item and adding “Not available at my school” to the option list to capture the
information originally collected through the previous skip pattern item (VE015623).
2. For sub-item b, the word “translate” is vague (used in an analogous sense). ETS
recommends revising the word to “interpret and incorporate” to clearly reflect the kind of
service that a “science specialist or coach” provides.
Recommendation:
1. Replace the trend item (VE015624) with this revised version of the item in the 2014 pilot
administration.
83
[Grade 12 Science–School #3]
New
In the last school year, to what extent did does your school provide students with the following
opportunities? Fill in one oval on each line.
e.
f.
g.
h.
Not at all
Small
extent
(1–3
times)
Moderate
extent (4–
7 times)
A
A
A
B
B
B
C
C
C
Large
extent
(8 times
or
more)
D
D
D
A
B
C
D
Science fairs
Science clubs
Science competitions
Trips to science museums
(including traditional museums
and other similar sites)
Cognitive lab results:
1. All administrators interpreted the item as asking about science-related opportunities that
the school provides beyond traditional classroom instruction.
2. Three of the five administrators agreed that there is a distinction between a “science fair”
and a “science competition.”
3. It appears that grade 12 administrators further differentiate “science museums,” including
traditional museums, such as a natural history museums, and locations like aquariums or
other similar sites.
4. Most administrators (3 of 5) expressed a preference for having response options with
frequencies, such as certain times a week/a semester/a school year. One administrator
suggested adding ranges to quantify what was meant by a small, moderate and large
extent. One administrator indicated that the best option would be to base answers on a
school year.
Rationale:
1. Standing Committee (06/19/2012) has suggested developing more items asking about
students’ science activities outside of school. ADC (07/05/2012) specifically suggested
out-of-school activities could include science fairs, national competitions, robotics
competitions, etc. To address these suggestions, ETS developed relevant items for all
student, teacher, and school administrator questionnaires and tested these items in
cognitive lab.
2. Cognitive lab results reveals that school administrators differentiate “science fair” and
“science competition”, ETS recommends keeping them as two separate sub-items.
84
New
New
New
New
3. Phrase “in the last school year” and range of the option “small extent”, “moderate
extent”, and “large extent” haven been added to the item stem and options to further
quantify the answers.
Recommendation:
1. Add to the 2014 pilot administration.
85
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
Cognitive Laboratories Report and Findings
(Eureka Facts)
86
Cognitive Interviews of Background Questionnaire for
NAEP 2012
Science Assessment
Final Report
January 22, 2013
Contents
1.
2.
Introduction ......................................................................................................................... 1
1.1
Sample ......................................................................................................................... 1
1.2
Recruitment and Interview Procedures ......................................................................... 3
Cognitive Interview Results ................................................................................................. 4
2.1
Student Background Questionnaire .............................................................................. 4
2.1.1.
Grade 4 Students.................................................................................................. 4
2.1.2.
Grade 8 Students.................................................................................................24
2.1.3.
Grade 12 Students ...............................................................................................41
2.2
Teacher Background Questionnaire ............................................................................ 58
2.2.1.
Grade 4 Teachers ................................................................................................58
2.2.2.
Grade 8 Teachers ................................................................................................68
2.3
Administrator Background Questionnaire .................................................................... 77
2.3.1.
Grade 4 Administrators ........................................................................................77
2.3.2.
Grade 8 Administrators ........................................................................................82
2.3.3.
Grade 12 Administrators ......................................................................................87
Cognitive Interviews Report for BQ CogLab
Date of Report:
January 22, 2013
Interview Dates:
11/27/2012 – 01/04/2013
1.
Introduction
1.1
Sample
Cognitive interviews designed to test survey items developed for the National Assessment of Educational
Progress (NAEP) Science Assessment were conducted with 15 school administrators, 10 teachers, and 30
students. Each group of respondents was further divided into sub-groups based on grade level: school
administrators for fourth, eighth, and twelfth-grade students; teachers of fourth and eighth-grade students;
and students in fourth, eighth, and twelfth grades.
The administrators, teachers and students who participated in the cognitive interview study were recruited
from the greater Washington D.C., and Baltimore, Maryland metropolitan areas.
Administrator Sample
Five school administrators of fourth-grade students, five school administrators of eighth-grade students, and
five school administrators of twelfth-grade students participated in the cognitive interviews. Demographics of
participating administrators and their schools are presented in Table 1.
Table 1: Participant and School Demographics for Administrator Respondent Group
Respondent
Group
th
4 grade
Administrators
Participant Demographics
School Demographics
Total
Gender
SES
School Size
White: 2
Black: 2
Low: 3
High: 2
Large: 3
Small: 2
5
Male: 3
Female: 2
White: 2
Black: 1
Other: 2
Low: 2
High: 3
Large: 3
Medium: 2
5
12 grade
Administrators
Male: 3
Female: 2
White: 3
Black: 2
Low: 2
High: 3
Large: 5
5
Total
Male: 6
Female: 9
White: 9
Black: 5
Other: 1
Low: 7
High: 8
Large: 11
Medium: 2
Small: 2
15
th
8 grade
Administrators
th
Female: 5
Race/Ethnicity
Prepared by EurekaFacts, LLC
1
ETS CogLab Report: BQ Science 2013
Teacher Sample
Five fourth-grade teachers and five eighth-grade teachers who teach science during the current school year
participated in the cognitive interviews. Demographics of participating teachers and their schools are
presented in Table 2.
Table 2: Participant and School Demographics for Teacher Respondent Group
Respondent Group
Participant Demographics
School Demographics
Gender
Race/Ethnicity
SES
th
Male: 2
Female: 3
White: 4
Other: 1
Low: 4
High: 1
8 grade Teachers
th
Female: 5
White: 3
Black: 2
Low: 2
High: 3
Total
Male: 2
Female: 8
White: 7
Black: 2
Other: 1
Low: 6
High: 4
4 grade Teachers
School Size
Large: 2
Medium: 2
Small: 1
Large: 1
Medium: 2
Small: 2
Large: 3
Medium: 4
Small: 3
Total
5
5
10
Student Sample
Ten fourth-grade students, ten eighth-grade students, and ten twelfth-grade students participated in cognitive
interviews. The demographics of participating students are presented in Table 3.
Table 3: Participant Demographics for Student Respondent Group
Respondent Group
SES
Gender
th
Low: 4
High: 6
Male: 4
Female: 6
th
Low: 5
High: 5
Male: 4
Female: 6
12 grade Students
Low: 5
High: 5
Male: 7
Female: 3
Total
Low: 14
High: 16
Male: 15
Female: 15
4 grade Students
8 grade Students
th
Prepared by EurekaFacts, LLC
Race/Ethnicity
White: 5
Hispanic: 2
Black: 3
White: 4
Hispanic: 2
Black: 4
White: 8
Black: 1
Asian: 1
White: 17
Black: 8
Hispanic: 4
Asian: 1
Total
10
10
10
30
2
ETS CogLab Report: BQ Science 2013
1.2
Recruitment and Interview Procedures
Cognitive interviews began on November 27, 2012 and were completed on January 4, 2013.
EurekaFacts recruited participants through email and telephone outreach. Trained interviewers screened
candidates who responded to the outreach using the screening questionnaire approved by OMB (Appendix
A). Parents of student candidates were informed that they would be compensated with two $25 gift cards,
one for the student participant and one for the parent of the student. Administrators and teachers were
informed that they would be compensated with a $40 gift card.
The cognitive interview process in this study followed the OMB approved script and interview protocol as
presented in Appendices A and B. Written consent was obtained from participants or the parents/legal
guardians of participants before interviews were conducted. Participants were informed about the purpose of
the study and reassured that their participation was voluntary and that their answers would be used only for
research purposes and would not be disclosed, or used, in identifiable form for any other purpose except as
required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].
Interview sessions lasted for one hour. During the sessions, interviewers explained the think-aloud process,
conducted a think aloud practice question exercise, and then asked a series of verbal questions.
Interviewers used several different cognitive interviewing techniques, including general think-aloud, questionspecific probes, and interviewers’ observations.
At the end of the session, participants were briefly interviewed about their overall impressions of the survey
items. Participants were thanked, remunerated, and asked to sign a receipt for their incentive payment.
The interview procedure for the fourth-grade student group included pairing of a sub-set of the survey items
to reduce possible order effect. Items 2, 3, 4, 5, and 6 were presented in paired order, following the design
presented in Table 4. Based on this paired order, six versions of questionnaire were developed; these
versions were assigned to participants from the fourth-grade student group according to a predefined order.
Table 4: Paired Design for Fourth-grade Student Questionnaire
Questionnaire
version
1
2
3
4
5
6
Question Order
2
2
2
3
3
3
Prepared by EurekaFacts, LLC
3
3
3
2
2
2
4
4
5
5
6
6
5
6
4
6
4
5
Participants
6
5
6
4
5
4
st
th
1 and 7
nd
th
2 and 8
rd
3
th
th
4 and 9
th
th
5 and 10
th
6
3
ETS CogLab Report: BQ Science 2013
2.
Cognitive Interview Results
2.1
Student Background Questionnaire
2.1.1. Grade 4 Students
Grade 4 Science–Student—Item 1
New (Revised based on VC519362)
In this school year, how often have you done activities or projects to learn about electricity (for
example, circuits and energy)?
A
B
C
D
Never
Rarely
Sometimes
Often
Reading
Interviewer notes report the following observations:
Three students had trouble with the word “circuits:” two mispronounced the word, while the third took an
unusually long pause at first, and then read the word properly.
Comprehension
Two students understood the item as asking about projects both at school and at home.
When asked to define “activities or projects,” eight of the ten students described this as “things” or “stuff”
to do. Three of these eight further described them as “fun,” while two described them as “work.” One
student only provided examples of activities; one student could not define the phrase.
When asked what “circuits and energy” meant to them, three students said that this meant “light” or
“power,” while two could not define the phrase. Four students did not define the phrase, but rather
provided an example (e.g., “wires”).
Response Process
Four students reported that they would have responded to the item differently if the term “batteries and
light” were used in place of “circuits and energy.” Two students explicitly said they would change
responses because “batteries and light” is clearer to them, and would have changed their responses
from “rarely” to “sometimes.”
Recommendations for Improvement
Although only two students reported not understanding the term “circuits and energy,” nearly half (4 of
10) would have changed their response if the more concrete example “batteries and light” were used. In
addition, there was much variation in how students understood the phrase “circuits and energy.” Thus,
Prepared by EurekaFacts, LLC
4
ETS CogLab Report: BQ Science 2013
we suggest that the tested alternative phrase, “batteries and light,” would be a more relevant example in
this item.
Prepared by EurekaFacts, LLC
5
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 2
VC315219
In this school year, how often have you done activities or projects to learn about chemicals (for
example, mixing sugar or salt in water)?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Response Process
When asked how they would respond to this item if they had used chemicals only at certain times during
the school year, students provided a variety of responses. Three students would have chosen “never or
hardly ever,” two would have chosen “about once a week,” and two would have chosen “two or three
times a week.” The remaining response options (“once every few weeks” and “every day or almost every
day”) would have been chosen by one student each. One student did not understand the probe.
Only three students reported the time when they had carried out any activities, providing such time
frames as “10 weeks into the year,” “more towards the beginning [of the year],” and “2 weeks ago.” Only
two students provided specific examples of the activities, such as:
o
Watching videos in class about what happens when you touch or drink unknown chemicals.
o
Looking at and smelling different liquids.
More than half of students (6 of 10) preferred the response options in item two to those of item three.
Three students stated that they liked item two because the choices were more concrete than those of
item three.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
6
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 3
New (Revised based on VC315219)
In this school year, how often have you done activities or projects to learn about chemicals (for
example, mixing sugar or salt in water)?
A
B
C
D
Never
Rarely
Sometimes
Often
Comprehension
When asked to paraphrase the item, the students displayed a good understanding of the item.
When asked to define “rarely,” “sometimes,” and “often,” the students used different approaches:
o
Four students defined the answer choices using subjective, vague terminology: rarely means
“hardly ever,” sometimes means “once in a while,” and often means “a lot” or “all the time.”
o
Two students defined the response options in terms of comparisons to the other categories:
"rarely” means between “often” and “never;” “sometimes” is very similar to “rarely;” “often means
more than sometimes, and is between sometimes and always.”
o
Two students defined response options in terms of frequencies: “sometimes” as once or twice a
week; “often” as either every day or every other day.
o
Two students used a mix of approaches described above.
Response Process
When asked how they would respond if they had used chemicals only at certain times during the school
year, three students chose “rarely,” three chose “sometimes,” and three chose “often.” The remaining
student misunderstood the probe.
Four students preferred the response options in item three to those in item two. One student said that
these options were easier to read; another said they were “less to read” and therefore made more
sense.
Recommendations for Improvement
Six of the ten students preferred the response options in item two to those in item three. Only four
students preferred the response options provided in item three. It appears that the students had difficulty
differentiating between “rarely,” “sometimes,” and “often,” and also with matching their internally
generated answers to the more abstract response options in item three. Based on these difficulties with
the response options in item three, we suggest that the response options used in item two are more
appropriate for fourth grade students.
Prepared by EurekaFacts, LLC
7
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 4
New (Revised based on VC315222, VC315227, and VC315229)
In this school year, how often have you done science activities using scientific tools?
A
Never or hardly ever
B
Once every few weeks
C
About once a week
D
E
Two or three times a week
Every day or almost every day
Comprehension
Interviewer notes report the following observations:
When asked to paraphrase the item, the students displayed a good understanding of the item.
When asked to define “scientific tools,” students had difficulty explaining this term, and instead provided
examples of such tools.
o
Four students identified “magnifying glass” as an example of a scientific tool; two students listed
“pH testing strip;” two students mentioned “dropper.”
Response Process
When asked how they would respond if they had only used tools at certain times of the year, three
students would have responded “once every few weeks,” two would have answered “two or three times
a week,” one would have answered “about once a week,” two would have answered “never or hardly
ever,” and one did not understand the probe.
Of the three students who viewed item four last, after items five and six, only one preferred the response
options in item four, while two preferred item six. These two students explained that the response
options in item four were confusing, and one added that option A and B in item four seemed to be the
same.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
8
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 5
New (Revised based on VC315222, VC315227, and VC315229)
In this school year, how often have you done science activities using scientific tools?
A
B
C
D
Never
Rarely
Sometimes
Often
Comprehension
When asked to paraphrase the item, the students displayed a good understanding of the item and
instructions.
When asked to define “scientific tools,” the students had difficulty explaining this term and instead
provided examples of such tools.
o
Students reported the following examples of scientific tools: “magnifying glass,” “magnet,” and
“dropper.”
When asked to define “rarely,” “sometimes,” and “often,” students defined the response options either
using subjective, vague terminology (rarely means “hardly ever,” sometimes means “once in a while,”
and often means “a lot” or “all the time”), or in terms of frequencies (“often” as either every day or every
other day; “sometimes” as once or twice a week; “often” as either every day or every other day).
Response Process
When asked how they would answer if they had only used tools at certain times of the year, four
students would have answered “rarely,” three would have answered “sometimes,” and one would have
answered “never.” One student was not asked this probe, and one student could not provide an answer.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
9
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 6
New (Revised based on VC315222, VC315227, and VC315229)
Have you used scientific tools in school this year?
A
No
B
Yes, a little
C
Yes, a lot
Response Process
When asked how they would answer if they had only used tools at certain times of the year, half of the
students (5 of 10) would have answered “yes, a little,” two would have answered “no,” and two would
have answered “yes, a lot.” One student did not understand the probe.
Between items four, five, and six, half of the students (5 of 10) preferred item six, three students
preferred item four, and two preferred item five. Three students indicated that they did not prefer item
five because the words of the response options were hard for them to understand. Students typically
preferred item six because it was simpler than item four, and easier to understand than item five.
Recommendations for Improvement
Student verbal reports and responses to the probing questions show that item six was the preferred
version of this question, and easiest for fourth grade students to understand.
Prepared by EurekaFacts, LLC
10
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 7
VC315239
In this school year, how often have you read a book or magazine about science?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Comprehension
When asked to paraphrase the item, the students displayed a good understanding of the item. However,
they did not mention the time period included in the item stem.
Response Process
When asked if they had included material that they read outside of school in their answers, five students
indicated that they had not, three indicated that they did include readings outside of school, and two
students did not read science books at all. Half of the students (5 of 10) also indicated that they did not
read materials about science outside of school.
Of the five students who responded that they had read a book or magazine about science in this school
year, all indicated that they read the most often in the classroom.
Only four students stated that they read books about science that are not for school work.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
11
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 8
VC315259
In this school year, how often have you presented what you learned about science to your class?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Reading
Interviewer notes report the following issues with reading:
Two students showed hesitation in pronouncing the word “presented.”
Comprehension
When asked to paraphrase the item, the students displayed a good understanding of the item. However,
they did not mention the time reference included in the item.
When asked to define “presented,” students varied in their responses. Half of the students (5 of 10) said
that it involved “showing” something, while two used the word “share” instead. Six of the students stated
that it involved showing something in front of others, and five explicitly referred to the classroom.
When asked if “to show what you have learned” was different from “to present what you have learned,”
six students said that these were the same, while one student said that they differed. Two students did
not answer, and one student did not understand what “show what you’ve learned” meant.
Response Process
Six of the ten students indicated that they have presented something to their class. Of these six, four
presented information that they learned, and the other two presented a project that they worked on.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
12
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 9
VC315288
In this school year, how often have you talked about measurements or results from your science
activities or projects?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Comprehension
When asked to paraphrase the item, three students stated that it asked if they had used measurements,
and did not note that the item asks whether they have talked about them.
When asked to define “talked about,” the majority of students (8 of 10) answered in a similar manner,
stating that it meant to “say something” or “talk to someone.” Only one student did not know how to
respond.
When asked to define the term “measurements or results,” the students showed good understanding of
these terms, and explained them by referring to results, the end-effect or product of experiments, and
solutions to problems. Two students used examples to help illustrate what they were trying to say.
Response Process
Seven of the ten students stated that they “never or hardly ever” talked about measurements or results.
However, when probed, two of these students reported talking with others: one with a teacher, and the
other with classmates.
The remaining three reported that they talked about measurements or results with a teacher,
classmates, or the whole class.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
13
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 10
VC315265
In this school year, how often have you written a report on your science activities or projects?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Reading
Interviewer notes report no major problems with reading in this item.
Comprehension
Interviewer notes report no major problems with comprehension in this item.
When asked to paraphrase the item, the students displayed a good understanding of the item. However,
they did not mention the time period referenced in the item stem.
All students but one defined “report” in a similar manner, as writing or sharing information from
measurements or science projects with other people.
Response Process
Half of the students (5 of 10) responded that they had “never or hardly ever” written a report this school
year; two responded “once every few weeks;” three said “two or three times a week.”
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
14
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 11
New (Revised based on VC315265)
In this school year, how often have you been asked to write about your science activities or projects?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Comprehension
All students had a similar understanding of the phrase “science journal,” describing it as something you
write in to keep track of information you have for science topics.
The descriptions that students provided for the phrase “lab write-up” were less consistent.
o Six students either did not know what the phrase meant or defined it incorrectly; another student
had never heard of it, but had an intuitive grasp of its meaning.
o
Of the students that did define the term, three described it more as a report, where they write the
results of a project or test, and one described it more as a journal, where general science notes
and information go.
Response Process
Four of the ten students would call the written work they do for a science activity or project a “science
journal” (these same four students were unable to define a “lab write-up”). Two students would call the
work a “report,” while two students would call it something else (a “writing piece” and “results”), and one
was unsure what to call it. Only one student said he/she would call the work a “lab write-up,” but did not
know why.
Half of the students (5 of 10) reported having “never or hardly ever” been asked to write about science
activities or projects. However, only two students who answered “never or hardly ever” to the previous
item (item ten) responded the same way to this item (item eleven).
Recommendations for Improvement
Verbal reports and student responses to item-specific probes show that over half of the students did not
understand the phrase “lab write-up,” but all students had at least a rudimentary grasp of the phrase
“science journal.”
Although the interviewer notes for the previous item (item ten) show that all but one student had a good
understanding of the term “report,” the more generic formulation of item eleven (“asked to write about
your science activities or projects”) produced a different answer pattern among students, suggesting that
item ten did not capture all possible information from students on this topic. We suggest that it might be
helpful using the question formulation from item eleven, with the addition of specific examples of written
deliverables, such as reports, science journals, and lab write-ups.
Prepared by EurekaFacts, LLC
15
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 12
VC315266
In this school year, how often have you used library resources for science?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Reading
Interviewer notes report the following issues with reading:
In the phrase “library resources for science,” two students had trouble pronouncing “resources,” and
another skipped reading the phrase.
Comprehension
When asked, five students interpreted the item as asking if they had ever used library resources for
science.
Two students could not define the phrase “library resources.” Of the remaining eight students, all gave
books as an example of a library resource. Three also gave the example of computers, and two others
gave the example of magazines.
Only after being probed did students acknowledge the internet as being a library resource. After being
asked, six students said that internet was a library resource, while three said that it was not. When
probed, two students also acknowledged computers as library resources; another student was unsure,
because computers in his/her school were not for fourth graders.
Response Process
Nine students were asked whether they thought of their school library, the public library, or both. Of the
nine, seven thought only of the school library, one thought of both, and one answered that they “do not
go to the library.”
When explicitly asked about individual resources, three students included encyclopedias, and three
included videos as library resources.
Recommendations for Improvement
Only two students were unable to explain the meaning of the term “library resources.” However, verbal
reports and student paraphrasing of item twelve show that a majority of students thought only of books
as library resources. We suggest including in the item, in parentheses, examples of different types of
library resources (such as magazines, computers, internet, and videos), in order to aid understanding of
this term and remind students of library resources other than books.
Prepared by EurekaFacts, LLC
16
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 13
VC315410
How often do you do science activities that are not for schoolwork?
A
B
C
D
Never or hardly ever
Sometimes
Often
Always or almost always
Comprehension
On the whole, the students understood the phrase “science activities” in a similar manner: eight of ten
described it as activities or projects that one does, that are related to a science topic. Only two did not
explicitly refer to a science topic.
Students demonstrated a good and consistent understanding of the phrase “not for schoolwork:” five
students said it meant activities done at home, and three said it meant being “out of school.”
Response Process
When asked to provide examples of activities they had done that were not for schoolwork, two students
gave technology/engineering examples (e.g., fixing a TV), two gave natural science examples (e.g.,
studying bugs), and two gave chemical examples (e.g., mixing substances together). Two students who
answered “never or hardly ever” could not remember any examples of activities they had done that were
not for schoolwork.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
17
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 14
New
In this school year, have you participated in a science fair?
A
B
Yes
No
Comprehension
When asked to define a “science fair,” six students described it as related to making experiments. Of
these six, two also stated that “science fair” involved showing their work to other people. One student
defined it as showing something, but did not mention experiments.
Two students defined science fairs as having a competition element; one said first place gets a prize,
and the other said science fair was synonymous with science competition.
Two students did not understand what a “science fair” was; they knew only that it had something to do
with science.
Response Process
When asked to provide an example of a science fair, there were few prevailing trends. Six students
stated that they had not participated in one; two referred to “event nights” at their schools where science
activities took place, but they did not present their own experiments.
Recommendations for Improvement
Verbal reports and answers to the probing questions show that more than half of students had not
participated in a science fair and had a vague understanding of the term. Thus, we suggest that “science
fair” might not be an appropriate term to use.
Prepared by EurekaFacts, LLC
18
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 15
New
In this school year, have you participated in a science club?
A
B
Yes
No
Comprehension
When asked to define a “science club,” all students provided responses indicating that science clubs had
to do with people meeting together to learn more about science or do science activities. Additionally, four
students explicitly indicated they were after-school activities.
Response Process
When asked to provide an example of a science club, four students provided concise examples of
specific clubs, three could only give examples of activities they had done in the clubs, and one student
had never participated in a club. Two students could not recall any examples of science clubs. Examples
provided were:
o
STEM club, robotics club, Hands-On Science, and Mad Science.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
19
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 16
New
In this school year, have you participated in a science competition?
A
B
Yes
No
Comprehension
When asked to define a “science competition,” five students gave responses indicating that this activity
involves “prizes” or “first place, second place, etc.” Six students said it involved either creating or
developing their own project, or that it was like a science fair.
Response Process
When asked to provide an example of a science competition, three students either gave “science fair” as
their example, or gave their same example from item fourteen (which asked for an example of a science
fair). One student said fairs and competitions were similar, but not the same.
Of the five students that were asked, all had heard of a robotics competition, but only one indicated that
they had participated in a robotics competition.
Students were asked whether or not science fairs, science clubs, and science competitions were
different things. Over half (6 of 10) said that they were all different things; three said they were all the
same; one said that fairs and competitions were the same, but clubs were different. All three of the
students who indicated that they were all the same gave the reason that they all dealt with science.
Recommendations for Improvement
Verbal reports and answers to the probing questions show that while a majority of the students had a
very vague understanding of the term “science fair,” a majority had quite a good understanding of the
term ”science competition.” Thus, it is likely that fourth graders do not have much experience with
science fairs, so “science competition” may be a more appropriate term to use.
Prepared by EurekaFacts, LLC
20
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 17
New
In this school year, have you visited a science museum?
A
B
Yes
No
Comprehension
Because “science museum” is a relatively vague term, there was some variance in the definitions that
students provided. However, most (7 of 10) indicated that it either involved scientific activities, lessons,
and/or technology, or gave appropriate examples. Two students were unsure of the meaning, and one
student seemed to include non-scientific museums in their definition.
Nine students were asked what specific museums they considered to be science and non-science
museums. Two students could not remember any science museums; two students could not remember
any non-science museums.
When asked to give an example of a science museum that they had been to, three students could not
remember any museums.
Response Process
When students were asked what museums they thought of as science museums, examples included:
o
When students were asked what museums they thought of as non-science museums, examples
included:
o
The National History Museum, the Wax Museum, and the Native American Museum.
When asked to give an example of a science museum they had been to, three students stated that they
had not been to any museums. One student provided an example that was not a museum, and another
provided three examples, one of which was a non-science museum. Examples of relevant museums
included:
o
The Museum of Natural History, the Space Museum, and an animal museum called Amazing
Animals.
Museum of Natural History, Air and Space Museum, and the Animal Museum.
Only one student was asked who he/she had visited a science museum with, and responded, “the entire
fourth grade.”
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
21
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 18
New
In this school year, have you visited a science museum on a school trip?
A
B
Yes
No
Comprehension
When asked to define “on a school trip,” three students stated it was a trip involving their class and/or
grade; three students mentioned riding on a school bus; three students said it was a field trip; two
students simply re-worded the phrase (e.g., “a trip for school”).
Of the seven students asked to provide an example of a science museum that they visited on a school
trip, two could not think of any examples.
Of the five students who provided examples, two referenced museums, and two could only discuss the
activities they did at the museums (one simply referenced a field trip to the Chesapeake Bay). The
referenced museums were:
o
The Animal Museum and the Kid’s Museum in Baltimore Harbor.
Only one student had been to a science museum on a school trip this year.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
22
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Student—Item 19
New
In this school year, have you visited a science museum that was not on a school trip?
A
B
Yes
No
Comprehension
When asked to define “not on a school trip,” six of the ten students stated it was a trip that did not
include school individuals or classmates, while five students described it as involving family and/or
friends.
Even though five students gave examples of trips they went on that were not for school, four of them
were unable to provide specific names of museums. Of these four students, two described what the
museum was like, and the other two could only say that they had been to a museum.
Five students provided the following examples of trips that were not for school:
o
The Air and Space Museum, the Natural History Museum, and the American Indian Museum.
Only one student indicated that they had visited a science museum this year that was not on a school
trip.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
23
ETS CogLab Report: BQ Science 2013
2.1.2. Grade 8 Students
Grade 8 Science–Student—Item 1
VC304986
In your science class this year, have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
Yes
a. Living things (for example, plants, animals,
bacteria)
b. Electricity (for example, circuits and energy)
No
VC304988
A
B
Revised sub-item
based on VC304989
A
B
c. Chemicals (for example, mixing or dissolving
sugar or salt in water)
A
B
d. Rocks or minerals (for example, identifying
types)
A
B
VC305007
e. Magnifying glass or microscope (for looking
at small things)
A
B
VC305008
f. Thermometer or barometer (for making
measurements)
A
B
VC305009
g. Simple machines (for example, pulleys and
levers)
A
B
VC305012
VC304991
Reading
Interviewer notes report the following observations:
Three students misread “barometer” in sub-item “f.”
Three students skipped over the examples in parentheses for all sub-items.
Comprehension
Four students did not understand the reference period of the item to be “this school year;” instead, their
responses reflected prior school years as well.
When asked what the phrase “hands-on activities or projects” meant to them, half of the students (5 of
10) indicated that this meant working on assigned tasks. Two students understood this to mean working
Prepared by EurekaFacts, LLC
24
ETS CogLab Report: BQ Science 2013
with their peers on an activity. The remaining three students provided limited, vague definitions for the
phrase.
When asked the meaning of the phrase “circuits and energy,” students’ responses varied greatly. While
two students understood a circuit as something consisting of wires, another two understood it as
something that electricity moves through. Two students understood energy to be synonymous with the
term “power.” Four students provided a limited and vague definition for the phrase, showing a partial
understanding that “circuits and energy” referred to electricity. Two of the students were unable to
provide a definition.
Response Process
Of the four students who answered “yes” to sub-item “b” (“electricity”), three recalled that the activity they
thought of when formulating their answer was related to using wires to light a light bulb. The remaining
respondent recalled learning about electricity, but could not provide examples of specific activities
related to this topic.
When asked how they would respond to sub-item “b” if the example had been “batteries and light”
instead of “circuits and energy,” half (5 of 10) responded that they would have changed their answers.
Four of these students would have changed their answers from “no” to “yes,” while the fifth would have
changed from “yes” to “no.”
Recommendations for Improvement
There was significant variation in how students understood the phrase “circuits and energy.” In terms of
their recollections of activities relating to electricity, students predominantly recalled activities where they
had used batteries, wires, and/or light bulbs. In addition, probing revealed that if sub-item “b” included
the example of “batteries and light,” instead of “circuits and energy,” twice as many students would have
answered “yes” to this sub-item. Thus, we recommend that the tested alternative phrase, “batteries and
light,” would be more a relevant example to include in this item.
A large proportion of the students did not appear to understand that the item referred only to “this year,”
and included in their answers activities from previous school years. The term “this year” can be
interpreted as “this school year,” “this calendar year,” or “at any time between now and last year at the
same time.” If it is important that this measurement reflect a specific time period, we suggest providing a
more specifically defined time reference in the item’s stem.
Prepared by EurekaFacts, LLC
25
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 2
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
a. Living things (for example,
plants, animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving sugar
or salt in water)
d. Rocks or minerals (for
example, identifying types)
e. Magnifying glass or
microscope (for looking at
small things)
f. Thermometer or barometer
(for making
measurements)
g. Simple machines (for
example, pulleys and
levers)
Never or
hardly
ever
Once
every
few
weeks
A
B
About
once a
week
Two or
three
times a
week
Every
day or
almost
every
day
C
D
E
VC304988
Revised subitem based on
VC304989
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
VC305007
A
B
C
D
E
VC305008
A
B
C
D
E
VC305009
A
B
C
D
E
VC305012
VC304991
Reading
Interviewer notes report the following observations:
Three students misread “barometer” in sub-item “f.”
Response Process
When asked how they would respond if they had only done activities or projects at certain times during the
school year (e.g., doing an activity every day for three weeks, but not since then), students provided a variety
of responses.
Four students said they would respond in the lower half of the scale. Of these four, three students said
they would answer “once every few weeks,” while the fourth would answer “never or hardly ever.”
Two students said they would choose the middle response option, “about once a week.”
Prepared by EurekaFacts, LLC
26
ETS CogLab Report: BQ Science 2013
Two students said they would answer in the upper half of the scale. One indicated that they would
answer “two or three times a week;” the other would choose “every day or almost every day.”
One student said they would arbitrarily choose either the first or last response option on the scale,
because the item’s reference period lacked specificity. This student suggested that the item should ask
how often they did each of these things in “each science unit,” rather than the entire school year.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
27
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 3
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
Never
Rarely
Sometimes
Often
a. Living things (for example, plants,
animals, bacteria)
A
B
C
D
b. Electricity (for example, circuits
and energy)
A
B
C
D
c. Chemicals (for example, mixing or
dissolving sugar or salt in water)
A
B
C
D
d. Rocks or minerals (for example,
identifying types)
e. Magnifying glass or microscope
(for looking at small things)
f. Thermometer or barometer (for
making measurements)
g. Simple machines (for example,
pulleys and levers)
A
B
C
D
VC305007
A
B
C
D
VC305008
A
B
C
D
VC305009
A
B
C
D
VC305012
VC304988
Revised
sub-item
based on
VC304989
VC304991
Reading
Interviewer notes report the following observations:
Three students skipped over the examples in parentheses for all sub-items.
Comprehension
Two students did not understand the reference period of the item to be “this school year;” instead, their
responses reflected prior school years as well.
Response Process
Six students defined the response options using more subjective and vague terminology, with “rarely”
meaning “not much,” “sometimes” meaning “once in a while,” and “often” meaning “a lot.” Of these six
students, four defined “rarely” in terms of comparisons to the other categories (“not often,” “less than
sometimes”). The definitions for “sometimes” and “often” varied among these six students, but their
definitions showed that they understood the concept in all cases.
Four students defined the response options according to more objective descriptions. Among these
respondents, the term “rarely” was defined by all four students as either “once” or “once a month;” the
term “sometimes” had varying definitions; the term “often” was defined by all four students as once or
multiple times a week.
Prepared by EurekaFacts, LLC
28
ETS CogLab Report: BQ Science 2013
When asked how they would respond if they had only done activities or projects at certain times during
the school year, a majority of students (7 of 10) said that they would respond “sometimes.”
When asked which version of the response options they preferred, eight of the ten students said they
preferred the response options used in this item. Many implied that answering the item with the
dichotomous response options used in item one was easy, but that these options were too broad to
accurately assess their internally generated answers. Moreover, the students explained that it was
difficult to answer the item with frequencies, as used in item two, because the response options were too
specific.
Recommendations for Improvement
Verbal reports and answers to probing questions show that the students strongly preferred the response
option scale used in item three, as compared to item two. It appears that the students could more easily
fit their internally generated answers into the response options provided in item three; in addition, these
response options placed a lower overall cognitive burden upon students. When asked how they would
respond if they had only done activities or projects at certain times during the school year, student
responses showed lower variability in item three, as compared to item two, which may indicate that the
response scale in item three is able to produce responses with higher reliability.
Prepared by EurekaFacts, LLC
29
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 4
VC305292
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
About
once a
week
Two or
three
times a
week
Every
day or
almost
every
day
B
C
D
E
VC546510
A
B
C
D
E
Revised subitem based
on
VC305295
c. Use the Internet to learn
about science topics
A
B
C
D
E
Revised subitem based
on
VC720562
d. Watch a movie, video, or
DVD about science topics
A
B
C
D
E
Revised subitem based
on
VC305307
Never or
hardly
ever
Once
every
few
weeks
a. Read a science textbook, in
class or at home
A
b. Read a book or magazine
about science topics
Comprehension
When asked to state the item in their own words, the students displayed a good understanding of the
item or instructions. However, further probing revealed that four students missed, or did not take into
account, that their answer should focus on activities performed for science class, and they consequently
considered media that was not for school work when answering this item.
When asked what the phrase “science topics” meant, five students implied that this meant subjects
studied in science class, two students gave discrete examples of specific science topics, and two
answered that this phrase meant “anything” that has to do with science.
Response Process
When asked if they would respond differently if the item used the term “science” instead of “science
topics,” seven said they would answer differently. The most common reason given for this change of
response among students (4 of 7) was that the term “science” was broader than “science topics,” and
would allow them to consider more types of media as related to “science” in their responses.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
30
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 5
VC720622
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
Never or
hardly
ever
Once
every
few
weeks
About
once a
week
Two or
three
times a
week
Every
day or
almost
every
day
a. Identify questions that can be
addressed through science
experiments
A
B
C
D
E
VC720623
b. Design a science experiment
A
B
C
D
E
VC720634
c. Talk about measurements
you took for your science
project or activity
A
B
C
D
E
VC720635
d. Talk about the results of your
science project or activity
A
B
C
D
E
VC720636
e. Watch your teacher do a
science experiment or
activity
A
B
C
D
E
VC720638
f. Make graphs or charts of the
results from your science
project or activity
A
B
C
D
E
VC720639
g. Write a report on your
science project or activity
A
B
C
D
E
VC720641
h. Write a lab write-up on your
science project or activity
A
B
C
D
E
New
i. Write a science journal
A
B
C
D
E
New
Comprehension
When asked, half of the students (5 of 10) indicated that the phrase “talk about” in sub-items “c” and “d”
implied summarizing what they did or learned in class. The other five students defined the phrase as
generally talking with people (students or teachers) in their class.
When asked what is meant by the term “lab write-up,” half of the students (5 of 10) described it as
something akin to a report, describing a science lab that they had completed. Three students described
it as another type of class assignment unrelated to a science lab. Two students were unable to provide a
definition.
Prepared by EurekaFacts, LLC
31
ETS CogLab Report: BQ Science 2013
When asked the meaning of the term “science journal,” responses varied greatly, and included the
following: where they write their “warm-up” exercise (2 students); where to write notes (2 students);
where to write about experiments (2 students); something that is used to describe what happened in
class (2 students); where one can write about science in general (2 students).
Recommendations for Improvement
Interview notes revealed considerable variability in students’ understanding of the terms “lab write-up”
and “science journal.” In order to avoid any potential confusion in the future and to ensure consistent
interpretation among students, we suggest providing a standardized definition of these terms.
Prepared by EurekaFacts, LLC
32
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 6
VC315266
In this school year, how often have you used library resources for science?
A
Never or hardly ever
B
Once every few weeks
C
About once a week
D
Two or three times a week
E
Every day or almost every day
Comprehension
Three students indicated during their think-aloud that they found the item and the term “library
resources” to be vague.
Response Process
When probed, seven students stated that they were thinking about their school’s library when answering
this item and three stated that they were thinking of both their school’s library and a public library.
In regards to what students thought of as library resources, the top library resource identified by students
was books (9 of 10). Other resources that students mentioned, unaided, were computers (4 students),
internet (2 students), encyclopedias (2 students), dictionaries (2 students), videos (1 student), and
magazines (1 student).
When asked directly about each resource, eight students said they thought of computers as resources
while answering the item, seven students thought of encyclopedias, five thought of the internet, five of
magazines, and four thought of videos.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
33
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 7
VC305330
Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in
one oval on each line.
Strongly
disagree
Disagree
Agree
Strongly
agree
a. I do science-related activities that are
not for schoolwork.
A
B
C
D
VC305348
b. I like science.
A
B
C
D
VC305350
c. Science is one of my favorite subjects.
A
B
C
D
VC305351
d. I take science only because I have to.
A
B
C
D
VC305352
e. I take science only because it will help
me in the future.
A
B
C
D
VC305353
Comprehension
Interviewer notes do not indicate any consistent problems in students’ understanding of the item or
instructions.
When asked what was meant by the phrase “science-related activities” in sub-item “a,” the students
either provided general, broad descriptions related to activities that had “anything to do with science,” or
offered some examples such as measurements, experiments, and research.
When asked what was meant by the phrase “not for schoolwork,” five students emphasized in their
responses that these activities were done outside of school, or were not assigned schoolwork. Four
students emphasized in their explanations that these activities were done based on personal interests.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
34
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 8
VC304978
In your science class this year, which of the following topics have been covered? Fill in one oval on each
line.
Yes
No
a. Life science (for example, biology, the human body, or ecology)
A
B
VC304982
b. Physical science (for example, energy, physics, or chemistry)
A
B
VC304983
c.
A
B
VC304984
A
B
VC304985
Earth and space science (for example, geology or astronomy)
d. Engineering and technology (for example, designing solutions to
problems)
Reading
Interviewer notes report the following observations:
Three students had trouble reading the word “ecology” in sub-item “a.”
Comprehension
When asked what was meant by the term “covered” in the item stem, seven students provided a general,
broad definition for the term (e.g., “gone over,” “what we learned”). In their definitions, three of the
students included a term that described the extent of the learning (“we learned about it completely,” “we
know it thoroughly”).
Students that answered “yes” to any of the sub-items were asked to describe how the material was
“covered” in their class. A majority of the students (8 of 9) answered this by providing the names of the
topics that were covered. All of the topics mentioned by students fit within the existing sub-item’s
category. One student described the types of assignments and materials that were used to learn each
topic.
Response Process
When probed, seven students said that they would prefer a larger scale that asked to what extent the
topics were covered, as opposed to “yes” and “no” response options. The main reason for this
preference, according to five of these seven students, was the ability to more accurately assess the
extent to which these topics were covered. Of the three students who preferred the yes/no format, two
had previously included a term that described the extent of learning in their description of the term
“covered.”
Recommendations for Improvement
Based on the students’ verbal reports and answers to probes, we suggest that a change in item scale
would be relevant, from a yes/no scale to a Likert scale that measures the extent to which a topic was
covered. Overall, the students preferred the Likert scale-based response options and seem capable of
accurately assessing sub-items on this scale.
Prepared by EurekaFacts, LLC
35
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 9
New
In this school year, have you participated in any of the following activities? Fill in one oval on each line.
Yes
No
a. Science fair
A
B
b. Science club
A
B
c.
A
B
Science competition
Comprehension
When asked what was meant by the term “science fair” in sub-item “a,” all ten students provided
explanations which indicated that science fairs had to do with “experiments” or “projects.” Seven of the
ten explanations also indicated that these projects were shown or talked about to others.
When asked what was meant by the term “science club” in sub-item “b,” students had varying answers.
Five students indicated that these clubs occurred after school, one student said that they occur annually,
and the other four students did not discuss when clubs occurred. Four students included in their
definition that the purpose of the club was to do science activities, and three noted that the purpose of
the club was to discuss or talk about science.
When asked what was meant by the term “science competition” in sub-item “c,” nine students provided a
definition that described it as similar to a science fair, where one makes a project, invention, or
experiment and competes against others. One student mentioned that people also compete by
answering questions. One student was unable to define the term.
Response Process
Eight students were able to provide an example of a science fair that either they or a classmate had
participated in.
o
Three students cited examples from elementary school, one referred to a science expo, one
student referred to the school STEM fair.
Five students were able to provide an example of a science club that either they or a classmate had
participated in. Examples included:
o
Technology club, life science club, a science club involving robotics, and Project Change.
Only one student was able to provide an actual example of a science competition, which they described
as a “science club competition, where they had to build things to compete in certain challenges.” One
student answered that a geography bee was a science competition. The other eight students were
unable to retrieve a cogent example.
All ten students answered “no” to sub-items “a” and “c,” and nine answered “no” to sub-item “b.”
Prepared by EurekaFacts, LLC
36
ETS CogLab Report: BQ Science 2013
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
37
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 10
New
In this school year, have you visited a science museum?
A
B
Yes
No
Comprehension
All ten students interpreted the item as asking about trips to museums that involve science.
When asked what was meant by the term “science museum,” more than half of students (6 of 10) gave
general and vague definitions for the term (e.g., “a museum that has to do with science of any kind” and
“a museum about science”). The other four students provided examples of what might be found in a
science museum (e.g., “a collection of experiments” and “rocks and dinosaurs”).
When asked what they think of as science museums, eight students were able to provide examples of
actual science museums, including: Air and Space, Smithsonian, and Port Discovery. Only one student
provided an example that was not an actual science museum.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
38
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 11
New
In this school year, have you visited a science museum on a school trip?
A
B
Yes
No
Comprehension
The students interpreted the item as asking if they have visited a museum on a school trip, or under the
supervision of the school.
When asked what was meant by the phrase “on a school trip,” all ten students either found the term
synonymous with the term “field trip,” or described traveling to a location with people from their school.
When asked if they had been on a school trip to a science museum, six students described a situation
where they had been on a school trip. Of those six, three provided descriptions of the specific trip,
including traveling to a science museum with classmates, the class, or the teacher on a bus.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
39
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Student—Item 12
New
In this school year, have you visited a science museum that was not on a school trip?
A
B
Yes
No
Comprehension
When asked what was meant by the phrase “not on a school trip,” seven of the students stated that it
meant visiting the museum without others from their school, and three said it meant visiting in the
presence of family or friends.
Six students described a time when they visited a science museum that was not on a school trip, and
four students said they could not remember a visit to a science museum that was not on a school trip.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
40
ETS CogLab Report: BQ Science 2013
2.1.3. Grade 12 Students
Grade 12 Science–Student—Item 1
VC304986
In your science class this year, have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
Yes
No
a. Living things (for example, plants, animals,
bacteria)
A
B
b. Electricity (for example, circuits and energy)
A
B
c. Chemicals (for example, mixing or dissolving
sugar or salt in water)
A
B
d. Rocks or minerals (for example, identifying
types)
A
B
VC305007
e. Magnifying glass or microscope (for looking
at small things)
A
B
VC305008
f. Thermometer or barometer (for making
measurements)
A
B
VC305009
g. Simple machines (for example, pulleys and
levers)
A
B
VC305012
VC304988
Revised sub-item based
on VC304989
VC304991
Comprehension
Half of the students (5 of 10) described “hands on activities” as actively interacting with what they were
learning. The other half of students (5 of 10) thought the term “hands on activities” referred to labs and
research.
When the students were asked what they thought was meant by the term “circuits and energy,” half the
students (5 of 10) attempted to explain the concept of how circuits and energy are related (“circuits carry
electrical energy from one place to another”). The other half of students named tangible components of
circuits such as wires, batteries, and light bulbs.
Of the three students that answered “yes” to sub-item “b,” two recalled activities in class related to
electricity or circuits. The other described a chemistry lab where reactions were measured as a function
of heat applied by an electric heater.
Prepared by EurekaFacts, LLC
41
ETS CogLab Report: BQ Science 2013
Response Process
When probed, eight of the ten students would not change their response to sub-item “b” if instead of
“circuits and light” the example was “batteries and light.” Five students noted they believed that asking
the item this way would have changed its intent. Three of these five thought that “light” did not fit with the
concept of electricity.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
42
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 2
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
a. Living things (for example,
plants, animals, bacteria)
b. Electricity (for example,
circuits and energy)
c. Chemicals (for example,
mixing or dissolving sugar
or salt in water)
d. Rocks or minerals (for
example, identifying types)
e. Magnifying glass or
microscope (for looking at
small things)
f. Thermometer or barometer
(for making
measurements)
g. Simple machines (for
example, pulleys and
levers)
Never or
hardly
ever
Once
every
few
weeks
About
once a
week
Two or
three
times a
week
Every day
or almost
every day
A
B
C
D
E
VC304988
Revised subitem based
on
VC304989
A
B
C
D
E
A
B
C
D
E
A
B
C
D
E
VC305007
A
B
C
D
E
VC305008
A
B
C
D
E
VC305009
A
B
C
D
E
VC305012
VC304991
Response Process
When asked how they would respond if they had only done activities or projects at certain times during the
school year (e.g., if they had done an activity every day for three weeks, but they have not done the activity
since then) seven of ten students provided a clear indication of the response options they would choose.
Six of these seven students selected options on the lower end of the scale such as “never or hardly
ever” (2 of 6) or “once every few weeks” (4 of 6).
Of the five students who would answer in the lower end of the scale, three explained their choices by
describing a process similar to averaging the frequency of the activity over the course of a school year.
The other two students explained that the response option “about once a week” or higher would be a
poor way of describing this situation.
Prepared by EurekaFacts, LLC
43
ETS CogLab Report: BQ Science 2013
One student chose “every day or almost every day” explaining that they would consider the period of
time that they were working on that project rather than the entire year.
The remaining three students did not specify the response option they would choose and explained that
their answer would depend on how often they studied that topic or how well they would remember it.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
44
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 3
New (Revised based on VC304986)
In your science class this year, how often have you done hands-on activities or projects with any of the
following? Fill in one oval on each line.
a. Living things (for example, plants,
animals, bacteria)
Never
Rarely
A
B
Sometimes
Often
VC304988
C
D
Revised subitem based on
VC304989
b. Electricity (for example, circuits
and energy)
A
B
C
D
c. Chemicals (for example, mixing or
dissolving sugar or salt in water)
A
B
C
D
d. Rocks or minerals (for example,
identifying types)
e. Magnifying glass or microscope
(for looking at small things)
f. Thermometer or barometer (for
making measurements)
g. Simple machines (for example,
pulleys and levers)
A
B
C
D
VC305007
A
B
C
D
VC305008
A
B
C
D
VC305009
A
B
C
D
VC305012
VC304991
Response Process
The students were asked to define what the terms “rarely,” “sometimes,” and “often” meant to them. Five
of the students provided objective, quantifiable answers, and five students provided subjective answers.
o
For the term “rarely,” three of the five who answered objectively indicated that the term means
“once,” and two described it as “once every few weeks.” Of the five who answered subjectively,
three compared the term to the other response categories (“between never and sometimes”).
o
For the term “sometimes,” three of the five who answered objectively indicated that it meant
more than once during the week, and two described it as more than once in a month. Of the five
that answered subjectively, three compared it to the other response categories (”between rarely
and often”).
o
For the term “often,” three of the five who answered objectively indicated that it meant every day,
and two described it as once or more in a week. Of those who answered subjectively, all used a
term to describe a large amount of use such as “very frequently.”
When asked how they would respond if they had only done activities or projects at certain times during
the school year, six of the ten students provided a clear indication of which of the response options they
would choose.
Prepared by EurekaFacts, LLC
45
ETS CogLab Report: BQ Science 2013
o
All six students provided answers at the higher end of the scale by choosing either “sometimes”
(4 of 6) or “often” (2 of 6).
o
Two of these students explained that they based their answers on the amount of time that they
were working on that project rather than the entire school year. Two explained that if they had
spent three weeks on a project they would have learned a lot, and one stated that “rarely” would
not be applicable in the situation described in the probe.
When asked which of the first three item formats they preferred, six students preferred item three, two
preferred item two, and two preferred item one.
o
Of the six students that preferred item three, five stated they believed these response options
better matched their answers compared to the other items.
o
The two students that preferred item two stated that they liked the specificity of the response
options.
o
The two students that preferred item one stated that they liked that the item was straight forward
and easy to answer.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
46
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 4
VC305292
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
About
once a
week
Two or
three
times a
week
Every
day or
almost
every
day
B
C
D
E
VC546510
A
B
C
D
E
Revised subitem based on
VC305295
c. Use the Internet to learn
about science topics
A
B
C
D
E
Revised subitem based on
VC720562
d. Watch a movie, video, or
DVD about science topics
A
B
C
D
E
Revised subitem based on
VC305307
Never or
hardly
ever
Once
every
few
weeks
a. Read a science textbook, in
class or at home
A
b. Read a book or magazine
about science topics
Comprehension
When asked what the phrase “science topics” meant, six students explained that it means academic
subjects such as physics, chemistry, and biology. Four students answered that this phrase means
anything that has to do with science.
Response Process
Five students indicated they read books or magazines about science topics that are not required for
schoolwork.
Four of the five students who indicated they read books or magazines that were not for schoolwork did
not include these books or magazines when answering sub-item “b.” The reason provided for not
including these books or magazines was that they were not being read for their science classes.
When asked if they would respond differently if the item used the term “science” instead of “science
topics,” four of the ten students responded that they would answer the item differently. One student
explained that they would have included activities done at home in addition to those completed in class.
The explanations of the other two students suggest that the phrase “science topics” was interpreted as
referring to more narrow and strictly defined science subjects, while the term ”science” would include a
wider range of subjects.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
47
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 5
VC720622
In your science class this year, how often do you do each of the following? Fill in one oval on each line.
About
once a
week
Two or
three
times a
week
Every
day or
almost
every
day
B
C
D
E
VC720623
A
B
C
D
E
VC720634
c. Talk about measurements you
took for your science project
or activity
A
B
C
D
E
VC720635
d. Talk about the results of your
science project or activity
A
B
C
D
E
VC720636
e. Watch your teacher do a
science experiment or activity
A
B
C
D
E
VC720638
f. Make graphs or charts of the
results from your science
project or activity
A
B
C
D
E
VC720639
g. Write a report on your science
project or activity
A
B
C
D
E
VC720641
h. Write a lab write-up on your
science project or activity
A
B
C
D
E
New
i. Write a science journal
A
B
C
D
E
New
Never
or
hardly
ever
Once
every
few
weeks
a. Identify questions that can be
addressed through science
experiments
A
b. Design a science experiment
Comprehension
When asked what the phrase “talk about” meant in sub-item “c” and “d,” all ten students described it as
“discussing” with the entire class, classmates, or teachers.
When probed, most students (7 of 10) provided similar definitions of the term “lab write-up” in sub-item
“h.”
Prepared by EurekaFacts, LLC
48
ETS CogLab Report: BQ Science 2013
o
These seven students provided explanations that were centered on the idea that it was a written
piece that summarized a science lab. However, three of these students expressed some
confusion about the meaning of the term “lab write-up.”
o
Two students believed that the term “lab write-up” meant answering questions.
o
One student was unable provide a definition.
When initially answering, three students expressed some confusion about the meaning of the term
“science journal” in sub-item “i.” When asked directly what the term meant, the students had varying
responses. While providing a definition, six of the students expressed that they were unsure what the
term meant; of these six, two were unable to guess a definition. Two students believed the term could
mean a long science report. Two other students believed a science journal may be where students keep
their notes for science. Two students thought the term might mean a short daily class or homework
assignment. One student thought a science journal might be where short brief constructed response
items were answered. Another student thought that it might be used to provide a summary of an
experiment for publication.
Response Process
Nine students answered that they had at some point “talked about” measurements or results in subitems “c” or “d.” Of these nine, five gave examples of discussing their results with a teacher and six
students provided an example of talking with classmates.
Of the eight students reporting that they had done a lab write-up, six gave examples of specific labs
where they had completed this activity by summarizing the results of the lab. One student described
doing a lab write-up as answering questions about the lab and one student could not remember any
specifics, but remembered the students had to write a hypothesis.
One student stated that he had written a science journal. He described a middle school assignment for
which he was asked to write a nightly entry about something related to science. He specifically explained
providing a nightly description of the moon.
Recommendation for Improvement
There was considerable variation in how students understood the phrase “science journal.” More than
one half of the students interviewed (6 of 10) expressed that they were unsure of the meaning of the
term. Only one student was able to provide an example of having written a science journal and this
example was drawn from an experience in middle school. We suggest providing a standardized
definition of the term “science journal,” in order to avoid any potential misunderstandings in the future.
Prepared by EurekaFacts, LLC
49
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 6
VC315266
In this school year, how often have you used library resources for science?
A
B
C
D
E
Never or hardly ever
Once every few weeks
About once a week
Two or three times a week
Every day or almost every day
Reading
Overall, students did not exhibit consistent reading problems in item six.
One student skipped reading the response options and proceeded to answer the item.
Comprehension
Overall, students did not exhibit consistent comprehension problems. A majority of students (8 of 10)
stated the item was asking how often information from the library - books, an online resource, or a
librarian - was used for science class. Only one student mentioned that this took place this school year.
When answering the probe, “did you think about your school’s library, a public library, or both,” there was
an even split among students.
o
Five students thought only of the school library resources when answering.
o
Five students thought of both the school and public libraries and their resources.
When asked to provide examples of library resources that they have used for science class, all students
were able to provide at least one example. Some of these included:
o
A majority of students (9 of 10) considered computers and the internet as library resources.
o
Biographies, websites, online journals, internet databases, magazines, autobiographies,
encyclopedia Britannica, and textbooks.
While thinking about his answer, one student was not sure whether the probe was asking about
the use of computers in general, or solely as a means to access the internet.
A majority of students (9 of 10) did not think of videos as a library resource. One of these students
responded to the probe by indicating that while not mentioning videos as a library resource originally, he
had used the library computers to watch online videos.
Recommendations for Improvement
The item does not provide instructions regarding the type of library it referring to, resulting in inconsistent
responses among the students. We suggest that specifying the type of library (school, public or both)
might improve the validity of responses captured by the item.
Prepared by EurekaFacts, LLC
50
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 7
VC305330
Please indicate how much you DISAGREE or AGREE with the following statements about science. Fill in
one oval on each line.
Strongly
Disagree
Disagree
Agree
Strongly
Agree
a. I do science-related activities that are not
for schoolwork.
A
B
C
D
VC305348
b. I like science.
A
B
C
D
VC305350
c. Science is one of my favorite subjects.
A
B
C
D
VC305351
d. I take science only because I have to.
A
B
C
D
VC305352
e. I take science only because it will help
me in the future.
A
B
C
D
VC305353
f. When I graduate from high school, I
would like to have a job related to
science.
A
B
C
D
VC720596
Comprehension
Overall, students did not exhibit consistent comprehension problems. Most students (7 of 10) stated that
the item asked if they agreed or disagreed with these statements about science. Three students stated
that the item is asking about their opinions and feelings about science.
When probed what the phrase “science-related activities” meant to them, a majority of students (9 of 10)
reported that it meant doing anything related to science such as watching a movie or conducting an
experiment.
When asked what the phrase “not for schoolwork” meant to them, a majority of students (9 of 10) stated
that it was something that was not required by the school (something extracurricular).
When asked to provide an example of “science-related activities that you or your classmates have done
or might do that are not for schoolwork,” more than half of the students (6 of 10) were able to provide an
example, such as:
o
Using a telescope, working at NIH, reading Stephen Hawking books, nature treks, and science
fairs.
o
More than half of the students (6 of 10) stated that these activities were not for schoolwork
because a teacher did not instruct them to perform these activities, they would not be graded,
and the activities would not be discussed in class.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
51
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 8
New
In this school year, have you participated in any of the following activities? Fill in one oval on each line.
a. Science fair
Yes
A
No
B
b. Science club
A
B
c.
A
B
Science competition
Comprehension
A majority of students (7 of 10) indicated that the item asked whether they had done the listed activities
during the current school year. Three students thought it referred to all extracurricular activities.
A majority of students (8 of 10) stated that they did not participate in any of the listed activities. Of the
two that had participated in the listed activities, one had participated in a science club and a science
competition; the other student had participated in a science fair. Those who reported participating in the
listed activities also specified that they participated in the activities during the current school year.
When asked what the phrase “science fair” means to them, most students (7 of 10) explained that it
refers to creating a research project and presenting it to other people.
o
Two students did not differentiate the terms “science fair” and “science competition.”
o
One student had never participated in a science fair but had attended one.
Over half of students (6 of 10) were able to provide an example of a science fair, but only two students
mentioned science fairs that were taking place this year. Four students provided examples of science
fairs ranging chronologically from elementary school to their first year of high school. Four students were
unable to provide specific examples of science fairs.
When asked what the phrase “science club” meant to them, students had varied ideas about whether the
club was a school sponsored, extracurricular, or an activity separate from the school altogether. All
students stated that a science club was a club where students meet and discuss science-related topics.
When asked to provide an example of a science club that they or their classmates have participated in or
might participate in, students provided the following examples:
o
Future Doctors and Scientists, Science Bowl club, Medical Masterminds, turtle club, biology,
physics, or chemistry clubs, and an environmental club.
o
Three students were unable to provide an example, or stated that a science club was the same
as a science fair or competition.
Most students (7 of 10) stated that a science competition includes conducting an experiment and being
judged against other students, explaining that it was different from a fair. Two students considered a
science competition to be the same as a science fair or club.
Prepared by EurekaFacts, LLC
53
ETS CogLab Report: BQ Science 2013
When asked to provide an example of a science competition that they or their classmates have
participated in or might participate in, some examples included:
o
Science bowl, Siemens competition, NIH competition, and county competitions.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
54
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 9
New
In this school year, have you visited a science museum?
A
B
Yes
No
Comprehension
Overall, students did not exhibit any consistent comprehension problems.
A majority of students (9 of 10) stated that they were being asked if they had been to a museum that
focuses on science. Only two students indicated that the visit had to take place during the current school
year.
When asked what the phrase “science museum” meant to them, all students stated that it is a museum
that focuses on any field of science.
Response Process
When asked to provide examples of science museums, all students responded and listed:
o
When asked to provide examples of museums that are not for science, all students responded and
listed:
o
The National Gallery of Art, Spy Museum, Native American Museum and Holocaust Museum.
When asked to provide an example of a science museum that they or their classmates have visited or
might visit, all students responded and provided these examples:
o
The National Air and Space Museum, the Natural History Museum, and the Cryptologic
Museum.
The Baltimore Science Museum, the Natural History Museum, the National Air and Space
Museum, and the National Aquarium. One student described “Bodies: the Exhibition,” but was
unable to name it.
Of the four students who were asked who they went to the science museum with this school year, two
went with their class, one went with friends, and one went with family.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
55
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 10
New
In this school year, have you visited a science museum on a school trip?
A
B
Yes
No
Comprehension
Overall, students did not exhibit any consistent comprehension problems.
All students understood that the item was asking about field trips for science class.
All students stated that “on a school trip” means a trip that is sponsored by the school with school
supervision. Most students (8 of 10) were able to report an example of a school trip. They included:
o
The Medical Museum in Walter Reed, the Natural History Museum, the National Aquarium, and
the Air and Space Museum.
None of the students had visited a museum with their school during the current school year.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
56
ETS CogLab Report: BQ Science 2013
Grade 12 Science–Student—Item 11
New
In this school year, have you visited a science museum that was not on a school trip?
A Yes
B No
Comprehension
Overall, students did not exhibit any consistent comprehension problems.
A majority of students (9 of 10) were able to identify that the item was asking if they have visited a
science museum on a trip that was not sponsored by the school.
All students stated that “not on a school trip” meant that the visit was not sponsored by the school, and
done on the students own time.
Response Process
More than half (6 of 10) of students gave examples of museum trips with their families. Some of the
museums mentioned were:
o
The National Air and Space Museum, and the Smithsonian.
A majority of students (9 of 10) reported that they have not visited a museum this year.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
57
ETS CogLab Report: BQ Science 2013
2.2
Teacher Background Questionnaire
2.2.1. Grade 4 Teachers
Grade 4 Science–Teacher—Item 1
New (Revised based onVC305014)
To what extent do you use each of the following student groupings for science instruction in your
classroom? Fill in one oval on each line.
a. Groupings based on students’ interest in
science/science-related topics
b. Groupings based on students’ learning
preferences
c. Groupings based on students’ readiness
needs
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
New
A
B
C
D
New
A
B
C
D
New
Comprehension
The teachers interpreted this item as asking about how students were placed in smaller groups to work
during class and how this related to students’ learning.
All teachers interpreted the phrase “students’ learning preferences” in sub-item “b” to refer to students’
learning styles; one teacher noted that she initially thought it meant “what the student wanted to learn
about,” but changed her interpretation after re-reading.
Teachers interpreted “students’ readiness needs” in sub-item “c” to mean students’ ability levels related
to reading or content comprehension. One teacher noted some confusion with the term, calling it “weird
wording,” and wondered if this referred to how English learners were grouped with other students.
Response Process
Most teachers (3 of 5) noted during probing that they did not group their students during science
lessons, or that they only occasionally had students work in pairs that were not based on any of the
criteria mentioned in this item. However, this did not affect their comprehension of the item.
When asked if it would be efficient for the item to ask “Do you create groups within this class for science
instruction on the basis of ability,” solely with “yes” or “no” response options, all five teachers responded
that they preferred the way the item was currently asked, noting that this would allow them to better
explain how they taught.
One teacher mentioned that he would have preferred to have the ability to provide an open-ended
response.
Prepared by EurekaFacts, LLC
58
ETS CogLab Report: BQ Science 2013
Recommendations for Improvement
Although only one teacher had difficulty interpreting the terms “students’ learning preferences” and
“students’ readiness needs,” we suggest including definitions of these two terms within each sub-item, to
avoid any potential misunderstanding in the future and ensure consistent interpretation of the item
among teachers.
Prepared by EurekaFacts, LLC
59
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Teacher— Item 2
VC970876
To what extent do you emphasize each of the following objectives in teaching science to your fourth-grade
class? Fill in one oval on each line.
Not at
Small
Moderate
Large
all
extent
extent
extent
a. Increase students’ interest in science
A
B
C
D
VC970917
b. Teach scientific facts and principles
A
B
C
D
VC970919
c. Teach scientific methods
A
B
C
D
VC970920
A
B
C
D
Revised subitem based on
VC970922
A
B
C
D
VC970923
f. Develop problem-solving (design) skills
A
B
C
D
VC970925
g. Develop skills in lab techniques
A
B
C
D
VC970926
h. Increase awareness of the importance
of science in daily life
i. Develop systematic observation skills
A
B
C
D
VC970928
A
B
C
D
VC970929
j. Learn about applications of science to
environmental issues
k. Develop scientific writing skills
A
B
C
D
VC970930
A
B
C
D
VC970931
d. Equip students with the knowledge and
skills needed for studying science in
upper grade levels
e. Develop inquiry skills
Comprehension
One teacher did not understand the meaning of the phrase “inquiry skills” in sub-item “e.”
One teacher described the phrase “develop skills in lab techniques” in sub-item “g” as “weird wording.”
One teacher had some difficulty with the phrase “scientific writing skills” in sub-item “k.” She interpreted it
as referring to “lab write-up skills” which, she noted, would apply only to older students. She therefore
found this sub-item to be irrelevant to elementary school teachers and did not appear to comprehend the
sub-item’s intent.
Response Process
One teacher called the term “objectives” in the item problematic, or not relevant to him, because
teachers had to follow rigid county-approved objectives in their teaching.
One teacher stated that sub-item “f” (“develop problem-solving (design) skills”) was asking two different
questions at the same time.
Prepared by EurekaFacts, LLC
60
ETS CogLab Report: BQ Science 2013
A majority of teachers (4 of 5) noted that they interpreted “upper grade levels” in sub-item “d” to refer to
any grade above fourth grade, including middle school; one teacher indicated that it meant seventh
grade or above.
All teachers mentioned referring to school, county, or standards and curricula to understand what
knowledge and skills students would need for studying science in upper grade levels.
Most teachers (3 of 5) stated that they would have answered sub-item “d” the same way if it had stated
“prepare students for further study” instead of “equip students with the knowledge and skills needed.”
One teacher felt that the phrase “knowledge and skills” was necessary for clarification, and another
noted that if the sub-item had only said “prepare students for further study,” she would have also
included “study skills that will transfer to the next grade levels,” and would consequently have selected a
higher frequency response option.
Recommendations for Improvement
Although only one teacher had difficulty interpreting “develop problem-solving (design) skills” in sub-item
“f,” we suggest removing the word “design” or breaking this item into two parts, to avoid any potential
confusion. If separated into two parts, we suggest one sub-item about problem solving skills and another
about design skills. We also suggest keeping the current formulation of sub-item “d,” “Equip students
with the knowledge and skills needed for studying science in upper grade levels.”
Prepared by EurekaFacts, LLC
61
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Teacher— Item 3
VC970932
How much of the following instructional materials and other resources does your school system provide you
with to teach science to your fourth-grade class? Fill in one oval on each line.
None
Little
Some
A Lot
a. Science textbooks
A
B
C
D
VC970953
b. Science magazines and books
A
B
C
D
VC970954
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
A
B
C
D
VC970955
A
B
C
D
VC970956
e. Space to conduct science labs
A
B
C
D
VC970957
f. Computers for students’ use in class
A
B
C
D
VC970958
g. Computer labs
A
B
C
D
VC970959
h. Computers for teachers’ use
A
B
C
D
VC970960
i. Computerized science labs for
classroom use
j. Audiovisual materials
A
B
C
D
VC970961
A
B
C
D
VC970962
k. Science kits
A
B
C
D
VC970963
A
B
C
D
VC970964
l. Scientific measurement instruments
(e.g., telescopes, microscopes,
thermometers, or weighing scales)
Comprehension
One teacher found sub-item “j” and the term “audiovisual” to be “vague.”
There were two occasions where teachers appeared somewhat confused, likely due to forgetting what
the item stem was originally asking. One teacher stated that “computerized science labs for classroom
use” in sub-item “i” was confusing because she could not tell whether it referred strictly to science or to
overall teaching. Another teacher had difficulties answering sub-item “j,” stating that she had audiovisual
materials, but was not provided these resources by her school system.
Prepared by EurekaFacts, LLC
62
ETS CogLab Report: BQ Science 2013
Response Process
When describing what the item was asking, most teachers (3 of 5) said it asked what the “school
system” provided. One teacher specifically noted that this was different from the “school.” Another
teacher, however, said the item asked what the school provided. One teacher did not indicate whether
the question referred to the school system or the school, and merely stated that the question asked
“what kind of resources are available for teaching.”
One teacher found it difficult to choose between response options “B” (“little”) and “C” (“some”) for subitems “c” and “d” (supplies or equipment for science demonstrations, or labs). He indicated that these
sub-items were “hard” because the county (the school system) will no longer pay for “consumables,”
meaning that they do not have all the material they need and he obtains some materials on his own. This
teacher appeared to understand the sub-item, but had difficulty recalling the necessary information to
confidently provide an accurate response.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
63
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Teacher— Item 4
New (Revised based on VC970932)
To what extent does your school provide the following to you? Fill in one oval on each line.
Not at
all
Small
extent
Moderate
extent
Large
extent
a. Science textbooks
A
B
C
D
VC970953
b. Science magazines and books
A
B
C
D
VC970954
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
A
B
C
D
VC970955
A
B
C
D
VC970956
e. Space to conduct science labs
A
B
C
D
VC970957
f. Computers for students’ use in class
A
B
C
D
VC970958
g. Computer labs
A
B
C
D
VC970959
h. Computers for teachers’ use
A
B
C
D
VC970960
i. Computerized science labs for
classroom use
j. Audiovisual materials
A
B
C
D
VC970961
A
B
C
D
VC970962
k. Science kits
A
B
C
D
VC970963
A
B
C
D
VC970964
l. Scientific measurement instruments
(e.g., telescopes, microscopes,
thermometers, or weighing scales)
Comprehension
Two teachers indicated that the different wording of this item, as compared to the previous item, did not
change how they answered this item. However, two other teachers stated that their answers for this item
differed; both noted that they saw a distinction between “school system” (used in previous item) and
“school” (used in the current item), and the role of each entity in providing them with materials. Another
teacher said he had answered a couple of sub-items differently than before, but could not explain why.
Response Process
All teachers stated that they found the options “not at all,” “small extent,” “moderate extent,” and “large
extent” appropriate and sufficient response options.
When asked what the response options “small extent,” “moderate extent,” and “large extent” meant, all
teachers indicated that “small extent” meant there were not enough resources, “moderate extent” meant
Prepared by EurekaFacts, LLC
64
ETS CogLab Report: BQ Science 2013
there was a sufficient amount for class work with students, and “large extent” meant there were more
than enough. Although teachers used different ways of quantifying the amount of resources that they
considered sufficient or insufficient for their needs, they shared a similar interpretation of the response
options.
When asked which version of the response options they preferred, between the third and fourth items, a
majority of teachers (4 of 5) did not seem to understand this probe and selected one particular response
option (such as “small extent”) as their reply. The single teacher who understood the probe stated she
would choose the item with the response options “small extent, etc.” because “it sounds more
professional.”
When asked if they could suggest other response options that would provide more specific information
about resources, one teacher suggested adding the response option “Not Applicable.” Another teacher
had several suggestions: ask separately about science magazines and science textbooks, ask about a
separate lab classroom devoted to science, and ask about safety equipment.
Recommendations for Improvement
The previous item (item three) includes the term “school system,” while item four includes the term
“school.” Because of this difference, two teachers provided different answers for item three and item
four. They inferred that item three was asking about resources provided by the county or school district,
while item four asked about resources from the school. Their responses suggest that they tried to
differentiate between resources provided by the school system (county or school district) and those
provided by the school itself. Teachers’ verbal reports and answers to items three and four show that the
formulation of item three, with the phrase “school system,” may cause confusion among teachers. We
suggest that the final item version provide a more clear and unequivocal formulation of the question.
Interviewer notes suggest teachers had a consistent interpretation of the response options used in item
four, and stated that they are appropriate and sufficient for describing the extent of resources provided
by the school.
Prepared by EurekaFacts, LLC
65
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Teacher— Item 5
VC767811
When you teach science to your fourth-grade class, do you do any of the following? Fill in one oval on each
line.
a. Use a different set of methods in
teaching some students
b. Supplement the regular course
curriculum with additional material for
some students
c. Vary the pace of instruction for some
students
d. Have some students engage in
different classroom activities
e. Set different achievement standards for
some students
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
VC767814
A
B
C
D
VC767820
A
B
C
D
Revised sub-tem
based on
VC767821
A
B
C
D
VC767823
A
B
C
D
VC767824
Comprehension
One teacher said that the meaning of the word “activities” in sub-item “d” was not clear.
Response Process
All teachers understood that the item was asking about different instructional methods they use with their
students.
All teachers interpreted the phrase “vary the pace of instruction for some students” to refer to whether
students could learn at a faster or slower pace based on how quickly they understood the material or
lesson.
All teachers stated that they would have answered sub-item “c” the same way if it had instead read
“pace my teaching differently for some students.” Only one teacher said she would prefer the question
use this alternate wording, because it “probably makes more sense.”
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
66
ETS CogLab Report: BQ Science 2013
Grade 4 Science–Teacher— Item 6
New
To what extent do you provide students with the following learning experiences? Fill in one oval on each
line.
Not at
Small
Moderate
Large
all
extent
extent
extent
a. Science fairs
A
B
C
D
New
b.
Science clubs
A
B
C
D
New
c.
Science competitions
A
B
C
D
New
d. Trips to science museums
A
B
C
D
New
Reading
Overall, teachers did not exhibit consistent reading problems or any difficulty understanding the words in
the item.
Comprehension
A majority of teachers (4 of 5) said they made a distinction between “science fair” and “science
competition.” Three seemed to feel that science fairs were more informal and involved a larger number
of students exhibiting their work. One teacher said that he had not previously thought there was a
distinction, but now did; however, he did not provide further explanation. One teacher stated that, after
thinking about it, she was confused by what “science competition” meant, and how it differed from a
“science fair.” She suggested the questionnaire provide examples of each.
Response Process
Two teachers specifically mentioned that the item asked what outside opportunities teachers were
providing, while others indicated that it asked what the teachers or the school provided. However, all
teachers answered in regard to activities or opportunities the school organized, rather than what the
teacher personally provided.
All teachers felt that the response options provided were appropriate, though one teacher commented
that it would be better to ask teachers to quantify the frequency of events per school year. When asked
the follow-up probe of whether it would better to use frequencies such as certain times a week, a
semester, or a school year, a majority of teachers (4 of 5) said it would be better.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
67
ETS CogLab Report: BQ Science 2013
2.2.2. Grade 8 Teachers
Grade 8 Science–Teacher—Item 1
New (Revised based onVC305014)
To what extent do you use each of the following student groupings for science instruction in your
classroom? Fill in one oval on each line.
a. Groupings based on students’ interest in
science/science-related topics
b. Groupings based on students’ learning
preferences
c. Groupings based on students’ readiness
needs
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
New
A
B
C
D
New
A
B
C
D
New
Comprehension
Three of the teachers interpreted this item as asking about how they group students in the classroom,
and how these groupings are related to students’ learning. Two interpreted this item as asking how they
identify student groupings based on their abilities, learning preferences and interests. However, one
teacher also stated that it is not clear if the item is asking about interests, learning preferences, or
academic factors.
All teachers interpreted the term “student groupings” to mean separating the students into work groups
to collaborate on assignments.
The teachers defined the phrase “students’ learning preferences” in two different ways. Three teachers
explained that the phrase refers to learning style typology, such as auditory, visual or kinetic learners,
while the remaining two teachers interpreted the phrase as referring to more generally understood
interests and preferences.
Three teachers expressed confusion regarding the phrase “students’ readiness needs” in sub-item “c.”
One noted that it could mean "two different things: the desire and willingness to learn," or whether
students "have a background without holes in things like math and reading abilities, and their conceptual
knowledge is on par.” Two stated that the phrase meant students’ academic preparedness to understand
the curriculum content and complete tasks. Two other teachers equated it with either content
accessibility or students’ abilities.
Response Process
A majority of teachers (4 of 5) stated that they use all three grouping methods in their teaching day, but
that they also take into consideration student behaviors to avoid a volatile study environment. Two stated
that this item could be restructured as a yes or no question, asking only whether they use groupings by
student ability in their science class instruction. Three indicated that because the item is not, in one
teacher’s words, “black and white,” it was in fact necessary to ask about various grouping methods.
All teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and “large
extent,” are appropriate for describing the extent of instructional grouping strategies.
Prepared by EurekaFacts, LLC
68
ETS CogLab Report: BQ Science 2013
Recommendations for Improvement
The teachers offered two different interpretations of the phrases “students’ readiness needs” and
“students’ learning preferences.” Therefore, we suggest including standardized definitions of these
phrases within sub-item “b” and “c,” to ensure consistent interpretation of the item among respondents
and avoid any potential confusion in the future.
Prepared by EurekaFacts, LLC
69
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Teacher—Item 2
VC976013
To what extent do you emphasize each of the following objectives in teaching science to your eighth-grade
class? Fill in one oval on each line.
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
VC976015
b. Teach scientific facts and principles
A
B
C
D
VC976017
c. Teach scientific methods
A
B
C
D
VC976018
A
B
C
D
Revised subitem based on
VC976019
A
B
C
D
VC976020
f. Develop problem-solving (design) skills
A
B
C
D
VC976021
g. Develop skills in lab techniques
A
B
C
D
VC976022
h. Increase awareness of the importance
of science in daily life
i. Develop systematic observation skills
A
B
C
D
VC976023
A
B
C
D
VC97625
j. Learn about applications of science to
environmental issues
k. Develop scientific writing skills
A
B
C
D
VC970926
A
B
C
D
VC970927
a. Increase students’ interest in science
d. Equip students with the knowledge and
skills needed for studying science in
upper grade levels
e. Develop inquiry skills
Comprehension
One teacher had difficulty interpreting a number of sub-items, asking how sub-item “e” differed from
asking about the teaching of scientific methods in sub-item “c.” For sub-item “h” (“increase awareness of
the importance of science in daily life”) she noted that it was very similar to what was asked in sub-item
“a” (“increase students’ interest in science”).
A majority of teachers (4 of 5) stated that the item asks how much they focus on different areas and
topics when teaching science.
When asked what the phrase “upper grade levels” means to them, a majority of teachers (4 of 5) stated
that it meant high school level material. One teacher stated that it meant honors biology, chemistry, and
physics, as well as AP high school and college material.
A majority of teachers (4 of 5) stated that the phrase “studying science” meant taking physics, chemistry,
and biology in high school.
Prepared by EurekaFacts, LLC
70
ETS CogLab Report: BQ Science 2013
When asked what the phrase “knowledge and skills needed for studying science in upper grade levels”
meant, all teachers explained that it meant the principles and techniques students need to be successful
in high school science.
Response Process
A majority of teachers (4 of 5) said they determine what knowledge and skills will be needed beyond 8th
grade by following state curriculum standards. One stated that he uses a tie-in with current events, such
as conservation and streams and the Chesapeake Bay.
Most teachers (3 of 5) reported that if the item had been phrased “prepare students for further study in
science,” their answer would change slightly but remain similar. They also stated that “prepare students
for further study in science” means to prepare students for scientific studies in the future.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
71
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Teacher—Item 3
VC976028
How much of the following instructional materials and other resources does your school system provide you
with to teach science to your eighth-grade class? Fill in one oval on each line.
None
Little
Some
A Lot
a. Science textbooks
A
B
C
D
VC976031
b. Science magazines and books
A
B
C
D
VC976032
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
A
B
C
D
VC976034
A
B
C
D
VC976035
e. Space to conduct science labs
A
B
C
D
VC976036
f. Computers for students’ use in class
A
B
C
D
VC976037
g. Computer labs
A
B
C
D
VC976039
h. Computers for teachers’ use
A
B
C
D
VC976040
i. Computerized science labs for
classroom use
j. Audiovisual materials
A
B
C
D
VC976041
A
B
C
D
VC976042
k. Science kits
A
B
C
D
VC976043
A
B
C
D
VC976045
l. Scientific measurement instruments
(e.g., telescopes, microscopes,
thermometers, or weighing scales)
Comprehension
The teachers interviewed displayed inconsistent interpretations of this item. Two interpreted the item as
asking about the resources they have available to carry out all the tasks required for their students to
learn throughout the school year. Three teachers interpreted the item as asking what resources were
provided by their school system, with two teachers specifically mentioning the county or school district.
One teacher was unsure about the meaning of sub-item “j” (Audiovisual materials), and whether it meant
particular software or the actual technology.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
72
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Teacher—Item 4
New (Revised based on VC976028)
To what extent does your school provide the following to you? Fill in one oval on each line.
Not at
all
Small
extent
Moderate
extent
Large
extent
a. Science textbooks
A
B
C
D
VC976031
b. Science magazines and books
A
B
C
D
VC976032
c. Supplies or equipment for science
demonstrations
d. Supplies or equipment for science labs
A
B
C
D
VC976034
A
B
C
D
VC976035
e. Space to conduct science labs
A
B
C
D
VC976036
f. Computers for students’ use in class
A
B
C
D
VC976037
g. Computer labs
A
B
C
D
VC976039
h. Computers for teachers’ use
A
B
C
D
VC976040
i. Computerized science labs for
classroom use
j. Audiovisual materials
A
B
C
D
VC976041
A
B
C
D
VC976042
k. Science kits
A
B
C
D
VC976043
l. Scientific measurement instruments
(e.g., telescopes, microscopes,
thermometers, or weighing scales)
A
B
C
D
VC976045
Comprehension
Three teachers stated that item four is asking the same question as item three. However, one of these
respondents paraphrased the item using the term “school district.”
Two teachers were not entirely clear if the item was asking about their school or their school system.
Both stated that item three and item four were making the distinction between materials provided by the
school district and those provided by the school itself. One of these respondents stated that item three
asks about resources provided by the county, while item four asks about in-house resources and not
those from the county science office. Another noted that the term "school" can have multiple
interpretations in the context of this item.
Prepared by EurekaFacts, LLC
73
ETS CogLab Report: BQ Science 2013
Response Process
Three teachers reported that the way the item four was phrased compared to the way item three was
phrased would not make a difference in how they would answer any of the sub-items listed.
A majority of teachers (4 of 5) stated that the response options “not at all,” “small extent,” “moderate
extent,” and “large extent” are appropriate and sufficient for describing the extent to which resources are
provided by the school.
When asked what the phrases “small extent,” “moderate extent,” and “large extent” meant, a majority of
teachers (4 of 5) said that “small extent” meant to have some resources but not enough, “moderate
extent” meant an adequate amount of resources, and “large extent” meant more than enough resources.
When asked which version of the response options they preferred between the third and fourth items, all
teachers misinterpreted the probe and selected one particular response option (such as “small extent”)
as their reply. A majority of teachers (4 out of 5) stated that they would answer moderate to large extent.
Two teachers gave suggestions for other answer choices that would provide more specific information
about the resources provided by the school. One suggested that the item should be asked in terms of
usage. Another suggested the scale should be none, some, etc.
Recommendations for Improvement
The previous item (item three) includes the term “school system,” while this item includes the term
“school.” Because of this difference two teachers considered item three and item four as two entirely
different questions. They deduced that item three asks about the resources provided by the county,
which is different from item four which asks about available “in-house” school resources. While
answering, they tried to differentiate between resources provided by the school system (county or school
district), and those provided by the school itself. Respondents’ verbal reports and answers to item four
and item three show that the formulation of the item three, which utilizes the phrase “school system,”
may cause confusion among respondents. We suggest that item four provides a more clear and
unequivocal formulation of the question.
Interviewer notes provided no evidence suggesting a preferential type of response scale between items
three and four for teachers. Three of the five teachers indicated they would answer item four the same
as item three, and four of the five stated that the response scales are appropriate and sufficient for
describing the extent of resources provided by the school.
Prepared by EurekaFacts, LLC
74
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Teacher—Item 5
VC976085
When you teach science to your eighth-grade class, do you do any of the following? Fill in one oval on each
line.
a. Use a different set of methods in
teaching some students
b. Supplement the regular course
curriculum with additional material for
some students
c. Vary the pace of instruction for some
students
d. Have some students engage in different
classroom activities
e. Set different achievement standards for
some students
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
VC976086
A
B
C
D
VC976088
A
B
C
D
Revised subitem based on
VC976091
A
B
C
D
VC976092
A
B
C
D
VC976094
Comprehension
A majority of teachers (4 of 5) stated that item five is asking about different instructional methods, and
how they plan and change lesson presentation methods to accommodate students with varying levels of
understanding.
A majority of teachers (4 of 5) also stated that the phrase “vary the pace of instruction for some students”
meant providing students with different amounts of time to complete tasks, based on their different
performance levels. One indicated that the phrase meant more time with computer-based activities.
One teacher did not like the wording “set different achievement standards for some students,” describing
it as a hard question because its wording would suggest that she holds some students to lower
standards; she felt this was untrue, because some students would still master the content, but in a
different way than others.
Three teachers stated that they would have answered sub-item “c” (“vary the pace of instruction for
some students”) the same way if it had instead said “pace my teaching differently for some students.”
Two stated that their answers would be different because they interpreted the first version to be more
student-specific, instead of referring to their overall teaching approach.
All teachers stated that they prefer the current phrasing: “vary the pace of instruction for some students.”
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
75
ETS CogLab Report: BQ Science 2013
Grade 8 Science–Teacher—Item 6
New
To what extent do you provide students with the following learning experiences? Fill in one oval on each
line.
a. Science fairs
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
New
b.
Science clubs
A
B
C
D
New
c.
Science competitions
A
B
C
D
New
d. Trips to science museums
A
B
C
D
New
Comprehension
A majority of teachers (4 of 5) stated that item six asks about science-related activities outside of the
classroom.
When asked if they make a distinction between “science fair” and “science competition,” three teachers
described the judging component as the difference between a science fair and competition. Two
teachers did not find the two to be different.
When asked of different activities offered by the school, teachers provided the following examples:
o
Robotics club, STEM club, and Recycling club
o
Visits to the Koshland Science Museum, USA Science and Engineering Festival, and the
Smithsonian Air and Space Museum.
When asked what types of science museums they take students to, teachers provided the following
examples:
o
Children’s Museum, the National Aquarium, and the Smithsonian Air and Space Museum.
Three teachers stated that the four answer choices, “not at all,” “small extent,” “moderate extent,” and
“large extent,” were sufficient to describe the extent of the learning experience provided to the students.
One thought the choices were sufficient but felt that the term “science museum” was too limiting and
suggested to add science field trips such as taking kids to Chesapeake Bay and testing compounds.
In regards to whether frequencies, such as certain times a week/a semester/a school year, would be
better than the response options provided for this item, opinions were divided. While two teachers
rejected the idea that frequencies would be more relevant response options, two noted that they would
be preferred. One respondent specifically suggested the following options: per quarter, per semester, or
per year.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
76
ETS CogLab Report: BQ Science 2013
2.3
Administrator Background Questionnaire
2.3.1. Grade 4 Administrators
Grade 4 Science–School—Item 1
New (Revised based on VE013369)
Is there a science specialist or coach available (full- or part-time) to fourth-grade teachers at your
school?
A
B
C
Yes, available full-time at my school.
Yes, available part-time at my school.
No
Comprehension
Administrators were not unified in their understanding of what the role and responsibilities would be for a
“science specialist or coach.” One understood “specialist” and “coach” to have different meanings,
explaining that a specialist works with students to teach them concepts directly, while a coach works with
teachers to improve teaching practices and planning.
Response Process
Although a majority of administrators (4 of 5) indicated that they did not have a part-time or full-time
science specialist or coach in their school, they emphasized that the responsibilities they associated with
the role were taken on by classroom teachers and/or other types of specialists (e.g. instructional support
teacher).
The one administrator who indicated that she did have a STEM (Science, Technology, Engineering, and
Mathematics) teacher on staff indicated that it was a full-time staff member, and explained that the
STEM teacher was responsible for helping teachers plan their lessons and demonstrations.
Of the administrators who indicated that they did not have a science specialist or coach on staff, a
majority (3 of 4) indicated that the responsibilities were divided among classroom teachers at their
schools. One administrator indicated that she does have an “instructional support teacher” in the fields of
literacy and science that she would consider as fulfilling the role of a science specialist or coach.
Recommendations for Improvement
As interpretations of the term “science specialist or coach” seemed to vary among administrators, it may
be helpful to include a short definition. It was also important to the administrators to qualify a “no”
response to this item by explaining that although they did not have a designated full or part-time science
specialist or coach, they did have individuals who provided the services and responsibilities they
associate with this role. In order to capture this information, we recommend expanding the range of the
response options to include additional answer choices, such as “Yes, available on an as needed basis,”
and/or “No, but other individuals within the school provide the services of a science specialist or coach.”
Prepared by EurekaFacts, LLC
77
ETS CogLab Report: BQ Science 2013
Grade 4 Science–School—Item 2
New (Revised stem based on VE013372)
To what extent are any of the following a responsibility of the science specialist or coach available to fourthgrade teachers at your school? Fill in one oval on each line.
a. Provide technical assistance/support to
individual teachers about science content
or the teaching of science
b. Conduct professional development for
groups of teachers about science content
or the teaching of science
c. Provide science instruction to students
on various topics
d. Translate science standards and current
research into classroom practice
e. Provide science enrichment to some
student groups
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
VE013373
A
B
C
D
VE013382
A
B
C
D
VE013383
A
B
C
D
Revised sub-item
based on
VE013384
A
B
C
D
VE013385
Comprehension
All administrators interpreted sub-item “d” (“Translate science standards and current research into
classroom practice”) as meaning the ability to: understand and internalize science standards, use that
ability to create learning plans and objectives easily understood by students, and use appropriate
vocabulary and teaching methods.
When asked what sub-item “e” (“Provide science enrichment to some student groups”) meant to them,
all administrators indicated that it meant some student groups would receive more attention or different
activities than the rest of the class. A majority of administrators (4 of 5) indicated that the enrichment
activities would apply to students who wanted to go beyond what was being taught in the classroom.
One understood “some student groups” as students who may be struggling with the curriculum (such as
those enrolled in ESOL or remedial classes), and who need additional activities to help them catch up to
their peers.
Response Process
All administrators felt that the sub-items covered the majority of the responsibilities they would consider
part of the role of a science specialist or coach. Additional responsibilities mentioned by administrators
included the incorporation of literacy skills into science instruction, and motivating trends within the
school to assess how well students understood the core science standards.
As mentioned earlier, only one out of five administrators reported having a full- or part-time science
specialist or coach. This resulted in a majority of administrators (4 of 5) feeling unsure of how to provide
appropriate responses to the sub-items. However, when probed about the response options, all
administrators felt that they were appropriate.
Prepared by EurekaFacts, LLC
78
ETS CogLab Report: BQ Science 2013
o
Most administrators (3 of 5) who did not have a science specialist or coach on staff answered
the sub-items as if they referred to the staff that performed each responsibility (teachers,
administrators, etc.).
o
One administrator mentioned that she would have replied “not applicable” for each of the subitems if the response option had been available, since she did not have a science specialist or
coach on staff.
Recommendations for Improvement
Depending on the item’s intent, there are two ways to address administrators’ response process
challenges. First, if the intent of the item is to measure to what extent these responsibilities are
performed at the school, we recommend modifying item so that it does not specifically refer to a “science
specialist or coach.” Second, if the intent of the item is to ask specifically about the activities of a
“science specialist or coach,” we recommend an additional “not applicable” response category for those
administrators who do not have one on staff.
Prepared by EurekaFacts, LLC
79
ETS CogLab Report: BQ Science 2013
Grade 4 Science–School—Item 3
New
To what extent does your school provide students with the following opportunities? Fill in one oval on
each line.
a. Science fairs
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
New
b.
Science clubs
A
B
C
D
New
c.
Science competitions
A
B
C
D
New
d. Trips to science museums
A
B
C
D
New
Comprehension
All administrators interpreted this item as asking about science-related opportunities the school provides
beyond the traditional classroom instruction.
Although most administrators (3 of 5) agreed that there is a distinction between a “science fair” and a
“science competition,” definitions of each term differed.
o
Two characterized a “science fair” as being open to all students in the school, and indicated that
students pursued individual interests.
o
Two characterized a “science competition” as being more structured, with students receiving a
certain question or task and competing against one another.
o
One did not differentiate between the two terms, but guessed that a “science competition”
involved asking students questions about science. Another administrator did not understand the
term “science competition,” but guessed that it would be similar to a spelling or geography bee,
and that students would be asked questions related to science.
When asked for examples of how the school provided opportunities for students, administrators
mentioned included school and county-based science fairs, gardening and recycling clubs, and other
environmental clubs.
When asked specifically about trips to science museums, administrators mentioned the National
Aquarium, the Smithsonian Natural History Museum, the Baltimore Science Center, NASA, and the
Smithsonian Air and Space Museum.
Response Process
A majority of administrators (4 of 5) preferred the response options of “large extent, moderate extent,
and small extent,” versus frequencies such as certain times a week/semester/school years.
One administrator suggested merging the two categories of response options, changing the question to
ask “To what extent does your school provide students with the following opportunities per school year?”
She also suggested adding ranges to qualify what was meant by a small, moderate and large extent
Prepared by EurekaFacts, LLC
80
ETS CogLab Report: BQ Science 2013
(e.g. “small”, 0-1 times per school year, “moderate,” 2-3 times per school year, “large,” 4 or more times
per school year).
Recommendations for Improvement
As the definitions for “science fair” and “science competition” differed among administrators, it is
recommended that descriptions or definitions be provided for these terms to ensure consistent
understanding of this term across administrators.
Prepared by EurekaFacts, LLC
81
ETS CogLab Report: BQ Science 2013
2.3.2. Grade 8 Administrators
Grade 8 Science–School—Item 1
New (Revised based on VE013964)
Is there a science specialist or coach available (full- or part-time) to eighth-grade teachers at your
school?
A
B
C
Yes, available full-time at my school.
Yes, available part-time at my school.
No
Response process
All administrators selected option “C,” because their schools do not have a science specialist or coach
available to eighth-grade teachers. Three administrators noted that a teacher or administrator would
perform some responsibilities of a science coach at certain times; however, they did not have a
specialist or coach dedicated solely to science.
One administrator displayed a strong discomfort stemming from the fact that his school does not have a
science specialist or coach position, and defended and explained the school’s situation.
When asked to describe the position or role of a “science specialist or coach,” administrators identified
responsibilities such as providing support and help with development of curricula, lesson plans, and
meeting standards for science instruction.
When asked whether a different term is used in their school to describe individuals with the
responsibilities of a science specialist, most administrators (3 of 5) answered that a similar type of
position existed in other subject areas (e.g., math, social studies), and was described as either an
“instructional lead teacher,” “instructional coach,” or “mentor teacher.” Two administrators noted that the
responsibilities associated with a “science specialist” were performed by the principal or the assistant
principal.
Recommendations for Improvement
Although all administrators responded that their schools do not have a science specialist or coach
available full- or part-time to eighth-grade teachers, further probing revealed that the responsibilities of
this role are fulfilled by teachers or administrators who do not hold an official position as a science
specialist or coach. In order to improve the collection of this information, we suggest expanding the
range of response options to include “Yes, available on an as-needed basis.”
Prepared by EurekaFacts, LLC
82
ETS CogLab Report: BQ Science 2013
Grade 8 Science–School—Item 2
New (Revised stem based on VE013966)
To what extent are any of the following a responsibility of the science specialist or coach available to eighthgrade teachers at your school? Fill in one oval on each line.
a. Provide technical assistance/support to
individual teachers about science
content or the teaching of science
b. Conduct professional development for
groups of teachers about science
content or the teaching of science
c. Provide science instruction to students
on various topics
d. Translate science standards and current
research into classroom practice
e. Provide science enrichment to some
student groups
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
VE013967
A
B
C
D
VE013968
A
B
C
D
VE013969
A
B
C
D
Revised subitem based on
VE013970
A
B
C
D
VE013971
Comprehension
Two administrators exhibited some confusion or difficulty understanding this item. One stated that subitem “a” is “too wordy” and needed to reread it in order to understand it. The use of the word “translate”
in sub-item “d” was also confusing to this administrator. The other administrator was unsure about the
intent of the item, and tried to clarify whether it was referring to what his school currently offers or what
he envisions a science specialist would do if there were such a role at his school.
Response Process
Given his school’s lack of an available science specialist for eighth-grade teachers, one administrator
answered the item based on the extent to which the services described in each sub-item are available to
teachers. For the same reason, another administrator based his answers on the role of the science
department chair; this individual serves as an instructional coach but does not fit the administrator’s
definition of a science coach.
When probed about the phrase “translate science standards and current research into classroom
practice,” administrators stated that it means explaining or understanding “what the standard is asking us
[teachers, students] to do,” explaining what the standards “mean for day to day instruction,” and how “to
help map these standards into daily objectives,” or measureable objectives.
A majority of administrators (4 of 5) had consistent and similar interpretations of “provide science
enrichment to some student groups,” as additional activities that go beyond the standards or
requirements set in curriculum. These activities may be provided to a small group of students needing
additional practice, or to a group that has already mastered the curricula standards and requires more
Prepared by EurekaFacts, LLC
83
ETS CogLab Report: BQ Science 2013
differentiated materials. As examples, administrators provided such activities as “dissection, fun
experiments,” and scientific investigations.
Most (3 of 5) administrators agreed that the activities presented in the sub-items cover all main
responsibilities of a science specialist or coach. Two stated that sub-items “a” and “c” are not very
relevant to the situation in their schools, and are specific to the person with training in science.
Two of the five administrators agreed that the four answer choices, “not at all,” “small extent,” “moderate
extent,” and “large extent,” are sufficient for describing the extent of a science specialist’s or coach’s
responsibilities in their school. One administrator stated a preference for a numeric scale, and another
suggested a three-point scale that would combine small and moderate extent response options, because
he had a problem deciding which option is more relevant to his school. The last administrator
misinterpreted the probe.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
84
ETS CogLab Report: BQ Science 2013
Grade 8 Science–School—Item 3
New
To what extent does your school provide students with the following opportunities? Fill in one oval on each
line.
a. Science fairs
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
New
b.
Science clubs
A
B
C
D
New
c.
Science competitions
A
B
C
D
New
d. Trips to science museums
A
B
C
D
New
Comprehension
Overall, administrators did not exhibit any consistent problems with this item.
The administrators interpreted the item as asking if the school provides opportunities to participate in
science related activities beyond the regular classroom and outside of school hours.
Response Process
All administrators made a clear distinction between “science fair” and “science competition,” and defined
the differences between these two concepts as follows:
o
The science fair has a larger scale and extent, and gives all students the opportunity to
participate.
o
The competition is limited to the science clubs and some specific topics or areas of
specialization.
Administrators reported the following examples of opportunities provided by the school:
o
Trips: outdoor education at Camp Schmidt with hydroponic gardens, Howard Owens Science
Center for Astronomy.
o
Science competitions: robotics tournaments, Science Olympia competition.
o
Science clubs: robotics club, science club, environmental club.
Administrators reported the following examples of field trips to science museums organized by the
school:
o
The Owens Science Center, Bladensburg Waterfront Park, Maryland Science Museum in
Baltimore, Smithsonian museums in Washington DC (e.g., Air and Space Museum, Museum of
Natural History).
Four of the five administrators agreed that the four answer choices, “not at all,” “small extent,” “moderate
extent,” and “large extent,” are sufficient for describing the extent of the opportunities their school
provides; the last administrator misinterpreted the probe.
Prepared by EurekaFacts, LLC
85
ETS CogLab Report: BQ Science 2013
Administrators had diverse opinions about the use of frequencies, such as certain times a week/
semester/school year, and whether these would be better response options for this item. While one
administrator noted that it would be appropriate for the science fair and clubs, another stated it would not
be applicable for science fairs, because they happen only once a year. Two administrators indicated a
preference for frequency-based response options, accompanied by a numeric scale.
Recommendations for Improvement
There are no recommendations for improvement for this item based on the evidence collected from the
interviews.
Prepared by EurekaFacts, LLC
86
ETS CogLab Report: BQ Science 2013
2.3.3. Grade 12 Administrators
Grade 12 Science–School—Item 1
New (Revised based on VE015623)
Is there a science specialist or coach available (full- or part-time) to twelfth-grade teachers at
your school?
A Yes, available full-time at my school.
B Yes, available part-time at my school.
C No
Comprehension
Overall, the administrators did not exhibit any consistent problems with this item. One administrator
indicated that he did not understand what the term “science specialist or coach” was meant to convey.
However, when asked to describe the role of a “science specialist or coach,” he indicated that it was a
“person that helps with the content: an expert with a toolbox of resources to develop lesson plans, field
trips, etc., and who observes classes to coach teachers on better practices.”
All administrators gave similar definitions of a “science specialist or coach,” indicating that an individual
in this role would be a subject matter expert who helps to coach and provide instructional support to
teachers.
Response Process
Two administrators answered “no,” explaining that although they did have an individual fulfilling the role
of a “science specialist or coach,” they did not consider this person as being employed either full- or parttime. One of these administrators indicated that the “science specialist or coach” was available on an as
needed basis, and the other explained that the “science specialist or coach” was not dedicated solely to
th
12 grade.
Two administrators selected the response option, “yes, available part-time.” One explained that the
individual taught class in the morning and was available as a “science specialist or coach” in the
afternoon. The other administrator explained that the role of the “science specialist or coach” was fulfilled
by two individuals – one who works with more than one school, and another with a full teaching load,
who also took on the responsibilities of a “science specialist or coach.”
One administrator chose not to select any of the response options, indicating that none of these options
fit his school’s situation. He further explained that although a resource teacher is available, this teacher
th
is responsible for all high schools in the county, and that the 12 grade teachers in his school rarely ask
for support from this individual.
When asked if there is a different term used in their school for describing individuals with the
responsibilities of a “science specialist or coach,” administrators noted a “resource teacher,” “science
department chairperson,” and “science coordinator.”
Prepared by EurekaFacts, LLC
87
ETS CogLab Report: BQ Science 2013
Recommendations for Improvement
Although administrators were fairly unified in their definitions of the roles and responsibilities of a
“science specialist or coach,” it may be helpful to provide a specific definition or additional descriptive
terms.
Overall, administrators felt that the response options did not fit their internally generated answer. The
addition of a response option such as “yes, on an as-needed basis” may help to alleviate this difficulty.
Prepared by EurekaFacts, LLC
88
ETS CogLab Report: BQ Science 2013
Grade 12 Science–School—Item 2
New (Revised stem based on VE015624)
To what extent are any of the following a responsibility of the science specialist or coach available to
twelfth-grade teachers at your school? Fill in one oval on each line.
a. Provide technical assistance/support to
individual teachers about science content
or the teaching of science
b. Conduct professional development for
groups of teachers about science content
or the teaching of science
c. Provide science instruction to students on
various topics
d. Translate science standards and current
research into classroom practice
e. Provide science enrichment to some
student groups
Not at
all
Small
extent
Moderate
extent
Large
extent
A
B
C
D
VE015625
A
B
C
D
VE015626
A
B
C
D
VE015627
A
B
C
D
Revised
sub-items
based on
VE015628
A
B
C
D
VE015629
Comprehension
Overall, administrators were able to comprehend the item and sub-items. One administrator expressed
confusion over the term “science enrichment” in sub-item “e.” However, when probed, his definition
matched those of the other administrators.
All administrators interpreted “translate science standards and current research into classroom practice”
as meaning the ability to understand and internalize science standards, and using that knowledge to
create learning plans and objectives easily understood by students, and include appropriate vocabulary
and teaching methods. When asked for terms that might be more meaningful for school administrators,
one administrator suggested “curriculum review or standards based instructional development.”
When asked what “provide science enrichment to some student groups” meant to them, all
administrators indicated that it meant that some student groups would receive more attention or different
activities than the rest of the class, particularly opportunities that may go beyond the classroom. Most
administrators (3 of 5) indicated that the enrichment activities would apply to students who excelled in
the subject matter and wanted to expand their knowledge base, while two administrators also explained
that “some student groups” could also be interpreted as students who may be struggling with the
curriculum (such as those enrolled in remedial classes) and are in need of additional activities to help
them catch up to their peers.
Response Process
All administrators felt that the sub-items covered the majority of responsibilities they would consider as
parts of the role of science specialist or coach. Additional responsibilities mentioned by administrators
included facilitating training regarding new technology, determining the best resource materials
(textbooks, secondary documents, etc.) to use, and facilitating hands-on experiences and field trips.
Prepared by EurekaFacts, LLC
89
ETS CogLab Report: BQ Science 2013
The two administrators who reported that they did not have a “science specialist or coach” seemed
unsure about appropriate responses to the sub-items. However, when probed about the response
options (not at all, small extent, etc.), all administrators felt that they were appropriate.
o
One administrator mentioned that he would have replied “not applicable” for each of the subitems if that response option had been available, since he did not have a science specialist or
coach on staff.
o
One administrator suggested using numbers (1-4) to represent the response options, rather than
letters (“A”-“D”).
Recommendations for Improvement
Depending on the item’s intent, two alternatives could address administrators’ response process
concerns. If the intent of the item is to measure to what extent these responsibilities are performed at the
school, we recommend changing item so that it does not specifically identify a “science specialist or
coach.” If the intent of the item is to ask specifically about the role of the science specialist or coach, we
recommend a “not applicable” response option for those administrators who do not have one on staff.
Prepared by EurekaFacts, LLC
90
ETS CogLab Report: BQ Science 2013
Grade 12 Science–School—Item 3
New
To what extent does your school provide students with the following opportunities? Fill in one oval on each
line.
a. Science fairs
Not at
all
A
Small
extent
B
Moderate
extent
C
Large
extent
D
New
b.
Science clubs
A
B
C
D
New
c.
Science competitions
A
B
C
D
New
d. Trips to science museums
A
B
C
D
New
Comprehension
All administrators interpreted this item as asking about science-related opportunities that the school
provides beyond traditional classroom instruction.
Two administrators indicated that the term “science museums” is unclear, since it could incorporate both
traditional museums, such as a natural history museum, and locations like aquariums or other similar
sites. Both of these administrators suggested that the sub-item should be clarified.
Most administrators (3 of 5) agreed that there is a distinction between a “science fair” and a “science
competition.”
o
Two administrators characterized a “science fair” as an on-site exhibition where students
demonstrate a science project or experiment to others.
o
All administrators characterized a “science competition” as where students compete against
each other; two of the five described it as more structured, where students may be given a
certain question or task.
o
One administrator did not differentiate between the two terms, explaining that in her school the
“science fair” is a competition where students can win first place, second place, and honorable
mentions.
Response Process
When asked for examples of how the school provided opportunities for students, administrators
mentioned robotics competitions, natural history museums, aquariums, environmental science clubs,
annual science fairs, and electronic science fairs.
When asked specifically about trips to science museums, administrators mentioned the National
Aquarium, the Chesapeake Bay Foundation, the National Electronics Museum, the Smithsonian Natural
History Museum, and the National Institutes of Health.
Most administrators (3 of 5) expressed a preference for having response options with frequencies, such
as certain times a week/a semester/a school year, explaining that it would be clearer and easier to
Prepared by EurekaFacts, LLC
91
ETS CogLab Report: BQ Science 2013
quantify their answers. One administrator indicated that the best option would be to base answers on a
school year, since it would be harder to quantify in weeks.
One administrator suggested adding ranges to qualify what was meant by a small, moderate and large
extent. He suggested small extent would mean one to three times per year, moderate extent would
mean four to seven times per year, and large extent would be eight or more.
Recommendations for Improvement
Verbal reports and answers to probing questions show that the sub-items do not provide administrators
with a chance to report all types of trips used for science classes. It may be helpful to specify how
narrowly or widely the term “science museum” should be interpreted within the context of this item, and
whether to include non-traditional museums (e.g., aquariums and arboretums). Alternately, additional
sub-items could be provided to capture other types of trips that can be used for science instruction.
Lastly, most administrators seemed to have difficulty using the response options of “small, moderate and
large extent,” explaining that the terms were subjective. Overall, administrators preferred frequencies in
the response options instead.
Prepared by EurekaFacts, LLC
92
This page was intentionally left blank.
This page was intentionally left blank.
2015 Science Survey Questionnaire Cognitive Interview Study, January 2013
Appendices
183
Contents
Interview Protocol: Generic Probes and Think-Aloud Hints for the Interviewer ............................... 185
Science Student Questionnaire Interview Protocal: Items and Probes.............................................. 187
Recruitment and Operational Forms, Letters, and Scripts ................................................................. 188
Parent or Legal Guardian Consent Form ..................................................................................... 188
Student (age 18 or older) Consent Form..................................................................................... 189
Teacher and School Administrator Consent Form ...................................................................... 190
Phone Script Recruiting Parents or Legal Guardians of Student Participants ............................ 191
E-mail Recruiting Parents or Legal Guardians of Student Participants ....................................... 194
Letter Recruiting Parents or Legal Guardians of Student Participants ....................................... 195
Phone Script Recruiting Teachers and School Administrators .................................................... 196
E-mail Recruiting Teachers and School Administrators .............................................................. 199
Letter Recruiting Teachers and School Administrators............................................................... 200
Email/Letter Confirmation to Parent or Legal Guardian of Student (under age 18)
Participants .................................................................................................................................. 201
Email/Letter Confirmation to Student (age 18 or older) Participants ........................................ 202
Thank You Letter to Parent or Legal Guardian of Student (under age 18) Participants ............. 203
Thank You Letter to Student (age 18 or older) Participants ....................................................... 204
Thank You Letter to Teachers and School Administrators .......................................................... 205
184
Interview Protocol: Generic Probes and Think-Aloud Hints for
the Interviewer
If the participant is silent for 5 or more seconds, use the following as a guideline for encouraging
the participant to read the question aloud and say his/her thoughts, or to help him/her elaborate
the responses.
•
•
•
Please read the question aloud and tell me what you are thinking.
You mentioned __________, tell me more about that.
You mentioned __________, what was that like for you?
We are interested in capturing all the participant’s mental processes while answering these
questions. Your goal is to have the participant speak aloud all his or her thoughts while
answering the questions. By asking follow-up probes after each item, you will ensure that the
data collected are as complete as possible. Provide positive feedback by telling the participant
when he or she has given helpful information. The participant may provide more detailed
responses if given positive feedback such as:
•
•
•
This information is very helpful.
The example you just gave is very useful.
Thank you for providing such a thorough response.
If a participant is continually providing short responses or not answering, use “continuers” to
encourage the participant to be more descriptive. The goal is to get participants to verbalize their
thoughts without “putting words in their mouth.” Do not lead the participant’s response by
asking a question that implies or contains its own answer such as:
•
•
Leading Question: Do you dislike science class like most students?
Neutral Question: How do you feel about science class?
You have to be as objective and unbiased as possible, but you may offer a verbal “nudge,” such
as:
•
•
•
What are you thinking now?
Any other thoughts?
Tell me how you came to pick that answer.
Use your best judgment. If a participant exhibits difficulty reading or understanding an item,
probe the participant without biasing the response.
•
•
Can you tell me in your own words what that question was asking?
[comprehension and interpretation probe]
What does the word [term] mean to you as it’s used in this question?
[comprehension and interpretation probe]
185
Use your best judgment. If a participant appears to have guessed or seems less than certain of the
answer he/she has given, probe the participant.
•
•
•
•
•
•
•
How much would you say you know about [topic]? [confidence probe]
How sure are you of your answer? [confidence probe]
How did you come up with that answer? [recall strategy and bias probe]
Was this easy or hard to answer? [comprehension and recall probe]
How easy or difficult is it to remember [topic]? [recall probe]
Did you find the list of answer choices easy or hard? [response probe]
How easy or hard was it to choose an answer from that list of choices? [response
probe]
Use your best judgment. If a participant appears uncomfortable with a word or answering a
question, probe the participant.
•
•
The question uses the word [term]. Does that sound OK to you, or would you
choose something different? [sensitivity probe]
In general, how do you feel about this question? [sensitivity probe]
Use your best judgment. After testing an item, using the above generic probes to understand why
the question as written is not clear, you may reword the question in a way that the participant
will understand based on shared interviewer feedback or previous interview experiences. If this
is done, be sure to document precisely the reworded question that was tested and test the item
using relevant generic probes.
In addition to generic probes, interviewer protocols will include item-specific probes. Not all
items have an item-specific probe. Item-specific probes are provided only if the researchers
deemed it important to learn more about a cognitive process or issue that might not be raised
naturally from the think-aloud process or generic probing. If the issue to be inquired through the
item-specific probe was addressed, or if time is running short, the item-specific probe does not
need to be asked. Use your best judgment.
186
Science Student Questionnaire Interview Protocal: Items and
Probes
Interviewer Note: Proceed in the following manner for each of the survey items in this section:
•
•
•
•
Direct the participant to the item and ask him/her to read the question (and any preceding
or following instructions) out loud
Instruct the participant to answer the question and to think aloud while doing so
Record the participant’s comments
Ask generic and item-specific probes, as appropriate, before going on to the next item
Note: Some items are grade specific and will be asked only of participants for that grade. The
grade, subject, and respondent associated with the items are identified in brackets (e.g., [Grade 4
Science–Student #1]) preceding the item.
Note: Each item to be tested is presented on its own page.
187
Recruitment and Operational Forms, Letters, and Scripts
Parent or Legal Guardian Consent Form
Dear Parent or Legal Guardian:
On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of
Education, Educational Testing Service and EurekaFacts are conducting voluntary research
interviews with grade students about survey questions for the National
Assessment of Educational Progress (NAEP). Each student interview will take no more than one
hour. The purpose of these interviews is not to test or grade students. NCES wants to hear their
thoughts to improve the survey questions and make the questions easier to understand.
The interviewer will meet one-on-one with each student to ask for his or her input. These
sessions will be recorded for the researchers on this study to review. No one else will see or hear
the recording. Your child’s answers will be used only for statistical purposes and may not be
disclosed or used, in identifiable form, for any other purpose except as required by law
[Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].
These interviews are an important stage of the NAEP survey development process. The
comments students provide help NCES improve the survey questions to make the best survey
possible. Your child will be given a $25 gift card as thanks for his or her time. In addition, you
will be given a $25 gift card for your time to take the participating student to and from the
interview site.
We hope that you will give your consent for your son or daughter to participate in the voluntary
NAEP interview by signing this form. Without your consent, your child will not be able to
participate in the interview.
Respectfully,
Laurie Policastro
NAEP Survey Development
Educational Testing Service
I hereby give my permission for _____________________________________________ to
(Student’s Name)
participate in the NAEP survey interview.
____________________________________
Signature
________________________
Date
188
Student (age 18 or older) Consent Form
Dear Student:
On behalf of The National Center for Education Statistics (NCES), part of the U.S. Department
of Education, Educational Testing Service and EurekaFacts are conducting voluntary research
interviews with students about survey questions for the National Assessment of Educational
Progress (NAEP). Each student interview will take no more than one hour.
NCES needs to know if the survey questions are clear. Your feedback will help NCES improve
the survey so students like you understand all the questions. You will be given questions that are
very similar to the actual questions that will be asked during the national testing. The interviewer
will pay very close attention to your opinions and feelings about all aspects of the survey. He or
she will also ask what you think about the questions after you complete them.
You will not be judged on how well you answer the questions, and you will not receive a grade.
Other students like you will also participate in one-on-one sessions just like this. This session
will be recorded for the researchers on this study to review. No one else will see or hear the
recording. Your answers will be used only for statistical purposes and may not be disclosed or
used, in identifiable form, for any other purpose except as required by law [Education Sciences
Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].
These interviews are an important stage of the NAEP survey development process. Your
comments, along with the comments of other students, will help NCES improve the survey. You
will be given a $25 gift card in compensation for your time and effort. In addition, NCES is
offering a gift card of $25 for a parent or legal guardian to remunerate him or her for the time
involved and to help offset the travel/transportation costs of taking you to and from the
EurekaFacts cognitive laboratory site. We thank you for your voluntary participation and ask
you to please sign the form below.
Thank you,
Laurie Policastro
NAEP Survey Questionnaires
Educational Testing Service
I hereby accept the terms described above and agree to participate voluntarily in the NAEP
survey interview.
__________________________________
Print Name
____________________________________
Signature
________________________
Date
189
Teacher and School Administrator Consent Form
Dear Educator:
On behalf of The National Center for Education Statistics (NCES), part of the U.S. Department
of Education, Educational Testing Service and EurekaFacts are conducting voluntary research
interviews with teachers and school administrators about survey questions for the National
Assessment of Educational Progress (NAEP). Each teacher and school administrator interview
will take no more than one hour.
NCES needs to know if the survey questions are clear. Your feedback will help NCES improve
the survey so educators like you understand all the questions. You will be given questions that
are very similar to the actual questions that will be asked of teachers and school administrators
whose students participate in future NAEP assessments. The interviewer will pay very close
attention to your opinions and feelings about all aspects of the survey. He or she will also ask
what you think about the questions after you complete them.
Other educators like you will also participate in one-on-one sessions just like this. This session
will be recorded for the researchers on this study to review. No one else will hear the recording,
or see your responses or comments. Your answers will be used only for statistical purposes and
may not be disclosed or used, in identifiable form, for any other purpose except as required by
law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].
These interviews are an important stage of the NAEP survey development process. Your
comments, along with the comments of other educators, will help NCES improve the survey. In
addition, you will be given a $40 gift card in compensation for your time and effort. We thank
you for your voluntary participation and ask you to please sign the form below.
Thank you,
Laurie Policastro
NAEP Survey Questionnaires
Educational Testing Service
I hereby accept the terms described above and agree to participate voluntarily in the NAEP
survey interview.
__________________________________
Print Name
____________________________________
Signature
________________________
Date
190
Phone Script Recruiting Parents or Legal Guardians of Student Participants
Good morning/afternoon. May I please speak to ?
Good morning/afternoon. My name is and I am contacting you from
EurekaFacts, a local research company. We are conducting an important study for the U.S.
Department of Education’s National Center for Education Statistics (NCES).
We are recruiting students to participate in voluntary research interviews supporting the
development of survey questions for the National Assessment of Educational Progress (NAEP).
The survey will be administered to students throughout the United States. The research
interviews will explore how students work through the survey questions to ensure that they are
clear and understandable to a broad range of students. The students who qualify for and
participate in the study will be given a $25 gift card as thanks for their time. In addition, you will
be given a $25 gift card for your time to take the participating student to and from the interview
site.
The interviews will take place at the EurekaFacts cognitive laboratory site in Rockville,
Maryland and will last no more than one hour. During the interview, your child will be asked to
respond to several survey questions. NCES simply wants to know how he or she would answer
the questions if they were included on an actual survey given to students after taking a NAEP
test.
These interviews are an important stage of the NAEP survey development process. The
comments students provide help NCES improve the survey questions in order to create the best
survey possible.
Would your child be interested in participating in this research project?
[If No] Thank you for taking the time to speak with us today about this important project. Have a
wonderful day/evening.
[If Yes] Great, thank you very much. To ensure that we interview a broad mix of students, I have
a few brief screening questions to ask about you and your child. It should only take a few
minutes of your time. First, I want to collect some additional information about your child to be
certain he or she is a good fit for participating.
Screener
[Interviewer: Determine recruitment needs in advance. Record all responses. End screener at
whatever point the parent or legal guardian gives a response that confirms that their group has
already been sampled adequately. End screener script: Based on the requirements of this study,
we are not able to include your child in the interviews at this time.]
191
1. What is your child’s name?
2. Is a boy or a girl?
3. What grade is currently in? (If the student is in twelfth-grade, ask
if he or she is age 18 or over. This information will be used later in the script.)
4. What is your child’s race or ethnicity? [Interviewer: Only read list if parent doesn’t
immediately reply] Are you…?
a. American Indian or Alaska Native
b. Asian
c. Black or African American
d. Hispanic or Latino
e. Native Hawaiian or other Pacific Islander
f. White or Caucasian
5. What is the highest level of education that you have completed? [Interviewer: Do not read
this list].
a. Less than high school degree
b. High school degree or GED
c. Some college
d. Trade school degree
e. Associate’s or Bachelor’s degree, or
f. Post graduate degree?
6. Just to be sure we’re interviewing a broad range of students for this study, what is your
annual household income?
If the respondent indicates they prefer not to disclose this information, the interviewer
should say “That’s fine”, and move on to the next question.
7. What is current zip code?
8. And has said would be interested in participating in
the interview session which will take no more than one hour?
Eligible Student Participant Script
Thank you for taking the time to answer the screening questions. Your child is eligible to
participate in the research interviews.
In the session, we will be asking to respond to several survey questions.
NCES needs to make sure that students will understand what the survey is asking them to do.
Participation is entirely voluntary, and your child’s answers will only be used only for statistical
purposes and may not be disclosed, or used, in identifiable form for any other purpose except as
192
required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].
Interviewer note: this reference is for informational purposes only. If asked what law is being
referenced, please provide the information contained within this bracketed text.]
Your child’s input will help NCES improve the survey that will be given nationwide. To thank
, will receive a $25 gift card. In addition, you will receive a
$25 gift card for your time to take the to and from the interview site.
Again, the interview will last no more than one hour and will be held at the EurekaFacts
cognitive laboratory site in Rockville, Maryland. Do you think would like
to participate?
[If No] Thank you for taking the time to speak with us today about this important project.
[If Yes (and the student is under age 18)] And would you be willing to submit a signed form onsite, indicating your approval for to participate?
[If No] Thank you for taking the time to speak with us today about this important project.
[If Yes] Are you and your child available for the interview at