Pilot Test Report

National Center for Education Stataistics
School Climate Surveys
Pilot Test

Attachment 5
Cognitive Interviews and Usability Testing Report

1

American Institutes for Research

Contents
1. Introduction..................................................................................................................................................... 4
Cognitive interviews ............................................................................................................................... 4
Usability Testing ..................................................................................................................................... 6

1.2 Methodology .......................................................................................................7
Cognitive interviews ............................................................................................................................... 7
Usability testing ...................................................................................................................................... 8

1.3 Sample .................................................................................................................8
1.4 Data and Analytic approach ........................................................................................................................ 9
Data collection: Cognitive interviews ..................................................................................................... 9
Data analysis methods: Cognitive interviews ....................................................................................... 10
Data analysis methods: Usability testing............................................................................................... 10
2. Findings and Recommendations .................................................................................................................. 11
2.1 Cognitive Interviews ................................................................................................................................... 11
2.1.1 Student survey ............................................................................................................................ 11
Demographic/Background Items ......................................................................................................... 11
Domain: Engagement .......................................................................................................................... 11
Domain: Environment ......................................................................................................................... 12
Domain: Safety ..................................................................................................................................... 14
2.1.2 Instructional staff survey .............................................................................................................. 17
Demographic/Background Items ......................................................................................................... 17
Domain: Engagement .......................................................................................................................... 17
Domain: Environment ......................................................................................................................... 18
Domain: Safety ..................................................................................................................................... 19
2.1.3 Noninstructional staff survey ....................................................................................................... 21
Demographic/Background Items ......................................................................................................... 21

2

Domain: Engagement .......................................................................................................................... 21
Domain: Environment ......................................................................................................................... 21
Domain: Safety ..................................................................................................................................... 22
2.1.4 Parent survey ................................................................................................................................ 25
Demographic/Background Items ......................................................................................................... 25
Domain: Engagement .......................................................................................................................... 25
Domain: Environment ......................................................................................................................... 26
Domain: Safety ..................................................................................................................................... 27

2.2 Usability Testing...............................................................................................29
2.2.1 Survey entry interface testing ....................................................................................................... 29
2.2.2 Administrative functions testing .................................................................................................. 31
2.2.3 Other Issues .................................................................................................................................. 36
Appendix A - Item by Item Analysis ............................................................................................................... 37
Appendix B – Recruitment and Interview Protocols ..................................................................................... 38

3

1. Introduction
This report summarizes the findings and recommendations from cognitive and usability testing
conducted to support the development of the School Climate Surveys (SCLS) and survey
administration platform. The School Climate Surveys are a suite of survey instruments being
developed for schools, school districts, and states by the U.S. Department of Education’s
National Center for Education Statistics (NCES). Through the SCLS, schools nationwide will
have access to survey instruments and a survey platform that will allow for the collection and
reporting of school climate data across stakeholders at the local level. The surveys can be used to
produce school-, district- and state-level scores on various indicators of school climate from the
perspectives of students, teachers and staff, principals, and parents and guardians. The
development of the SCLS survey instruments started in 2013 with a review of the existing school
climate literature and survey items in the Position Paper on SCLS Content. Based on the paper,
the Technical Review Panel (TRP) meeting was held in early 2014 to recommend items to be
included in the SCLS. Then the SCLS Concept Design Paper contains the draft survey items was
created building on the foundation of the Position Paper on SCLS Content and recommendations
from the TRP. In the summer of 2014, cognitive lab testing, include cognitive interviews and
usability testing, was conducted on the draft items and the survey platform.
Cognitive and usability testing were conducted to evaluate the quality of the survey items being
developed and the data collection platform itself. Cognitive laboratory testing of items and scales
investigates the cognitive processes that respondents use to answer survey questions. The goal of
the cognitive testing for the National Center for Education Statistics (NCES) School Climate
Surveys (SCLS) is to ensure that all items included in the final surveys are easily understood by
respondents, with their interpretation consistently aligned with the concept being measured. The
usability testing of the SCLS data collection platform was designed to confirm that major
navigational features can be used as intended and that participants are able to progress
appropriately through the surveys and other functions of the survey system.
Both the cognitive interviews and usability tests were conducted one-on-one with individual
participants. The items tested in the cognitive interviews were developed for the SCLS and are
intended to be administered via an online survey platform developed by Sanametrix. The
usability testing was conducted to observe respondents’ experience with the survey entry
platform as well as the interaction of the primary survey administrators—school principals,
assistant principals, and other administrators who regularly collect data as part of their job
duties—with the administrative functions dashboard.
Cognitive interviews

The items tested in the cognitive interviews included three categories of items: (1) new items that
had not been fielded previously; (2) items that had been fielded, but for which validation
information was not available; and (3) items from validated scales that were revised based on the
TRP’s recommendation. In instances where many items were changed in the same way (e.g.,
4

reversing the order of “agree-disagree” response options), we tested a representative subsample
of the items in order to minimize the level of respondent burden. The protocols for students and
parents were also translated into Spanish, and cognitive interviews were conducted in Spanish by
trained bilingual (Spanish/English) interviewers. Participants were recruited by the target of each
of the four questionnaires: middle and high school students, teachers and instructional staff,
principals and noninstructional staff, and parents of middle and high school students, we
included at two fifth grade students in the student sample. Eligible participants included students
attending public schools, parents of children attending public schools, staff that work at public
schools, and teachers and other instructional staff at public schools.
Each cognitive interview was conducted using a structured protocol in a one-on-one interview
drawing on best practices and methods from cognitive science. These interviews were designed
to identify problems of ambiguity or misunderstanding in question wording and problems in the
translatability of items.1 The goal was to ensure that all items included in the final surveys are
easily interpreted by respondents and that these interpretations are consistently aligned with the
concepts being measured.
A total of 309 items underwent cognitive testing; included 70 items in the student survey, 40
items in the parent survey, 89 items in the instructional staff survey, and 108 items in the
noninstructional staff survey (including 20 items specifically for principals). The selected survey
items were tested with members of the appropriate target respondent group to determine their
understandability, the need for revision, and recommendations for retention in or exclusion from
the surveys. Due to the large number of items that needed cognitive testing, we split the items
into two groups (for adults) and three groups (for students) by domains or topical areas. We
interviewed at least five potential respondents for each item to explore their understanding of the
items and answer options.
Based on the results of the cognitive interviews, we determined that 270 items are acceptable as
tested, 35 items require revisions to improve comprehension, and at least 4 items need to be
dropped completely. Table 1 contains a breakdown of these totals by respondent group.

1

Translatability relates to the extent to which a concept is similarly conveyed and understood in all relevant
languages (in this case, English and Spanish). It takes into account not only the technical aspects of translation, but
cultural and linguistic variation that may affect interpretation by respondents.

5

Table 1. Total number of SCLS items tested and number of items by recommendation and
respondent group
Number of items

Type of respondent
Total
Students
Instructional staff
Noninstructional
staff
Parents

Total No change
309
270
70
50
89
83
108
42

104
33

Revision
needed
35
16
6

Drop
4
4
0

4
9

0
0

Note: Item wording or formatting changes for purposes of consistency across surveys are not included in the number
of items for “revision needed”.

Based on the issues found during the cognitive interviews, we have made recommendations for
each survey item’s inclusion, exclusion, or revision in the SCLS (see appendix A for the itemby-item analysis). In order to maintain consistency within and across surveys, any recommended
changes to items with parallel items in a different survey will be used across all affected items.
For example, if a wording change is recommended for a student item and the same item is asked
of instructional and noninstructional staff, the recommended change from the student item will
be applied to the other two respondent group items. A summary of the findings is in section 2.1
below. In addition to item by item recommendations we also include revisions that are not item
specific, such as general respondent instructions and the use of skip patterns.
Usability Testing

Parallel to the development of survey items is the development of an online survey platform that
allows integrated data collection and reporting. The success of the SCLS relies in part on the
easy use of the platform, as it is the intention that survey setup and administration, as well as data
collection, occur without the need of technical assistance. The platform allows users to interact
with the SCLS in two ways—responding to survey questions through the survey entry interface
as respondents; and setting up and monitoring the survey through the administrative dashboard
as administrators.
The usability testing, which took place after the development team had created basic working
interaction elements for users to test, explored the interaction that respondents had with the
prototype of the SCLS platform. The usability tests were conducted with target respondent group
members to understand their experiences with the platform, to identify difficulties, and to ensure
that the goal of ease of use was met.
The participants in the survey entry interface testing took an abbreviated version of the survey.
They were asked to complete tasks designed to simulate a respondent’s interaction with the
website from survey invitation to survey submission (e.g., logging into the survey entry
6

interface, toggling the questionnaire language from English to Spanish, and re-accessing the
survey after taking a break).
In addition, a group of school administrators was asked to test the administrative functions of the
platform, including survey setup, monitoring, and reporting functions. For this special group,
usability testing tasks included setting up a survey, monitoring an in-progress survey, generating
a summary report of survey results, exporting a survey report, and locating item-level response
frequencies. The administrative function testing participants were not provided with a user’s
guide and had to complete the tasks based on their assumptions of how the dashboard worked.
Users’ success or difficulty in completing assigned tasks was analyzed to determine which
information or control elements were missing or insufficient to allow successful completion of
the anticipated tasks. After each set of tasks was completed, respondents were asked to answer
questions on the ease of using the features or navigating through the platform. They were also
asked for any comments they had about the task they had just completed. All comments were
recorded and analyzed to help guide the development of the SCLS platform.

1.2 Methodology
Cognitive interviews

The interviews were conducted by interviewers who received training in the cognitive interview
technique by a senior researcher at AIR. The interviewers used a topic guide (or protocol)
developed by AIR to ensure that all objectives of the study were met (see appendix B for the
different versions of the questionnaire tested in the study). Researchers from AIR and NCES
observed to some of the interviews and provided guidance and feedback throughout the study.
During each interview, the study participant was welcomed by the interviewer and led to an
interview room. The participant was then told that the purpose of the interview was to help
answer questions about how people respond to School Climate Surveys items. All participants
were assured that their participation was voluntary and that their answers would be used only for
research purposes and not disclosed, or used, in identifiable form for any other purpose except as
required by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Interviewers
explained the think-aloud process (see below) and conducted a practice session with a sample
question. Participants were then given a paper copy of the questionnaire to fill out.
The interviews were conducted using “concurrent probing,” a technique in which interviewers
asked respondents to “think aloud” as they answered the survey questions. The cognitive
interviews also included a verbal probing component, conducted after completion of the thinkaloud portion for a given questionnaire. The verbal probes included a combination of preplanned, item-specific questions identified before the session as important and ad hoc questions
that the interviewer identified as important from observations during the interview, such as
clarifications or expansions on points raised by the participant. In addition, the interview
protocol contained generic prompts to be applied flexibly by the interviewer to facilitate and
7

encourage participants to verbalize their thoughts. For example: “I see you’re looking at the
answer. What are you thinking?” or “Are there any questions or words that seem confusing
here?” Interviewers took notes on what participants said, and the sessions were audio-recorded.
Usability testing

The usability testing was also conducted one-on-one with respondents, but instead of filling out a
paper questionnaire, participants sat in front of a computer. The participants in the survey entry
interface testing were told that we were more interested in their interaction with the interface
than in their responses to the survey items. The interviewer then presented an overview of the
survey administration process and asked the participant to complete certain tasks. Participants
could use either a concurrent think-aloud or a retrospective think-aloud process to let the
interviewer know what they were thinking while they worked on the tasks as well as about any
confusion they had. The interviewer did not provide assistance to participants or answer
questions directly related to the tasks during the think-aloud process. If a participant was unable
to move forward for a prolonged period of time or asked for help, the interviewer only responded
with a generic prompt, such as “Is there anything else you can do to help you move forward?” or
“What do you think should happen/be available here?” After the think-aloud activity was
completed, the interviewer conducted a verbal probing component on the ease of navigating
through the interface and participants’ experiences with and expectations of the interface’s
features. Interviewers used probing questions such as “Are any of the buttons/links confusing on
this page?”
For the administrative functions testing, participants were asked to complete survey setup,
monitoring, and reporting tasks. During this testing, the interviewer could provide assistance to
respondents since instructions on how to perform the task were not available. When assistance
was needed, the interviewer would have the respondent explain why the task was initially
confusing and what information could be provided to making performing the task easier.
In all usability testing interviews, interviewers took notes on what participants said, and the
sessions were audio-recorded.

1.3 Sample
A total of 110 interviews were conducted in three locations: the Washington, DC, metropolitan
area, California (including San Mateo, San Francisco, and Sacramento), and Houston, Texas. In
total, we conducted 78 cognitive interviews and 32 usability tests. We sought to recruit a diverse
set of respondents. Table 2 provides more detailed information on the characteristics of the
respondents.

8

Table 2. Number of participants in SCLS cognitive interviews and usability testing, by
selected characteristics: 2014
Cognitive interviews
Usability testing
Total
78
32
Respondent type
Student
25
9
Parent
21
8
Instructional
16
3
Principal
5
9
Noninstructional
11
3
Gender
Female
55
20
Male
23
12
1
Race/ethnicity
White
22
12
Black
20
9
Hispanic/Latino
32
7
Asian or other Pacific Islander
3
1
Two or more races
1
2
Location
DC
51
25
CA
17
7
TX
10
0
Language interview was conducted
English
53
26
Spanish
25
6
2
School type
Public
63
20
Charter
8
5
Private
7
3
1
One respondent did not provide information on race/ethnicity and is not reported here.
2
School type is missing for four respondents.

1.4 Data and Analytic approach
Data collection: Cognitive interviews

Cognitive interview respondents were recruited through local recruitment firms, Craigslist posts,
and newspapers ads. Respondents were asked to complete a paper version of the questionnaire;
while doing this, they were asked to “think aloud” their responses to the questions. While the
respondent filled out the questionnaire, the interviewer took handwritten notes in an interview
protocol that contains the items and probes for the cognitive interview. . The questionnaire was
9

kept by the interviewer as a record of the respondent’s responses to the questions. Once the
interview was completed, the interviewer was required to enter notes in an electronic note
template. This template is a standardized format for qualitative data entries, which allowed the
team to process and analyze the data in a data processing software system. The interview notes
and an audio recording of the interview were then saved in a secure server for analysis.
Data analysis methods: Cognitive interviews

Once the interview notes were completed by an interviewer, a senior researcher would check the
data for errors and accuracy. The interview notes were also checked to ensure that the most
accurate information was included about the errors, issues, and concerns that respondents had
about the items in the questionnaire. After the interview notes had been checked for quality, each
item was coded into the following themes:
a. No problems
b. Respondent refused to answer the question
c. Response error
i.
Translatability issues
ii.
Respondent was unable to find a response that matched his or her experience
iii. Respondent had trouble understanding the question in general
iv.
Respondent had trouble understanding a phrase or word in the question
These themes allowed the analysis team to determine the types of problems that each item has,
by the type of respondent, and by the language the interview was conducted in. Once we were
able to identify item comprehension themes, we created subthemes to analyze how respondents
interpret concepts in an item or set of items. Once we were able to determine the item-by-item
issues, we were able to do a higher level analysis of domain-level problems or item clusters. Our
recommendations are based on this multi-stage analysis.
Data analysis methods: Usability testing

Usability test notes were reviewed to identify problems encountered by respondents while they
interacted with the survey platform in order to complete the tasks given to them by the
interviewer. Recommendations for changes to the platform were formulated based on the
problems that the respondents encountered.

10

2. Findings and Recommendations
2.1 Cognitive Interviews
2.1.1 Student survey

Demographic/Background Items
Three demographic items were tested in the student cognitive interviews. Of the 25 respondents
who answered these items, none had difficulty understanding them or finding a response that
matched their experience.
Domain: Engagement
A total of 18 items were tested in the engagement domain. Of these 18 items, five were in the
“cultural and linguistic competence” topical area, seven were in the “participation” topical area,
and six were in the “relationship” topical area.
Cultural and Linguistic Competence:
Analysis of the data collected from the cognitive interviews shows that two of the five cultural
and linguistic competence items were problematic. For these items, respondents had difficulty
understanding key vocabulary within the question or had difficulty understanding the question
overall. For example, one item asks students whether “teachers and other adults treat all students
respectfully.” In several cases, students had difficulty understanding what exactly the question
was asking, and did not understand who was treating whom respectfully. In order to provide
clarification, it is recommended that the wording be changed to “adults working at this school
treat all students respectfully” (Appendix A, page A.1.3).
The other item asks “people of different cultures, races, or ethnicities do not get along.” For this
item, respondents had trouble answering a negatively worded item on the strongly agree to
strongly disagree scale. It is recommended that the wording be changed to “people of different
cultural backgrounds, races, or ethnicities get along well at this school” (Appendix A, page
A.1.6).
Additionally, cognitive interviews revealed that one item in the cultural and linguistic
competence topical area had translatability issues. The item asks respondents about content they
encounter during class lessons, translated to “las lecciones de clase” for the cognitive interviews.
This formal Spanish translation caused confusion, and as a result, it is recommended that this
phrase just be translated to just “las clases.”
Participation:
Of the seven items in the participation topical area, five items presented no comprehension
problems during the cognitive interviews, and respondents found them easy to answer.

11

The remaining two items included a word or phrase that was difficult to understand in the
context of the question. Both items dealt with the issue of student opinion and input in school
decisions, and each included a word or phrase that respondents found ambiguous and, therefore,
caused confusion. For example, one question asks whether “students can help make decisions at
this school.” Many respondents were unclear as to what “decisions” this question is asking about
and did not understand the scope of the decisions that they are or are not making. The same issue
came up in another question, “students are given the chance to help solve school problems,”
where it is unclear what “problems” the question asks about. We recommend that these items be
removed from the survey (Appendix A, pages A.1.8 and A.1.9).
Relationship:
The relationship topical area contains six questions. Three of these items presented no problems
during the cognitive interviews. Respondents found these items easy to answer, and there was
little or no evidence of difficulty or a lack of understanding of concepts.
Respondents found specific vocabulary in two items to be confusing, leaving students unclear as
to the question’s meaning. For example, one question asks whether “it is easy to talk with
teachers.” Several respondents had difficulty understanding the term “teacher,” and interpreted
the term broadly include anyone an individual may learn from. In this question, respondents
interpreted “teacher” to include parents and other adults in their lives, in addition to school
instructional staff. A similar issue was found for the question “If I am absent, there is a teacher or
some other adult at school who will miss me,” which asks if a teacher or adult will “miss” a
student if he or she is absent. This brought about confusion, as many respondents interpreted the
question as asking about the emotional impact their absence might have on school staff, rather
than staff simply noticing their absence. For both items, it is recommended that the wording be
changed to account for these issues. It is recommended that the wording for these items be
changed to “it is easy to talk with teachers at this school,” and “if I am absent, there is a teacher
or some other adult at school that will notice my absence” respectively (Appendix A, pages
A.1.12 and A.1.16).
In the last relationship question, respondents had difficulty understanding a key phrase within the
question. This question asks students whether “adults near where I live support this school.” The
phrase “near where I live” was interpreted in several different ways, spanning from a narrow
interpretation of the people the respondent lives with at home, to a broad interpretation asking
about the quality of the school overall. It is recommended that this item be removed from the
survey (Appendix A, page A.1.14).
Domain: Environment
A total of 23 items were tested in the environment domain. Of these 23 items, five were in the
“instructional environment” topical area, three were in the “mental health” topical area, seven

12

were in the “physical environment” topical area, one was in the “discipline” topical area, and
seven were in the “physical health” topical area.
Instructional Environment:
Of the five items in the instructional environment topical area, three items showed no issues
during the cognitive interviews. Respondents found these items easy to answer, and we found
minimal confusion in respondents’ understanding of them.
The wording of one item caused confusion among respondents. The item asks “the schoolwork I
am assigned is meaningful and important.” Respondents were unclear about what “meaningful”
schoolwork meant, and interpreted the question differently. It is recommended that the item
wording be changed to “my schoolwork is important to my future” (Appendix A, page A.1.20).
Another instructional environment item caused confusion among respondents. The question asks
students “classes at this school really make me think.” Respondents were unclear what exactly
“make[s] me think” meant. While some thought this meant concentration or focus during class,
others thought the question meant the class made them think about their future. In order to
address the issues of both rigor and engagement, it is recommended that the item be revised to
“classes at this school are challenging,” and that another item “I’m really learning a lot in my
classes” be added (Appendix A, page A.1.21).
Mental Health:
Respondents had no difficulties with the three items tested in the mental health topical area.
Respondents found these items easy to answer.
Physical Environment:
Of the seven items in the physical environment topical area, six items presented no difficulties
during the cognitive interviews. Respondents found these items easy to answer, with few
comprehension problems.
The wording of one item caused confusion among respondents. This question asks students if
“this school has a bright and pleasant appearance.” Respondents found the phrase “bright and
pleasant” to be confusing, and one respondent interpreted the question as asking about the
appearance of students and others within the school. It is recommended that this item be revised
to “this school looks nice and pleasant” (Appendix A, page A.1.28).
Discipline:
Respondents had comprehension issues with the one item that was tested in the discipline topical
area. The question asks “discipline problems are handled fairly at this school.” Many
respondents had trouble understanding the question, and specifically interpreted the word
“discipline” in several different ways. While some respondents understood the word to mean
13

showing respect, others interpreted discipline to mean the reaction time of teachers and other
staff in the school to intervene during a conflict or fight. In order to address issues of both equity
and fairness of punishment, it is recommended that the item be separated into two items; one
question asks “school rules are applied equally,” and the other question asks “the punishment for
breaking school rules is fair” (Appendix A, page A.1.33).
Physical Health:
Of the seven items in the physical health topical area, six items presented no difficulties during
the cognitive interviews. Respondents found these items easy to answer, with few
comprehension problems.
Respondents had comprehension issues with the remaining physical health item. The question
asks “how often do you go to gym class or participate in other physical activity during the school
day?” Two respondents were not sure what “physical activity” meant in context of the question,
and one of them did not consider sports to be a physical activity. It is recommended that the
wording be changed to “how often do you go to gym class or participate in other physical
activity during the school day (e.g., running, playing sports)” (Appendix A, page A.1.52).
Domain: Safety
A total of 26 items were tested in the safety domain. Of these 26 items, 10 were in the
“emotional health” topical area, three were in the “emergency readiness and management”
topical area, seven were in the “substance abuse” topical area, and six were in the “bullying”
topical area.
Emotional Health:
No issues were found in the 10 items tested in the emotional health topical area. Respondents
found these items easy to answer, and there were no comprehension problems.
Emergency Readiness and Management:
Of the three items in the emergency readiness and management topical area, two item showed no
issues during the cognitive interviews. Respondents found these items easy to answer and there
were no comprehension issues, but mentioned the two items are very similar.
Respondents found the remaining one item difficult to understand. The item asks “if students
hear about a threat to school or student safety, they have a way to report it to someone without
giving their name.” Some respondents were unable to understand what the question was asking,
specifically who was hypothetically reporting on whom. In order to clarify the question, it is
recommended that the item be split into two separate items - the first one “if students hear about
a threat to school or student safety, they have a way to report it to someone in authority,” and the
second one “students can report a threat to school or student safety without giving their names”
(Appendix A, page A.1.36).
14

Substance Abuse:
One of the seven items in the substance abuse topical area presented no difficulties during the
cognitive interviews. Respondents found this item easy to answer, and there were no
comprehension problems.
Respondents found three items confusing due to a combination of unclear wording and varied
interpretation. One item asks “Based on your experience, how much of a problem at this school
is student drug use?” The other questions are “Based on your experience, how much of a
problem at this school is student use of tobacco (cigarettes, chew, cigars),” and “Based on your
experience, how much of a problem at this school is student alcohol use.” All three of these
items ask respondents about their perception of how much a problem these substances are “based
on your experience.” Many respondents interpreted this as asking them about their personal use
of tobacco and alcohol. We recommend changing the wording of these items to the following:
“At this school, how much of a problem is student drug use (e.g., Oxy, Ritalin, marijuana,
ecstasy),” “At this school, how much of a problem is student use of tobacco (e.g., cigarettes,
chew, cigars, electronic cigarettes),” and “At this school, how much of a problem is student
alcohol use (e.g., beer, wine, liquor)” (Appendix A, pages A.1.37, A.1.38, and A.1.39).
Respondents found the rest of the substance abuse items difficult due to specific wording. One
question asks “students think it is okay to have one or two drinks of any alcoholic beverage
nearly every day.” The item asks about alcohol consumption over or within a specific timeframe,
and the wording used to indicate frequency and time caused confusion, with some interpreting
the questions as asking about scheduled or periodic use. The same issue caused confusion for
two other items, asking “students at this school think it is okay to try drugs once or twice,” and
“students at this school think it is okay to use drugs once a month or more.” It is recommended
that the last item be removed from the survey, and that the other two items be changed to
“students at this school think it is okay to get drunk” and “students at this school think it is okay
to try drugs” (Appendix A, pages A.1.41, A.1.42, and A.1.43). We will also confirm with TRP
that it is not essential to ask about the frequency of usage.
Bullying:
Of the six items tested in the bullying topical area, five items showed no issues during cognitive
interviews. Respondents found these items easy to answer, and there were no comprehension
problems.
Respondents had comprehension problems with one of the items in the bullying topical area. The
question asks “students at this school are teased or picked on about their sexuality.” Respondents
had trouble understanding the concept of sexuality, and interpreted the word as referring to
gender and sex. In order to clarify, it is recommended that the item be split into two separate
ones -“students at this school are teased or picked on about their dating or sexual behavior,” and
the other question asks “students at this school are teased or picked on about their sexual
15

orientation” (Appendix A, page A.1.57). Additionally, in the Spanish translation of the bullying
items it was revealed that the most appropriate word to use is the English word “bullying”
instead of the formal Spanish translation “acoso.” As a result, it is recommended that the word
“bullying” be added in parentheses to “acoso” in the Spanish version of the questionnaire.

16

2.1.2 Instructional staff survey

Demographic/Background Items
A total of four demographic items were tested in the cognitive interviews. Of these four items,
respondents found three of these items easy to understand, and had not issues answering.
Respondents found one background item difficult to answer. The question asks “Is your main
assignment/responsibility at this school to provide instruction or other support services to any of
these types of students - Special Education, English Language Learners, Gifted and Talented
Education students, or Migrant Education?” Respondents had trouble answering the question,
because while they did provide instruction or support services to these students, it was not
necessarily their primary responsibility. It is recommended that the formatting of the item be
changed, such that “main assignment/responsibility” in the question is displayed in bold to
respondents (Appendix A, page A.2.3).
Domain: Engagement
A total of 21 items were tested in the engagement domain. Of these 21 items, eight were in the
“cultural and linguistic competence” topical area, nine were in the “participation” topical area,
and four were in the “relationship” topical area.
Cultural and Linguistic Competence:
Six of the eight items in the cultural and linguistic competence topical area presented no
difficulties to respondents during the cognitive interviews. Respondents found these items easy
to answer.
Respondents had comprehension problems with the remaining two items. One item asks “at this
school, closing the racial/ethnic achievement gap is considered a high priority.” Some
respondents interpreted the question as asking about a diverse student body, while others were
unclear to what “achievement gap” refers. A similar issue occurred with the other item, which
asks “this school provides effective supports for teaching culturally and linguistically diverse
students with IEPs.” Some respondents did not understand what this phrase means, causing
confusion over how to respond. For both items, it is recommended that the wording be changed
to clarify the meaning of these concepts. The first item will be revised to “at this school, closing
the racial/ethnic academic achievement gap is considered a high priority,” and the second item
will be revised to “this school provides effective supports (e.g., student assessment and
evaluation, collaboration among teachers and specialists, staff training) for teaching culturally
and linguistically diverse students with Individualized Education Programs (IEPs)” (Appendix
A, pages A.2.5 and A.2.11).

17

Participation:
Respondents found all nine items in the participation topical area easy to answer. We
recommend keeping these items as currently designed.
Relationship:
Respondents found all four items in the relationship topical area easy to answer. We recommend
keeping these items as currently designed.
Domain: Environment
A total of 29 items were tested in the environment domain. Of these 29 items, three were in the
“discipline” topical area, six were in the “instructional environment” topical area, six were in the
“mental health” topical area, seven were in the “physical environment” topical area, and seven
were in the “physical health” topical area.
Discipline:
Respondents found all three items in the discipline topical area easy to answer. We recommend
keeping these items as currently designed.
Instructional Environment:
Respondents found all five of the six items in the instructional environment topical area easy to
answer. We recommend keeping these items as currently designed.
One item caused comprehension issues among respondents. The item says “we have so many
different programs at this school that I can’t keep track of them all.” Respondents found the
question wording vague, and were unable to determine to what programs the question refers. It is
recommended that the wording be changed to “the programs and resources at this school are
adequate to support students’ learning” (Appendix A, page A.2.32).
Mental Health:
Respondents found all six items in the mental health topical area easy to answer. We recommend
keeping these items as currently designed.
Physical Environment:
Of the seven items in the physical environment topical area, no items showed issues during the
cognitive interviews. Respondents found these items easy to understand and to answer.
Physical Health:
Six of the seven items in the physical health topical area presented no difficulties to respondents
during the cognitive testing. Respondents found these items easy to answer.
18

Respondents did have comprehension issues with one item due to what appears to be ambiguity
in the meaning of a phrase within the question. The item asks “this school provides quality
physical health and nutrition services for students.” Some respondents were not sure what these
services would include and, therefore, had difficulty answering the question. It is recommended
that the item wording be changed to “this school provides quality physical health and nutrition
services (e.g., school nurse, free and reduced price meals, before and after school meals) for
students” (Appendix A, page A.2.49).
Domain: Safety
A total of 35 items were tested in the safety domain. Of these 35 items, 12 were in the “bullying”
topical area, 10 were in the “emotional health” topical area, four were in the “emergency
readiness and management” topical area, and nine were in the “substance abuse” topical area.
Bullying:
The 12 items in the bullying topical area presented no problems to respondents during the
cognitive interviews. Respondents found these items easy to understand and answer.
Emotional Health:
The 10 items in the emotional health topical area presented no problems to respondents during
the cognitive interviews. Respondents found these items easy to understand and answer.
Emergency Readiness and Management:
Of the four items in the emergency readiness and management topical area, three presented no
difficulties to respondents during the cognitive interviews. Respondents found these items easy
to understand and answer.
One item caused comprehension issues among respondents. The question asks “this school or
school district provides effective training in safety procedures to staff (e.g., how to handle
emergencies).” Some respondents did not understand to what “safety procedures” the question
refers. It is recommended that the item wording be changed to “this school or school district
provides effective training in safety procedures (e.g., how to handle emergencies, lockdown
training, fire drills) to staff” (Appendix A, page A.2.80).
Substance Abuse:
Respondents had no difficulty with seven of the nine items in the substance abuse topical area.
Respondents found these items easy to answer.
The remaining three questions caused some confusion due to the fact that respondents were
unable to find a response that matched their experience. For example, one question asks whether
“this school provides effective confidential support and referral services for students needing
help because of substance abuse.” Several respondents mentioned that they are not familiar with
19

the services provided by their school and thought the question should include a “don’t know”
response. The same issue came up in two other questions; one asks whether “this school has
programs that address substance use among students”; the other asks whether “at this school,
first-time violations of alcohol or other drug policies are punished by at least an out-of-school
suspension.” Respondents did not know if these programs or policies were provided by their
school and could not find a response that matched their experience. While these issues were
found through the cognitive interviewing process, it is recommended that these items not be
changed, as they are not required items and respondents are able to skip the questions if they
wish. Additionally, as it is only worthwhile to include “don’t know” responses in surveys if they
are analytically useful and provide actionable information, we will not include a “don’t know”
option.2 We will also evaluate the item missing using pilot data.

2

For more detailed information regarding the inclusion rule for a “don’t know” option, please see page 17 of the
concept design paper.

20

2.1.3 Noninstructional staff survey

Demographic/Background Items
A total of four demographic items were tested in the cognitive interviews. Of the 15 respondents
who answered these items, none had any issues understanding them or finding responses that
matched their experience.
Domain: Engagement
A total of 33 items were tested in the engagement domain. Of these 33 items, eight were in the
“cultural and linguistic competence” topical area, 14 were in the “relationship” topical area, and
11 were in the “participation” topical area.
Cultural and Linguistic Competence:
Of the eight items given to noninstructional staff in the cultural and linguistic competence topical
area, seven items showed no problems during the cognitive interviews. Respondents found these
items easy to answer, and there were minimal comprehension problems.
One item caused confusion among respondents due to varying interpretations of a phrase within
the question. The item asks whether “this school provides instructional materials that reflect
students’ culture, ethnicity and identity.” There were varying interpretations of “instructional
materials”—with some respondents thinking it referred to just curriculum and standards and
others including textbooks, handouts, and media—leading to differences in responses. As a
result, we recommend that the question be changed to “this school provides instructional
materials (e.g., textbooks, handouts, software applications) that reflect students’ cultural
background, ethnicity and identity” (Appendix A, page A.3.8).
Relationship:
Of the 14 items in the relationship topical area, nine were given to all noninstructional staff and
five were given only to principals. None of the 14 items presented any problems during the
cognitive interviews. Respondents found these items easy to understand and to answer.
Participation:
Of the 11 items in the participation topical area, nine were items given to all noninstructional
staff, and two items were given to principals only. None of the 11 items presented any problems
during the cognitive interviews. Respondents found these items easy to understand and to
answer.
Domain: Environment
A total of 39 items were tested in the environment domain. Of these 39 items, 10 were in the
“physical environment” topical area, nine were in the “instructional environment” topical area,

21

seven were in the “physical health” topical area, seven were in the “mental health” topical area,
and six were in the “discipline” topical area.
Physical Environment:
Of the 10 items in the physical environment topical area, eight were items given to all
noninstructional staff, and two items were given to principals only. All 10 items had no issues
during the cognitive interviews. Respondents found these items easy to understand and to
answer.
Instructional Environment:
Of the nine items in the instructional environment topical area, seven were items given to all
noninstructional staff, and two items were given to principals only. None of the nine items
presented any problems during the cognitive interviews. Respondents found these items easy to
understand and to answer.
Physical Health:
Of the seven items in the physical health topical area, five were items given to all
noninstructional staff, and two items were given to principals only. All seven items showed no
issues during the cognitive interviews. Respondents found these items easy to answer and no
comprehension issues were found during cognitive testing among noninstructional staff.
Mental Health:
Of the seven items in the physical health topical area, five were given to all noninstructional staff
and two were given only to principals. None of the seven items presented any problems during
the cognitive interviews. Respondents found these items easy to understand and to answer.
Discipline:
Of the six items in the discipline topical area, four were items given to all noninstructional staff,
and two items were given to principals only. None of the six items presented any problems
during the cognitive interviews. Respondents found these items easy to understand and to
answer.
Domain: Safety
A total of 32 items were tested in the safety domain. Of these 32 items, 10 were in the
“emotional health” topical area, six were in the “bullying” topical area, nine were in the
“substance abuse” topical area, and seven were in the “emergency readiness and management”
topical area.

22

Emotional Health:
None of the 10 items given to noninstructional staff in the emotional health topical area
presented any problems during the cognitive interviews. Respondents found these items easy to
understand and to answer.
Bullying:
None of the six items given to noninstructional staff in the bullying topical area presented any
problems during the cognitive interviews. Respondents found these items easy to understand and
to answer.
Substance Abuse:
Of the nine items in the substance abuse topical area, seven of the items showed no issues during
the cognitive interviews. Respondents found these items easy to understand and to answer.
One item was problematic because many respondents interpreted the question differently than
intended. The item asks if “substance abuse prevention is an important goal at this school.”
While some respondents understood the question to be about if the school provides substance
abuse prevention services, others interpreted it to be about if the school provides information
about such services, or believed the question asked about health classes and counselors focus on
substance abuse. Due to the variety of interpretations, it is recommended that the wording be
changed to “this school has programs, resources, and/or policies to prevent substance abuse”
(Appendix A, page A.3.51).
Respondents had comprehension issues with another item in the substance abuse topical area.
The question asks “at this school, first-time violations of alcohol or other drug policies are
punished by at least an out-of-school suspension.” Some respondents interpreted the question as
asking about cigarette smoking on school grounds, while another respondent understood the
question to ask about drug use generally at the school. It is recommended that the formatting of
the item be changed, such that “alcohol or other drug policies” in the question is displayed in
bold to respondents (Appendix A, page A.3.54).
Emergency Readiness and Management:
Of the seven items in the emergency readiness and management topical area, four were given to
all noninstructional staff and three were given only to principals. All four of the noninstructional
items and two of the principal-only items presented no difficulties during the cognitive
interviews. Respondents found these items easy to understand and to answer.
One item targeted to principals caused confusion over specific wording within the question. The
item asks if the school “has agreements with various agencies and services to aid timely
communications.” One respondent was unclear what “timely” meant in the question, and other
23

respondents had trouble thinking of or identifying the “agencies and services” to which the
question refers. In order to provide clarity to respondents, it is recommended that the wording be
changed to “This school has agreements with various agencies and services (e.g., local police
and fire departments, community organizations) to aid timely communication” (Appendix A,
page A.3.98). We will also consult TRP as to whether “timely” is an appropriate word in this
context.

24

2.1.4 Parent survey

Demographic/Background Items
A total of two demographic items were tested in the cognitive interviews. Of the 21 respondents
who answered the demographic items, none had any difficulty answering them.
Domain: Engagement
A total of 10 items were tested in the engagement domain. Of these 10 items, seven were in the
“relationship” topical area, and three were in the “cultural and linguistic competence” topical
area.
For the engagement domain, six of the 10 questions in this domain presented no problems during
the cognitive interviews. Respondents found these items easy to answer.
Respondents had comprehension issues with one item in the relationship topical area. The item
asks whether “this school helps me figure out what social and emotional skills my child needs to
develop.” Some respondents did not understand the term “social and emotional skills,” while
others were not sure whether the question referred to direct or indirect help from the school.
Another respondent did not know how to respond, as the situation had never come up in their
experience at their child’s school. It is recommended that the wording be changed to “this school
helps me figure out what social and emotional skills (i.e., knowledge, attitudes, and skills
necessary to deal with social and emotional interactions) my child needs to develop” (Appendix
A, page A.4.36).
Respondents also had comprehension issues with three items in the cultural and linguistic
competence topical area. One question asks “this school communicates how important it is to
respect the beliefs and practices of all cultures.” Respondents interpreted this item in several
ways, with some believing “all cultures” to strictly refer to religious practices, while other
respondents focused on the school’s goal of acceptance and respect of diversity rather than how
well the school communicates the importance of that respect. Another question that created
comprehension issues asks “this school provides instructional materials that reflect my child’s
culture, ethnicity, and identity.” While most respondents understood what “culture,” “ethnicity,”
and “identity” mean separately, they had trouble understanding what the question was asking
overall. It is recommended that the wording for these two items be changed to “this school
communicates how important it is to respect the practices of all cultures,” and “This school
provides instructional materials (e.g., textbooks, handouts, software applications) that reflect
students’ cultural background, ethnicity, and identity” (Appendix A, pages A.4.38 and A.4.39).
The last item that had comprehension issues is “this school encourages all students to take
challenging courses no matter their race, ethnicity, nationality, or culture.” Respondents had
trouble understanding what would classify as a “challenging course,” and some respondents did
not know enough about the school to answer. It is recommended that the wording be changed to
“this school encourages students to take challenging courses (e.g., honor level courses, AP or IB
25

courses, and dual credit courses) no matter their race, ethnicity, nationality, and/or cultural
background” (Appendix A, page A.4.37).
Domain: Environment
A total of 14 items were tested in the environment domain. Of these 14 items, two were in the
“physical health” topical area, two were in the “physical environment” topical area, two were in
the “mental health” topical area, five were in the “instructional environment” topical area, and
three were in the “discipline” topical area.
10 of the 14 questions in the environment domain presented no difficulties to respondents in the
cognitive interviews. Respondents found these items easy to understand and to answer.
Respondents had comprehension issues with one item in the physical health topical area. The
item asks whether “this school provides quality physical health and nutrition services for
students.” Several respondents had trouble understanding what “physical health and nutrition
services” meant in the context of the question, and one respondent interpreted the question as
asking if schools should provide such services. It is recommended that the wording be changed
to “this school provides quality physical health and nutrition services (e.g., school nurse, free and
reduced price meals, before and after school meals) for students” (Appendix A, page A.4.16).
Respondents also had comprehension issues in one item in the mental health topical area. The
question asks whether “this school has enough programs that address students’ emotional and
social development.” One respondent had trouble understanding what exactly “development”
meant in the context of the question, while others had trouble understanding what “programs”
the question refers to. It is recommended that the wording be changed to “this school has enough
programs for students (e.g., mentoring programs, group problem solving, team building) that
address emotional and social development” (Appendix A, page A.4.20).
One item in the instructional environment topical area proved problematic during testing. The
item asks “this school has high expectations for student learning.” Several respondents had
differing interpretations of what “high expectations” meant in context of student learning; one
respondent took the item to mean that the school had high standards generally, while another
respondent thought it meant specifically that the school assigned homework. It is recommended
that the wording be changed to “this school has high expectations for students” (Appendix A,
page A.4.25).
Respondents had trouble answering one last question for the environment domain, in the
discipline topical area. The question asks “discipline problems are handled fairly at this school.”
One respondent was not sure how to answer the question, as she had never dealt with this issue at
her child’s school. Another respondent was not sure if the question was asking about discipline
from the perspective of one student or a group of students. In order to address issues of both
equity and fairness of punishment, it is recommended that the item be separated into two items;

26

one question asks “school rules are applied equally,” and the other question asks “the
punishment for breaking school rules is fair” (Appendix A, page A.4.27).
Domain: Safety
A total of 16 items were tested in the safety domain. Of these 16 items, three were in the
“substance abuse” topical area, four were in the “physical safety” topical area, three were in the
“emergency readiness and management” topical area, three were in the “emotional health”
topical area, and three were in the “bullying” topical area.
13 of the 16 questions in the safety domain presented no difficulties to respondents in the
cognitive interviews. Respondents found these items easy to answer.
Respondents had issues answering one item in the substance abuse topical area. The question
asks, “Based on your experience, how much of a problem at this school is student drug use?”
The responses varied depending on how the particular respondent interpreted the question; one
respondent understood the question to ask about drug problems generally in society, while
another respondent understood the question to ask about their family’s drug problems. Several
respondents could not answer the question because they did not know one way or another if
drugs were a problem. While these issues were found through the cognitive interviewing process,
it is recommended that this item not be changed, as it is not a required item and respondents are
able to skip the question if they wish. Additionally, as it is only worthwhile to include “don’t
know” responses in surveys if they are analytically useful and provide actionable information,
we will not include a “don’t know” option. We will also evaluate the item missing using pilot
data. (Appendix A, page A.4.3).
One item in the physical safety topical area proved problematic during testing. The question asks
whether “physical fighting between students is a problem at this school.” Several respondents
were unable to answer the question because they were unsure whether “physical fighting”
referred only to gang-related fighting or whether it could also include bullying. While these
issues were found through the cognitive interviewing process, it is recommended that this item
not be changed, as it is not a required item and respondents are able to skip the question if they
wish. Additionally, as it is only worthwhile to include “don’t know” responses in surveys if they
are analytically useful and provide actionable information, we will not include a “don’t know”
option. We will also evaluate the item missing using pilot data. (Appendix A, page A.4.4).
Respondents had comprehension issues with one item in the bullying topical area. One question
asks whether “this school has helped me be more aware of bullying and cyberbullying of
students.” Respondents had difficulty understanding the meaning of “cyberbullying,” and one
respondent did not see a difference between cyberbullying and other forms of bullying. This
respondent appeared to have comprehension problems at the conceptual level. It is recommended
that the term cyberbullying be added to another item “bullying of students via electronic means
or devices is a problem at this school” to define cyberbullying. Therefore, the revised item
27

“bullying of students via electronic means or devices is a problem at this school (cyberbullying)”
will be placed before the other cyberbullying item “this school has helped me be more aware of
bullying and cyberbullying of students” (Appendix A, page A.4.14).
Across multiple items, respondents to the parent survey had comprehension issues with the term
“this school.” Some respondents interpreted this to mean all schools, while others did not
understand to what school the question(s) referred. As a result, it is recommended that a general
instruction be added to the beginning of the parent survey to clarify which school the survey will
specifically address.

28

2.2 Usability Testing
Usability testing was conducted to observe respondent interaction with the survey entry interface
and the school administrator interaction with administrative function dashboard. A total of 32
usability tests were conducted, including six survey entry interface usability tests with Spanish
speaking respondents and seven administrative functions usability tests with administrative staff.
2.2.1 Survey entry interface testing

As part of the usability testing of the survey entry interface, respondents were given a simulation
of the expected survey process from survey invitation to survey submission. All respondents
were given a series of tasks that guided them through the actions respondents would encounter
when taking the survey on the platform: accessing the survey site, logging into the survey
interface, and answering the survey items. Respondents were also asked to log out of the
interface before completing the survey. They were then asked to reenter the interface to complete
the survey and submit their responses. In order to give the respondents sufficient time to go
through the entire process, each survey was shortened to 20 items. The detailed tasks, findings
and recommendations are summarized below.
1. Accessing the survey from the invitation:
a. Task description: Parents and school staff (principals, instructional and
noninstructional) respondents began the usability test by reading an example of an email invitation from the survey administrator. The e-mail contained a description of
the survey, a link to the survey interface site, and a unique username. Since it is likely
that student survey administrations will take place in school computer labs, student
usability testing began with the survey interface home page already opened and the
students being given their username.
b. Findings: English speaking respondents did not have problems with this task. One
Spanish speaking parent had a problem with going to the website.
c. Recommendations: No changes needed.
2. Logging in to the survey platform:
a. Task description: Respondents were asked to log in to the survey interface with the
username provided. They were also asked if they could identify who to contact if they
encountered a problem. At the Informed Consent screen, respondents were asked how
they would exit the survey if they wanted to wait to begin the survey. The
interviewers observed whether the respondents wrote down the PIN number provided
as it would later be needed to reenter the survey.
b. Findings: Issues found
i. Respondents were asked how they would exit the survey at the Informed
Consent screen, (where they have the options of clicking on the “Yes, I wish
to participate” or the “No, I do not wish to continue” buttons or clicking the
Log Out link with the intention of returning to complete the survey at another
29

time). Most respondents indicated that they would click on “No, I do not wish
to continue” (which takes them to a screen to confirm that they want to opt out
of the survey).
c. Recommendations: Respondents should be given clear directions on how to exit the
survey at the Informed Consent screen if they want to return to take the survey at
another time. On the Informed Consent screen, change the text in the last paragraph
of the informed consent to:
i. If you understand this information and are willing to take this survey, please
select “Yes, I am ready to begin”. If you are not willing to take the survey,
please select “No I do not want to participate.”
ii. Change the Navigation buttons at the bottom accordingly.
3. Taking the survey:
a. Task description: Respondents were asked to answer survey items using the survey
interface. They were observed to see how they interacted with the navigation buttons.
Respondents were asked to use to the “back” and “clear” buttons if they did not use
them on their own.
b. Findings: Respondents did not encounter any significant issues with answering the
survey questions.
c. Recommendations: No changes needed.
4. Exiting the survey prior to completion (and survey reentry after break-off):
a. Task description: After answering approximately 10 items, respondents were asked to
exit the interface. They were then asked to then reenter the interface. Interviewers
observed whether respondents used their PIN to reenter the survey.
b. Findings: Issues found
i. While most respondents wrote down their PIN, they were not aware that if
they lost or did not record their PIN they would not be able to reenter the
survey. Most respondents believed that the survey administrator would be able
to resend their PIN or reopen the survey for them.
ii. Some respondents entered their PIN in the username field when returning to
the interface. When they got an error message that the PIN was invalid they
then correctly entered their username on the home screen and their PIN on the
next page.
iii. When reentering the survey, most respondent were not aware that they could
not go back to previous questions.
c. Recommendations: A message should be added to the PIN screen that informs
respondents that they will not be able to reenter the survey if they do not record their
PIN and that the PIN cannot be retrieved by the survey administrator. On the PIN
page, add the text below:
30

i. Please make note of the PIN below.
It will allow you to log back into the survey if the session times out or you
need to stop and come back later. The session will time out if left idle for
more that xx minutes.
PIN:XXXX
For your security, we will not be able to reset the PIN if lost or forgotten.

5. Completing and submitting the survey:
a. Task description: After reentering the survey interface, respondents were asked to
respond to the remaining items. Respondents were asked if they realized that they
could not return to previous questions after logging out. Upon reaching the end of the
survey, respondents were asked if they could tell that it had been submitted.
b. Findings: Issues found
i. Upon reaching the end of the survey, the respondents were unsure from the
closing message that their responses had been submitted. The closing message
tells them that they have completed the survey and they have the option of
going back to previous questions or logging out.
c. Recommendations: Additional text should be added to the closing screen that informs
the respondents that their responses have been submitted and that they can either exit
the survey or go back to review their answers up to the point they last logged in. on
the last page, replace the current text with:
i. Thank you! You have successfully submitted your survey. If you wish, you
may go back to review your responses up to the last point you logged in. If
you do not want to review your responses, you may log out the survey at this
time.
2.2.2 Administrative functions testing

In addition to the principals recruited for the survey interface usability testing, seven school
principals and assistant principals were recruited to test the administrative functions features of
the platform. The administrative functions platform was designed to be used with minimal
technical assistance, so one of the goals of the usability testing was to determine how intuitive it
was to perform the tasks and to identify what information needed to be provided to
administrators to perform the tasks. It was expected that usability testing would discover reveal
what aspects of the administrative function dashboard would need to be clarified for respondents;
these findings will be used in the creation of a user guide that will provide users with detailed
instructions on using the platform.

31

In practice, the use of all of the administrative functions for a given survey will occur on a period
of time. To field a survey, an administrator has to schedule the data collection period, generate
and distribute usernames to respondents, monitor the submission status while data collection
period is occurring, and create reports once the data collection period is over. In order to
maximize the time allotted for usability testing, the platform developer included data in the test
accounts that allowed the administrators to test all of the administrative functions.
1. Logging in to the platform:
a. Task description: Respondents were given a simulated e-mail from a district
administrator and asked to log into the platform. The e-mail included a link to the
platform, a username and a temporary password. Upon logging in respondents were
required to change their password.
b. Findings: Moderate issues
i. When respondents clicked on the new password field, a floating message field
appeared that contains the criteria needed for the new password. While most
respondents indicated that they saw the floating message field, they did not
read the criteria and created invalid new passwords. Several respondents took
two or more attempts to create a valid password.
c. Recommendations: The password criteria, which currently appear in a floating text
box on the dashboard, should be moved to the password change page. More special
characters should be added as acceptable characters.

2. Setting up the school information:
a. Task description: Upon logging into the platform after changing their password,
respondents were then required to enter his or her school contact information and
NCES ID. In our usability testing, a specific school name and its NCES ID were prefilled because the data used later in the testing for generating reports had been
prepared for the school. Instead, respondents in our testing were asked if they knew
the NCES ID for his or her school or if they would need to look it up. Respondents
were also asked if they noticed that a link was provided for them to access the NCES
website where they could look up the NCES ID for his or her school.
b. Findings: Respondents had no problems completing this task. All respondents
reported that they would have to look up the NCES ID required for the setup (and
they noticed that there was a link to a site to look up the ID number). Some
respondents suggested that the link to the ID look-up site should be labeled more
explicitly.
c. Recommendations: Provide a shorter label for the links to look up IDs.
o Find Your District ID (instead of "Find NCES School District ID")
o Find Your School ID (instead of "Find NCES School ID")
32

3. Setting up a data collection:
a. Setting survey dates and respondent groups:
i. Task description: Respondents were asked to set up two data collection
periods for their school. The set up required the respondents to enter in the
collection dates and the respondent group. After entering in this information
respondents needed to press the “add” icon3 to complete the setup.
ii. Findings: Moderate issues
1. The fields in which to enter the data collection set-up were located at
the bottom of the web page. Respondents first attempted to modify
existing data collections; they were then directed to the bottom of the
page where they saw the blank fields that needed to be modified.
2. Some respondents clicked on the plus sign icon for “Add data
collection” before entering the study criteria, which caused them to be
unable to select a respondent group.
iii. Recommendations: The fields used to enter information to set up a data
collection should be moved from the bottom to the top of the page.
b. Generating usernames:
i. Task description: Respondents were asked to generate a specific number of
user names for each respondent group.
ii. Findings: Minor issues
1. No issues were encountered in generating usernames.
2. Most respondents had minor issues in figuring out how to look at the
generated usernames. The function button 4needed to show usernames,
is labeled “export” that label was not intuitive to selected for the task
but since the other buttons in that function area were labeled
“Generate” and “Disseminate, respondents were able to select
“Export” after some deliberation.
iii. Recommendations: Change “Export” to “View.”
c. Disseminating usernames:
i. Task description: Respondents were asked to upload a prepared username file
into the platform in order to generate the automatic e-mails that would be sent
to respondents. Respondents were asked what information they would include

3

Icon refers to a small picture on a computer screen that represents a program or function. In the platform icon may
or may not be accompanied by test on the screen. In the instance when they appear along their stated function can be
seen when the cursor is placed over it.
4

Function button refers to large icons that include the name of the intended function.

33

in the message they would create, and what information they expected to be
automatically included in the email.
ii. Findings: Moderate issues encountered
1. Respondents were able to disseminate the usernames contained in the
file prepared for usability testing to the e-mails. However, we didn’t
test if respondents knew how to prepare the file if we didn’t provide it
for them.
2. When describing the information they would include in the email,
most respondent did not mention the link to the website. When asked
why, the respondents reported that they thought it would automatically
be included in the email.
3. Although the sample email text showing how the dynamic field
“[username]” should be embed was located above the email field, it
was not clear to most respondents that they needed to include the
dynamic field “[username]” in the message in order for the usernames
to be sent out to survey users. Some respondents didn’t include it,
thinking it would be automatically included, or put a static “username”
in the message.
iii. Recommendations: Use sample text with the required fields entered into the email text field. This instruction will be added on top of the e-mail field: Please
insert the link to your survey to the e-mail text. You can customize the text for
your data collection. However, please do not modify this line – “User name:
[username]”.
Sample email:
Hello,
You are invited to participate in the School Climate Survey. Results from
the survey will provide us with a basis for understanding and improving
the educational climate for the students at our school. Below you will find
a link to the survey site and your user name.
Survey site: {Please insert the link to your survey}
User name: [username]
Thank you in advance for your participation!

34

d. Modifying data collection:
i. Task description: Respondents were asked to modify the dates of one of the
data collection that they created and delete the other.
ii. Findings: Minor issues encountered
1. Respondents clicked on the icon labeled “edit” before they made
changes to the data collection setup. It was not clear that they needed
to make their changes first, and then click on the icon.
iii. Recommendations: Change
icons to
Text for the icon for
editing should say “Save changes” and for the icon for “adding a data
collection” fields it should say “Save data collection”.

4. Creating data collection reports:
a. Survey status reports:
i. Task description: Respondents were asked to generate a survey status report
using the platform.
ii. Findings: Minor issues encountered
1. Respondents were able to generate the reports without issues.
2. Respondent were asked to explain what they thought the numbers in
each column of the report referred to. Respondents reported unsure
about the meaning of the data because they didn’t understand column
headers.
3. When asked to generate a case disposition report and then filter to look
for the usernames with specific dispositions, some respondents did not
notice that there was a drop-down menu they could use. Other
respondents clicked on the column header thinking that it would be a
way to filter instead of sort.
4. It was not clear to respondents that they could only view survey status
reports for data collections currently in the field.
Survey Submission Report Column Headers (Wording change and
column reorder).
iii. Recommendations: Use the following column headers in this order:
1.
2.
3.
4.
5.

Total Usernames Generated
Number of Submitted Surveys
Number of Incomplete Surveys
Number of Unused Usernames
Survey Submission Rate

35

a. Survey results reports:
i. Task description: Respondents were asked to generate a survey results report
using the platform.
ii. Findings: Minor issues encountered
1. Respondents were able to generate the reports. It was not clear that
reports could only be generated for data collections that were already
closed, but respondents were able to correctly surmise the reason for
the limitation.
iii. Recommendations:
1. Add the following text to pages for the submission rate and case
disposition reports: To maintain respondent confidentiality these
reports can only be run on data collections that are currently open.
2. Add the following text to pages for the item frequency and scale
reports: To maintain respondent confidentiality these reports can only
be run on data collections that are closed.

3. Import and export of survey data:
a. Task description: Respondents were asked to import data from a file prepared for the
usability test. They were also asked to export data from a previously closed data
collection.
b. Findings: No issues encountered.
c. Recommendations: No changes needed.
2.2.3 Other Issues

An issue that arose from the usability testing was the potential problem of technology literacy
gap between respondents who are familiar with computer and related technologies and those who
are not. During our testing, one Spanish speaking parent who had little experience using the
computer initially did not understand that she needed to click on the link in the e-mail message to
open the survey website. However, the participant was able to answer the survey items once
inside the platform. Special efforts may be needed from either schools or districts to ensure that
the type of respondents will be able to reach the survey website.

36

Appendix A - Item by Item Analysis

37

Appendix B – Recruitment and Interview Protocols

38

Appendix A - Item by Item Analysis
Appendix A.1 Student survey - Item by item recommendations

Question 1: Sdemo149 Are you of Hispanic or Latino origin?
 Yes
 No
Probe: There is no probe for this item.
Summary of the responses:
25 students received this question; none had any difficulty answering it or finding a response that
matched their experience.
Recommendations:
No change.

A.1.1

Question 2: Sdemo151 What grade are you currently in at this school?
 5th Grade
 6th Grade
 7th Grade
 8th Grade
 9th Grade
 10th Grade
 11th Grade
 12th Grade
Probe: There is no probe for this item.
Summary of the responses:
25 students received this question; none had any difficulty answering it or finding a response that
matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, to account for
students that are not graded, the following wording change will be made to the response options:
5th grade, 6th grade, 7th grade, 8th grade, 9th grade, 10th grade, 11th grade, 12th grade,
Ungraded

A.1.2

Question 3: Sengclc4 Teachers and other adults at this school treat all students respectfully.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




Can you tell me what this question is asking you?
What does it mean to be treated respectfully?
What does “other adults” mean to you in this question?

Summary of the responses:
14 students received this question during cognitive testing. Three students had trouble
understanding the question, specifically the phrase “treat all students respectfully.” One of the
three also misread the question as “whether teachers are disrespectful to students” and another
treated the question as “if teachers/other adults treats kids with the same respect as they treat
other adults.”
Recommendations:
Wording change: Adults working at this school treat all students respectfully.

A.1.3

Question 4: Sengclc5 There are examples of different racial, ethnic, or cultural backgrounds in
the class lessons at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “class lesson” mean in your own words, and how did you come up with that
answer?
Can you give me an example of the type of thing this question is asking about?

Summary of the responses:
14 students received this question. Four students were unsure about the intent of the question,
two of them specifically had difficulty with the phrase “class lessons.” Another respondent
thought the question was asking if “there are racist comments or activities in the class.” One
student was unable to find a response that matched experience.
Recommendations:
No change.

A.1.4

Question 5: Sengclc6 A teacher or other adult at this school has disrespected students because of
their race, ethnicity, or culture.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me what this question is asking you.
What do you think it means to be disrespected because of race, ethnicity, or culture?

Summary of the responses:
14 students received this question; none had any difficulty answering it or finding a response that
matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
Adults working at this school has disrespected students because of their race, ethnicity, or
cultural background.

A.1.5

Question 6: Sengclc7 People of different cultures, races, or ethnicities do not get along well at
this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What are these questions asking?
What does the term “different cultures” mean? OR when you hear “different cultures”
what do you think about?

Summary of the responses:
14 students received this question. Two students had trouble with the double negative formed by
the stem and the third and fourth answer options.
Recommendations:
Wording change: People of different cultural backgrounds, races, or ethnicities get along well at
this school.

A.1.6

Question 7: Sengclc8 Students from different backgrounds get along well at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What are these questions asking?
What does the term “different cultures” mean? OR when you hear “different cultures”
what do you think about?

Summary of the responses:
14 students received this question. One respondent commented that “different cultures” could be
a reference to “geeks, smart, people, and girls who thing they’re all that” but understood that
different cultures meant “different races.”
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
Students from different cultural backgrounds get along well at this school.

A.1.7

Question 8: Sengpar41 Students can help make decisions in this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What sorts of decisions are students able to help make?

Summary of the responses:
10 students received this question Five respondents did not understand the scope of “decision”
for their examples: one student compared them to “going to King’s Dominion and kids wanting
to go off on their own but the teachers said they had to stay with a partner and the kids who went
off alone got lost,” another student said that “if there is not enough interest among students for a
field trip (thus not a lot of payments) the field trip gets cancelled.”
Recommendations:
Drop the item.

A.1.8

Question 9: Sengpar42 Students are given the chance to help solve school problems.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What sorts of problems are students able to help solve?

Summary of the responses:
10 students received this question. Six respondents had trouble understanding the question;
specifically, they were unclear on the scope of “problems,” comparing them to “tests and
whether they can retake them” or “students helping with homework” or breaking up fights when
the adults don’t come to help. One respondent initially thought that this question was the same as
the one before “Students can help make decisions in this school.”
Recommendations:
Drop the item.

A.1.9

Question 10: Sengpar43 At this school, the principal asks students what their ideas are.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What kind of ideas does the principal ask student about?
Who were you thinking about when you thought of the principal?

Summary of the responses:
10 students received this question. Two respondents had trouble understanding the question and
were unable to come up with specific examples of ideas the principals ask students about and
their tone was “doubtful.”
Recommendations:
No change.

A.1.10

Question 11: Sengrel10 Teachers and other adults at this school seem to take a real interest in
my future.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



The question is asking about “teachers and other adults”. When you answered this
question, which teachers were you thinking about?
Which “other adults” were you thinking about?

Summary of the responses:
12 students received this question. One respondent had trouble understanding the question,
saying that “the other adults they were thinking about were mostly their parents”.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
Adults working at this school seem to take a real interest in my future.

A.1.11

Question 12: Sengrel12 It is easy to talk with teachers.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What does the word “teacher” mean to you?
Who do you consider a teacher (ask for examples if the student is having a hard time
telling you who is a teacher).
When you were answering this question which teachers were you thinking about?

Summary of the responses:
12 students received this question. The three student who had trouble with the question said that
“it is easy to talk to teacher because if you misbehave you don’t have to say it or you can lie
about it.” One respondent was confused by the term “teacher,” stating that they consider their
dad a teacher because he has taught them a lot.
Recommendations:
Wording change: It is easy to talk with teachers at this school.

A.1.12

Question 13: Sengrel16 Teachers at this school help us children with our problems.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me what this question is asking?
Summary of the responses:
13 students received this question. One respondent who had trouble understanding the question
said the question meant, “how does the teacher help you solve your problems.”
Recommendations:
No change.

A.1.13

Question 14: Sengrel25 Adults near where I live support this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me what this question is asking?
Summary of the responses:
13 students received this question. Many respondents had trouble with was “near where I live”
with some believing this meant “whoever I live with” or “teachers and staff at the school like
counselors, nurses, and lunch people”. Other respondents believed that this question was asking
about the quality of the school, for example one thought this question was asking if adults in her
neighborhood thought this was a good school.
Recommendations:
Drop the item.

A.1.14

Question 15: Sengrel26 When there are events at this school, lots of families come.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When you hear “events at school”, what do you think the question is talking about?
Summary of the responses:
12 students received this question; none had any difficulty answering it or finding a response that
matched their experience.
Recommendations:
No change.

A.1.15

Question 16: Sengrel29 If I am absent, there is a teacher or some other adult at school who will
miss me.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:


Which teacher or other adult were you thinking of, who would miss you?

Summary of the responses:
12 students received this question. Three respondents had trouble with the word “missing” with
one respondent interpreting it as “feeling longing for someone rather than noticing their
absence.” Other respondents had difficulty knowing whether or not other people would miss
them. One respondent pointed out that “miss me” can be interpreted in different ways.
Recommendations:
Wording change: If I am absent, there is a teacher or some other adult at school that will notice
my absence.

A.1.16

Question 17: Senvins108 Other students often disrupt class.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about your answer to this question?
Summary of the responses:
17 students received this question; none had any difficulty answering it or finding a response that
matched their experience.
Recommendations:
No change.

A.1.17

Question 18: Senvins109 I get distracted from doing schoolwork in my classes because other
students are misbehaving, for example, talking or fighting.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is this question asking you in your own words?
Are you thinking of any particular distractions?

Summary of the responses:
17 students received this question; none had trouble understanding it.
Recommendations:
No change.

A.1.18

Question 19: Senvins113 My teachers give me individual attention when I need it.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When you answer this question and think about teachers, who are thinking about?
Summary of the responses:
17 students received this question. Three students had some issues understanding this question.
One respondent misinterpreted the question as meaning that teachers help students stay focused
“by snapping their fingers or telling them to look at the board. “Additionally one respondent was
“also thinking about their parents” and for another it “was not necessarily the teachers
themselves” but students who would assist the teachers after school.
Recommendations:
No change. The question sequence can provide the clarity that students need to answer these
questions—we suggest that this question is sequenced at the end of a set of questions about
teachers.

A.1.19

Question 20: Senvins115 The schoolwork I am assigned is meaningful and important.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When you think about schoolwork that is assigned, what does this mean to you?
Summary of the responses:
17 students received this question. 13 students had trouble understanding the question,
misinterpreting it as asking them if they were assigned homework to do at home, or if they had
regular homework.
Recommendations:
Wording change: My schoolwork is important to my future.

A.1.20

Question 21: Senvins119 Classes at this school really make me think.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What is this question asking you in your own words?
Summary of the responses:
17 students received this question. Many respondents interpreted “really make me think” as
understanding, concentrating/focusing, “wondering if I am right or wrong”, and whether the
class is good or bad. For others the question, made them “think about things related to” his or her
“future”.
Recommendations:
Wording change designed to get at academic rigor: Classes at this school are challenging.
Add this item: I'm really learning a lot in my classes. From the same survey (Survey of Chicago
Public Schools, 2012)

A.1.21

Question 22: Senvmen132 I can talk to my teachers about problems I am having in class.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is the question asking you in your own words?
What kind of “problems” did you think about when you were answering this question?

Summary of the responses:
15 students received this question; none had trouble understanding it.
Recommendations:
No change.

A.1.22

Question 23: Senvmen133 I can talk to a teacher or other adult at this school about something
that is bothering me.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



When we refer to “teacher or… adult” what comes to mind for you?
Who would you think about in answering this question?

Summary of the responses:
15 students received this question; no students had trouble understanding the question. During
probing students were asked about the phrase “teacher or other adult, “which students interpreted
as meaning teachers, social workers, principals, security personnel, or counselors.
Recommendations:
No change.

A.1.23

Question 24: Senvmen136 Students at this school think it’s ok to fight if someone insults them.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What is the question asking you in your own words?
Can you tell me at times when this happens?
Were you thinking of those times when you answered this question?

Summary of the responses:
15 students received this question. Two respondents were unable to find a response that matched
their experience. One respondent said “yes and no, it’s kind of in between,” the second
respondent said “I don’t really know whether to choose agree or disagree.” The respondents who
were unable to find a response understood the question as asking if they had engaged in the
behavior.
Recommendations:
No change.

A.1.24

Question 25: Senvpenv100 The bathrooms in this school are clean.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When answering these questions, how did you come up with the answers?
Summary of the responses:
15 students received to this question and none had a problem with it.
Recommendations:
No change.

A.1.25

Question 26: Senvpenv101 This school is clean and well-maintained.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When answering these questions, how did you come up with the answers?
Summary of the responses:
15 students received this question and none had a problem with it.
Recommendations:
No change.

A.1.26

Question 27: Senvpenv102 The temperature in this school is comfortable all year round.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When answering these questions, how did you come up with the answers?
Summary of the responses:
15 students received to this question, and none had trouble understanding it.
Recommendations:
No change.

A.1.27

Question 28: Senvpenv103 This school has a bright and pleasant appearance.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
15 students received this question. One student interpreted the question to mean a student’s
appearance and “the dress code is important” while another had “trouble with what bright and
pleasant meant.”
Recommendations:
Wording change: This school looks nice and pleasant.

A.1.28

Question 29: Sengpar44 I regularly attend school-sponsored events, such as school dances,
sporting events, student performances, or other school activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:





Tell me what you are thinking when you read this question?
What comes up for you when you hear school-sponsored events
Can you tell me, in your own words, what this question is asking?
You chose [response], tell me more about it.

Summary of the responses:
Nine students received this question and none had a problem with it.
Recommendations:
No change.

A.1.29

Question 30: Sengpar46 At this school, students have lots of chances to help decide things like
class activities and rules.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What kind of activities does this question make you think about? [Note to interviewer: if
student states that the activities that he/she is involved in is not listed].
Are there activities you are involved in at school?
o Tell me about those activities.
What about the rules in the question?
o What rules were you thinking of?

Summary of the responses:
10 students received this question. Although all students appeared to interpret the question
correctly, some students used examples that went beyond the classroom to include sports and the
dress code.
Recommendations:
No change.

A.1.30

Question 31: Sengpar47 There are lots of chances for students at this school to get involved in
sports, clubs, and other school activities outside of class.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How easy or hard was it to choose an answer?
Summary of the responses:
10 students received this question. One respondent mentioned activities that “students
did…outside of school” while another tied participation to behavior: “I agree because those that
misbehave during class are not allowed to participate.”
Recommendations:
No change.

A.1.31

Question 32: Sengpar48 I have lots of chances to be part of class discussions or activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight students received this question; none had trouble understanding it.
Recommendations:
No change.

A.1.32

Question 33: Senvdis147 Discipline problems are handled fairly at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:





Who or what is the question referring to?
What do you think this question is asking?
What does “discipline” mean to you?
Can you tell me in your own words how you describe fair?

Summary of the responses:
15 students received this question. One student would have liked an unsure or not applicable
option. Nine respondents had trouble understanding the question; specifically, many students did
not know what “discipline” meant and while others associated “discipline” with “being
respectful” and “growing up.” Other respondents thought the question meant “whether or not the
teacher/principal/security react fast enough to stop a problem or fight.”
Recommendations:
Split the question into two parts:
Question 1: School rules are applied equally.
Question 2: The punishment for breaking school rules is fair.

A.1.33

Question 34: Ssafemo58 I feel loved and wanted.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




Is this OK to talk about in a survey if your answers are kept private or is it too
uncomfortable?
In general how do you feel about being asked these questions?
Tell me more about what this question means to you.

Summary of the responses:
15 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.34

Question 35: Ssaferm96 This school has told students what to do if there is an emergency,
natural disaster (tornado, flood) or a dangerous situation (e.g. violent person on campus) during
the school day.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



When you answered this question, who were you thinking about?
Who has told you about what to do during an emergency?

Summary of the responses:
20 students received this question. One respondent said that it was “confusing and that it’s really
three questions in one” and said that the question should be “Do student know what to do if
there’s an emergency?”
Recommendations:
No change.

A.1.35

Question 36: Ssaferm98 If students hear about a threat to school or student safety, they have a
way to report it to someone without giving their name.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is this question asking, in your own words?
You answered [response]; how did you choose this response?

Summary of the responses:
21 students received this question. Two respondents refused to answer the question. One was
unable to find a response that matched experience. Two had trouble understanding the question.
Two were confused about the phrase “without giving their name.” One respondent said that he
was confused about “who was reporting to whom.” Another respondent thought that the question
had to do with knowing drills.
Recommendations:
Split into two different questions, one about reporting and one about anonymity:
Question 1: If students hear about a threat to school or student safety, they have a way to report it
to someone in authority.
Question 2: Students can report a threat to school or student safety without giving their names.

A.1.36

Question 37: Ssafsub84 Based on your experience, how much of a problem at this school is
student drug use?
 Not a problem
 Small problem
 Somewhat a problem
 Large problem
Probe:



Tell me more about what you are thinking about [question #].
What does “drug use” mean to you in this question?

Summary of the responses:
15 students received this question. One had trouble with the vocabulary in the question
(marijuana, LSD, cocaine, ecstasy) and appeared to be unfamiliar with these terms. One
respondent said she was in between two response option: not a problem and small problem.
Three students refused to answer the question and several students misunderstood the questions
as asking them if they had a problem or had engaged in drug use, for example one respondent
said “it is asking if I have a problem with this, isn’t it?”
Recommendations:
Wording change: At this school, how much of a problem is student drug use (e.g., Oxy, Ritalin,
marijuana, ecstasy)?

A.1.37

Question 38: Ssafsub85 Based on your experience, how much of a problem at this school is
student use of tobacco (cigarettes, chew, cigars).
 Not a problem
 Small problem
 Somewhat a problem
 Large problem
Probe:


Tell me more about what you are thinking about [question #].

Summary of the responses:
14 students received this question. One student thought that since the question began with “based
on your experience” it was asking about his “personal use of tobacco” and since he wasn’t a
smoker, he didn’t know how to answer. Three students added that hookah was also a type of
tobacco.
Recommendations:
Wording change: At this school, how much of a problem is student use of tobacco (e.g.,
cigarettes, chew, cigars, electronic cigarettes)?

A.1.38

Question 39: Ssafsub86 Based on your experience, how much of a problem at this school is
student alcohol use?
 Not a problem
 Small problem
 Somewhat a problem
 Large problem
Probe:



Tell me more about what you are thinking about [question #].
What does “alcohol” use mean to you in this question?

Summary of the responses:
15 students received this question. One respondent would have liked an unsure or not applicable
option. For one respondent the beginning of the question “Based on your experience” meant his
personal use of alcohol and “since he is not an alcohol drinker, he didn’t know how to answer.”
Recommendations:
Wording change: At this school, how much of a problem is student alcohol use ( e.g., beer, wine,
liquor)?

A.1.39

Question 40: Ssafsub92 Students at this school think it is okay to smoke one or more packs of
cigarettes a day.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:


In this question it refers to “smoke one or more packs of cigarettes.” Can you tell me
what you think a “pack of cigarettes” is?
o How about cigarettes?
o What kinds of things do you include in cigarettes?

Summary of the responses:
22 students received to this question and none had trouble understanding it.
Recommendations:
No change.

A.1.40

Question 41: Ssafsub93 Students at this school think it is okay to have one or two drinks of any
alcoholic beverage nearly every day.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In this question they use the term “alcoholic beverage” what does this term mean to you?
To you what are some types of alcoholic beverages?

Summary of the responses:
22 students received to this question and none had trouble understanding it.
Recommendations:
Wording change: Students at this school think it is okay to get drunk.

A.1.41

Question 42: Ssafsub94 Students at this school think it is okay to try drugs once or twice.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:


In your own words, what is this question asking?



What does “once or twice” mean for you?

Summary of the responses:
21 students received this question Eight students were confused by the phrase “once or twice,” as
respondents asked “a month, a day, at school?” suggesting the time frame for this question need
clarification.
Recommendations:
Wording change: Students at this school think it is okay to try drugs.

A.1.42

Question 43: Ssafsub95 Students at this school think it is okay to use drugs once a month or
more.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In your own words, what is this question asking?
Summary of the responses:
21 students received this question during cognitive testing. Nine students had a variety of
problems with the item. The first major problem had to do with the time reference in the question
“once a month or more.” Respondents answered with many versions of “once a month or more”:
setting times to do drug, doing it once then waiting a month before doing it again, continuous
use, and regular basis but respondents did not use the time reference that the question sought to
elicit. The other set of comprehension problems were around the interpretation of the question as
asking if it is ok to do drugs.
Recommendations:
Drop the item

A.1.43

Question 44: Senvpenv104 Overcrowding is a problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In this list of questions we use the word “overcrowding.”
In your words, what does this word mean to you?

Summary of the responses:
14 students received to this question and none had trouble understanding it.
Recommendations:
No change.

A.1.44

Question 45: Senvpenv107 Broken things at this school get fixed quickly.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When answering these questions, how did you come up with the answers?
Summary of the responses:
14 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.45

Question 46: Senvpenv99 The school buildings are pleasant and well maintained.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
15 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.46

Question 47: Senvphea123 How often do you eat fruit at school?
 Never
 Rarely
 Sometimes
 Often
Probe:



You said you [answer] eat fruit at school. How did you choose your response to this
question?
Can you give an example of a fruit students eat at school?

Summary of the responses:
15 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.47

Question 48: Senvphea124 How often do you eat vegetables at school?
 Never
 Rarely
 Sometimes
 Often
Probe: Can you give an example of a vegetable students eat at school?
Summary of the responses:
15 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.48

Question 49: Senvphea125 How often do you eat breakfast on school days?
 Never
 Rarely
 Sometimes
 Often
Probe: You said you [answer] eat breakfast on school days. What time period are you referring
to?
Summary of the responses:
15 students received to this question. Two students interpreted the question to mean how many
times did they eat breakfast at school.
Recommendations:
No change.

A.1.49

Question 50: Senvphea126 How often do you eat candy at school?
 Never
 Rarely
 Sometimes
 Often
Probe: Tell me more about what you are thinking about [question #].
Summary of the responses:
15 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.50

Question 51: Senvphea127 How often do you drink soda at school?
 Never
 Rarely
 Sometimes
 Often
Probe: Tell me more about what you are thinking about [question #].
Summary of the responses:
14 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.51

Question 52: Senvphea128 How often do you go to gym class or participate in other physical
activity during the school day?
 Never
 Rarely
 Sometimes
 Often
Probe: What does “physical activity” mean in these questions?
Summary of the responses:
14 students received this question. Two students had difficulty with “physical activity;” for
example, one did not consider sports like soccer to be physical activity.
Recommendations:
Wording change: How often do you go to gym class or participate in other physical activity
during the school day (e.g., running, playing sports)?

A.1.52

Question 53: Senvphea129 How often do you stay after school to participate in sports or other
physical activity?
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “physical activity” mean in these questions?
Summary of the responses:
15 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.53

Question 54: Ssafbul74 Students at this school are teased or picked on about their race or
ethnicity.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Your answered [answer] to the first question, can you Tell me more about your answer.
What does “race or ethnicity” mean in this question for you?

Summary of the responses:
15 students received this question. Four had some difficulty when probed for the meaning for
ethnicity—three did not know what ethnicity was and one said it referred to religion.
Recommendations:
No change.

A.1.54

Question 55: Ssafbul75 Students at this school are teased or picked on about their culture or
religion.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “culture or religion” mean in this question for you?
Summary of the responses:
14 students received this question and none had trouble understanding it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
Students at this school are teased or picked on about their cultural background or religion.

A.1.55

Question 56: Ssafbul76 Students at this school are teased or picked on about their physical or
mental disability.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “disability” mean to you in this question? [If student has trouble you may ask
for examples: can you give me some examples of what you mean by physical disability? By
mental disability?]
Summary of the responses:
15 students received this question. One student did not know the definition of disability.
Recommendations:
No change.

A.1.56

Question 57: Ssafbul77 Students at this school are teased or picked on about their sexuality.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
14 students received this question. Five students had trouble with the concept of sexuality and
linked the word sexuality to gender and sex. One student refused to answer the question.
Recommendations:
Split this question into two separate questions; one focusing on bullying based on perceived
sexual promiscuity and the other focused on bullying based on sexual orientation.
Question 1: Students at this school are teased or picked on about their dating or sexual behavior.
Question 2: Students at this school are teased or picked on about their sexual orientation.

A.1.57

Question 58: Ssafbul82 Students at this school are often cyber bullied (e.g. receiving a
threatening or hurtful message from another student in an email, on a website, on a cell phone, or
in instant messaging).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



The question uses the word “cyber bullied,” what does “cyber bullied” mean in your own
words?
In what ways does cyber bullying or teasing happen? [Note to interviewer: try to find out
if the list above includes everything they are thinking about; want to know what we are
missing].

Summary of the responses:
14 students received this question. One of these students did not understand the example “text
messaging.” One student understood cyber bullying to be “when someone goes to a website and
meets another person that they want to have a relationship with” in addition to saying back things
or offending others on the internet. Two students mentioned Facebook and twitter as additional
modes of cyber bullying—one mentioned Instagram and one mentioned Twitter.
Recommendations:
No change.

A.1.58

Question 59: Ssafbul83 Students often spread mean rumors or lies about others at this school on
the internet (i.e., Facebook™, Myspace™, email, and instant message).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is this question asking in your own words? In what ways are rumors spread?
Are there things missing in this list? [Note to interviewer: try to find out if the list above
includes everything they are thinking about; want to know what we are missing].

Summary of the responses:
14 students received this question and all had a clear understanding of the question; although,
one student was confused by the “TM” symbols.
Recommendations:
No change.

A.1.59

Question 60: Ssafemo49 Students at this school get along well with each other.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “getting along well” mean to you?
How did you choose an answer?

Summary of the responses:
Eight students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.60

Question 61: Ssafemo50 Students at this school can tell their teachers if they feel confused
about something in class.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Who were you thinking of when you thought about “students” in these questions—which
students?
Summary of the responses:
Eight students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.61

Question 62: Ssafemo51 Students at this school are sensitive to the feelings of other students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does it mean to you to be “sensitive to the feelings of other students?”
Summary of the responses:
Eight students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.62

Question 63: Ssafemo52 At this school, students talk about the importance of understanding
their own feelings and the feelings of others.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is this question asking in your own words?
What does it mean to you to “understand your own feelings?”

Summary of the responses:
Eight students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.63

Question 64: Ssafemo53 At this school, students work on listening to others to understand what
they are trying to say.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.64

Question 65: Ssafemo54 I am happy to be at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.65

Question 66: Ssafemo55 I feel close to people at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.66

Question 67: Ssafemo56 I feel like I am part of this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
13 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.67

Question 68: Ssafemo57 I feel socially accepted.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
13 students received this question. One student did not understand what “socially” meant.
Recommendations:
No change.

A.1.68

Question 69: Ssaferm97 Students know what to do if there is an emergency, natural disaster
(tornado, flood) or a dangerous situation (e.g. violent person on campus) during the school day.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What is this question asking, in your own words? You answered [response], how did you
choose this response?
Summary of the responses:
21 students received this question. Two respondents refused to answer the question. The rest did
not have trouble understanding the question. Many students mentioned that this question initially
seemed the same as Ssafenm96.
Recommendations:
No change.

A.1.69

Question 70: Sdemo150 What is your race?
 White
 Black or African-American
 Asian
 American Indian or Alaskan Native
 Native Hawaiian or Pacific Islander
 Other
Probe: There is no probe for this item.
Summary of the responses:
25 students received this question and none had trouble understanding it.
Recommendations:
No change.

A.1.70

Appendix A.2 Teacher/instructional staff survey – Item-by-item
recommendations
Question 1: Idemo139 Are you of Hispanic or Latino origin?
 Yes
 No
Probe: There is no probe for this item.
Summary of the responses:
Fifteen respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.2.1

Question 2: Idemo140 What is your race?
 White
 Black or African-American
 Asian
 American Indian or Alaska Native
 Native Hawaiian or Pacific Islander
 Other
Probe: There is no probe for this item.
Summary of the responses:
Fifteen respondents received this question. One respondent had trouble responding to it because
Latino was not listed as an option for race.
Recommendations:
No change.

A.2.2

Question 3: Idemo141 Is your main assignment/responsibility at this school to provide
instruction or other support services to any of these types of students - Special Education,
English Language Learners, Gifted and Talented Education students, and Migrant Education?
 Yes
 No
Probe:



Can you tell me more about your assignment at school and the responsibilities that you
have?
What does “this school” mean to you in the context of this question?

Summary of the responses:
Thirteen respondents received this question. Three teachers answered “yes” because of the
presence of these students in their class, although they noted that this wasn’t necessarily their
“main responsibility.”
Recommendations:
Formatting change (bolding the phrase, as shown):
Is your main assignment/responsibility at this school to provide instruction or other support
services to any of these types of students - Special Education, English Language Learners, Gifted
and Talented Education students, and Migrant Education?

A.2.3

Question 4: Idemo142 How many years have you been working at this school?
 1-3 years
 4-9 years
 10-19 years
 20 or more years
Probe:



You said [response]; how did you come up with that answer?
How certain are you of your response?

Summary of the responses:
Fifteen respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.2.4

Question 5: lengclc1 At this school, closing the racial/ethnic achievement gap is considered a
high priority.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




How well does this question apply to your school?
Can you tell me more about what comes to mind: Is there a gap at the school that you
work at?
How do you read/understand the racial/ethnic concept?

Summary of the responses:
Nine teachers received this question. Two respondents misunderstood the racial/ethnic
achievement gap as having to do with the presence of different cultures at school.
Recommendations:
Wording change: At this school, closing the racial/ethnic academic achievement gap is
considered a high priority.

A.2.5

Question 6: lengclc2 At this school, all students are treated equally, regardless of whether their
parents are rich or poor.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What do you think this question is asking you?
Summary of the responses:
Nine teachers received this question. One teacher initially wanted to answer “somewhat,” but
eventually answered “disagree.”
Recommendations:
No change.

A.2.6

Question 7: lengclc3 This school encourages all students to enroll in challenging courses
regardless of their race, ethnicity, or nationality.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



The question uses the term “challenging courses.”
In your own words, what does this term mean? Tell me more about why you chose the
[response].

Summary of the responses:
Nine teachers received this question. One respondent skipped it, saying “this question is
irrelevant to elementary school because there is a prescribed course of the work and students do
not make the choice to enroll in classes.”
Recommendations:
No change.

A.2.7

Question 8: Iengclc4 This school provides instructional materials that reflect students' culture,
ethnicity and identity.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What is this question asking you?
What does “instructional materials” mean to you?
In your own words what does “culture, ethnicity and identity” mean in this question?

Summary of the responses:
Nine teachers received this question, and all had a clear understanding of it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
This school provides instructional materials (e.g., textbooks, handouts, software applications)
that reflect students’ cultural background, ethnicity and identity.

A.2.8

Question 9: Iengclc5 This school fosters an appreciation of student diversity and respect for
each other.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: For you, what does it mean to foster an appreciation for diversity?
Summary of the responses:
Nine teachers received this question, and one skipped it.
Recommendations:
No change.

A.2.9

Question 10: Iengclc6 This school emphasizes showing respect for all students’ cultural beliefs
and practices.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does it mean for you to show respect to all students?
Summary of the responses:
Nine teachers received this question, and none had any difficulty understanding it.
Recommendations:
No change.

A.2.10

Question 11: Iengclc7 This school provides effective supports for teaching culturally and
linguistically diverse students with IEPs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:





Let’s talk about this question; how did you arrive at your answer?
Can you tell me about the term “IEPs?”
What does it refer to for you and at your school?
What do “effective supports” mean to you?

Summary of the responses:
Nine teachers received this question. One respondent did not understand the “culturally diverse”
aspect of the question. One respondent interpreted the phrase “culturally and linguistically
diverse” to mean English language learners.
Recommendations:
Wording change: This school provides effective supports (e.g., student assessment and
evaluation, collaboration among teachers and specialists, staff training) for teaching culturally
and linguistically diverse students with Individualized Education Programs (IEPs).

A.2.11

Question 12: Iengclc8 This school provides effective supports for students needing alternative
modes of communication (e.g., manual signs, communication boards, computer-based devices,
picture exchange systems, Braille).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you arrive at your answer?
What were you thinking as you picked a response?

Summary of the responses:
Nine teachers received this question. None had trouble answering it.
Recommendations:
No change.

A.2.12

Question 13: Iengpar29 My level of involvement in decision making at this school is fine with
me.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




How well does this question apply to you?
How did you come up with an answer?
What kind of decisions were you thinking about?

Summary of the responses:
Nine teachers received this question. One respondent said he was very unclear as to what
“decisions” the question was asking about.
Recommendations:
No change.

A.2.13

Question 14: Iengpar31 Staff at this school have many informal opportunities to influence what
happens within the school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What types of informal opportunities come to mind when you answer this question?
When we say “influence” in this question, what comes to mind for you?

Summary of the responses:
Nine teachers received this question. One teacher was unclear as to what “informal” means.
Recommendations:
No change.

A.2.14

Question 15: Iengpar32 At this school, students are given the opportunity to take part in decision
making.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What does “take part in decision making” mean to you?

Summary of the responses:
Nine teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.15

Question 16: Iengpar33 Students at this school are encouraged to help solve problems at this
school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What problems were you thinking of?

Summary of the responses:
Nine teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.16

Question 17: Iengpar35 Administrators consistently seek input from staff.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




The question uses the term “administrators” and “staff.” In your own words, what do
these terms mean?
Tell me more about why you chose the [response].
What “input” were you thinking of?

Summary of the responses:
Nine teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.17

Question 18: Iengpar37 This school’s administration invites students to share their ideas about
the school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What types of ideas does the administration invite students to share?
When you think of the administration, what comes to mind? Whom does that include?

Summary of the responses:
Nine teachers were asked this question. One respondent skipped it, saying she didn’t see any
evidence of this practice.
Recommendations:
No change.

A.2.18

Question 19: Iengpar39 Staff at this school make it easy for students to suggest activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you arrive at your answer?
Tell me more about why you chose [response].

Summary of the responses:
Nine teachers received this question. None had any difficulty understanding it.
Recommendations:
No change.

A.2.19

Question 20: Iengpar42 This school provides students with opportunities to take a lead role in
organizing programs and activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you arrive at your answer?
Tell me more about why you chose [response].

Summary of the responses:
Nine teachers received this question. One teacher initially found the phrase “lead role”
confusing. After reading the question a second time, the respondent was able to answer the
question without difficulty.
Recommendations:
No change.

A.2.20

Question 21: Iengpar48 Students are encouraged to get involved in extra-curricular activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you arrive at your answer?
Tell me more about why you chose [response].

Summary of the responses:
Eight teachers were asked this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.21

Question 22: Iengrel10 Staff do a good job helping parents understand when their child needs to
learn social, emotional, and character skills.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In this question we use the term “parents.” What does that term refer to for you in the
context of your position at this school?
How about “staff” – who comes to mind when you read this term?

Summary of the responses:
Nine teachers received this question. All had a clear understanding of it.
Recommendations:
No change

A.2.22

Question 23: Iengrel13 Staff do a good job showing parents how to keep track of their child’s
progress.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does it mean to you for the staff to do a good job? – A good job for whom?
Summary of the responses:
Nine teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.23

Question 24: Iengrel14 This school asks families to volunteer at the school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What types of volunteer activities come to mind when you answer this question?
When we say “families” in this question, what comes to mind for you?

Summary of the responses:
Nine teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.24

Question 25: Iengrel15 This school communicates with parents in a timely and ongoing basis.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What types of communication are you thinking of?
What, to you, does “timely and ongoing” mean in this question?

Summary of the responses:
Nine teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.25

Question 26: Ienvdis134 Discipline problems are handled fairly at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “discipline problems” mean to you?
Can you tell me, in your own words, how you describe “fairly?”

Summary of the responses:
Three teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.26

Question 27: Ienvdis135 This school effectively handles student discipline and behavior
problems.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “effectively handled” mean to you?
Summary of the responses:
Three teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.27

Question 28: Ienvdis136 Staff at this school work together to ensure an orderly environment.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: We use the term “orderly environment.” What does this term mean for you as a teacher?
Summary of the responses:
Three teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.28

Question 29: Ienvins104 The students in my class(es) attend class regularly.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about your answer to this question. For you, what does it mean to “attend
class regularly”?
[Ask respondents if they teach more than one class. If “yes,” ask]: Were you thinking of
all your classes, or just a few of your classes, when you answered this question?

Summary of the responses:
Six teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.29

Question 30: Ienvins105 The students in my class(es) come to class prepared with the
appropriate supplies and books.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is this question asking, in your own words?
For you, what does it mean to come to class “prepared”?

Summary of the responses:
Six teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.30

Question 31: Ienvins106 The students in my class(es) actively participate in class activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How often do you take stock of participation in your classes?
When you think about participation, how do you think about it?

Summary of the responses:
Six teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.31

Question 32: Ienvins108 We have so many different programs at this school that I can’t keep
track of them all.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What do you think is meant by “programs at this school?”
Can you tell me about the programs?

Summary of the responses:
Six teachers received this question. Three respondents pointed out that the question was vague.
For example, one respondent said that when thinking about the question, she only considered
programs that applied to her (and that she doesn’t have to keep track of all of the programs at her
school because she does not participate in all of them). She suggested that the wording of the
question could be changed to account for that. Another respondent stated that for her the question
was asking two different things: Are there many different programs at your school? Can you
keep track of them all? One respondent said it might be good to clarify what is meant by
“programs,” since it was not clear to the respondent whether the question was referring to
academic or more nonacademic programs.
Recommendations:
Wording change: The programs and resources at this school are adequate to support students’
learning.

A.2.32

Question 33: Ienvins110 Teachers at this school feel responsible to help each other do their best.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Six teachers received this question, and all had a clear understanding of it. Two teachers
mentioned that they had a positive environment where they could help each other out, but did not
necessarily “feel responsible” to do so.
Recommendations:
No change.

A.2.33

Question 34: Ienvins112 Teachers at this school feel responsible when students at this school
fail.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Six teachers received this question. All had a clear understanding of it.
Recommendations:
No change.

A.2.34

Question 35: Ienvmen123 This school provides quality counseling or other services to help
students with social or emotional needs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What does the term “quality counseling or other services” mean for you?
Did you find a response category that fits your experience?
Can tell me more about your answer?

Summary of the responses:
Four teachers received this question. One teacher said she didn’t know what “quality counseling”
meant.
Recommendations:
No change.

A.2.35

Question 36: Ienvmen125 This school provides the materials, resources, and training necessary
for me to support students’ social or emotional needs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me what you think this question is asking you?
Were you thinking about the materials, resources, and training provided specifically for
you?

Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.36

Question 37: Ienvmen126 This school places a priority on addressing students’ mental health
needs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about why you chose [answer].
Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.37

Question 38: Ienvmen127 This school places a priority on social and emotional development.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does the term “emotional development” mean for you?
Tell me more about why you chose [answer].

Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.38

Question 39: Ienvmen128 This school places a priority on teaching students strategies to
manage their stress levels.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about why you chose [answer].
Summary of the responses:
Four teachers received this question. One respondent answered “disagree,” stating that “it is done
but not [considered] a priority.” Another respondent answered “disagree” and explained that she
chose this answer because “they are not against it, it’s just that they haven’t done anything [yet].”
Recommendations:
No change.

A.2.39

Question 40: Ienvmen137 This school places a priority on helping students with their social,
emotional, and behavioral problems.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does the term “social, emotional, and behavioral problems” mean for you?
Tell me more about why you chose [answer].

Summary of the responses:
Four teachers received this question. One respondent viewed this as the same as Ienvmen127
(Noninstructional staff question 38) “This school places a priority on social and emotional
development.”
Recommendations:
No change.

A.2.40

Question 41: Ienvpenv100 My teaching is hindered by poor heating, cooling, and/or lighting
systems at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about your answer?
What do you think this question is asking?

Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.41

Question 42: Ienvpenv101 My teaching is hindered by a lack of instructional space (e.g.,
classrooms) at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about your answer?
What do you think this question is asking?

Summary of the responses:
Eight teachers received this question. One respondent incorporated supplies into their answer:
“No overcrowding, but the issue is with a lack of science teaching supplies and lab equipment.”
One respondent answered “Strongly Agree,” but said that her instructional area seemed “OK.”
Recommendations:
No change.

A.2.42

Question 43: Ienvpenv102 My teaching is hindered by a lack of textbooks and basic supplies at
this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “basic supplies” mean to you in this question?
Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.43

Question 44: Ienvpenv103 My teaching is hindered by inadequate or outdated equipment or
facilities at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Which “equipment or facilities” were you thinking of when you answered this question?
Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.44

Question 45: Ienvpenv97 This school has a bright and pleasant appearance.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me what a “bright and pleasant appearance” means to you.
Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
This school looks clean and pleasant.

A.2.45

Question 46: Ienvpenv98 This school is an inviting work environment.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me about how you came up with your answer to this question.
Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.46

Question 47: Ienvpenv99 Overcrowding is a problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: We use the word “overcrowding”; in your own words, what does this word mean?
Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.47

Question 48: Ienvphea117 Staff at this school promote students’ physical health and nutrition.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can tell me more about your answer?
Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.48

Question 49: Ienvphea118 This school provides quality physical health and nutrition services for
students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me about your answer?
What comes to mind when you read “quality services”?

Summary of the responses:
Four teachers received this question. Two teachers were unclear as to what should be considered
“quality physical health and nutrition services.”
Recommendations:
Wording change: This school provides quality physical health and nutrition services (e.g., school
nurse, free and reduced price meals, before and after school meals) for students.

A.2.49

Question 50: Ienvphea119 This school provides the materials, resources, and training necessary
for me to support students’ physical health and nutrition.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about why you chose [answer].
How can you tell if your school places a “priority” on this issue?

Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.50

Question 51: Ienvphea120 This school places a priority on making healthy food choices.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about why you chose [answer].
How can you tell if your school places a “priority” on this issue?

Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.51

Question 52: Ienvphea121 This school places a priority on students’ health needs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about why you chose [answer].
How can you tell if your school places a “priority” on this issue?

Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.52

Question 53: Ienvphea122 This school places a priority on students’ physical activity.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about why you chose [answer].
How can you tell if your school places a “priority” on this issue?

Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.53

Question 54: Ienvphea138 This school provides quality physical health and nutrition instruction.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me how you chose your answer?
Summary of the responses:
Four teachers answered this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.54

Question 55: Isafbul68 I think that bullying is a frequent problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.55

Question 56: Isafbul69 I think that cyberbullying is a frequent problem among students at this
school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Five teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.56

Question 57: Isafbul70 I think that racial/ethnic tension or discrimination among students is a
frequent problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Five teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.57

Question 58: Isafbul71 Students at this school would feel comfortable reporting a bullying
incident to a teacher or other staff.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What “teacher or other staff” were you thinking of when you answered this question?
Summary of the responses:
Five teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.58

Question 59: Isafbul72 Staff at this school communicate to students that bullying is
unacceptable.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me more about you think this question is asking you?
Summary of the responses:
Six teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.59

Question 60: Isafbul73 Staff at this school always stop bullying when they see it.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me about how you arrived at your answer?
Summary of the responses:
Six teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.60

Question 61: Isafbul74 Staff at this school know what to say or do to intervene in a bullying
situation.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you come up with your answer to this question?
Can you tell me more about intervening in a bullying situation?

Summary of the responses:
Six teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.61

Question 62: Isafbul75 This school provides bullying prevention.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In your own words, what is this question asking you?
How do you think your school provides bullying prevention?

Summary of the responses:
Five teachers received this question. One respondent asked, “Does it refer to an organization
coming in to speak to students? Does it refer to the school’s core values? Simple
communication?” Ultimately, she decided that bullying prevention is taking place because the
school tells students that bullying is not acceptable and prevention is in the school’s core values.
Recommendations:
No change.

A.2.62

Question 63: Isafbul79 Staff at this school are teased or picked on about their race or ethnicity.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Four teachers answered this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.63

Question 64: Isafbul80 Staff at this school are teased or picked on about their culture or religion.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Five teachers received this question. All indicated a clear understanding of it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
Adults working at this school are teased or picked on about their cultural background or religion.

A.2.64

Question 65: Isafbul81 Staff at this school are teased or picked on about their physical or mental
disability.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Five teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.65

Question 66: Isafbul82 Staff at this school are teased or picked on about their sexuality.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Four teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.66

Question 67: Isafemo49 This school is an emotionally safe place for students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In this question, we use the term “emotionally safe.” What does this term mean, in your
own words?
Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.67

Question 68: Isafemo50 Students get along well with each other.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “get along well” mean to you?
How did you choose an answer?

Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.68

Question 69: Isafemo51 This school is an emotionally safe place for staff.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In this question, we use the term “emotionally safe.” What does this term mean, in your
own words?
Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.69

Question 70: Isafemo52 I feel like I belong.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What does it mean for you to “belong?”
Were you thinking of belonging at this school, or belonging in general?
Tell me more about your answer.

Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.70

Question 71: Isafemo53 I feel satisfied with the recognition I get for doing a good job.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




You answered [answer]; tell me more about your answer.
How do you get recognized?
In what ways do you feel recognized?

Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.71

Question 72: Isafemo54 I feel comfortable discussing feelings, worries, and frustrations with my
supervisor.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




In your own words, what is this question asking you?
How well does this question apply to you?
How did you come up with an answer?

Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.72

Question 73: Isafemo55 This school inspires me to do the very best at my job.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




In your own words, what is this question asking you?
How well does this question apply to you?
How did you come up with an answer?

Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.73

Question 74: Isafemo56 People at this school care about me as a person.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




In your own words, what is this question asking you?
How well does this question apply to you?
How did you come up with an answer?

Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.74

Question 75: Isafemo57 I can effectively work with defiant or disruptive students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



To you, what does the term “defiant or disruptive students” mean?
Can you tell me more about how you work with them effectively?

Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.75

Question 76: Isafemo58 I can manage almost any student behavior problem.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about managing student behavior problems.
Summary of the responses:
Eight teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.76

Question 77: Isaferm92 I know what to do if there is an emergency, natural disaster (tornado,
flood) or a dangerous situation (e.g., violent person on campus) during the school day.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What is this question asking, in your own words?
How do you know what to do?
You answered [response]. How did you choose this response?

Summary of the responses:
Nine teachers received this question. All indicated a clear understanding of it.
Recommendations:
No change.

A.2.77

Question 78: Isaferm93 This school has a written plan that describes procedures to be performed
in shootings.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me what you were thinking when you read and answered this question.
Summary of the responses:
Nine teachers received this question. One respondent seemed to be confused by the term
“shootings” and said that there is a marked difference in procedure depending on whether the
incident is a domestic terrorist attack, an intruder who is unarmed, or an active shooter on
campus.
Recommendations:
No change.

A.2.78

Question 79: Isaferm94 This school has a written plan that clearly describes procedures to be
performed in natural disasters (e.g., earthquakes or tornadoes).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When you answered this question, what were you thinking about?
Summary of the responses:
Nine teachers received this question. One respondent said she thought the question should be
split into two parts: (1) Does the school have a written plan? (2) Does it clearly describe the
procedures?
Recommendations:
No change.

A.2.79

Question 80: Isaferm95 This school or school district provides effective training in safety
procedures to staff (e.g., how to handle emergencies).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



You answered [response]. Can you explain/tell me more about your answer?
What emergencies were you thinking of?

Summary of the responses:
Nine teachers received this question. One teacher indicated that she wasn’t sure how to answer
because she hadn’t been in one of these emergencies. One teacher was unclear on what safety
procedures and emergencies the question referred to. One teacher was unclear on how training
should be defined.
Recommendations:
Wording change: This school or school district provides effective training in safety procedures
(e.g., how to handle emergencies, lockdown training, fire drills) to staff.

A.2.80

Question 81: Isafsub83 Based on your experience, how much of a problem at this school is
student drug use?
 Not a Problem
 Small Problem
 Somewhat a Problem
 Large Problem
Probe:




What is this question asking, in your own words?
You answered [answer]. Can you describe what this means to you?
What does it mean for students to engage in drug use? [Interviewer: Try to get at
respondents’ understanding of “use” – how often, how much, etc.]

Summary of the responses:
Ten teachers received this question. All indicated a clear understanding of it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student drug use (e.g., Oxy, Ritalin, marijuana,
ecstasy)?

A.2.81

Question 82: Isafsub84 Based on your experience, how much of a problem at this school is
student use of tobacco (such as cigarettes, chew, cigars)?
 Not a Problem
 Small Problem
 Somewhat a Problem
 Large Problem
Probe:



You answered [answer]. Can you describe what this means to you?
What does it mean that students use tobacco? Can you tell me about the things you took
into account for this question?

Summary of the responses:
Ten teachers received this question. All indicated a clear understanding of it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student use of tobacco (e.g., cigarettes, chew, cigars,
electronic cigarettes)?

A.2.82

Question 83: Isafsub85 Based on your experience, how much of a problem at this school is
student alcohol use?
 Not a Problem
 Small Problem
 Somewhat a Problem
 Large Problem
Probe: You answered [answer]. Can you describe what this means to you?



What does it mean that students use alcohol?
Can you tell me about the things you took into account for this question?

Summary of the responses:
Ten teachers received this question. One respondent said it was hard to reliably choose between
“small problem,” “somewhat a problem,” and” large problem” because she wasn’t sure to what
extent it was a problem.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student alcohol use (e.g., beer, wine, liquor)?

A.2.83

Question 84: Isafsub86 This school collaborates well with community organizations to help
address youth substance use problems.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What kind of community organization are you thinking about?
When you think of collaboration, what does that mean to you?
What were you thinking about when you read “substance use?”

Summary of the responses:
Nine teachers received this question. One respondent suggested that this question should be split
into two: “Does the school collaborate with community organizations? Yes/No. If yes, how
well?”
Recommendations:
No change.

A.2.84

Question 85: Isafsub87 This school has adequate resources to address substance use prevention.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In your own words, what is this question asking you?
What “adequate resources” were you thinking of?

Summary of the responses:
Ten teachers received this question. All indicated a clear understanding of it.
Recommendations:
No changes.

A.2.85

Question 86: Isafsub88 This school provides effective confidential support and referral services
for students needing help because of substance abuse.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:





When you hear the term “substance abuse,” what does that refer to?
What kind of support are you thinking about?
What does “confidential” mean to you?
What does “confidential” mean in terms of your school? What about “referral services?”

Summary of the responses:
Ten teachers received this question. Four respondents thought there should be a “don’t know”
response.
Recommendations:
No change because instructional staff can always skip this question.

A.2.86

Question 87: Isafsub89 At this school, first-time violations of alcohol or other drug policies are
punished by at least an out-of-school suspension.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In this question, we use the term “suspension.” Can you tell me what this means for you?
If respondent answers “Strongly Disagree” or “Disagree,” ask: What does happen to firsttime offenders?

Summary of the responses:
Ten teachers received this question. Three respondents indicated that they would prefer to
answer “don’t know.”
Recommendations:
No change because instructional staff can always skip the question. However, for consistency
purposes, the following formatting change will be made to the item (bolding the phrase, as
shown):
At this school, first-time violations of alcohol or other drug policies are punished by at least an
out-of-school suspension.

A.2.87

Question 88: Isafsub90 Substance abuse prevention is an important goal at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me about how you arrived at your answer?
Summary of the responses:
Ten teachers received this question. All indicated a clear understanding of it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
This school has programs, resources, and/or policies to prevent substance abuse.

A.2.88

Question 89: Isafsub91 This school has programs that address substance use among students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “substance use” mean to you?
What kind of programs are you thinking about?

Summary of the responses:
Ten teachers received this question. One respondent said she would prefer a “not applicable” or
“don’t know” option, and another respondent indicated that substance use is the same as abuse.
Recommendations:
No change.

A.2.89

Appendix A.3 Principal/non-instructional staff survey – Item-by-item
recommendations
Question 1: Ndemo146 How many years have you been working at this school?
 1-3 years
 4-9 years
 10-19 years
 20 or more years
Probe:



You said [response]; how did you come up with that answer?
How certain are you of your response?

Summary of the responses:
Fifteen respondents received this question during the cognitive interviews; none had issues
answering this item.
Recommendations:
No change.

A.3.1

Question 2: Ndemo145 Is your main assignment/responsibility at this school to provide
instruction or other support services to any of these types of students - Special Education,
English Language Learners, Gifted and Talented Education students, and Migrant Education?
 Yes
 No
Probe:



Can you tell me more about your assignment at school and the responsibilities that you
have?
What does “this school” mean to you in the context of this question?

Summary of the responses:
Fifteen respondents received this question. One respondent initially thought she was being asked
if she worked with all of the types of students listed; thus, she felt that the question did not apply
to her. Once she reread the question, she realized it meant one of the types of students.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following formatting change will be made to the item (bolding the phrase, as
shown):
Is your main assignment/responsibility at this school to provide instruction or other support
services to any of these types of students - Special Education, English Language Learners, Gifted
and Talented Education students, and Migrant Education?

A.3.2

Question 3: Ndemo143 Are you of Hispanic or Latino origin?
 Yes


No

Probe: There is no probe for this item.
Summary of the responses:
Fifteen respondents received this question; none had any difficulty answering it.
Recommendations:
No change.

A.3.3

Question 4: Ndemo144 What is your race?
 White
 Black or African-American
 Asian
 American Indian or Alaska Native
 Native Hawaiian or Pacific Islander
 Other
Probe: There is no probe for this item.
Summary of the responses:
Fifteen respondents received this question; none had any difficulty answering it.
Recommendations:
No change.

A.3.4

Question 5: Nengclc2 At this school, all students are treated equally, regardless of whether their
parents are rich or poor.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In this question, we use the term “parents.” What does that term refer to for you in the
context of being a principal/staff at the school?
Summary of the responses:
Eight respondents received this question; none had any difficulty answering the question or
finding a response that matched their experience.
Recommendations:
No change.

A.3.5

Question 6: Nengclc5 This school fosters an appreciation of student diversity and respect for
each other.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is this question asking you?
What does “student diversity” mean to you in this question? How about “this school”?

Summary of the responses:
Eight respondents received this question; none had any difficulty answering the question or
finding a response that matched their experience.
Recommendations:
No change.

A.3.6

Question 7: Nengclc6 This school emphasizes showing respect for all students’ cultural beliefs
and practices.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What is this question asking you? In your own words, what does “cultural beliefs and
practices” mean in this question?
How did you decide how to answer this question?

Summary of the responses:
Eight respondents received this question; none had any difficulty answering the question or
finding a response that matched their experience.
Recommendations:
No change.

A.3.7

Question 8: Nengclc4 This school provides instructional materials that reflect students’ culture,
ethnicity and identity.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What is this question asking you?
What does “instructional materials” mean to you?
In your own words, what does “culture, ethnicity and identity” mean in this question?

Summary of the responses:
Eight respondents received this question. Three respondents interpreted the term “instructional
materials” as referring to curriculum.
Recommendations:
Wording change: This school provides instructional materials (e.g., textbooks, handouts,
software applications) that reflect students’ cultural background, ethnicity and identity.

A.3.8

Question 9: Nengclc7 This school provides effective supports for teaching culturally and
linguistically diverse students with IEPs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree

Probe:




Let’s talk about this question; how did you arrive at your answer?
Can you tell me about the term “IEP”? What does it mean to you?
What does the term “cultural and linguistic diversity” mean to you?

Summary of the responses:
Eight respondents received this question. One respondent said she didn’t know what IEP meant.
She was also confused by “teaching culturally.” Another respondent had trouble with the phrase
“culturally and linguistically diverse” and with the term IEP. Both of these respondents were
members of the kitchen staff at the school.
Recommendations:
No change.

A.3.9

Question 10: Nengclc8 This school provides effective supports for students needing alternative
modes of communication (e.g., manual signs, communication boards, computer-based devices,
picture exchange systems, Braille).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




How did you arrive at your answer?
What were you thinking as you picked a response?
What do you think of when you see “students needing alternative modes of
communication”?

Summary of the responses:
Eight respondents received this question; none had any difficulty answering it or finding a
response that matched their experience.
Recommendations:
No change.

A.3.10

Question 11: Nengclc3 This school encourages all students to enroll in challenging courses
regardless of their race, ethnicity, or nationality.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



The question uses the term “challenging courses.” In your own words, what does this
term mean?
Tell me more about why you chose the [response].

Summary of the responses:
Eight respondents received this question; none had any difficulty answering it or finding a
response that matched their experience.
Recommendations:
No change.

A.3.11

Question 12: Nengclc1 At this school, closing the racial/ethnic achievement gap is considered a
high priority.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How well does this question apply to your school?
Can you tell me more about what comes to mind when you hear “achievement gap?”

Summary of the responses:
Seven respondents received this question; none had any difficulty answering it or finding a
response that matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, closing the racial/ethnic academic achievement gap is considered a high priority.

A.3.12

Question 13: Nengrel17 Staff at this school do a good job helping parents to support their
children’s learning at home.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How easy or difficult is it to come up with an answer?
Summary of the responses:
Six respondents received this question; none had any difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.13

Question 14: Nengrel21 Staff at this school do a good job showing families how to keep track of
their child’s progress.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you decide how to answer this question?
What does it mean to you for the staff to “do a good job?”

Summary of the responses:
Six respondents received this question; none had any difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.14

Question 15: Nengrel16 This school helps parents find community supports for their students
who need them.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “community supports” mean to you?
Summary of the responses:
Six respondents received this question; none had any difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.15

Question 16: Nengrel18 Staff at this school do a good job helping parents understand when their
child needs to learn social and emotional skills.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you arrive at your answer?
In this question, we use the term “parents.” What does that term refer to for you in the
context of being a principal/staff member at school?

Summary of the responses:
Six respondents received this question; none had any difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.16

Question 17: Nengrel24 At this school the staff get along well.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What do you think “get along well” means in this question?
Summary of the responses:
Six respondents received this question; none had any difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.17

Question 18: Nengrel25 At this school there is a feeling of trust among the staff.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Six respondents received this question; none had any difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.18

Question 19: Nengrel26 At this school staff are willing to help each other out.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How did you come up with your response to this question?
Summary of the responses:
Six respondents received this question; none had any difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.19

Question 20: Nengrel27 At this school the staff respect each other.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Six respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.3.20

Question 21: Nengrel29 At this school staff care about students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.21

Question 22: Nengrel30 At this school students get along well with the staff.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.22

Question 23: Nengpar35 My level of involvement in decision making at this school is fine with
me.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What does “involvement in decision making” mean to you?

Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.23

Question 24: Nengpar34 Administrators ask staff to be involved in making decisions.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Who are you thinking about when you think about the term “administrators?”
Summary of the responses:
Six respondents received this question. One respondent said that it depends: On very serious
issues, like the school improvement plan, he would say “yes,” because they have formal and
informal meetings and gatherings; however, on less important matters, like professional
development, his response might be different.
Recommendations:
No change.

A.3.24

Question 25: Nengpar37 Staff at this school have many informal opportunities to influence what
happens within the school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What types of informal opportunities come to mind when you answer this question?
When we say influence in this question, what comes to mind for you?

Summary of the responses:
Six respondents received this question. One respondent did not understand what “informal
opportunities” referred to. One respondent explained that “informal” makes him think more
about off the record or by pure luck or accident (for example, talking in the hall rather than
formal meetings).
Recommendations:
No change.

A.3.25

Question 26: Nengpar38 At this school, students are given the opportunity to take part in
decision making.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What does “take part in decision making” mean to you?

Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.26

Question 27: Nengpar39 Students at this school are encouraged to help solve problems at this
school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me, in your own words, what this question is asking?
Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.27

Question 28: Nengpar41 This school’s administration invites students to share their ideas about
the school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What types of “ideas” does the administration invite students to share?
When you think of the “administration,” what comes to mind? Whom does that include?

Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.28

Question 29: Nengpar43 Staff at this school make it easy for students to suggest activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What types of activities do students suggest? ?
When you think of the staff, what comes to mind? Whom does that include?

Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.29

Question 30: Nengpar44 This school provides students with opportunities to take a lead role in
organizing programs and activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



How did you arrive at your answer?
Tell me more about why you chose [response].

Summary of the responses:
Six respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.30

Question 31: Nengpar47 Students are encouraged to get involved in extra-curricular activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How did you arrive at your answer? Tell me more about why you chose [response].
Summary of the responses:
Six respondents received this question. One respondent is a staff member at a school that
requires students to be involved in extracurricular activities; therefore, for him this question’s
response option would be “not applicable.”
Recommendations:
No change.

A.3.31

Question 32: Nsafemo48 This school is an emotionally safe place for students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In this question, we use the term “emotionally safe.” What does this term mean in your
own words?
Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.32

Question 33: Nsafemo49 Students get along well with each other.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What does the term “get along well” mean, in your own words?

Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.33

Question 34: Nsafemo50 This school is an emotionally safe place for staff.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In this question, we use the term “emotionally safe.” What does this term mean, in your
own words?
Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.34

Question 35: Nsafemo51 I feel like I belong.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:





What does it mean for you to “belong?”
What kind of references does the term “belong” bring up for you?
Tell me more about your answer.

Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.35

Question 36: Nsafemo52 I feel satisfied with the recognition I get for doing a good job.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



You answered [answer]; tell me more about your answer.
What does it mean for you to be recognized?

Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.36

Question 37: Nsafemo53 I feel comfortable discussing feelings, worries, and frustrations with
my supervisor.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In your own words, what is this question asking you?
Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.37

Question 38: Nsafemo54 This school inspires me to do the very best at my job.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In your own words, what is this question asking you?
Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.38

Question 39: Nsafemo55 People at this school care about me as a person.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In your own words, what is this question asking you?
Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.39

Question 40: Nsafemo147 I can effectively work with defiant or disruptive students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



To you, what does the term “defiant or disruptive students” mean?
Can you tell me more about how you work with them effectively?

Summary of the responses:
Four respondents received this question; none had issues answering it or finding a response that
matched their experience.
Recommendations:
No change.

A.3.40

Question 41: Nsafemo148 I can manage almost any student behavior problem.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about managing student behavior problems.
Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.41

Question 42: Nsafbul69 Staff at this school communicate to students that bullying is
unacceptable.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “bullying” mean to you in this question?
What kind of communication were you thinking of?

Summary of the responses:
Three respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.42

Question 43. Nsafbul72 This school provides bullying prevention.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In your own words, what is this question asking you?
How were you thinking this school provides prevention?

Summary of the responses:
Three respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.43

Question 44: Nsafbul76 Staff at this school are teased or picked on about their race or ethnicity.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Three respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.44

Question 45: Nsafbul77 Staff at this school are teased or picked on about their culture or
religion.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Three respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
Adults working at this school are teased or picked on about their cultural background or religion.

A.3.45

Question 46: Nsafbul78 Staff at this school are teased or picked on about their physical or
mental disability.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Three respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.46

Question 47: Nsafbul79 Staff at this school are teased or picked on about their sexuality.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Three respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.47

Question 48: Nsafsub80 Based on your experience, how much of a problem at this school is
student drug use?
 Not a Problem
 Small Problem
 Somewhat a Problem
 Large Problem
Probe:



What is this question asking, in your own words? What does it mean for students to
engage in drug use?
You answered [response]; can you describe what this means to you?

Summary of the responses:
Ten respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student drug use (e.g., Oxy, Ritalin, marijuana,
ecstasy)?

A.3.48

Question 49: Nsafsub81 Based on your experience, how much of a problem at this school is
student use of tobacco (such as cigarettes, chew, cigars)?
 Not a Problem
 Small Problem
 Somewhat a Problem
 Large Problem
Probe:




You answered [response]; can you describe what this means to you?
What does it mean that students use tobacco?
Can you tell me about the things you took into account for this question?

Summary of the responses:
Eleven respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student use of tobacco (e.g., cigarettes, chew, cigars,
electronic cigarettes)?

A.3.49

Question 50: Nsafsub82 Based on your experience, how much of a problem at this school is
student alcohol use?
 Not a Problem
 Small Problem
 Somewhat a Problem
 Large Problem
Probe:




You answered [response]; can you describe what this means to you?
What does it mean that students use alcohol?
Can you tell me about the things you took into account for this question?

Summary of the responses:
Eleven respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student alcohol use (e.g., beer, wine, liquor)?

A.3.50

Question 51: Nsafsub87 Substance abuse prevention is an important goal at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me about how you arrived at your answer?
Summary of the responses:
Ten respondents received this question. Two respondents thought the question was asking
whether the school offered substance abuse services to those who needed them. Another
respondent thought the question was asking whether the school offered information about
substance abuse services, and another thought it was asking whether health classes and
counselors work with students to understand substance abuse.
Recommendations:
Wording change: This school has programs, resources, and/or policies to prevent substance
abuse.

A.3.51

Question 52: Nsafsub88 This school has programs that address substance use among students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “substance use” meant to you?
What kind of programs are you thinking about?

Summary of the responses:
Ten respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.52

Question 53: Nsafsub84 This school has adequate resources to address substance use
prevention.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



In your own words, what is this question asking you?
What “adequate resources” were you thinking of?

Summary of the responses:
Twelve respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
This school has programs, resources, and/or policies to prevent substance abuse.

A.3.53

Question 54: Nsafsub86 At this school, first-time violations of alcohol or other drug policies are
punished by at least an out-of-school suspension.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: In this question, we use the term “suspension.” Can you tell me what this means for you?
Summary of the responses:
Ten respondents received this question. One respondent spoke of cigarette smoking on school
property, one respondent referred to bringing guns to school, and one respondent answered
“disagree” because she believed the school had no alcohol or drug problems.
Recommendations:
Formatting change (bolding the phrase, as shown):
At this school, first-time violations of alcohol or other drug policies are punished by at least an
out-of-school suspension.

A.3.54

Question 55: Nsafsub85 This school provides effective confidential support and referral services
for students needing help because of substance abuse.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe






When you hear the term “substance abuse,” what does that refer to?
What kind of support are you thinking about?
What does “confidential” mean to you?
What does “confidential” mean in terms of your school?
What about “referral services?”

Summary of the responses:
Eleven respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.55

Question 56: Nsafsub83 This school collaborates well with community organizations to help
address youth substance use problems.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:





What kind of community organization are you thinking about?
When you think of collaboration, what does that mean to you?
What are you thinking about when you read “substance use?
[IF RESPONDENT IS NOT A COUNSELOR OR PRINCIPAL]: How did you decide
how to answer this question?

Summary of the responses:
Ten respondents received this question. One respondent interpreted community organizations as
everything that was not school related: church or glee club, summer job or winter job. Another
respondent described DARE and the police as community organizations.
Recommendations:
No change.

A.3.56

Question 57: Nsaferm91 This school has a written plan that clearly describes procedures to be
performed in natural disasters (e.g., earthquakes or tornadoes).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



When you answered this question, what were you thinking about?
What emergencies were you thinking of?

Summary of the responses:
Eight respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.57

Question 58: Nsaferm90 This school has a written plan that describes procedures to be
performed in shootings.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me what you were thinking as you read and answered this question.
Summary of the responses:
Eight respondents received this question. One respondent interpreted the question as asking if
there had been a shooting at the school they currently work at.
Recommendations:
No change.

A.3.58

Question 59: Nsaferm89 I know what to do if there is an emergency, natural disaster (tornado,
flood) or a dangerous situation (e.g., violent person on campus) during the school day.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What is this question asking, in your own words?
How do you know what to do?
You answered [response]; how did you choose this response?

Summary of the responses:
Nine respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.59

Question 60: Nsaferm92 This school or school district provides effective training in safety
procedures to staff (e.g., how to handle emergencies).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: You answered [response]; can you explain/tell me more about your answer?
Summary of the responses:
Eight respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.60

Question 61: Nenvpenv102 This school has a bright and pleasant appearance.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me about how you came up with your answer to this question.
Summary of the responses:
Eight respondents received this question, and none had trouble understanding it.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
This school looks clean and pleasant.

A.3.61

Question 62: Nenvpenv103 This school is an inviting work environment.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.62

Question 63: Nenvpenv104 My workspace at this school is comfortable.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me about how you came up with your answer to this question. What were you
thinking of when you read “this school”? How about “my workspace”?
Summary of the responses:
Eight respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.3.63

Question 64: Nenvpenv96 Overcrowding is a problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.64

Question 65: Nenvpenv97 My work is hindered by poor heating, cooling, and/or lighting
systems at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight respondents received this question. One respondent had trouble understanding the word
“hindered.”
Recommendations:
No change.

A.3.65

Question 66: Nenvpenv98 My work is hindered by insufficient workspace at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.66

Question 67: Nenvpenv99 My work is hindered by a lack of materials and basic supplies at this
school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.67

Question 68: Nenvpenv100 My work is hindered by inadequate or outdated equipment or
facilities at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Eight respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.68

Question 69: Nenvins140 Staff at this school feel that it is a part of their job to prepare students
to succeed in college.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Can you tell me what this question means to you?
Summary of the responses:
Seven respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.69

Question 70: Nenvins110 Staff at this school feel responsible when students at this school fail.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How did you come up with your response to this question?
Summary of the responses:
Seven respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.70

Question 71: Nenvins141 Staff at this school expect students to do their best all the time.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about your answer to this question.
What were you thinking “all the time” meant when you were considering how to answer?

Summary of the responses:
Seven respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.71

Question 72: Nenvins109 Staff at this school feel responsible to help each other do their best.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How did you come up with your response to this question?
Summary of the responses:
Seven respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.3.72

Question 73: Nenvins111 The programs and resources at this school are adequate to support
students with special needs or disabilities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “adequate” mean to you in this question?
What “needs” are you thinking about while answering this question?

Summary of the responses:
Seven respondents received this question, and none had difficulty answering it.
Recommendations:
No change.

A.3.73

Question 74: Nenvins107 Once we start a new program at this school, we follow up to make
sure that it’s working.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “follow up” mean to you in this question?
How does this school follow up?

Summary of the responses:
Seven respondents received this question. One respondent interpreted “follow up” to mean that
once a program has started, it continues until the program is completed.
Recommendations:
No change.

A.3.74

Question 75: Nenvphea118 This school places a priority on students’ health needs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



You said [answer]. Tell me more about why you chose [answer].
How can you tell if your school places a “priority” on this issue?

Summary of the responses:
Five respondents received this question; none had issues answering it or finding a response that
matched their experience.
Recommendations:
No change.

A.3.75

Question 76: Nenvphea117 This school places a priority on making healthy food choices.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



You said [answer]. Tell me more about why you chose [answer].
How can you tell if your school places a “priority” on this issue?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.76

Question 77: Nenvphea119 This school places a priority on students’ physical activity.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



You said [answer]. Tell me more about why you chose [answer].
How can you tell if your school places a “priority” on this issue?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.77

Question 78: Nenvphea116 This school provides quality physical health and nutrition services
for students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: There is no probe for this item.
Summary of the responses:
Five respondents received this question. One respondent misinterpreted the word “quality” as
“quantity.” Another respondent interpreted the question to be asking about her own experience
with health and nutrition services and described the services she had access to.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
This school provides quality physical health and nutrition services (e.g., school nurse, free and
reduced price meals, before and after school meals) for students.

A.3.78

Question 79: Nenvphea115 This school provides the materials, resources, and training necessary
for me to support students’ physical health and nutrition.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me what you think this question is asking you?
Were you thinking about the “materials, resources, and training” provided specifically for
you?

Summary of the responses:
Four respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.79

Question 80: Nenvmen125 This school places a priority on teaching students strategies to
manage their stress levels.1
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about why you chose [answer].
Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

1

In addition to non-instructional staff, four instructional staff received this question. Please refer to the write up for
question Ienvmen128.

A.3.80

Question 81: Nenvmen122 This school places a priority on addressing students’ mental health
needs.2
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about why you chose [answer].
Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

2

In addition to noninstructional staff, four instructional staff received this question. Please refer to write up for
question Ienvmen126

A.3.81

Question 82: Nenvmen123 This school places a priority on social and emotional development.3
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does the term “social and emotional development” mean to you?
Tell me more about why you chose [answer].

Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

3

In addition to noninstructional staff four instructional staff received this question. Please refer to the write up for
question Ienvmen127

A.3.82

Question 83: Nenvmen127 This school provides quality counseling or other services to help
students with social or emotional needs.4
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does the term “quality counseling or other services” mean for you?
Can you tell me more about your answer?

Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

4

In addition to noninstructional staff four instructional staff received this question. Please refer to write up for
question Ienvmen123

A.3.83

Question 84: Nenvmen126 This school provides the materials, resources, and training necessary
for me to support students’ social or emotional needs. 5
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me what you think this question is asking you?
Were you thinking about the “materials, resources, and training” provided specifically for
you?

Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

5

In addition to noninstructional staff four instructional staff received this question. Please refer to write up for
question Ienvmen125

A.3.84

Question 85: Nenvdis134 Discipline problems are handled fairly at this school.6
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What do you think this question is asking?



What does “discipline problems” mean to you?
Can you tell me, in your own words, how you describe “fairly”?

Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

6

In addition to noninstructional staff four instructional staff received this question. Please refer to write up for
question Ienvdis134

A.3.85

Question 86: Nenvdis136 This school effectively handles student discipline and behavior
problems.7
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



When this question refers to “this school,” who or what comes to mind?
What does “effectively handles” mean to you?

Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

7

In addition to noninstructional staff four instructional staff received this question. Please refer to write up for
question Ienvdis135

A.3.86

Question 87: Nenvdis137 Staff at this school work together to ensure an orderly environment.8
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: We use the term “orderly environment;” what does this term mean for you in the context
of your job at the school?
Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

8

In addition to noninstructional staff three instructional staff received this question. Please refer to write up for
question Ienvdis136.

A.3.87

Question 88: Nenvdis133 Staff at this school assign consequences that help students learn from
their behavior.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
What kind of “consequences” were you thinking of?

Summary of the responses:
Two respondents received this question; neither had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.88

Question 89: NPengrel10 Staff at this school teach students strategies to manage emotions.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Can you tell me, in your own words, what this question is asking?
You answered [response]; what did you think about to come up with your answer?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.89

Question 90: NPengrel11 Staff at this school let students know how they should behave when
dealing with each other.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: When answering this question, what are you thinking about to help you come up with
your answer?
Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.90

Question 91: NPengrel12 Staff at this school teach positive conflict resolution strategies to
students.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “positive conflict resolution strategies” mean to you?
Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.91

Question 92: NPengrel14 Staff at this school help students see what is the right thing to do and
help them understand it.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What are you thinking about to help you come up with your answer?
Summary of the responses:
Five respondents received this question; none had issues answering it or finding a response that
matched their experience.
Recommendations:
No change.

A.3.92

Question 93: NPengrel9 Staff at this school regularly give students individualized attention and
help.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How did you arrive at your answer?



In this question, we use the term “staff.” What does that term refer to for you in the
context of being the principal of a school?
What do you think “individualized attention and help” means in this context?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.93

Question 94: NPengpar31 Staff at this school give students opportunities to get involved in
service projects or activities.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



You said [answer]. Can you tell me more about that?
Are you thinking of specific activities?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.94

Question 95: NPengpar32 I ask students about their ideas.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:


I see you answered [answer]. Can you tell me about this?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.95

Question 96: NPsaferm93 This school notifies parents in case of a school-wide emergency.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What is this question asking, in your own words?
You answered [response]; how did you choose this response?
What kinds of things are “school-wide emergencies?”

Summary of the responses:
Five respondents received this question, none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.96

Question 97: NPsaferm94 This school provides a structured anonymous threat reporting system
(e.g., online submission, telephone hotline, or written submission via drop box).
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



When you answered this question, what were you thinking about?
What do “structured” and “anonymous” mean to you in this question?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.97

Question 98: NPsaferm95 This school has agreements with various agencies and services to aid
timely communication.
 Strongly Agree
 Agree
 Disagree


Strongly Disagree

Probe: Can you tell me about how you arrived at your answer?



What does “timely” mean to you in this question?
What kind of “agreements” were you thinking of?

Summary of the responses:
Five respondents received this question. One respondent had trouble with the word “timely.”
Two other respondents had trouble with the phrase “agencies and services” and felt they could
not identify the appropriate agencies that provided these services and thus declined to answer the
question.
Recommendations:
Wording change: This school has agreements with various agencies and services (e.g., local
police and fire departments, community organizations) to aid timely communication.

A.3.98

Question 99: NPenvpenv105 This school’s facilities are adequate for meeting the needs of the
children at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What does “adequate” mean to you in this question?
What “needs” are you thinking about while answering this question?

Summary of the responses:
Five respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.3.99

Question 100: NPenvpenv106 The following are a problem in the neighborhood where this
school is located: garbage, litter, or broken glass in the street or road, on the sidewalks, or in
yards.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: This question refers to “problems”; can you tell me about what you understand the
question is asking you?
Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.100

Question 101: NPenvins112 Poor student attendance is a problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: How do you define “poor attendance” being a problem?
Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.101

Question 102: NPenvins113 This school has a formal process to obtain parental input on
policies related to curriculum and instructional programs.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



What do you consider a “formal process?”
What are ways that “parental input” is obtained?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.102

Question 103: NPenvphea120 Poor student nutrition is a problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:



Tell me more about why you chose [answer].
What does being a “problem” mean to you in this question?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.103

Question 104: NPenvphea121 Lack of student physical activity is a problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: Tell me more about why you chose [answer].



What does being a “problem” mean to you in this question?
What does “physical activity” mean to you?

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.104

Question 105: NPenvmen128 Student depression or other mental health problems are a problem
at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What does “student depression or other mental health problems” mean to you?
Summary of the responses:
Five respondents received this question. One respondent did not provide an answer.
Recommendations:
No change.

A.3.105

Question 106: NPenvmen129 Lack of student social-emotional skills is a problem at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What do you think this question is asking about?
Summary of the responses:
Five respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.3.106

Question 107: NPenvdis138 Order and discipline are maintained satisfactorily at this school.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe: What do you think this question is asking?



What does “order and discipline” mean to you?
Can you tell me, in your own words, how you describe “satisfactorily?”

Summary of the responses:
Five respondents received this question; none had difficulty answering it or finding a response
that matched their experience.
Recommendations:
No change.

A.3.107

Question 108: NPenvdis139 This school has a formal process to obtain parental input on
policies related to school crime and discipline.
 Strongly Agree
 Agree
 Disagree
 Strongly Disagree
Probe:




What do you consider a “formal process?”
What does “school crime” mean to you?
How did you come up with your answer?

Summary of the responses:
Five respondents received this question, and none had trouble understanding it.
Recommendations:
No change.

A.3.108

Appendix A.4 Parent survey - Item by item recommendations
Question 1: Psafsub42 Based on your experience, how much of a problem at this school is
student alcohol use?
o Not a Problem
o Small Problem
o Somewhat a Problem
o Large Problem
Probe:




You answered [answer]; can you describe what this means to you?
What does it mean that students use alcohol?
Can you tell me about the things you took into account for this question?

Summary of the responses:
16 students received this question during cognitive testing. One respondent wanted an unsure or
not applicable answer option; two respondents answered the question generally for all schools
and not specific to their children’s schools; and one respondent didn’t understand which school
the question was referring to.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student alcohol use (e.g., beer, wine, liquor)?

A.4.1

Question 2: Psafsub41 Based on your experience, how much of a problem at this school is
student use of tobacco (cigarettes, chew, cigars)?
o Not a Problem
o Small Problem
o Somewhat a Problem
o Large Problem
Probe:




You answered [answer], can you describe what this means to you?
What does it mean that students use tobacco?
Can you tell me about the things you took into account for this question?

Summary of the responses:
16 parents received this question during cognitive testing. Four parents interpreted the question
as asking if it was ok for students to use or chew tobacco in general, not specific to their child’s
school. One respondent suggested adding a “not applicable” or “unsure” option.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student use of tobacco (e.g., cigarettes, chew, cigars,
electronic cigarettes)?

A.4.2

Question 3: Psafsub40 Based on your experience, how much of a problem at this school is
student drug use?
o Not a Problem
o Small Problem
o Somewhat a Problem
o Large Problem
Probe:




You answered [answer], can you describe what this means to you?
What does it mean for students to use drugs?
What does it mean to you for a student to use drugs, tobacco, or alcohol?

Summary of the responses:
16 parents received this question. One respondent said “small problem” though he didn’t see
anyone using drugs. Another respondent changed her answer from “small problem” to “large
problem” because she was thinking about drugs being a problem in general—not particularly at
her child’s school. Three respondents said couldn’t answer this question because they didn’t
know.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
At this school, how much of a problem is student drug use (e.g., Oxy, Ritalin, marijuana,
ecstasy)?

A.4.3

Question 4: Psafpsaf34 Physical fighting between students is a problem at this school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



What is this question about?
How well does it apply to you or your experience?

Summary of the responses:
18 parents received this question. Four respondents couldn’t answer the question because they
were not sure if physical fighting refers to gang-related fighting or if it includes bullying. One
respondent chose “agree” though she saw a fight in a cafeteria once.
Recommendations:
No change.

A.4.4

Question 5: Psafpsaf33 Gang-related activity is a problem at this school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



What is this question about?
How well does it apply to you or your experience?

Summary of the responses:
18 parents received this question. One respondent felt that the response options were difficult to
interpret. Two respondents answered generally that they “agree” that gangs at schools could be a
problem.
Recommendations:
No change.

A.4.5

Question 6: Psafpsaf32 Racial/ethnic conflict among students is a problem at this school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



What does “racial/ethnic conflict” mean to you?
You said [answer]. Can you tell me more about that?

Summary of the responses:
Seven parents received this question. One parent didn’t seem to understand the question
completely. She didn’t see any problems in the school and said children are more open-minded
in an elementary school and that racial/ethnic conflict evoked ideas of groups such as Latinos,
Whites, Asians or groups who identify based on what they wear (e.g., baggy pants or girls who
wear long sleeved sweaters in summer identifies a person into a group).
Recommendations:
No change.

A.4.6

Question 7: Psaferm47 This school has made it clear to my child what he/she should do if there
is an emergency, natural disaster (tornado, flood) or a dangerous situation (e.g. violent person on
campus) during the school day.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t Know
Probe:
 What are you thinking of to help you determine if your child has a clear understating of
what to do in an emergency?
 How easy or hard was it to choose an answer?
 If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.
Summary of the responses:
15 parents received this question and none had any problems understanding it.
Recommendations:
No change.

A.4.7

Question 8: Psaferm45 This school takes effective measures to ensure the safety of students.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:



What does “effective measures” mean to you?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the responses:
16 parents received this question. One parent didn’t answer the question and just defined what
“effective measures” means to him and spoke about how schools and no one are perfect.
Recommendations:
No change.

A.4.8

Question 9: Psaferm44 This school notifies parents effectively in the case of a school-wide
emergency.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:





In this question we use the term “notify effectively.”
Can you tell me what that means to you?
What do you think “school-wide emergency” means in this question?
You said [answer]. Can you tell me more about that?

Summary of the responses:
16 parents received this question. One respondent requested a “not applicable” option because a
school-wide emergency never happened to him. Two respondents answered “agree” but their
probing suggests they should’ve marked “disagree” because the school did not notify them
effectively in the past (possible problem with the scale).
Recommendations:
No change.

A.4.9

Question 10: Psafemo29 Staff at this school care about what families think.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




In this question we refer to “staff.”
Who did you include in your answer?
How about “families?” Who does that include?

Summary of the responses:
19 parents received this question. 19 parents had no problems with this question.
Recommendations:
No change.

A.4.10

Question 11: Psafemo28 This school is a friendly place overall.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



Can you tell me in your own words what that question was asking?
Tell me more about your answer.

Summary of the responses:
19 parents received this question. One respondent requested specificity for “this school”—staff,
students, both, actual place (this concern is something to consider as a couple of respondents
answered this question based on how student behave and treat their kids while others based it on
staff’s presence and the building).
Recommendations:
No change.

A.4.11

Question 12: Psafemo27 At this school, the staff really cares about my child.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




What does “really cares about my child” mean to you?
Who are you thinking about when you think about “the staff?”
What are you thinking about when trying to decide if the staff cares about your child?

Summary of the responses:
19 parents received this question. Two parents seem to interpret this question as “how much
care” the child receives instead of whether or not the staff at the school really cares about their
child.
Recommendations:
No change.

A.4.12

Question 13: Psafbul39 This school has helped me be more aware of bullying and cyberbullying
of students.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




Tell me more about what you are thinking when coming up with your answer.
In what ways has the school helped you to become more aware of bullying?
What does “cyberbullying” mean to you?

Summary of the response:
18 parents received this question. One respondent didn’t answer the question and he spoke about
how teachers complain about students not paying attention or bothering other students. Another
respondent didn’t see the difference between cyberbullying, social networks, and emails. One
chose “agree” though she never heard anything from the school regarding bullying. Another
respondent said the school didn’t make him “more aware” because he already knew about this.
Recommendations:
No change.

A.4.13

Question 14: Psafbul37 Bullying of students via electronic means or devices is a problem at this
school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



Tell me what you are thinking when I ask about harassment or bullying via electronic
means or devices.
Tell me more about your opinions on this type of harassment or bullying at school.

Summary of the response:
18 parents received this question. One respondent disagreed that children should have cellphones
(not understanding the question).
One respondent wanted a “don’t know” or “not applicable” answer because this question does
not apply to younger grades (1st and 2nd grade). Two respondents were unsure because they
hadn’t seen it nor heard about it. Two respondents answered the question generally, not
particular to their children’s schools. One said “agree” that bullying is wrong and teachers need
to do more about it in general. One respondent had trouble understanding the response options.
Recommendations:
Wording change: Bullying of students via electronic means or devices is a problem at this school
(cyberbullying).

A.4.14

Question 15: Psafbul36 Bullying of students at school or school activities is a problem at this
school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



Tell me what you are thinking when I ask about harassment or bullying.
Tell me more about your opinions on harassment and bullying at school.

Summary of the response:
18 parents received this question. One respondent thought about their feelings on bullying in
general, not specific to their child’s school. One respondent disagreed that bullying was a
problem though his son was a victim of bullying; the respondent said “it wasn’t that bad”. One
respondent didn’t provide a response and said his son never had bullying problems.
Recommendations:
No change.

A.4.15

Question 16: Penvphea56 This school provides quality physical health and nutrition services for
students.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:




Did you find a response category that fit your experience?
Tell me more about your answer.
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
13 parents received this question. Five respondents had trouble understanding “physical health
and nutrition services”. One respondent thought this question should be broken into two
questions.
One respondent answered generally that she “agreed” that all schools should provide quality
services for students.
Recommendations:
Wording change: This school provides quality physical health and nutrition services (e.g., school
nurse, free and reduced price meals, before and after school meals) for students.

A.4.16

Question 17: Penvphea55 Students have enough healthy food choices at this school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:




What do you consider “healthy food choices?”
You answered [response]; can you tell me more about that response?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
13 parents received this question. One parent generally didn’t agree with the foods being served
at schools and remarked there is an extensive list of options for healthy choices. One respondent
said she “agrees” that all schools should have healthy food choices.
Recommendations:
No change.

A.4.17

Question 18: Penvpenv49 The school building is clean and well-maintained.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




Can you tell me in your own words what that question was asking?
What are you thinking about in order to come up with your answer.
What parts of the school building are you thinking about when coming up with your
answer?

Summary of the response:
14 parents received this question. One respondent didn’t provide an answer but confirmed that
the school was clean during probing. A few respondents commented on how this question item is
similar to a previous question item (most likely Penvpenv48).
Recommendations:
No change.

A.4.18

Question 19: Penvpenv48 This school has a bright and pleasant appearance.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



What does “bright and pleasant appearance” mean to you?
What parts of the school are you thinking about when coming up with your answer?

Summary of the response:
14 parents received this question. 14 parents had no problems with this question.
Recommendations:
No changes were recommended based on cognitive interview data. However, for consistency
purposes, the following wording change will be made to the item:
This school looks clean and pleasant.

A.4.19

Question 20: Penvmen58 This school has enough programs that address students' emotional and
social development.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:




What does the term “emotional and social development” mean to you? Tell me more
about why you chose [answer].
What programs were you thinking of?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
12 parents received this question. One respondent didn’t know how to answer this question—her
child doesn’t require these programs; and the school doesn’t offer assistance to help students
with problems such as depression; the school only offers chapel services and the open door
policy. Another respondent struggles to answer this question; thought it was similar to a previous
question. One respondent didn’t know how to answer this question; she struggled with the term
“development”; she requested examples. Another respondent struggles to explain emotional and
social development during probing. One respondent was puzzled by the term “programs” and
didn’t know what that meant. One respondent didn’t notice the “don’t know” option response.
Recommendations:
Wording change: This school has enough programs for students (e.g., mentoring programs,
group problem solving, team building) that address emotional and social development.

A.4.20

Question 21: Penvmen57 This school provides high quality services to help students with social
or emotional needs.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:






What does the term “high quality services” mean for you?
What or who comes to mind when you read “this school” in this question?
Did you find a response category that fit your experience?
Can you tell me more about your answer?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
13 parents received this question. One respondent didn’t understand the question and solely
discussed his children and what “high quality services” could entail in general, not specific to
services that help students with social and emotional needs). One respondent was confused about
what “this school” referred to.
Recommendations:
No change.

A.4.21

Question 22: Penvins54 The programs and resources at this school are adequate to support
students with special needs or disabilities.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



How easy or hard is it for you to answer this question?
Where you able to find a response that matches your experience?

Summary of the response:
13 parents received this question. One respondent lost focus and answered generally about the
subject, not specific to her child’s school; she agreed that every school should have these
programs and resources. Three respondents would’ve benefited from a “Don’t know” option.
One of these three respondents conjured up an answer (“disagree”) because the school told her
that her child had to be focused at one time (she had trouble recalling this story) and she also
mentioned talking to the school about her child’s depression which she interpreted as a special
need or disability. The two other respondents said “agree” and “strongly agree” but they really
didn’t know anything about, never heard, never assessed, and never used these types of services
for their child.
Recommendations:
No change.

A.4.22

Question 23: Penvins53 My child's teachers make themselves available to me.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



What does “make themselves available” mean to you?
Which teachers were you thinking about when you answered this question?

Summary of the response:
13 parents received this question. 13 parents had no problems with this question.
Recommendations:
No change.

A.4.23

Question 24: Penvins52 This school sees me as a partner in my child's education.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



In your own words, what is this question asking?
What kinds of things are you including in this answer?

Summary of the response:
13 parents received this question. One respondent confused the question to be asking about her
child instead of her and changed her answer from “agree” to “strongly agree” because she does
all that she can do to get what her child needs. This respondent suggested changing the question
to “Do I, the parent, do what I need to help my child?” (change “partner” to “parent”).
Recommendations:
No change.

A.4.24

Question 25: Penvins51 This school has high expectations for student learning.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



Can you tell me what “high expectations” are in your own words?
When thinking about high expectations are you thinking of something the school has
defined as a high expectation, or is it based on something you think is a high expectation?

Summary of the response:
13 parents received this question. One respondent seemed to disregard “for student learning” in
the question and focused on “high expectations”. Another respondent seems to have interpreted
the question as the respondent having the high expectations of the school (like the school has
high expectations of itself). One respondent thought “high expectations” meant students
achieving knowledge but also growing up as individuals in society. Another respondent didn’t
read the entire question and responded after reading “high expectations” and spoke generally
about how “high expectations” means to do well and exceed. One respondent didn’t understand
“expectations” and answered “strongly agree” because her son gets good grades. Another
respondent doesn’t talk to her daughter’s teachers much so she doesn’t know what the
expectations are; she doesn’t know how the school defines “high expectations”. One respondent
interpreted this question as in what the teachers have to offer instead of about what the children
are being expected to do; she said she agrees because all of her son’s teachers are great and push
the students to thrive. To another respondent, “high expectations” meant doing homework. For
one respondent, “high expectations” is when schools help develop good character that will help
the child stay on a good path.
Recommendations:
Wording change: This school has high expectations for students.

A.4.25

Question 26: Penvins50 Attending school every day is important for my child to do well in
his/her classes.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe: Tell me what you were thinking you read and answered this question.
Summary of the response:
13 parents received this question. 13 parents had no problems with this question.
Recommendations:
No change

A.4.26

Question 27: Penvdis61 Discipline problems are handled fairly at this school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




What do you think this question is asking?
What does “discipline problems” mean to you?
Can you tell me in your own words what it means for problems to be “handled fairly?”

Summary of the response:
Nine parents received this question. One respondent requested a “Don’t know” option because
she never had to deal with the subject; this same respondent said “agree” even though during
probing she mentioned how her daughter once got into trouble for talking too much which
wasn’t fair because her daughter said she didn’t do it. The other respondent was unsure about
from whose perspective it was fair and offered some examples: 1) If a student hits a student who
hits back, then both students get in trouble which is presented as “fair”, but 2) if a few students
talk in the classroom and the whole class gets punished and has detention, then that is “unfair” to
all other students. This respondent “disagreed” to this question because he was thinking in terms
of the students.
Recommendations:
Add a “Don’t Know” option.
Split into two different items:
Question 1: School rules are applied equally.
Question 2: The punishment for breaking school rules is fair.

A.4.27

Question 28: Penvdis60 This school communicates school policies and procedures clearly to
parents.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




Can you tell me what this question is asking in your own words?
What are you thinking about to help you come up with your answer?
How easy or hard is it to come up with your answer?

Summary of the response:
Nine parents received this question. One respondent didn’t answer the question but the probing
indicates that he understood the question and would’ve picked “disagree” or “strongly disagree”.
Recommendations:
No change.

A.4.28

Question 29: Penvdis59 When my child does something good at school, I usually hear about it
from the school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




What does doing “something good” mean to you?
What ways do you hear from the school?
Tell me more about why you chose [answer].

Summary of the response:
11 parents received this question. One respondent was puzzled by the phrase “something good”
and needed clarification on what behavior to focus on (e.g., good grades or helping others in
class) but his answer “agree” matched the fact that he thought the teacher was very good at this
(but not the school as a whole).
Recommendations:
No change

A.4.29

Question 30: Pengrel25 This school promptly responds to my phone calls, messages, or e-mails.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:





Have you had occasion to call or email this school, or leave a message?
o If not, how did you decide how to answer this question?
What do you consider a prompt response?
Think about the term “this school” in this question, are you thinking of an individual
person or the school in general?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
19 parents received this question. 19 parents had no problems with this question.
Recommendations:
No change.

A.4.30

Question 31: Pengrel23 This school has quality programs for my child's talents, gifts, or special
needs.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:





What kind of programs are you thinking about here?
Can you tell me about the programs that your child is involved in?
What does “quality programs” mean to you?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
19 parents received this question. One respondent refused to answer the question because she
didn’t know about any of these types of programs.
Recommendations:
No change.

A.4.31

Question 32: Pengrel16 I feel comfortable talking to someone at this school about my child's
behavior.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:





Can you tell me in your own words what that question was asking?
What does “someone at this school” mean to you?
What kinds of behaviors or things did you think about when you answered this question?
You said [answer]. Can you tell me more about that?

Summary of the response:
19 parents received this question. One respondent answered this question too broadly and spoke
about how one can identify with certain types of people who have the same values as you.
Recommendations:
No change.

A.4.32

Question 33: Pengrel15 This school encourages me to be an active partner in educating my
child.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




In your own words, what is this question asking?
Can you tell me about how the school encourages you to participate?
What kinds of things are you including in this answer?

Summary of the response:
19 parents received this question. One respondent didn’t understand the question and interpreted
it as making friends with the parents of students and discussing about the children’s behaviors
and being a good student. For the other respondent, there might’ve been a translation issue where
“encourage” is interpreted as an emotion and not in other forms (like action).
Recommendations:
No change.

A.4.33

Question 34: Pengrel13 I feel welcome at this school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:




Tell me what you were thinking when you read this question, about feeling welcomed?
[If Respondent answered positively] What kinds of things make you feel welcomed at
your child’s school?
[If Respondent answered negatively] Can you tell me more about that your response?

Summary of the response:
19 parents received this question. One respondent wanted a “Neutral” answer option because she
felt her child was welcomed as a new student but the student body was not welcoming so the
respondent found this question difficult to answer.
Recommendations:
No change.

A.4.34

Question 35: Pengrel11 At this school, my child feels he/she belongs.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe: Could you tell me what you think this question is asking?
Summary of the response:
Nine parents received this question. One parent didn’t know; respondent said that he/she didn’t
go to school; that the respondent’s child didn’t’ really like the school
Recommendations:
No change.

A.4.35

Question 36: Pengrel10 This school helps me figure out what social and emotional skills my
child needs to develop.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:



What does “social and emotional skills” mean to you?
Can you explain to me how you came up with your answer?

Summary of the response:
Nine parents received this question. Two respondents didn’t understand “social and emotional
skills”. Another parent had trouble answering this question because she didn’t know if the
question was referring to “direct” or “indirect” help. For example, the school doesn’t directly
help with her child’s social/emotional needs but indirectly, the school gives her child homework
which indirectly helps. One respondent didn’t know how to respond because this situation never
came up. Another respondent never had to use the school’s services for these issues but she
knows a person is available if needed so she said “agree”.
Recommendations:
Wording change: This school helps me figure out what social and emotional skills (i.e.,
knowledge, attitudes, and skills necessary to deal with social and emotional interactions) my
child needs to develop.

A.4.36

Question 37: Pengclc9 This school encourages all students to take challenging courses no matter
their race, ethnicity, nationality, or culture.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
Probe:


The question uses the term “challenging courses.” In your own words, what does this
term mean?
o Tell me more about why you chose the [response].

Summary of the response:
Nine parents received this question. One respondent thought “challenging courses” meant
“classes that students can’t handle that are above the student’s capabilities” so she disagreed
because she didn’t think the school would place a student in a class that they couldn’t handle.
This respondent didn’t mention anything about “no matter their race, ethnicity, nationality, or
culture”. One respondent wanted a “don’t know” option because she didn’t know enough about
the school. Another respondent “strongly agreed” but said that the school does not have
challenging courses but they do encourage the student to try their best. Another respondent
didn’t’ understand “challenging courses” and didn’t understand why some groups would be left
out from participation.
Recommendations:
Wording change: This school encourages students to take challenging courses (e.g., honor level
courses, AP or IB courses, and dual credit courses) no matter their race, ethnicity, nationality,
and/or cultural background.

A.4.37

Question 38: Pengclc6 This school communicates how important it is to respect the beliefs and
practices of all cultures.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:





What methods of communication are you thinking of?
Can you tell me about the methods of communication that the school uses?
You answered [response]; can you tell me more about why you chose this answer?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
Nine parents received this question. One respondent first assumed the question asked if her
child’s beliefs are accepted by the school but later realized her answer wasn’t correct because she
doesn’t know if the school communicates this or not. Two respondents spoke about the school’s
practice of respecting beliefs/practices instead of the school’s communication of it. One
respondent took “all cultures” to mean Catholics vs. Christians (in a religious way). One
respondent assumed that this question is asking about the school communicating to the kids; she
had difficulty answering this question because it’s a delicate subject and she wasn’t sure if the
school had time to talk to students about these issues at their grade level (yet she answered
“agree”). One respondent didn’t know how to answer because the school never speaks on this
topic. Another respondent changed her answer from don’t know to disagree to agree. She was
thrown off by the term “beliefs”; she also wasn’t familiar with the race groups at the school
which made her hesitate and change her answer.
Recommendations:
Wording change: This school communicates how important it is to respect the practices of all
cultures.

A.4.38

Question 39: Pengclc5 This school provides instructional materials that reflect my child's
culture, ethnicity, and identity.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe:





What is this question asking you?
What does “instructional materials” mean to you?
In your own words what does “culture, ethnicity and identity” mean in this question?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the
categories?
o Tell me more about your answer.

Summary of the response:
Nine parents received this question. One respondent understood all of the terms separately but
didn’t understand the question and what it was asking. Another respondent didn’t understand the
question; she focused on the “identity” part of the question and her son’s IEP mentor. Note that
several parents understood this question to mean other’s culture/ethnicity/identity as well as their
child’s.
Recommendations:
Wording change: This school provides instructional materials (e.g., textbooks, handouts,
software applications) that reflect students’ cultural background, ethnicity, and identity.

A.4.39

Question 40: Pdemo64 What is your race?
o White
o Black or African-American
o Asian
o American Indian or Alaska Native
o Native Hawaiian or Pacific Islander
o Other
Probe: There is no probe for this item.
Summary of the response:
21 parents received this question. One respondent commented that these questions are always
confusing and answered “other” by process of elimination. Another respondent was from
Colombia and said “other” because he wasn’t white or black. One respondent chose “Other”
because she didn’t’ find another option that applies to her. One respondent couldn’t find
“Hispanic” option so she chose “other”. Two respondents found this question confusing because
she already said she was “Hispanic” in the previous question. One respondent didn’t understand
why there isn’t a “Hispanic” option. She said, “you’re addressing the white people, you’re
addressing the other cultures, but you don’t address Hispanics, so I’m thinking I don’t like using
the option of White, so I would put other and then write in Hispanic. I think it should be added as
one of the races because there are a large number of Hispanics that live in California.” One
respondent didn’t know if she was “white” or “black” so she chose “white”. Another respondent
remarked that she didn’t see a “check all that apply” option; it was difficult to only pick one.
Two respondents looked for “Mexican” option. One respondent chose “white” based on her skin
color. Another respondent didn’t read “check all that apply”.
Recommendations:
No change.

A.4.40

Question 41: Pdemo63 Are you of Hispanic or Latino origin?
o Yes
o No
Probe: There is no probe for this item.
Summary of the response:
21 parents received this question. Two parents did not like the use of “Hispanic” or “Hispano”.
One respondent did not feel “Hispanic” and identified more with “Latino” and the other
respondent said his/her nationality wasn’t Hispanic.
Recommendations:
No change.

A.4.41

Question 42: Psafpsaf30 My child is safe at this school.
o Strongly Agree
o Agree
o Disagree
o Strongly Disagree
o Don’t know
Probe: What does safe mean to you? Tell me more about your answer?
Summary of the response:
19 parents received this question. One respondent could not find a response category that
matched her experience. She said, “I agree and disagree” because “I don’t think anyone is safe,
no matter where you are.”
Recommendations:
No change.

A.4.42