School Climate Surveys Cognitive Interviews and Usability Testing

NCES Cognitive, Pilot, and Field Test Studies System

Volume 2 School Climate Surveys Cog Labs & Usability Testing

School Climate Surveys Cognitive Interviews and Usability Testing

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [docx]

Download: docx | pdf

Shape1

Table of Contents

Cognitive Laboratory: Protocols for Student Survey 3

Cognitive Laboratory: Protocols for Parent Survey 25

Cognitive Laboratory: Protocols for Instructional Staff Survey 46

Cognitive Laboratory: Protocols for Non-Instructional Staff Survey 68

Cognitive Laboratory: Protocols for Student Survey

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. Also, keep track of the time so that your total interview time with the student lasts 1 hour or less.

Before starting the interview, use this table to confirm the participant info that we received from the recruitment firm. If the respondent indicates that some of the listed information is incorrect, please cross it out and enter the correct information in the relevant cell.

Background Information		Confirmed 
Date & Time
Name of Interviewer
Student First Name
Student ID Number
Student’s Gender	 Male  Female
Student’s Grade
Type of School	 Public  Charter  Private
Name of School

Before starting the interview make sure that you have a signed consent form from the child’s parent or guardian either (1) already in your files, or (2) handed to you by the parent/guardian on the day of the interview.

School Climate Surveys

STUDENT SURVEY
2014

U.S. Department of Education
National Center for Education Statistics

MATERIALS NEEDED FOR INTERVIEW

INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)
RESPONDENT QUESTIONNAIRE
DO YOU HAVE THE CORRECT VERSION OF THE BOOKLET AND QUESTIONNAIRE?
CONSENT FORM (TWO COPIES)
INCENTIVE
PAYMENT RECEIPT
DIGITAL RECORDER AND EXTRA BATTERIES
PENS AND PENCILS

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the student]:

You are here today to help us out with a new survey that we are working on with the U.S. Department of Education. This survey asks about your opinions of the school you currently attend. A survey is just a set of questions about your experiences. This is not a test. I will ask you to take the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. I will not be grading your work. As you are answering the questions, I will ask you about the survey questions. I will also stop you once in a while to ask you questions. This will help us make the survey better and clearer for other students just like you.

You can choose whether or not you want to participate. We have discussed this research with your parent(s)/guardian and they know that we are also asking you for your agreement. If you are going to participate in the research, your parent(s)/guardian also have to agree. But if you do not wish to take part in the research, you do not have to do so, even if your parents have agreed. Because the information you provide is so important, I am going to be taking notes while you are working and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today].

Also, this interview will be recorded so researchers can review the tapes later. Your responses, and any information that refers to you or the name of your school will not be included in our report, and your responses will be used for research purposes only and will not be disclosed, or used for any other purpose except as required by law.

Remember, you are not being tested or graded; we want you to tell us what you think about the survey. Would you like to be a part of this study? [If respondent agrees, continue, if not, end the interview.]

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the student may ask.]

PROVIDE RESPONDENT WITH A COPY OF THE ASSENT FORM. ASK THE RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.

SIGNED ASSENT FORM COLLECTED
COPY OF ASSENT FORM GIVEN TO RESPONDENT

IF THE RESPONDENT HAS ASSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

While you fill out the survey I want you to think out loud about how you are answering some of the questions. Hearing you talk about how you figure out your answers to the questions will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give me. To help you get started on thinking aloud, we are going to do a practice activity. I’m going to ask you a question and ask you to think aloud as you decide on your answer.

How many windows are there in the house or apartment where you live?

Please tell me what you are thinking while you are figuring out your answer.

Shape2

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. While you are answering the questions, please think about the school you currently attend. I’d like you to think aloud while you decide on your answer. Please read aloud anything that you would normally read to yourself if I were not here.

Remember that I am very interested in hearing you talk about what you are thinking about as you figure out your answers to the questions.

Probe bank (from Willis)

Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking?
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are thinking.”
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: “Tell me what just happened. Tell me about what you were just doing.”

Marking Directions

Shape4

PLEASE READ EACH QUESTION CAREFULLY. Here is an example of the type of items you will see in this survey:

I. MARK ONE RESPONSE

1. What is the color of your eyes?

 Brown

 Blue

 Green

 Another color

If the color of your eyes is green, you would mark the circle beside green.

THIS IS THE END OF THE EXAMPLES.

THIS SURVEY IS NOT A TEST.

WE HOPE YOU WILL ANSWER EVERY QUESTION, BUT YOU MAY SKIP ANY QUESTION YOU DO NOT WANT TO ANSWER.

PLEASE GO TO THE NEXT PAGE TO BEGIN THE SURVEY.

Say to respondent: Please answer these questions thinking about the school you went to in the 2013-14 school year. [If respondent is confused due to summer administration, clarify that we want to know about the school the school they attended most recently – the one they attended right before summer break.]

Mark One Response.

1. What grade are you currently in at this school?

 5^th grade

 6^th grade

 7^th grade

 8^th grade

 9^th grade

 10^th grade

 11^th grade

 12^th grade

Mark One Response On Each Line

2.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Teachers and other adults at this school treat all students respectfully.				
	b) There are examples of different racial, ethnic, or cultural backgrounds in the class lessons at this school.				
	c) A teacher or other adult at this school has disrespected students because of their race, ethnicity, or culture.				
	d) People of different cultures, races, or ethnicities do not get along well at this school.				
	e) Students from different backgrounds get along well at this school.				

Note to interviewer: probe after this set of items.

Item-specific probes:

a) Can you tell me what this question is asking you? What does it mean to be treated respectfully? What does “other adults” mean to you in this question?

b) What does class lesson mean in your own words, how did you come up with that answer? Can you give me an example of the type of thing this question is asking about?

c) Tell me what this question is asking you. What do you think it means to be disrespected because of race, ethnicity, or culture?

d) and e) What is this question asking you? What does the term “different cultures” mean? OR when you hear “different cultures” what do you think about?

Note to interviewer: after individually probing a-d stop student and probe overall.

Who were you thinking of when you thought about “students” in these questions—which students?
In these questions we use the phrase “this school,” when you hear this, what do think it’s referring to?

Mark One Response On Each Line.

3.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Teachers and other adults at this school seem to take a real interest in my future.				
	b) It is easy to talk with teachers.				
	c) Teachers at this school help us children with our problems.				
	d) Adults near where I live support this school.				
	e) When there are events at this school, lots of families come.				
	f) If I am absent, there is a teacher or some other adult at school who will miss me.				

Note to interviewer: probe after this set of items.

Item-specific probes:

The question is asking about “teachers and other adults”. When you answered this question, which teachers were you thinking about? Which “other adults” were you thinking about?
What does the word “teacher” mean to you? Who do you consider a teacher (ask for examples if the student is having a hard time telling you who is a teacher). When you were answering this question which teachers were you thinking about? [just at this school or all teachers?]
Can you tell me what this question is asking?
Can you tell me what this question is asking?
When you hear “events at school”, what do you think the question is talking about?
[If the student has a hard time remembering if she is absent] how easy or hard was it to choose an answer? Which teacher or other adult were you thinking of, who would miss you?

Note to interviewer: after individually probing a-f stop student and probe overall.

How easy or hard was it to choose an answer for each of these questions?

Mark One Response On Each Line.

4.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	Students can help make decisions in this school.				
	Students are given the chance to help solve school problems.				
	At this school, the principal asks students what their ideas are.				
	I regularly attend school-sponsored events, such as school dances, sporting events, student performances, or other school activities.				
	At this school, students have lots of chances to help decide things like class activities and rules.				
	There are lots of chances for students at this school to get involved in sports, clubs, and other school activities outside of class.				
	I have lots of chances to be part of class discussions or activities.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Can you tell me, in your own words, what this question is asking? [Note to interviewer: if respondent says they are able to help make decisions] What sorts of decisions are students able to help make?
Can you tell me, in your own words, what this question is asking? [Note to interviewer: if respondent says they are able to help solve problems] What sorts of problems are students able to help solve?

What kind of ideas does the principal ask student about? Who were you thinking about when you thought of the principal? [Note to interviewer: if the R tells you that they are thinking of the principal at the school continue probing to find out if the student understands who the principal is. You can ask: What is the name of your principal? What other kinds of things does the principal do at your school?]
Tell me what you are thinking when you read this question? What comes up for you when you hear school-sponsored events? [Note to interviewer: you may ask for examples if respondent is having trouble describing school-sponsored events]. Can you tell me, in your own words, what this question is asking? You chose [response], tell me more about it.
What kind of activities does this question make you think about? [Note to interviewer: if student states that the activities that he/she is involved in is not listed]. Are there activities you are involved in at school? Tell me about those activities. What about the rules in the question? What rules were you thinking of?
How easy or hard was it to choose an answer?

Mark One Response On Each Line.

5.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Students at this school get along well with each other.				
	b) Students at this school can tell their teachers if they feel confused about something in class.				
	c) Students at this school are sensitive to the feelings of other students.				
	d) At this school, students talk about the importance of understanding their own feelings and the feelings of others.				
	e) At this school, students work on listening to others to understand what they are trying to say.				
	f) I am happy to be at this school.				
	g) I feel close to people at this school.				
	h) I feel like I am part of this school.				
	i) I feel socially accepted.				
	j) I feel loved and wanted.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What does “getting along well” mean to you? How did you choose an answer?
What does it mean to you to be “sensitive to the feelings of other students”?
What is this question asking in your own words? What does it mean to you to “understand your own feelings”?
What is this question asking in your own words? What does it mean to you to “work on listening to others”?

Note to interviewer: after individually probing a-d stop student after item e and probe overall a-e.

Who were you thinking of when you thought about “students” in these questions—which students?

f-j) Note to interviewer: these questions may be sensitive to some students. Use the probes below to address the questions overall. Have the student answer all questions and then probe for all questions.

If the student appears uncomfortable ask: Is this OK to talk about in a survey if your answers are kept private or is it too uncomfortable? In general how do you feel about being asked these questions?
[Note to interviewer: take notice on any non-verbal cues and follow up on those questions that the student paused, or said anything about]. Tell me more about what you are thinking about [item #].

i-j) Tell me more about what this question means to you. [getting at whether they are just thinking of school or their entire life]

Mark One Response On Each Line.

6.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Students at this school are teased or picked on about their race or ethnicity.				
	b) Students at this school are teased or picked on about their culture or religion.				
	c) Students at this school are teased or picked on about their physical or mental disability.				
	d) Students at this school are teased or picked on about their sexuality.				
	e) Students at this school are often cyber bullied (e.g. receiving a threatening or hurtful message from another student in an email, on a website, on a cell phone, or in instant messaging).				
	f) Students often spread mean rumors or lies about others at this school on the internet (i.e., Facebook™, Myspace™, email, and instant message).				

Note to interviewer: these questions may be sensitive to some students. Use the probes below to address the questions overall. Have the student answer all questions and then probe for all questions.

If the student appears uncomfortable ask: is this OK to talk about in a survey if your answers are kept private or is it too uncomfortable? In general how do you feel about being asked these questions?

Note to interviewer: probe after this set of items.

Item-specific probes:

Your answered [answer] to the first question, can you tell me more about your answer? What does “race or ethnicity” mean in this question for you?
What does “culture or religion” mean in this question for you?
What does “disability” mean to you in this question? [If student has trouble you may ask for examples: can you give me some examples of what you mean by physical disability? By mental disability?]
What does the word “sexuality” mean in your own words? Could you give an example of how someone could be teased or picked on about their sexuality? The question uses the word sexuality does that sound ok to you, or would you choose something different?

The question uses the word “cyber bullied,” what does “cyber bullied” mean in your own words? In what ways does cyber bullying or teasing happen? [Note to interviewer: try to find out if the list above includes everything they are thinking about; want to know what we are missing].

What is this question asking in your own words? In what ways are rumors spread? Are there things missing in this list? [Note to interviewer: try to find out if the list above includes everything they are thinking about; want to know what we are missing].

Note to interviewer: after individually probing a-f stop student and probe overall.

How much thought would say you have given to these questions when you were answering?
How well do these questions apply to you?

Mark One Response On Each Line.

7.	How much of a problem are the following at this school:	Not a Problem	Small Problem	Somewhat a Problem	Large Problem
	a) Based on your experience, how much of a problem at this school is: Students' drug use (such as marijuana, LSD, cocaine, ecstasy).				
	b) Based on your experience, how much of a problem at this school is: Students' use of tobacco (such as cigarettes, chew, cigars).				
	c) Based on your experience, how much of a problem at this school is: Students' alcohol use (such as beer, wine, liquor).				

Note to interviewer: these questions may be sensitive to some students. Use the probes below to address the questions overall. Have the student answer all questions and then probe for all questions.

If the student appears uncomfortable ask: is this OK to talk about in a survey if your answers are kept private or is it too uncomfortable? In general how do you feel about being asked these questions?

Note to interviewer: probe after this set of items.

Item-specific probes:

Tell me more about what you are thinking about [question #]. (a) What does “drug use” mean to you in this question? [If student has trouble you may ask for examples: can you give me some examples of what you mean by drug use?]

What does “tobacco” mean to you? When you hear the word tobacco what do you think this question is referring to? [If student has trouble you may ask for examples: can you give me some examples of what you mean by tobacco?]

Tell me more about what you are thinking about [question #]. What does “alcohol” use mean to you in this question? [If student has trouble you may ask for examples: can you give me some examples of what you mean by alcohol use?]

Note to interviewer: after individually probing a-c stop student and probe overall.

These questions ask you to respond “based on your experience”. What does this mean to you? How did you come up with your answer to these questions?

Mark One Response On Each Line.

8.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Students at this school think it is okay to smoke one or more packs of cigarettes a day.				
	b) Students at this school think it is okay to have one or two drinks of any alcoholic beverage nearly every day.				
	c) Students at this school think it is okay to try drugs once or twice.				
	d) Students at this school think it is okay to use drugs once a month or more.				

If the student appears uncomfortable ask: is this OK to talk about in a survey if your answers are kept private or is it too uncomfortable? In general how do you feel about being asked these questions?

Note to interviewer: probe after this set of items.

Item-specific probes:

In this question it refers to “smoke one or more packs of cigarettes.” Can you tell me what you think a “pack of cigarettes” is? How about cigarettes? What kinds of things do you include in cigarettes?
In this question they use the term “alcoholic beverage” what does this term mean to you? To you what are some types of alcoholic beverages?
In your own words, what is this question asking? What does “once or twice” mean for you?
In your own words, what is this question asking?

Mark One Response On Each Line.

9.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school has told students what to do if there is an emergency, natural disaster (tornado, flood) or a dangerous situation (e.g. violent person on campus) during the school day.				
	b) Students know what to do if there is an emergency, natural disaster (tornado, flood) or a dangerous situation (e.g. violent person on campus) during the school day.				
	c) If students hear about a threat to school or student safety, they have a way to report it to someone in authority without giving their name.				

Note to interviewer: probe after this set of items.

Item-specific probes:

When you answered this question, who were you thinking about? Who has told you about what to do during an emergency?
What is this question asking, in your own words? You answered [response], how did you choose this response?

a-b) What does a “violent” person refer to in these questions? Tell me what you were thinking when you read and answered this question? When you think of a person in authority, who is that person?

Mark One Response On Each Line.

10.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) The school buildings are pleasant and well maintained.				
	b) The bathrooms in this school are clean.				
	c) This school is clean and well-maintained.				
	d) The temperature in this school is comfortable all year round.				
	e) This school has a bright and pleasant appearance.				
	f) Overcrowding is a problem at this school.				
	g) Broken things at this school get fixed quickly.				

Note to interviewer: stop student after this set of items and probe overall.

When answering these questions, how did you come up with the answers?
Are there any questions that were hard to answer?
In this list of questions we use the word “overcrowding.” In your words, what does this word mean to you? How about “well-maintained”?

Mark One Response On Each Line

11.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Other students often disrupt class.				
	b) I get distracted from doing schoolwork in my classes because other students are misbehaving, for example, talking or fighting.				
	c) My teachers give me individual attention when I need it.				
	d) The schoolwork I am assigned is meaningful and important.				
	e) Classes at this school really make me think.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Tell me more about your answer to this question? [Note to interviewer: trying to understand how the student understands disruptions].

What is this question asking you in your own words? Are you thinking of any particular distractions? [Note to interviewer: here you are trying to determine if the student is reading the whole question, interpreting the question to include only talking and/or fighting].
When you answer this question and think about teachers, who are thinking about?
When you think about schoolwork that is assigned, what does this mean to you?
What is this question asking you in your own words?

Mark One Response On Each Line.

12.	How often do the following occur?	Never	Rarely	Sometimes	Often
	a) How often do you eat fruit at school?				
	b) How often do you eat vegetables at school?				
	c) How often do you eat breakfast on school days?				
	d) How often do you eat candy at school?				
	e) How often do you drink soda at school?				
	f) How often do you go to gym class or participate in other physical activity during the school day?				
	g) How often do you stay after school to participate in sports or other physical activity?				

Note to interviewer: these questions may be sensitive to some students. Use the probes below to address the questions overall. Have the student answer all questions and then probe.

If the student appears uncomfortable ask: is this OK to talk about in a survey if your answers are kept private or is it too uncomfortable? In general how do you feel about being asked these questions?

Note to interviewer: probe after this set of items.

Item-specific probes:

Tell me more about what you are thinking about [question #]. [Note to interviewer: ask for each question what time period, how did you calculate how often you each these things].

You said you [answer] eat fruit at school. How did you choose your response to this question? Can you give an example of a fruit students eat at school?

How about vegetables, can you give an example of a vegetable students eat at school?

You said you [answer] eat breakfast on school days. What time period are you referring to?

f)-g) What does “physical activity” mean in these questions?”

Mark One Response On Each Line.

13.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) I can talk to my teachers about problems I am having in class.				
	b) I can talk to a teacher or other adult at this school about something that is bothering me.				
	c) Students at this school think it’s ok to fight if someone insults them.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What is the question asking you in your own words? What kind of problems did you think about when you were answering this question? [note to interviewer: Try to get at whether they are thinking about “math problems” or bigger picture problems with schoolwork in general.]
When we refer to “teacher or… adult” what comes to mind for you? Who would you think about in answering this question?
What is the question asking you in your own words? Can you tell me at times when this happens? Were you thinking of those times when you answered this question?

Note to interviewer: stop student after this set of items and probe overall.

How easy or hard is it to choose an answer for each of these questions?

Mark One Response On Each Line

14.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Discipline problems are handed fairly at this school.				

Note to interviewer: probe after this set of items.

Item-specific probe:

Who or what is the question referring to? What do you think this question is asking? What does “discipline” mean to you? Can you tell me in your own words how you describe fair?

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort student out of the interviewing room and back to his or her parent. Please give the student an incentive [$25] and the parent an incentive [$40 if participant, $25 if not a participant and provided transportation for child to and from the interview], and thank them for time and effort.

Cognitive Laboratory: Protocols for Parent Survey

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. Also, keep track of the time so that your total interview time with the parent lasts 1 hour or less.

Background Information		Confirmed 
Date & Time
Name of Interviewer
Parent First Name
Parent’s Gender	 Male  Female
Child Enrolled in School	 Yes  No
Name of School
Type of School	 Public  Charter  Private
Student’s Grade
Age of Student
Is Student Hispanic or Latino	 Yes  No  Prefer not to answer
Student Ethnic Background

School Climate Surveys

PARENT SURVEY
2014

U.S. Department of Education
National Center for Education Statistics

MATERIALS NEEDED FOR INTERVIEW

INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)
RESPONDENT QUESTIONNAIRE
DO YOU HAVE THE CORRECT VERSION OF THE BOOKLET AND QUESTIONNAIRE?
CONSENT FORM (TWO COPIES)
INCENTIVE
PAYMENT RECEIPT
DIGITAL RECORDER AND EXTRA BATTERIES
PENS AND PENCILS

STEP 1: WELCOME AND INDTRODUCTION SCRIPT

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the parent]:

You are here today to help us out with a new survey that we are working on with the U.S. Department of Education. This survey asks about your opinions of [child]’s school. A survey is a set of questions about your experiences. This is not a test. I will ask you to take the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will ask you about the survey questions. I will also stop you once in a while to ask you questions. This will help us make the survey better and clearer for other parents just like you.

Because the information you provide is so important, I am going to be taking notes while you are working and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today].

Also, this interview will be recorded so researchers can review the tapes later. Any information that refers to you or your family personally, like your name, your child’s name, or name of your child’s school, will not be included in our report, and your responses will be used for research purposes only and will not be disclosed, or used for any other purpose except as required by law.

Remember, you are not being tested; we want you to tell us what you think about the survey. Would you like to be a part of this study? [If respondent agrees, continue, if not, end the interview.]

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the parent may ask.]

PROVIDE RESPONDENT WITH A COPY OF THE INFORMED CONSENT FORM. ASK THE RESPONDENT TO READ THE FORM (OR READ IT TO HIM/HER, DEPENDING ON RESPONDENT PREFERENCE), ANSWER ANY QUESTIONS, AND HAVE THE RESPONDENT SIGN THE FORM. LEAVE A SEPARATE COPY OF THE FORM WITH THE RESPONDENT.

SIGNED CONSENT FORM COLLECTED
COPY OF CONSENT FORM GIVEN TO RESPONDENT

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

How many windows are there in the house or apartment where you live?

Please tell me what you are thinking while you are figuring out your answer.

Shape19

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. While you are answering the questions, please think about the school that [child] attends. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would have read to yourself if I were not here.

Remember that I am very interested in hearing you talk about what you are thinking about as you figure out your answers to the questions.

Probe bank (from Willis)

Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking?
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are thinking.”
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: “Tell me what just happened. Tell me about what you were just doing.”

Marking Directions

Shape21

PLEASE READ EACH QUESTION CAREFULLY. Here is an example of the type of items you will see in this survey:

I. MARK ONE RESPONSE

1. What is the color of your eyes?

 Brown

 Blue

 Green

 Another color

If the color of your eyes is green, you would mark the circle beside green.

THIS IS THE END OF THE EXAMPLES.

THIS SURVEY IS NOT A TEST.

WE HOPE YOU WILL ANSWER EVERY QUESTION, BUT YOU MAY SKIP ANY QUESTION YOU DO NOT WANT TO ANSWER.

PLEASE GO TO THE NEXT PAGE TO BEGIN THE SURVEY.

Mark one response for each question.

1.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school provides instructional materials that reflect my child's culture, ethnicity, and identity.					

Item-specific probes:

What is this question asking you? What does “instructional materials” mean to you? In your own words what does “culture, ethnicity and identity” mean in this question?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

2.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school communicates how important it is to respect the beliefs and practices of all cultures.					

Item-specific probes:

What methods of communication are you thinking of? Can you tell me about the methods of communication that the school uses? You answered [response]; can you tell me more about why you chose this answer?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

3.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school encourages all students to take challenging courses no matter their race, ethnicity, nationality, or culture.				

Item-specific probes:

The question uses the term “challenging courses.” In your own words, what does this term mean? Tell me more about why you chose the [response].

4.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school helps me figure out what social and emotional skills my child needs to develop.				

Item-specific probes:

What does “social and emotional skills” mean to you? Can you explain to me how you came up with your answer?

5.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	At this school, my child feels he/she belongs.				

Item-specific probes:

Could you tell me what you think this question is asking?

Note to interviewer: probe after items 1-5.

6.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	I feel welcome at this school.				

Item-specific probes:

Tell me what you were thinking when you read this question, about feeling welcomed? [If R answered positively]: What kinds of things make you feel welcomed at your child’s school?
[If answered negatively]: Can you tell me more about that your response?

7.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school encourages me to be an active partner in educating my child.				

Item-specific probes:

In your own words, what is this question asking? Can you tell me about how the school encourages you to participate? What kinds of things are you including in this answer?

[Note to interviewer: you are trying to determine understanding of the question and concept understanding].

8.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	I feel comfortable talking to someone at this school about my child's behavior.				

Item-specific probes:

Can you tell me in your own words what that question was asking? What does “someone at this school” mean to you? What kinds of behaviors or things did you think about when you answered this question? You said [answer]. Can you tell me more about that?

9.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school has quality programs for my child's talents, gifts, or special needs.					

Item-specific probes:

What kind of programs are you thinking about here? Can you tell me about the programs that your child is involved in? What does “quality programs” mean to you?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

10.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school promptly responds to my phone calls, messages, or e-mails.					

Item-specific probes:

Have you had occasion to call or email this school, or leave a message? If not, how did you decide how to answer this question? What do you consider a prompt response? Think about the term “this school” in this question, are you thinking of an individual person or the school in general?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

Note to interviewer: probe after items 6-10.

11.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	At this school, the staff really cares about my child.				

Item-specific probes:

What does “really cares about my child” mean to you? Who are you thinking about when you think about “the staff”? What are you thinking about when trying to decide if the staff cares about your child?

12.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school is a friendly place overall.				

Item-specific probes:

Can you tell me in your own words what that question was asking? Tell me more about your answer.

13.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Staff at this school care about what families think.				

Item-specific probes:

In this question we refer to “staff.” Who did you include in your answer? How about families, who does that include?

14.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	My child is safe at this school.				

Item-specific probes:

What does safe mean to you? Tell me more about your answer?

15.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Racial/ethnic conflict among students is a problem at this school.				

Item-specific probes:

What does “racial/ethnic conflict” mean to you? You said [answer]. Can you tell me more about that?

Note to interviewer: probe after items 11-15.

16.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Gang-related activity is a problem at this school.				

Item-specific probes:

What is this question about? How well does it apply to you or your experience?

17.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Physical fighting between students is a problem at this school.				

Item-specific probes:

What is this question about? How well does it apply to you or your experience?

18.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Harassment or bullying of students at school or school activities is a problem at this school.				

Item-specific probes:

Tell me what you are thinking when I ask about harassment or bullying. Tell me more about your opinions on harassment and bullying at school.

19.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Harassment or bullying of students via electronic means or devices (i.e., text message, email, or social networks) is a problem at this school.				

Item-specific probes:

Tell me what you are thinking when I ask about harassment or bullying via electronic means or devices. Tell me more about your opinions on this type of harassment or bullying at school.

20.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school has helped me be more aware of bullying and cyberbullying of students.				

Item-specific probes:

Tell me more about what you are thinking when coming up with your answer. In what ways has the school helped you to become more aware of bullying? What does “cyberbullying” mean to you?

Note to interviewer: probe after items 16-20.

21.	How much of a problem is the following at this school.	Not a Problem	Small Problem	Somewhat a Problem	Large Problem
	Based on your experience, how much of a problem at this school is: students' drug use (such as marijuana, LSD, cocaine, ecstasy)?				
22.	How much of a problem is the following at this school.	Not a Problem	Small Problem	Somewhat a Problem	Large Problem
	Based on your experience, how much of a problem at this school is: students' use of tobacco (such as cigarettes, chew, cigars)?				
23.	How much of a problem is the following at this school.	Not a Problem	Small Problem	Somewhat a Problem	Large Problem
	Based on your experience, how much of a problem at this school is: students' alcohol use (such as beer, wine, liquor)?				

Note to interviewer: probe after items 21-23

Item-specific probes:

You answered [answer], can you describe what this means to you? What does it mean for students to use drugs? What does it mean to you for a student to use drugs, tobacco, or alcohol? [Interviewer: try to get at their understanding of “use” – how often, how much, etc.]
You answered [answer], can you describe what this means to you? What does it mean that students use tobacco? Can you tell me about the things you took into account for this question [Interviewer: try to get at their understanding of “use” – how often, how much, etc. also trying to understand what the respondent includes in the tobacco]

You answered [answer], can you describe what this means to you? What does it mean that students use alcohol? Can you tell me about the things you took into account for this question [Interviewer: try to get at their understanding of “use” – how often, how much, etc. also trying to understand what the respondent includes in alcohol use.]

Note to interviewer: after respondent is done with all three items ask the following questions.

How well do these questions apply to your experience at the school your child attends? Are these questions asking about your children at your child’s school specifically or more broadly about all children?

24.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school notifies parents effectively in the case of a school-wide emergency.				

24.

Using the scale provided, please indicate the extent to which you agree or disagree with the statement.

Strongly Agree

Agree

Disagree

Strongly Disagree

This school notifies parents effectively in the case of a school-wide emergency.



Item-specific probes:

In this question we use the term “notify effectively.” Can you tell me what that means to you? What do you think “school-wide emergency” means in this question? You said [answer]. Can you tell me more about that?

25.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school takes effective measures to ensure the safety of students.					

25.

Using the scale provided, please indicate the extent to which you agree or disagree with the statement.

Strongly Agree

Agree

Disagree

Strongly Disagree

Don’t Know

This school takes effective measures to ensure the

safety of students.



Item-specific probes:

What does “effective measures” mean to you?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

26.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school has made it clear to my child what he/she should do if there is an emergency, natural disaster (tornado, flood) or a dangerous situation (e.g. violent person on campus) during the school day.					

Item-specific probes:

What are you thinking of to help you determine if your child has a clear understating of what to do in an emergency? How easy or hard was it to choose an answer?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

27.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school has a bright and pleasant appearance.				

Item-specific probes:

What does “bright and pleasant appearance” mean to you? What parts of the school are you thinking about when coming up with your answer?

28.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	The school building is clean and well-maintained.				

Item-specific probes:

Can you tell me in your own words what that question was asking? What are you thinking about in order to come up with your answer. What parts of the school building are you thinking about when coming up with your answer?

Note to interviewer: probe after items 24-28.

29.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Attending school every day is important for my child to do well in his/her classes.				

Item-specific probes:

Tell me what you were thinking you read and answered this question.

30.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school has high expectations for student learning.				

Item-specific probes:

Can you tell me what “high expectations” are in your own words? When thinking about high expectations are you thinking of something the school has defined as a high expectation, or is it based on something you think is a high expectation?

31.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school sees me as a partner in my child's education.				

Item-specific probes:

In your own words, what is this question asking? What kinds of things are you including in this answer?

[Note to interviewer: you are trying to determine understanding of the question and concept understanding].

32.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	My child's teachers make themselves available to me.				

Item-specific probes:

What does “make themselves available” mean to you? Which teachers were you thinking about when you answered this question?

33.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	The programs and resources at this school are adequate to support students with special needs or disabilities.				

Item-specific probes:

How easy or hard is it for you to answer this question? Where you able to find a response that matches your experience?

Note to interviewer: probe after items 29-33.

34.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	Students have enough healthy food choices at this school.					

Item-specific probes:

What do you consider “healthy food choices”? You answered [response]; can you tell me more about that response?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

35.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school provides quality physical health and nutrition services for students.					
Item-specific probes: Did you find a response category that fit your experience? Can tell me more about your answer? If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?
36.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree	Don’t Know
	This school provides high quality services to help students with social or emotional needs.					

Item-specific probes:

What does the term “high quality services” mean for you? What or who comes to mind when you read “this school” in this question? Did you find a response category that fit your experience? Can tell me more about your answer?
If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer?

37.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree		Strongly Disagree		Don’t Know
	This school has enough programs that address students' emotional and social development.							
Item-specific probes: What does the term “emotional and social development” mean to you? Tell me more about why you chose [answer]. What programs were you thinking of? If Respondent answers Don’t Know: was it easy or difficult to choose one of the categories? Tell me more about your answer.
38.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree		Disagree		Strongly Disagree
	When my child does something good at school, I usually hear about it from the school.						

Item-specific probes:

What does doing “something good” mean to you? What ways do you hear from the school? Tell me more about why you chose [answer].

Note to interviewer: probe after items 34-38.

39.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	This school communicates school policies and procedures clearly to parents.				

Item-specific probes:

Can you tell me what this question is asking in your own words? What are you thinking about to help you come up with your answer? How easy or hard is it to come up with your answer?

40.	Using the scale provided, please indicate the extent to which you agree or disagree with the statement.	Strongly Agree	Agree	Disagree	Strongly Disagree
	Discipline problems are handled fairly at this school.				

Item-specific probes:

What do you think this question is asking? What does “discipline problems” mean to you? Can you tell me in your own words what it means for problems to be “handled fairly”?

Note to interviewer: probe after items 39-40.

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort parent out of the interviewing room and to the lobby. Please give the respondent incentive envelope [$40], and thank them for their time and effort.

Cognitive Laboratory: Protocols for Instructional Staff Survey

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. Also, keep track of the time so that your total interview time with the instructional staff lasts 1 hour or less.

Background Information		Confirmed 
Date & Time
Name of Interviewer
Staff First Name
Staff Gender	 Male  Female
Staff’s Type of School	 Elementary  Middle  High School
Staff’s Type of School	 Public  Charter  Private
Staff Title	 Core  Non-Core  Instructional Staff
School Location	 Urban  Suburban  Rural  Don’t Know
Years Working in Current Title	 2 or less  3-6  7 or more
Is Staff Hispanic or Latino	 Yes  No  Prefer not to answer
Staff Ethnic Background
Staff Primary Position

School Climate Surveys

INSTRUCTIONAL STAFF SURVEY
2014

U.S. Department of Education
National Center for Education Statistics

MATERIALS NEEDED FOR INTERVIEW

INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)
RESPONDENT QUESTIONNAIRE
DO YOU HAVE THE CORRECT VERSION OF THE BOOKLET AND QUESTIONNAIRE?
CONSENT FORM (TWO COPIES)
INCENTIVE
PAYMENT RECEIPT
DIGITAL RECORDER AND EXTRA BATTERIES
PENS AND PENCILS

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the teacher]:

You are here today to help us out with a new survey that we are working on with the U.S. Department of Education. This survey asks about your opinions of the school where you work: [school name]. A survey is a set of questions about your experiences. This is not a test. I will ask you to take the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will ask you about the survey questions. I will also stop you once in a while to ask you questions. This will help us make the survey better and clearer for other teachers and school instructional staff just like you.

Also, this interview will be recorded so researchers can review the tapes later. Any information that refers to you or the name of your school will not be included in our report, and your responses will be used for research purposes only and will not be disclosed, or used for any other purpose except as required by law.

Remember, you are not being tested; we want you to tell us what you think about the survey. Would you like to be a part of this study? [If respondent agrees, continue, if not, end the interview.]

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the participant may ask.]

SIGNED CONSENT FORM COLLECTED
COPY OF CONSENT FORM GIVEN TO RESPONDENT

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

How many windows are there in the house or apartment where you live?

Please tell me what you are thinking while you are figuring out your answer.

Shape23

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. While you are answering the questions, please think about the school you currently work at: [school name]. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would have read to yourself if I were not here.

Remember that I am very interested in hearing you talk about what you are thinking about as you figure out your answers to the questions.

Probe bank (from Willis)

Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking?
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are thinking.”
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: “Tell me what just happened. Tell me about what you were just doing.”

Marking Directions

Shape25

PLEASE READ EACH QUESTION CAREFULLY. Here is an example of the type of items you will see in this survey:

I. MARK ONE RESPONSE

1. What is the color of your eyes?

 Brown

 Blue

 Green

 Another color

If the color of your eyes is green, you would mark the circle beside green.

THIS IS THE END OF THE EXAMPLES.

THIS SURVEY IS NOT A TEST.

WE HOPE YOU WILL ANSWER EVERY QUESTION, BUT YOU MAY SKIP ANY QUESTION YOU DO NOT WANT TO ANSWER.

PLEASE GO TO THE NEXT PAGE TO BEGIN THE SURVEY.

Mark One Response.

1. Is your main assignment/responsibility at this school to provide instruction or other support services to any of these types of students - Special Education, English Language Learners, Gifted and Talented Education students, and Migrant Education?

 Yes

 No

Item-specific probes:

Can you tell me more about your assignment at school and the responsibilities that you have? What does “this school” mean to you in the context of this question?

2. How many years have you been working at this school?

 1-3 years

 4-9 years

 10-19 years

 20 or more years

Item-specific probes:

[Note to interviewer: we are interested in recall, try to find out if the respondent has a hard time remembering how long they have working at school or if there is a confusion with how long they have been a teacher/instructional staff member. Also try to get at whether they were thinking of how long they have worked at this specific school or of their entire career at any school].

You said [response]; how did you come up with that answer? How certain are you of your response?

Mark One Response On Each Line.

3.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) At this school, closing the racial/ethnic achievement gap is considered a high priority.				
	b) At this school, all students are treated equally, regardless of whether their parents are rich or poor.				
	c) This school encourages all students to enroll in challenging courses regardless of their race, ethnicity, or nationality.				
	d) This school provides instructional materials that reflect students' culture, ethnicity and identity.				
	e) This school fosters an appreciation of student diversity and respect for each other.				
	f) This school emphasizes showing respect for all students’ cultural beliefs and practices.				
	g) This school provides effective supports for teaching culturally and linguistically diverse students with IEPs.				
	h) This school provides effective supports for students needing alternative modes of communication (e.g., manual signs, communication boards, computer-based devices, picture exchange systems, Braille).				

Note to interviewer: probe after this set of items.

Item-specific probes:

How well does this question apply to your school? Can you tell me more about what comes to mind when you hear “achievement gap”? [Note to interviewer: here you are trying to determine two things, 1. is there a gap that applies to the school the teacher works at, and 2. how the teacher read/understood the racial/ethnic concept].
In this question we use the term “parents.” What does that term refer to?
The question uses the term “challenging courses.” In your own words, what does this term mean? Tell me more about why you chose the [response].
What is this question asking you? What does “instructional materials” mean to you? In your own words what does “culture, ethnicity and identity” mean in this question?
What is this question asking you?
How did you come up with your answer to this question?
Let’s talk about this question, how did you arrive at your answer? Can you tell me about the term “IEPs?” What does it refer to for you and at your school? What do “effective supports” mean to you?
How did you arrive at your answer? What were you thinking as you picked a response?

Mark One Response On Each Line.

4.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Staff do a good job helping parents understand when their child needs to learn social, emotional, and character skills.				
	b) Staff do a good job showing parents how to keep track of their child's progress.				
	c) This school asks families to volunteer at the school.				
	d) This school communicates with parents in a timely and ongoing basis.				

Note to interviewer: probe after this set of items.

Item-specific probes:

How did you arrive at your answer? In this question we use the term ‘parents.’ What does that term refer to for you in the context of your position at this school? How about “staff” – who comes to mind when you read this term?
What does it mean to you for the staff to do a good job? – A good job for whom?
What types of volunteer activities come to mind when you answer this question? When we say families in this question, what comes to mind for you?
What types of communication are you thinking of? What, to you, does “timely and ongoing” mean in this question?

Mark One Response On Each Line.

5.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) My level of involvement in decision making at this school is fine with me.				
	b) Staff at this school have many informal opportunities to influence what happens within the school.				
	c) At this school, students are given the opportunity to take part in decision making.				
	d) Students at this school are encouraged to help solve problems at this school.				
	e) Administrators consistently seek input from staff.				
	f) This school’s administration invites students to share their ideas about the school.				
	g) Staff at this school makes it easy for students to suggest activities.				
	h) This school provides students with opportunities to take a lead role in organizing programs and activities.				
	i) Students are encouraged to get involved in extra-curricular activities.				

Note to interviewer: probe after this set of items.

Item-specific probes:

How well does this question apply to you? How did you come up with an answer? What kind of decisions were you thinking about?
What types of informal opportunities come to mind when you answer this question? When we say “influence” in this question, what comes to mind for you?

Can you tell me, in your own words, what this question is asking? What does “take part in the decision making process” mean to you?

Can you tell me, in your own words, what this question is asking? What problems were you thinking of?

The question uses the term “administrators” and “staff.” In your own words, what do these terms mean? Tell me more about why you chose the [response]. What input were you thinking of?

What types of ideas does the administration invite students to share? When you think of the administration, what comes to mind? Whom does that include?
How did you arrive at your answer? Tell me more about why you chose [response].
How did you arrive at your answer? Tell me more about why you chose [response].

How did you arrive at your answer? Tell me more about why you chose [response].

Mark One Response On Each Line.

6.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school is an emotionally safe place for students.				
	b) Students get along well with each other.				
	c) This school is an emotionally safe place for staff.				
	d) I feel like I belong.				
	e) I feel satisfied with the recognition I get for doing a good job.				
	f) I feel comfortable discussing feelings, worries, and frustrations with my supervisor.				
	g) This school inspires me to do the very best at my job.				
	h) People at this school care about me as a person.				
	i) I can effectively work with defiant or disruptive students.				
	j) I can manage almost any student behavior problem.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What types of ideas does the administration invite students to share? When you think of the administration, what comes to mind? Whom does that include?

Note to interviewer: ask this probe after respondent has completed item c.

and c). In this question we use the term “emotionally safe,” what does this term mean in your own words?

d) What does it mean for you to “belong?” Were you thinking of belonging at this school, or belonging in general? Tell me more about your answer?

e) You answered [answer]; tell me more about your answer? How do you get recognized? In what ways do you feel recognized?

f) – h). In your own words what is this question asking you? How well does this question apply to you? How did you come up with an answer?

i) To you, what does the term “defiant or disruptive students” mean? Can you tell me more about how you work with them effectively?

j) Tell me more about managing student behavior problems.

Mark One Response On Each Line.

7.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) I think that bullying (e.g., physical, verbal, and/or social bullying) is a frequent problem at this school.				
	b) I think that cyberbullying is a frequent problem among students at this school.				
	c) I think that racial/ethnic tension or discrimination among students is a frequent problem at this school.				
	d) Students at this school would feel comfortable reporting a bullying incident to a teacher or other staff.				
	e) Staff at this school communicate to students that bullying is unacceptable.				
	f) Staff at this school always stop bullying when they see it.				
	g) Staff at this school know what to say or do to intervene in a bullying situation.				
	h) This school provides harassment or bullying prevention.				
	i) Staff at this school are teased or picked on about their race or ethnicity.				
	j) Staff at this school are teased or picked on about their culture, or religion.				
	k) Staff at this school are teased or picked on about their physical or mental disability.				
	l) Staff at this school are teased or picked on about their sexuality.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Note to interviewer: ask probe after item b:

a) and b). What does bullying mean to you in this question? How about cyber bullying – what do you think of when you read this?

Note to interviewer: ask probe after item c:

b) and c). What does it mean to you for something to be a frequent problem?

d) What “teacher or other staff” were you thinking of when you answered this question?

e) What kind of communication were you thinking of?

f) Can you tell me about how you arrived at your answer?

g) How did you come up with your answer to this question? Can you tell me more about intervening in a bullying situation?

h) In your own words, what is this question asking you? How do you think this school provides prevention?

Note to interviewer: ask probe after item j:

i-j). Who comes to mind when you think of “staff?” How would you be aware of staff being teased or picked on? If participant appears uncomfortable ask: Is it okay to talk about these topics in a survey, as long as it’s confidential, or is it too uncomfortable? In general, how do you feel about being asked these questions?

Note to interviewer: stop instructional staff after this set of items and probe overall.

How well do these questions apply to you?

Mark One Response On Each Line.

8.	How much of a problem are the following at this school:	Not a Problem	Small Problem	Somewhat a Problem	Large Problem
	a) Based on your experience, how much of a problem at this school is: Students' drug use (such as marijuana, LSD, cocaine, ecstasy)?				
	b) Based on your experience, how much of a problem at this school is: Students' use of tobacco (such as cigarettes, chew, cigars)?				
	c) Based on your experience, how much of a problem at this school is: Students' alcohol use (such as beer, wine, liquor)?				

Note to interviewer: probe after this set of items.

Item-specific probes:

What is this question asking, in your own words? You answered [answer], can you describe what this means to you? What does it mean for students to engage in drug use? [Interviewer: try to get at their understanding of “use” – how often, how much, etc.]

You answered [answer], can you describe what this means to you? What does it mean that students use tobacco? Can you tell me about the things you took into account for this question? [Interviewer: try to get at their understanding of “use” – how often, how much, etc. also trying to understand what the respondent includes in the tobacco]

You answered [answer], can you describe what this means to you? What does it mean that students use alcohol? Can you tell me about the things you took into account for this question? [Interviewer: try to get at their understanding of “use” – how often, how much, etc. also trying to understand what the respondent includes in alcohol use.]

Note to interviewer: stop instructional staff after this set of items and probe overall.

How well do these questions apply to your experience at the school your work?

Mark One Response On Each Line.

9.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school collaborates well with community organizations to help address youth substance use problems.				
	b) This school has adequate resources to address substance use prevention.				
	c) This school provides effective confidential support and referral services for students needing help because of substance abuse.				
	d) At this school, first-time violations of alcohol or other drug policies are punished by at least an out-of-school suspension.				
	e) Substance abuse prevention is an important goal at this school.				
	f) This school has programs that address substance use among students.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What kind of community organization are you thinking about? When you think of collaboration, what does that mean to you? What were you thinking about when you read “substance use?”
In your own words, what is this question asking you? What adequate resources were you thinking of?
[Note to interviewer: you are probing on substance abuse in this question which is different than substance use from item a. You are trying to ascertain if the respondent is able to differentiate between substance use and substance abuse]. What kind of support are you thinking about? What does “confidential” mean to you? What does “confidential” mean in terms of your school? When you hear the term “substance abuse,” what does that refer to? What about “referral services?” Did you find a response category that fit your experience? What does it mean to you for the provided support to be “effective;” is this different than “adequate support?”
In this question we use the term “suspension.” Can you tell me what this means for you? Were you thinking of only out of school suspensions? If R answered Strongly Disagree or Disagree, ask: What does happen to first-time offending students at this school? [Note to interviewer: you are trying to determine what type of suspension the R is thinking about].
Can you tell me about how you arrived at your answer?
What kind of programs are you thinking about? Did you find a response category that fit your experience?

Mark One Response On Each Line.

10.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) I know what to do in if there is an emergency, natural disaster (tornado, flood) or a dangerous situation (e.g. violent person on campus) during the school day.				
	b) This school has a written plan that describes procedures to be performed in shootings.				
	c) This school has a written plan that clearly describes procedures to be performed in natural disasters (e.g., earthquakes or tornadoes).				
	d) This school or school district provides effective training in safety procedures to staff (e.g., how to handle emergencies).				

Note to interviewer: probe after this set of items.

Item-specific probes:

What is this question asking, in your own words? How do you know what to do? You answered [response], how did you choose this response?
Tell me what you were thinking when you read and answered this question.
When you answered this question, what were you thinking about?
You answered [response]. Can you explain/tell me more about your answer? What emergencies were you thinking of?

Mark One Response On Each Line

11.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school has a bright and pleasant appearance.				
	b) This school is an inviting work environment.				
	c) Overcrowding is a problem at this school.				
	d) My teaching is hindered by poor heating, cooling, and/or lighting systems at this school.				
	e) My teaching is hindered by a lack of instructional space (e.g., classrooms) at this school.				
	f) My teaching is hindered by a lack of text books and basic supplies at this school.				
	g) My teaching is hindered by inadequate or outdated equipment or facilities at this school.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Note to interviewer: ask probe after item c:

c) We use the word “overcrowding”; in your own words, what does this word mean?

a)-c). Tell me about how you came up with your answers to these questions.

f) What does “basic supplies” mean to you in this question?

g) Which “equipment or facilities” were you thinking of when you answered this question?

Note to interviewer: ask probe after item g:

d)-g). These questions refer to “my teaching.” What does that refer to? What does your teaching involve or include? What were you thinking of when read your teaching being “hindered?”

Note to interviewer: stop instructional staff after this set of items and probe overall.

When answering these questions, how did you come up with the answers?
Are these any questions that were hard to answer?

Mark One Response On Each Line.

12.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) The students in my class(es) attend class regularly.				
	b) The students in my class(es) come to class prepared with the appropriate supplies and books.				
	c) The students in my class(es) actively participate in class activities.				
	d) We have so many different programs at this school that I can't keep track of them all.				
	e) Teachers at this school feel responsible to help each other do their best.				
	f) Teachers at this school feel responsible when students at this school fail.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Tell me more about your answer to this question. For you, what does it mean to attend “class(es) regularly”? [Ask R if they teach more than one class. If Yes, ask]: Were you thinking of all your classes, or just a few of your classes, when you answered this question?
What is this question asking in your own words? For you, what does it mean to come to class well “prepared”?
How often do you take stock of participation in your classes? When you think about participation, how do you think about it?
Can you tell me more about your answer? What do you mean by the programs in the school? Can you tell me about the programs?

Note to interviewer: ask probe after item f:

e)-f). Can you tell me what this question means to you?

Mark One Response On Each Line

13.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Staff at this school promote students’ physical health and nutrition.				
	b) This school provides quality physical health and nutrition services for students.				
	c) This school provides the materials, resources, and training necessary for me to support students' physical health and nutrition.				
	d) This school places a priority on making healthy food choices.				
	e) This school places a priority on students’ health needs.				
	f) This school places a priority on students' physical activity.				
	g) This school provides quality physical health and nutrition instruction.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Did you find a response category that fit your experience? Can tell me more about your answer? What were you thinking about how this school “promotes” student health and nutrition?
Did you find a response category that fit your experience? Can you tell me about your answer? What does comes to mind when you read “quality services”?
Can you tell me what you think this question is asking you? Were you thinking about the materials, resources, and training provided specifically for you?
How much would you say you know about [topic]? Tell me more about why you chose [answer].
How much would you say you know about [topic]? Tell me more about why you chose [answer].
How much would you say you know about [topic]? Tell me more about why you chose [answer].

Note to interviewer: ask probe after item f:

d)-f). How can you tell if your school places a “priority” on these things?

Can you tell me how you chose your answer?

Mark One Response On Each Line.

14.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school provides quality counseling or other services to help students with social or emotional needs.				
	b) This school provides the materials, resources, and training necessary for me to support students' social or emotional needs.				
	c) This school places a priority on addressing students’ mental health needs.				
	d) This school places a priority on social and emotional development.				
	e) This school places a priority on teaching students strategies to manage their stress levels.				
	f) This school places a priority on helping students with their social, emotional, and behavioral problems.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What does the term “quality counseling or other services” mean for you? Did you find a response category that fit your experience? Can tell me more about your answer?
Can you tell me what you think this question is asking you? Were you thinking about the materials, resources, and training provided specifically for you?
How much would you say you know about [topic]? Tell me more about why you chose [answer].
What does the term “emotional development” mean for you? How much would you say you know about [topic]? Tell me more about why you chose [answer].

How much would you say you know about [topic]? Tell me more about why you chose [answer].

What does the term “social, emotional, and behavioral problems” mean for you? How much would you say you know about [topic]? Tell me more about why you chose [answer].

Mark One Response On Each Line

15.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Discipline problems are handled fairly at this school.				
	b) This school effectively handles student discipline and behavior problems.				
	c) Staff at this school work together to ensure an orderly environment.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What do you think this question is asking? What does “discipline problems” mean to you? Can you tell me in your own words how you describe “fairly?”

What does “effectively handle” mean to you?

We use the term “orderly environment.” What does this term mean for you as a teacher?

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort teacher out of the interviewing room and to the lobby. Please give the respondent incentive envelope [$40], thank them for their time and effort.

Cognitive Laboratory: Protocols for Non-Instructional Staff Survey

Interviewer Instructions: Please fill out the table below before starting Part 1 of the interview. Also, keep track of the time so that your total interview time with the non-instructional staff lasts 1 hour or less.

Background Information		Confirmed 
Date & Time
Name of Interviewer
Staff First Name
Staff Gender	 Male  Female
Staff’s Type of School	 Elementary  Middle  High School
Staff’s Type of School	 Public  Charter  Private
Staff Title
School Location	 Urban  Suburban  Rural  Don’t Know
Years Working in Current Title	 2 or less  3-6  7 or more
Is Staff Hispanic or Latino	 Yes  No  Prefer not to answer
Staff Ethnic Background
Staff Primary Position

School Climate Surveys

NON-INSTRUCTIONAL STAFF SURVEY
2014

U.S. Department of Education
National Center for Education Statistics

MATERIALS NEEDED FOR INTERVIEW

INTERVIEWER PROTOCOL BOOKLET (THIS BOOKLET)
RESPONDENT QUESTIONNAIRE
DO YOU HAVE THE CORRECT VERSION OF THE BOOKLET AND QUESTIONNAIRE?
CONSENT FORM (TWO COPIES)
INCENTIVE
PAYMENT RECEIPT
DIGITAL RECORDER AND EXTRA BATTERIES
PENS AND PENCILS

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the principal/staff member]:

Also, this interview will be recorded so researchers can review the tapes later. Any information that refers to you or the name of your school, will not be included in our report, and your responses will be used for research purposes only and will not be disclosed, or used for any other purpose except as required by law.

Remember, you are not being tested; we want you to tell us what you think about the survey. Would you like to be a part of this study? [If respondent agrees, continue, if not, end the interview.]

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions about what I just said? [Answer any questions the participant may ask.]

SIGNED CONSENT FORM COLLECTED
COPY OF CONSENT FORM GIVEN TO RESPONDENT

IF THE RESPONDENT HAS CONSENTED TO RECORDING, START THE RECORDER.

STEP 2: THINK-ALOUD PRACTICE

How many windows are there in the house or apartment where you live?

Please tell me what you are thinking while you are figuring out your answer.

Shape40

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

HAND RESPONDENT A COPY OF THE QUESTIONNAIRE AND A PEN.

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. While you are answering the questions, please think about the school where you currently work: [school name]. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would have read to yourself if I were not here.

Remember that I am very interested in hearing you talk about what you are thinking about as you figure out your answers to the questions.

Probe bank (from Willis)

Can you tell me in your own words what that question was asking?
What does the word [term] in this question mean to you?
How easy or difficult is it to remember [topic]?
How easy or hard was it to choose an answer?
Tell me what you are thinking?
How did you come up with that answer?
You said [answer]. Can you tell me more about that?
[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?
[If you pick up on a visual cue like a furrowed brow or a puzzled look]: “Tell me what you are thinking.”
[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: “Tell me what just happened. Tell me about what you were just doing.”

Marking Directions

Shape42

PLEASE READ EACH QUESTION CAREFULLY. Here is an example of the type of items you will see in this survey:

I. MARK ONE RESPONSE

1. What is the color of your eyes?

 Brown

 Blue

 Green

 Another color

If the color of your eyes is green, you would mark the circle beside green.

THIS IS THE END OF THE EXAMPLES.

THIS SURVEY IS NOT A TEST.

WE HOPE YOU WILL ANSWER EVERY QUESTION, BUT YOU MAY SKIP ANY QUESTION YOU DO NOT WANT TO ANSWER.

PLEASE GO TO THE NEXT PAGE TO BEGIN THE SURVEY.

Mark One Response.

 Yes

 No

Item-specific probes:

Can you tell me more about your assignment at school and the responsibilities that you have?

2. How many years have you been working at this school?

 1-3 years

 4-9 years

 10-19 years

 20 or more years

Item-specific probes:

You said [response]; how did you come up with that answer? How certain are you of your response?

Mark One Response On Each Line.

3.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) At this school, closing the racial/ethnic achievement gap is considered a high priority.				
	b) At this school, all students are treated equally, regardless of whether their parents are rich or poor.				
	c) This school encourages all students to enroll in challenging courses regardless of their race, ethnicity, or nationality.				
	d) This school provides instructional materials that reflect students' culture, ethnicity and identity.				
	e) This school fosters an appreciation of student diversity and respect for each other.				
	f) This school emphasizes showing respect for all students’ cultural beliefs and practices.				
	g) This school provides effective supports for teaching culturally and linguistically diverse students with IEPs.				
	h) This school provides effective supports for students needing alternative modes of communication (e.g., manual signs, communication boards, computer-based devices, picture exchange systems, Braille).				

Note to interviewer: probe after this set of items.

Item-specific probes:

How well does this question apply to your school? Can you tell me more about what comes to mind when you hear “achievement gap?” [Note to interviewer: here you are trying to determine several things, 1. How does R understand “achievement gap” and is there a gap that applies to the school the R works at, 2. how does the R read/understand racial/ethnic concept, and 3. How did R decide how to answer].
In this question we use the term “parents.” What does that term refer to for you in the context of being a principal/staff at the school?
The question uses the term “challenging courses.” In your own words, what does this term mean? Tell me more about why you chose the [response].
What is this question asking you? In your own words what does “culture, ethnicity and identity” mean in this question? How did you decide how to answer this question?
What is this question asking you? What does “student diversity” mean to you in this question? How about “this school”?
Let’s talk about this question; how did you arrive at your answer? Can you tell me about the terms “cultural beliefs and practices” and “IEPs?” What does it refer to for you and at your school?
How did you come up with your answer to this question?
How did you arrive at your answer? What were you thinking as you picked a response? What do you think of when you see “students needing alternative modes of communication?”

Mark One Response On Each Line. (PRINCIPAL ONLY)

4.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Staff at this school regularly give students individualized attention and help.				
	b) Staff at this school teach students strategies to manage emotions.				
	c) Staff at this school let students know how they should behave when dealing with each other.				
	d) Staff at this school teach positive conflict resolution strategies to students.				
	e) Staff at this school help students see what is the right thing to do and help them understand it.				

Note to interviewer: probe after this set of items.

Item-specific probes:

How did you arrive at your answer? In this question we use the term “staff.” What does that term refer to for you in the context of being the principal of a school? What do you think “individualized attention and help” means in this context?

Can you tell me what this question is asking in your own words? You answered [response]; what did you think about to come up with your answer?

to e). What are you thinking about to help you come up with your answer? Can you give me some of the examples you are thinking of to help you decide how to answer? [INTERVIEWER: Make sure to probe on clarity of e) in particular]

What does “positive conflict resolution strategies” mean to you?

Mark One Response On Each Line.

5.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school helps parents find community supports for their students who need them.				
	b) Staff at this school do a good job helping parents to support their children's learning at home.				
	c) Staff at this school do a good job helping parents understand when their child needs to learn social and emotional skills.				
	d) Staff at this school do a good job showing families how to keep track of their child's progress.				
	e) At this school the staff get along well.				
	f) At this school there is a feeling of trust among the staff.				
	g) At this school staff are willing to help each other out.				
	h) At this school the staff respect each other.				
	i) At this school staff care about students.				
	j) At this school students get along well with the staff.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What does “community supports” mean to you?

How easy or difficult is to think about how the staff help parents?

How did you arrive at your answer? In this question we use the term “parents.” What does that term refer to for you in the context of being a principal/staff member at school?
How did you decide how to answer this question? What does it mean to you for the staff to “do a good job?” – A good job for whom?
What do you think “get along well” means in this question?

Note to interviewer: ask probe after item j:

f)-j). How easy or difficult is it to think about how to answer these questions? Are you able to think of examples that helped you come up with your answer?

Mark One Response On Each Line. (PRINCIPAL ONLY)

6.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Staff at this school give students opportunities to get involved in service projects or activities.				
	b) I ask students about their ideas.				

Note to interviewer: probe after this set of items.

Item-specific probes:

You said [answer]. Can you tell me more about that? Are you thinking of specific incidents?

Can you tell me more about your answer? I see you answered [answer]. Can you tell me about this?

Mark One Response On Each Line.

7.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Administrators ask staff to be involved in making decisions.				
	b) My level of involvement in decision making at this school is fine with me.				
	c) Staff at this school have many informal opportunities to influence what happens within the school.				
	d) At this school, students are given the opportunity to take part in decision making.				
	e) Students at this school are encouraged to help solve problems at this school.				
	f) This school’s administration invites students to share their ideas about the school.				
	g) Staff at this school make it easy for students to suggest activities.				
	h) This school provides students with opportunities to take a lead role in organizing programs and activities.				
	i) At this school staff care about students.				
	j) Students are encouraged to get involved in extra-curricular activities.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Who are you thinking about when you think about the term “administrators?”
How well does this question apply to you? How did you come up with an answer? What kind of decisions were you thinking about?
What types of informal opportunities come to mind when you answer this question? When we say influence in this question, what comes to mind for you?

Can you tell me, in your own words, what this question is asking? What does “take part in decision making” mean to you?

Can you tell me, in your own words, what this question is asking?

The question uses the terms “administrators” and “staff.” In your own words, what do these terms mean? Tell me more about why you chose the [response]. What input were you thinking of?

What types of ideas does the administration invite students to share? When you think of the administration, what comes to mind? Whom does that include?
How did you arrive at your answer? Tell me more about why you chose [response].
How did you arrive at your answer? Tell me more about why you chose [response].

How did you arrive at your answer? Tell me more about why you chose [response].

Mark One Response On Each Line.

8.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school is an emotionally safe place for students.				
	b) Students get along well with each other.				
	c) This school is an emotionally safe place for staff.				
	d) I feel like I belong.				
	e) I feel satisfied with the recognition I get for doing a good job.				
	f) I feel comfortable discussing feelings, worries, and frustrations with my supervisor.				
	g) This school inspires me to do the very best at my job.				
	h) People at this school care about me as a person.				
	i) I can effectively work with defiant or disruptive students.				
	j) I can manage almost any student behavior problem.				

Note to interviewer: probe after this set of items.

Item-specific probes:

and c). In this question we use the term “emotionally safe.” What does this term mean in your own words?
Can you tell me in your own words what this question is asking? Which students were you thinking about when you were answering this question?

d) What does it mean for you to “belong?” Were you thinking of belonging at this school, or belonging in general? Tell me more about your answer?

e) You answered [answer]; tell me more about your answer. How do you get recognized? In what ways do you feel recognized?

f)–h). In your own words what is this question asking you?

i) To you what does the term “defiant or disruptive students” mean? Can you tell me more about how you work with them effectively?

j) Tell me more about managing student behavior problems.

Shape50

Mark One Response On Each Line.

9.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Staff at this school communicate to students that bullying is unacceptable.				
	b) This school provides harassment or bullying prevention.				
	c) Staff at this school are teased or picked on about their race or ethnicity.				
	d) Staff at this school are teased or picked on about their culture, or religion.				
	e) Staff at this school are teased or picked on about their physical or mental disability.				
	f) Staff at this school are teased or picked on about their sexuality.				

Note to interviewer: probe after this set of items.

Item-specific probes:

a) What does “bullying” mean to you in these questions? What kind of communication were you thinking of?

b) In your own words, what is this question asking you? How were you thinking this school provides prevention?

Note to interviewer: ask probe after item f:

c)-f). Who comes to mind when you think of “staff?” How could you be aware of staff being teased or picked on? If participant appears uncomfortable ask: Is it okay to talk about these topics in a survey, as long as it’s confidential, or is it uncomfortable? In general, how do you feel about being asked these questions?

Mark One Response On Each Line

10.	How much of a problem are the following at this school:	Not a Problem	Small Problem	Somewhat a Problem	Large Problem
	a) Based on your experience, how much of a problem at this school is: Students' drug use (such as marijuana, LSD, cocaine, ecstasy)?				
	b) Based on your experience, how much of a problem at this school is: Students' use of tobacco (such as cigarettes, chew, cigars)?				
	c) Based on your experience, how much of a problem at this school is: Students' alcohol use (such as beer, wine, liquor)?				

Note to interviewer: Wait until respondent is done with all three items. Here you are trying to determine how the respondent conceptualizes the response categories: not a problem, small problem, somewhat a problem, large problem.

Item specific probes:

What is this question asking, in your own words? You answered [answer]; can you describe what this means to you? What does it mean for students to engage in drug use? [Interviewer: try to get at their understanding of “use” – how often, how much, etc.]

You answered [answer], can you describe what this means to you? What does it mean that students use tobacco? Can you tell me about the things you took into account for this question [Interviewer: try to get at their understanding of “use” – how often, how much, etc. also trying to understand what the respondent includes in the tobacco]

You answered [answer], can you describe what this means to you? What does it mean that students use alcohol? Can you tell me about the things you took into account for this question [Interviewer: try to get at their understanding of “use” – how often, how much, etc. also trying to understand what the respondent includes in alcohol use.]

Note to interviewer: stop non-instructional staff after this set of items and probe overall.

How well do these questions apply to your experience at the school your work?

Mark One Response On Each Line

11.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school collaborates well with community organizations to help address youth substance use problems.				
	b) This school has adequate resources to address substance use prevention.				
	c) This school provides effective confidential support and referral services for students needing help because of substance abuse.				
	d) At this school, first-time violations of alcohol or other drug policies are punished by at least an out-of-school suspension.				
	e) Substance abuse prevention is an important goal at this school.				
	f) This school has programs that address substance use among students.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What kind of community organization are you thinking about? When you think of collaboration, what does that mean to you? What are you thinking about when you read “substance use?”
In your own words, what is this question asking you? What adequate resources were you thinking of?
What kind of support are you thinking about? What does “confidential” mean to you? What does “confidential” mean in terms of your school? When you hear the term “substance abuse,” what does that refer to? What about “referral services?” Did you find a response category that fit your experience? What does it mean to you for the provided support to be “effective;” does this differ from “adequate support?”
In this question we use the term “suspension.” Can you tell me what this means for you? Were you thinking of only out of school suspensions? [Note to interviewer: you are trying to determine if the term is something that is used in school, if they understand the suspension and what type of suspension the R is thinking about].
Can you tell me about how you arrived at your answer?
What kind of programs are you thinking about? Did you find a response category that fit your experience?

Note to interviewer: stop non-instructional staff after this set of items and probe overall.

[IF RESPONDENT IS NOT A COUNSELOR OR PRINCIPAL]: How did you decide how to answer these questions?

Mark One Response On Each Line. (PRINCIPAL ONLY)

12.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school notifies parents in case of a school-wide emergency.				
	b) This school provides a structured anonymous threat reporting system (e.g., online submission, telephone hotline, or written submission via drop box).				
	c) This school has agreements with various agencies and services to aid timely communication.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What is this question asking, in your own words? You answered [response]; how did you choose this response? What kinds of things are “school-wide emergencies?”
When you answered this question, what were you thinking about? What do “structured” and “anonymous” mean to you in this question? Were you thinking of a reporting system that goes out from the school or comes into the school, or both?
Can you tell me about how you arrived at your answer? What does “timely” mean to you in this question? What kind of agreements were you thinking of?

Mark One Response On Each Line.

13.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) I know what to do if there is an emergency, natural disaster (tornado, flood) or a dangerous situation (e.g. violent person on campus) during the school day.				
	b) This school has a written plan that describes procedures to be performed in shootings.				
	c) This school has a written plan that clearly describes procedures to be performed in natural disasters (e.g., earthquakes or tornadoes).				
	d) This school or school district provides effective training in safety procedures to staff (e.g., how to handle emergencies).				

Note to interviewer: probe after this set of items.

Item-specific probes:

What is this question asking, in your own words? How do you know what to do? You answered [response]; how did you choose this response?
Tell me what you were thinking as you read and answered this question.
When you answered this question, what were you thinking about? Which emergencies were you thinking of?
You answered [response]; can you explain/tell me more about your answer?

Mark One Response On Each Line. (PRINCIPAL ONLY)

14	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school's facilities are adequate for meeting the needs of the children at this school.				
	b) The following are a problem in the neighborhood where this school is located: garbage, litter, or broken glass in the street or road, on the sidewalks, or in yards.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What does “adequate” mean to you in this question? What “needs” are you thinking about while answering this question?
This question refers to several problems; are you thinking about them as a whole or are you thinking about each individual problem? If some of these problems are found in your school’s neighborhood and others aren’t, how would that affect how you answer the question?

Mark One Response On Each Line.

15.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Overcrowding is a problem at this school.				
	b) My work is hindered by poor heating, cooling, and/or lighting systems at this school.				
	c) My work is hindered by insufficient work space at this school.				
	d) My work is hindered by a lack of materials and basic supplies at this school.				
	e) My work is hindered by inadequate or outdated equipment or facilities at this school.				
	f) This school has a bright and pleasant appearance.				
	g) This school is an inviting work environment.				
	h) My workspace at this school is comfortable.				

Note to interviewer: probe after this set of items.

Item-specific probes:

a) We use the word “overcrowding;” in your own words, what does this word mean to you?

b)–e). These questions mention to “my work.” What does that refer to? What does your work involve or include? What were you thinking of when you read your “work is hindered?”

d) What does “basic supplies” mean to you in this question?

e) Which “equipment or facilities” were you thinking of when you answered this question?

f)-h). Tell me about how you came up with your answers to these questions. What were you thinking of when you read “this school?” How about “my workspace?”

Note to interviewer: stop non-instructional staff after this set of items and probe overall.

Are there any items that were hard to answer?

Mark One Response On Each Line. (PRINCIPAL ONLY)

16.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Poor student attendance is a problem at this school.				
	b) This school has a formal process to obtain parental input on policies related to curriculum and instructional programs.				

Note to interviewer: probe after this set of items.

Item-specific probes:

How do you define “poor attendance” being a problem?
What do you consider a “formal process?” What are ways that parental input is obtained?

Mark One Response On Each Line.

17.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Once we start a new program at this school, we follow up to make sure that it's working.				
	b) Staff at this school feel responsible to help each other do their best.				
	c) Staff at this school feel responsible when students at this school fail.				
	d) The programs and resources at this school are adequate to support students with special needs or disabilities.				
	e) Staff at this school feel that it is a part of their job to prepare students to succeed in college.				
	f) Staff at this school expect students to do their best all the time.				

Note to interviewer: probe after this set of items.

Item-specific probes:

a) What does “follow up” mean to you in this question? How does this school follow up?

b)-c). How did you come up with your response to this question? Can you give an example for each?

d) What does “adequate” mean to you in this question? What “needs” are you thinking about while answering this question?

e) Can you tell me what this questions means to you? Can you give an example?

f) Tell me more about your answer to this question? What were you thinking “all the time” meant when you were considering how to answer?

Mark One Response On Each Line. (PRINCIPAL ONLY)

18.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Poor student nutrition is a problem at this school.				
	b) Lack of student physical activity is a problem at this school.				

Note to interviewer: probe after this set of items.

Item-specific probes:

a)-b). How much would you say you know about [topic]? Tell me more about why you chose [answer]. What does being a “problem” mean to you in this question?

Mark One Response On Each Line

19.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school provides the materials, resources, and training necessary for me to support students' physical health and nutrition.				
	b) This school provides quality physical health and nutrition services for students.				
	c) This school places a priority on making healthy food choices.				
	d) This school places a priority on students’ health needs.				
	e) This school places a priority on students' physical activity.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Can you tell me what you think this question is asking you? Were you thinking about the “materials, resources, and training” provided specifically for you?
Did you find a response category that fit your experience? Can you tell me about your answer? What does come to mind when you read “quality services?”
How much would you say you know about [topic]? Tell me more about why you chose [answer].
How much would you say you know about [topic]? Tell me more about why you chose [answer].
How much would you say you know about [topic]? Tell me more about why you chose [answer].

Note to interviewer: on items c – e you are trying to find out more about how well the respondent can answer this item and if the response categories match his/her experience.

Ask probe after item e:

c)-e). How can you tell if your school places a “priority” on these things?

Mark One Response On Each Line. (PRINCIPAL ONLY)

20.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Student depression or other mental health problems are a problem at this school.				
	b) Lack of student social-emotional skills is a problem at this school.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What does “student depression and other mental health problems” mean to you? How can you tell when this issue is a problem?
What do you think this question is asking about? How can you tell when this issue is a problem?

Mark One Response On Each Line.

21.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) This school places a priority on addressing students’ mental health needs.				
	b) This school places a priority on social and emotional development.				
	c) This school places a priority on teaching students strategies to manage their stress levels.				
	d) This school provides the materials, resources, and training necessary for me to support students' social or emotional needs.				
	e) This school provides quality counseling or other services to help students with social or emotional needs.				

Note to interviewer: probe after this set of items.

Item-specific probes:

How much would you say you know about [topic]? Tell me more about why you chose [answer].

What does the term “social and emotional development” mean to you? How much would you say you know about [topic]? Tell me more about why you chose [answer].

How much would you say you know about [topic]? Tell me more about why you chose [answer].

Can you tell me what you think this question is asking you? Were you thinking about the materials, resources, and training provided specifically for you?

What does the term “quality counseling or other services” mean for you? Did you find a response category that fit your experience? Can tell me more about your answer?

Mark One Response On Each Line. (PRINCIPAL ONLY)

22.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Order and discipline are maintained satisfactorily at this school.				
	b) This school has a formal process to obtain parental input on policies related to school crime and discipline.				

Note to interviewer: probe after this set of items.

Item-specific probes:

What do you think this question is asking? What does “order and discipline” mean to you? Can you tell me in your own words how you describe “satisfactorily?”
What do you consider a “formal process?” What does “school crime” mean to you? How did you come up with your answer?

Mark One Response On Each Line.

23.	How strongly do you agree or disagree with the following statements?	Strongly Agree	Agree	Disagree	Strongly Disagree
	a) Staff at this school assign consequences that help students learn from their behavior.				
	b) Discipline problems are handled fairly at this school.				
	c) This school effectively handles student discipline and behavior problems.				
	d) Staff at this school work together to ensure an orderly environment.				

Note to interviewer: probe after this set of items.

Item-specific probes:

Can you tell me in your own words what that question was asking? What kind of consequences where you thinking of?
What do you think this question is asking? What does “discipline problems” mean to you? Can you tell me in your own words how you describe fairly?

When this question refers to “this school,” who or what comes to mind? What does “effectively handle” mean to you?

We use the term “orderly environment;” what does this term mean for you in the context of your job at the school?

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for making this survey better.

Interviewer Instructions: Collect materials and escort participant out of the interviewing room and to the lobby. Please give the respondent incentive envelope [$40], thank them for their time and effort.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	111313_version
Subject	NAEP BQ
Author	Sandoval Giron, Anna
File Modified	0000-00-00
File Created	2021-01-28