2020-2021 National Teacher and Principal Survey (NTPS) Cognitive Interviews

National Center for Education Statistics

2020-2021 National Teacher and Principal Survey (NTPS)

Cognitive Interviews

OMB# 1850-0803 v.259

Attachment 5

Cognitive Interview Protocol

November 2019

Section 1: Introduction

Participant ID #: |___|___|___|__|__| __|___|___|

Interview Date: |__|__| / |__|__| / |__|__|__|__| (mm/dd/yyyy)

Interviewer initials: _____

Start Time: ____________ AM / PM End Time: ____________ AM / PM

Welcome participant and thank them for their time.

Read or paraphrase: Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study. [IF OBSERVER PRESENT]: I also want to let you know that this interview is being observed by [DESCRIBE OBSERVER’S ROLE: e.g., someone who works on the NTPS]. They are observing the interview because they are also interested in how the questions work for people like you.

We are working with the National Center for Education Statistics, within the U.S. Department of Education, to develop and refine questions for the National Teacher and Principal Survey. Before surveys are conducted, it’s important to test the questions with the help of people such as yourself.

It is important that the questions make sense, are easy to answer, and that everyone understands the questions the same way. If you agree to take part in this study, I will hand you a questionnaire to complete. After you complete it, I will ask you some questions about the answers you gave. There are no right or wrong answers. In this study, I am less interested in your actual answers to the questions and more interested in what you thought about the questions and how you understood them. After you have gone through the questionnaire, I will ask some questions about your answers, such as what you understood the questions to mean and whether the questions reflect your actual circumstances.

In order to concentrate on what you are saying rather than having to take notes, and to make sure that I don’t miss anything you say, I would like to audio record this interview. Is that okay with you? [IF NOT OKAY WITH RECORDING]: Okay, that’s no problem.

I want you to know that your participation is voluntary and all of your responses are protected from disclosure by federal statute. I want you to know that you can speak freely. Also, you do not have to answer any question you don’t want to and can stop the interview at any time if you wish.

Your participation in this interview is very important because it will help us improve the questionnaire. We’re scheduled to be here for about [60] minutes, and I won’t keep you any longer. As a thank you for your help, we will give you [$75] cash after the interview is completed.

Do you have any questions?

[Write down questions]

Section 2: Informed Consent

Before we start, I’m going to give you a written consent form. This document explains the purpose of the study, how we will protect your information, and who you can call if you have questions.

[Hand the Informed Consent document to the participant to read.]

In addition to the point I have just mentioned about confidentiality I would like to draw your attention to the fact that:

• Your participation is voluntary and you may stop the interview at any time

• You can refuse to answer any of the questions asked or take a break at any time

• All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151)

[When completed:

1. Ask participant if (s)he has any questions.

2. Ask participant if (s)he agrees to participate and agrees to be audio recorded.

3. If participant agrees, obtain the respondent’s signature to proceed – on two (2) copies.

4. Make sure respondent takes one copy with them when the interview is over.

5. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED:] I will now turn on the audio recorder.

Section 3: Cognitive Interviewing for Paper Mode

This is the School Questionnaire of the National Teacher and Principal Survey. If this were the real survey, you would receive this questionnaire from a school coordinator or in your mailbox at your school. It would have your name and school address in the address label.

I am going to ask you to fill out sections of this questionnaire just as you would if you had received it at school but with one difference.

I would also like you to think aloud as you answer the questions. While I am interested in your answers, I am even more interested in the process you go through when you answer the questions. So, as much as possible, I’d like you to think aloud for me; meaning, talk me through your thought process as you answer each question. From time to time, I may also ask you what you are thinking to remind you to do this.

I find it helps people to get the idea of thinking aloud while they are answering a question if they do a practice question first. Let’s begin with a practice question. Remember to think aloud as you answer.

How many windows are there in the house or apartment where you live?

_____________ WINDOWS

[IF NEEDED: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

PROBES:

• How did you come up with that answer?

• Tell me more about that. Why did you say [ANSWER]?

• I noticed that you hesitated. Tell me what you were thinking.]

As you complete the questionnaire, I may stop you in a couple of places to skip you to the next set of questions to answer. After you complete some sections, I am going to ask you about some specific questions that you answered.

Before we begin, do you have any questions about the process?

[Let respondent begin modules we are testing. In addition to the scripted probes that are included for each questionnaire as shown in Section 5, additional general probes that may be used as needed are presented below:]

• What did you think this question was asking?

• Can you rephrase the question in your own words?

• How did you go about answering this question?

• What were you thinking about when you answered this question?

• How did you decide on your answer?

• What was the reason for that answer?

• How easy or difficult was it to answer this question?

• What is an example?

• How did you feel about answering this question?

• What does this term or phrase mean to you?

• Can you tell me more about that?

• You seemed to hesitate / take a moment before providing your response. Can you tell me what you were thinking about?

• What do you mean by…?

• Is there any additional information that we should have provided to help you answer that question?

• Does this term / question make sense to you, or is there something else that would make more sense?

Section 4: Debriefing Procedures

• Ask for any last observations / comments / concerns.

• Stop the digital audio recorder.

• Thank the participant for coming in and for helping to test the survey.

• Give the $75] incentive and their copy of the consent form, have the respondent sign the voucher, and escort them from the interview room.

Section 5: Scripted Probes

School Survey

ask after they complete section 1

[If needed to verify:] Does your school offer any virtual or online classes?

If they have online classes

Please tell me more about the set up of online classes for your school?

[If needed] Who teaches the classes?

What are the responsibilities of the online teachers?

Who employs the online teachers?

Who evaluates them?

Who provides the curriculum for online classes?

If they say the teacher: Does the teacher create the curriculum themselves or use something provided by another entity?

Are these classes in certain subjects that students need to graduate or are the classes offered just for additional advancement?

What kinds of classes or subjects can students take online?

[If needed] Do the classes use live sessions, pre-recorded video, text-based, or a combination?

[if any video] Can you tell me more about the videos that students have access to? How are they set up? Are the recordings or sessions typically done by the teachers assigned to the class? Or are the videos recorded from different people?

How are students expected to turn in assignments or work on exams?

How are students expected to contact the teachers, if needed?

[for blended/brick-and-mortar] How is the classroom physically set up for online class?

Are there any other staff who assist with online students? [If yes] What is their role?

Please tell me more about what groups of students can take online? Is there a particular subset of students who are more likely to take classes online?

[if blended/brick-and-mortar school] How are the online classes offered at your school managed?

Question 1-7

Only ask if they appear to struggle or mark something other than “Regular School”

• Can you tell me more about your answer?

Ask everyone

• Did you notice the apple instruction under “Alternative/ Other school?”

• Can you tell me in your own words what you think the apple instruction is saying?

Q1-12

• Can you tell me more about your answer to question 1-12?

• Can you tell me in your own words what this question is asking?

• Would it have made more sense if we said “in person” rather than face to face?

Question 1-13&1-14

Only ask if they appear to have contradicted the information they gave at the beginning of probing.

• Can you tell me more about your answer to question 1-13?

Question 1-15

• Now let’s turn to 1-15. Can you tell me more about how you came to your answer?

• Can you tell me in your own words what this question is asking?

Question 1-16

• Now let’s turn to 1-16. Can you tell me how you came up with your answer to this question?

• Did you struggle with which boxes you should mark?

Now I am going to ask you to complete section 2 remembering to think aloud while you do.

AFTER SECTION 2 ASK

Question 2-2

• How did you decide on your answer?

• Were there any parts of this question that you had trouble answering?

Question 2-4

• How did you come to your answer for this question?

• What does “this school’s hiring authority” mean to you in this question?

• About how many teaching vacancies were there in this school?

• Can you tell me more about your answers to this question?

• What timeframe were you thinking of when you answered this question?

Now please complete the survey, remember to think aloud.

Question 4-2

• Now turning to question 4-2, Was this question easy or difficult to answer?

• Is this sort of information kept for all IEP students at this school?

• Did you have any students you struggled to classify in this question?

After they complete the questionnaire hand them the appropriate school coordinator letter and TLF.

Now I would like you to read this letter, remembering to think aloud, but rather than having you do the task I would like you to tell me what you would do next.

[If needed] What staff would you include on the TLF?

Would you include your online teachers?