National Center for Education Statistics
Attachment II
Cognitive Interview Protocol and Questionnaire Items
2022 School Crime Supplement to
the National Crime Victimization Survey (SCS:22/ NCVS)
Cognitive Interviews
OMB# 1850-0803 v.282
November 2020
2022 SCS to the NCVS
Cognitive Interview Protocol
Round 1
Cognitive Testing of NCVS SCS
PARTICIPANT ID #: _________________________ DATE: ____ / ____ / ___
INTERVIEWER’S NAME:_________________________________________________
START TIME: ______: ______ AM / PM
Hello, my name is ______ and I work for the Census Bureau. It’s nice to meet you and thank you very much for helping us out today.
IF NECESSARY, CREATE SMALL TALK TO BUILD RAPPORT WITH THE STUDENT BY ASKING A QUESTION, SUCH AS:
What is your favorite subject in school? RESPOND ACCORDINGLY (e.g., if the student responds with “science,” ask them what they are studying in their science class).
Let me start by telling you a little bit about what we're doing here. The Census Bureau administers many surveys in addition to counting people in the US every 10 years. Since so many agencies rely on data from the Census Bureau, it is extremely important that this information be as accurate as possible.
In order to help us improve our surveys, we turn to students like you to find out if our questions make sense and are easy to understand and answer. The survey we'll talk about today is conducted by the National Center for Education Statistics, which is part of the U.S. Department of Education. The research we’re doing today will help us figure out if changes we have made to the questionnaire are working for students like you, and it will help us improve the accuracy of data.
I'm going to go through the questionnaire like a real interview, but I'm mainly interested in your reactions to the questions. So after we go through the survey questions, I’m going to ask you some follow up questions about the survey questions you answered. For example, I might reread a question from the survey, and ask you what certain words or phrases in the question mean to you, or what you were thinking about when you were answering. Whenever I reread a question from the survey, I will show the question wording on the screen for you to reference. There are no right or wrong answers, we just want to know your thoughts and reactions to the questions to help us make them better so that other students like you can easily understand them. The things we will talk about in the interview today will only be used in our research to help us improve the survey. Your name will not be attached to anything you say. Direct quotes may be used in research papers and professional presentations, but your name and any names you might mention today will never be used in our reports and presentations.
I'd like to ask for your permission to audio tape the session today. The main reason we tape these interviews is so that we don't have to rely on notes or our memories later. This allows me to concentrate on what you're saying during the interview. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).
Do you have any questions about the recording?
AFTER ANSWERING QUESTIONS AND GIVING FURTHER EXPLANATION, CONTINUE. IF THE PARTICIPANT IS NO LONGER INTERESTED IN PARTICIPATING, THANK THE PARTICIPANT FOR HIS/HER TIME AND END THE INTERVIEW.
[If student is under 18]:
Your parent signed a form giving you permission to participate today. I am going to tell you about what was in that document. The document that your parent signed explains the purpose of the study and how we will protect your information. In addition to what I have just mentioned about never sharing your name or any names you say during the interview, I would like to draw your attention to the fact that:
Your participation is voluntary and you may stop the interview at any time
You can refuse to answer any of the questions asked or take a break at any time
All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).
As stated in the form, we would like to record this interview to assist in our ability to write a report summarizing the interview.
[If student is 18]:
Before we start, I want to remind you of the consent form you signed electronically.
That document explained the purpose of the study and how we will protect your information. In addition to what I have just mentioned about never sharing your name or any names you say during the interview, I would like to draw your attention to the fact that:
Your participation is voluntary and you may stop the interview at any time
You can refuse to answer any of the questions asked or take a break at any time
Your responses in this study may be used only for statistical purposes and will not be disclosed, or used, in identifiable form for any other purpose, except as required by law. (ESRA 2002, U.S.C. 20 § 9573)
As stated in the form, we would like to record this interview to assist in our ability to write a report summarizing the interview.
Thank you for signing the consent form.
ASK PARTICIPANT IF (S)HE HAD ANY QUESTIONS ABOUT THE CONSENT FORM.
[All students]:
Are you comfortable with me recording this interview?
IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED:
I’m going to turn on the recorder now, and once it is on, I’m going to ask for your permission to record today, and then we will get started.
TURN ON TAPE RECORDER, ASK FOR PERMISSION TO RECORD ON THE TAPE, AND BEGIN THE INTERVIEW.
OK, let’s begin.
INTERVIEWER INSTRUCTIONS:
Note any confusion or difficulties respondents have with the questions.
RETROSPECTIVE PROBING (R=respondent):
IF YOU NOTICED R HAVING COGNITIVE DIFFICULTY ANSWERING ANY OF THE SURVEY QUESTIONS THAT HAVE PROBES, MAKE SURE TO MAKE A NOTE OF R’S DIFFICULTY FOR LATER FOLLOW UP. WHEN ASKING THE SCRIPTED RETROSPECTIVE PROBES, BE SURE TO PROBE ON ANY OF THE RESPONDENT’S APPARENT DIFFICULTY OR HESITATION THAT YOU NOTICED.
2022 SCHOOL CRIME SUPPLEMENT (SCS)
(See Attachment II for the full text of the 2022 School Crime Supplement (SCS) to be tested. Interviewers will administer the full survey, reading questions out loud. There will be no probing during the survey. If a respondent asks for clarification during the survey, interviewers will encourage the respondent to answer the question as they would if the interviewer was not able to provide clarification. After the full survey has been administered, the interviewer will administer the probing questions in Section 5. )
[Note: Interviewers will be trained to use the scripted probes as necessary to elicit the desired information. If a student’s responses to scripted probes are not sufficient to answer the research questions, interviewers will ask additional unscripted probes in an attempt to gather more information. Similarly, if students provide ample information in response to an earlier probe, interviewers may skip subsequent probes that are asking for redundant information. When necessary, interviewers may ask a redundant probe to confirm that they correctly understood what the student previously explained.]
Research Questions:
Do the new and revised questions accurately capture the different types of schooling children are receiving?
Are all virtual students visiting the school buildings for extra-curricular activities or non-class related reasons?
Are students “virtually” learning in pods? If so, how do they answer the questions?
Probes (GENERAL SCHOOLING): [ALL RESPONDENTS]
Thank you! Now I have a few questions about some of the items you just answered.
All respondents:
First, can you tell me a little bit about how you have attended school so far this school year?
[If R doesn’t mention in person versus virtual] Did you attend classes in the school building, online, both, or somewhere else? [If needed] Can you tell me a little bit more about that?
Probes (E_ATTENDSCHOOL A-C): [ALL RESPONDENTS]
Earlier, when I asked you these questions, you said yes to ____. [SCREEN SHARE POWERPOINT SLIDE WITH E_ATTENDSCHOOL A-C TEXT]
All respondents:
How easy or difficult was it to answer these questions? [If needed] Can you tell me more about that?
Were you unsure of how to answer any of those questions? [If needed] What made you unsure? How did you decide how to answer?
[If R’s explanation of schooling in so far suggests that they answered one or more items incorrectly, probe to find out why] How did you come up with your answer to question [A/B/C]?
[If needed] Did the type of school you receive this year change?
What does the phrase “virtual or remote instruction” mean to you in question B?
Virtual respondents:
[If R said yes to E_ATTENDSCHOOL_B] How does your school refer to virtual instruction?
Research Question (Homeschooling)
Are the new follow-up questions for fully homeschooled students performing without issues?
Are students aware of the decision making process their parents/guardians went through when deciding to homeschool?
Probes (E_HOMESCHOOL_COVID): [HOMESCHOOLED RESPONDENTS]
This was another question I asked: [SCREEN SHARE POWERPOINT SLIDE WITH E_HOMESCHOOL_COVID TEXT]. Earlier, when I asked you these questions, you said ____.
Can you tell me what this question is asking in your own words?
How did you come up with your answer to this question?
[If needed] Were you unsure of how to answer this question?
[If R said yes to E_HOMESCHOOL_COVID] Did your school give you and your parents/guardians the option to choose between virtual instruction and attending classes in-person?
Probes (E_HOMESCHOOL_WHY): [HOMESCHOOLED RESPONDENTS]
This was the next question I asked: [SCREEN SHARE POWERPOINT SLIDE WITH E_HOMESCHOOL_WHY TEXT]. Earlier, when I asked you these questions, you said yes to ____.
How did you come up with your answer to this question?
Were there any that you thought about saying yes to but didn’t? [If yes and if needed] How did you decide how to answer for [that category/those categories]?
[If needed] Did you feel like you could only choose one of the reasons?
What comes to mind when you hear category 2, “Due to concerns about the school environment, such as safety, drugs, or negative peer pressure?”
[If R hasn’t mentioned this yet] When did you start homeschooling?
[If R has not always been homeschooled and hasn’t mentioned yet] Can you tell me a bit about how the decision to start homeschooling was made?
[If needed] Whose idea was it for you to start homeschooling? [If needed] Was it your idea, or was it your parent or guardian’s idea? Someone else’s?
[If R said parents made the decision and R hasn’t mentioned yet] Do you know the reason your parents/guardians decided to start homeschooling?
IF R WAS FULLY HOMESCHOOLED THIS SCHOOL YEAR, SKIP TO DEBRIEFING
Research Questions:
What “scope” are students in different scenarios (in person, virtual, hybrid, homeschool) thinking of as they answer various questions?
Do minor revisions intended to adapt wording for different scenarios perform as intended?
Do all-virtual students comment on it being weird to be asked about things that can only happen in person?
Probes (E_DIFFSCHOOL_ATTENDED): [ALL RESPONDENTS]
This was another question I asked: [SCREEN SHARE POWERPOINT SLIDE WITH E_DIFFSCHOOL_ATTENDED TEXT]. Earlier, when I asked you these questions, you said ____.
Can you tell me what this question is asking in your own words?
How easy or difficult was it to answer this questions? [If needed] Can you tell me more about that?
[If R’s response to the Q seems incorrect based on their explanation of school this year] How did you come up with your answer to this question?
Virtual and hybrid respondents:
Were you unsure of how to count your virtual instruction when answering this question?
Probes (ENVIROMENTAL QUESTIONS): [VIRTUAL AND HYBRID RESPONDENTS]
This was another series of questions I asked: [SCREEN SHARE POWERPOINT SLIDE WITH INTRO_3 AND QS 4A-4D].
How easy or difficult was it to answer these questions? [If needed] Can you tell me more about that?
Probes (Q9): [ALL RESPONDENTS]
This is another set of questions I asked: [SCREEN SHARE POWERPOINT SLIDE WITH Qs 9a-9h].
If yes to any:
Earlier, when I asked you these questions, you said yes to ____.
Can you tell me about the different activities you’ve participated in this school year?
[If needed] Are these activities organized by your school? [If no] Who organizes them?
Are these activities in person, virtual, or both?
[If R if fully virtual but participates in person, and if needed] Can you tell me more about the in-person activities the school offers for students who do all of their schooling virtually?
If no to all:
Is your school offering any in-person activities this school year? What about virtual activities?
All respondents:
[If school is offering virtual activities] Have you participated in any virtual school sponsored activities that you didn’t include here?
[If yes] How did you decide whether or not to include the different activities when answering these questions?
Probes (SCS_INTRO_4_VIRTUAL): [FULLY VIRTUAL RESPONDENTS]
This is an introduction I read you: [SCREEN SHARE POWERPOINT SLIDE WITH SCS_INTRO_4_VIRTUAL].
Can you tell me in your own words what you considered to be “during school”?
Was there anything you were unsure about whether it counted as “during school”?
[If yes] How did you decide whether or not things counted as “during school”?
Are there other parts of virtual school that we should include here?
Probes (SCS_INTRO_4_INPERSON): [IN PERSON AND HYBRID RESPONDENTS]
This is an introduction I read you: [SCREEN SHARE POWERPOINT SLIDE WITH SCS_INTRO_4_INPERSON].
Can you tell me in your own words what you considered to be “at school”?
Was there anything you were unsure about whether it counted as “at school”?
[If yes] How did you decide whether or not something counted as “at school”?
Probes (F_KNOW_DRUGS_OR_ALCOHOL): [ALL RESPONDENTS RECEIVE PROBES, ARE SHOWN APPROPRIATE VERSION OF QUESTION TEXT]
This was another question I asked:
[FULLY VIRTUAL RESPONDENTS]: [SCREEN SHARE POWERPOINT SLIDE WITH F_KNOW_DRUGS_OR_ALCOHOL_VIRTUAL TEXT].
[IN PERSON AND HYBRID RESPONDENTS]: [SCREEN SHARE POWERPOINT SLIDE WITH F_KNOW_DRUGS_OR_ALCOHOL_INPERSON TEXT].
Earlier, when I asked you these questions, you said ____.
Can you tell me in your own words what this question is asking?
If R answered yes:
Where did you see another student under the influence of drugs or alcohol?
All respondents:
Were there any times you saw another student under the influence of drugs or alcohol but weren’t sure if it counted for this question? What made you unsure if it counted?
Probes (Q22a-Q22h): [ALL RESPONDENTS]
Earlier I asked you these questions: [SCREEN SHARE POWERPOINT SLIDE WITH Q22A-Q22H TEXT].
If any Q22 a – h= yes:
When I asked you if any student from your school had done any of those things that might make you feel bad or are hurtful to you, you said yes to items _______ [SAY WHICH LETTERS R SAID YES TO, DO NOT READ BEHAVIORS. Ex – “you said yes to items b and d”].
Can you tell me about what happened? As much as you feel comfortable talking about.
[If R did not specify] Where did [this/these things] happen?
[If R did not mention cyberbullying] Did anything happen online or by text that you didn’t think of when you were first answering that question? [IF YES] Can you tell me more about that?
If Q22 = no to all:
When I asked you if any student from your school had done any of those things that might make you feel bad or are hurtful to you, you said no to all items.
Did any of those things happen to you but you didn’t think they would count for this question? Can you tell me more about why you didn’t include them here?
[If cyberbullying not mentioned in response to probe d] Did any students from school do anything that made you feel bad or was hurtful to you online or by text that you didn’t think of when answering the question? [If yes] Can you tell me more about that?
Did other students do any other things not included in this list that made you feel bad or were hurtful to you? Can you tell me more about that?
Fully virtual respondents:
What places were you thinking about when you were answering these questions?
[If needed] Do you ever see any students from your school in-person during the school day? [If needed] Can you tell me more about that?
Hybrid respondents:
What places were you thinking about when you were answering these questions?
[If no mention of virtual classes] Did any students from school do anything that made you feel bad or was hurtful to you during virtual classes that you didn’t think of when answering the question?
Probes (G_BULLY_WHERE1 through G_BULLY_WHERE9): [ALL RESPONDENTS]
Next I asked you these questions: [DEPENDING ON WHICH CATEGORIES THEY WERE ASKED ABOUT, SHOW SLIDE WITH EITHER FULL LIST (IN PERSON OR HYBRID) OR PARTIAL LIST (FULLY VIRTUAL)]
All respondents:
Does this list include all the places where other students have done hurtful things to you? [If no] Where else did they occur?
Probes G_HATE_WORDS_VIRTUAL: [ALL RESPONDENTS]
This was another question I asked: [SCREEN SHARE POWERPOINT SLIDE WITH G_HATE_WORDSVIRTUAL TEXT].
Can you tell me what this question is asking in your own words?
What types of school sponsored websites or applications could someone post hate-related words, pictures, videos, or symbols on?
If R answered yes:
When I asked you this question, you answered yes. Where did you see the hate-related post?
All respondents:
Was there anything you thought about including here but didn’t? [If needed] Can you tell me more about that?
Probes (L_SKIP_CLASSES): [VIRTUAL AND HYBRID RESPONDENTS]
Research Question:
How do schools determine attendance for virtual learning? Does skipping one zoom call constitute skipping a class in the traditional sense?
This was another question I asked: [SCREEN SHARE POWERPOINT SLIDE WITH L_SKIP_CLASSES TEXT].
How easy or difficult was it to answer this question? [If needed] Can you tell me more about that?
[If needed] Were you unsure of how to answer for your virtual classes?
How does your school count attendance for virtual learning?
[If needed] If you don’t sign in to one virtual class, does your school consider it skipping a class the same way they would if it was an in-person class?
Debriefing Questions: [ALL RESPONDENTS]
Overall, what did you think of this survey? Was it easy or difficult?
Is there anything you wanted to say about the survey questions that you haven’t had a chance to talk about yet?
Those are all of the questions that I had for you today. Thank you very much for your participation. We will be mailing out $25 for you and $25 for the adult who helped you set up the interview in the next few days.
TURN OFF THE TAPE RECORDER.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Katherine N Mark (CENSUS/ADDP FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |