Justification

National Center for Education Statistics

Volume I

Supporting Statement

2017 School Crime Supplement to

the National Crime Victimization Survey (SCS:17/ NCVS)

Cognitive Interviews

OMB# 1850-0803 v.154

Attachments:

Attachment I – Recruitment Advertisements

Attachment II – Recruitment Protocol and Eligibility Screener

Attachment III – Sample Consent Forms

Attachment IV – Cognitive Interview Protocol and Questionnaire Items

April 2016

Submittal-Related Information

The following materials are being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting testing such as pilot tests, focus groups, and cognitive interviews.

This request is to conduct recruitment and cognitive interviews designed to evaluate new and revised questionnaire items for the 2017 School Crime Supplement to the National Crime Victimization Survey (SCS:17/NCVS). This package presents the question wording to be tested and describes plans and procedures for conducting the cognitive interviews. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (20 U.S. Code, Section 9543).

Background

The School Crime Supplement (SCS) to the National Crime Victimization Survey (NCVS) was co-designed by the National Center for Education Statistics (NCES) and the Bureau of Justice Statistics (BJS). The SCS collects data on school-related topics, including alcohol and drug availability, fighting, bullying and hate related behaviors, and fear and avoidance behaviors from students age 12 to 18 in U.S. public and private elementary, middle, and high schools. To date, the SCS was conducted in 1989, 1995, 1999, 2001, 2003, 2005, 2007, 2009, 2011, 2013, and 2015.

In 2013, the NCES and BJS initiated the revision of the SCS. Changes to the supplement included adjustments to question wording to be consistent with the Centers for Disease Control and Prevention’s (CDC) definition of “bullying,” including “cyber-bullying,” as well as changes to wording based on other research. The CDC modified its definition of bullying to include two additional components: repetition and a power differential. A technical review panel was convened to recommend changes that could meet the CDC’s definitional requirements. The 2015 SCS questionnaire included two versions of the question used to measure the rate of bullying. One version maintained the original question on bullying (consistent with 2013 SCS question to help preserve the time trend) and added two new follow-up questions to capture the two new components. In the other version, respondents were asked a single new question on bullying that included the two new components. Initial analyses of the two versions of bullying questions used in the 2015 SCS produced differing rates of bullying. In an effort to better understand the different components of bullying included in the CDC definition, the NCES, BJS, and Census agreed that more testing is needed in order to revise certain bullying questions for the 2017 SCS.

Cognitive interviews will be conducted to evaluate the new items for the SCS on the topic of bullying in schools. The cognitive interviews will enable the team to identify problems with question wording and suggest revisions to problematic questions.

Study Design, Context, and Respondent Burden

Exhibit 1 summarizes the number of interviews to be conducted within each round, as well as the recruitment burden. We will only test one version of the survey in each round since the proposed questions for this questionnaire only include two new questions; all other questions were included in the 2015 SCS. The cognitive testing will use an iterative process. Data from the interviews will be analyzed after each round to identify problematic questions. Those questions will be revised and tested in the subsequent round.

To test these instruments, we will recruit 30 students ages 12-18 who currently attend public or private middle schools and high schools. We will strive to recruit a sample with diverse demographic characteristics and school levels (middle school and high school). In addition to demographic and school level characteristics, we aim to recruit a sample of students who have experienced bullying at school and students who have not.

The interviews will be 60 minutes in length and will allow time to complete the SCS survey and administer a set of in-depth retrospective probes of the questions in the bullying section. From prior experience, we anticipate needing to recruit approximately 80 individuals to yield the 30 desired interviews to account for ineligible respondents and cancellations.

Exhibit 1. Estimated Respondent Burden by Round of Cognitive Interviews,

Response Type	Respondents per Round			Total # of respondents	Total # of responses	Time Estimate (minutes) per respondent	Total burden hours
	Round 1	Round 2	Round 3
Student recruitment	27	27	26	80	80	10	14
SCS Questionnaire
Middle School Students	5	5	5	15	15	60	15
High School Students	5	5	5	15	15	60	15
Total	27	27	26	80	110	--	44

Cognitive interview participants will be recruited using partnerships with student organizations in Maryland, Virginia, and the District of Columbia; through listservs; and through personal networks. See Attachments I and II for the language to be used in recruitment advertisements and for a sample recruitment script. The respondents will receive materials by email or on paper if their interview is scheduled in person, to remind them of their interview time and any other directions or information they need.

Researchers from the U.S. Census Bureau will conduct the cognitive interviews. Between five and ten staff members will be trained to conduct the interviews to allow the greatest flexibility possible in interview scheduling. All interviewers will be required to demonstrate strong understanding of the interview protocol before beginning interviews. With respondent permission, the interviews will be audio-recorded so that the responses may be reviewed as needed during analysis. The project leader from the Center for Survey Measurement (CSM) staff will conduct interviewer training, supervise staff, and monitor data quality throughout the data collection period.

We have planned for three rounds of cognitive interviewing. During each round, interviewers will administer the survey through the section on bullying (see the cognitive interview protocol in Attachment IV). The interviewer will observe the respondent as they answer the survey questions, noting any questions or problems that the respondent has with particular items. After completing these sections, respondents will be asked a set of probing questions about the items on bullying, and probe on any questions or problems the respondent had when answering the main survey questions. Interviewers will then ask a subset of the remaining survey questions in the SCS, and a set of debriefing questions. These items will be iteratively tested within each of the three rounds, with the protocol and question wording being adapted based on the results from the previous round.

Cognitive interviewing techniques allow researchers to evaluate potential response error and to assure that the survey provides valid data. In general, the goal of cognitive testing is to assess the respondents’ comprehension of the questionnaire items, including question intent and the meaning of specific words and phrases in the item. Data from cognitive interviews can identify potentially problematic questions that are not understood as intended. This testing can also examine the respondents’ retrieval of relevant information from memory, decision processes involved with answering a question, and questions that are difficult to answer due to cognitive burden. The cognitive interviews will assess issues such as:

• The subject’s understanding of terms in the survey

• How confident the subject is in their response

• How they remembered the information they provided in factual questions

• Whether they found a response choice that fit their answer

• How easy or difficult it is to answer a question

• Issues with sensitive questions

• Consistency of answers within the questionnaire and in comparison to the expected range of answers

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and their responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002; 20 U.S. Code, Section 9573). Participants will also be advised that the interview will be recorded and that the audio recording will only be reviewed for the purposes of report writing. Lastly, participants will be advised that direct quotes may be used in research papers and professional presentations, but names will never be attributed to anything a respondent says.

Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all materials together. The respondent ID will not be linked to the respondent in any way or form. If respondents are under the age of 18, their parents will be provided with a parental consent form; respondents who are 18 will be provided with the standard consent form used for adults (see Attachment III). The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study.

Estimate of Costs for Recruiting and Paying Respondents

To encourage participation and thank them for their time and effort, $25 will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site they will also receive $25 cash as a thank you for their time and effort and for allowing their child to participate in the study.

Cost to the Federal Government

The cost of conducting the cognitive interviews will be $65,000, under the NCES contract to CSM at the U.S. Census Bureau.

Project Schedule

Recruit participants through networks and advertisements	April – June 24, 2016
Conduct Round 1 cognitive testing	May 9 – 13, 2016
Iterative revisions to item wording	May 16 – 27, 2016
Conduct Round 2 cognitive testing	May 31 – June 3, 2016
Iterative revisions to item wording	June 6 – 17, 2016
Conduct Round 3 cognitive testing	June 20 – 24, 2016
Analysis and Final Recommendations	June 27 – July 1, 2016
Final Wording for Questionnaire	July 1, 2016
Final Report	August 1, 2016