Justification

National Center for Education Statistics (NCES)

School Survey on Crime and Safety (SSOCS) 2018 Cognitive Interviews 2016

Volume I – Supporting Statement

OMB #1850-0803 v.171

September 2016

Contents

Background and Study Rationale 1

Cognitive Interviews 1

Survey Items 2

Sample and Recruitment Plans 2

Data Collection Process 3

Analysis Plans 3

Consultations outside the Agency 4

Assurance of Confidentiality 4

Estimation of Hourly Burden 5

Estimate of Costs for Recruiting and Paying Respondents 5

Estimate of Cost Burden and Cost to Federal Government 5

Schedule 6

Appendixes:

Appendix A-E SSOCS 2018 Cog Labs 2016 Recruitment Materials

Appendix F SSOCS 2018 Cog Labs 2016 Protocol

Appendix G SSOCS 2018 Cog Labs 2016 Communication Materials for Testing

Appendix H SSOCS 2018 Cog Labs 2016 Questionnaire for Testing

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803), which allows NCES to conduct procedures to develop, test, and improve its data collection methodologies (e.g., cognitive interviews, focus groups, feasibility testing, etc.).

Background and Study Rationale

The School Survey on Crime and Safety (SSOCS), a nationally representative survey of elementary and secondary public schools, is one of the nation’s primary sources of school-level data on crime and safety. Conducted by the National Center for Education Statistics (NCES), within the U.S. Department of Education, SSOCS has been administered six times so far—covering the 1999–2000, 2003–04, 2005–06, 2007–08, 2009–10, and 2015–16 school years.

SSOCS is unique in that it is the only recurring federal survey collecting detailed information on the incidence, frequency, seriousness, and nature of violence affecting students and school personnel, as well as other indices of school safety from the schools’ perspective. As such, SSOCS fills an important gap in data collected by NCES and other agencies. Topics covered by SSOCS include school programs and practices, parent and community involvement, school security staff, school mental health services, staff training, limitations on crime prevention, the type and frequency of crime and violence, and the types of disciplinary problems and actions used in schools.

Principals or other school personnel designated by the principal as the person who is “the most knowledgeable about school crime and policies to provide a safe environment” are asked to complete SSOCS questionnaires. For administrations up until SSOCS 2010, completed questionnaires were received from about 2,650 public schools (with re-mailings and extensive telephone follow-up) with weighted unit response rates generally averaging around 80 percent. In SSOCS 2016, completed questionnaires were received from about 2,100 public schools for an unweighted unit response rate of 60%.

An updated SSOCS 2016 questionnaire, including approximately 40 new and modified items on school practices and programs, school security staff, school mental health services, staff training, and incidents of crime, was approved in August 2015 (OMB# 1850-0761 v.8) and fielded in spring 2016. With additional funding available, SSOCS will be conducted again in the spring of the 2017–18 school year. To the greatest extent possible, NCES would like to retain items contained in earlier collections to preserve trend lines. That said, some items should be modified to stay relevant, and new items should be considered to address new and emerging issues and interests.

As part of the SSOCS 2018 item development process, a portion of the new and modified survey items will be tested on target participants through cognitive interviews in fall 2016 to uncover comprehension issues and to measure overall understanding of the content surveyed. This document describes the types of testing we plan to conduct, the number of items to be tested, the sample and recruitment of participants, the data collection process, the hourly burden, and the cost of the testing.

Cognitive Interviews

In cognitive interviews, an interviewer uses a structured protocol in a one-on-one interview, drawing on methods from cognitive science. Cognitive interviews investigate the cognitive processes that respondents use to answer survey questions. In particular, these interviews will identify problems of ambiguity or misunderstanding in question wording. The main goal within this task is to ensure that all items included in the final survey are easily understood by respondents, with their interpretations consistently aligned with the concepts being measured.

The cognitive interviewing methods will consist of two key components: think-aloud interviewing and verbal probing techniques (these two methods are also known as concurrent and retrospective recall probing, respectively). With think-aloud interviewing, respondents are explicitly instructed to think aloud (i.e., describe what they are thinking) as they work through items. With verbal probing techniques, the interviewer asks probing questions, as necessary, to clarify points that are not evident from the “think-aloud” process, or to explore additional issues that have been identified a priori as being of particular interest. Cognitive interview studies produce qualitative data in the form of verbalizations made by participants during the think-aloud phase and in response to the interviewer probes. Both the think-aloud approach and probing techniques will be applied to all participants during the cognitive interviews.

Survey Items

Given the number of survey items to be tested, we estimate the length of the cognitive interview to be 60 minutes. NCES and its contractor for the SSOCS, the American Institutes for Research (AIR), estimate that approximately 13 items (32 sub-items) will be tested. Given the degree of item modification and section length, priority for the cognitive interviews will focus around the newly proposed items and refinements to the School Mental Health Services section. An additional set of questions will also be tested to measure participant responses to select communication materials and physical components of the survey package —the advance letter, the survey cover letter, the brochure, the FedEx envelope, and the free pen. Items may be split to test across participants so as to maximize the amount of content covered per interview. It is important to note that the research from the cognitive interviews will be iterative in that question wording and design may change during the testing period in response to early findings.

Sample and Recruitment Plans

NCES has contracted AIR to conduct the SSOCS cognitive laboratory testing in October 2016 through November 2016. Participants will include elementary, middle, and high school principals or persons designated by principals as “the person most knowledgeable about school crime and policies to provide a safe environment.” They will be recruited primarily from the District of Columbia, Chicago, San Mateo, Boston, and Austin metropolitan areas (near AIR offices) to maximize scheduling and interviewing efficiency and flexibility.

On behalf of NCES, AIR will recruit approximately 20 participants representing a range of characteristics (including schools in urban and suburban areas, schools serving different grade levels, and with varying enrollment sizes). Please note that although the sample will include a mix of characteristics, the results will not explicitly measure differences by these characteristics.

We will use multiple outreach methods and resources, such as contacts with schools and community organizations, marketing research companies, and newspaper/internet ads to recruit participants. E-mails and phone calls will be used to contact potential participants. Interested individuals will be screened to ensure they are eligible for participation in the interviews. AIR will utilize the services of marketing research companies as necessary in order to meet the target number of participants. Recruitment experiences with the cognitive interviews for SSOCS 2016 indicated that principals and primary administrative staff are a hard-to-reach population; thus, increased time and effort will be dedicated to meeting recruitment targets. Interviews will be scheduled during the beginning months of the school year in an effort not to conflict with school holiday schedules. Additionally, higher incentives will be offered to participants in the SSOCS 2018 cognitive interviews than were provided to participants in the SSOCS 2016 cognitive interviews in an effort to improve participation.

For the selected participants, AIR or a contracted marketing research company will confirm the interview date, time, and location via e-mail and phone. Participants will complete consent forms at the time of the interview and as a thank you for their time and participation, they will receive a $50 VISA or AMEX gift card for interviews conducted remotely (via telephone or video conference) or a $75 gift card for interviews conducted in person.

Data Collection Process

The cognitive interviews will take place in a variety of locations and through a range of modes. Interviews will be conducted primarily in the District of Columbia, Chicago, San Mateo, Boston, and Austin areas with facilities including (but not limited to) urban, suburban, and town schools in small, medium, and large districts. Participants will be given the option to participate in the interviews in person or remotely via telephone or video conference. The availability of the remote interview will serve to encourage participation by offering flexible options. In most cases, in-person interviews will be conducted within schools; however, in some cases participants may be invited to AIR offices where interviews will be conducted or testing may take place in another appropriate environment (depending on scheduling and the participants).

Given the diversity of locations across the country in which the cognitive interviews will be held, NCES and AIR expect that those who participate will better represent the target population of schools from SSOCS than participants sampled from the same region or city. To adequately test the survey instrument without introducing selection bias, it is also necessary to distribute the interviews across schools that represent a diverse cross section of the general population given their socio-demographic characteristics (i.e., racial makeup, percent eligible for free or reduced-price lunch, etc.).

This data collection process is consistent with the cognitive interviews that were conducted for SSOCS 2016, which utilized a multi-mode/locale strategy. Remote options allowed for reduced costs as opposed to strictly in-person interviews, which were more costly overall (due primarily to transit costs and additional staff hours for travel time). Using remote and in-person methods, cognitive interview participants were also able to be recruited from different regions and locales of the country (extending beyond the cities/states in which AIR offices were located), which resulted in a much more geographically representative sample than would have been feasible with in-person interviews only. At the same time, AIR recruiters and cognitive interviewers were able to be more flexible with the needs of participants and interviews. We anticipate similar accomplishments utilizing a similar process for the cognitive interviews this fall.

Analysis Plans

For the cognitive interview data collection, the key units of analysis are the survey items as well as select communication materials and physical components of the survey package: the advance letter, the survey cover letter, the brochure, the FedEx envelope, and the free pen. As the response rate for SSOCS 2016 was lower than expected, questions about the communication materials and physical survey package contents will be introduced to assess possible factors which may or may not contribute to the propensity of respondents and non-respondents in completing the survey. Items will be analyzed across participants and used to inform the development of these materials for the SSOCS 2018 data collection. The results of the cognitive interviews will include the following information about the questionnaire items, communication materials, and survey package components being tested:

• think-aloud verbal reports;

• responses to generic questions about respondents’ interpretation of the item or survey package components;

• responses to targeted questions specific to an item or survey package components; and

• additional comments volunteered by participants or observations noted by interviewers.

The deliverable from the analyses of the cognitive interviews will be a report which will contain results for each item and survey package component, along with a brief summary of results. This will include information from NCES’ contractor (AIR) as collected in interviewer notes. The general analysis approach will be to compile the different types of data to facilitate identification of patterns of responses or issues for specific items. This overall approach will help to ensure that the data are analyzed in a thorough and systematic way that enhances the identification of problems with items or tasks and provides recommendations for addressing them.

Consultations outside the Agency

NCES has an ongoing consultation with federal partners from the National Institute of Justice (NIJ) and the Office for Safe and Healthy Students (OSHS). Along with NCES, these partners are members of the Comprehensive School Safety Initiative Group established by NIJ. Since funding to test a web instrument for SSOCS in the spring of the 2017–18 school year is available through the NIJ, these partners shared input on priority areas that they recommend for SSOCS development. During the summer of 2016, NCES reached out to federal partners to gather feedback on proposed changes to the SSOCS 2018 questionnaire.

NCES has also been in consultation with some of the nation’s top experts in school crime, school safety, and mental health services in schools, including some of the previous SSOCS Technical Review Panel members. These experts provided insight from the perspective of researchers using the SSOCS data and from the perspective of potential respondents. In August 2016, experts provided written feedback on the content of the SSOCS 2018 questionnaire. Both the federal partners and subject matter experts were given the opportunity to review and provide suggestions to refine current SSOCS items, weigh in on recommendations for modifications or additions proposed by NCES and AIR, and propose items to address new and emerging issues and interests.

Assurance of Confidentiality

Participation is voluntary and respondents will sign a consent form before interviews are conducted. Participants will be notified that their participation is voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, (20 U.S.C. §9573)]. The consent form states the following: “I have read the attached consent form for the School Survey on Crime and Safety and associated activities; all of my questions have been answered. The above document describing the voluntary nature, response confidentiality, and procedures for this research study has been explained to me. I agree to participate.”

The interviews will be audio- or video-recorded solely for research purposes. The only identification included in the audio or video files will be the participant ID. The recorded files will be secured for the duration of the study – with access limited to key AIR project staff – and will be destroyed after the final report is submitted.

Estimation of Hourly Burden

NCES and AIR expect the cognitive interviews to last approximately 60 minutes in length. With approximately 20 participants recruited for the cognitive interviews, this will result in a maximum of 20 burden hours for the interviews. The initial contact and screening of potential participants is estimated at 3 minutes, or 0.05 hours. We anticipate 12 screening interviews will be needed to identify one eligible participant (thus an estimated 240 screenings to yield 20 participants). This will result in 12 hours of burden for the screener and an estimated total of 32 hours of respondent burden for this research.

NCES’s contractor (AIR) will be responsible for recruiting and screening cognitive interview participants. AIR will utilize the services of marketing research companies as necessary in order to meet the target number of participants.

Table 1. Estimation of hourly burden for SSOCS cognitive interviews

	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment (initial contact & screening)	240	240	0.05	12
Cognitive Interviews	20	20	1	20
Total	240	260	-	32

Estimate of Costs for Recruiting and Paying Respondents

Marketing research companies, such as Nichols Research, charge a management fee, in addition to a recruitment fee, for every participant obtained through their networks.

To encourage principals’ participation and to thank them for their time and effort, all participants will be offered an incentive. For the SSOCS 2016 cognitive interviews, participants were offered a $25 VISA gift card for interviews conducted remotely (via telephone or video conference) and a $40 gift card for interviews conducted in person. As discussed in the Sample and Recruitment Plans section above, we had difficulty reaching and recruiting principals at that incentive level and for the SSOCS 2018 cognitive interviews we would like to offer participants a $50 gift card for interviews conducted remotely (via telephone or video conference) and a $75 gift card for interviews conducted in person. We expect the higher incentive to allow us to contact fewer principals in order to reach our target number of participants, and thereby to minimize the total recruitment burden on principals for the SSOCS 2018 cognitive interviews.

Estimate of Cost Burden and Cost to Federal Government

The estimated cost to the federal government for the cognitive interviews, including development of a detailed protocol, a revised data collection instrument, the justification package, recruitment, data collection, data analysis, and preparation of a report on findings is $80,000.

Schedule

Table 2 (below) depicts the high-level activity schedule pertaining to the SSOCS 2018 cognitive interviews, including recruitment and scheduling, data collection, analyses, and reports.

Table 2. Schedule of activity milestones for SSOCS

Activity	Start Date	End Date
Recruitment & scheduling of cognitive interviews	10/14/16	11/30/16
Cognitive interviews (data collection)	10/14/16	11/30/16
Draft cognitive interview lab report	11/30/16	12/16/16
Final cognitive interview report & submission	By early 2017	By early 2017

NOTE: Dates are slightly flexible and may change at project staff discretion.