Justification

National Center for Education Statistics

Volume I

Supporting Statement

2019-2020 National Teacher and Principal Survey (NTPS)

Cognitive Interviews

OMB# 1850-0803 v.218

November 2017

Attachments:

Attachment 1 – Recruitment Advertisements

Attachment 2 – Recruitment Protocols with Eligibility Screeners

Attachment 3 – NTPS Cognitive Interviews FAQs

Attachment 4 – Participant Consent Forms

Attachment 5 – Cognitive Interview Protocols

Attachment 6 – Questionnaires to be Cognitively Tested

Submittal-Related Information

The following materials are being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, usability testing, and cognitive interviews.

Background

The National Teacher and Principal Survey (NTPS), conducted every two years by the National Center for Education Statistics (NCES), is a system of related questionnaires that provides descriptive data on the context of elementary and secondary education. Redesigned from the Schools and Staffing Survey (SASS) with a focus on flexibility, timeliness, and integration with other ED data, the NTPS system allows for school, principal, and teacher characteristics to be analyzed in relation to one another. NTPS is an in-depth, nationally representative survey of first through twelfth grade public school teachers, principals, and schools. Kindergarten teachers in schools with at least a first grade are also surveyed. NTPS utilizes core content and a series of rotating modules to allow timely collection of important education trends as well as trend analysis. Topics covered include characteristics of teachers, principals, schools, teacher training opportunities, retention, retirement, hiring, and shortages. The 2019-20 national data collection will be the third administration of NTPS (SASS was conducted from 1987 until 2011, and NTPS was conducted in 2015-16 and 2017-18). The NTPS teacher, principal, and school questionnaires each include core sections of content that are included in every survey administration. NTPS questionnaires also include modular, rotating content that is fielded in selected administrations, usually every other survey cycle. Although the modular survey content varies by administration, survey questions and sections are presented in a logical order and organized to maximize clarity. This request is to conduct up to four rounds of cognitive testing of the 2019-20 NTPS items that are part of the teacher, principal, and school questionnaires.

Most of the rotating question modules that were first fielded in the surveys’ 2015-16 administration will again be fielded in the 2019-20 administration. The purpose of the cognitive testing study is to retest and further refine the content of the rotating question modules, including identifying and correcting issues with question wording, organization, order, and web design and presentation. In addition to cognitive testing, edit and imputation data and internet paradata collected as part of the 2015-16 administration of these question modules will be used to inform changes and refinements for the 2019-20 NTPS. Data from the 2017-18 NTPS will also allow us to evaluate problematic core module questions requiring modifications. The rotating modules that were first fielded in the 2015-16 administration and that will be evaluated through the cognitive testing study described here include: “Teacher Working Conditions,” “Early Career Experiences,” and “School Climate and Attitudes” on the teacher questionnaire; “Working Conditions and Principal Perceptions” on the principal questionnaire; and “School Staffing” on the school questionnaire.

In addition to these question modules, a series of three new items on sexual orientation and gender identity will be included for testing in the demographic sections of the public principal and public teacher questionnaires. The private teacher and private principal questionnaires will only include one standard binary question on gender, and no questions about sexual orientation. Another new item will be added in the teacher and principal Contact Information Sections that asks them to consent to receive text messages on the Cell Phone Number they provide in the same section. The text messages are to serve as contact reminders for the Principal Follow-Up Survey (PFS) and Teacher Follow-Up Survey (TFS) that will be administered in 2021. Finally, the following two additional items will be included for testing in the School Climate and Teacher Attitudes Section of the teacher questionnaire:

• In general, would you say that your health is: Excellent, Very good, Good, Fair, or Poor?

• On average, how many hours of sleep do you get in a 24-hour period?

This request is to conduct four rounds of cognitive interviews with teachers and school staff and three rounds of interviews with principals from primary and secondary public and private schools to evaluate the new and revised items for the 2019-20 NTPS Teacher, School, and Principal questionnaires, respectively. The cognitive interviews will enable the NTPS team to identify problems with question wording and suggest revisions to problematic questions.

Study Design, Context, and Respondent Burden

The following versions of the 2019-20 NTPS instruments will be evaluated using cognitive testing:

• Teacher Questionnaire,

• Principal Questionnaire, and

• School Questionnaire (public and private questionnaire versions).

Cognitive testing will be used to evaluate wording and content of both core and rotating question modules that were previously included in the 2015-16 administration. For the majority of questions, testing will evaluate the same wording as included in the 2015-16 instruments, or wording that is modified only slightly based on expert review prior to testing. The cognitive testing will use an iterative process. Data from the interviews will be analyzed after each round to identify any issues and each questionnaire and its items will be revised accordingly and then tested in the subsequent round.

Rounds 1 (where applicable) and 2 of testing will be conducted using paper instruments. Rounds 3 and 4 will utilize paper and internet instruments to further evaluate any unresolved issues from the initial rounds of testing and to also assess internet navigation and design issues. Both types of testing will utilize one or more of the following techniques: a think-aloud protocol to identify any comprehension or usability concerns, retrospective probing and debriefing, audio and/or video recording, and observation notes as recorded by the interviewer, when required.

To test these instruments, we will recruit teachers, principals, and school staff members who in a national NTPS administration would be responsible for filling out their respective questionnaire in primary and secondary public and private schools. Exhibit 1 summarizes the estimated response burden associated with recruiting for and conducting the cognitive interviews. We will strive to recruit a diverse group of educators who range in age and years of experience, and who represent schools of varying sizes and demographic characteristics. Across interview rounds, our goal is to conduct 50 interviews with teachers, 35 with principals, and 36 with school staff, for a maximum of 121 interviews. From prior experience, we anticipate needing to screen approximately two teachers or school staff for each cognitive interview of these types, and approximately five additional principals to yield the desired number of completed interviews for each round (during the 2017-18 NTPS Portal Usability Testing, in response to our recruitment advertisements for the study, we received far fewer calls expressing interest in participation from principals as compared to teachers and school staff. However, of the principals that did call, many were eligible for participation based on the study screener). For the 2019-20 NTPS Cognitive Interview study, we anticipate needing to screen a total of 222 individuals in order to yield 121 completed interviews (accounting for ineligible respondents and cancellations), for a total respondent burden of 195 hours.

Exhibit 1. Estimated Respondent Burden by Round of Interview, Instrument, and School Characteristic

Respondent Type	Recruitment Time Estimate (minutes)	Respondents Screened per Round				Total # of Respondents	Total Burden Hours
		Round 1	Round 2	Round 3	Round 4
Teacher	10	20	30	30	20	100	17
Principal	10	0†	15	20	15	50	9
School Staff	10	16	16	24	16	72	12
Recruitment Subtotal						222	38
Respondent Type	School Type	Respondents Interviewed per Round (estimated duration of the interview)				Total # of Respondents	Total Burden Hours
		Round 1 (60 min)	Round 2 (60 min)	Round 3 (90 min)	Round 4 (90 min)
Teacher	Primary	5	8	8	5	26	33
	Secondary	5	7	7	5	24	30
Subtotal		10	15	15	10	50	63
Principal	Primary	-	5	7	5	17	23
	Secondary	-	5	8	5	18	25
Subtotal		0†	10	15	10	35	48
School Staff	Small (<500 students)	4	4	6	4	18	23
	Large (500+ students)	4	4	6	4	18	23
Subtotal		8	8	12	8	36	46
Interview Subtotal						121*	157
Total for the Study (343 total responses)						222	195

^† Principals will not be interviewed during the first round of testing.

* Duplicate counts of the same respondents contribute to the total number of responses but are not included in the total number of respondents.

Because education is heavily influenced by state policy, we will recruit and interview educators from a number of states including, but not limited to, the District of Columbia, Maryland, Pennsylvania, Virginia, and West Virginia. Respondents will be recruited using educational website ads, listservs, Craiglist, other professional networks, and personal networks. Characteristics for which we will attempt to recruit at least one teacher or principal include:

• Teachers with alternative certification;

• New teachers (those with less than five years of teaching);

• Teachers and principals from schools that include all grades K through 12; and

• Teachers who are employed full or part-time.

See Attachment 1 for the language to be used in recruitment advertisements and for a sample recruitment script.

Researchers from the Center for Survey Measurement (CSM) at the U.S. Census Bureau will carry out the cognitive interviews. The interviews will be conducted either (1) in person at Census headquarters in Suitland, MD; (2) at a secure location that is convenient for the respondent, such as a library meeting room; or (3) in the event we cannot recruit enough respondents for in person interviews, virtually over Skype. Between five and ten staff members will be trained to conduct the interviews to allow the greatest flexibility possible in scheduling. Interviewers will vary somewhat in their level of experience, however all will be required to demonstrate strong understanding of the interview protocol before beginning interviews. With respondent permission, the interviews will be audio- and/or video-recorded so that the responses may be reviewed as needed during analysis. The CSM project lead will conduct interviewer training, supervise staff, and monitor data quality throughout the field period.

During Rounds 1 and 2 of testing, respondents will be asked to complete a subset of current NTPS core sections of a paper version of the questionnaire for which they were recruited (teacher, principal, or school). For Rounds 3 and 4, most respondents will be interviewed using a web instrument while about one-third of respondents per questionnaire type (teacher, principal, or school) will still use a paper version of the instruments. Respondents will be trained on the think-aloud technique and will be instructed to complete portions of the instrument while thinking aloud so that their thought process can be recorded when they are first answering the questions. The interviewer will observe the respondent as they complete the survey and record any relevant information on their behavior, including any demonstrated difficulties. If needed, the interviewer will spontaneously probe on any problems or hesitation respondents may experience with particular questions. After the respondent has completed the survey, the interviewer will then review the instrument with the respondent using scripted and, when relevant, spontaneous probes to address any additional unforeseen issues that arose for the respondent (the cognitive interview protocol is provided in Attachment 5). The questionnaire items will be iteratively tested within each of the four rounds, with the protocol and question wording being adapted for the next round based on the results from the previous round.

Cognitive interviewing techniques allow researchers to evaluate potential response error and to assure that the survey provides valid data. In general, the goal of cognitive testing is to assess the respondents’ comprehension of the questionnaire items, including question intent and the meaning of specific words and phrases in the items. Data from cognitive interviews can identify potentially problematic survey questions that are not understood as intended. This testing can also examine the respondents’ retrieval of relevant information from memory, decision processes involved with answering a question, and questions that are difficult to answer due to cognitive burden.

The cognitive interviews will assess issues such as:

• The respondent’s understanding of terms in the survey;

• How confident the respondent is in their response;

• How they recalled the information they provided in factual questions;

• Whether they found a response choice that fit their answer;

• How easy or difficult it is for the respondent to answer each question;

• Issues with sensitive questions; and

• Consistency of answers within the questionnaire and in comparison to the expected range of answers.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that all of the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Participants will also be advised that the interview will be recorded and that the audio recording will only be reviewed for the purposes of report writing. Lastly, participants will be advised that direct quotes may be used in research papers and professional presentations, but names will never be attributed to anything a respondent says.

Participants will be assigned a unique participant identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant in any way or form. The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the cognitive testing study and will be destroyed after the final report is released. A consent form for each questionnaire is provided in Attachment 4.

Estimate of Costs for Recruiting and Paying Respondents

To help motivate their participation and to thank them for their time, effort, and inconvenience, teachers will be offered $40 for participation. Because principals are a small population that is difficult to recruit for participation, especially as their schedules are often filled with school-related activities, as in testing of the 2017-18 principal questionnaire, they will be offered $75 to participate. With regards to testing the school questionnaire, a $40 incentive was offered to school staff capable completing that survey during testing of the 2017-18 NTPS content, but CSM experienced great difficulty recruiting participants for that aspect of testing. At the center of the problem is that often the only staff members that are qualified to answer the school questionnaire are principals, vice principals, and/or assistant principals and to recruit them successfully, we need to use the same approach as for recruiting principals for testing the principal questionnaire. We will thus offer the same amount to prospective participants in testing the school questionnaire as in testing the principal questionnaire, an incentive of $75. Participants in cognitive testing of each of the three questionnaires will be paid in cash after completion of the interview.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to the Federal Government

The cost of conducting the cognitive interviews will be $372,000, under the NCES contract to CSM at the U.S. Census Bureau.

Project Schedule

Recruit participants through networks and advertisements	November 2017-July 2018
Conduct round 1 of cognitive testing of teacher and school questionnaires	December 2017-January 2018
Revision to teacher and school questionnaire item wording	January 2018
Conduct rounds 2-4 of cognitive testing of teacher, school, and principal questionnaires	January 2018- July 2018
Analysis of cognitive testing results and final recommendations	July 2018
Develop final wording for the 2019-20 NTPS questionnaires	August 2018