Volume I 2015-16 NTPS Cog Labs

National Center for Education Statistics

Volume I

Supporting Statement

2015-16 Schools and Staffing Survey (SASS) / National Teacher and Principal Survey (NTPS) Cognitive Interviews

OMB# 1850-0803 v.89

Attachments:

Attachment I – Draft SASS 2012 – NTPS 2015-2018 Crosswalk

Attachment II – Recruitment Protocol and Eligibility Screener

Attachment III – Sample Consent Form

Attachment IV – Cognitive Interview Protocol and Questionnaire Items

November 13, 2013

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews.

The request for approval described in this memorandum includes recruiting and screening activities and cognitive testing of the questionnaire items to be used in the 2015-16 National Teacher and Principal Survey (NTPS), which is a redesign of the NCES Schools and Staffing Survey (SASS). This package provides information about plans and procedures for conducting the cognitive testing, and presents the question wording to be tested in the cognitive interviews.

Background

The National Center for Education Statistics (NCES) sponsors the Schools and Staffing Survey (SASS), which has been the U.S. Department of Education’s (ED’s) primary source of information on what is happening in K-12 schools from teachers’ and administrators’ perspectives, since the survey began in 1987. In 2012, NCES initiated the redesign of SASS. NCES’s vision for the redesigned SASS, named the National Teacher and Principal Survey (NTPS), is for a highly flexible, timely collection that is integrated with other ED data. To that end, the new NTPS will include modular content that can be easily swapped in and out of the survey, will be fielded more frequently than SASS was fielded, and will provide data users with relevant extant data rather than burdening respondents with answering questions that have been asked in other ED collections. The SASS/NTPS will be fielded for the first time in 2015-16.

The content of the 2011-12 SASS forms the basis of the NTPS content, though many questions will be shifted to different questionnaire instruments or will be answered through the use of extant data sources. Teacher instruments will remain relatively similar to previous teacher instruments. Principal instruments, however, will include question items that were formerly fielded as separate school and district-level questionnaires. Several SASS items measuring school characteristics will be provided to data users from EDFacts data, Civil Rights Data Collection (CRDC) data, or Private School Universe Survey (PSS) data but will no longer be collected directly in the NTPS.

Cognitive interviews will be conducted to evaluate new items for the NTPS and to test the reconfigured question order on the new instruments. The cognitive interviews will enable the team to identify problems with items and with the organization and order of the instruments. The interviews will also allow us to evaluate how principals respond to questions about their schools and school districts in the context of an instrument that also asks about the principals’ experiences and characteristics.

Study Design, Context, and Respondent Burden

Four instruments are being prepared for cognitive testing:

• Teacher Questionnaire (public and charter school version)

• Teacher Questionnaire (private school version)

• Principal Questionnaire (public and charter school version)

• Principal Questionnaire (private school version)

Each instrument has Core sections with content that will be included in every survey administration. In addition, the instruments include rotating modules that will be fielded in selected rounds. Although from a procedural perspective these will be considered distinct sections for survey administrators, the respondents will receive instruments in which the items are grouped in a logical order that is organized for greatest clarity.

Exhibit 1 summarizes the number of interviews to be conducted with each instrument, as well as the recruitment burden. We are developing two versions of the teacher questionnaire and two versions of the principal questionnaire (one version for public or charter schools and one for private schools). To test these instruments, we will recruit teachers and principals from three types of schools (public, private, and charter) representing both elementary and secondary schools. We will strive to recruit both urban and suburban schools, educators from a variety of backgrounds who range in age and years of experience, and to select schools that vary across size and demographic characteristics. Although we do have specific recruitment goals for age, years of experience, school size, and demographic composition of school, we want to recruit educators from a variety of backgrounds to assure that the instruments are effective for a wide range of respondents. A maximum of 164 interviews will be completed. Our goal is to obtain 80 interviews with teachers and 84 interviews with Principals. In the first set of interviews, 45 minutes will allow time for think aloud responses to in-depth probes of key question items. In the second set, the time allotted will allow participants to review the entire survey and provide responses to retrospective probes.

Exhibit 1. Expected Burden and Breakdown of SASS Cognitive Interviews, by Instrument, Population, and Location
	school type	time estimate (minutes)	# of rounds	interviews per round in geographic area		Total # of respondents	Total burden hours
				Washington, DC	Chicago, IL
Teacher recruitment	all	10	1	-	-	120	20
Principal recruitment	all	10	1	-	-	140	24
subtotal						260	44
Teacher items in need of testing	public	45	2	4	4	16	12
	charter	45	2	3	3	12	9
	private	45	2	4	4	16	12
subtotal						44	33
Principal items in need of testing	public	45	2	5	5	20	15
	private	45	2	5	5	20	15
	charter	45	2	2	2	8	6
subtotal						48	36
Full teacher questionnaire	public	60	2	3	3	12	12
	charter	60	2	3	3	12	12
	private	60	2	3	3	12	12
subtotal						36	36
Full principal questionnaire	public	60	2	3	3	12	12
	private	60	2	3	3	12	12
	charter	60	2	3	3	12	12
subtotal						36	36
Total Burden (424 total responses)						260	185

Cognitive interview respondents will be recruited from elementary and secondary schools in the Washington, DC and Chicago, Illinois areas. Because education is heavily influenced by state policy, we will recruit teachers and principals from a diversity of states as a goal. In the D.C. region we will recruit from schools in Maryland, Virginia, and D.C. In the Chicago area, we will attempt to recruit teachers and principals from Wisconsin and Indiana in addition to Illinois. We will use the Common Core of Data and the Private School Universe Survey to identify schools in these areas. We will purposively select schools that meet the sample characteristics defined above. Although we are constructing a non-probability sample for the cognitive testing, we will select schools that as a group reflect the variety of school types of interest in the study.

To complement the school-based recruitment, we will also utilize personal and professional networks as well as advertisements on www.craigslist.org and education listserves. Characteristics for which we will recruit at least one teacher or principal include:

• Teachers in blended learning classrooms; principals in schools that utilize blended learning.

• Teachers who entered the profession mid-career.

• Teachers and principals from virtual schools that are on CCD.

• Principals from schools that share physical space with other entities.

• Principals from schools that have multiple buildings.

• New teachers

• Teachers with alternative certification

• Teachers and principals from urban, suburban and rural schools

We will contact the principal at each selected school. During this call, we will convey the following information about the study:

• Study purpose

• Sponsor

• What participation involves

• Time commitment

• Confidentiality and voluntary nature of participation

• $40 compensation for participation

A sample recruitment script is presented in Attachment II.

To gain efficiencies in the data collection process, we will try to recruit a principal and a teacher at each sampled school to participate in an interview. We will enlist the aid of the principal in identifying a teacher to interview. To avoid potentially biasing effects of having the principal select the teacher who will participate, we will first ask the principal for permission to contact teachers through email if the school publishes teacher email addresses on its website. If teacher email addresses are unavailable, we will work with the principal to select several teachers at random from the teacher roster. We will then recruit one of these teachers to participate.

To reduce travel time and costs, if interviewing both a principal and teacher at the same school, we will schedule both interviews for the same day if possible. We will plan on interviewing principals and teachers at their schools unless school policy requires that interviews be conducted off site. The principal and the teacher will receive materials in the mail (letter, FAQs) to reiterate the key information about the study listed above and to remind them of their interview date and time.

Researchers from the Washington, DC and Chicago areas will conduct the cognitive interviews. Between three and five staff members from each office will be trained to conduct the interviews to allow the greatest flexibility possible in the scheduling of interviews. Interviewers will vary somewhat in their level of experience with cognitive interviewing. However, they will be required to demonstrate strong understanding of the interview protocol before beginning interviews. With respondent permission, the interviews will be audio-recorded so that the interviews may be reviewed as needed during analysis. Senior Survey Methodologist Lisa Lee will conduct the trainings, supervise staff and monitor data quality throughout the field period.

We have planned for two sets of cognitive testing, each with 2 rounds. In the first set of testing, the instrument will include only selected items identified as problematic, newly developed items, and other items as needed to provide context and for analytic purposes. Once these items have been tested and refined, Set 2 will focus on testing the full instruments.

In Set 1, respondents will be trained on the “think-aloud” technique. They will then complete the questionnaire independently while thinking aloud. The interviewer will observe the respondent during survey completion, noting any questions that the respondent has or problems with particular items. After the respondent has completed the survey, the interviewer will then review the instrument with the respondent. The interviewer will step through the scripted probes (see Attachment IV for details) and also probe spontaneously about additional issues that arose for the respondent. Instruments will be revised based on findings from the first round, and a second round of interviews within this set will be conducted to test the revisions.

In Set 2, respondents will again begin by completing the instrument independently. In order to obtain an accurate timing of survey length, no probing will occur while the respondent completes the survey. After the respondent has finished, the interviewer will then conduct retrospective probing. Scripted probing will focus on items that have undergone development. Interviewers will also probe spontaneously as needed to explore other issues that arise during the interview. If needed, instruments will be revised after the first round and retested in the second round.

Cognitive interviewing techniques allow researchers to evaluate potential response error and to assure that the survey provides valid data. In general, the goal of cognitive testing is to assess the respondents’ comprehension of the questionnaire items, including question intent and the meaning of specific words and phrases in the item. Data from cognitive interviews can identify potentially problematic questions that are not understood as intended. This testing can also examine the respondents’ retrieval of relevant information from memory, decision processes involved with answering a question, and questions that are difficult to answer due to cognitive burden. For example, terms such as “state content standards” can be explored. Further, respondent ability to recall information such as days in their teaching contract or to estimate information such as years of teaching experience can be examined.

The cognitive interviews will assess issues such as:

• The subject’s understanding of terms in the survey

• How confident the subject is in their response

• How they remembered the information they provided in factual questions

• Whether they found a response choice that fit their answer

• How easy or difficult it is to answer a question

• Issues with sensitive questions

• Consistency of answers within the questionnaire and in comparison to the expected range of answers

The cognitive interview protocol is provided in Attachment IV.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and their responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002; 20 U.S. Code, Section 9573). Participants will also be advised that the interview will be recorded and that the audio recording will only be reviewed for the purposes of report writing. Participants will also be advised that direct quotes may be used in research papers and professional presentations, but names will never be attributed to anything a respondent says.

Participants will be assigned a unique teacher or principal identifier (ID), which will be created solely for data file management and used to keep all student and principal materials together. The respondent ID will not be linked to the respondent in any way or form. The signed consent forms will be kept separately from the interview files in a locked cabinet for the duration of the study and will be destroyed after the final report is released. A sample consent form may be found as Attachment III.

Estimate of Costs for Recruiting and Paying Respondents

The teacher and principal respondents will be offered $40 to participate, to both help motivate their participation and to thank them for their time, effort, and inconvenience. The incentive is also deemed necessary to ensure a diverse sample. Respondents will be paid by check after completion of the interview.

Estimate of Cost Burden

There are no direct costs to participants.

Cost to the Federal Government

The cost of conducting the cognitive interviews will be $93,305, under the NCES contract to NORC at the University of Chicago, which includes recruitment and securing consent (including school district-level IRB review if needed), interviewing, recording, analysis, report writing, and participant incentives of $40 each for 164 interviews.

Project Schedule

Contact schools for recruitment of principals and teachers	December 2013
Submit materials to district institutional review boards where necessary	December 2013
Recruit participants through schools and through a snowball sample	December 2013-March 2014
Conduct cognitive testing	December 2013-March 2014
Finalize revisions to item wording	March 2014-April 2014