Justification

National Center for Education Statistics

Volume I

Supporting Statement

2017-2018 National Teacher and Principal Survey (NTPS)

Portal Usability Testing

OMB# 1850-0803 v.189

Attachments:

Attachment 1 – Communication Materials: Contact, Consent, and Receipt Acknowledgement, Mailing Materials

Attachment 2 – Recruitment Screener

Attachment 3 – Protocol Materials

Attachment 4 – Portal Instrument Screen Shots

February 2017

Background

The National Teacher and Principal Survey (NTPS), conducted biennially by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), is a system of related questionnaires that provides descriptive data on the context of elementary and secondary education. Redesigned from the Schools and Staffing Survey (SASS) with a focus on flexibility, timeliness, and integration with other ED data, the NTPS system allows for school, principal, and teacher characteristics to be analyzed in relation to one another. SASS, the predecessor to NTPS, was conducted by NCES seven times between 1987 and 2011. After 2011-12, NCES redesigned SASS and named it NTPS to reflect the redesigned study’s focus on the teacher and principal labor market and on the state of K-12 school staff. NCES first conducted NTPS during the 2015-16 school year, and the next collection will be conducted during the 2017-18 school year. NTPS data collection is administered for NCES by the U.S. Census Bureau (Census), with the special contact district recruitment contracted to Avar Consulting and its subcontractor Westat.

The following materials are being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803) which provides for NCES to improve methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting pilot tests, focus groups, cognitive interviews, and usability testing.

This request is to conduct usability testing of the NTPS Respondent Portal internet instrument, which was developed for the 2015-16 NTPS and has since been refined for the 2017-18 NTPS.

Design

The sample for NTPS 2017-18 will include approximately 10,600 public schools and principals and 47,000 public school teachers, and approximately 4,000 private schools and 9,000 private school teachers. Data collection will begin with the sampled school. A notification letter, intended to inform the school of their selection for the survey and to verify their mailing address, will be sent to sampled schools early in the summer of 2017.

In July, all schools will receive an advance letter addressed to the principal at the school address. The letter includes instructions for completing a brief screener interview online using the NTPS Respondent Portal. The purpose of the screener interview is to determine the school’s eligibility for the NTPS and establish a survey coordinator. The survey coordinator will be asked to facilitate the completion of NTPS questionnaires within their school, and materials will be mailed to him or her throughout data collection. The NTPS Respondent Portal will serve as a valuable resource for the survey coordinator, as it provides the status of each of the school’s questionnaires, allows the survey coordinator to request replacement questionnaires, and provides the survey coordinator with materials to help encourage response within the school using a tool called the “Resource Center.” The Resource Center is a new addition to the NTPS Respondent Portal designed to enhance user experience, it was not part of the 2015-16 NTPS portal.

In late September 2017, schools will receive an initial school package addressed to the survey coordinator or principals at the school address. The package will contain a letter to the survey coordinator or principal, and three individual sealed envelopes that contain login information for completing the Teacher Listing Form (TLF), Principal Questionnaire, and School Questionnaire via the internet survey instruments. Principals and survey coordinators will also be contacted by email around the same time the initial packages are mailed to the sampled schools. The emails will contain the appropriate hyperlinks and User IDs to complete the relevant NTPS questionnaire(s) online.

The TLF obtains information about each teacher working in the school – including the name, email address, full- or part-time teaching status, and subject (high level). The primary purpose of the TLF is to build the teacher sampling frames (public and private). Teachers will be sampled from teacher rosters obtained through the TLFs submitted by sampled schools to Census. Teachers are ineligible for NTPS if they are short-term substitutes, student teachers, teacher’s aides, or do not teach any of grades K-12 or comparable ungraded levels.

The TLF instrument is included in the NTPS Respondent Portal instrument. The TLF internet instrument offers the respondent two options for completing their TLF electronically. Respondents may (1) download an Excel template, populate it with their TLF data, and upload it to the instrument or (2) enter their TLF information directly into the instrument. These two options were available to respondents in the analogous 2015-16 NTPS Respondent Portal instrument; however, to make it more user-friendly, improvements have been made for the 2017-18 NTPS.

The general functionalities of the instrument – status center, screener interview, and TLF application –remain unchanged from the 2015-16 version, aside from the addition of the “resource center” described above. Generally, the goal of the 2017-18 NTPS Respondent Portal development is to make adjustments to improve the functionality of the instrument – particularly the TLF instrument – ultimately making it more user-friendly for school staff.

After the majority of the instrument development is complete, a formative usability test will be conducted. During the test, the participants’ performance will be investigated using a think-aloud protocol, in order to identify usability problems and to better understand the causes of the problems.

The priorities and goals of the usability testing are as follows: (1) Participants providing TLF data electronically using their own equipment; (2) Participants completing the screener interview; and (3) Participants exploring the information provided in the Resource Center, including selecting the various tabs within the instrument.

Census will administer the usability testing interviews on behalf of NCES. Prior to beginning the primary task in the usability session, these interviewers will gather information about the current teacher information available within the school, whether this information is reported to other entities, and other general information about the data. During the testing, interviewers will observe the participant receiving mocked-up mailing materials about the survey and then accessing the NTPS Respondent Portal using their own device on site at the school.

Interviewers will observe the participant completing the TLF electronically as s(he) uses the method that (s)he arrives at naturally. Interviewers will ask probing questions as needed during the maximum forty minutes allocated for this task. Due to the nature of the information collected on the TLF, interviews ideally should be conducted onsite at the participant’s school, using the participant’s computer so that (s)he is able to access the necessary school records to complete the TLF as a respondent would during 2017-18 NTPS full-scale data collection. A Census laptop and “dummy data” will be brought to the interview site as back-up, but every effort should be made to have participants use their own equipment and school data during the usability testing.

After the completion of the TLF, the participant will be debriefed about his/her experience with the instrument. As time permits, the participant will be asked to complete as many as three additional tasks or “vignettes” related to instrument screener information, general instrument exploration, and completing the TLF electronically using the other completion method. If the task was already accomplished during the first part of the usability test, then the vignette is skipped. Each entire usability session is expected to last ninety minutes.

The following data collection methods will be used to collect participants’ performance data:

• Think-aloud protocol with minimal probing such as “Keep Talking;” “What are you thinking?” and acknowledgement tokens (linguists refer to this as backchannels) such as “Um-hum?”

• Observation notes;

• Satisfaction questionnaire;

• Retrospective Debriefing; and

• Audio and video recording.

Analysis of the data will include behavioral observations, spontaneous verbalizations and answers to debriefing questions in order to identify problems. Overall satisfaction ratings will also be produced.

Recruiting and Paying Respondents

To conduct usability testing on the NTPS Respondent Portal, we will recruit between 10 to 20 individuals who work at schools in the metropolitan DC area that offer at least one of grades 1-12. The participant may work at either public or private schools and must have access to the list of teachers for their school, including information such as teacher name, email address, subject(s) taught, and full or part-time status. We will attempt to recruit school staffs from varied sizes, as measured by the number of students enrolled in the school, in order to test the effect of school size responses (e.g. a principal in a small school vs. school secretary in a large school) on TLF completion method. The goal is to conduct five sessions in small schools (<350 students), five in medium-sized schools (350-750 students), and five in large schools (750+ students).

To assure that we are able to recruit participants from all desired populations and to thank them for completing the interview, each respondent will be offered $75. The $75 amount is requested because the target population is notoriously difficult to recruit and we will be asking for ninety minutes of their time. The 90 minutes are necessary to ensure adequate time for a participant to provide a TLF, complete a screener interview, and go through as many as three vignettes. By including all of these components in each usability testing session, we will be able to fully observe and evaluate any challenges participants encounter when accessing various parts of the instrument as a school would during the 2017-18 NTPS full scale data collection.

Participants will be recruited by Census using multiple sources, including flyers posted in libraries, social media/Craig’s List, personal and professional contacts, and Census’s recruiting database. Contact, consent, and receipt acknowledgment materials are provided in Attachment 1, the questions used to screen respondents for participation in Attachment 2, the usability interview protocols in Attachment 3, and screenshots of the portal instrument in Attachment 4. A testing session will be carried out either onsite at the participant’s school or in Census’s usability lab. The session will be conducted one-on-one, i.e., one participant and one test administrator (TA), with one note taker.

Assurance of Confidentiality

Participation is voluntary, and participants will read and sign a consent form before interviews are conducted. The consent form with a confidentiality statement are provided in Attachment 1. No personally identifiable information will be maintained after the usability testing interview analyses are completed. Primary interview data will be destroyed on or before December 31, 2024. Data recordings will be stored on Census’s secure data servers.

The interviews will be audio and video-recorded. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The only identification included in the audio files will be the participant ID. The recorded files will be secured for the duration of the study – with access limited to key Census and NCES project staff – and will be destroyed seven years after completion of the testing. Interviews may also be observed by key project staff. Participants will be informed when observers attend.

Estimate of Hour Burden

We expect the usability interviews to last approximately ninety minutes. Screening potential participants will require 10 minutes per screening. We anticipate needing to conduct 30 screening interviews to yield 20 participants for usability testing. This will result in an estimated total of 35 hours of respondent burden for this usability testing.

Table 1. Estimated response burden for 2017-18 NTPS Respondent Portal Suability Testing

Participants	Number of Respondents*	Number of Responses	Time Estimate (minutes)	Total Burden Hours
School staff recruitment	30	30	10	5
Usability Interviews	20	20	90	30
Total	30	50	-	35

* Duplicate counts of the same respondents are not included in the total number of respondents.

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

Recruitment will begin in February 2017, as soon as OMB approval is received. Interviewing is expected to be completed within 2 months of OMB approval.

Additionally, accessibility evaluations will be performed concurrently with the usability testing. For desktop and laptop screens, Section 508 regulations will be followed. Specifically, desktop/laptop versions of the portal will be evaluated with JAWS 17 and the IE 11 Web browser, running on Windows 7 for conformance to Section 508. Besides navigating through the electronic instruments, these evaluations will include performing the same tasks done in the test cases.

The Portal will be revised after the completion of usability testing and accessibility evaluations.

Table 2. Estimated Project Schedule

Activity	Start
Participant recruitment	February 2017
Usability testing	March 2017 (need to end before spring breaks begin*)
Quick report	May 2017
Accessibility report	May 2017
Final report	September 2017

* 2017 DC Metro Area spring break schedules: Montgomery County: April 7-14, Fairfax: April 10-14, Falls Church: April 10-14, Calvert: April 13-18, and PG: April 14-21.

Cost to the Federal Government

The cost to the federal government for this usability testing laboratory study is approximately $64,425.

5