2020-21 National Teacher and Principal Survey (NTPS) Screener and Respondent Portal Usability Testing

National Center for Education Statistics

Attachment 2 – Recruitment Screener

Attachment 3 – Protocol Materials

2020-2021 National Teacher and Principal Survey (NTPS)

Screener and Respondent Portal Usability Testing

OMB# 1850-0803 v.251

June 2019

2020-21 NTPS Screener and Portal Usability Testing

Attachment 2

Recruitment Screener

Hello. My name is [XXXXXX] and I work for the U.S. Census Bureau. The U.S. Department of Education is conducting a study of schools to evaluate the effects of school workplace conditions, salaries, and training opportunities on the educational workforce, among other things.

We are looking for volunteers to help us improve the web survey used in this study. We want to know how easy or hard it is to complete the web survey, and we need help from people like you to do that. The information gained from the study will be used for potential revisions to the web survey. Your participation is voluntary, and you will receive no sales pitches or follow-up calls based on your participation. I’d like to ask you a few questions to see if you qualify for the study.

The next few questions are for classification purposes only.

1. What is your name? (write in)

________________________________________________

2. What is your phone number? (write in)

_________________________________________________

3. What is your email address? (write in)

_________________________________________________

4. Do you currently work at a school?

___ Yes

___ No – Thank you for your interest, but we need participants who work in a school. END CALL

5. What is your current job description?

_________________________________________________

6. Are you a school administrator or staff person who can provide information about the teachers at your school, including teacher’s names, email addresses, and subject matter taught?

____ Yes

____ No – Thank you for your interest, but we need participants who have access to that data. END CALL

7. Do you have access to the files containing that information at the school?

____ Yes

____ No or Not sure - Thank you for your interest, but we need participants who have access to that data. END CALL

8. What is the name of the school where you work?

________________________________________________

9. What grades does your school serve?

________________________________________________

10. How many students are enrolled in your school?

________________________________________________

11. Is your school public or private or something else?

________________________________________________

12. Have you participated in a research study or focus group in the past 6 months? (RECRUIT NONE.)

___ Yes– Thank you for your interest, but we need participants who have not participated in a study recently. END CALL

___ No

13. Based on your answers, we would like to invite you to participate in our study. You will receive $75. The interview is approximately 90 minutes in length. We would like to conduct the session at your school so you have access to your records and can use your computer. Is that okay?

__Yes, it is okay to come to the school and I will have access to my records -> Go to 14

__No, I cannot conduct this session at my school

We can conduct the session at the U.S. Census Bureau Headquarters or at a local library, but I first need to contact the researcher who will work with you. What is your telephone number and a good time to call?

Telephone # _______________________________

Good time to call ___________________________

GET BACK TO THEM AFTER TALKING WITH RESEARCHER

14. We are conducting interviews between __________ and ___________. When are you available to participate in this research study during that time period?

	CIRCLE
Yes	1	 (CONTINUE, and collect date, time and location)
No	2	 (THANK AND TERMINATE)

Date:

Time:

Location:

2020-21 NTPS Screener and Portal Usability Testing

Attachment 3

Protocol Materials

Table of Contents

Usability Testing Protocol 6

Practice Think Aloud 7

Interview – Starting with Invitation letter for the screener 8

Interview – Starting with Invitation package for the teacher listing 8

Satisfaction Questions 8

Debriefing Questions 9

Vignette 10

Additional Debriefing Questions 10

Finishing up session 10

Interviewer Log Sheet 11

Questionnaire for User Interaction Satisfaction (QUIS) 12

Usability Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form

• Invitation Letter with User ID

• Satisfaction Survey

• Vignettes

• Incentive

• Payment Receipt

• Pens and Paper

• Webcam with gooseneck clamp to video the participant’s PC

• Laptop

• Guest Wireless Username and Password

• Extra Equipment and Charging Cords

INTERVIEWER USABILITY TESTING SCRIPT

Thank you for your time today. My name is XX and I work with usability lab staff at the U.S. Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult our various web survey products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Your name will not be associated with the recording or any of the other data collected during the session.

[Hand NCES consent form; give time to read and sign; sign own name and date. Show participants where the camera is and what part of the screen will be recorded start recording.]

OK great.

Thank you.

There are several parts to today’s session.

As you know when we talked to you to see if you were eligible to participate, we had to make sure you had access to the list of teachers at your school. As the first part of today’s session, we’d like you to describe that list in your own words.

Probe if they are not forthcoming:

Do you have a list?

(If yes, then ask remaining questions; If No, then ask what type of information they have about their teachers)

What do you call it?

What type of information is on it?

Where did that information come from?

Who created it?

Who uses it?

Do you have to report the information from that list to anyone? Who?

Do other schools use a similar format?

Thank you for that information.

The second part of today’s session is for you to help us evaluate the design of a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Teacher and Principal Survey. You will do this by completing it using your computer/this laptop.

If you run into any difficulties as you work in the web instrument, please don’t blame yourself. Any difficulties are the result of the design of the web instrument, not your skills or abilities. We are going to use your comments and experiences, as well as comments and experiences of other participants, to help improve the web instrument. I did not create it, so don’t feel like you have to hold back your thoughts to be polite to me. We appreciate your help so we can make the web instrument work well for everyone.

Think Aloud: I want you to respond to the web instrument exactly the way you would if you were alone at school, but with one major difference. I would like you to think aloud as you work within the instrument. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice Think Aloud

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.

Probes that will be used in study

• Keep talking

• Um-hum?

• What are you thinking?

• What are you including?]

Great, that’s what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

At the end of the session, I will have some additional questions about how satisfied you were with the survey. Now, we may open up some of the screens and talk about them.

Describe how we will video the screen of their computer. Point the gooseneck with webcam to their computer.

Interview – Starting with Invitation letter for the screener

[Hand participant the mailing materials (invitation letter 9L2 with their User ID info). Make certain to write down the PIN or any information that the user may need later in survey]

This is an example of mailing materials you would receive at school if your school was selected for the survey. Imagine that you received this at your school address. Please take a moment to read it. [(If needed) Notice that the address is not your real school address. If you were to receive the survey for real, the mailing materials would have your real school address. Since we cannot replicate that for the lab setting, you will have to pretend that this letter came to your school’s address.] I am interested in any comments you have on the letter, so please think aloud.

[If needed after reading the letter] Please go ahead and do what the letter says to get started. Remember that I am interested in what you have to say as you go about interacting with the web instrument. Please answer the survey questions as they apply to you and your school.

Interview – Starting with Invitation package for the teacher listing

[Hand participant the correct mailing material package –package depends on whether the data are pre-loaded or not. This will include: the coordinator letter (11L2C); the TLF letter (either pre-populated {11L(TLF)1} or blank{11L(TLF)2}); the 11L(PQ) letter; and the 11L(SQ) letter]

Now, pretend it is a few months later after you completed that survey and you receive this package at your school address. Go ahead and give us feedback on the package. What would you do when you received it? [(If needed) Again, you will have to pretend that this letter came to your school’s address, but that is the only part of the study that is pretend.]

Please answer the survey questions as they apply to you and your school. Please go ahead and do what the letter says to get started. Remember that I am interested in what you have to say as you go about interacting with the web instrument. We especially want to see how you enter the teacher information into the form, so please start with that piece.

[Interviewer instruction: If there is pre-loaded data, then that is the option for them; otherwise, it is either file upload or manual entry. If there is pre-loaded data and they have not edited the data, deleted or added a teacher, please ask them to perform those functions, even if it means adding a fake name, “John Doe”. Stop them about 40 minutes into the instrument. If they pause, tell them that we’d like them to work within the instrument for 30-40 minutes.]

Okay, we are ready to move onto the next task. We’d like you to logout of the instrument.

Satisfaction Questions

Now I would like you to answer these questions about the web instrument.

[Open the satisfaction survey on the laptop and have participant answer the questions.]

Debriefing Questions

So, now that you have worked with the web instrument for 40 minutes, in your own words, tell me the purpose of the instrument and the important features of the instrument.

If manual entry:

I noticed that you manually entered your teacher’s names. Why did you choose that way of entering teacher names? Do you have any comments on the data requested?

If file upload:

I noticed you used the file upload feature to enter your teacher list. Why did you choose that way of entering teacher names? Do you have any comments on the file structure or template?

If pre-filled data:

You worked with the data already provided for your school. Do you have any comments on that data or how you had to work with it?

Thank you.

Now we have a few more things we would like you to do.

Vignette

If they interacted with the pre-populated data during the session:

We’d now like you to enter your teacher list using another method. We’d like you to use the file upload method. Please walk us through what you would do using this new ID and PIN (hand ID and PIN to participant). Please think aloud.

(Be sure that the participant downloads the template, copies/pastes or enters their own data into it, and then uploads it. If time is a restriction, have them only upload 5 teachers.)

If there is no pre-loaded data, have them do a manual entry of 5 teachers or upload a file, whatever they did not do.

We’d now like you to [manually enter your teacher list/upload a file]. Please walk us through what you would do using this new ID and PIN (hand ID and PIN to participant) and enter at least five teachers. Please think aloud.

If they uploaded a file and there was pre-loaded data, provide them with the ID and PIN for the pre-populated data.

We’d now like you to interact with a pre-loaded teacher list. Please walk us through what you would do using this new ID and PIN (hand ID and PIN to participant). Please think aloud.

Additional Debriefing Questions

Do you have any questions about the Web instrument or how you would use it?

I have slides of the web instrument available. We can go over any of these screens you would like.

(If time allows) We would like you to read this letter (give 9L1). Please think aloud as you read it. Now, compare it with this other letter that you used earlier today (show 9L2). Do you have any comments on the two letters? Do you prefer one over the other? Why?

Finishing up session

[Give the respondent incentive envelope [$75], and thank them for their time and effort. Collect materials and leave or escort respondent out of the interviewing room.]

Okay, that’s all that I have today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $75.

Interviewer Log Sheet

Type of school: Size_______ Grades _______ Public/Private/Charter_______

P#___

TA:______________________

Date: _____________________

Location: ________________________

PIN:

Main data entry:

Manual OR File Upload OR Pre-populated (circle one)

Wrote PIN down/Answered security questions

Notes:

Vignette:

Manual OR File Upload OR Pre-populated (circle one)

Notes:

Pre-populated data:

Add

Edit

Delete

Questionnaire for User Interaction Satisfaction (QUIS)

Please Note: The questionnaire will be electronic with word labels on either end of 7 radio buttons. See image below for an idea of what the final format will look like.

1. Overall experience with the survey

Unpleasant to Pleasant

2. The survey screens are visually pleasing

Never to Always

3. The survey screens look professional

Never to Always

4. Characters (words/text) on the screen

Barely legible to Very legible

5. Instructions for correcting errors

Confusing to Clear

6. Entering your answers into the survey

Difficult to Easy

7. You knew what to do with this survey

Never to Always

8. Additional Comments__________________________________________________________________

18