2022 National Household Education Survey (NHES) Web Usability Testing
NCES System Clearance for Cognitive, Pilot, and Field Test Studies 2019-2022
Att 2 3 NHES 2022 Usability Testing Screener Protocols
2022 National Household Education Survey (NHES) Web Usability Testing
OMB: 1850-0803
2022 National Household Education Survey (NHES) Web Usability Testing
Attachments 2 & 3
Attachment 2 Recruitment Screener..p.2
Attachment 3 – Protocol Materials…..p.9
OMB# 1850-0803 v.265
National Center for Education Statistics (NCES)
April 2020
Attachment 2 Recruitment Screener
NHES: 2022 Usability Testing
Hello. My name is [XXXXXX] and I work for the U.S. Census Bureau. The U.S. Department of Education is conducting a study about children’s care and education. The Census Bureau is assisting with the research.
We are looking for adult volunteers to help us improve the web survey used in this study. We want to know how easy it is to complete the web survey, and we need help from people like you to do that. The information gained from the study will be used to revise and improve the web survey. Your participation is voluntary, and you will receive no sales pitches or follow-up calls based on your participation. I’d like to ask you a few questions to see if you qualify for the study.
Collect code if offered: _______________________________
What is your name? (write in First and Last name in separate fields)
________________________________________________
What is your daytime phone number?
(Probe for all 3: mobile, home (if not mobile), and work (if any))
_________________________________________________
How many people age 18 or older currently live in your household, INCLUDING yourself?
[ ]
How many people younger than 18 currently live in your household?
[ ]
Why do you think you qualify for this study? (Select all that apply)
[] At least one young child in one non-parental care situation
[] At least one young child in multiple non-parental care situations
[] At least one child being homeschooled
[] At least one young child with a diagnosed developmental disability
[] At least one young child where the parent is concerned about a developmental delay
CHECK: If none of situations a-e apply, then thank and terminate call for NHES. Otherwise, go to Q6.
Are you a parent or guardian of a child who is homeschooled currently and who was homeschooled prior to COVID-19?
Yes (go to Q7)
No (go to Q12)
We are interested in finding out more about the child who is homeschooled.
What is this child’s name? (You can provide initials if you would like instead of name.)
How old is this child?
Is this child male or female?
Is this child homeschooled for grades equivalent to kindergarten through grade 12?
Yes – What is this child’s grade equivalent?________
No
Do you have another child who is homeschooled now and was homeschooled prior to COVID-19?
Yes (go to Q7)
No (go to Q12)
Are you a parent or guardian of any child who is not yet in kindergarten and who is in child care?
Yes
No (Go to CHECK)
We are interested in finding out more about the child who is not yet in kindergarten and who is in childcare. Just to verify, your child typically is in some type of childcare, even if due to COVID-19 your child may be temporarily cared for at home. Correct?
What is this child’s name? (You can provide initials if you would like instead of name.)
How old is this child?
____ (if older than 5 go to Q20)
Is this child in kindergarten or a higher grade?
Yes (Go to Q20)
No
Is this child in a childcare arrangement, e.g., cared for by a relative other than the child’s parent or guardian; cared for by a person who is not a relative in the child’s or another home; or cared for in a day-care center, preschool, prekindergarten, Head Start or other early childhood program?
Yes -
Please explain the type of care arrangement the child is in. If your child is in multiple care arrangements, please tell us all the different care arrangements you have. (400 characters)
No
Have you been told your child has a speech or language impairment, Attention Deficit Hyperactivity Disorder (also known as ADHD), or autism or autism spectrum disorder?
Yes
No
Has a health care provider told you that your child has a developmental delay?
Yes
No
Do you have concerns about your child’s development? For example, concerns that the child might have a speech, social or emotional delay?
Yes
No
Do you have another child who is not yet in kindergarten and who is in child care?
Yes (Go to Q13)
No
CHECK: If no children, then thank and terminate call. If children, determine if they meet the current recruiting needs and then either proceed or thank and terminate call.
What is your relation to the child/children?
o Mother
o Father
o Grandmother
o Other: Please describe________________________
Are you employed by the Federal Government? (if federal contractor, mark NO)
o Yes
o No
In what city, state, and ZIP code do you currently live?
Do you own a cell phone?
o Yes
o No Go to Q25
Some cell phones are called “smartphones” because they connect to the internet. Is your cell phone a smartphone, such as an iPhone, Android, Blackberry, or Windows phone, or are you not sure?
o Yes
o No
o Not sure
What type of smartphone do you have?
o iPhone
o Android
o Blackberry
o Windows phone
o Not sure
Do you own a desktop or laptop computer, or both?
o Desktop computer
o Laptop computer
o Both desktop and laptop computer
o Neither
Do you own a tablet computer such as an iPad, Samsung Galaxy Tab, Google Nexus, or Kindle Fire?
o Yes
o No
Have you used the Internet for at least one year?
o Yes
o No End interview
a. How often do you use the Internet on a desktop or laptop computer? Several times a day, about once a day, a few times a week, a few times a month, or a few times a year?
o Several times a day
o About once a day
o A few times a week
o A few times a month
o A few times a year
o Not at all [do not read to respondent]
How did you hear about this research opportunity?
[ ] E-mail link
[ ] Census Bureau employee
[ ] Friend or family member (but not a Census Bureau employee)
[ ] Craigslist
[ ] Google advertisement
[ ] Flyer
[ ] Newspaper advertisement
[ ] Other – specify [ ]
Are you a U.S. citizen? You do not have to be a U.S. citizen to participate in our studies.
o Yes
o No - cannot come to headquarters
Have you participated in a research study or focus group in the past 6 months? (RECRUIT NONE.)
Yes (Thank and terminate.)
No (Continue)
What is the email address we should use to contact you, should you qualify for a study? (write in)
_________________________________________________
All recruits MUST be articulate, able to answer open ended questions, have at least one year of Internet experience, have a child that fits into the needed recruiting criteria, and able to provide explanations for their responses.
Based on your answers, we would like to invite you to participate in an interview session (to discuss your child’s care and education). You will receive $40 as a thank you for completing the interview. The interview is about 60 minutes in length.
Remote:
The session will be conducted over the telephone using an application called Skype. You will use your computer at home and the researcher will be at the census and watch you complete the survey on your computer using Skype. A researcher will send you an email with a link to download Skype, a form explaining your rights as a participant, and different postal options for receiving the $40. Then they will set up a time to test Skype with you before the session. Is this okay?
-
o Yes
(CONTINUE – Great. In the next two days a researcher will email you a link to download Skype.. Please check your email for this notification and further instruction.)
o No
(thank you for your time, but at this time we are not conducting in person sessions. We will notify you if that changes. )
In person:
The session will be conducted (at the U.S. Census Bureau headquarters in Suitland, Maryland or at a local library or community center).
Are you willing to come to Suitland, Maryland to the Census Bureau headquarters for the session?
-
o Yes
(CONTINUE)
o No
(participant prefers other location – gather general location that would work for participant and CONTINUE)
We are conducting interviews between __________ and ___________. Are you available to participate in this research study during those hours?
-
o Yes
(CONTINUE AND GIVE DIRECTIONS)
O No
(THANK AND TERMINATE)
RECRUITER: Check off all of the criteria this adult meets in order to categorize the interviews we will conduct.
Interview Criteria:
Parent of child < 6 NOT enrolled in school and in day care center / child care center / preschool / prekindergarten |
Parent of child < 6 NOT enrolled in school and in child care with a relative, e.g., grandparent, sibling, other relative |
Parent of child < 6 NOT enrolled in school and in a home child care setting with non-relative, e.g., nanny, in home care, neighbor, au pair |
Parent of child < 6 with a known developmental delay (diagnosed by a health care provider) |
Parent of child < 6 with a parent that is concerned their child MIGHT have a developmental delay |
Parent of child < 6 where parent DOES NOT have any concerns about the child’s development |
Parent of homeschool child (K-12 equivalent; age 5-17) |
|
|
|
|
|
|
|
|
Attachment 3 – Protocol Materials
NHES: 2022 Usability Testing
Table of Contents
In-person only - Eye tracking 10
In-person only - Eye tracking of alternate design for branching. 11
Debriefing slides and debriefing questions PFI 15
Debriefing slides and debriefing questions ECPP 27
Test Administrator Notes Pages 44
Usability Testing Protocol
MATERIALS NEEDED FOR INTERVIEW
Interviewer Protocol Script
Consent Form
Laptop with the Demographic and Satisfaction Survey and Tobii software and/or Camtasia software
Link to NHES Qualtrics Survey
Eye tracking hardware: Tobii x2-60
Incentive
Payment Receipt
Pens and Paper
Charging Cords and surge protector
Antibacterial disinfectant wipes
INTERVIEWER USABILITY TESTING SCRIPT
Thank you for your time today. My name is XX and I work at the U.S. Census Bureau. I will be working with you today. In this lab, we evaluate how easy or difficult our various web survey products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group.
[In person] Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Your name will not be associated with the recording or any of the other data collected during the session.
[Hand NHES consent form; give time to read and sign; thank participant; sign own name and date. Show participants where the camera is and what part of the screen will be recorded. Then start recording.]
[Remote] We would like to take a video recording of the device as you use it along with the audio from this session to get an accurate record of your feedback. Your name will not be associated with the recording or any of the other data collected during the session. We are not capturing a video of your face, only the screen. We emailed you the consent form, but for our purposes, we can capture your consent orally. [Turn on Camtasia] Please state, “I give consent to be audio recorded and my screen to be recorded.”
Thank you.
There are several parts to today’s session.
The first task is to complete a brief questionnaire asking about your background. Then I’ll have you complete the survey about your child.
[In person: Have participant fill out the background questionnaire on the laptop.
If remote, ask questions orally.]
Thank you.
The second part of today’s session is for you to complete a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate the National Household Education Survey. You will complete it on [your PC/on this laptop].
If you run into any difficulties as you work on the survey, please don’t blame yourself. Any difficulties are the result of the design of the Web survey, not your skills or abilities. We are going to use your comments and experiences as well as comments and experiences of other participants to help improve the survey. I did not create the online survey, so don’t feel like you have to hold back on your thoughts to be polite to me. We appreciate your help so we can make the online survey work well for everyone.
I want you to respond to the web instrument exactly the way you would if you were at home, but with one major difference. I would like you to think aloud as you work within the instrument. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.
Practice Think Aloud
Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?
[Probe if they fall silent; if they just give a number ask them to do it again but this time so that you can understand how they came up with the number; after they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.]
Probes to use in study
Keep talking
Um-hum?
What are you thinking?
Great. That is what I want you to do throughout our session. I will remind you to think aloud if you get quiet.
In-person only - Eye tracking
We are also going to record where you look on the screen as you complete the survey. Now we will do a simple task that will allow the computer to find your eyes. Please follow the dot with your eyes.
[Calibrate the participants’ eyes]
Interview
I am going to open up the survey that we would like you to complete. Remember that I am interested in what you have to say as you answer the survey questions.
Remote To open up the survey, I need you to give me control of your device for a moment so I can type in the survey link. At the top, please select give control automatically. Now I will type in the survey link.
Satisfaction Questions
After final SUBMIT is clicked, have participant answer satisfaction questions.
The next task is for you to answer a short questionnaire about your experience completing the survey you just finished.
Remote To open up the survey, I need you to give me control of your device for a moment so I can type in the satisfaction survey link. At the top, please select “give control automatically”. Now I will type in the link.
Debriefing Questions
The next task is to go over some of the screens you saw. I’m going to bring up a PowerPoint slide deck.
Printout of slides and debriefing questions are found in section below
Bring up slide deck and follow questions in the notes portion of that slide deck.
In-person only - Eye tracking of alternate design for branching.
The survey asked you about where your child lived using one design. Here is another design for this question. Take a look at the question and please answer it as if you were at home.
[Open the eye tracker and have user answer the question.]
Final Debriefing
Is there any additional feedback you would like to share with me that we have not already discussed?
Were there any other questions that you found confusing or difficult to answer?
Last question: Is there anything else you’d like to mention that we haven’t talked about?
Finishing up session
[Give the respondent incentive envelope [$40], and thank them for their time and effort. Collect materials and escort respondent out of the interviewing room if in person.]
In person: Okay, that’s it for today. Many thanks for your work on this project. The last thing I need you to do is sign this form that says I am giving you $40.
Remote: Okay, that’s it for today. I’m going to be sending you $40 via certified mail/postal mail. Let me confirm your address. Where would you like the cash mailed?
Enter address:
Background Questionnaire
1. In what month and year were you born? _________________ __________________
month year
2. What is the highest grade or year of school you have completed?
Less
than high school
Completed
high school
Some
college, no degree (indicate area of study:
____________________________________)
Associate’s
degree (AA/AS) (indicate area of study:
__________________________________)
Bachelor’s
Degree (BA/BS) (indicate area of study:
__________________________________)
Degree
above Bachelor’s (indicate degree and area of study:
________________________________)
3. Are you male or female?
Male
Female
4. Are you of Hispanic, Latino, or Spanish origin?
Yes
No
5. What is your race? Choose one or more races.
White
Black
or African American
American
Indian or Alaska Native
Asian
Native
Hawaiian or Other Pacific Islander
Satisfaction Questionnaire
Please Note: The questionnaire will be electronic with word labels on either end of 7 radio buttons. See image below for an idea of what the final format will look like.
Unpleasant to Pleasant
|
Never to Always
|
Hard to understand to Easy to understand
|
Always bothered me to Never bothered me
|
Confusing to Clear
|
Difficult to Easy
|
Debriefing slides and debriefing questions PFI
*During the session, participants will only see the screen, and the text will be not shown to the participant. The TA will read the text aloud.
Debriefing slides and debriefing questions ECPP
*During the session, participants will only see the screen, and the text will be not shown to the participant. The TA will read the text aloud.
Test Administrator Notes Pages
Completed from screening information prior to the start of the session
P#___
PFI or ECPP
TA:______________________
Date: _____________________
Location: Headquarters / Other :________________________
Characteristics from screener:
Description of relationship between child and participant:
[] Child age is less than 5 years old / - type of care arrangement________________________________
[] Child is home schooled – grade equivalent____
[] Child’s health – parent answer from recruiter (circle)
[] No health concerns
[] Parent has development concerns
[] Health care suggests delayed development
[] Child has been diagnosed with _______________________________
All
Were there any spontaneous comments on the two languages offered?
Yes – what were they_____________________________________
No
Were any of these comments negative?
Yes
No
All
What names did the P record?
Did P enter names incorrectly –
like entering multiple names on the same line? Yes or No
like entering names, nicknames and initials together? Yes or No
Were there any spontaneous comments on question asking for child’s name? Yes or No
Comments_______________________________________________________________
Were there any spontaneous comments on the number of fields (there are currently 5)? Yes or No
Was there a match with # of children & listing out children’s names? Yes or No
ECPP
Record the answers selected on this screen.
Was there any confusion with the enable/disable feature on this page?
Yes – what were they_____________________________________
No
Were there any spontaneous comments on the options provided?
Yes – what were they_____________________________________
No
How about the term home child care provider?
Yes – what were they__________________________
No
Did any of the comments indicate confusion?
Yes
No
As far as you could tell from the think aloud data, did the P double count any care provider?
Yes - Explain
No
Did the R talk about the arrangement in a way that is consistent with how we categorize relative care arrangements, nonrelative care arrangements, and center-based care arrangements, respectively?
Yes
No – Explain more
ECPP = Young child health questions
Record the answers selected on this screen.
Were there any spontaneous comments on any of the disabilities questions?
Yes - Which ones and what were the comments
No
Was there any hesitation on any of the disabilities questions?
Yes - Which ones
No
Did data match the screening criteria for P?
Yes
No – what didn’t match?
PFI
Record the answers selected on this screen.
P is supposed to enter all grades for which the child was homeschooled. What happened?
Did P select multiple grades?
Yes
No
Did P select intermittent grades?
Yes
No
Were there any spontaneous verbalizations on the word grade ?
Yes – What occurred?
No
PFI
Record answers on both screens
Any spontaneous comments?
Record answers on both screens
Any spontaneous comments?
Record answers on both screens
Any spontaneous comments?
If R comments about this question being irrelevant to their situation , try to determine if that is a product of the artificial nature of this test. In the NHES:2019, a homeschooling parent would have received this item in a section about the child’s school if they had previously answered YES to a question asking if the child also attends a school. So in the real survey not all parents of homeschooled children get this question.)
Did P appear to report per week or per day?
Per week
Per day
Not sure
Were there any spontaneous comments on this screen?
Yes – What were they
No
All
Were there any spontaneous comments on the greyed out branching?
Yes – What were they?
No
Was there any indication of confusion because of the greyed out branching?
Yes – What did you observe?
No
All - Parent sections:
Were there any spontaneous comments on fills for parents/esp second parents?
Yes – What were they?
No
To the best of your knowledge, did the P get confused and answers for the wrong parent?
Yes – what happened
No
Were there any spontaneous comments about “other mother” or “other father” if in a same-sex relationship?
Yes – what were they?
No
Was there any non-verbal feedback during this section of questionnaire?
Yes – what happened?
No
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Low, Kathryn (Temp) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |
© 2024 OMB.report | Privacy Policy