Justification

Volume I

2017 National Household Education Surveys Program (NHES)

Usability Testing

OMB# 1850-0803 v.157

May 2016

Justification

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. Such topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement.

Beginning in 1991, NHES was administered roughly every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates in all RDD surveys, NCES decided to conduct a series of field tests to determine if a change to self-administered mailed questionnaires would improve response rates. After a 5-year hiatus in data collection for this developmental work, NCES conducted the first full-scale mail-out administration with NHES:2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. In 2016, the NHES is fielding the PFI and ECPP surveys along with the first administration of the Adult Training and Education Survey (ATES). NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

The PFI, previously conducted in 1996, 2003, 2007, and 2012, surveys families of children and youth enrolled in kindergarten through 12^th grade or homeschooled for these grades, with an age limit of 20 years, and addresses specific ways that families are involved in their children’s school; school practices to involve and support families; involvement with children’s homework; and involvement in education activities outside of school. Parents of homeschoolers are asked about their reasons for choosing homeschooling and resources they used in homeschooling. Information about child, parent, and household characteristics is also collected. To minimize response burden and potential respondent confusion, both enrolled and homeschool versions of the PFI questionnaire were created for self-administration. This submission includes both PFI-Enrolled and PFI-Homeschooled instruments.

The ECPP, previously conducted in 1991, 1995, 2001, 2005, and 2012, surveys families of children ages 6 or younger who are not yet enrolled in kindergarten and provides estimates of children’s participation in care by relatives and non-relatives in private homes and in center-based daycare or preschool programs (including Head Start and Early Head Start). Additional topics addressed in ECPP interviews have included family learning activities; out-of-pocket expenses for nonparental care; continuity of care; factors related to parental selection of care; parents’ perceptions of care quality; child health and disability; and child, parent, and household characteristics.

The ATES surveys adults ages 16 to 65 who are out of high school and provides new measures of adults’ educational and occupational credentials. It identifies adults who have educational certificates, including the subject field of the certificate, its perceived labor market value, and its role in preparing for occupational credentialing; and counts adults who have an occupational certification or license, including the number of such credentials, type of work they are for, their perceived labor market value, and the role of education in preparing for these occupational credentials. To get a comprehensive picture of adult education and training, the survey also includes brief sections on adult participation in work experience programs (such as apprenticeships) and college classes. The 2017 Web Test will include a new section for adults who are in the process of obtaining a certification.

NHES:2017 Web Test

NCES is planning an NHES:2017 Web Test to help determine the feasibility of moving forward with web as a primary mode of data collection in the next full-scale NHES collection in 2019. The web test will experiment with:

• respondent contact strategies,

• asking respondents to complete two topical surveys instead of one, and

• increasing the precision and usability of the web-based screener instrument.

The web instruments are based on the paper and pencil and web versions approved in August 2015 (OMB# 1850-0768 v.11), though no paper questionnaires will be offered as part of this test. Unlike the paper and pencil versions, however, skip patterns in the web instruments will be invisible to the respondent. This test will build on the experiment with the web mode conducted as part of NHES:2016. As with the previous test, this experiment will include “on the fly” sampling between the screener and the topical stages of data collection. The functionality of the web interface will permit immediate sampling of a household member for a topical survey, and if the screener respondent is the sampled adult respondent, the web instrument will allow him or her to continue immediately to the topical survey(s).

Unique to this test, the web instrument will also include the functionality to sample more than one household member for topical surveys. If two topicals are completed by eligible households, NHES will gain sample efficiency and cost savings. However, impacts on response rates and breakoff rates must be assessed before a two-topical strategy is deployed in a full NHES administration.

The 2017 test will include new contact procedures, wherein email addresses will be collected at the screener level and used for nonresponse follow-up to any unfinished topicals. The 2017 web test will allow us to assess the relative costs and benefits in terms of response rates of asking for email addresses and using them for nonresponse follow-up.

The 2017 web instrument will also be mobile optimized such that we expect web respondents to be able to complete the surveys on mobile devices such as smart phones and tablets with ease. In addition, a redesigned version of the screener instrument will be compared to the NHES:2016 web screener in terms of data quality and response rates.

This request is to conduct usability interviews to refine the functionality of the survey platform for the 2017 web data collection. Usability testing has been used for the ED School Climate Survey (EDSCLS), for the National Teacher and Principal Survey (NTPS), for the 2016 NHES web experiment, and for other NCES surveys in past years.

The objective of the proposed testing on the 2017 web instrument is threefold. First, testing will determine design inconsistencies and usability problem areas within the application (e.g., navigation errors, presentation errors – failure to locate information in screens, or control usage problems such as improper button usage). Second, testing will exercise the application under controlled test conditions with users. Data such as timing calculations and item-missing data will be used to assess whether usability goals regarding an effective, efficient, and well-received user interface have been achieved.

The interviews should result in an application that is easy to understand for respondents and therefore less burdensome, while also yielding accurate information. Usability will be evaluated in terms of respondent’s effectiveness and efficiency in survey completion, and satisfaction with the experience of survey completion. The primary deliverable from this study will be the revised, final online application. A report highlighting key findings will also be prepared.

Design

After the instrument development is complete, a formative usability test will be conducted. During the test, the participants’ performance will be investigated using a think aloud protocol, in order to identify usability problems and to better understand the causes of the problems.

In the formative testing, the participant will be required to complete two topical surveys. During response to the survey, the participant will be asked to think aloud (verbalizing what he/she is thinking). Interviewers will ask probing questions as needed. In addition, the eye-tracking technique will be employed to record the participant’s visual scan and gaze pattern so that the design of the instrument may be evaluated. After the completion of the survey, the participant will be debriefed about his/her experience with the instrument. One session is expected to last ninety minutes.

Since there are four topical surveys that can be self-administered on a desktop/laptop computer or on a mobile device, we adopted a 2-factor between-subject design. One factor is topical survey sequence (Survey) with 12 (4x3) permutations, the other is device type (Device) with two categories (laptop, smartphone). The 12 permutations balance out the possible order effect introduced through the dual-topical survey paradigm. With this factorial design, there will be 24 (12x2) combinations of conditions. The minimum sample size to fulfill this design is 24. With the sample size of 24, each cell will have one participant, but each topical survey will be tested on six participants with laptop and on the other six participants with smartphone. Table 1 shows the 24 condition combinations.

Table 1. Condition combinations of the Formative Usability Test

Laptop	Smartphone
Adult, Public/private school	Adult, Public/private school
Adult, Homeschool	Adult, Homeschool
Adult, Early childhood education	Adult, Early childhood education
Public/private school, Adult	Public/private school, Adult
Public/private school, Homeschool	Public/private school, Homeschool
Public/private school, Early childhood education	Public/private school, Early childhood education
Homeschool, Adult	Homeschool, Adult
Homeschool, Public/private school	Homeschool, Public/private school
Homeschool, Early childhood education	Homeschool, Early childhood education
Early childhood education, Adult	Early childhood education, Adult
Early childhood education, Public/private school	Early childhood education, Public/private school
Early childhood education, Homeschool	Early childhood education, Homeschool

The rationale for choosing laptop and smartphone as testing devices follows. Desktop and laptop computers share similar screen display and manual control technology, with a laptop typically being smaller. If a participant can successfully complete a survey on a laptop computer, the participant is likely to be successful on a desktop as well. Likewise, mobile devices share similar screen display and manual control technology, with smartphones being more difficult to operate because of their smaller size. If one can complete a survey with a smartphone, a success can be expected with a larger mobile device.

Each participant will complete the NHES:2017 screener and 2 topical surveys. During the performance of survey completion, the participant will be asked to carry out the following two tasks: login/logout and re-login; read or review help/FAQ depending on a created vignette and sponsor objectives.

The following data collection methods will be used to collect participants’ performance data:

• Think-aloud protocol with minimal probing such as “Keep Talking;” “What are you thinking?” and acknowledgement tokens (linguists refer to this as backchannels) such as “Um-hum?”

• Observation notes

• Satisfaction questionnaire

• Retrospective Debriefing

• Audio and video recording

• Eye tracking recording

Analysis of the data will include behavioral observations, spontaneous verbalizations and answers to debriefing questions in order to identify problems. We will also produce gaze patterns on PC to investigate whether participants attended to or ignore important parts of the screens. We will investigate whether we can produce gaze patterns on smartphones. Finally, we will compute the overall satisfaction ratings.

Before the instrument goes to production, a summative usability test will also be conducted. During the summative testing, the participant will complete the survey on his/her own without interruption or probes. The purpose of this component is to provide evidence for the estimate of the burden imposed upon respondents and identify any unresolved issues from formative testing.

The summative testing will be conducted in the same fashion as the formative testing aside from the following differences. The participant will complete the survey alone, without assistance or interruption. Think-aloud and probing during the survey will not be conducted. Survey completion time and survey responding errors will be recorded. Survey responding errors will be further defined.

The summative test design will be the same as that of the formative testing, namely, a 2-factor between-subject design, except for the factor of topical survey sequence (Survey). The topical survey sequence factor will have only six permutations, for the sake of reducing workload due to time constraints. With this partial factorial design, there will be 12 (6x2) combinations of conditions. The minimum sample size to fulfill this design is 12. Table 2 shows the 12 condition combinations for the Summative Usability Test. If results of the formative usability testing suggest that certain topical-survey sequences are especially problematic for users, the condition combinations below may be revised to ensure that the problematic topical-survey sequences are included in summative usability testing.

Table 2. Condition combinations of the Summative Usability Test

Laptop	Smartphone
Adult, Public/private school	Adult, Public/private school
Adult, Homeschool	Adult, Homeschool
Public/private school, Homeschool	Public/private school, Homeschool
Homeschool, Public/private school	Homeschool, Public/private school
Early childhood education, Adult	Early childhood education, Adult
Early childhood education, Public/private school	Early childhood education, Public/private school

The participant will complete the 2017 NHES Screener and two topical surveys as it pertains to their real-life situation. The data collection methods will be the same as the formative testing, except that there will be no think aloud and debriefing may be conducted accordingly. Data analysis will be the same as the formative testing, with two additional quantitative measures: average completion time and error rate.

Recruiting and Paying Respondents

To assure that we are able to recruit participants from all desired populations and to thank them for completing the interview, each respondent will be offered $60 for participation in a ninety-minute interview. The longer interview is necessary to ensure adequate time for a participant to complete the screener interview and two topical interviews -- a scenario that will be feasible in the 2017 NHES and that will allow us to evaluate any challenges participants encounter when moving from one topical to another, and to assess potential break-off from fatigue. In addition, participants will be asked to bring their own mobile device for use during the study. This request is consistent with requests previously approved for studies conducted by both the U.S. Census Bureau and the Bureau of Labor Statistics (see table 3).

Table 3. Examples of Studies Approved for $60 Incentive

OMB Control Number	Study Name	Concluded Date	Current Expiration Date
0607-0725	Decennial Language Research – Usability Study of the Online ACS	01/24/2014	08/31/2016
	Online ACS Mobile Device
	Usability of 2017 National Survey of College Graduates (NSCG) online questionnaire
	2016 Census Test Online Questionnaire
	2015 National Content Test Usability and Cognitive Testing
	2015 Site Test April 2015
	2016 COMPASS NRFU
1220-0141	CE Validation Study	02/27/2009	02/29/2012

Participants will be recruited by the U.S. Census Bureau, using multiple sources, including the U.S. Census Bureau’s recruiting database, flyers posted in libraries, social media/Craig’s List, and personal and professional contacts. Recruitment contact materials are included in Attachment 1. The questions used to screen respondents for participation are included in Attachment 2. The usability interview protocols are included in Attachment 3. A testing session will be carried out in either the U.S. Census Bureau usability lab or a community library. The session will be conducted one-on-one, i.e., one participant and one test administrator (TA), with one note taker.

Assurance of Confidentiality

Participation is voluntary, and respondents will read a confidentiality statement and sign a consent form before interviews are conducted. The confidentiality statement and consent form are provided in Attachment 1. No personally identifiable information will be maintained after the usability testing interview analyses are completed. Primary interview data will be destroyed on or before December 31, 2023. Data recordings will be stored on the U.S. Census Bureau’s secure data servers.

The interviews will be audio and video-recorded. Participants will be assigned a unique identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The only identification included in the audio files will be the participant ID. The recorded files will be secured for the duration of the study – with access limited to key U.S. Census Bureau and NCES project staff – and will be destroyed seven years after completion of the testing. Interviews may also be observed by key project staff. Participants will be informed when observers attend.

Estimate of Hour Burden

We expect the usability interviews to last approximately ninety minutes. Screening potential participants will require 20 minutes per screening. We anticipate needing to conduct 252 screening interviews to yield the needed participants (an estimated 168 screenings to yield 24 participants for formative testing and an estimated 84 screenings to yield 12 participants for summative testing). This will result in an estimated total of 137 hours of respondent burden for this study.

Table 4. Estimated response burden for NHES:2017 web usability tests

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	252	252	0.33	83
Formative Usability Interviews	24	24	1.5	36
Summative Usability Interviews	12	12	1.5	18
Total	252	288	-	137

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

Recruitment will begin upon OMB approval. Interviewing is expected to be completed within 4 months of OMB approval. The data collection instrument will be revised after the completion of formative testing and summative testing.

Cost to the Federal Government

The cost to the federal government for this usability testing laboratory study is approximately $128,000.