Justification

Volume I

2019 National Household Education Surveys Program (NHES) Adult Training and Education Survey (ATES) Spanish Language Cognitive Interviews

OMB# 1850-0803 v.205

August 2017

National Center for Education Statistics (NCES)

Justification

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, focusing on topics that are appropriate for household surveys rather than institutional surveys. NHES topics have covered issues such as early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, participation in before- and after-school activities, participation in adult and career education, parents’ involvement in education, school choice, homeschooling, and civic involvement. The NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. The NHES typically fields 2 to 3 topical surveys per administration, although the number has varied across administrations. Surveys are administered in English and in Spanish. NHES is currently on a 3-year administration cycle and is administered using paper instruments mailed to households (although a new administration method is being tested, as indicated below).

The Adult Training and Education Survey (ATES) is a relatively new NHES topical survey, first administered in 2016, that focuses on the attainment of non-degree credentials. It identifies adults who have an occupational certification or license, including the number of such credentials, type of work they are for, their perceived labor market value, and the role of education in preparing for these occupational credentials. The survey also includes a brief section on postsecondary certificates and a section on adult completion of work experience programs (such as apprenticeships). Out-of-high-school adults ages 16 to 65 are eligible to receive this topical survey.

The ATES topical survey has undergone extensive development work over the past six years. To develop the ATES instrument, NCES has: conducted focus groups in 2013 on characteristics of educational certificates and participation in work-related training; conducted cognitive interviews in 2013 and 2014 to test English versions of the ATES instrument; administered a response rate pilot study in early 2013; administered a feasibility study in 2014 to test processes for incorporating ATES into the NHES program; and conducted a series of concept interviews and cognitive testing in 2015 to develop the 2016 ATES instrument. After the first administration of the ATES instrument in 2016, a series of focus groups (in Spanish only), concept interviews, cognitive testing in both English and Spanish, and usability interviews were conducted to revise the instrument for a 2017 pilot test of an NHES web administration. These efforts led to the development of the current version of the ATES instrument, to be part of the 2019 NHES administration.

Since the last cognitive testing of the ATES instrument (for the 2017 web pilot; OMB# 1850-0803 v.149), a small number of new and revised items have been added to the survey that require additional testing. New items have been added to find out more about individuals who have the same licenses in multiple states. A few items also have been revised in the certifications and licenses section and the section describing preparation for new certifications and licenses, in order to collect better information on those credentials. Finally, the Department of Health and Human Services (DHHS) has requested that two questions about the child care workforce be added to ATES, as a cost-savings measure (to avoid administration of a separate national survey of adults) – these items are modified versions of items asked on the 2012 household survey component of the National Survey of Early Care and Education (NSECE).

This request is to conduct cognitive interviews in Spanish to test the new and revised questions for inclusion in the ATES instrument in the 2019 NHES administration. In order to reduce respondents’ cognitive and time burdens, the cognitive test instrument includes only sections of the ATES questionnaire that contain items requiring testing. Some background questions inquiring about education and employment have also been included to provide context to certification and licensure items.

The primary deliverable from this study will be the revised, final version of the ATES items for inclusion in the 2019 NHES administration. A report highlighting key findings will also be prepared.

Design

Cognitive Interviews

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just answered. Techniques include asking probing questions to clarify points that are not evident from the think-aloud comments, and responding to scenarios. In these interviews, probes will be used to:

• verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating);

• check respondents’ understanding of the meaning of specific terms or phrases used in the questions; and

• identify experiences or concepts that the respondent did not think were covered by the questions but that we consider relevant.

Interviews are expected to last about 1 hour and will be conducted by trained cognitive interviewers. This submission includes the protocol that will be used to conduct the interviews and the questionnaire to be tested (Attachments 3 and 4¹). It is expected that the instrument and interview protocol will evolve during testing. The research will be iterative, in that question wording and format design may change during the testing period in response to problems identified during the interviews.

To adequately test the survey, it is necessary to distribute the cognitive interviews across respondents who represent the primary experiences of the target population while also obtaining sufficient numbers of respondents with similar characteristics to test each survey section. We propose to conduct 30 interviews, all in Spanish. These interviews will be held with adults ages 18 to 65 who are part of the work force (e.g., not retired).

The interviews will be distributed as follows:

• 30 adults who hold at least one current certification or license:

  • 10 who hold multiple certifications or licenses from more than one state;

  • 10 who hold one or more certifications; and

  • 10 who hold one certification or license and are currently preparing for an additional certification or license.

The interviews will be conducted in two rounds, with one round of revisions to the items after the first round, based on the results of interviews conducted up to that point. If at any point during a round an item is perceived as not working—as evidenced by at least 3 respondents making the same or similar error or comment on that item—revisions may be made to that item on a case-by-case basis. Interviews will be audio-recorded. At their discretion, NCES staff may observe some of the interviews through a video-stream using GoTo Meeting video conferencing, hosted by the American Institutes for Research (AIR).

Consultations Outside the Agency

The ATES instrument was initially developed through the efforts of a government interagency working group – the Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA). For more information about GEMEnA’s developmental guidance on the ATES survey, please visit nces.ed.gov/surveys/gemena.

Recruiting and Paying Respondents

Participants will be recruited by AIR using multiple sources, including Craigslist ads, flyers on local community college and trade school campuses, and through personal and professional contacts. An example recruitment advertisement and e-mail are included in Attachment 1. People who have participated in cognitive studies or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating. Interested individuals will be asked to complete an online or phone screening questionnaire (Attachment 2). The 30 interviews projected to occur will take place in the AIR offices in the DC-Metro area, Austin, Texas, and San Mateo, California. To assure that participants agree to take part in the interviews and to thank them for their time and for completing the interview, each respondent will be offered $40.

Assurance of Confidentiality

Interviewers will read a confidentiality statement to respondents, and respondents will be given the opportunity to provide or decline consent before any interview is conducted (Attachment 1). The consent form includes the following statement in Spanish:

This cognitive interview has been approved by the Office of Management and Budget (OMB# 1850-0803). Your participation is voluntary. The information you give us will be combined with the responses of others in a summary report that does not identify you as an individual. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

No personally identifiable information will be maintained after the interview analyses are completed. Data recordings will be stored on AIR’s secure data servers.

Estimate of Hour Burden

We expect the interviews to be approximately one hour in length. Screening potential participants will require 3 minutes per screening. We anticipate it will require 12 screening interviews per eligible participant (thus an estimated 360 screenings to yield 30 participants). This will result in an estimated total of 48 hours of respondent burden for this study.

Table 1. Estimated response burden for ATES cognitive interviews and focus groups

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	360	360	0.05	18
Cognitive Interviews	30	30	1.0	30
Total	360	390	-	48

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

The project schedule calls for recruitment to begin in August 2017, as soon as OMB approval is received. Interviewing is expected to be completed within 2 months of OMB approval. After the interviews are completed, survey items will be finalized for the 2019 NHES administration.

Cost to the Federal Government

The cost to the federal government for this study is approximately $46,404.

1 Note that all attachments include both English and Spanish text for review purposes. Only the Spanish text will be used for the study.