Volume 1 NHES 2015 ATES Cog Labs

Volume I

Request for Clearance for the

2015 National Household Education Survey (NHES)

Adult Training and Education Survey (ATES)

Cognitive Interviews and Concept Interviews

OMB# 1850-0803 v.92

December 12, 2013

Justification

Beginning in January 2015, the National Household Education Survey (NHES) is expected to field the first full-scale administration of the Adult Training and Education Survey (ATES). This will be a two-stage mail study. In the first stage households will be screened to determine if they contain eligible members. If eligible members are in the household, within household sampling will be performed. Finally, topical surveys will be administered to the selected household members. Out-of-high-school adults ages 16 to 65 will be eligible to receive the ATES.

The purpose of this submission is to conduct cognitive interviews and additional qualitative concept interviews to refine the ATES instrument for the 2015 data collection. This research focuses specifically on formal on-the-job training experiences (such as apprenticeships and internships) certificate holders, individuals with low levels of literacy, and a new Spanish language translation. We will also conduct limited retesting of certification and license items. Previously, ATES conducted focus groups on characteristics of education certificates and participation in work-related training, cognitive interviews to test new questions, and a response rate pilot study in early 2013, and is using early results to evaluate responses and skip patterns as input to questionnaire revision. The proposed research will build on these activities.

Cognitive and qualitative testing has been used for ATES and other NHES surveys in past years. The primary objective of the interviewing in 2014 is to identify and correct problems of ambiguity or misunderstanding in question wording for characteristics that appear to have been over-reported in prior testing (certificates, apprenticeships), improve comprehension for a variety of respondents that have not been included in prior testing (Spanish language speakers, adults with low levels of English literacy), and develop new items that will broaden the focus of the apprenticeship section to cover a wider variety of formal on-the-job training experiences. The interviews should result in a set of questionnaires that are easier to understand and therefore less burdensome for respondents, while also yielding more accurate information. The primary deliverable from this study will be the revised questionnaires. A report highlighting key findings will also be prepared.

The interviews will focus on three key areas:

Understanding of concepts. For formal work-related training, we will conduct concept interviews to determine respondents’ understanding of key characteristics of formal training experiences that occur on the job. The objective of this research is to broaden the focus of the current apprenticeship section to cover internships, residencies, and externships, concepts that overlap with apprenticeships and that respondents tend to report in the apprenticeship section even when asked to exclude them. These interviews will ensure that the characteristics and terminology used in the new items for this broader section capture the key information about formal training experiences as understood by those who have actually participated in them and that the questions needed to distinguish these various work experience programs will be included in the survey.

Question wording. The cognitive interviews conducted for the feasibility study showed some problems with question wording and comprehension, particularly for the certificates section and for people with lower reading ability. These items were revised for the feasibility study and tested with a small number of respondents. The current research seeks to test these items further with individuals with low educational attainment, certificate holders, and respondents who are misreporting having an educational certificate. Earlier waves of testing have identified considerable over-reporting of certificates. Speaking with respondents who appear to be falsely reporting that they have a certificate will allow us to further refine the main certificate item with the aim of reducing these over-reports.

Spanish translation. The ATES instrument will be translated into Spanish for the 2015 NHES administration. Approximately 5% of ATES returns are expected to be in Spanish. The questionnaire has not been tested in Spanish yet; these interviews will help ensure that the questionnaire items remain clear after they have been translated to Spanish.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions, or to answer a series of questions about the items they just answered. Techniques include (1) asking probing questions, as necessary, to clarify points that are not evident from the think-aloud comments and (2) responding to scenarios. Probes that will be used include:

• probes to verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating),

• probes about the meaning of specific terms or phrases used in the questions, and

• probes for experiences or ideas that the respondent did not think were covered by the question but we would have considered relevant.

This research will also include qualitative concept interviews (conducted via telephone) about formal work-related training that will focus on respondent language used to describe the characteristics of their training experiences. It is important to conduct these interviews because prior cognitive testing has shown that apprenticeships are not clearly distinguishable from internships and externships, and these other types of formal training experiences have not previously been measured in ATES.

All interviews are expected to last about 1 hour and will be conducted by trained interviewers. This submission includes the questionnaire to be tested for the initial cognitive interviews, and the protocol for conducting the concept interviews. The research will be iterative in that question wording and forms design may change during the testing period in response to early findings.

To adequately test the instruments, it is necessary to distribute the interviews across respondents who represent the major variations of experience in the target population and, correspondingly, to set the total number of participants so as to obtain sufficient numbers of similarly-situated respondents.

We propose to conduct a total of 60 interviews, in two rounds.

Round 1 will consist of 36 interviews:

Formal on-the-job training

• We will conduct 12 phone concept interviews with people who are in or have completed a formal on-the-job training experience, such as an apprenticeship, internship, residency, or externship (including student teaching). We will include a variety of occupations.

Low literacy

• We will conduct interviews with 6 people who have a high school education or less. We will focus on respondents with occupations such as construction, cosmetology, health-care support, or skilled trades that are likely to require the credentials that are asked about in ATES.

Certificate holders

• We will conduct 8 to 12 interviews with people who report that they have a certificate in response to the main certificate question. Their highest level of education will be some college or less (no degree). Half of these will be people who seem to be falsely reporting a certificate based on their response to the hours of instruction question in the screener; individuals who report having a certificate that required less than 40 hours to complete will be considered likely false reporters. The remainder will be anyone who reports a certificate of 40 hours or more.

Spanish-language

• We will conduct 6 interviews with individuals whose primary language is Spanish. We will focus on respondents with occupations such as construction, cosmetology, health-care support, or skilled trades that are likely to require the credentials that are asked about in ATES.

Round 2 will consist of 24 interviews:

The second round interviews will explore issues identified in the first round of interviews. For example, the on-the-job training concept interviews and the Round 1 cognitive interviews will result in revisions to the apprenticeship section, so the Round 2 interviews will include additional cognitive testing of this section. Also, if the Round 1 interviews with certificate holders result in changes to the certificate section, the Round 2 interviews will include additional cognitive testing of this section. Finally, if the Round 1 interviews with low literacy individuals or early ATES Feasibility Study responses suggest that respondents are having trouble with the grids in the certification and training sections, the Round 2 interviews may test how easily respondents navigate revised sections.

Additional Spanish-language interviews will be conducted with at least 6 primarily Spanish-speaking respondents to ensure that the Round 1 changes to the English-language version are clear in Spanish.

Consultations Outside the Agency

NCES has an ongoing consultation with a government interagency working group – the Federal Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA) – that has been involved with the survey design since its inception. GEMEnA conducted 3 focus groups in January 2013 on characteristics of educational certificates and in May 2013 on work-related training to assist in questionnaire development. Additionally, over 40 cognitive interviews were conducted prior to the 2014 feasibility study. Finally, NCES conducted a nationally representative response rate pilot for adults in early 2013, the National Adult Training and Education Survey or NATES, and is evaluating response options and skip patterns based on the early results.

Recruiting and Paying Respondents

To assure that participants from all desired populations agree to take part in the cognitive interviews and to thank them for their time and for completing the interview, each will be offered $40.

Participants will be recruited by AIR, using multiple sources, including listservs and personal contacts. An example recruitment e-mail is included below. People who have participated in cognitive studies or focus groups in the past 6 months will not be eligible to participate. The items used to screen respondents for participation are included in the submission. We anticipate it will take 3 minutes per screening interview. Interviews will take place in the DC-Metro area; respondents will be reimbursed for parking expenses.

Dear [contact]:

The American Institutes for Research is assisting the National Center for Education Statistics, of the U.S. Department of Education, and other federal statistical agencies in learning more about how adults get the education, training, and credentials they need for work. We are specifically looking for adults to participate in a one-time, voluntary, research interview about these topics.

Participants will take part in a 60-minute interview on [DATE] and will receive $40 for their participation.

If you or someone you know is interested in participating please contact [contact info].

Thank you for your time and help with this important study.

Sincerely,

Recruitment Contact Info

Assurance of Confidentiality

Participation is voluntary and respondents will sign a consent form before interviews are conducted. The consent form states the following: “The American Institutes for Research is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act [US code title 20, section 9543]. Your participation is voluntary. Your responses are protected from disclosure by federal statute [US code title 20, section 9573]. All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.”

No personally identifiable information will be maintained after the cognitive laboratory analyses are completed.

Estimate of Hour Burden

We expect the concept and cognitive interviews to be approximately one hour in length. This will result in a maximum of 60 burden hours for the interviews. Screening potential participants will require 3 minutes per screening. We anticipate it will require 12 screening interviews to identify one eligible participant (thus an estimated 720 screenings to yield 60 participants). This will result in 36 hours of burden for the screener and an estimated total 96 hours of respondent burden for this research.

Table 1. Estimated response burden for ATES Concept and Cognitive Interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	720	720	0.05	36
Concept and Cognitive Interviews	60	60	1.0	60
Total	720	780	-	96

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

The project schedule calls for the cognitive research focusing on the topical interviews to begin by early January 2014 and to be completed by April 12. After the cognitive interviews are completed, data collection instruments will be revised and submitted to OMB as part of a request for clearance for the January 2015 NHES study.

Consent Form

The American Institutes for Research is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act [US code title 20, section 9543]. Your participation is voluntary. Your responses are protected from disclosure by federal statute [US code title 20, section 9573]. All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law.

Please sign below to indicate that you have read the assurance of confidentiality.

Print Name ___________________________________

Signature _____________________________________

Date_________________________________________