Volume 1 NHES 2016 ATES Cog Labs

Volume I

2016 National Household Education Surveys Program (NHES)

Adult Training and Education Survey (ATES)

Cognitive Interviews

OMB# 1850-0803 v.127

February 04, 2015

Justification

The National Household Education Survey (NHES) is a data collection program of the National Center for Education Statistics (NCES) aimed at providing descriptive data on the educational activities of the U.S. population, with an emphasis on topics that are appropriate for household surveys rather than institutional surveys. Such topics have covered a wide range of issues, including early childhood care and education, children’s readiness for school, parents’ perceptions of school safety and discipline, before- and after-school activities of school-age children, participation in adult and career education, attainment of non-degree credentials, parents’ involvement in their children’s education, school choice, homeschooling, and civic involvement. NHES uses a two-stage design in which sampled households complete a screener questionnaire to enumerate household members and their key characteristics. Within-household sampling from the screener data determines which household member receives which topical survey. NHES typically fields 2 to 3 topical surveys at a time, although the number has varied across its administrations. Surveys are administered in English and in Spanish.

Beginning in 1991, NHES was administered roughly every other year as a landline random-digit-dial (RDD) survey. During a period of declining response rates in all RDD surveys, NCES decided to conduct a series of field tests to determine if a change to self-administered mailed questionnaires would improve response rates. After a 5-year hiatus in data collection for this developmental work, NCES conducted the first full-scale mail-out administration with NHES:2012, which included the Early Childhood Program Participation (ECPP) and the Parent and Family Involvement in Education (PFI) surveys. In 2016, the NHES is expected to field the first full-scale administration of the Adult Training and Education Survey (ATES). This will be a two-stage mail study. In the first stage, households will be screened to determine if they contain eligible members. If eligible members are in the household, within household sampling will be performed. Finally, topical surveys will be administered to the selected household members. Out-of-high-school adults ages 16 to 65 will be eligible to receive the ATES.

The ATES provides new measures of adults’ educational and occupational credentials. It identifies adults who have educational certificates, including the subject field of the certificate, its perceived labor market value, and its role in preparing for occupational credentialing; and counts adults who have an occupational certification or license, including the number of such credentials, type of work they are for, their perceived labor market value, and the role of education in preparing for these occupational credentials. To get a comprehensive picture of adult education and training, the survey also includes brief sections on adult participation in work experience programs (such as apprenticeships) and college classes.

The purpose of this submission is to conduct cognitive interviews to refine the ATES instrument for the 2016 data collection. To develop the ATES instrument, NCES has: conducted focus groups in 2013 on characteristics of education certificates and participation in work-related training; conducted cognitive interviews in 2013 and 2014 to test English versions of the ATES instrument; administered a response rate pilot study in early 2013; and administered a feasibility study in 2014 to test processes for incorporating ATES into the NHES program. The proposed research will build on these activities by conducting a final series of cognitive testing of the 2016 ATES instrument. This testing will be in English only. A separate OMB package will be submitted for testing the Spanish version of the instrument after the English version has been finalized.

Qualitative cognitive testing has been used for ATES and other NHES surveys in past years. Since the last cognitive testing of the ATES instrument, however, a number of changes have been made to the instrument that require additional testing. These include slightly changing wording in a few items (mainly to simplify questions), switching to asking about the “most important” certification rather than “the last” certification, changing from a grid format to a repeating series of questions about up to three certifications, and adding a new series of questions on work-experience programs. The objective of the proposed set of cognitive interviews is to test the instrument in its new format, and to specifically identify and correct problems of ambiguity or misunderstanding in question wording. The interviews should result in a questionnaire that is easy to understand for respondents and therefore less burdensome, while also yielding accurate information. The primary deliverable from this study will be the revised, final English-language version of the ATES questionnaire. A report highlighting key findings will also be prepared. This request is to conduct cognitive interviews to test and revise the ATES questionnaire, focusing on potential issues with items that were not included in previous ATES testing.

Design

Cognitive interviews are intensive, one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions or to answer a series of questions about the items they just answered. Techniques include asking probing questions, as necessary, clarifying points that are not evident from the think-aloud comments, and responding to scenarios. In these interviews, probes will be used to:

• verify respondents’ interpretation of the question (e.g. asking for specific examples of activities in which the respondent reports participating);

• check respondents’ understanding of the meaning of specific terms or phrases used in the questions; and

• identify experiences or concepts that the respondent did not think were covered by the questions but we consider relevant.

Interviews are expected to last about 1 hour and will be conducted by trained cognitive interviewers. This submission includes the protocol that will be used to conduct the interviews and the questionnaire to be tested. It is expected that the instrument and interview protocol will evolve during testing. The research will be iterative, in that question wording and format design may change during the testing period in response to problems identified during the interviews.

To adequately test the survey, it is necessary to distribute the cognitive interviews across respondents who represent the primary experiences of the target population while also obtaining sufficient numbers of respondents with similar characteristics to test each survey section. We propose to conduct 49 interviews with adults ages 18 to 65 who are part of the work force (e.g., not retired, not full-time students), with an emphasis on adults who have less than a Bachelor’s degree. The interviews will be distributed such that at least:

• 21 adults have participated in a work experience program;

• 14 adults have at least one professional certification or license;

  • Of these 14, 7 adults have two or more professional certifications or licenses;

• 14 adults have at least one educational certificate;

• 5 adults have taken at least one college-level course in the past 12 months;

• 5 adults do not have these credentials (to represent “general population”) and are currently employed;

Note: The sum of the categories listed above is greater than the total because several respondents are expected to fall into multiple categories.

Revisions to the questionnaire will be made on an ongoing basis, depending on the results of interviews conducted up to that point. Typically, we expect to conduct at least 3 interviews prior to making a change to question wording. Interviews will be audio-recorded. NCES staff may also observe interviews either in person or through a video-stream using WebEx video conferencing, hosted by the American Institutes for Research (AIR). The WebEx is an optional method for observing interviews and will only be used at the direction of NCES.

Consultations Outside the Agency

The ATES instrument was initially developed through the efforts of a government interagency working group – the Interagency Working Group on Expanded Measures of Enrollment and Attainment (GEMEnA). GEMEnA has been involved with the survey design since its inception and continues to provide expert advice on both policy and methodological issues. For more information about GEMEnA’s developmental guidance on the ATES survey, please visit nces.ed.gov/surveys/gemena.

Recruiting and Paying Respondents

To assure that we are able to recruit participants from all desired populations and to thank them for their time, travel, and for completing the interview, as in previous rounds of ATES cognitive interviews, each will be offered $40. Participants will be recruited by AIR, and its subcontractor, Shugoll Research, using multiple sources, including company databases, social media/Craig’s List, and personal and professional contacts. An example recruitment e-mail is included in Attachment 1. People who have participated in cognitive studies or focus groups in the past 6 months and employees of the firms conducting the research will be excluded from participating. The questions used to screen respondents for participation are included in Attachment 2. The cognitive interview protocol and the ATES questionnaire are included in Attachments 3 and 4. We anticipate it will take 3 minutes to complete each screening interview. Interviews will take place in the AIR offices in the DC-Metro area (30 interviews), and San Mateo, California, office (14 interviews) and in quiet, public places, such as a library or community center in the Grand Rapids, Michigan area (5 interviews).

Assurance of Confidentiality

Participation is voluntary and respondents will read a confidentiality statement and sign a consent form before interviews are conducted. The confidentiality statement and consent form are provided in Attachment 1. No personally identifiable information will be maintained after the cognitive interview analyses are completed. Primary interview data will be destroyed on or before December 31, 2015. Data recordings will be stored on AIR’s secure data servers.

Estimate of Hour Burden

We expect the cognitive interviews to last approximately one hour. Screening potential participants will require 3 minutes per screening. We anticipate it will require 12 screening interviews per eligible participant (thus an estimated 588 screenings to yield 49 participants). This will result in 30 hours of burden for the screener, and an estimated total 79 hours of respondent burden for this study.

Table 1. Estimated response burden for ATES cognitive interviews

Respondents	Number of Respondents	Number of Responses	Burden Hours per Respondent	Total Burden Hours
Recruitment Screener	588	588	0.05	30
Cognitive Interviews	49	49	1.0	49
Total	588	637	-	79

Estimate of Cost Burden

There is no direct cost to respondents.

Project Schedule

The project schedule calls for recruitment to begin as soon as OMB approval is received. Interviewing is expected to be completed within 3 months of OMB approval. After the interviews are completed, the data collection instrument will be revised.

Cost to the Federal Government

The cost to the federal government for this cognitive laboratory study is approximately $62,000.

Attachment 1

NHES 2016 ATES Cognitive Interviews Communication Materials

Table of Contents

Email text for personal and professional contacts

Text for advertisements on social media and/or Craig’s List

Informed Consent Form

Email text for personal and professional contacts

Dear [contact]:

The American Institutes for Research and Shugoll Research are assisting the National Center for Education Statistics, of the U.S. Department of Education. We are specifically looking for adults ages 18 to 65 who have a credential for work, such as a certification, license, or post-high-school certificate, to participate in a one-time, voluntary, research interview.

Eligible participants will take part in a 60-minute interview and will receive $40 for their participation.

If you or someone you know is interested in participating, please contact [contact info].

Thank you for your time and help with this important study.

Sincerely,

Recruitment Contact Info

Text for advertisements on social media and/or Craig’s List

Do you have a work credential, certification, or certificate?

You may be eligible to earn $40 by participating in a research interview about your education and work credentials.

About the interviews:

American Institutes for Research (AIR) is conducting interviews for a research study sponsored by the National Center for Education Statistics (NCES) of the U.S. Department of Education. The research interviews will be conducted between XXXX dates, 2015 and will be held at a central location convenient to major highways or public transportation. The interview will last approximately 60 minutes, during which time a researcher will ask you to answer written survey questions, including providing your interpretation of the survey questions. Your participation and responses are voluntary.

Eligibility to Participate:

We are looking for adults ages 18 to 65. Please call us between 9 a.m. and 5 p.m. weekdays to complete a brief telephone screening in order to determine if you are eligible. The telephone screening will take approximately 3 minutes to complete.

Contact:

<CONTACT NAME>

<CONTACT INFORMATION>

INFORMED CONSENT - ADULT

Informed Consent -Adult

The American Institutes for Research (AIR) is assisting with data collection for a study being conducted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education. We are interviewing adults ages 18 to 65 to gather information that will improve a national survey about credentials for work. For this study, a staff member will meet with participants individually for 60 minutes at <LOCATION>. Participants will be given $40 to thank them for their time and participation.

Participants will be asked to think aloud while they read the survey questions. Additionally, the interviewer will ask targeted questions to gain a better understanding of the survey questions. Participants’ responses will be audio-recorded; the information collected during this study will be for research purposes only. Your participation is voluntary, you can skip any question you do not want to answer, and you may request to stop at any time during the session. We do not anticipate that any of the questions will make you uncomfortable or upset. However, you may refuse to answer any question or take a break at any time. There are no direct benefits to you for participating in this study, but these interviews will help us develop and improve the survey instrument.

Before you are eligible to participate, the attached consent form is required.

If you have questions about this research study, please contact xxxx xxxxx at the American Institutes for Research at xxx-xxx-xxxx.

Thank you,

<CONTACT NAME>

<CONTACT INFORMATION>

Name (Please Print): ________________________

I have read the attached consent form for the National Household Education Survey study and all of my questions at this time have been answered. I understand that my session will be audio-recorded.

This cognitive interview has been approved by the Office of Management and Budget (OMB# 1850-0803). Your participation is voluntary. The information you give us will be combined with the responses of others in a summary report that does not identify you as an individual. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C. § 9573].

I agree to participate in this study.

_____________________________

______________________________