Employment and Earnings Survey Cog Labs Volume I

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: January 15, 2015

TO: Shelly Martinez, OMB

FROM: Sharon Boivin, NCES

THROUGH: Kashka Kubzdela, NCES

SUBJECT: Employment and Earnings Survey 1^st Round Cognitive Testing (OMB# 1850-0803 v.123)

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803 v.123) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pilot tests, focus groups, and cognitive interviews. This request is to conduct the first round of cognitive interviews to aid in developent of employment and earnings items for the National Household Education Survey (NHES) program. These items will also inform the U.S. Department of Education’s development of a new gainful employment survey that will be used by postsecondary programs to survey graduates about their earnings. The data collection for this study is being carried out for NCES by RTI International.

The NHES collects earnings information as part of the Adult Training and Education Survey. This survey is designed to collect nationally-representative data on the attainment of non-degree credentials and the labor market characteristics of individuals with such credentials. As part of its submission for OMB clearance for the full scale 2016 NHES, NCES will propose a small pilot test to compare collecting point estimates versus categories for respondent earnings. If it is feasible to collect point estimates, NCES can derive variables reflecting poverty status to include on public use files and licensed analysts will have more sensitivity for modeling the relationship between credential attainment and labor market outcomes. The cognitive interviews described in this submittal will test the proposed survey item language. For ease of reference, the items proposed here for testing are referred to as the Employment and Earnings Survey.

The proposed cognitive interviews will also inform the gainful employment survey, which will be used by institutions to appeal program-level debt-to-earnings ratios calculated by the U.S. Department of Education under the gainful employment regulations (34 CFR Parts 600 and 668). The earnings component of the debt-to-earnings ratio (D/E ratio) is provided by the Social Security Administration, but institutions may calculate an alternative earnings measure by administering a survey to their program graduates.

NCES will use cognitive interviews to refine the the Employment and Earnings Survey questions to maximize the quality of data collected and provide information on issues with important implications for the survey design, such as the following:

• The manner and degree to which respondents comprehend the questions and instructions;

• Memory search strategies commonly used and cues found to facilitate recall;

• Memory demands of the questions and the answering strategies used;

• Respondents’ ability to make calculations and judgments;

• Sources of burden and respondent stress; and

• Response domain coverage by the provided response options.

This request is to recruit and screen potential respondents and to conduct cognitive testing of the Employment and Earnings Survey. A Technical Review Panel (TRP) met on December 2, 2014 (TRP participants are listed in Attachment I), and provided input on the survey items drawn primarily from similar items from the American Community Survey (ACS). A summary report of the cognitive testing results will become available in February 2015. This submission includes recruitment and screening materials, respondent informed consent form, the survey items to be tested, and the protocol for conducting the cognitive interviews.

Design and Context

Cognitive testing of the Employment and Earnings Survey will be conducted for NCES by its contractor, RTI International (RTI), in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago. Participants will be recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. RTI will recruit sufficient numbers of graduates to ensure 15 completed interviews. To recruit graduates, RTI will advertise on a variety of websites including Facebook and Craigslist, as well as use direct outreach such as postings at area businesses. In addition, institutional representatives on the TRP have volunteered to assist with recruiting their graduates. Attachment II presents the materials that will be used for recruitment of cognitive interview participants.

Once a candidate expresses interest in participating in the cognitive interview, one of the designated recruiters will use a set of scripted screening questions (Attachment V) to determine his/her eligibility for the cognitive interview. The script contains questions on the participant’s current enrollment status, completion of a program within the reference period, type of program, and institution attended.

The desired sample will include a mix of respondents who are (1) recipients of a degree or certificate from any level of private for-profit institutions and (2) recipients of a certificate, associates degree, or other vocational credential at public and private not-for-profit, 2-year and less-than-2-year institutions. To help inform gainful employment regulations, the sample will overrepresent graduates from private for-profit less-than-2-year and 2-year institutions. Such schools had higher than average metric failure rates. Table 1 provides a distribution of the desired minimum number of interview participants by control and level of institution.

Table 1: Desired minimum number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Minimum number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	4
	2-year	Vocational certificates and associates	4
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	2
Public	Less-than-2-year and 2-year	Vocational certificates and associates	3

Cognitive testing will involve one-on-one interviews in which the respondent is asked to “think aloud” as he or she answers survey questions and to answer additional questions about the items they just answered. The survey will be developed and tested as a paper-and-pencil instrument. The survey items to be tested are provided in Attachment VII; item justifications are provided in Attachment IV. Each testing session will last about 40 minutes. In order to encourage participation in this study, potential respondents will be offered a $40 incentive for their participation. Arrangements will be made to allow NCES staff to observe the testing and a recording of each interview will be available to NCES for review.

The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer survey questions and to pinpoint potential sources of question misinterpretation in their responses. A retrospective protocol can provide a valuable source of evidence about participants’ comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To evoke relevant responses, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions, to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes – e.g., “How certain were you of your answer?” or “How easy or difficult was it to answer this question?” – seek to verify respondent interpretations, investigate the meaning of specific, potentially ambiguous phrases, or elicit notions that the respondent thought were critically relevant to, but absent from, the question. A cognitive interviewing protocol (Attachment VI) will be used to guide the flow and content of the cognitive interviews. Instructions for cognitive interview respondents and probes inquiring about the meaning of key terms or constructs are included in the protocol. Interviewers will refer to the protocol to guide the content of the interviews, but will also be free to deviate from the guide should participants have difficulty answering questions that do not have scripted probes or prompts.

Immediately following the conclusion of each interview, methodologists will review the recordings and notes and highlight emerging themes. Following each interview, the digital audio recording will be archived for qualitative analysis. RTI will organize its observations and summarize the themes, insights, and ideas gleaned from the interviews into a report to be submitted to NCES.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form (Attachment III) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study and destroyed after the final report is submitted.

Schedule for Development of the Employment and Earnings Survey

RTI will begin recruiting for the cognitive testing round upon receiving OMB clearance, and the testing is scheduled to begin as soon as possible in January of 2015. Recommended changes will be made and the revised survey items presented to a panel of experts convened by the National Institute of Statistical Sciences in February 2015. Based on their feedback, a second draft of the survey will undergo another round of cognitive testing in early March 2015. The final draft of the survey will be used in a pilot test with approximately 3,400 program graduates beginning in April 2015. All survey item development activities need to be completed by the end of June 2015 for inclusion in the clearance request for the 2016 collection of NHES and to inform the gainful employment regulations that will be posted in the Federal Register by July 1, 2015. The requests for the second round of cognitive interviews and for the pilot test will be submitted to OMB as soon as all required materials become available.

Estimate of Respondent Burden

To yield 15 completed interviews, we anticipate screening up to 60 individuals for eligibility to achieve the desired distribution of program graduates. The screening process, on average, is estimated to take about 5 minutes per person.

Table 2: Estimate respondent burden

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Screening	60	60	5	5
Cognitive interview	15	15	40	10
Study Total	60	75		15

Cost to Federal Government

The cost to the federal government for developing and cognitive and pilot testing the survey will be $555,570, which includes contractor staff time, incentives, and project materials.

3