Summary Report

National Center for Education Statistics

Attachment IV
Cognitive Interviewing, Round 1 – Summary of Results and Recommendations

1. Recruitment and Participation

Cognitive testing was conducted from January 26 to February 11, 2015, in three of RTI’s U.S. office locations – Research Triangle Park, NC; Washington, DC; and Chicago. To recruit graduates, RTI posted web advertisements on Craigslist and used direct outreach though advertisement postings at area businesses. A total of 42 individuals responded to the posted advertisements; of these, 19 were ineligible, 15 were eligible and participated, and 8 were eligible but did not participate due to cancellation, difficulty scheduling a time to participate, or not coming to their scheduled appointment. Participants were recent graduates (up to 4 years following completion of the credential) from institutions in the vicinity of RTI’s offices, including online program graduates residing in the same geographic areas. Of those who participated, nine were female and six male. Table 1 provides a distribution of the final number of interview participants, by control and level of institution.

Table 1: Number of respondents, by institution control and level

Institution control	Institution level	Credential type(s)	Number of respondents
Private for-profit	Less-than-2-year	Vocational certificates	3
	2-year	Vocational certificates and associates	2
	4-year	All	2
Private not-for-profit	Less-than-2-year and 2-year	Vocational certificates and associates	1
Public	Less-than-2-year and 2-year	Vocational certificates and associates	7

Participants were given the short paper survey and asked to complete it on their own. Upon completion, they were asked to think aloud retrospectively in order to identify potential problems related to question comprehension, recall, information editing due to sensitivity or social desirability and response mapping. On average, the interviews took 32 minutes to complete.

2. Results and Recommendations

2. 1 Introduction

The purpose of the Graduate Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment (34 CFR Parts 600 and 668).

The information on this survey will be used to report average earnings for program graduates to the U.S. Department of Education. Your individual earnings will never be disclosed or used in any other manner unless required by law.

While most interviewees read the introduction, three appeared only to skim the introduction, and two indicated that they had not read it at all. Of those who read the introduction, respondents generally agreed that the introduction was an informative and adequately succinct summarization, clearly communicating the purpose of the survey. Two participants thought the survey applied to people who received federal financial aid and a few commented that the gainful employment legislation part of the introduction was confusing.

Recommendation:

Remove reference to gainful employment legislation. Revise first sentence of the introduction to read:

The purpose of the Graduate Employment and Earnings Survey is to collect information about the average earnings of graduates from programs whose students receive federal financial aid and are subject to regulations on gainful employment (34 CFR Parts 600 and 668).

2. 2 Survey Questions

1. Please verify that you are _______________________________________. Is this correct? If you have legally changed your name, but once went by ______________________, please answer “Yes” below.

Yes

No  Please stop filling out the survey. It is important that you still return the survey in the envelope provided so that we will not contact you again.

Most interviewees noted that if there were a minor spelling error in their name, they would make the correction and continue filling out the survey. Several respondents noted that if there was an error in their name, they might return the survey if provided a postage-paid return envelope. However, six interviewees mentioned that, were the listed name completely incorrect, they would throw away the survey rather than return it.

Recommendation:

Shorten the instructions associated with the “No” response option to increase the chances of people reading them. Revise to read:

Thank you. Please return the survey in the attached envelope, so that we can remove you from our list.

2. According to our information, you completed the _______________________ program at _____________________________ between _______________ and ___________________. Is that correct?

Nearly all of the interviewees mentioned that they would have no difficulty recalling the year they began and ended their program; however, some reported that recalling the months might be more difficult. Several interviewees noted their ease in recalling the date was due to the recentness or significance of program completion.

Recommendation:

None.

3. Did you work at a job or business at any time during 2014?

Seven interviewees interpreted this question as asking about a job or business related to their degree program, despite the question not referencing this in particular. Their confusion may be due to the question’s proximity to the prior question which references the student’s degree program completion. The remainder of respondents generally saw no difference between “job” or “business,” and interpreted the question as “Did you have anywhere you earned income in 2014?” These respondents generally mentioned interpreting this question as excluding any unpaid work (e.g. unpaid internships or volunteer work).

Recommendation:

Revise to read: During 2014, did you work for pay at a job or business? (Source: modified Q29 of the American Community Survey—PAPI version).

4. Did you receive any wages or salary in 2014?

Respondents had no difficulty with “calendar year” and all interpreted as meaning January 1, 2014 – December 31, 2014. However, interpretations of the terms “salary” and “wages” were inconsistent. Seven interviewees interpreted “salary” as referring to a fixed yearly rate and wages as hourly pay, seven noted that they saw no difference between the two terms, and one interviewee interpreted salary as hourly pay and wages as a fixed yearly rate. In addition, some participants with more than one job (for example, a salaried job and part-time jobs, such as baby sitting, tutoring, or participating in studies) did not consider their part time jobs when answering this question.

Recommendation:

Start the series of questions on income with a common lead-in: “The next set of questions are about the different types of income you might have received in 2014.”

Additionally, include instructions to the question: “Please include both full-time jobs and part-time jobs.” Alternatively, revise the question to read: “In 2014, did you receive any wages or salary from full-time or part-time jobs?”

4a. How much did you receive in wages and salary from ALL jobs before taxes and other deductions in 2014? Be sure to include wages and salary from ALL jobs in 2014

Nearly all interviewees expressed having difficulty with this question. Interviewees were generally confused as to what to include and what to exclude in this amount, especially after encountering the remaining questions (5 – 6). Three respondents revised their initial answers to this question once they attempted to answer the following questions (5 – 6), mentioning that they thought first 4a included everything, but were then unsure. Several interviewees noted that they excluded part-time jobs or irregular pay (e.g. payment for babysitting, participation in surveys, gambling winnings, tips, bonuses).

Most interviewees mentioned that the amount they provided was a rough estimate; however, two said they had simply reported what was on their W-2, Wage and Tax Statement, which they had recently received. Two interviewees arrived at an amount by first calculating approximate monthly earnings before taxes and then multiplying by 12 months.

A majority of respondents noted that the amount which they provided excluded taxes and deductions, and had no trouble interpreting the term “deductions.” Two interviewees expressed confusion over how to enter the numeric amount given the “.00” at the end of the blank field.

All but one of the interviewees noted that they had no reservations providing income information, and did not feel the questions infringed on their privacy.

Recommendation:

Restructuring the questionnaire to include all income questions as subparts of one question, using a common introduction (see recommendation to question 4) should eliminate double counting as it will become obvious to respondents that questions 4a-6a are subcategories that should sum up to their total annual income.

To further ensure tips and bonuses are not double counted, follow the approach of the American Community Survey and combine questions 4a and 5a, asking respondents to report on “wages, salary, commissions, bonuses, or tips from all jobs.” Include instructions that all jobs should be considered whether or not they are related to the program of study.

In addition, use a box, rather than a blank field, where the requested dollar amount can be entered.

5. Did you receive any additional tips, overtime pay, bonuses, or commissions?

Three respondents indicated “No” on this question because they had already reported overtime and commission pay in the previous question (4a). Two interviewees treated this question as a catch-all category and interpreted it as asking about money made in addition to salary and wages, including “under the table” pay or for money made in small temporary jobs. One interviewee indicated “No” because she did not report her tips for tax purposes, and one mentioned excluding tips and commissions due to their job not being related to their degree program.

Recommendation:

Include tips, bonuses, overtime pay, and commissions as part of the overall income from wages and salaries in questions 4a.

5a. How much did you receive in tips, overtime pay, bonuses, or commissions from ALL jobs before taxes and other deductions in 2014? Be sure to include ALL tips, overtime pay, bonuses, or commissions in 2014.

Similarly to the previous question, the interpretation of this question and its relationship to question 4a was puzzling to most respondents. Five of the eight eligible to answer this question noted that they had already included tips, overtime pay, bonuses and commissions in their response to question 4a. Several interviewees mentioned that this figure would be hard to come up with; some of the recall strategies mentioned were estimation based on a minimum yearly figure, and estimation based on a monthly figure.

A self-employed person was confused whether to include his commission under 4a, or 6a, but never considered it in response to this question, that explicitly states commissions.

Recommendation:

Include instructions in question 4a to include this figure in the overall income from wages and salaries.

6. Did you receive any self-employment income? Report income from own business (farm or non-farm) including proprietorships and partnerships.

Interviewees generally had no issues determining how to answer this question. One interviewee who holds a part-time job in a salon, was unsure how to answer this question, as she rents space in the salon and pays for her own supplies. Several interviewees who were not self-employed, expressed difficulty determining the difference between proprietorships and partnerships when probed on the meaning by interviewers.

Recommendation:

None.

6a. What was the amount you received in self-employment income before taxes and other deductions in 2014? Report net income after operating expenses. If net income was a loss, mark the “Loss” box to the right of the dollar amount.

Four interviewees expressed confusion over what to enter in this field because they had already reported their self-employment income in question 4a.

While most of the interviewees had no issue identifying what net income after operating expenses was conceptually, two noted that they had reported the amount before operating expenses.

Three interviewees noted confusion over what “loss” would mean, with one believing that expenses for equipment would be considered a loss, and one thinking that “loss” pertained only to those dependent on the stock market for their income.

Recommendation:

Restructure the questionnaire to include all income questions as subparts of one question, using a common introduction (see recommendation for question 4). Graphically (using a different font or underlining) emphasize that the requested amount should be reported after business expenses. Remove the instructions for “loss” from the question stem, as they seem to confuse rather than help respondents.

Overall Impressions

Interviewees were generally pleased by the brevity and straightforwardness of the survey. Most noted that they had no privacy concerns about any of the questions and would leave none blank, but several emphasized that they would participate only if the survey looked official and was clearly from their prior institution or the U.S. Department of Education. Four interviewees emphasized their confusion about where to report what income, and what should be included and excluded in each question.

4