2015 National Household Education Survey (NHES) Adult Training and Education Survey (ATES) Cognitive Interviews and Concept Interviews

Cognitive Interview Protocol

Adult Training and Education Survey (ATES)

December 12, 2013

General Introduction

I would like to begin by thanking you for coming here today to help us out. We’re scheduled to be here for about 1 hour, and I won’t keep you any longer.

The reason we’ve asked you here is to help with a survey that is run by the U.S. Department of Education to describe adult’s education and training experiences. Today I’m going to ask you to fill out the survey questionnaire and provide feedback. This will help the Department of Education make sure the questionnaire will be understood by all different types of people. I don’t work for the Department of Education, so feel free to be honest with your comments. My role is to find out how YOU interpret the questions.

• [IF OBSERVER PRESENT]: One [a few] of my colleagues will be observing our discussion from behind this mirror. They are here to make sure that I don’t forget to ask you any of the important questions that you and I are supposed to cover. Toward the end of the interview, I will step out to ask them if there is anything I forgot to discuss with you. Is that OK with you? [IF NOT, ASK OBSERVERS TO LEAVE]

• Because it would be hard to keep track of everything you say today, I would like to audio-record this session. Is that OK with you? [IF NOT, CONTINUE WITHOUT RECORDING]

• And I’ll also be taking notes, so you might notice me writing as we are talking.

• I want you to know that your participation is voluntary and your responses are protected from disclosure by federal statute [US code title 20, section 9573]. All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose, unless otherwise compelled by law. We want you to know that you can speak freely. Also, you do not have to answer any question you don’t want to and can stop the interview at any time if you wish.

• There are no right or wrong answers. Today we are more interested in how you answer the questions than what the answers are. Your answers will not be counted as part of the real survey.

In a few minutes I'm going to hand you the questionnaire. I'd like you to fill it out the same way you would if it came to you at home in the mail. Unlike being at home, though, I’ll ask you to stop periodically so that we can talk about what you were thinking as you answered the questions. If any questions are unclear to you as you go through, please stop and let me know.

Do you have any questions before we begin? [TURN ON THE AUDIO RECORDER NOW.]

Specific Probes for Main Questionnaire

INTERVIEWER - If you feel there was uncertainty about a respondent’s answer, probe retrospectively on those items. Pay attention to respondent behavior here. Do they hesitate? Ask questions? Make remarks that indicate the question is awkward? If yes, PROBE.

Pay attention to SKIP errors. Do they have trouble finding the start of the next section? Do they have trouble figuring out which question to go to next? Do they go to the wrong question? If you notice they go to the wrong question, give them a chance to figure it out on their own, but if they don’t, point out to them that they skipped to the wrong question, PROBE about what was confusing.

Pay attention to how they answer the multiple response items. Do they notice when questions change from “mark one” to “mark all that apply”?

Pay attention to how they answer the grids. Do they fill in the correct cells?

After the respondent completes each section of interest [Certification and Licensure/Certificates/Apprenticeships/Other training]:

General Probes
Problem items	What is this question asking, in your own words?
	What were you thinking when you answered this question?
Problem with instructions	Tell me what this introduction/instruction is telling you.
Problems with clarity	Can you tell me in your own words what this question is asking? What does this word [term] mean to you? [in reference to a particular word or technical term] Tell me what you were thinking when I asked you about this [topic, time period, degree, certification, etc].
Skip and marking issues	I noticed you skipped this/checked this. How did you choose this answer and not this other option?
	Can you tell me how you chose this category? How easy or hard was it to find your answer on that list?

	CERTIFICATION AND LICENSURE Qs 3-9 – STOP RESPONDENT AFTER 9
I’d like you to stop here.
Q5b	How easy or difficult was it to use table B to answer this question? Would other words or labels be useful in that table?
Q5g	Was it easy or difficult to remember the year that you first earned the certification or license?
	EDUCATIONAL CERTIFICATES Qs 10 to 25 – STOP RESPONDENT AFTER 25
I’d like you to stop here.
Q10	Did you read all of the question? Can you tell me in your own words what this question is asking? Note any cues that the respondent did not read all of the text (ex: answering so quickly there is no way they read it all)
Q11	When you were trying to remember how many hours it took, (probe to find out if respondent included time spent studying, time in class, how respondent accounts for online courses, practicum), what did you include in that calculation? How easy or difficult was it to come up with your answer to this question? Tell me about this certificate. Why and how did you get it? Who gave it to you? (If does not seem to be an educational certificate: Why did you decide to count this as an educational certificate?)
Table C	INTERVIEWER: Turn to Table C. Note if the respondent used this table. If the respondent used the table, ask: How easy or difficult was it to use Table C to answer this question? Were you able to find an option that fits your siuation, or do you think there were any options missing? Can you tell me more about how you chose your answer from this table? If chose ‘other’: What was the field of study for your certificate?
Q19	When you were trying to remember how many hours it took, (probe to find out if respondent included time spent studying, time in class, how respondent accounts for online courses, practicum), what did you include in that calculation? How easy or difficult was it to come up with your answer to this question? Tell me about this certificate. Why and how did you get it? Who gave it to you? (If does not seem to be an educational certificate: Why did you decide to count this as an educational certificate?)
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from question 25.

	ON-THE-JOB TRAINING PROGRAMS Qs 26 to 35 – STOP RESPONDENT AFTER 34
I’d like you to stop here.
Q26	Was this question easy or difficult to answer? Why did you select [________]? [If Q26=“yes”]: Tell me about your program. What job was it for, how was the program structured, and why did you participate in it?
Q27	What does “sponsored the program” mean to you? Were you able to find an option that fits your situation, or do you think there were any options missing?
Q29	What does “educational program” mean to you?
Q30	[If one of the “yes” options selected]: Can you tell me more about how you chose your response to this question? [If only one response selected]: Did you notice that you can choose more than one response? To you, what does it mean “to take classes from an employer or joint employer-union sponsor”?
Q31	Were you able to find a response option that fits your experience? (If not, what would be a better option for you?) [If one of the wage options selected]: Can you tell me more about the wage you received during this program?
Q34	In your own words, what is this question asking you?
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from question 35.

	OTHER TRAINING FOR WORK Qs 45 to 51 – STOP RESPONDENT AFTER 52
I’d like you to stop here.
Q45	Were there any parts of this question that you found confusing or did not understand? Note any cues that the respondent did not read all of the text (ex: answering so quickly there is no way they read it all) Can you tell me in your own words what the term “work-related training” means? What types of experiences do you think you should include in your answer?
Q46	How easy or difficult was it to answer this question?
Q47	INTERVIEWER: note anything the respondent says, comments, or long pauses while trying to fill out the grid. Also note any questions the respondent asks you. Let’s take a look at this page. What did you think when you saw the grid? Was this grid easy or difficult for you to complete? IF DIFFICULT: what made it difficult? How could we make this easier for you to complete? Even if respondent tells you that it was not difficult but you noticed that the respondent paused or took a long time to respond, ask the respondent what they were thinking while they paused.
Q47a-b	INTERVIEWER: note whether respondent used table D. How easy or hard is it to remember your most recent training?
Q47c	How easy or difficult was this to answer? Can you tell me more about how you came up with an answer? (probe on how respondent calculated the number of hours)
Q47d-f	How easy or hard was it to choose an answer?
Table D	INTERVIEWER: Turn to Table D. Can you tell me more about how you chose your answer from this table? Are any of these categories confusing? If the respondent used the table, ask: Were you able to find an option that fits your situation, or do you think there were any options missing?
Q48	How easy or difficult was this to answer? How did you come up with your answer?
Q50	Can you tell me in your own words what this question is asking?
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from Question 52.

	EMPLOYMENT Qs 52 to 70 – STOP RESPONDENT AFTER 70
I’d like to stop you here. Were there any questions so far in this section that were confusing or a bit difficult for you to answer? Which ones?
You can continue.
Once respondent has completed question 71: I’d like you to stop here.
Q71	Can you tell me in your own words what this questions was asking? In this question, what does “training path”mean to you?
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

IF TIME IS RUNNING SHORT, INTERVIEW CAN END HERE.

Go ahead and finish the rest of the survey now and let me know if any questions are confusing or difficult to answer in any way or if you are unsure of anything.

	BACKGROUND Qs 73 to 81 – STOP RESPONDENT AFTER 81
Were any questions in this section confusing or a bit difficult for you to answer? Which ones? PROBE.
FINAL OBSERVATIONS
	Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention? REMEMBER TO ASK NCES STAFF FOR INPUT, GIVE GIFT CARD AT END OF INTERVIEW
	Thank you!

Additional Notes:

7