2014 National Household Education Survey (NHES) Adult Training and Education Survey (ATES) Cognitive Interviews

NHES ATES Cognitive Interview Protocols

July 19, 2013

Part B. Adult Training and Education Survey (ATES)

General Introduction

I would like to begin by thanking you for coming here today to help us out. [IF OBSERVER IS PRESENT: This is my colleague from the U.S Department of Education who will be listening in on the interview today.] We’re scheduled to be here for 60 minutes, and I won’t keep you any longer. The reason we’ve asked you to come is that the US Department of Education is conducting a survey to describe adults’ education and job training experiences. Today I’m going to ask you to fill out a questionnaire that is being developed for 2014 and to provide feedback so that we can help the Department of Education make sure the questionnaire is clear and easy to fill out. I don’t work for the Department of Education, so feel free to be honest with your comments. My role is to find out from you how YOU interpret the questions.

• Because it would be hard to keep track of everything you say today, we're going to audio-record this session.

• And I’ll also be taking notes, so you might notice me writing as we are talking.

• I want you to know that your responses are voluntary and confidential. You do not have to answer any question you don’t want to and can stop the interview at any time if you wish.

In just a minute I'm going to hand you the questionnaire and I'd like you to fill it out the same way you would if it came to you at home in the mail. Unlike being at home, though, I’ll ask you to stop periodically so that we can talk about what you were thinking as you answered the questions.

I would also like you to think “out loud” as you answer. That is, tell me how you go about answering a question, tell me if something doesn’t sound right, is hard to answer, etcetera.

Now because some people aren't used to reading, thinking and expressing their feelings aloud, I'd like to begin today with a short practice question. I’d like you to practice with this by reading the question I’m about to give you out loud and telling me your thoughts and feelings from the moment I hand you the question until you are finished explaining your answer.

QUESTION: How many times did you eat a meal out at a restaurant in the past week?

Thank you…

Okay. Now here is the survey, please read it and fill it out as if you were actually responding to the survey. As you go through, if any question sticks out to you in any way (maybe you think a question is hard to answer, hard to understand, in a strange place, or you just don’t like it)—anything at all that you would like to say about an item, please tell me so that we can discuss it. Remember that I’ll be stopping you here and there as you complete the survey so that we can discuss it.

(GIVE QUESTIONNAIRE TO RESPONDENT)

Specific Probes for Main Questionnaire

INTERVIEWER - If you feel there was uncertainty about a respondent’s answer, probe retrospectively on those items. Pay attention to respondent behavior here. Do they hesitate? Ask questions? Make remarks that indicate the question is awkward? If yes, PROBE.

Pay attention to SKIP errors. Do they have trouble figuring out which question to go to next? Do they go to the wrong question? If you notice they go to the wrong question, give them a chance to figure it out on their own, but if they don’t, point out to them that they skipped to the wrong question, PROBE about what was confusing about it. NOTE these errors.

Pay attention to how they answer the multiple response items. Do they notice when questions change from “mark one” to “mark all that apply”?

Pay attention to how they answer the grids. Do they fill in the correct cells?

After the respondent completes each section:

General Probes
Problem items	What is this question asking, in your own words?
	What were you thinking about when you answered this question?
Skip and marking issues	I noticed you skipped this/checked this. Did you notice the instructions here?
	What were you thinking about when you answered this question?

SECTION 1	Qs 1 to 3 – STOP RESPONDENT AFTER 3
I’d like you to stop here.
ALL	Did you read the “Survey Overview”? What message did you take away from the overview? Were there any questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE. Was it easy or difficult to use the table of field-of-study codes? Could they find the code they needed?

You can continue from question 4 and remember to tell me what you are thinking.

SECTION 2	CERTIFICATION & LICENSURE Qs 4 to 14 – STOP RESPONDENT AFTER 14
Please stop here for a minute so I can ask you a few questions.
Q4	Let’s look at question 4. Were there any parts of this question that you did not understand? What does the term “business license” mean to you in this question?
Q5	Now let’s look at question 5. What does the term “work-related” mean to you in this question? How did you determine whether any of your certifications or licenses was work-related? If all for personal interest: Can you tell me the name of your certification or license?
Q6	Now let’s look at question 6. Can you tell me more about how you arrived at your answer? In particular, note if respondents seem to count a renewal as a second/third/etc certification or license.
Q7	Let’s look at question 7. Can you tell me what this question means in your own words? If ‘no’, probe on what, if anything, they need to do to maintain or renew their certification or license to determine whether the correct response was ‘yes’, but they refer to this using different terms.
Q8	Let’s look at question 8. What did you think when you saw the grid? Did it look like something that would be easy or difficult to complete? Was it easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete?
Q8b	If response category does not seem to line up with name provided in 8a: Let’s look at question 8b. Can you tell me more about how you came up with your answer?
Q8c	Let’s look at question 8c. Were you able to find a suitable response option, or do you think there were any options missing?
Q9	Let’s look at question 9. If more than one certification or license entered in Q8: Can you tell me more about how you came up with your answer? What does the term “most recently” mean to you in this question?
Q10	Let’s look at question 10. Were there any parts of this question that you did not understand or did not know how to answer? Can you tell me what “keeping you marketable” means in your own words? Were there any benefits or outcomes related to your certification or license that you felt were missing? Were you able to find a suitable response for each of the questions? Were any of the response options confusing? If selects “too soon to tell”, probe to find out when earned certification or license. If selects “not applicable”, probe to find out why this is not applicable.
Q11	Let’s look at question 11. Were there any parts of this question that you did not understand or did not know how to answer? Were you able to find a suitable response option, or do you think there were any options missing?
Q12	Let’s look at question 12. Were there any parts of this question that you did not understand or did not know how to answer? Were you able to find a suitable response option, or do you think there were any options missing? What does “since 2010” mean to you in this question? Note if the respondent seems to have trouble with the recall period.
Q14	Let’s look at question 14. Can you tell me more about how you arrived at your answer? If you paid the renewal costs and your employer reimbursed you, would you include that amount here?
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from question 15 and remember to tell me what you are thinking.

SECTION 3	EDUCATIONAL CERTIFICATES Qs 15 to 20 – STOP RESPONDENT AFTER 20
I’d like you to stop here.
Q15	Let’s look at question 15. Were there any parts of this question that you did not understand or found confusing?
Q16	Let’s look at question 16. What does the term “earned most recently” mean to you in this question? What did you think when you saw the grid? Did it look like something that would be easy or difficult to complete? Was this grid easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete?
Q16a	Let’s look at question 16a. Was it easy or difficult to go back to Table 1? Were you able to find a suitable response option, or do you think there were any options missing?
Q16d	Now let’s look at question 16d. Can you tell me what this question is asking in your own words?
Q16e	Now let’s look at question 18. Can you tell me what this question is asking in your own words? How did you come up with your answer? If not clear, probe on whether or not included time spent studying (as opposed to just including time spent in class).
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from question 17 and remember to tell me what you are thinking.

SECTION 4	APPRENTICESHIPS Qs 17 to 22 – STOP RESPONDENT AFTER 22
I’d like you to stop here.
Q17	Let’s look at question 17. Were there any parts of this question that you did not understand or found confusing?
Q20	Let’s look at question 20. Were there any parts of this question that you did not understand or did not know how to answer? Were there any other important benefits or outcomes of completing your apprenticeship that are not mentioned here? If not already asked: Can you tell me what “keeping you marketable” means in your own words? If not already asked: Were you able to find a suitable response? Were any of the response options confusing? If selects “too soon to tell”, probe to find out when completed apprenticeship. If selects “not applicable”, probe to find out why this is not applicable.
Q21	Were you able to find a suitable response option, or do you think there were any options missing?
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from question 23 and remember to tell me what you are thinking.

SECTION 5	COLLEGE AND OTHER COURSES Qs 23 to 33 – STOP RESPONDENT AFTER 33
I’d like you to stop here.
Q23	Let’s look at question 23. Were there any parts of this question that you did not understand or did not know how to answer? Can you tell me in your own words what the third part of this question is asking about [classes to improve basic reading…]? Is it clear or confusing about what kinds of classes should be included here? For respondents who indicate they have taken one or more of these courses: How difficult was it to remember whether or not you took these classes in the past 12 months? If you started a class more than 12 months ago but finished it less than 12 months ago, how do you think you would have responded to this question?
Q24	Now let’s look at question 24. In your own words, what is this question asking about? Is it clear or confusing about what kinds of classes should be included here?
Q26	Let’s look at question 26. How did you decide whether or not you received college credit for the classes you took?
Q31	If took more than one class in past 12 months: Let’s look at question 31. What does the term “most recent” mean to you in this question? Was it easy or difficult to go back to Table 1? Were you able to find a suitable response, or did you feel that something was missing?
Q32	Let’s look at question 32. Were you able to find a suitable response? Were any of the response options confusing? Are there other important motivations for taking your non-credit course that are not listed here?
Q33	Let’s look at question 33. Were there any parts of this question that you did not understand or did not know how to answer? Are there other important benefits or outcomes of your non-credit course that are not listed here? If not already asked: Can you tell me what “keeping you marketable” means in your own words? If not already asked: Were you able to find a suitable response? Were any of the response options confusing? If selects “too soon to tell”, probe to find out when completed class. If selects “not applicable”, probe to find out why this is not applicable.
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from question 34 and remember to tell me what you are thinking.

SECTION 6	OTHER TRAINING FOR WORK Qs 34 to 38 – STOP RESPONDENT AFTER 46
I’d like you to stop here.
Q34	Let’s look at question 34. Were there any parts of this question that you found confusing or did not understand? Can you tell mean in your own words what the term “formal work-related training” means?
Q35 instructions	Let’s look at the instructions for question 35. Were there any parts of this that you found confusing or did not understand?
Q35	Let’s look at question 35. What did you think when you saw the grid? Did it look like something that would be easy or difficult to complete? Was it easy or difficult for you to complete? If difficult, what made it difficult? Is there a better way that would make this easier for you to complete?
Q35a	Let’s look at the response options for question 35a. Were you able to find a suitable response? Were any of the response options confusing?
Q35b	Let’s look at question 35b. Was this easy or difficult to answer? Can you tell me more about how you came up with your response?
Q35c	Let’s look at question 35d. How difficult did you find this question to answer? (Pay attention especially to how difficult this is for self-employed respondents.)
Q35d	Let’s look at question 35d. How difficult did you find this question to answer? (Pay attention especially to how difficult this is for unemployed respondents.)
Q36	Let’s look at question 36. Can you tell me more about how you arrived at your answer? If you paid for the training and your employer reimbursed you, would you include that amount here?
Q37	Let’s look at question 37. Were there any parts of this question that you did not understand or did not know how to answer? Are there other important benefits or outcomes of your training that are not mentioned here? If not already asked: Can you tell me what “keeping you marketable” means in your own words? If not already asked: Were you able to find a suitable response? Were any of the response options confusing? If selects “too soon to tell”, probe to find out when completed training. If selects “not applicable”, probe to find out why this is not applicable.
Q38	Let’s look at question 38. Were there any parts of this question that you did not understand or did not know how to answer? Are there other important motivations for your training that are not mentioned here?
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

You can continue from question 39 and remember to tell me what you are thinking.

SECTION 7	EMPLOYMENT Qs 39 to 63 – STOP RESPONDENT AFTER 63
If time is running short, have respondent skip Q39-56; go to next row. Otherwise, once respondent has completed question 56: I’d like to stop you here. Were there any questions so far in this section that were confusing or a bit difficult for you to answer? Which ones?
You can continue from question 57 and remember to tell me what you are thinking.
Once respondent has completed question 63: I’d like you to stop here.
Q57	Let’s look at question 57. Were you able to find a suitable response? Were any of the response options confusing?
Q59	Let’s look at question 59. How difficult was this to answer? Can you tell me about how you came up with your answer?
Q61	Let’s look at question 61. Can you tell me about how you came up with your answer? Probe to determine who respondent is thinking of when referring to “employer”.
Q62	Let’s look at question 62. How difficult was this question to answer? Does the concept of a “training path” make sense to you?
Q63	Let’s look at question 63. Were there any parts of this question that you did not understand or did not know how to answer?
ALL	Were there any other questions in this section that were confusing or a bit difficult for you to answer? Which ones? PROBE.

(If time is running short, interview can end here.) You can continue from question 64 and remember to tell me what you are thinking.

SECTION 8	BACKGROUND Qs 64 to 75 – STOP RESPONDENT AFTER 82
Go ahead and finish the rest of the survey now and let me know if any questions are confusing or difficult to answer in any way or if you are unsure of anything.
FINAL OBSERVATIONS
	Do you have anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention? REMEMBER TO ASK NCES STAFF FOR INPUT, GIVE GIFT CARD AT END OF INTERVIEW
	Thank you…

Additional Notes:

8