2019 National Household Education Surveys Program (NHES) Adult Training and Education Survey (ATES) Cognitive Interviews

Attachment 2

NHES 2019 ATES Cognitive Interviews Recruitment Screener

Hello. My name is [XXXXXX] and I work for American Institutes for Research. The U.S. Department of Education is conducting a brief but important study about adults’ education and work credentials. American Institutes for Research (AIR) is assisting with the research. Your participation is voluntary, and you will receive no sales pitch or follow-up calls based on your participation.

We are looking for adults to help us understand how they interpret survey questions about adults’ education and work credentials. The objective of the current study is to make sure the survey materials are clear and easy to understand, and we need your help to do that. The information gained from the study will be used to revise the survey. I’d like to ask you a few questions to see if you qualify for the study (takes about 5 minutes).

The next few questions are for classification purposes only.

1. What is your name? (write in)

________________________________________________

2. What is your phone number? (write in)

_________________________________________________

3. What is your email address? (write in)

_________________________________________________

[Interviewer circle/highlight responses]

4. Have you participated in a research study or focus group in the past 6 months? (Recruit none.)

1. Yes (Thank and terminate.)

2. No (Continue)

5. Do you work for the American Institutes for Research, or the U. S. Department of Education?

   1. Yes (Thank and Terminate)

   2. No (Continue)

   3. Don’t know / Refused (Thank and Terminate)

6. Which of the following describes your age? (READ LIST)

1. under 18 (Thank and terminate)

2. 66 or older (Thank and terminate)

3. 18 to 24 (Continue; recruit a mix of ages)

4. 25 to 34 (Continue; recruit a mix of ages)

5. 35 to 44 (Continue; recruit a mix of ages)

6. 45 to 65 (Continue; recruit a mix of ages )

7. (Do not read) Refused (Thank and terminate)

(Note: Recruit a mix of ages with a higher proportion of younger population 18-34)

7. Which of the following best describes your employment status? (READ LIST)

1. Working full time or part time (Continue)

2. Retired or otherwise not working (Thank and terminate)

3. (Do not read) Don’t know / refused (Thank and terminate)

8. Do you have a currently active professional certification or a state or industry license?

A professional certification or license shows you are qualified to perform a specific job and includes things like Licensed Realtor, Certified Medical Assistant, Certified Teachers, or an IT certification. These do not include business licenses, such as a liquor license or vending license.

1. Yes (Go to Q9)

2. No (Thank and terminate)

1. Do you have more than one currently active professional certification or state or industry license?

   1. Yes (Go to Q10)

   2. No (Got to Q12)

10. Are your certifications or licenses valid anywhere in the U.S. or only in a particular state or states? (Mark all that apply – the R may have both a national level CN and state level CN)

    1. Anywhere in the U.S. (i.e., these are likely certifications, not licenses; Go to Q12)

    2. In particular state(s) only (Continue)

    3. Both (Continue)

11. Do you have the same certification(s) or license(s) in more than one state?

    1. Yes (Go to Q12 – Recruit 25 who responded “In particular state(s) only” to Q10; recruit 15 who responded “Anywhere in the U.S.” or “Both” to Q10)

    2. No (Continue)

12. What is YOUR HIGHEST EDUCATION level?

1. Less than High School (Continue; recruit a mix of education levels)

2. High School Graduate (Continue; recruit a mix of education levels

3. Some College, trade, technical or vocational school (Continue; recruit a mix of education levels)

4. College degree (Continue; recruit a mix of education levels)

5. Graduate education (Continue; recruit a mix of education levels)

(Note to recruiter: Recruit a few participants that do not have a college degree)

13. Are you currently working towards a certification or a state or industry license?

    1. Yes (Select one)

       1. Currently working towards your first certification or license (Recruit none. Thank and terminate – note there should be no cases for this response since they reported they had a certification or license.)

       2. Currently working towards getting a new one in addition to the certifications or licenses you already have (Continue – Recruit 10)

    2. No (Continue)

14. Are you a licensed day care provider?

    1. Yes (Continue – Recruit 2-3 who fall under any of the three groups being recruited)

    2. No (Continue)

15. RECRUITER: INDICATE GENDER (Recruit a mix of males and females.)

    1. Male

    2. Female

Based on your answers, we would like to invite you to participate in an interview (to discuss your education and work credentials). You will receive $40 to thank you for completing the interview. The interview is about 1 hour in length and will be conducted at XXXXX. If you drive, please park in our garage and bring your ticket with you for validation. We will pay for your parking.

We are conducting interviews on __________ and ___________. Are you available to participate in this research study?

	CIRCLE
Yes	1	 (CONTINUE AND GIVE DIRECTIONS)
No	2	 (THANK AND TERMINATE)

Attachment 3

NHES 2019 ATES Cognitive Interview Protocol

Interview ID:		Date of Interview:
First Name:		Interviewer

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us with a survey that we are working on for the U.S. Department of Education. This survey asks about adults’ education and training experiences. I will ask you to complete the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will ask you about your answers. This will help us make the survey better and clearer for other adults like you.

To make sure we learn everything we can from this session, I am going to be taking notes while you are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today behind the glass/by video].

Also, this interview will be audio recorded so researchers can review the recording later. Any information that refers to you personally, like your name or employer, will not be included in our report, and your responses will be used for research purposes only. Is all of this OK?

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

[Provide respondent with a copy of the informed consent form. Ask the respondent to read the form (or read it to him/her, depending on respondent preference), answer any questions, and have the respondent sign the form. Leave a separate copy of the form with the respondent.

If the respondent has consented to recording, start the recorder.]

STEP 2: THINK-ALOUD PRACTICE

While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you decide on your answers will help me understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give. To help you get started on thinking aloud, we are going to do a practice activity. [Show an example by asking yourself the question and answering it using ‘think-aloud’]

I’m going to give you a piece of paper with a question on it. I’m going to ask you to read out loud the question and think aloud as you decide on your answer.

[Give respondent handout with think aloud practice.]

How many glasses of water did you drink yesterday?

Please tell me what you are thinking while you are figuring out your answer.

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

[Hand respondent a copy of the questionnaire and a pen or pencil.]

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would read to yourself if I were not here.

Remember that I am interested in hearing you talk about what you are thinking as you figure out your answers to the questions.

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

Useful Probes—Use only as needed when response is unclear or R appears confused or unsure

Can you tell me in your own words what that question was asking?

What does the word [term] in this question mean to you?

How easy or difficult is it to remember [topic]?

How easy or hard was it to choose an answer?

Tell me what you are thinking.

How did you come up with that answer?

You said [answer]. Can you tell me more about that?

[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Tell me what you are thinking.

[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: Tell me what just happened. Tell me about what you were just doing.

Education
STOP RESPONDENT AFTER Q2 IF NEEDED – Probe on Q1-2	NOTES
ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY
Instructions: It is not necessary to probe on the instructions. Note if the respondent has any problems if they read them.
Q1-Q2: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Certifications and Licenses
STOP RESPONDENT AFTER Q11 – Probe on Q3-11	NOTES
Q3-6: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q7:  Is your most important certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work? Who issued your certification or license? Did you interpret this item to mean that you should answer yes ONLY IF the certification or license is required by a state board? Why do you think you interpreted it that way?
Q8: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q9: In the process of getting your most important certification or license, did you do any of the following? How easy or difficult was it to answer this question? Are any of these options confusing? Are there any other activities you participated in that are not listed here? IF R FIRST SAYS THEY TOOK CLASSES (“Yes” to 14a-c): How easy or difficult was it to remember the classes, trainings, or other activities you participated in?
Q10: Did you prepare for your most important certification or license through a FREE state or city program? How easy or difficult was it to answer this question? If YES, tell us more about the program. Who offered it and who paid the program costs?
Q11: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q3 through 11 If the R did not explain from the think aloud and probes above probe further: Tell me a little bit more about your most important certification or license—in your own words, what was the process you went through in order to get it?
STOP RESPONDENT AFTER Q17 – Probe on Q12-17	NOTES
Q12: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q13: We have some questions about your SECOND-MOST-IMPORTANT certification or license. Is your second-most-important certification or license the same as your most important one, but in a different state? In your own words, what does this question mean to you? How easy or difficult was it to answer this question? IF YES: Tell me more about this second CN or license and how it relates to your most important CN or license
Q14-15: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q16:  Is your second-most-important certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work? Who issued your certification or license? (If R seems confused) How easy or hard was it to answer this question?
Q17: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q12 through 17 If the R did not explain from the think aloud and probes above probe further: Tell me a little bit more about your second most important certification or license—in your own words, what was the process you went through in order to get it?
STOP RESPONDENT AFTER Q23 – Probe on Q18-23	NOTES
Q18: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q19: We have some questions about your THIRD-MOST-IMPORTANT certification or license. Is your third-most important certification or license the same as your most important or second most important one, but in a different state? How easy or difficult was it to answer this question? IF YES: Tell me more about this third CN or license and how it relates to your most important or second-most important CN or license.
Q20-23: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Preparation for New Certifications and Licenses
STOP RESPONDENT AFTER Q29 – Probe on Q24-29	NOTES
Q24: Are you currently working on RENEWING a professional certification or license? What does it mean to you to be currently working on renewing a certification or license?
Q25: Are you currently working on getting a NEW professional certification or license? In your own words, how does this question differ from the previous question? What, to you, is a “new” certification or license? (If R happens to mention currently being enrolled in school during talk-aloud portion of the Education section, try to probe on whether they are in a program that leads to certification or licensure (e.g., teaching, nursing, cosmetology, computer networking, electrician) and if so, whether they consider their current studies to be a part of preparing for certification or licensing in their field.)
Q26-27: It is not necessary to probe on these questions. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q28:  Is this certification or license required by a federal, state, or local government agency (such as a state board) in order to do that kind of work? Who issued your certification or license? (If R seems confused) How easy or hard was it to answer this question?
Q29: In preparing for your MOST IMPORTANT new certification or license, are you currently participating in or have participated in any of the following activities? Tell me a bit more about the process that you have to go through to get this certification or license. Is there anything else important that you are doing or have done to get this certification or license that is not listed here? IF RESPONDENT CHECKS THE BOX FOR CERTIFICATE OR DEGREE: Can you tell me a bit more about the certificate or degree that you are required to complete to get this certification or license?
Employment
STOP RESPONDENT AFTER Q37 – Probe on Q30-37	NOTES
Q30-35: It is not necessary to probe on this section. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
Q36: Do you spend 5 or more hours each week IN YOUR HOME looking after any children under age 13 who are not your own? Can you tell me in your own words what this question was asking? How easy or difficult was it to answer this question? If yes, tell me a bit more about this. How many children to you look after? How many hours each week do you look after them? And these are NOT your children?
Q37: Are you paid for watching those children? How easy or difficult was it to answer this question? Looking at questions 36 and 37 does their placement make sense or do they seem out of place?

FINAL QUESTION

Is there anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?

(IF APPLICABLE) REMEMBER TO ASK OBSERVER FOR INPUT/ADDITIONAL QUESTIONS

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for improving this survey.

Interviewer Instructions: Collect materials and escort respondent out of the interviewing room and to the lobby. Please give the respondent incentive envelope [$40], and thank them for their time and effort.

12