2021 Adult Training and Education Survey (ATES) Cognitive Interviews – First Round

Attachment 2 and 3

2021 Adult Training and Education Survey (ATES)

Cognitive Interviews – First Round

Recruitment Screener and Cognitive Interview Protocol

OMB# 1850-0803 v. 227

March 2018

National Center for Education Statistics (NCES)

Attachment 2

ATES 2021 Cognitive Interviews Recruitment Screener

Hello. My name is [XXXXXX] and I work for American Institutes for Research. The U.S. Department of Education is conducting a brief but important study about adults’ education and work credentials. American Institutes for Research (AIR) is assisting with the research. Your participation is voluntary, and you will receive no sales pitch or follow-up calls based on your participation.

We are looking for adults to help us understand how they interpret survey questions about adults’ education and work credentials. The objective of the current study is to make sure the survey materials are clear and easy to understand, and we need your help to do that. The information gained from the study will be used to revise the survey. I’d like to ask you a few questions to see if you qualify for the study (takes about 5 minutes).

The next few questions are for classification purposes only.

1. What is your name? (write in)

________________________________________________

2. What is your phone number? (write in)

_________________________________________________

3. What is your email address? (write in)

_________________________________________________

[Interviewer circle/highlight responses]

4. Have you participated in a research study or focus group in the past 6 months?

1. Yes (Thank and terminate.)

2. No (Continue)

5. Do you work for the American Institutes for Research, the U. S. Department of Education, or the National Science Foundation?

   1. Yes (Thank and Terminate)

   2. No (Continue)

   3. Don’t know / Refused (Thank and Terminate)

6. Which of the following describes your age? (READ LIST)

1. under 18 (Thank and terminate)

2. 66 or older (Thank and terminate)

3. 18 to 24 (Continue; recruit a mix of ages)

4. 25 to 34 (Continue; recruit a mix of ages)

5. 35 to 44 (Continue; recruit a mix of ages)

6. 45 to 65 (Continue; recruit a mix of ages )

7. (Do not read) Refused (Thank and terminate)

(Note: Recruit a mix of ages with a higher proportion of younger population 18-34)

7. Which of the following best describes your employment status? (READ LIST)

1. Working full time or part time (Continue)

2. Retired or otherwise not working (Thank and terminate)

3. (Do not read) Don’t know / refused (Thank and terminate)

8. Do you work in a job that uses science, medicine, engineering, mathematics, or computer science?

   1. Yes (Recruit 10)

   2. No

9. What is YOUR HIGHEST EDUCATION level?

1. Less than High School (Continue)

2. High School Graduate (Continue)

3. Some College, trade, technical or vocational school (Continue)

4. College degree (Continue)

5. Graduate education (Continue)

(Note: Recruit a mix of education levels)

10. Are you currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school?

    1. Yes (Recruit 8)

    2. No

11. Some people enroll at a community college, trade school, or other school to earn a certificate rather than a degree. It is sometimes called a vocational diploma. Have you ever earned this type of certificate? Do not count a teaching certification or other certification.

    1. Yes (Recruit 10)

    2. No

12. Do you have a currently active professional certification or a state or industry license?

A professional certification or license shows you are qualified to perform a specific job and includes things like Licensed Realtor, Certified Medical Assistant, Certified Teachers, or an IT certification. These do not include business licenses, such as a liquor license or vending license.

1. Yes (Recruit 22)

2. No (Go to Q13)

1. What is the name of your certification or license?

__________________________________________________________________________

14. Have you ever completed an internship, apprenticeship, practicum, clerkship, externship, residency, clinical experience, or similar program?

    1. Yes (Recruit 20)

    2. No

15. Have you recently switched jobs?

    1. Yes

    2. No (Go to Q16)
       
       

16. When did you switch jobs?

_____________________________________ (recruit recent job changers [since 2/1/18])

17. Do not read: RECRUITER: INDICATE GENDER (Recruit a mix of males and females.)

    1. Male

    2. Female

Based on your answers, we would like to invite you to participate in an interview (to discuss your education and work credentials). You will receive $40 to thank you for completing the interview. The interview is about 1 hour in length and will be conducted at XXXXX. If you drive, please park in our garage and bring your ticket with you for validation. We will pay for your parking.

We are conducting interviews on __________ and ___________. Are you available to participate in this research study?

	CIRCLE
Yes	1	 (CONTINUE AND GIVE DIRECTIONS)
No	2	 (THANK AND TERMINATE)

RECRUITER: Check off the criteria that this adult fits in order to categorize the cognitive interviews we will conduct and assure that the targets are met.

Interview Criteria:

Currently enrolled (8)	Certificate (10)	Certification/ License (22)	Work experience (20)

NHES 2021 ATES Cognitive Interview Protocol

Interview ID:		Date of Interview:
First Name:		Interviewer

STEP 1: WELCOME AND INTRODUCTION SCRIPT

Interviewer Instructions: The following scripts should NOT be read verbatim. Text in italics is suggested content with which the interviewer should be thoroughly familiar in advance. Text in brackets [ ] presents instructions for the interviewer.

Hello, my name is <your name > and I work for American Institutes for Research. It’s nice to meet you, thanks for coming to help us out today.

[Create small talk to build rapport with the respondent]

You are here today to help us with a survey that we are working on for the U.S. Department of Education. This survey asks about adults’ education and training experiences. I will ask you to complete the survey as if you were taking a real survey. There are no right or wrong answers, and it’s ok if you do not know some of the answers. As you are answering the questions I will ask you about your answers. This will help us make the survey better and clearer for other adults like you.

To make sure we learn everything we can from this session, I am going to be taking notes while you are answering the survey and while we are talking [INSERT IF THERE IS AN OBSERVER: and my colleague will also be observing our session today behind the glass/by video].

Also, this interview will be audio recorded so researchers can review the recording later. Any information that refers to you personally, like your name or employer, will not be included in our report, and your responses will be used for research purposes only. Is all of this OK?

Remember, you are not being tested; all we want is to learn how well the survey works.

If at any time you want to stop, just let me know.

[If, for any reason, the participant is no longer interested in participating, thank the participant for his/her time and end the interview.]

Before we continue, do you have any questions? [Answer any questions they may ask.]

[Provide respondent with a copy of the informed consent form. Ask the respondent to read the form (or read it to him/her, depending on respondent preference), answer any questions, and have the respondent sign the form. Leave a separate copy of the form with the respondent.

If the respondent has consented to recording, start the recorder.]

STEP 2: THINK-ALOUD PRACTICE

While you fill out the survey I want you to talk about how you are answering the questions. Hearing you talk about how you decide on your answers will help us understand how to make the questions better. I am more interested in how you arrive at your answers than in the answers you give. To help you get started on thinking aloud, we are going to do a practice activity. [Show an example by asking yourself the question and answering it using ‘think-aloud’]

I’m going to give you a piece of paper with a question on it. I’m going to ask you to read out loud the question and think aloud as you decide on your answer.

[Give respondent handout with think aloud practice.]

How many glasses of water did you drink yesterday?

Please tell me what you are thinking while you are figuring out your answer.

Probe as needed for detail: How did you arrive at that answer? What were you visualizing when you were deciding on that answer? Can you tell me more about what you are including in your count?

When complete: Thank you; that’s the kind of detail I’m looking for when you talk to me about your answers to the questions in the survey.

[Hand respondent a copy of the questionnaire and a pen or pencil.]

STEP 3: COMPLETION OF THE QUESTIONNAIRE

Now I would like you to complete the questionnaire. I’d like you to think aloud while you decide on your answer. Please read aloud anything you would read to yourself if I were not here.

Remember that I am interested in hearing you talk about what you are thinking as you figure out your answers to the questions.

[REMEMBER TO TELL THEM THROUGHOUT THE QUESTIONNAIRE THAT THEY ARE DOING A GREAT JOB THINKING ALOUD, AND THEIR ANSWERS TO YOUR QUESTIONS ARE REALLY HELPFUL.]

Useful Probes—Use only as needed when response is unclear or R appears confused or unsure

Can you tell me in your own words what that question was asking?

What does the word [term] in this question mean to you?

How easy or difficult is it to remember [topic]?

How easy or hard was it to choose an answer?

Tell me what you are thinking.

How did you come up with that answer?

You said [answer]. Can you tell me more about that?

[In response to R actions or comments]: Tell me what happened? Tell me what you were doing?

[If you pick up on a visual cue like a furrowed brow or a puzzled look]: Tell me what you are thinking.

[If you pick up on a visual cue that indicates confusion like paper flipping or rereading an old question before answering this question]: Tell me what just happened. Tell me about what you were just doing.

Education
Education Level: STOP RESPONDENT AFTER QA4 IF NEEDED – Probe on QA1-A4	NOTES
ASK QUESTIONS AND USE PROBES ONLY AS NECESSARY
Instructions: It is not necessary to probe on the instructions. Note if the respondent has any problems if they read them.
QA3: What was the field of study for your highest degree? How easy or difficult was it to answer this question?
QA4: Thinking back to when you first got this degree, which ONE of the following BEST describes why you got it? How easy or difficult was it to answer this question? Are any of these options confusing? Are there any other reasons you got this degree that are not listed here?
Certificates: STOP RESPONDENT AFTER QB5 – Probe on QB1-B5	NOTES
QB1: Some people enroll at a community college, trade school, or other school after high school to earn a certificate rather than a degree. Have you ever earned this type of certificate? It is sometimes called a vocational diploma. Do not count a teaching certification or other certification. In your own words, what does this question mean to you? If yes, what is the name of the certificate you earned?
QB2: Who gave you your certificate? If you have more than one, answer for your most recent one. What was the name of the institution? (If it is not clear what type of institution it is, probe on whether it was a school, a business, or something else.) Were any of these options confusing? Are there any other options you would like to see listed?
B4: Which ONE of the following was required for enrolling in your last certificate program? How easy or difficult was it to answer this question?
QB5: What was the field of study for your certificate? [Probe only if this is the first field of study question answered] How easy or difficult was it to answer this question?
QB6: Thinking back to when you first got this certificate, which ONE of the following BEST describes why you got it? How easy or difficult was it to answer this question? Are any of these options confusing? Are there any other reasons you got this certificate that are not listed here?
Enrollment: STOP RESPONDENT AFTER QC7 – Probe on QC1-C7	NOTES
QC1: Are you currently enrolled in or taking courses from a college, university, technical or trade school, or other school after high school? If you are/were taking online classes, would/did you include them in your answer?
QC3: Are you currently enrolled in a degree program? How easy or difficult was it to answer this question? Is the phrase “transfer to a degree program” clear or confusing?
QC5: What is your primary field of study? [Probe only if this is the first field of study question answered] How easy or difficult was it to answer this question?
QC6: Which ONE of the following BEST describes why you are in school? How easy or difficult was it to answer this question? Are any of these options confusing? Are there any other reasons why you are in school that are not listed here?
QC7: Is your current schooling intended to prepare you to earn a professional license or certification? In your own words, what does this question mean to you? How easy or difficult was it to answer this question?
Employment
Employment Status: STOP RESPONDENT AFTER QD7 – Probe on QD1-D7	NOTES
QD1: Were you working for pay or profit during the week of February 1, 2018? Can you describe your work situation during the week of February 1? [If respondent struggles to answer describe work situation during the week of February] Can you tell me about your work situation last week?
QD2: What is the MAIN reason you were not working? How easy or difficult was it to answer this question?

Current Employment: STOP RESPONDENT AFTER QE17 – Probe on QE1-E17	NOTES
QE1: During the week of February 1, 2018, how many jobs did you have? Count the same work with multiple clients as one job. How clear or confusing was the instruction “count the same work with multiple clients as one job”?
QE6: We have some questions about the job at which you worked the MOST HOURS during the week of February 1, 2018. First, which one of the following best describes your employment at the job at which you worked the MOST HOURS during the week of February 1, 2018? How easy or difficult was it to answer this question?
QE10: Please rate your satisfaction with that job's… e. Working conditions E13e. Tell me a bit more about why you gave that satisfaction rating. What specific working conditions were you thinking about when you answered this question?
Licenses and Certifications
Licenses: STOP RESPONDENT AFTER QF12 – Probe on QF1-F12	NOTES
QF1:  On February 1, 2018 did you have a license or certification that was REQUIRED BY A GOVERNMENT AGENCY (such as a state board) to work in that job? For example, state boards of education require that public school teachers be certified. How easy or difficult was it to answer this question? Who issued your certification or license?
QF2: What is the name of that license or certification? Please write out the full name; do not use abbreviations. If you were required to have more than one, report the one you obtained first. [Note if respondent struggles to answer due to having more than one license or certification and probe on how they made a decision] How easy or difficult was it to answer this question?
QF3: Thinking back to when you first got this license or certification, which ONE of the following BEST describes why you got it? How easy or difficult was it to answer this question? How did you come to select your answer? Would you have preferred to mark more than one answer?
QF4: Which type of government agency required that you have this license or certification? Please mark Yes or No for each type. How easy or difficult was it to answer this question?

QF5: We will refer to meeting this government requirement as "being licensed". On February 1, 2018, in how many states were you licensed for that job? In your own words, what does this question mean to you? How did you come to select your answer? How easy or difficult was it to answer this question? IF LICENSED IN MORE THAN 1 STATE, how did you become licensed in the secondary state(s)? (e.g., reciprocity, testing, automatic, full application, etc.)
QF8: In order to be licensed in this occupation, did you first have to get a professional certification? Certifications are usually awarded by professional or trade associations, and include things like certified medical assistant, Microsoft certified professional, and ASE certification. In your own words, what does this question mean to you? How did you come to select your answer? How easy or difficult was it to answer this question? IF YES, please tell me more about this certification and its relationship to the license you have for this job.
QF10: What type of program did you complete? IF R STRUGGLES WITH Q: Is there a different type of program you completed to earn your license?
QF11: To get the knowledge and skills needed when you were FIRST licensed for this job, did you take… [IF NO to F11a, b, & c] How did you get the knowledge and skills needed when you were first licensed in this occupation?
QF12: Did you prepare to get licensed through a FREE state or city program? How easy or difficult was it to answer this question? What does a “FREE state or city program” mean to you? If YES, tell us more about the program. Who offered it and who paid the program costs?
Certifications: STOP RESPONDENT AFTER QG11 – Probe on QG1-G11	NOTES
QG1: For this same job, on February 1, 2018 did you have a professional certification that is NOT required by a government agency? Certifications are usually awarded by professional or trade associations, and include things like certified medical assistant, Microsoft certified professional, and ASE certification. In your own words, what does this question mean to you? How did you come to select your answer? How easy or difficult was it to answer this question?
QG2. How many certifications did you have for this job on February 1, 2018? (If R answered Yes to F8) Did you include the certification(s) obtained to get a license in this occupation as well as other certifications?
QG3: What is the name of your MOST IMPORTANT certification for this job? Tell me more about this certification. Who awarded it?
QG4: If you also reported a license in section F, did you get your most important certification in order to get that license? In your own words, what does this question mean to you?
QG5: Thinking back to when you first got your most important certification, which ONE of the following BEST describes why you got it? How easy or difficult was it to answer this question? Are there any other reasons why you got this certification that are not listed here?
QG7: What type of program did you complete? IF R STRUGGLES WITH Q: Is there a different type of program you completed to earn your certification?
QG8: To get the knowledge and skills needed for your most important certification, did you take… [IF NO to G8a, b, and c] How did you get the knowledge and skills needed to get your most important certification?
G9: Did you prepare for your most important certification through a FREE state or city program? [IF R skipped F12] How easy or difficult was it to answer this question? What does a “FREE state or city program” mean to you? If YES, tell us more about the program. Who offered it and who paid the program costs?
QG10: Does your employer require that you have this certification? How easy or difficult was it to answer this question? Do all employers require this certification?
QG11: Do you get paid more for having this certification? Tell me a bit about pay in your field for people who have this certification versus those who do not have it. Does everyone in your job who has this certification get paid more or does it depend on the situation? Tell me about your situation.
QG1 through G11 If the R did not explain from the think aloud and probes above probe further: Tell me a little bit more about your most important certification—in your own words, what was the process you went through in order to get it?
Other Licenses and certifications: STOP RESPONDENT AFTER QH5 – Probe on QH1-H5	NOTES
QH1: Do you have a license that is REQUIRED by a government agency FOR ANY OTHER JOBS? In your own words, tell me what you think “FOR ANY OTHER JOBS” means. Did you consider jobs outside of the week of February 1, 2018? IF YES, please tell me more about this license (name, who it was issued by, for what field and job).
QH2: FOR ANY OTHER JOBS, do you have a professional certification that is NOT required by a government agency? IF YES, please tell me more about this certification (name, who it was issued by, for what field and job).
QH3: Are you currently working to get a new license or certification? In your own words, what does this question mean to you? IF YES, tell me a bit more about the process that you are going through to get this certification or license. (If YES, try to determine if R is currently working towards a license or certification, or has not yet started the process of working towards a license or certification.)
QH4: Are you getting this license or certification for your main job? IF R STRUGGLES WITH Q: How easy or difficult was it to answer this question?
Work Experience Programs
STOP RESPONDENT AFTER QI17 – Probe on QI1-I17	NOTES
QI2: We will refer to internships, apprenticeships, and similar programs as "work experience programs". Was your LAST work experience program for the main job you had during the week of February 1, 2018? In your own words, what does this question mean to you? How easy or difficult was it to answer this question?
QI4: In what year did you complete your last work experience program? How easy or difficult was it to answer this question?
QI5: It is not necessary to probe on this question. Let the respondent continue the think aloud and only probe if respondent says something that indicates a potential problem, or if there is confusion or an inconsistent interpretation compared to other respondents.
QI6. What wage did you earn during your last work experience program? [If R received a wage] How did you come up with your answer? What does “steady wage” mean to you? Did you get any type of salary adjustment during the program, such as a cost of living adjustment, bonus, or pay raise for good work? [If YES, received salary adjustment] Did you consider that when reporting your answer here or did you not consider it?
QI7: Which ONE of the following BEST describes your last work experience program? How easy or difficult was it to answer this question? Was this question clear or confusing in any way?
QI10: How useful was this training for giving you the skills you needed? How did you come to select your answer? Why did you find this to be helpful/or not? Did your answer refer to the training specifically, or to the entire work experience program?
QI13: At the END of your last work experience program, were you hired by the company or business that sponsored the program? What does “sponsored the program” mean to you? Is the question clear or confusing? If NO, were you offered a job?
QI14: How long did you work for that employer? Would you or would you not have preferred to report a fraction of a year?
QI16. Thinking back to when you first completed the program, would you say it was a good launching pad for your career? How did you come to select your answer? What does “launching pad” mean to you? Does this question capture the role of the program in your career? Are the response options appropriate? (INTERVIEWER: Try to get a sense of what stage in their career R is currently at and whether the work experience program was to enter a field or for advancement in the field.)
QI1-I16: If the R did not explain from the think aloud and probes above then probe further: Tell me a little bit more about your last work experience program. Where did you complete it, how long was it, how did it help your career development, etc.

FINAL QUESTION

Is there anything else you would like to tell me about this questionnaire that you haven’t had a chance to mention?

(IF APPLICABLE) REMEMBER TO ASK OBSERVER FOR INPUT/ADDITIONAL QUESTIONS

STEP 4. THANK YOU FOR PARTICIPATING

We are finished. Thank you for helping us. The information you’ve provided will be very helpful for improving this survey.

Interviewer Instructions: Collect materials and escort respondent out of the interviewing room and to the lobby. Please give the respondent incentive envelope [$40], and thank them for their time and effort.

17