Certificates Focus Groups and National Adult Training and Education Survey (NATES) Cog Labs

Email to Local Businesses, College and University Alumni Associations, and Personal Contacts to Recruit for Focus Groups and Cognitive Interviews

Dear [contact]:

The American Institutes for Research and Avar Consulting are assisting the National Center for Education Statistics and other federal statistical agencies in learning more about job education, skills, training, preparation, and other similar qualifications people may have. We are specifically looking for [JOB TITLE] to participate in a one-time, voluntary, research interview.

Participants will take part in a [90-minute focus group] [60-minute cognitive interview] on [DATE] and will receive [$60/$40] for their participation.

I would appreciate if you could circulate this invitation to [employees or contractors at your business] [former students] and ask them to email or call me if they are interested in participating. My contact information is below. Please do not hesitate to contact me if you have any questions.

Thank you for your time and help with this important study.

Sincerely,

Recruitment Contact Info

Recruitment Screener for Focus Groups and Cognitive Interviews

Thank you for contacting us. I just have a few question to ask you to verify that you qualify for the research study.

 

1. What is your current occupation? [verify it matches our occupational field criteria]

 

2. What kind of qualifications, like a certificate, certification, or license, you have in order to work in your current occupation? [verify they have a qualification in the occupational field of interest]

 

IF ELIGIBLE: We would like to invite you to participate in the research study. Your participation is voluntary. You will be paid $40/$60 for your time. We are [interviewing people]/[conducting the focus group] on X date at X time in downtown Washington, DC. Are you able to attend the interview/focus group at that time?

 

Thank you and we look forward to seeing you on X date at X time. Directions and other instructions are attached. Please do not hesitate to contact me if you have any questions. [If on phone, ask for email or mail information to send directions].

 

IF NOT ELIGIBLE or places are full: Unfortunately, we are unable to invite you to participate at this time. Thank you again for contacting us.

Focus Groups Protocol

The moderator will introduce herself or himself, go over rules and logistics, reference the observation room, and explain the purpose of the group.

Purpose: Thank you for taking the time to join our discussion today. The American Institutes for Research has been asked by the National Center for Education Statistics and other federal statistical agencies to help them understand more about job education, skills, training, preparation, and other similar qualifications people may have. You were invited here because you are local residents who may have these types of qualifications. I and the federal statistical agencies that are involved are pleased we are able to get your input on this important topic.

Opener – the opening “question” is intended to be friendly, conversational, somewhat relevant to the topic, and easy so respondents feel relaxed and part of a positive discussion.

Let’s get started with some introductions. Tell us your name, what you do for a living, and if you are working in the field you thought you would be when you were in high school.

Overview questions – these questions are to get the respondent talking about their qualifications and how they are perceived in the industry. We start with a broad question and move onto more specific questions.

MODERATOR: We are interested in the language that participants use to talk about their qualifications (e.g. “certification” “certificate” “license” etc.) Listen carefully for the language and terms participants use. Once the language has been established, use that language throughout the session.

For the next hour or so we are going to talk about your work, and education, training, skills, and qualifications that you or your colleagues might have.

To start, I’d like to hear what you think is the most important qualification people need to get a job like yours.

MODERATOR: Probe on what a person’s occupation is if it isn’t clear from the intro and the discussion

What kinds of training or preparation did you have to undertake to qualify for your job?

MODERATOR: Only if necessary, probe with “training, qualifications, preparation, certifications, certificates, licenses”.

Are these things required for your job?

Where or how did you get your [USE PARTICIPANT LANGUAGE]?

Is where or how you got it important in your field? [Probe on sponsorship – that is, is the sponsor different from the provider?]

How long did it take you to get these qualifications? [Probe about credit hours, part-time, full-time, and sequencing – all at once or periodically]

Are there other training, qualifications, preparation, etc. in your field that are good to have, but not required? Probe: Why are they good to have?

Do most of your colleagues have these qualifications?

When you applied or were recruited for your job, did your employer ask you if you had certain qualifications? Do you think having these improved your chances of getting the job? Does it increase your chances of promotion and marketability in general?

What other things are important in your field to help you advance?

Probe about the order they got them in.

Which one did you get first, next, last? Was one necessary or helpful for the other (e.g., certificate provided knowledge or skills needed to pass licensing exam)? Did one make another obsolete or irrelevant?

If you knew someone who wanted to do the job you do, what qualifications would you tell them to get?

Language: the purpose of this set of questions is to get participants to present key information about the qualifications and to use the different terms and language they might use formally and informally.

If you had to describe these types of industry qualifications or (USE PARTICIPANT LANGUAGE) to someone like me who doesn’t know a lot about them or someone just starting out in your field, how would you describe them?

What about if you are talking about these qualifications to people who have been in your profession a while? Do you use different terms for it? Like an abbreviation or acronym or something like that?

Do you have some qualifications that are important to your employer, but that are not relevant to other employers? When talking to others, how would you distinguish these qualifications from those that are recognized by employers in general?

Probe on commonality of use of terms that come up.

Resume activity: this is a concrete example intended to focus respondents on short, succinct descriptions of their qualifications which could provide insight into what is important to highlight in the short, succinct survey questions we are interested in designing.

MODERATOR -- list on flip chart:

If you were reviewing or preparing a resume or job application form, under what section heading would you list or expect to see the kinds of qualifications we have been talking about?

What, specifically, would you look for or put in that section? Probe: Use the specific language you would use or expect to see.

Let’s look at some of these, how would someone demonstrate on a resume or job application form that they had these qualifications and skills?

[Probe for grades, course hours, exam pass/fail]

Benefits focus: this section asks some specific questions about properties of the qualification/certification.

Here is a list of some benefits that you may or may not have received from getting your qualification. What is missing from this list? Are there things on this list that don’t apply to you? Why don’t they apply? [Ask how important these benefits were in motivating them to get the qualification? Would they have got the qualification otherwise?]

Check to see if there are any follow-up questions needed from observation room and END.

Cognitive Interview Protocol

Introduction and Warm-Up

• Who I am

• What I do

• Length

• Purpose of interview

• Review confidentiality statement

• Be candid; no right or wrong answers

• Gratuity for your time and opinions

• Colleagues listening in on the interview (if any)

• As you are answering questions, “think out loud” when responding. That is, if a question is unclear, tell me and tell me why it is unclear. Tell me what is going through your mind as you answer a question

INTERVIEWER INSTRUCTIONS:

Interviews will consist of concurrent think-aloud and follow-up probes and retrospective probing. The purpose is to let the respondent flow naturally through the materials unless they encounter a problem or difficulty. Retrospective probing will be used to follow-up on some specific possible problem areas.

Think-aloud

Concurrent think-aloud for Screener, Letters, and Questionnaire

[Order is 1-Letters, 2-Screener, 3-Questionnaire]

Allow the respondent to proceed through each document until they make a comment or ask a question. Use non-directive probing to inquire about problems.

Tell me in your own words what this letter is saying?

What do you think this question is asking?

What does that mean to you?

Tell me more about that/the people in your household.

Specific probes for main questionnaire

[Interviewer instruction] If you feel there was uncertainty about a respondent’s answer, probe retrospectively on those items. Pay attention to respondent behavior here. Do they hesitate? Ask questions? Make remarks that indicate the question is awkward? If yes, probe.

Let’s go back and review some of the answers you just gave me.

Did you read all of the text in [question number]? Tell me in your own words what this question is asking about. Were there any words or phrases you didn’t understand?

What about [question number]? In your own words, what is this asking about? Did you read all of the text in this question? Were there any words or phrases you didn’t understand?

Let’s look at this question [question number]. Tell me what you think is meant by “xxxx”.

How did you come up with your most recent [certificate/certification/etc. ]?

[Interviewer instruction] Pay attention to whether or not there are problems with people confusing certificates and certifications and probe.

Did you find it easy or difficult to answer this question about why you got your [certificate/certification/etc.] ? Do these reasons for getting it apply to you? Are there any answer choices that are missing?

What about this question about who awarded it? Did you find it easy or difficult to answer this question? Are there any answer choices that are missing?

How did you come up with your answer for how long it took you to get the certificate?

Were there any other questions in this section that were difficult for you to answer?

How about the question asking about continuing education units and other professional development credits? Did that make sense to you? Were you sure of how to answer that question?

How about the question asking about your most recent activity in order to meet professional development requirements? Did the list of activities make sense? Are any activities missing?

What does the phrase “professional or trade association” mean to you?

The first question about certificates mentions these kinds of schools: ”trade school, community or technical college, or other college or university”. Can you give me some examples of these? Are you familiar with other types of schools that could be included in this list?

Question X asks about who provided the instruction for your continuing education activity. Tell me more about you answer to that

How did you remember what training or instruction you’ve had in the past 12 months? Walk me through how you answered about that. Do you think you may have forgotten some things when you responded to that question? Are there other types of training we should list in that question?

Draft NATES Letter – for think aloud

I am pleased to inform you that the U.S. Census Bureau has selected your household to participate in the 2012 National Adult Training and Education Survey, which we are conducting on behalf of the U.S. Department of Education. This survey collects information about training and education of adults in the United States. The study provides vital information that is used to improve education for everyone – information that is not available anywhere else. The results will help policymakers, researchers, and educators understand how education and training help people get the skills they need for work and life. This survey is about all of us!

Your household was chosen at random as part of this scientific study, and we cannot replace you with someone else. Although your participation is voluntary, your answers will make sure that households like yours are counted. Every type of household in America needs to be included in the survey.

When completing the enclosed surveys, please answer about all adults in the household.

This national education study will provide much-needed information about the kinds of education and training programs that adults participate in to develop new skills for work and life. It also asks about certificates, certifications, and licenses that people get to show employers that they have skills for work. Your response to this survey is important as it will help provide a clear picture of how education and training contribute to jobs and the economy. Please be assured that your participation is voluntary and all information you give may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Section 9573, 20 U.S. Code). Your responses will be combined with those of others to produce national statistics about adult’s educational experiences.

The National Center for Education Statistics, within the Department of Education, is authorized to conduct this survey (Section 9543, 20 U.S. Code). The U.S. Census Bureau is administering the survey on behalf of the Department of Education. More details about the study are provided on the back of the enclosed survey and online at http://surveys.nces.ed.gov/nates/.

We have enclosed $ as a token of appreciation for your participation in this study. Thank you in advance for your continued participation. The survey should only take about 20 minutes to complete. Please return the completed survey in the enclosed envelope. If you have any questions about the study, or need replacement materials please call us toll-free at 1–888–840–8353.

Sincerely,

Robert M. Groves

Director

12