2008-18 Baccalaureate and Beyond Longitudinal Study (B&B:08/18) Cognitive and Usability Testing

National Center for Education Statistics

Attachments

2008-18 Baccalaureate and Beyond Longitudinal Study (B&B:08/18) Cognitive and Usability Testing

OMB# 1850-0803 v.204

August 2017

Attachments:

Attachment I – Recruitment Procedures and Materials

Attachment II – Eligibility Screening Questions

Attachment III – Consent to Participate in Research

Attachment IV – Interview Protocol

Attachment I

Recruitment Procedures and Materials

Identification of Respondents

Shugoll Research is continuously developing its database of over 100,000 people in the Washington, DC area. The database was developed originally through cold-calling and recruiting from mall-intercept locations and has continued to grow both through cold-calling from directory listings as well as through referrals, word-of-mouth, and social media. The database contains demographic information such as age, gender, employment (for example, occupation type), and education (for example, education level) so that the contact information can be sorted based on the needs of each individual study performed by Shugoll.

Respondents will be recruited by Shugoll Research using its database of individuals in the Washington, DC area who have at one time expressed interest or participated in Shugoll’s research activities.

Sample Recruitment Advertisements

Advertisements will be targeted toward individuals qualified for this particular study, describing briefly the purpose of the cognitive and usability testing and the opportunity for respondents to contribute to the development of a survey. The advertisements will identify the need for 2007-2008 college grads, the time commitment of the interview, and the incentive amount for participation. Shugoll Research may also use a variety of other methods to recruit potential participants.

Sample advertisements using a variety of methods are presented below. These methods include networking and referrals; e-blasting from the database and online screening; and a variety of social media outlets that may include Facebook, Twitter, and LinkedIn.

Advertisement 1:

2007-2008 college grads needed.

Answer a few questions about the impact of your education and if you qualify and participate in a follow up interview you will RECEIVE $40 for your time! Click here (link to online eligibility screener shown in Attachment II).

Advertisement 2:

#DC 2007-2008 college grads. We are looking for participants for a study about the impact of your bachelor’s degree on your life experiences after college. Click here (link to online eligibility screener shown in Attachment II).

Advertisement 3:

Shugoll Research is looking for 2007-2008 college grads with a bachelor’s degree for a study about what you have been doing since you graduated.

Email us at XXXXX@XXXXXX.XXX or call 1-8XX-XXX-XXXX

Your responses will be used to refine survey questions.

On average, these sessions take about 90 minutes.

Attachment II

Eligibility Screening Questions

All initial eligibility screening will be conducted through the online recruitment screener. The recruitment screener, a 3-5-minute online survey, will be used to determine whether individuals qualify to participate in the study that cognitively tests a subset of B&B:08/18 survey items. Potential respondents who contact Shugoll via telephone or email in response to the recruitment advertisements will receive an email containing the online recruitment screener survey link. Respondents who click the survey link provided in the targeted social media ads and emails will go directly to the online recruitment screener. The responses to the online recruitment screener survey questions will determine their eligibility for the project.

If the online recruitment screener determines that potential respondents are eligible, respondents will be contacted via telephone for the final portion of screening and interview scheduling.

Email Text (when screener link is emailed to potential respondents who email in or call in to inquire about study)

Shugoll Research is conducting a study among college graduates and would greatly value your opinions. Qualified participants who participate in a 90-minute online video interview will receive $40 as a thank you for their time and opinions. If you are interested, please click on the link below to take a brief survey to determine your eligibility.

(Link to online eligibility screener shown in Attachment II)

Please do not share this email or link as it is associated with your specific database entry.

Thank you,

Shugoll Research

Online Recruitment Screener

Introduction to online recruitment screener:

We are conducting a survey among college graduates and would greatly value your opinions. This information will help us improve an existing national survey of college graduates. First, we would like to ask you a few questions to determine if you are eligible to participate. The screening is very brief and is expected to take approximately 3-5 minutes.

Shugoll Research and RTI International are carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 4 minutes per response, including the time to review instructions, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this form, please write to: The 2008/18 Baccalaureate and Beyond Longitudinal Study (B&B:08/18), National Center for Education Statistics, PCP, 550 12th St., SW, 4th floor, Washington, DC 20202

(Responses to questions about demographic characteristics of the respondent and of their college name and major will be used to help recruit a mixture of respondents. Responses to questions about additional enrollment, employment, teaching, and resumes will be used to identify groups of interest, such as teachers and those with resumes.)

1. What is your name?

2. What is a phone number we can contact you?

3. What is your preferred email address?

4. Were you enrolled for a bachelor’s degree at a postsecondary institution at any time between July 1, 2007 and June 30, 2008?

Yes

No (Ineligible)

5. When did you complete the requirements for your bachelor’s degree?

Month/Year

If between July 1, 2007 and June 30, 2008 (Eligible)

Any other time (Ineligible)

6. What is the name of the college or university where you received your bachelor’s degree?

7. What was the primary major for your bachelor’s degree?

8. Have you enrolled in any college or university for an additional degree or certificate, including enrollment for any graduate or additional undergraduate degree or certificate programs, since earning your bachelor's degree?

Yes

No

9. Are you currently employed?

Yes

No

10. If no to 6:

Have you ever been employed since July 2012?

Yes

No (Ineligible)

11. Since July 2012, have you worked as a teacher at the K-12 level?

Yes

No

12. What is your age? Are you…

Younger than 25

25-29

30-34

35-39

40 or older

13. Are you of either Hispanic or Latino origin?

Yes

No

14. What is your race?

Choose one or more.

White

Black or African American

Asian

American Indian or Alaska Native

Native Hawaiian or Other Pacific Islander

15. What is your sex?

16. Would you be willing to provide us with your current résumé and have a copy available during the interview to answer résumé-specific questions?

Yes

No

17. On a scale of “1” to “10,” where a score of “10” means that the statement describes you extremely well and “1” means that the statement does not describe you at all, how well does the following statement describe you? (SELECT ONE NUMBER BELOW)

	EXTREMELY WELL										NOT AT ALL WELL
I feel comfortable expressing my opinion in a group or interview		10	9	8	7	6	5	4	3	2		1

If ineligible in online recruitment screener based upon questions 1 - 17:

Thank you for taking our survey. Unfortunately based on your answers, you do not meet the criteria for this particular study. In the event that our client changes the criteria of this study, our staff will review your answers to determine your eligibility. If you do qualify, someone from our office will contact you to set up an appointment.

If eligible in online recruitment screener based upon questions 1 – 17 the following will be asked:

We are conducting online video interviews with respondents like you to evaluate some of the questions that will be used in an education survey with special topics.

The interview will take about one and a half hours to complete and respondents will receive $40 after the interview as a thank you for their participation. The interview will be conducted using online video technology and you can participate from your desktop or laptop computer.

A. Would you like to participate in an interview?

Yes	1	(CONTINUE)
No	2	(TERMINATE)

B. Does the desktop or laptop computer at the location where you would participate in the interview have high speed internet?

Yes	1	(CONTINUE)
No	2	(TERMINATE)

C. The computer at the location where you would participate in the interview must be indoors in a quiet location without any noise that would be disruptive to the discussion. For example, we cannot have loud machinery, dogs barking, or children crying in the background. Will you be able to find a location to meet these requirements?

Yes	1	(CONTINUE)
No	2	(TERMINATE)
Don’t know	3

D. Does the computer at the location where you would participate in the interview have a webcam?

Yes	1	(SKIP TO Question F)
No	2	(CONTINUE)

E. If you do not have a webcam on this computer, we will send you one. Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

__________________________________________________ (MAILING ADDRESS)

F. Some of the interviews will involve filling out a questionnaire using a smart phone or tablet. If you qualify to participate, we will send you a document camera to record the screen on your mobile device as you are using it. Do you have a smart phone or tablet that you would be willing to use for this purpose? (OF THE 30 COMPLETED INTERVIEWS, ABOUT 10 RESPONDENTS WILL COMPLETE THE SURVEY USING A SMART PHONE OR TABLET.)

(Check all that apply.)
Yes, have a smart phone	1	(RECRUIT 5 FOR SMART PHONE MOBILE USABILITY AND CONTINUE)
Yes, have a tablet	2	(RECRUIT 5 FOR TABLET MOBILE USABILITY AND CONTINUE)
No, I have neither that can be used	3	(RECRUIT 20 FOR COMPUTER USABILITY AND CONTINUE)

G. (ASK IF Question F CODES 1 OR 2) In order to complete the survey using a smart phone or tablet, we will need to send you a document camera to use to record the screen on your device as you are using it. Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

__________________________________________________ (MAILING ADDRESS)

If ineligible in online recruitment screener based upon items A - G:

Thank you for taking our survey. Unfortunately based on your answers, you do not meet the criteria for this particular study. In the event that our client changes the criteria of this study, our staff will review your answers to determine your eligibility. If you do qualify, someone from our office will contact you to set up an appointment.

If eligible in online recruitment screener based upon items A - G:

Thank you for taking the time to complete this survey. You have pre-qualified for this study. We will call you to either invite you to participate or for any additional necessary screening. Please be aware that if we do not contact you, that simply means that you did not qualify for an open segment/quota for this particular study. But we do sincerely appreciate you taking the time to complete the screening questions. Please click “Proceed” to finalize your answers.

Telephone contact for final recruitment question and interview scheduling:

Introduce yourself and then say: You have pre-qualified for this study. As a reminder, all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (if asked provide: 20 U.S.C. §9573 and 6 U.S.C. §151). We have one more question in order to confirm your eligibility.

Now that you have been out of school for a few years, how has your education impacted your career? (RECORD VERBATIM. PROBE AND CLARIFY.)

NOTE: IT IS EXTREMELY IMPORTANT THAT RESPONDENTS IN THIS STUDY BE ABLE TO PROVIDE AND CLEARLY EXPRESS INFORMATION ABOUT THEIR EXPERIENCE. IF RESPONDENT CANNOT OR WILL NOT EASILY GIVE A ONE TO TWO SENTENCE UNPROMPTED ANSWER IN WELL UNDERSTOOD ENGLISH, PLEASE TERMINATE. (The information is used only to determine whether they meet the articulation requirements in order to be eligible to participate in an interview. We do not want to recruit someone who cannot clearly express their thoughts because he/she will not be a good cognitive interview research participant given the conversational approach used during these interviews.)

(If acceptable answer is provided, schedule interview.) INTERVIEWER: RECORD APPOINTMENT DATE AND TIME.

If for any reason your plans change and you cannot keep your appointment, please call and let me know in advance. My telephone number is (INSERT NUMBER).

Finally, we will be sending you a “consent to participate in research form” via email, which we will need you to sign and scan back to us in order to confirm your participation.

Thank you for your participation. We will send you login instructions for the interview session and one of our technical partners from Interactive Video Productions (IVP) will contact you by email or phone to set up an appointment before the session to ensure your webcam, document camera, and video software are working properly for you.

Consent to participate in research form email

Dear <First Name>

Thank you for your commitment to participate in an online video interview for the National Center for Education Statistics.

• We look forward to your participation in the interview on <Date>.

• The interview will begin at <Beginning Time>. Please login to your online session 15 minutes prior to the start of the session.

• The interview will last approximately 90 minutes.

• According to our records, the email address and telephone number at the location where you will be participating in the interview are <email address> and <telephone number>. If this information is incorrect, please contact us as soon as possible at XXX-XXX-XXXX.

• The mailing address where we will send the webcam or document camera (if needed) is <Address>.

• As a token of our appreciation, a $40 check will be mailed to you after the completion of the interview.

• We will provide you with login instructions for the session and one of our technical staff will contact you to set up an appointment before the session to ensure your webcam and the online video software are working properly for you.

Your participation is completely voluntary. Please reply to this email to confirm receipt of this information. If we are unable to confirm your attendance, we reserve the right to cancel your participation and select another participant.

You must sign and return the attached consent form by scanning then emailing the form to XXX@XXX.XXX prior to your scheduled interview. Without this, you will be unable to participate.

Consent form: (Insert link to consent form)

Thank you,

(Insert name and contact information)

Attachment III

Consent to Participate in Research

Title of Research: Baccalaureate and Beyond Longitudinal Study (B&B:08/18) – Cognitive and usability Interviews

Introduction and Purpose

You, along with others, are being asked to participate in in-depth interviews being carried out by RTI International and Shugoll Research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. The purpose of the interviews is to obtain your feedback on some of the questions that will be used in the Baccalaureate and Beyond Longitudinal Study.

Procedures

You are one of approximately 30 respondents who will take part in the research.

The interviews will be audio and video recorded to make sure we do not miss anything that you say and to help us write a report summarizing the results of the interviews. Upon completion of the written report, the recordings will be destroyed. Your name will never be used in the report that we write.

Study Duration

Your participation in an interview will take about 90 minutes.

Possible Risks or Discomforts

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

Benefits to you

You will receive $40 to thank you for your participation in this study.

Benefits for Other People

We hope that these interviews will help us develop and improve the questionnaire used in this study so that they are clear and can be easily understood and answered.

Confidentiality

Shugoll Research and RTI International are carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Future Contacts

We will not contact you in the future as part of this study.

Your Rights

Your decision to take part in this research study is completely voluntary. You can refuse any part of the study and you can stop participating at any time.

Your Questions

If you have any questions about the study, you may call Cara Shugoll at Shugoll Research (301-656-0310) or Melissa Cominole at RTI International (919-990-8456). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).

PLEASE SIGN/DATE THIS FORM AND SCAN IT BACK TO SHUGOLL RESEARCH AT XXX@XXX.COM

Your signature below indicates that you have read the information provided above, have received answers to your questions, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.

Attachment IV

Interview Protocol

READ OR PARAPHRASE:

Section 1: Introduction

WELCOME respondent and thank them for their time.

INTRODUCE SELF: (Name/Shugoll Employee.)

This study is being carried out by Shugoll Research on behalf of the National Center for Education Statistics, which is part of the U.S. Department of Education. We are currently planning to conduct a survey to learn about the experiences of individuals who completed their bachelor’s degree requirements in the 2007-08 academic year. Today, I am hoping to get your honest feedback on information about issues related to your employment, debt and finances, family formation, civic engagement, and volunteerism experiences.

The interview will be audio and video recorded so that I can concentrate on what you are saying rather than having to take notes, and to make sure that we do not miss anything that you say. The recording will be destroyed once the project is completed. Do I have your permission to record? START RECORDING.

Section 2: Informed Consent (See Attachment III)

There are observers watching and listening via video streaming. These people are not here to observe you in particular but to learn from what you say about your experience of answering the questions.

However, I can assure you that all of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). You will not be identified by name in any report we produce.

Your participation in this interview is very important because it will help us improve an education study with topics related to respondents like you. Once we start, the interview will last about an hour and a half. When we are done, we will mail you a $40 check as a thank you for your participation.

Section 3: Cognitive and usability Interviewing

There are no right or wrong answers. This is not a test. It is your experiences, views, and opinions that I am interested in. Some of the questions I will not address further but other questions I may ask about in more detail. For questions that I want to probe for more detail, an on-screen prompt will appear to alert you and I to pause and take a moment to talk further about this question. So I may ask you what you understood a particular term or phrase to mean, or if you could rephrase a question in your own words. However, please feel free to tell me anything that comes to mind or ask me anything you are unclear about.

ASK THE PARTICIPANT TO BEGIN THE ONLINE SURVEY ON THE DEVICE THAY ARE ASSIGNED TO USE (note: 5 participants will use smart phones, 5 tablets, and 20 a laptop or a desktop computer).

CONDUCT INTERVIEW. ASK GENERAL PROBES AT QUESTIONS WHERE THE RESPONDENT HAS GIVEN SOME INDICATION OF DIFFICULTY WITH THE QUESTIONING BASED ON ANSWERS/BEHAVIORS, AS THE INTERVIEW PROGRESSES.

EXAMPLES OF GENERAL PROBES:

• What did you think the question was asking?

• Can you rephrase the question in your own words?

• How did you go about answering?

• What were you thinking of?

• How did you decide on your answer?

• What was the reason for that answer?

• How easy or difficult was it to answer the question?

• Probe for examples.

• How did you feel about answering the questions?

• Could you explain what you understand by this term?

• Can you tell me more about that?

• You seemed to hesitate/take a moment before giving/entering your response. Can you tell me what you were thinking about?

• What do you mean by…?

• Is there any additional information that we should have provided to help you answer that question?

• Does this term/question sound ok to you, or would you prefer something different?

EXAMPLES OF USABILITY PROBES:

• Was there anything in the survey that made it particularly easy or difficult to complete?

• Were specific questions more difficult to read than others? What could make it easier to answer these questions?

• Are there any specific changes or revisions that you think could be made to the survey to make it easier to complete by computer, phone, and tablet?

Section 4: Debriefing Procedures

• Ask for any last observations/comments/concerns.

• Stop recording.

• Thank the respondent for helping test the survey items.

• Confirm that the respondent will receive the $40 token of appreciation and end interview.

19