2016-17 Baccalaureate and Beyond Longitudinal Study (B&B: 16/17) Cognitive and Usability Testing

Attachment I

Recruitment Procedures and Materials

Identification of Respondents

Shugoll Research has been actively developing its database of over 100,000 people in the Washington, DC area since the mid-1970’s. The database was developed originally through cold-calling and recruiting from mall-intercept locations and has continued to grow both through cold-calling from directory listings as well as through referrals, word-of-mouth, and social media. The database contains demographic information such as age, gender, and employment/education so that the contact information can be sorted based on the needs of each individual study performed by Shugoll.

Respondents will be recruited by Shugoll Research using its consumer database of individuals in the Washington, DC area who have at one time expressed interest or participated in Shugoll’s research activities.

Shugoll Research may also use a variety of other recruiting methods in order to enhance its consumer database. These methods include networking and referrals; e-blasting from the database and online screening; and a variety of social media outlets that may include Facebook, Twitter, and LinkedIn.

Sample Twitter Post:

#DC Individuals who received their bachelor’s degree in 2014-2015. We are looking for participants for a study about the impact of your education on your career. Find out more here (appropriate link).

Sample Facebook Post:

Shugoll Research is looking for 2014-2015 college grads for a post graduate career study.

Sample Recruitment Advertisement

Advertisements will be targeted toward individuals qualified for this particular study, describing briefly the purpose of the cognitive and usability testing and the opportunity for respondents to contribute to the development of a survey. The advertisements will identify the need for 2014-2015 college grads, the time commitment of the interview, and the compensation amount for participation. Sample advertisement is presented below:

2014-2015 college grads needed. Proof required. Call to be screened for qualifications.

Answer a few questions about the impact of your education and if you qualify and participate in a follow up interview you will RECEIVE $40 for your time!

Email us or Call: 1-8XX-XXX-XXXX

Your responses will be used to refine survey questions.

On average, these sessions take about 90 minutes.

Attachment II

Consent to Participate in Research

Title of Research: Baccalaureate and Beyond Longitudinal Study (B&B:16/17) – Cognitive and usability Interviews

Introduction and Purpose

You, along with others, are being asked to participate in in-depth interviews being carried out by RTI International and Shugoll Research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. The purpose of the interviews is to obtain your feedback on some of the questions that will be used in the Baccalaureate and Beyond Longitudinal Study.

Procedures

You are one of approximately 30 respondents who will take part in the research.

The interviews will be audio and video recorded to make sure we don’t miss anything that you say and to help us write a report summarizing the results of the interviews. Upon completion of the written report, the recordings will be destroyed. Your name will never be used in the report that we write.

Study Duration

Your participation in an interview will take about 1½ hours.

Possible Risks or Discomforts

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

Benefits

Your Benefits You will receive $40 to thank you for your participation in this study.

Benefits for Other People We hope that these interviews will help us develop and improve the questionnaire used in this study so that they are clear and can be easily understood and answered.

Confidentiality

Shugoll Research and RTI International are carrying out this research for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. NCES is authorized to conduct this study under the Education Sciences Reform Act (20 U.S.C., § 9543). Your responses may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

Future Contacts

We will not contact you in the future as part of this study.

Your Rights

Your decision to take part in this research study is completely voluntary. You can refuse any part of the study and you can stop participating at any time.

Your Questions

If you have any questions about the study, you may call Merrill Shugoll at Shugoll Research (301-656-0310) or Natasha Janson at RTI International (919-316-3394). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).

YOU WILL BE GIVEN A COPY OF THIS CONSENT FORM TO KEEP.

Your signature below indicates that you have read the information provided above, have received answers to your questions, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.

Attachment III

Eligibility Screening Questions

The recruitment screener, a 3-5 minute phone script, will be used to determine if individuals qualify for study participation in order to test the items identified for cognitive and usability testing for the B&B:16/17 survey. Recruiters will reach out to respondents or potential respondents will call a toll-free number mentioned in an ad and an experienced recruiter will screen the caller for eligibility for the project.

Intro: Thank you for inquiring about this study. I need to ask you a few questions to determine if you are eligible to participate. The screening is expected to take approximately 3-5 minutes.

1. Were you enrolled for a bachelor’s degree at a postsecondary institution at any time between July 1, 2014 and June 30, 2015?

Yes

No (Ineligible)

2. When did you complete the requirements for your bachelor’s degree?

Month/Year

If between July 1, 2014 and June 30, 2015 (Eligible)

Some other time (Ineligible)

3. What is the name of the college or university where you received your bachelor’s degree?

4. Have you ever been employed since completing your bachelor’s degree requirements?

Yes

No (Ineligible)

5. Since receiving your bachelor’s degree, are you currently or have you ever worked:

As a pre-kindergarten teacher Yes No (Recruit 5)

As a kindergarten-12^th grade (K-12) teacher Yes No (Recruit 5)

Or in some other occupation (Specify)__________ Yes No (Recruit 20)

6. (Ask if K-12 teacher) What grade(s) do you teach?

7. What is your age? Are you…

18-24

25-29

30-34

35-39

40 or older

8. Are you of either Hispanic or Latino origin?

Yes

No

9. What is your race?

Choose one or more.

White

Black or African American

Asian

American Indian or Alaska Native

Native Hawaiian or Other Pacific Islander

10. What is your gender?

11. Now that you have been out of school for a few years, how has your education impacted your career? (WRITE VERBATIM. PROBE AND CLARIFY)

12. On a scale of “1” to “10,” where a score of “10” means that the statement describes you extremely well and “1” means that the statement does not describe you at all, how well does the following statement describe you? (CIRCLE ONE NUMBER PER ROW)

	CIRCLE ONE
	EXTREMELY WELL										NOT AT ALL WELL
I feel comfortable expressing my opinion in a group or interview		10	9	8	7	6	5	4	3	2		1

NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT RESPONDENTS IN THIS STUDY BE ABLE TO PROVIDE AND CLEARLY EXPRESS INFORMATION ABOUT THEIR EXPERIENCE. IF RESPONDENT CANNOT OR WILL NOT EASILY GIVE A ONE TO TWO SENTENCE UNPROMPTED ANSWER IN WELL UNDERSTOOD ENGLISH, PLEASE TERMINATE.

If ineligible: Thank you for inquiring about this study. If you are eligible to participate, we will contact you at a later date.

If eligible (Invitation to participate):

We are conducting online video interviews with respondents like you to evaluate some of the questions that will be used in an education survey with special topics. This is not a sales meeting of any kind.

The interview will take about one and a half hours to complete and respondents will receive $40 after the interview as a thank you for their participation. The interview will be conducted using online video technology and you can participate from your computer.

Can I schedule an interview at your convenience?

	CIRCLE ONE
Yes	1	(ASSIGN TIME SLOT AND CONTINUE)
No	2	(THANK AND TERMINATE)

Does the computer at the location where you will be participating in the interview have a webcam?

	CIRCLE ONE
Yes	1	 (GO TO NEXT QUESTION)
No	2	 (CONTINUE)

If you don’t have a webcam on this computer, we will send you one. Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

______________________________________________ (MAILING ADDRESS)

Part of the interview will involve filling out a questionnaire using a smart phone or tablet. Do you have a smart phone or tablet that you would be willing to use for this purpose? (ABOUT HALF OF RESPONDENTS WILL COMPLETE THE SURVEY USING A MOBILE DEVICE)

	CIRCLE ONE
Yes, has a smart phone	1	 (CONTINUE)
Yes, has a tablet	2	 (CONTINUE)

No, have neither that can be used   3 (RECRUIT FOR COMPUTER USABILITY, GO TO NEXT QUESTION)

In order to complete the survey using a smart phone or tablet, we will need to send you a document camera to use to record the screen on your device as you are using it.  Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

____________________________________________ (MAILING ADDRESS)

If for any reason your plans change and you can't keep your appointment, please call and let me know in advance. My telephone number is (INSERT NUMBER).

Finally, we will be sending you a “consent to participate in research form” via email that we will need you to sign and scan back to us in order to confirm your participation.

INTERVIEWER: RECORD APPOINTMENT DATE AND TIME ON FRONT OF SCREENER. COMPLETE NAME AND CONTACT INFORMATION.

Obtain the phone number and email address of the respondent and confirm that the phone number is the best one to reach him/her when they participate in the interview.

(RECORD BELOW.)

_______________________________________ (RECORD E-MAIL ADDRESS)

_______________________________________ (RECORD TELEPHONE NUMBER)

THANK YOU FOR YOUR PARTICIPATION.

 WE WILL SEND YOU LOGIN INSTRUCTIONS FOR THE SESSION AND ONE OF OUR TECHNICAL STAFF WILL CONTACT YOU TO SET UP AN APPOINTMENT BEFORE THE SESSION TO ENSURE YOUR WEBCAM, DOCUMENT CAMERA, AND THE ONLINE VIDEO SOFTWARE ARE WORKING PROPERLY FOR YOU.

Attachment IV

Interview Protocol

READ OR PARAPHRASE:

Section 1: Introduction

WELCOME respondent and thank them for their time.

INTRODUCE SELF: (Name/Shugoll Employee.)

This study is being carried out by Shugoll Research on behalf of the National Center for Education Statistics, which is part of the U.S. Department of Education. We are currently planning to conduct a survey to learn about the experiences of individuals who completed their bachelor’s degree requirements in the 2014-15 academic year. Today, I’m hoping to get your honest feedback on information about issues related to your employment, debt and finances, family formation, and undergraduate experiences.

The interview will be audio and video recorded so that I can concentrate on what you are saying rather than having to take notes, and to make sure that we don’t miss anything that you say. The recording will be destroyed once the project is completed. Do I have your permission to record? START RECORDING.

Section 2: Informed Consent (See Attachment II)

There are observers watching and listening via video streaming. These people are not here to observe you in particular but to learn from what you say about your experience of answering the questions.

However, I can assure you that everything you say will only be used for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, Section 9573). You will not be identified by name in any report we produce.

Your participation in this interview is very important because it will help us improve an education study with topics related to respondents like you. Once we start, the interview will last about an hour and a half. When we are done, we will mail you a $40 check as a thank you for your participation.

Section 3: Cognitive and usability Interviewing

There are no right or wrong answers. This is not a test. It is your experiences, views, and opinions that I am interested in. Some of the questions I ask I will not address further but other questions I may ask about in more detail. So I may ask you what you understood a particular term or phrase to mean, or if you could rephrase a question in your own words. However, please feel free to tell me anything that comes to mind or ask me anything you are unclear about.

CONDUCT INTERVIEW. ASK GENERAL PROBES AT QUESTIONS WHERE THE RESPONDENT HAS GIVEN SOME INDICATION OF DIFFICULTY WITH THE QUESTIONING BASED ON ANSWERS/BEHAVIORS, AS THE INTERVIEW PROGRESSES.

EXAMPLES OF GENERAL PROBES:

• What did you think the question was asking?

• Can you rephrase the question in your own words?

• How did you go about answering?

• What were you thinking of?

• How did you decide on your answer?

• What was the reason for that answer?

• How easy or difficult was it to answer the question?

• Probe for examples.

• How did you feel about answering the questions?

• Could you explain what you understand by this term?

• Can you tell me more about that?

• You seemed to hesitate/take a moment before giving/entering your response. Can you tell me what you were thinking about?

• What do you mean by…?

• Is there any additional information that we should have provided to help you answer that question?

• Does this term/question sound ok to you, or would you prefer something different?

EXAMPLES OF USABILITY PROBES:

• Was there anything in the survey that made it particularly easy or difficult to complete?

• Were specific questions more difficult to read than others? What could make it easier to answer these questions?

• During testing, did you notice the navigation and help aids that are available in the survey? Were these easy or difficult to use? What makes them difficult to use?

• What did you think about the labeling of the navigation and help aids? Did you understand the terms? What would be better wording to use?

• Are there any specific changes or revisions that you think could be made to the survey to make it easier to complete by computer, phone, and tablet?

Section 4: Debriefing Procedures

• Ask for any last observations/comments/concerns.

• Stop recording.

• Thank the respondent for helping test the survey items.

• Confirm the respondent will receive the $40 token of appreciation and end interview.

III-14