2012/17 Beginning Postsecondary Students Longitudinal Study (BPS:12/17) Cognitive and Usability Testing – Second Round

Attachment III
Eligibility Screening Questions

Intro: Thank you for inquiring about this study. I need to ask you a few questions to determine if you are eligible to participate. The screening is expected to take approximately 5 minutes.

1. What is your full name?

____________________________________________________

2. How did you hear about this study?

• Craigslist posting

• Facebook posting

• Other online posting

• Flyer

• Word-of-mouth

• Other_____________

3. Have you taken any classes since high school such as a vocational-technical or certification program, 4-year or 2-year college?

• No, never taken classes in a post high school education program –not eligible

• Yes, currently enrolled in a post high school education program

• Yes, post high school education program, but left before completing

• Yes, completed a post high school education program

Between

July 1, 2010 & June 30, 2012?

Yes

No (Not Eligible)

7/1/10-6/30/12?

YES NO

7/1/10-6/30/12?

YES NO

4. In what month and year did you FIRST attend any college, university, or trade school or certification program after you completed your high school requirements?

Month ______________

Year _______________

(If July 1, 2010 through June 30, 2012, go to question 6; If prior to July 1, 2010 or after June 30, 2012, Not eligible)

5. What is the name of the first college, university, or trade school you attended after completing your high school requirements, and in what city and state is it located?

School: ______________________________________City/State_______________________________

6. What degree or certificate were you working on at [FIRST SCHOOL NAME] between July 1, 2010 and June 30, 2012?

• Associate’s degree (usually a 2-year degree)

• Bachelor’s degree (usually a 4-year degree)

• Undergraduate certificate or diploma (usually less than 2 years), including those leading to a license (example: cosmetology)

• Undergraduate level classes

• Other__________________ (Not eligible)

7. After you completed your high school requirements and before attending [FIRST SCHOOL NAME], did you earn a degree or certificate at any college, university or trade school?

• Yes (Not eligible)

• No

8. Are you currently attending any college, university, or trade school?

• Yes

• No if no are you on a break between semesters, terms, or quarters, such as on fall break, and not currently attending classes or something else?

• On break between classes

• Completed program

• Left college/university/trade school prior to completing program

• Something else?

9. Have you ever taken out any undergraduate student loans?

(Include money borrowed from the federal government or from a private lender for all schools you attended. Do not include Parent PLUS loans, grants or scholarships, or money borrowed from family or friends.)

• Yes

• No

10. Are you currently married?

• Yes

• No

Thanks for those answers, now I just need some contact information.

11. What is the best telephone number to reach you (include area code)?

__________________________

12. Is it ok to leave a voice message at this number?

• Yes

• No

13. Is it ok to send a text message to this number?

• Yes

• No

14. What is your e-mail address?

____________________________________________

If ineligible: Thank you for inquiring about this study. If you are eligible to participate, we will contact you at a later date.

If eligible: Thank you for answering our questions. You are eligible to participate in this study.

What days of the week and times of day would you be available to participate in an interview?

___________________________________________________________

Thank you. We will contact you soon to schedule an interview time and answer any questions you may have about the study.

Participation will involve answering questions and reviewing a survey for an upcoming study that will be used to learn about the experiences and choices of individuals who entered college at the same time, whether or not they are still enrolled in college. During the interview, we will ask you to complete a survey and give us your thoughts on the questions you are being asked. In addition, we will ask you questions as you go, in order to measure your understanding and interpretation of the questions in the survey. 

Each full interview is expected to last approximately 90 minutes. The interviews will be recorded so that we can review your ideas and comments. We will prepare a report that does not single out any one person, but will summarize what we heard from the group as a whole. We will keep the recording only until our report is finished. No names will be mentioned in the report.

After your interview has been scheduled, you will receive a follow-up message to confirm the date, time, and location. At that point, you must RSVP to the invitation to guarantee an interview time. Upon completion of the interview you will receive $40 to thank you for your time.

Attachment IV
Post Eligibility Script for Device Determination:

Administered to screened respondents who meet the eligibility criteria and are interested in participating in the study

Thank you again for your interest in our study. You have been selected to participate in the study, if you would still like to take part.

You may recall, the purpose of the study is to obtain your feedback on some of the questions that will be used in a national study by the U.S. Department of Education. For some of the interviews we are asking the participants to access questions on a smartphone or tablet, such as an iPhone or Android phone or tablet. Do you have a smartphone or tablet with a data plan [IF NEEDED: internet access is part of the plan], that you would be willing to use during the interview?

Yes - Smartphone

• iPhone

• Android

Yes – Tablet

Brand_________________

No - Not willing to use my device/no data plan

No - Do not have smartphone/tablet

Have you used a smartphone or tablet in the past?

• Yes

• No

Attachment V
Cognitive Interview Protocol

Section 1: Introduction

Welcome participant and thank them for their time.

Introduce self: [Name/RSS employee]

Read or paraphrase: As we mentioned in our earlier conversations, we are working with the National Center for Education Statistics, within the U.S. Department of Education, to develop and refine questions for the Beginning Postsecondary Students Longitudinal Study (BPS). BPS tracks students’ paths through postsecondary education and helps answer questions of policy interest, such as why students leave school, how financial aid influences persistence and completion, and what percentages of students complete various degree programs. Today, I’m hoping to get your honest feedback on the survey questions we are planning to use.

During the interview, I will ask you to read the survey questions and respond to them. There are no right or wrong answers. In this study, I am less interested in your actual answers to the questions and more interested in what you thought about the questions and how you understood them. After you have gone through the survey, I will ask some questions about your answers, such as what you understood the questions to mean and whether the questions reflect your actual circumstances.

In order to concentrate on what you are saying rather than having to take notes, and to make sure that we don’t miss anything that you say, we would like to audio record this interview. Is that okay with you? [IF NOT OKAY WITH RECORDING: Okay, that’s no problem.]

I want you to know that your participation is voluntary and your responses are protected from disclosure by federal statute. All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law. We want you to know that you can speak freely. Also, you do not have to answer any question you don’t want to and can stop the interview at any time if you wish.

Your participation in this interview is very important because it will help us improve the questionnaire. We’re scheduled to be here for about 90 minutes, and I won’t keep you any longer. As a thank you for your help, we will give you $40 cash after the interview is completed.

Do you have any questions?

Write down questions

Section 2: Informed Consent

Before we start, I’m going to give you a written consent form. This document explains the purpose of the study, who can participate in the interviews, how we will protect your information, and who you can call if you have questions.

Hand the Informed Consent document to the participant. Read the document to participant or let participant read it (as appropriate). If participant chooses to read the document, tell him or her:

In addition to the points I have just mentioned about confidentiality and being observed I would like to draw your attention to the fact that:

• You are 1 of 20 people taking part in the interviews;

• Your participation is voluntary and you may stop the interview at any time

• You can refuse to answer any of the questions asked or take a break at any time

• Your responses are protected from disclosure by federal statute (20 U.S.C., § 9573). All responses that relate to or describe identifiable characteristics of individuals may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.”

When completed:

1. Ask participant if (s)he has any questions.

2. Ask participant if (s)he agrees to participate and agrees to be audio recorded.

3. If participant agrees, obtain the respondent’s signature to proceed – on two (2) copies.

4. Make sure respondent takes one copy with them when the interview is over.

Section 3: Cognitive Interviewing

If consent was given, start recorder

Okay, let’s get started. As you proceed though the interview I would like you to try and remember your thought process for the answer to each question. That is, be able to tell me how you went about answering the question, tell me if something didn’t sound right, was hard to answer, there were words you didn’t understand, etc. If any questions are unclear to you as you go through, please stop and let me know.

Do you have any questions before we begin?

If necessary, remind the respondent to “think aloud”.

Ask unscripted probes for questions where the respondent has given some indication of difficulty with the questioning based on answers/behaviors.

Examples of general probes you may use:

• What did you think the question was asking?

• Can you rephrase the question in your own words?

• How did you go about answering?

• What were you thinking of?

• How did you decide on your answer?

• What was the reason for that answer?

• How easy or difficult was it to answer the question?

• PROBE for examples.

• How did you feel about answering the questions?

• Could you explain what you understand by this term?

• Can you tell me more about that?

• You seemed to hesitate/take a moment before giving/entering your response. Can you tell me what you were thinking about?

• What do you mean by…?

• Is there any additional information that we should have provided to help you answer that question?

• Does this term/question sound okay to you, or would you prefer something different?

Section 4: Debriefing Procedures

• Ask for any last observations/comments/concerns.

• Stop the digital audio recorder.

• Thank the participant for coming in and for helping test the interview.

• Give the $40 incentive and escort them from the interview room.