Cognitive Test Report

Second Follow-up of the High School Longitudinal Study of 2009 (HSLS:09)

Cognitive Interviews - Round 2

Alisú Schoua-Glusberg, PhD

Katherine Kenward

Lisa Andrews

Daniela Glusberg

Patricia Marnell

Dánae Corado

Jessica Engel

Findings

Research Support Services, Inc.

September 16, 2015

Introduction

This report summarizes findings of the cognitive testing conducted by Research Support Services Inc. for the High School Longitudinal Study of 2009 (HSLS:09) Second Follow-Up field test survey instrument. Forty cognitive interviews were conducted in Illinois, in the Chicago Standard Metropolitan Statistical Area from June 15, 2015 through July 30, 2015.

The following table displays the respondents’ characteristics:

Case ID	Sex	Sexual orientation	Ethnic.	Race	Educ.	Post HS	Type	Job	HSLS Case ID	Device type used
208	Female	Straight	Not Hispanic	White	HS Diploma	Currently Enrolled	4 Year	Part Time	69990301	I phone 6
217	Male	Straight	Not Hispanic	White	HS Diploma	Left Before Completing		Full Time	69990202	Iphone
210	Male	Gay or lesbian	Not Hispanic	Black/ African American	HS Diploma	Currently Enrolled	4 Year	Looking for work	69990303	I phone 4
211	Female	Straight	Not Hispanic	Black/ African American	HS Diploma	Currently Enrolled	4 Year	Not employed not looking	69990203	I phone 4
215	Male	Gay or lesbian	Not Hispanic	White	HS Diploma	Currently Enrolled	4 Year	Part Time	69990204	Ipad
216	Female	Other: pan-sexual, bisexual but not adhering to gender binary	Not Hispanic	White	HS Diploma	Currently Enrolled	4 Year	Part Time	69990304	I phone 5C
218	Male	Bisexual	Hispanic	Black/ African American	No HS or GED	Never enrolled		Looking for work	69990205	LG Android
209	Female	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Completed	2 Year	Part Time	69990302	Laptop
229	Male	Bisexual	Not Hispanic	Black/ African American	GED	Completed	Vocational	Looking for work	69990305	Laptop
233	Female	Straight	Not Hispanic	Black/ African American	HS Diploma	Currently Enrolled	2 Year	Part Time	69990206	Windows phone, Nokia 63
235	Male	Straight	Not Hispanic	White	HS Diploma	Never enrolled		Part Time	69990207	Samsung Galaxy S4 Android
241	Female	Other: Not sure/ bisexual	Not Hispanic	Black/ African American	HS Diploma	Currently Enrolled	2 Year	Looking for work	69990306	Nexus 4 Android
239	Female	Bisexual	Not Hispanic	Black/ African American	HS Diploma	Currently Enrolled	2 Year	Looking for work	69990308	Laptop
242	Male	Straight	Not Hispanic	Black/ African American	No HS or GED	Never enrolled		Looking for work	69990209	Laptop
243	Female	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	2 Year	Part Time	69990208	Ipad mini
245	Female	Bisexual	Not Hispanic	Black/ African American	HS Diploma	Currently Enrolled	4 Year	Full Time	69990311	Ipad
254	Female	Gay or lesbian	Not Hispanic	Black/ African American	HS Diploma	Left Before Completing	2 Year	Looking for work	69990210	I phone 5s
258	Male	Straight	Not Hispanic	Black/ African American	GED	Left Before Completing	2 Year	Full Time	69990309	Samsung Galaxy Tablet
247	Female	Straight	Not Hispanic	Black/ African American	HS Diploma	Completed	Vocational	Full Time	69990310	Ipad
261	Female	Other: Pan-sexual	Not Hispanic	White	HS Diploma	Currently Enrolled	4 Year	Part Time	69990211	I phone 5s
282	Female	Straight	Not Hispanic	Asian	HS Diploma	Currently Enrolled	4 Year	Full Time	69990312	I phone 5s
246	Male	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	4 Year	Part Time	69990214	Laptop
289	Male	Straight	Not Hispanic	Black/ African American	No HS or GED	Never enrolled	Vocational	Looking for work	69990212	Android
292	Female	Gay or lesbian	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	4 Year	Full Time	69990313	LG Android
293	Gender non-conforming	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	4 Year	Full Time	69990213	Ipad
244	Male	Straight	Not Hispanic	Black/ African American	HS Diploma	Left Before Completing		Not employed not looking	69990314	Android HTC1
290	Female	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	4 Year	Part Time	69990215	Ipad
300	Female	Straight	Not Hispanic	Black/ African American	HS Diploma	Completed	4 Year	Looking for work	69990315	Iphone S5
231	Male	Straight	Not Hispanic	White	HS Diploma	Currently Enrolled	4 Year	Full Time	69990216	Laptop
296	Male	Straight	Not Hispanic	Black/ African American	HS Diploma	Left Before Completing		Looking for work	69990217	Laptop
306	Female	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	4 Year	Part Time	69990317	Laptop
309	Male	Straight	Not Hispanic	Black/ African American	HS Diploma	Left Before Completing	4 Year	Full Time	69990316	Laptop
230	Male	Straight	Hispanic	White	HS Diploma	Currently Enrolled	4 Year	Full Time	69990318	I phone 6
308	Male	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	4 Year	Full Time	69990219	Ipad
315	Male	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Completed	4 Year	Full Time	69990319	Ipad
313	Male	Straight	Hispanic	Hispanic- No race provided	HS Diploma	Left Before Completing	4 Year	Full Time	69990220	Ipad
321	Female	Straight	Not Hispanic	Verbatim response:"Mixed Black/ White"	HS Diploma	Currently Enrolled	4 Year	Part Time	69990221	Ipad
323	Male	Straight	Not Hispanic	White	HS Diploma	Never enrolled		Full Time	69990321	Ipad
325	Male	Bisexual	Hispanic	Hispanic- No race provided	HS Diploma	Currently Enrolled	2 Year	Part Time	69990222	Ipad
326	Male	Straight	Not Hispanic	White	HS Diploma	Currently Enrolled	4 Year	Looking for work	69990322	Ipad

Participants were recruited through online postings and flyers targeted at various postsecondary institutions and organizations. Flyers (shown on Appendix A) were handed out to students on five college campuses (including Chicago community colleges), and were posted in a variety of community venues, including job search centers, and community organizations serving youth --including those specifically serving lesbian, gay, bisexual, transgender, and LGBTQ¹ youth.

Interested candidates had to respond to a telephone screener (see Appendix B). Of the forty total participants, twelve identified as LGBTQ and twenty-eight as straight or heterosexual. Thirty-five respondents were currently or previously enrolled in a postsecondary program with seven of these having left prior to completion, while five had never enrolled in a postsecondary program. Thirty-five respondents had received their high school diploma, three had no high school credentials, and two had received a GED. Thirteen respondents were not working as of their interview date, eighteen were working while enrolled in a postsecondary program and nine were working and not enrolled in any program. Twenty-one of the forty respondents were male, eighteen were female and one was gender non-conforming. Thirty belong to racial/ethnic minority groups.

Interviews were conducted in two locations: an interview room in a Chicago area hotel and in a rented office in downtown Chicago. Each interview began with an introduction and administration of a consent form (shown on Appendix C). Then the respondent self-administered the questions on a laptop, tablet or smartphone. During the telephone screening process, participants were asked about the devices they use for accessing the Internet, and were asked to bring the device to the interview for web survey administration. Once respondents arrived at the interview, the interviewer determined whether the respondent’s device or an RSS laptop or tablet would be used. An effort was made to conduct testing on a variety of devices. Twenty-three large screen devices were used, and seventeen smartphones were used.

Large Screen Devices	23	58%
Ipad	13
Laptop	9
Samsung Tablet	1
Smartphones	17	43%
Iphone	10
Android	6
Windows Smartphone	1

The interviewer watched for usability issues throughout, including any difficulties the respondent had and made note of any spontaneous comments. Whenever the respondent reached a question of particular cognitive interest, the interviewer asked scripted probes about it, as well as any spontaneous probes warranted by the respondent answers. With NCES' and RTI's assistance, each question to be probed was pre-identified as high, medium, or low priority for probing, in recognition of the fact that self-administration and probes might make for too long an interview, beyond the OMB-approved 90 minutes (See Appendix D). Thus, interviewers kept track of the time to decide if they should skip low priority probes. The probing followed what is known as concurrent probing, that is, after each question of particular interest was asked, follow-up cognitive probes were administered to shed light on the mental process the respondent followed to provide an answer. The Interview Protocol appears on Appendix E. Showcards (see Appendix F) were used to have the respondent select one or multiple answers from a list.

Findings from Cognitive/Usability Testing

A01

Item-Specific Probes:

• PA01a What do you think they mean here by 'high school equivalency'?

• PA01b Which of these -- GED, HiSET, TASC -- were you familiar with? FOR ANY YES: Tell me what you know about them.

Most respondents (n=35) provided a definition of high school equivalency. However, they were often somewhat vague. For example:

"Something that is almost as good as a high school diploma"

"Same thing as high school"

"Certification of the same type of value as a high school diploma"

"A substitute for high school"

"An education equivalent to what you would have gotten if you had finished high school"

"Homeschooling or some type of education that can be proven to be equivalent to a high school education"

While five respondents were not sure about the meaning of high school equivalency, everyone had heard of the GED. For several respondents, it was closely related to their definition of high school equivalency. In fact, many defined high school equivalency by mentioning the GED. Other definitions included:

"A test to replace a diploma"

"You have to take a test in case you dropped out of high school"

"A test that gives you a substitution to a diploma"

"A test taken by people unable to complete high school after being enrolled"

"A program usually at a community college for those who dropped out of high school to earn a certificate that is like a high school diploma."

No one among the forty respondents had ever heard of the HiSET, but one thought he had heard of TASC and that it was similar to the GED.

There were no usability issues in this question, except that in a few cases the interviewer noted that the system was slow.

A04

Item-Specific Probes:

• PA04a: What do you think this question is asking?

• PA04b: Have you ever attended any program that you think would be considered an alternative program for adults or a high school completion program for adults? IF YES: When you answered <MONTH, YEAR> in this question, were you thinking of this program for adults?

• PA04b: What does “alternative high school for teenage students” mean to you?

• PA04c: And “traditional high school”?

• Usability Probes: UAO4c: ANY ISSUES WITH MONTH/YEAR ENTRY?

None of the respondents exhibited difficulty processing or answering this question. Because it only applied to those who did not have a high school diploma, it was only probed in a handful of cases. Those who did, thought the question was asking when they graduated or last attended.

Three respondents were asked about the meaning of 'alternative' high school. Two were able to define it ("schools willing to give you a second chance;" "for people who misbehaving or couldn't keep up.") and a third one said she was not sure what it meant.

No usability issues were observed regarding entry of month and year. One respondent went back to correct the month but had no issues doing so.

A08

Item-Specific Probes:

• PA08a: How did you figure what to answer? IF NECESSARY: Did you answer based on your cumulative GPA?

• PA08b: IF YES TO GPA: Was that your straight GPA or was that after your school adjusted it based on the level of the classes you took?

• PA08c: Were your grades consistent throughout your high school year(s)?

This question was probed in all but one interview. Respondents utilized various recall or estimation strategies, but all were able to answer. The vast majority gave their unweighted GPA; however, three said they had given a weighted GPA, one was not sure if it was weighted or unweighted, and one said his weighted and unweighted GPAs were the same. Finally, one respondent attended two different schools, one which weighted and one which did not weight grades, and the respondent said he considered both in answering.

While most respondents simply retrieved from memory their final GPA, others utilized different strategies to formulate a response. One respondent remembered her GPA was higher than 3.33 because she had a scholarship that required maintaining at least a 3.33 GPA. Three respondents could not remember the GPA so they gave their answer based on the letter grades offered in parentheses.

Some respondents had very consistent grades throughout high school while others did better some years than others and some were inconsistent within year. This made it more problematic when estimating. A respondent said he never looked at his GPA and instead "did a mental overview of my grades…that was the most recent average of my last two years." There was another respondent who only remembered his GPA from his sophomore year and senior years and picked a number in between. He added that he thought that he "low balled" his GPA in the process.

A10

Item-Specific Probes:

• PA10a: Were you able to find an answer that worked for your situation? What other answers did you consider, if any?

• PA10b: How well does the answer you chose capture your situation?

None of the respondents expressed difficulties finding a response while they were answering. During probing they explained their choices. Some were very straightforward, others required them to make a decision.

Two respondents selected ‘Algebra 1’, one without difficulty and the other because it was "close enough." This respondent took "Algebra and Algebra extended" and felt ‘Algebra 1’ captured it best. Two respondents picked ‘Geometry’ without difficulty or substantial comments. Two of six respondents that selected ‘Algebra 2’, did so without much need to consider. One other respondent remembered that in senior year she took a class called "College Algebra" and also remembered that it was sometimes called "Algebra 2" so she picked ‘Algebra 2’ from the choices offered. Another respondent was able to find an answer that fit and considered ‘Algebra 3’ but realized it was ‘Algebra 2’. Another respondent was unsure of the level to choose, but after asking if the levels of math were ranked or in order on the list, the respondent picked ‘Algebra 2’. Finally, the last one that chose ‘Algebra 2’, made the point of saying that he started ‘Algebra 3’ but dropped out of high school before actually completing the class, and was thinking the question probably wanted the highest level completed.

Several respondents selected ‘Algebra 3’ without much hesitation. Two selected it because they took ‘Trigonometry’. Another one took a class called "Advanced Algebra" so she picked ‘Algebra 3’, because the class she took came after taking ‘Algebra 2’; thus, she guessed this must be the right choice based on the list provided. Another respondent whose highest class was one that included some "smaller lessons" on pre-Calculus but, because he took it as part of Trigonometry, he choose ‘Algebra 3, Trigonometry, or Probability and/or Statistics’, since these were the things his class was mostly focused on.

Four respondents selected ‘Pre-Calculus’. None had difficulty making a selection.

Among the seven respondents choosing ‘Calculus’, 3 answered without issues. Another took AP Calculus and was looking for a choice specific to the level they took, but settled for ‘Calculus’. However, one more in that situation chose ‘Other’, and wrote in "AP Calculus."

Two more respondents selected ‘Other’. One specified ‘College Algebra’ for his AP-level algebra course. The other wrote in that the highest math he took was Differential Equations.

Generally, other than the issues described above, respondents felt the list was comprehensive and was not missing anything.

No usability issues to report in this question.

A11

Item-Specific Probes:

• PA11a IF YES: Tell me about it. What kept you out of school?

• PA11b IF NO: Tell me about any time stayed out of high school during the school year for at least 2 weeks. IF SOMETHING REPORTED: How do you remember it was two weeks?

• PA11c FOR BOTH YES: What grade were you in? How do you remember that?

All but one of the 40 respondents were asked the probes for this question. Of the 39, only one answered ‘Yes’ to the original question. The reason that kept him out of school for that long was that he was expelled.

Four other respondents would have answered differently if the reference period of this question had been two instead of four weeks:

• A respondent missed school for at least two weeks when he had bronchitis in junior year. He remembered it well because he was in the lacrosse team and tried to hide his illness so as not to miss any games.

• A respondent had a medical issue that caused her to be out of school for 3 weeks during her junior year of HS.

• A respondent was suspended for two weeks in the spring of 2009.

A respondent missed two weeks of school due to surgery in 11th grade. She remembered the circumstances clearly.

A12

Item-Specific Probes:

• PA12: What does the word ‘transfer’ mean in this question? Can you give me an example?

Six respondents transferred schools during high school for different reasons, such as household moves, or switching from private to public school. This was an easily understood question, and there was only one respondent who was not sure about how to answer. He asked the interviewer if the question was "referring to a mid-year transfer or just going to another high school the next year?" The respondent was thinking about the fact that he attended private school for first two years of high school and then switched to public school for the last two years of high school. He wanted to make sure this counted as a transfer as opposed to just changing schools.

For another respondent, "transferring" in high school means having to move with your family and implies lack of choice for the student. By contrast, in college, he thinks of transferring as when you choose to leave a school to go to a different one.

All others provided definitions of transfer that evidenced clear understanding:

• "moving schools in the middle of the year;" "a move from one high school to another"

• "permanently leave your high school to attend another one"

• "transferring your credits to another high school, for reasons such as moving to a different town"

• "to switch schools"

• "completing different semesters at different high schools"

• "It's when you have your transcript sent to another school that you now attend"

• "you spend some time at one school and then you move or something happens and you go to another school"

No usability issues in A12.

A13

Item-Specific Probes:

• PA13: What do you think they mean by “a high school completion program for adults”. Have you ever heard of a program like that? Where would these programs be held?

Most respondents were skipped out of this question based on prior answers. Six that answered it were probed on the meaning of 'a high school completion program for adults.' All mentioned the GED in their response and said classes can be offered at a high school or at a community college.

No usability issues observed.

A16

Item-Specific Probes:

• PA16a: “How did you come up with your answer?”

• PA16b: Were you familiar with all of these tests? FOR ANY YES: Tell me about them.

• PA16c: What type of tests did you take?

• PA16d: Were there any tests that you were unsure of whether you should include?

• PA16e: Did you include or exclude them and why?

• PA16f: Did you count each subject test separately or together as one test? Why?

• PA16g IF COUNTED AS ONE TEST: If you were instructed to count each subject test separately what would be your answer?

• PA16h: Did you ever retake a test or a subject test? If so, how did you count that?

• PA16i: Did you ever take parts of the exams at different times or different days?

Five respondents indicated they had taken the GED. However, the first of these had in fact only taken a GED pretest exam but not the actual GED exam.

Two respondents said they took the exam once. In both cases they noted in probing that the test was divided into sections with breaks in between, but that they saw it as a single test.

One respondent said he took the test three times. This respondent also counted the sections as part of a single GED test and noted that the subject tests were spread over multiple days but that they were a single test. To determine his answer, he thought about how many times he took GED preparation programs.

A final respondent said he took the test four times. Unlike the others, this respondent counted each of the four sections as an individual test. Since he had taken two sections each day over two days he came up with his response of four. However he also noted that he had to take section tests multiple times but counted each one only once.

No usability issues observed.

B05

Item-Specific Probes:

• PB05a: IF YES: What were you thinking of? Tell me about the college you attended? PROBE TO UNDERSTAND IF COLLEGE PROGRAM, ISOLATED CLASSES, DEGREE/non-DEGREE, OR ADULT ED CLASSES TAUGHT IN A COLLEGE ENVIRONMENT.

• PB05b: What do they mean by “college” in this question? What other kinds of schools could they also mean?

  • IF NECESSARY: Would you say yes if you had attended a technical institute?

  • What about a trade school?

Although the introduction to the post-secondary section defines what the survey considers college, many respondents either did not read the definition or did not retain it. Nineteen respondents said they would include colleges, technical/vocational and trade schools in their answer. They felt the question meant to include "anything postsecondary, including technical or trade school," "anything higher than high school," "any enrollment in any program after high school." One respondent clicked ‘Yes’, thinking about culinary school and Job Corps training. About five respondents mentioned the definition at the beginning of section B.

The other 21 respondents did not believe that 'college' includes everything in the definition, in particular, some did not think it includes technical school, and many did not think it includes trade school: "any school after high school...except like obviously some music class or something wouldn't count; trade school sounds more like training to me." One respondent specifically said that if the survey wanted to include a trade school the question itself should say "college or trade school."

One respondent did not know what trade school is. Another one stated she did not know too much about either technical schools or trade schools, and a third one seemed fuzzy on the concept.

One respondent answered ‘No’ in error, and was routed down the wrong path for their situation. The interviewer noticed and made the respondent back track and correct the answer.

B28

Item-Specific Probes:

• PB28, 29, 30, 32a: I noticed in the survey you said that you [did not attend college/took a break after high school/left one college to enroll in another/no longer are attending] due to [REASONS].

• PB28, 29, 30, 32b: Tell me about what led to that decision? GET NARRATIVE REASONS.

• PB28, 29, 30, 32c: Did the reasons offered in the survey adequately cover your situation? Tell me more about that.

Three respondents were asked probes in this question. Two respondents selected 'Financial reasons.' One of them explained that an uncle had offered to pay for college but a feud in the family prevented it; he felt no other option applied to the situation. The other explained that, because of previous choices he made in life, his parents would not support him financially in going to college. He considered also picking 'Personal or family reasons' but ultimately decided that 'financial reasons' covered it adequately. He commented that he liked that those were separate categories.

Finally, the third respondent selected 'Personal or family reasons' and 'Financial reasons'; he explained that he did not have the money and "I grew-up by myself, didn't have a mom and dad to help me out."

B07A

Usability Probes:

potential usability issue -- DID R HAVE TROUBLE FINDING “ENTER” – probe specific issue and if R had expected something different if not clear.

Usability/Cognitive

Most respondents had no difficulty in this question. They entered their school name and found it on the list. They clicked ‘Enter’ to continue. One respondent did comment that it was odd to have the ‘Enter’ button. Another respondent said she was not sure why she needed to click on ‘Enter,’ and then on the arrow, but as soon as she clicked ‘Enter,’ she understood why. Another respondent thought it might be difficult to find the school on the list or to trigger the right names on the list, for persons who attended schools where the names are often abbreviated in different ways.

For one of the respondents, the system froze after she entered her school, but she hit Refresh and it was resolved.

Four respondents had issues with finding the right college in the listings of schools.

After typing in the name of his college, a respondent was unsure if he should select the main college campus (CITY2) that came up because, "There are two campuses: the one in CITY1 and the one in CITY2 and they don’t have my campus (CITY1) listed here." He eventually decided to go ahead with selecting the only option because, "It's the same institute, but like....different locations."

A second respondent typed in his school but could not find it in the list, so he selected ‘None of the above.’ [Interviewer note: This was a large community college; however, the screen was hard to see and it was not clear if he had mistyped the school name].

A third respondent was not sure which city his school was located in, he clicked ‘List cities’ and scrolled through trying to find the right city. He noted there were too many cities to figure it out but finally did find it and clicked ‘Enter.’

Finally, a respondent typed in the name of his college, a large public university with a geographical name. The selection box that popped up listed four colleges with very different names and all from a neighboring state (it is possible he entered the wrong state abbreviation and the interviewer did not notice.) The respondent hit ‘Next’ and continued to the next screen. When probed, he said he thought the list was “what if you didn’t mean this college” (the one you typed in) to consider a different one; since he had meant his original college, he did not review. He did not see the ‘None of the above’ option. The options on the following screens (without the correct school) did not have the option ‘four year public.’ The respondent had seen the help screen ‘keep option or go on’ as if you want to keep what was originally typed, rather than a selected school.

B10

Item-Specific Probes:

• PB10a: What do you think they mean by school-scheduled break here?

• PB10b: What do you think they mean by taking classes here?

• PB10c: Let's say you are in a program where all the classes are online. How would you answer this question?

• PB10c: Let's say you are in college and are currently doing a semester abroad. How would you answer this question?

• PB10d: Let's say all you are doing this semester is an internship for school credit. How would you answer this question?-

• PB10e: Let's say you are doing an independent study or working on a research project for credit. How would you answer this question?

• PB10f: If the question asked: Are you still enrolled at [post-secondary institution]? How would this question be the same or different from Are you still attending [post-secondary institution]?

For the vast majority of the respondents, a 'school-scheduled break' was a summer break or time between school terms, and even a few mentioned holidays, or 'time when school is not in session.' However, a few saw it differently. One respondent who is not currently enrolled and is taking a break from school but has the option to return, answered 'on school-scheduled break' because he chose to leave and planned it. One respondent defined the break as possibly including "a leave of absence." Another defined the break as "being on probation." Another brought up the notion of deferment to define a school-scheduled break. Three respondents considered internships, research projects, and independent study as school-scheduled breaks.

One respondent on summer break first answered the first option, but when she saw the second one, she changed her answer. She appeared confused and asked the interviewer whether she was answering correctly.

Most respondents interpreted 'taking classes' as being enrolled and doing something for which you get credit. For them, taking classes does include going to class in person, taking online courses, and doing a semester abroad. Internships and independent study were not as clear cut for some respondents, because there is no classroom environment involved.

For several respondents, you can be enrolled in a program but not necessarily taking classes at the moment. For instance, you are in a program but are taking a semester off. However, you have the option to return after a break. Enrolled would also include internships, independent study, semester abroad, as well as classes (in person or online), or doing a research project for credit. One respondent explained that at her community college, if a student does not take any classes for three years, they need to re-enroll. For a few, enrolled and attending mean the same.

Usability:

In one case, the question loaded so slowly that the respondent had to answer it twice. (case 69990203)

B16A

Item-Specific Probes:

• PB16Aa: Tell me what you were thinking when you answered this question? INTERVIEWER, IF R ANSWERED 4, PROBE TO SEE IF R WAS TRULY NOT ENROLLED.

R answered that he was "not enrolled in a program but taking classes" because he is just taking classes but has not figured out what kind of degree he can get or the requirements for graduating. He added that he "is not on a pathway of classes towards a degree". Two respondents expected a more specific set of answers. Both were in BFA programs and ended up choosing B.A.

A respondent who is in an Associate's Degree program but now on summer break, was hesitant to answer because she will not actually be enrolled until the fall.

While a majority of respondents were able to answer the question as intended, there were five respondents for whom this question was problematic in some way:

• A respondent not currently enrolled and not taking classes answered ‘Not enrolled in a program, but taking classes,’ because he found no better choice. He answered "No" when asked if he was still attending the college he named. He had taken a psychology class in the past, but was never enrolled in any type of program.

• A respondent enrolled in a dual BA/MA program where she had just completed and received a bachelor's degree and is now continuing toward the master's at the same school and in the same program. This respondent was confused with the instrument since she is both currently enrolled and has graduated. She would have wanted to select 'in Graduate program' but that was not an option. She chose 'Certificate or diploma program...’ because of the mention of occupational training, given that her master's program is for an occupation leading to a license. She recognized that this choice didn't really fit and that it was intended for 'trade school,' but it was a better fit than the others. [After continuing through several questions after this one that did not fit her situation she went back and clicked that she is currently completing her bachelor’s --even though she already has-- just so she could get through the section.]

• A respondent who is currently doing an internship for an MA found no response choice that suited her situation. She selected ‘Not enrolled but taking classes’ because she thought it was the best option.

• A respondent who had already graduated said that she would answer the question as if she were in residence taking classes.

• A respondent who was taking classes to fulfill his general education requirements before moving from a 2-year college to a 4-year university answered ‘Associate's degree.’ He said he was first going to answer 'Not enrolled in a program but taking classes,’ but "when I thought about how long I was going to stay there, that made me choose 'Associate's degree program.’"

B21A

Item-Specific Probes:

• PB21Aa: Tell me more about your answer.

• PB21Ab: Did you have any hesitation as to what answer to select?

• PB21Ac: What do you think they mean by a degree requirement here?

• PB21Ad: What do you think is the difference between the first and second response choices?

• PB28: Which of the following reasons best describe why you have not attended in college?

Only a couple of respondents were probed on this question. One of them chose the first option because he did not take this class for any particular reason; he just wanted to get a "feel" for college. Another respondent chose "degree requirement" because her goal is not to "take classes just to take classes;" she is working towards a Master's degree.

Usability:

One of the respondents brought up the HELP screen to find a definition of requirements. She said it was not really helpful for making an answer selection. She was anticipating the HELP screen would tell her if the question was asking about general requirements or major field requirements and it did not tell her that. (case 69990211)

B29

Item-Specific Probes:

• PB28, 29, 30, 32a: I noticed in the survey you said that you [did not attend college/took a break after high school/left one college to enroll in another/no longer are attending] due to [REASONS].

• PB28, 29, 30, 32b: Tell me about what led to that decision? GET NARRATIVE REASONS.

• PB28, 29, 30, 32c: Did the reasons offered in the survey adequately cover your situation? Tell me more about that.

This question was not probed for any case.

B30

Item-Specific Probes:

• PB28, 29, 30, 32a: I noticed in the survey you said that you [did not attend college/took a break after high school/left one college to enroll in another/no longer are attending] due to [REASONS].

• PB28, 29, 30, 32b: Tell me about what led to that decision? GET NARRATIVE REASONS.

• PB28, 29, 30, 32c: Did the reasons offered in the survey adequately cover your situation? Tell me more about that.

Respondents who transferred from one college to another had no difficulty clicking ‘Yes’ to this question. For instance, one respondent clicked ‘Yes’ since she transferred schools, given that the program she was in was not well developed and the school was located in a city with few opportunities.

One respondent who clicked ‘No’ was probed about his situation. The interviewer asked, based on his prior answers, why switching from his community college to the state university did not qualify for a ‘Yes’ response. The respondent explained that he had been in a special program where if a student did well at a community college near the state school, they automatically moved him over, and the community college would not even appear on the student's transcript. Thus, he did not think he should answer ‘Yes.’

Two respondents who changed schools but did so when they graduated from college and started at a graduate program elsewhere, were not sure about how to answer. Two other respondents appeared confused about the question. One was able to process it and answer after re-reading. The other clicked ‘Yes,’ but upon probing she indicated she was confused about what the question was asking: "I thought of classes not counting to how soon I would graduate. Or now I'm thinking it's whether you transferred or not." She then said she did switch from one community college to another because of bad experiences with professors.

Usability: No issues observed

B31

Item-Specific Probes:

• PB28, 29, 30, 32a: I noticed in the survey you said that you [did not attend college/took a break after high school/left one college to enroll in another/no longer are attending] due to [REASONS].

• PB28, 29, 30, 32b: Tell me about what led to that decision? GET NARRATIVE REASONS.

• PB28, 29, 30, 32c: Did the reasons offered in the survey adequately cover your situation? Tell me more about that.

Only one of the respondents missed the instruction indicating that multiple answers can be selected. The rest either chose multiple responses or at least indicated they saw one could do that. One respondent chose all answers except for 'other,' since he felt they all partly covered his situation.

Generally respondents felt their situation was adequately covered in the available response choices. One, for example, answered he left one school for another due to financial reasons: he transferred to a less expensive college, closer to home. Another one who picked the same response, cited higher costs than she had expected.

Two respondents chose 'academic reasons.' One of them, for example, transferred because she was moving on from her AA degree to study for her BA.  

Finally, one respondent picked 'other' and wrote in her answer, but the interviewer did not probe.

Usability:

The respondent who was enrolled in an M.A. program and answered 'taking classes' at question B16A, for lack of a better option, was routed to this question but did not feel it applied to her. She answered 'other' to move on. She did not realize she could have left questions blank that did not apply to her.

B32

Item-Specific Probes:

• PB28, 29, 30, 32a: I noticed in the survey you said that you [did not attend college/took a break after high school/left one college to enroll in another/no longer are attending] due to [REASONS].

• PB28, 29, 30, 32b: Tell me about what led to that decision? GET NARRATIVE REASONS.

• PB28, 29, 30, 32c: Did the reasons offered in the survey adequately cover your situation? Tell me more about that.

Two respondents answered these probes. One of them felt that the choices offered did not cover what he was thinking. Under 'Other,' he entered "Did not need the classes to achieve my goals."

The other respondent thought the options given were sufficient for him to make a choice. He answered that he was no longer attending school for both personal/family and financial reasons, explaining that he did not have a lot of support/guidance from his family and that he was going to school on his own.

Usability:

The respondent who answered 'Other' had to shorten his answer and could not list all the reasons he wanted because the write-in form only allowed him to type in a few characters. This respondent was using a laptop.

A respondent tapped 'Select/Unselect All' instead of 'Continue' to leave the screen on his Android HTC1 smartphone.

B79

Item-Specific Probes:

• PB79a: How did you figure your answer?

• PB79b: Tell me about the situations of the people you counted.

• PB79c: When you answered the question, whom did you include as close friends or family members in your thinking?

• PB79d: Were there any people you were unsure of whether or not to include?

• PB79e: Did you include them or exclude them and why?

Respondents utilized different strategies to provide an answer. Some counted and some estimated. One respondent entered '0' because everybody says they will return to school at some point. The responses were as high as 15.

Generally they focused on nuclear family plus members of the extended family they maintain contact with. One respondent was not sure if a step-father should be included. As far as friends, while some kept the number down and included their innermost circle only, others seemed to include a large group of friends. A couple of respondents decided to answer only about close family and not get into their “friends' business.”

For respondents who know many people who dropped out of college or who have large groups of close friends and family, this seemed like a time-consuming, hard-to-answer question. A few others commented on the fact that it is difficult to keep track over time if someone who had dropped out is now back in school.

Respondents described a variety of reasons why their family and friends left school without graduating. These included financial reasons and mental health issues.

There were a few confusions. Two respondents misread or misinterpreted the question. One thought the question asked about those who had attended college or gotten a degree. The other thought the question asked who had "left with a degree." Two others had to read the question multiple times to be sure what was asked. A fifth respondent selected 'None of your close friends or family have gone to college,' when he really meant none had dropped out.

Usability: No issues observed. One respondent clicked on 'close friends and family members' and was successful in getting a definition from the help text.

B63

Item-Specific Probes:

• PB63a. Tell me more about your answer. IF AGREED OR STRONGLY AGREED TO COMPUTER ITEM: Do you think of yourself as a 'techie'?

• PB63b. Tell me more about your answer.

Most respondents when asked about their answers said that they agreed because they like using their devices, however both those who saw themselves as a ‘techie’ and those that didn’t said that a ‘techie’ likes figuring out the devices, fixing them, learning about them or being interested in the future and future technology. Of the 32 respondents who were asked if they see themselves as a ‘techie’ 19 answered no. Only one respondent disagreed with both questions, he noted that he grew up without computers or gaming systems in the home. No one who disagreed with question B63a, agreed with B63b however four people did agree with B63a but not B63b. One person noted that he sees ‘structural’ as different than ‘mechanical or electrical’ since ‘structural’ can refer to things like music and as a songwriter he likes that. He, along with two other people commented they wanted a third ‘neutral’ option.

Usability:

Four interviewers reported that the system was running slow.

On all smartphones, grids such as the four column grid in B63 and the five column grid in B38 required scrolling right when the device was held vertically. Interviewers reported that only the first two columns ‘strongly agree’ and ‘agree’ were visible on a vertically held smartphone screen with ‘disagree’ and ‘strongly disagree’ only visible after scrolling right. In addition, the formatting of the instrument on the smartphone for all questions was a dark blue/green border surrounding a white field with black text. When a respondent needed to scroll to the right, the white field did not travel to the additional response options. Instead, the remaining response options were seen as black text on the dark blue/green background field, similar to what one would see after highlighting text.

Despite the need to scroll, respondents generally did not rotate their smartphone to landscape view. Interviewers reported that almost all respondents maintained a vertical orientation and scrolled right for additional response categories. Two interviewers reported that 1-2 respondents held their phones either some or all of the time in landscape view.

On tablets, interviewers reported that scrolling was not necessary on some devices but necessary on others, however models requiring scrolling were not identified. Interviewers reported that most respondents held their tablets horizontally throughout the interview, one interviewer said two respondents held their tablet vertically. None reported that the tablet was turned (orientation switched) during administration.

For interviews administered on laptops, the horizontal orientation allowed all response options to be visible for all questions.

One respondent who had kept her phone vertical, was asked if she thought of turning her screen horizontally to see all answer categories, she said she had not considered doing that. She then turned her phone and saw that she could see all columns and thus all response categories; however, on all remaining questions she continued to hold the phone vertically and scrolled right to see the remaining response categories. This respondent was also asked in B38 if the blue/green dark field with black text was a problem and she responded that it was not (see B38).

B38

Item-Specific Probes:

• PB38a: What do you think they mean by ‘treated fairly’ in this question?

• PB38b: What situations did you consider when you answered? Can you give me some examples?

All respondents seemed to understand this question as intended. The term ‘treated fairly’ was described as without bias or favoritism, being graded the same; having equal time and attention in discussion and in the individual attention the teacher provides. Examples of being ‘treated fairly’ or not included showing bias, gender stereotyping, thinking boys (or girls) were better in a subject and showing favoritism to one's own gender. The only student who said treatment was different noted that the teacher gave the males more freedom to speak.

Usability:

Several interviewers reported that the system was slow.

When using a smartphone with a vertical orientation, respondents would have to scroll right to see the additional response columns on all questions with a grid format (see usability B63). The formatting of the instrument on the smartphone was a dark blue/green border surrounding a white field with black text, however whenever a respondent scrolled right the white field did not travel to the additional response options. Instead, the dark blue/green border became the background field with the additional response items remaining in black text. One interviewer thought this was more difficult to see, similar to reading through highlighted text. When she asked a respondent if the field color difference from light to dark when scrolling right was an issue, the respondent said it was not an issue but did make the site look ‘poorly formatted’.

B35

Item-Specific Probes:

• PB35a: How well did the response options capture your reasons for taking those courses?

• PB35b: Did you think you could select more than one reason?

• PB35c: In addition to the reason you chose, was there another reason? IF YES, what was that?

Most respondents were skipped out of this question.

Additional response categories offered by the respondents were “friends or family encouragement” and that they took additional courses "for the enjoyment of it."

Usability:

There were no usability issues although one R did point out that he recognized that he could select more than one reason by the shape of the response boxes.

B36

Item-Specific Probes:

• PB36a: How well did the response options capture your reasons for not taking those courses?

• PB36b: Did you think you could select more than one reason?

• PB36c: In addition to the reason you chose, was/were there another/other reason(s)? IF YES, what was that/ were they?

For most respondents the options captured their reasons. There were however several different reasons cited by respondents including, “not being academically prepared” and “cost.” Three Rs had not yet reached the point in their academics to take higher level courses but intended to and, three more had scheduling issues. In addition two respondents at specialty schools such as a culinary school noted there were no additional math courses available. The three respondents who felt that the categories did not capture their reasons included a student at a culinary institute, one said she does like math but it is very difficult for her and a third noted that she did not take them because they were computer based at her school and she did not like format.

Usability: No issues except a few noting a slow machine. 

B65

Usability Probes:

potential usability issue -- DID R HAVE TROUBLE FINDING “ENTER” – probe specific issue and if R had expected something different if not clear.

Generally, respondents were able to answer without difficulty, either by typing in an answer and clicking on Enter --which they found easily-- or checking the ‘Don't know’ option.

One respondent found the phrase 'most seriously considering' very helpful. Another remarked that the definition of Environmental Studies offered was "the best I have ever seen."

Usability:

One respondent appeared confused as to why there was an ‘Enter’ button. Another one scrolled up and down, unsure of what she had to do to complete this question; she eventually selected her major and the correct description.

Another respondent clicked ‘Enter’ and then quickly hit ‘Next’ without realizing there were two parts to the question, that is, that she first should type in a response and then choose from the list offered. The interviewer had to help.

B81

Item-Specific Probes:

• PB81a: What do you think they mean by “spending time with your family and friends”

• PB81b: How did you come up with your answer?

• PB81c: Do you live with family or friends? IF YES: Tell me more about how you figured your answer for option b.

• PB81d: Tell me how you figured what is a ‘typical week’ for answer d (extracurricular activities). Did you think about sports in a specific season?

Respondents used different strategies to calculate their answers and seemed to vary in the level of precision they tried to achieve.

The question about time with family or friends seemed to have more variability of interpretation than the other activities. Most defined it as “hanging out” with friends or family. However, respondents' living situation seemed to affect the way they decided what the question was asking them. A respondent decided not to count time spent with his girlfriend because he lives with her, so he only counted when he goes out with friends or to see family. Another respondent who lives with her two best friends was not sure whether to include the time when they are just home together sitting around the apartment. A respondent who lives with friends considered his responses in relation to how much time he spends socializing with them, which he considers quality time, as opposed to studying. Another respondent considered that time with friends meant time going out with them to parties, meals or other activities outside of the apartment where they live together.

What was actually included in 'spending time' varied by respondents as well. While some discounted what they considered non-quality time (e.g., the respondent who said that she only considered quality time when she and her family or friends were actually doing something and not just sitting around watching TV), another included time not spent face-to-face but in electronic or phone communications. Another counted only going out with family, not just regular time spent together.

A respondent indicated a 'typical' week is one in which he plays sports or works out. Another said it is a week without final exams. A third one thought of a typical week as one in which you are working and taking classes.

One respondent did not know what they meant by ‘extracurricular activities’ and wondered if volunteering would be included.

One respondent found counting the number of hours hard, and skipped answering.

Usability:

One respondent had difficulty seeing this screen on his tablet and often zoomed in and out.

Three respondents tried to enter double digits but all they could see on the phone screen was the first digit (e.g., could only see "2" when typing "25").

A respondent tried to enter a range but saw that was not allowed and had to enter just one number.

EMPLOYMENT SECTION

Now we are interested in learning about your employment experiences since you [received your high school diploma/received your certificate of attendance or completion/last attended high school].

C03 / C04

Item-Specific Probes:

• PC03/C04a: IF YES: Tell me about the internship/internships you have had?

• PC03/C04b: How is an internship different from a job? Different than volunteer work?

All respondents were able to process this question. Fifteen respondents initially reported having had an internship with most of those reporting they had a paid internship. One respondent was unsure if she should report her internship as paid or unpaid because she was paid from a grant through the school that covers wages for unpaid internships.

Thirteen respondents saw internships as a way to develop career skills or learn about a career. Almost as many differentiated it from a job in terms of school, noting it provided credit or learning opportunities. Four noted it provided mentorship opportunities or to build connections. One respondent noted that an internship was “less independent” than a job (more supervision) while another saw it as having more supervision than a job and one respondent described it as: "just a stamp on your forehead that says you are too young to be taken seriously.”

When asked the difference between an internship and volunteer work, three respondents said they were the same or similar but most respondents saw a difference although not all were able to describe the difference clearly. Volunteer work was described as different in that it provided social benefits, offered flexibility or convenience with scheduling; it was also described as less prestigious than internships or short term.

There were no usability issues although one interviewer reported that the system was slow.

C02

Item-Specific Probes:

• PC02a: What does a co-op job mean to you?

• PC02b: How is a co-op job different from an unpaid internship / paid internship?

• PC02c: IF YES: Tell me about your job?

• PC02d: Was the job through the school?

Twenty seven respondents were unfamiliar with the term ‘co-op’. The rest seemed to have only a slight understanding of the term and relied on the definition provided to help them come up with what it meant to them. It was described as a “real job” as compared to an internship or a high paying job. Other respondents saw it as a work/study arrangement or one with “more to do.” One respondent thought it might be “group work” with more than one person working together. One who did not know the term guessed that it was when you alternated work and classes each semester.

No one responded that they had a co-op job.

Usability: No issues except for one case that was consistently slow.

C33

Item-Specific Probes:

• PC33a: What does a work-study job mean to you?

• PC33b: How is a work-study job different from an internship or other job?

• PC33c: IF YES: Tell me about the work study job/s that you have had?

• PC33d: IF NO, did you try to get a work-study job? Was this opportunity available to you?

Respondents generally understood what work-study jobs are based either on their own knowledge or the question description. Both respondents who had work-study and those that did not described it as different than an internship. They described it as an on campus job used to pay for college costs rather than an internship which was to gain experience within a major or an off campus internship. Some respondents noted that internships can be unpaid whereas work study is paid. Respondents also noted that it was associated with financial aid. One respondent noted that work-study jobs are more accommodating to a student’s schedule than an internship.

One respondent misunderstood work-study as a job that is needed to fulfill certain requirements a student needs before s/he can apply for a specific program and that one would not attend school at the same time. One other respondent who had also not heard of work-study guessed that it would be in the field of study or help in getting a job after school.

Usability: No issues except for one case that was consistently slow.

C22

Item-Specific Probes:

• PC22a: What does an apprenticeship mean to you?

• PC22b: How is an apprenticeship different from an internship or job?

• PC22c: IF YES: Tell me about the apprenticeship/s that you have had?

One respondent answered ‘yes’ to apprenticeship however the interviewer believed this to be a false positive, the respondent had keyed into the concept of paid ‘on the job training,’ and since he had been paid during his job training he thought he should answer ‘yes.’

Six respondents noted that an apprenticeship was for the trades, vocational or technical training. One of these respondents also noted that one does not need college as you would with an internship.

Thirteen respondents noted that an apprenticeship was working directly under someone, in a mentorship relationship, or in a one-on-one training. Several of these respondents contrasted this with an internship that was less formal.

Respondents also saw an apprenticeship as job-specific training, one noted that an apprenticeship qualifies you for a job, “at the end of an internship you leave as an intern at the end of an apprenticeship you leave with a job.”

Twelve respondents mentioned they were unsure what an apprenticeship was with one noting that she had heard the term from the TV show ‘The Apprentice’ but didn’t really know what it meant.

Usability: No issues observed

B27

Item-Specific Probes:

• PB27a: IF YES: What type of certification or license have you received?

• PB27b: IF NO: Have you thought about getting any certification or license? Tell me more about that.

Eight respondents said they had received certifications and licenses. These included two food handlers, a CNA, a CDL, a pharmacy technician and a massage therapist. The remaining two include one respondent who has a certification as a soccer coach and one that received certification in MS Word and Excel while in high school. The respondent with the CNA also noted that she has a CPR certification. The respondent with the CDL also has a certification as a dishwasher but was unsure if the question was looking for that.

Of the twenty-seven respondents who clicked “No,” fifteen have considered receiving certifications or licenses, including two who will be getting master’s in Social Work and several who have considered but not pursued other fields such as beautician, bar tending, real estate, EMT and HVAC systems. One noted he would consider it when he determined what he wanted to do.

Usability: No issues except for one case that was consistently slow

C06

Item-Specific Probes:

• PC06a: How did you come up with your answer?

• PC06b: Are there any jobs you have had that you were not sure if you should include?

• PC06c: Have you worked any gigs, one-time jobs or temp-jobs? How did you count these?

• PC06d: If you work at the sandwich shop and move from making sandwiches to taking orders, would that count as one job or two?

No significant difficulties were observed among respondents in providing an answer to this question. While some respondents counted every work activity for which they got paid, others only counted 'official' jobs, but not short-term or temporary gigs, or odd jobs. One respondent based his answer on the number of W-2 forms he received; he did not include one time jobs or jobs where he was "paid under the table."

In one case, a respondent counted all her babysitting gigs for different clients as one self-employment job, but others counted every occasion in which they received payment. Others were not sure if they should consider a number of odd jobs (such as moving, or construction jobs) as one self-employment job. One respondent said: "They send me to a ton of employers so I wasn't sure if I should consider every single one or just the temp agency, so I just counted the temp agency." Another reported six jobs total, with five being regular employment and the sixth one "is when I don't have a job and I hustle, like I cut grass or do miscellaneous things around the neighborhood for other people".

Respondents had no difficulty including in their count the specific types of jobs listed in the question (paid apprenticeships, co-op, work-study, or internships). Summer jobs, however, were forgotten or excluded by some respondents, not deliberately in every case since upon probing the respondents remembered them and corrected their answer to include them.

Some respondents felt that a job had to have some minimal duration to be reported. For instance one person thought a job was worth including if it was at least for a couple of months. Among the jobs that were not counted, there were short term and temporary jobs in warehouses. Another respondent did not include one-time gigs because "they were side jobs, temporary jobs, not my real job." Another respondent did not include a two-day seasonal job. Another respondent did not include gigs such as dog sitting or dog walking which were "cash in hand" jobs. He thought of paid jobs as ones with official paperwork, where you have interviewed and they have your official identification, information, etc. On the opposite extreme, a respondent counted gigs lasting just a day or so as individual jobs. A respondent did not count gigs simply because he did not think about them until the probing.

One respondent was unsure about including her current job because the work hours are inconsistent. She did count it. One respondent said that she would count working at two different bookstores that were part of a chain as two separate jobs. Finally, a respondent thought that her retail job should not be included, because she thought the question was asking ONLY about internships, co-op jobs, self-employment, and work-study jobs. She thought so even after re-reading it. She did not see those as examples, but as a list to choose from.

Almost all of the respondents considered the sandwich shop scenario as one single shop because it was "in the same place," "for the same employer," "at the same company," "same boss and company, just changing positions," or "because it is a promotion." Only three did not follow the instruction and considered it as two jobs, saying that if the change in position also brought about a pay increase, then it is two jobs.

Usability: No issues except three cases reported that the system was slow.

C11

Item-Specific Probes:

• PC11a: IF YES: Tell me about the times you were not working?

• PC11b: What were the reasons that you were not working during this time?

Most respondents who were asked this question clicked ‘No’ and therefore were not probed. Among those who clicked ‘Yes,’ there was a variety of situations. One respondent took three weeks off to work at a different job, while remaining committed to the first job. Another respondent said the work at her job was irregular and they called her in only when there was work available. Another respondent did not work at various times due to the coaching season conflicting with her schedule.

Finally, a respondent reported not working as a temp while he is in school.

Usability: No issues except for one consistently slow case.

C14

Item-Specific Probes:

• PC14a: What were you thinking as you answered this question?

• PC14b: What do you think of as working “regularly?”

Respondents had no particular difficulties with this question. One respondent, however, was only thinking about the present when she clicked ‘No,’ but in probing she said she had worked while going to trade school.

'Working regularly' was defined by many in relation to the work schedule: "a consistent weekly schedule for the duration of the job," "having a schedule for work," "being on the schedule every week, no breaks in time," and "having a schedule each week outlining hours you expected to work." Others thought in terms of "working every week," "at least once a week," "either part time or full time," or "a lot of hours."

Usability: No issues observed, except for one consistently slow case.

C18

Item-Specific Probes:

• PC18a: How easy or difficult was it to choose one response?

• PC18b: Tell me how you selected your answer?

• PC18c: What is meant by “A career position?”

• PC18d: What is meant by “A way to explore a career option?”

• PC19e: What is the difference between options 1 & 2?

About ten respondents found it difficult to choose just one response. One of them suggested allowing multiple responses or adding 'all of the above.'

Respondents' narratives supported their response choice, indicating the response categories are working as intended.

Respondents had no difficulty defining the first two response options and distinguishing between 'a career position' and 'a way to explore a career option.' They defined it in different terms but generally showing a good understanding of both terms. For 'a career position' they said: "you are sure," "something I have the intention of doing for the rest of my life," "something that leads to something that I'm going to do for the rest of my life or for a really long time...," "the position you are in that is your career," "a career position is more permanent," one that might turn into a career for you," "you know what you want to do and you are doing it," "a job that is in your career field." For 'a way to explore a career option,' they explained it as "trying to see if you like it," "something I am trying out to see if that's what I want to do for the rest of my life," "a job where you are working to find out what is involved in the job," "that you are looking to see if you like it and may not be a career for you," "that you are testing something to see if you like it," "is a job in a field that you might be considering as another alternative choice for yourself," "that you are trying it out and if it does not become a career, it's a way to pay the bills" "seeing if that career is a fit for you." working a job that you think you might be interested in but do not want to commit to yet," "when you don't know where you want to go/work so you are looking at different jobs to see what you might want," and "a way to get exposure, not gonna stay there forever."

Usability:

One respondent wanted to click two options and was initially confused although was able to respond.

C32

Item-Specific Probes:

• PC32a: Tell me about your choice of response.

Among those who answered this question, the vast majority thought of themselves as 'a student who worked.' Some based that answer on the fact that were in school full-time when they looked for a part-time job. One respondent explained that she works part-time while attending school. She said that the “woman I take care of is a snowbird” so she does not work for her for some time during the winter.

Others spoke of priorities or what started first: "I prioritized school over the job.” Another said she came to town primarily to attend school but was also working. A respondent said she puts school first and would ask for time off from work if she needed it because of school. Another was thinking about work study and how in those jobs you are a student first. Another one explained: "I was a full time student and part time worker. So I am a student first and worker second." Another respondent explained he was in school before he had a job, so he considers himself a student with a job.

Another respondent clicked ‘an employee who decided to enroll in school’, basing his answer on the number of hours devoted to each activity: when he was in school he was working many more hours than he was spending on school.

Finally, one respondent said she answered randomly "A student who worked." She did not know which response would best apply to her. She wondered if there would be someone who would do both (work and go to school) equally.

Usability: No issues observed

C31

Item-Specific Probes:

• PC31a: What were you thinking of when you answered this question?

• PC31b: What do you think is meant by “assistance from a high school/college staff member?”

• PC31c: What type of person were you thinking of as “staff members” who might offer this kind of assistance?

• PC31d: What do you think they mean by a 'high school arranged program' and a 'program arranged by a college'?

Only three respondents answered these probes. None of them had ever benefitted from such a job assistance program.

None of the three was clear about what a 'high school arranged program,' or 'program arranged by a college' might be like. One guessed: "Maybe the high school has companies they work with the hire young adults directly and work with your school schedule possibly?" Another thought that 'high school arranged program' perhaps was something related to a GED program, while the third one thought it meant that a teacher might help you but said that he did not actually know what the terms meant.

One respondent interpreted 'assistance from a high school staff member' as meaning that "anyone who works at the high school helped you get the job." Another defined 'staff members' as "teachers, anyone who works at school, counselors, deans."

Usability: No issues observed

C21

Item-Specific Probes:

• PC21a: Tell me how you selected your answer?

• PC21b: How certain were you of the response you chose?

• PC21c: Were any of these response options unclear or confusing?

• PC21d: What is the difference between options 2 & 3 (for-profit vs. nonprofit)? Can you give me one or two examples?

Only three respondents answered the probes.

Two of the three respondents were confused by the response choices, primarily because they were not sure what each one meant exactly.

The first one was trying to categorize his job at a major department store and chose 'Another type of organization or business.' He was only 50/50 confident about his answer. He did not know what a federal, state or local government meant. When asked to define 'for-profit company,' he realized he worked for one and changed his answer, since "the goal of a for-profit company is to make money." When asked what a 'not-for-profit business' is, he struggled to come up with a response and guessed a “park district.”

A second respondent also answered 'Another type of organization or business' because he was not sure what the choices actually meant. He could not explain the difference between non- and for-profit companies.

Usability: No issues observed

C38

Item-Specific Probes:

• PC38a: What do you think they mean by “unemployment compensation?”

• PC38b: IF YES: Tell me about the time when you received unemployment compensation. What were the circumstances?

Three respondents were probed on this question. Two clicked ‘No,’ and one clicked ‘Yes.’ The latter said he had received unemployment when he was laid off from McDonald's, and described it as "I got what they owed me." In probing he explained that he had been laid off and was thus eligible to receive unemployment.

The other two respondents did not know clearly what ‘Unemployment Compensation’ exactly means. One said it was like a "disability check." The other one thought it means "when a job is over you get paid for it…from the years or months you worked, the taxes or something. You get paid for having worked."

Usability: No issues observed

C40

Item-Specific Probes:

• PC40a: What was the hardest part of answering this question? What made it hard?

• PC40b: IF NOT VOLUNTEERED: What about listing your job duties? How easy was that?

Several respondents found the question hard to answer simply because they do not really know what they want to be doing at age 30. Five selected 'You don't know.' One respondent said the hardest part of answering this question was thinking about age 30 because it was so close but so far away. "That's five years after I graduate and there is always the possibility I'll change my major." He was not sure what his duties would be either. Another respondent said that the question was hard in that she wants to learn other things related to working in a salon but the question was "as things stand now" so she "answered realistically.”

Two others said they did not know, but they provided an answer. The first one had a very hard time picking an answer to this question because he is unsure of what he wants to do at age 30. He entered "entrepreneur" and when the interviewer pointed out that he could have answered ‘you don't know,’ he said he did not see it but wanted to keep what he wrote in even though he did not know the specifics. The other said "I'm still trying to figure that out." but entered both a job and several duties.

In the remaining cases, some had difficulty coming up with the right job category among those offered by the program, while others were not sure of what job activities to enter.

One respondent struggled with this question: "What if I don't have no specifics? ... Hopefully I can own my own company...What if I don't want any job? I just want to own my own business so I don't have to worry about working anymore." He first entered that he wanted to own his own business in music and fashion, then selected in the dropdown 'producer' because "I'd like to do that too."

Other examples of the difficulties experienced included:

• A respondent was not sure what to enter for job duties but, after some thinking, he was finally able to come up with something that she was comfortable with.

• A respondent is in the music industry and had difficulty explaining how his future job would incorporate both playing music and the business side of music. It was easier for him to describe job duties than finding a job title.

• A respondent thought answering was hard because she wanted to come up with terms that would bring up the right kind of option to pick from (she had learned this mattered from previous write-in questions throughout the survey).

• A respondent answered "physician therapist," but was not sure what to answer for ‘Job Duties;’ she pressed ‘Enter’ and that confirmed what she had written as description of duties.

Usability: No issues observed

COMMUNITY SECTION

D28

Item-Specific Probes:

• PD28a: What does “marital status” mean in this question?

• PD28b: Who were you thinking of when you answered this question? How did you choose to think about that person/people?

• PD28C: In your own words, what is the question asking you to do if your parents are divorced when answering the question? What if your parents were never married? Would you answer differently?

A few respondents did not correctly interpret the intent of the question or what they were expected to do. A respondent had difficulty understanding the question and had to reread several times before answering.

Regarding the instruction about divorced parents, one said that if his parents had been divorced, he would have considered his two biological parents' relationship to each other when answering the question. Another one stated confusion about how to answer if her parents were divorced but this did not apply to her personally. They did not seem to clearly understand that the question was asking for their parents' current status, not just if they were married to each other.

Generally respondents understood 'marital status' as referring to the marriage or relationship status of the parents to each other, or "are your parents husband and wife?"

Usability: No issues observed

D35

Item-Specific Probes:

• PD35a: Were you familiar with <ASK FOR EACH TYPE OF BENEFIT>?

• PD35b: IF YES: Tell me about getting <BENEFIT>? Who received it?

• PD35c: IF YES: When did you begin receiving that benefit? Do you currently receive it?

About twelve respondents had at least heard of all the benefits listed in this question. Another seven had at least hard of SNAP, WIC and the Free & Reduced Lunch programs. SSI and TANF were less known to this population.

Three reported someone in their household receiving some benefits (2 said “SNAP,” and one “Free & Reduced Lunch).”

Usability: No issues observed

D36A

Item-Specific Probes:

• PD36Aa: For a, you said you [RESPONSE] worried about having enough money for regular expenses. When thinking about that, how did you come up with your answers?

• PD36Ab: Did you think about the whole year, or more recently?

• PD36Ac: If you had to give a number for how many times you worried about having enough money for regular expenses in the last 12 months, how many times would you say?

• PD36Ad: IF MORE THAN NEVER FOR C: About how many times have you carried a balance on a credit card? When you answered the question how far back did you think?

For this question, the 12-month reference period worked well. Respondents seemed to focus on the past year without difficulty. Only three answered thinking of a shorter period, between one and 6 months.

Three respondents who never have to worry about money explained their reasons, including having a trust fund, finding it easy to find a job, and "knowing how to stretch my money."

Among those answering 'sometimes,' their narratives suggest that they do not have serious financial pressures but once in a while they have some concerns about things like transportation or travel.

Among those answering 'often,’ there were concerns about paying for housing (living at school over the summer), and paying for books and other expenses while at school without family help. The frequency of worry was "100 days," "at least 20 times," and "every day (of the month), probably 30," and "6 to 8 times."

Those who worry very often painted a picture of rather constant worry. They answered the probe on how many times they worry, with numbers such as "200 times probably; almost every other day," "300," "at least once daily,1000 times," "I think about it every day, 365 times," "500," "365, every day I guess." Only two or three expressed 'very often' in much smaller numbers, ranging around 8 to 10 times in the past year.

Two respondents objected to the question asking number of times, thinking it was difficult to answer that way.

Only three respondents expressed uncertainty on how credit card balances work. Two were unsure about the meaning of "carrying a balance" on a credit card. One of them thought of 'balance' as in a debit card, which he was more familiar with. A third one answered “part b” thinking of his debit card and interpreting the question as asking if he had any money available in the debit account.

Five respondents who answered 'sometimes,' 'often,' or 'very often' were thinking about different periods of time when replying. One thought about the current calendar year, i.e. since January. Another thought back "about 6 months." The other three thought of the past 12 months as instructed in the question.

While they mostly answered with number of times, one respondent answered she had carried a balance "at least 300 days."

Usability: No issues observed

D36B

Item-Specific Probes:

• PD36Bb: What do you think they mean by “living expenses” in this question? Can you give me some examples?

• PD36Bc: What do you think they mean by “investigated” in this question? Can you think of some examples of what types of activities they could mean?

Respondents had no observable difficulties interpreting or answering this question. The definitions they provided for 'living expenses' included the types of expenses that would be expected, but varied depending on the kinds of things the respondents are responsible for in their daily lives (given their age and differential dependence on their parents, they do not all have to worry about the same things. The following are illustrative of the responses elicited: "room and board, tuition fees, and personal expenses (toiletries, food, transportation)," "gas, light, air conditioning," "food and rent," "rent, transportation, food, phone, cable bill, lights, garbage, water," "rent, clothing, food, water bills," "day to day things like food, rent, clothes," "transportation, food, clothes," "daily stuff, food," "clothes, personal hygiene items," "rent, housing, food, transportation," "just bills," "food, rent, utilities," "food, rent, toilet paper, etc.," "rent and utilities," "food, bills, rent," "food, necessary clothing, and necessary bills like utilities."

The concept of 'investigating' was clear to most respondents. One respondent said she did not know what it meant, and another did not at first but was able to figure it from context. Another respondent was surprised at hearing the term in the probe because she did not recall the word being in the question.

The vast majority defined it as "looking into" or "researching." Examples respondents gave included:

• "looking for loans or other support but not having gone through with it"

• "see if you could pick up a shift at work or more hours available and talk to the manager or something"

• " looked into or asked people to borrow money from them"

• "inquired about, like getting more hours".

• "search job ads"

• "searching for a way to get a loan"

Usability: No issues observed

D39

Item-Specific Probes:

• PD39a: Have you ever taken out any loan for your college education? IF YES: What type of loan it that? Why did you count/not count that loan when answering the question?

• PD39b: Other than private loans from private lenders, what other types of loans can you get for paying for college?

Most respondents answering these probes said that they had taken out federal loans, but no private loans. A few were slightly uncertain, in particular those with PLUS loans, but they checked the help text to make sure. One respondent knew virtually nothing about loans, except that they need to be paid back.

One respondent thought that he did not have a loan but rather got a reduced rate on his classes. However, these costs had not yet been paid at the time of the interview and he was unclear if this was a loan or not; he thought it was not, because he did not "go through a loan process."

Other types of loans they mentioned included federal loans both subsidized and unsubsidized, "Stanford loans," "Pell Grants," "Parent PLUS loans," "loans through the school," loans from family," and "a direct sponsor from a [music] studio or my mentors."

Usability: No issues observed

D45

Item-Specific Probes:

• PD45a: What loans were you thinking of when you answered this question?

From most responses to the probes, it appeared that the respondents were thinking of any and all money they already owed. Since a majority of the respondents were either close to completing or had already completed their undergraduate education, this is likely a good projected total amount.

In answering, most respondents were thinking of the types of loans they currently have (or had at graduation time; one respondent has borrowed for her graduate education but had no previous loans for undergraduate). Respondents included federal and other student loans, and appeared comfortable in selecting a range.

One respondent answered 'Don't know' because he did not see an option for 'no loans.' He did not realize a value of zero would be included in the range '$5000 or less.' Another respondent who had no student loans of any kind skipped the question without answering.

Only one respondent was explicit about including future debt. She was thinking of "future loans" and answered for the new debt she thinks she might acquire once she goes back to get her associate's degree (once she pays off the loans she has now).

Usability: No issues observed

D69

Item-Specific Probes:

• PD69a: In your own words what is this question asking?

• PD69b: IF NECESSARY: What do you think they mean by 'your sex assigned at birth’?

• PD69c: How do you think the doctor decides what to put in the birth certificate?

• PD69d: Can someone’s sex be left blank or unassigned on their birth certificate in the state in which you were born? If so, under what circumstances?

NOTE: D69 has been revised from the second follow-up field test instrument’s version of this question. The parenthetical “(what the doctor put on your birth certificate)” has been removed and response options 3 and 4 have been added. This cognitive interview will test a version that is presented in the Williams Institute Report (Conron et al., 2014).

This question posed no difficulties for respondents. They answered without trouble or hesitation. They understood the question as intended.

When probed about the meaning of 'your sex assigned at birth,' respondents answered with replies such as:

• "Birth assigned at birth is based on the genitals"

• "The doctor looks at your genitalia and decided if you were male or female based on if you have a penis or a vagina."

• "When I popped out of my mom they said, 'It's a boy!' because of the genitals of the baby, of me".

• "Are you male or female? Do you have a male organ or a female organ?"

• "The doctor decides gender by looking at 'your private parts'."

• "Was I born with a vagina or a penis?"

• "How many chromosomes you have and based on genitalia."

• "The doctor decides based on the reproductive organs of the baby."

• "The doctor assigns baby's sex based on your private areas."

• "What gender you were at birth…a little girl or a little boy, which is decided by doing science plus appearance." The respondent explained that 'doing science' was about genetics and chromosomes testing.

• "Are you male or female or somewhere in between, based on genitals?"

• "Your legal sex based on visual investigation at birth."

• "Possibly a chemical test could be done if there was any question."

• "What is your body defined as, pure anatomy."

About half of the respondents did not think that sex could be left blank on a birth certificate and almost another half was not sure. Two respondents thought that if a baby is born with a mixture of male and female organs, or two sets of organs, maybe the sex can be left blank on the certificate.

Usability: No issues observed

D70

Item-Specific Probes:

• PD70a: What do you think of this definition?: "Your gender is how you feel inside and can be the same or different than your biological or birth sex." Is this something you already knew?

• PD70b: IF NOT: Does it make sense to you? Tell me why/why not.

• PD70c: What does transgender mean to you?

• PD70d: How easily did you find one or more responses to choose? Tell me more about that.

• PD70e: Did you feel like there were any options missing here? Would you add anything?

• SHOW SHOWCARD

• PD70f: Now look at this question. (SHOWCARD) Do you prefer one version of the question? Which do you prefer and why?

NOTE: The order of D69 and D70 was varied in the cognitive interviews to test order effects as recommended by the Williams Institute Report (Conron et al., 2014).

The vast majority of respondents understood the question as intended, indicating they were previously familiar with the definition of gender provided in the question and that it made sense to them. Only one respondent said he was not familiar with that definition but took no issue, another one had heard it before but still found it odd, while two others simply did not agree with it. One said "I don't agree with that definition. I think your gender is what sex you are when you are born…I don’t think how you feel inside is relative to your biological or birth sex. Once you are born at birth you are that gender even if you decide to change it, you are always regarded as what you are born with." "I don't go with this whole whatever you are in your head is what you are on paper. What you are on paper is what you are physically;" adding that he had been raised to believe that his gender is what his body is and what his organs are.

Respondents were generally fine with the reference to transgender included in the gender definition in the question. They provided their own definitions of transgender that were generally in line with the one in the question. These included:

• "Gender doesn't agree with the sex assigned at birth.”

• "What they identify as is not the same as the gender they were assigned at birth.”

• "Someone who has taken steps to move away from the sex they were assigned at birth. Whether they just wear different clothes or got surgery."

• "When someone does something to their body to change it."

• "Transgender is one person becomes the opposite of what they are a guy wants to become a female…"

• "When you change from the sex you are assigned to the sex you feel or want to be.”

• "When someone is born as one gender and they don't identify as that gender.”

• Transgender means "the gray area…you're male or female or you're in the gray area…someone who was born one way and wants to switch, take the hormones or do the surgery.”

• "Transgender means you identify with a sex that is not your birth sex."

• "A person who dresses as a sex they are not.”

• "When you feel like you're not the sex that the doctor called you.”

• "Transgender = changing to opposite sex of what you are by appearance.”

• "The feeling someone has about being born in the wrong body."

• "Transgender is when your sex is one thing legally but the way you dress and have people reference you, the way you want to be seen and the reality you construct is a different sex."

A few respondents suggested additions to the response choices. One felt the options offered could be improved by adding “non-gender conforming” and “genderfluid” or “genderqueer,” but added these could all be included under 'something else'. Another suggested “non-binary” as an option for those who do not identify as either male or female.

Two respondents did not choose either male or female but neither chose transgender. One of them was screened in recruiting as 'gender non-conforming' and chose 'something else' and felt it was a good option for him. The other one chose 'female' during the survey, but in probing she added 'not sure'. She said she did not always conform to the binary.

Respondents were shown an alternate way of presenting this question on the following Showcard:

What is your current gender identity?

(Please choose all that apply)

 Male

 Female

 Transgender Male/Transgender Man

 Transgender Female/Transgender Woman

 Genderqueer/Gender Non-conforming

 Different Identity ____________

A good number of respondents were not familiar with the terms 'genderqueer’ and/or 'gender non-conforming.' They also preferred 'different identity' over 'something else,' and they liked the option to provide a write-in answer (one respondent mentioned that the original version response choice ‘Something else’ seemed to suggest the survey had no interest in learning what else the person considers him/herself.

Others who were familiar with all the response categories offered, had a very strong preference for the Showcard version of the question, primarily because it lists 'genderqueer' and 'gender non-conforming' as options, or "because it has more variety of choices." One respondent liked the level of specificity in the original question response options of "transgender, male-to-female" and "transgender, female-to-male."

A good number had a marked preference for the simplicity and directness of the Showcard question stem. A couple of respondents remarked on the word 'current' in the showcard version and indicated they liked it because it gives the sense that gender identity can indeed change over time. On the other hand, many liked the presence of the definition. Some were also mindful that their own preference might not be what persons in different gender communities might prefer to

see.

A couple of respondents found it difficult to compare the two versions and indicate which they preferred and why. Several others indicated like and dislikes in both question formats, and proposing a combination of both. For example, a respondent liked the question as worded in the original question but would add the ‘genderqueer’ option from the Showcard and remove/replace the ‘something else’ and ‘you are not sure’ options from the original question as well. She liked the question ‘What is your current gender identity’ but would add the definition if the Showcard version were used. Another respondent preferred the response options on the Showcard, especially the write-in option, but liked best the definition from the original question.

The respondent who was screened as ‘gender non-conforming’ said he preferred the original question because he was able to find 'something else,' which reflects how he feel inside. Another respondent who answered 'female' but in probing self-defined as gender non-conforming, liked the Showcard because "it gives 'more agency' to the people who are gender nonconforming rather than not sure."

Usability: No issues observed

Testing for Question Order Effects of Questions 69 and 70

Respondents were randomly assigned to one of two question order versions. Half of the respondents (n=20) were asked question 69 first, followed by question 70 (and its Showcard alternative) and the other half (n=20) were asked first question 70 and the Showcard alternative, followed by question 69.

The most notable difference was a preference for the Showcard version among those respondents who had not yet been asked question 69. There was no evident difference between those who stated they already knew the definition in question 70 and those who stated that they preferred having the definition.

Order of Admin:	69/70		70/69
Liked parts of each*	3	16.7%	4	21.1%
Original	3	16.7%	7	36.8%
Showcard	12	66.7%	8	42.1%

*Not all respondents indicated a preference

In addition, two respondents who received question 70 first appeared to see it as a combined sex/gender question whereas no respondents who received question 69 first mentioned this. The first one thought of both gender expression and biological sex when answering question 70.

The second one, a male respondent, when asked about his Showcard preference said he preferred the question he saw on the screen because "it incorporates your gender and sexual orientation both." He saw the response option 'you are not sure' as asking about someone's sexual orientation, not their gender identity. He stated that combining the current version and the Showcard would make it "a more in-depth question," because he saw the original version as including sexual orientation.

D71

Item-Specific Probes:

• PD71a: Tell me how you choose your answer.

• PD71b: What does the term “homosexual” mean to you?

• PD71c: What about the term “heterosexual”? What does that mean to you?

• PD71d: And “asexual”?

All respondents were able to identify a response. One respondent had reported 'straight' in the screener and in this question answered ‘something else,’ “after long days of meditation”. Another answered 'don’t know,' after choosing 'other' in the screening; this respondent said she often will choose 'bisexual' but since she hasn’t had a relationship with a woman she was not sure.

One person who identified as something else and one respondent who identified as bisexual noted that pansexual should be included.

Nineteen respondents had no issues answering this question and were able to define homosexual, heterosexual and asexual without issue.

Two respondents didn’t care for the terms, one wishing that pansexual was an option and the other noting that she doesn’t care for the terms heterosexual and homosexual and prefers using the terms gay and straight.

Three respondents who identified as straight were unsure what the term ‘heterosexual’ meant, one R who knew he was straight googled the term before responding, one noted that because it was in the same list as ‘straight’ it must mean straight.

Sixteen respondents were unsure of the term asexual although as one respondent noted “in this question if someone is asexual they will know what it means and pick it so it doesn't affect me". Another respondent noted that he was unfamiliar with the term in relation to humans but in animals meant reproducing within yourself. Guesses included: “without a specific sex,” “not liking sex,” “and loving just oneself.” One respondent who had not known the term heterosexual also thought asexual might also mean straight another wondered if it meant the same thing as pansexual. No respondent identified as asexual and the 16 respondents who were unsure of its meaning included those who identified as straight, gay and bisexuals.

Usability: Other than a couple slow systems, there were no reported usability issues.

1 LGBTQ: Lesbian, Gay, Bisexual, Transgender, Questioning/Queer

2