Appendix D Cognitive Testing Report

Qualitative Research Findings for the B&B:08/18 Cognitive Interviews: Field Test For The Baccalaureate And Beyond Longitudinal Study

Prepared for: RTI ON BEHALF OF NCES

Prepared by: SHUGOLL RESEARCH 7475 Wisconsin Avenue Suite 200 Bethesda, Maryland 20814 301-656-0310 w ww.shugollresearch.com

December 2016

TABLE OF CONTENTS

Section		Page
1.0	Background and Research Methodology	3
2.0	Research Objectives	7
3.0	Conclusions and Recommendations	8
4.0	Summary of Final Debriefing Questions	10
5.0	Detailed Findings by Probe	12

Supporting Materials

S-1

1.0 BACKGROUND AND RESEARCH METHODOLOGY

Background And Purpose

• RTI, on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, contracted with Shugoll Research to conduct qualitative in-depth interviews with respondents who graduated with their bachelor’s degree between July 1, 2006 and June 30, 2007 to obtain feedback on some of the questions that will be used in the Baccalaureate and Beyond Longitudinal Study. Feedback from the research will be used to improve the interview field test instrument.

Usability Methodology

• During the cognitive interviews, usability testing was conducted by allowing respondents to complete the survey either on a computer (PC or laptop) or on a mobile device (smartphone or tablet). This allowed respondents to provide feedback on versions of the questions formatted for different devices.

• Shugoll Research utilized a remote cognitive interviewing/usability methodology where respondents used their own computer or mobile device to complete the NCES survey. Remote testing is convenient/flexible for respondents because they can schedule the session to fit their needs and can participate from their home, work or other location. It allows respondents to use the questionnaire in a real-world environment rather than in a lab setting. It is also slightly more cost efficient than conducting in-person interviews in a research facility.

• Shugoll Research’s web-based remote interviewing/usability solution includes webcam technology, streaming video and an audio connection to provide real-time, face-to-face interaction between the respondent and interviewer/facilitator via a shared desktop on their computer. Tech checks were performed with respondents prior to each interview to minimize technological difficulty during the sessions.

• Approximately half of the respondents used their computer to complete the survey, and approximately half used their mobile device (either a tablet or smartphone). Respondents recruited for usability testing on a mobile device were sent a document camera so that screen shots of how the respondent proceeded through the mobile version of the survey were visible to the respondent, facilitator and client observers.

• At the end of each interview, the facilitator conducted a debrief to obtain feedback on how easy or difficult it was to use the computer or mobile device to answer the questions, and to ascertain ways the questions could be modified to improve mobile usability.

• Clients were able to log-in and watch all interviews remotely.

Cognitive Interview Methodology

• A total of 30, 90-minute, in-depth, virtual interviews were conducted between September 26^th and November 29^th, 2016 with respondents who completed their bachelor’s degree between July 1, 2006 and June 20, 2007. The 30 interviews were broken out into the following segments:

• 4 interviews with teachers (could have had any major in undergrad)

• 4 interviews with STEM majors (must never have been a teacher)

• 22 interviews with those who majored in “other” subject areas in undergrad (must never have been a teacher, must not be a STEM major or an education major)

• Respondents were recruited by Shugoll Research using its consumer database of individuals in the Washington, DC area.

• Shugoll Research also used a variety of other recruiting methods in order to enhance its consumer database. These methods include networking and referrals; e-blasting from the database and online screening; and a variety of social media outlets that include Facebook, Twitter, and LinkedIn.

• Shugoll Research has been actively developing its database of over 100,000 people in the Washington, DC area since the mid-1970’s. The database was developed originally through cold-calling and recruiting from mall-intercept locations and has continued to grow both through cold-calling from directory listings as well as through referrals, word-of-mouth, and social media. The database contains demographic information such as age, gender, and employment/education so that the contact information can be sorted based on the needs of each individual study performed by Shugoll. For these cognitive interviews, the database was sorted first by level of education, and then by other demographic characteristics (for example, age and gender) to balance the targeted respondent group.

• Respondents participating in this study had to meet the following recruiting specifications:

• Must have attended a college or university between July 1^st, 2006 and June 30^th, 2007.

• Must have completed the requirements for their bachelor’s degree between July 1^st, 2006 and June 30^th, 2007.

• Must have been employed at some point since graduated in 2006 or 2007.

• Must be over 18 years of age

• Respondents were recruited to represent a mix of undergraduate institutions and demographics such as gender and race.

R ecruiting Disposition

Limitations

• A qualitative research methodology seeks to develop direction rather than quantitatively precise or absolute measures. The limited number of respondents involved in this type of research means the results should be regarded as directional in nature, and they should be used to generate hypotheses for decision making.

• The non-statistical nature of qualitative research means the results cannot be generalized to the population under study with a known level of statistical precision.

2. RESEARCH OBJECTIVES

The specific study objectives are to:

• Obtain reactions to the mobile-optimized version of the survey and determine how easy or difficult it is to use a computer, tablet and mobile phone to answer the questions

• Identify questions that are confusing or ambiguous

• Identify specific terminology that is unfamiliar

• Understand how respondents interpret some of the questions in their own words

• Understand how respondents determine their answers to some of the questions

2. CONCLUSIONS AND RECOMMENDATIONS

• In general, respondents say they would be more cooperative when taking the survey if they had more information about the purpose of the survey and how the information is going to be used. RTI might consider providing some of this information to all respondents so they know why they’re being asked to respond to so many questions and provide such detailed information about themselves.

• Because the employment section of the survey is long and detailed, several suggest that RTI include instructions at the beginning of the survey for respondents to have their resumes handy so that they can quickly look up some of the information that is needed.

• From a usability standpoint, the survey is relatively simple and easy to complete.

• However, a few respondents taking the survey on their mobile devices say that some of the questions involve too much scrolling.

• In addition, those taking the survey on their mobile phones have trouble with some of the questions where they have to select options from an extensive drop-down list. There are times where they can see all the choices in the drop-down, but they are unable to actually click on the correct response because their phone keyboard gets in the way. This is especially true for the “occupation” and “industry” questions.

• RTI’s IT and programming teams should address the usability issues so these particular questions are more user-friendly for mobile survey takers.

• In general, the terminology used in the survey is clear and understandable.

• There are a few questions that are confusing or difficult for respondents to answer.

1. The “occupation” and “industry” questions are quite challenging for most respondents. Very few are able to find their exact occupation or industry in either version of the question, and instead get frustrated because they have to “settle” for a response they feel is close, but not completely accurate. RTI should consider modifying this section of the survey to make it more user-friendly. Consider the following:

• Allowing an open text field and accepting all answers typed by respondents

• Alphabetizing all potential answer choices so it’s easier for respondents to find what they are looking for

• Adding more options to the radio button version of the “industry” question

2. Many of the “employment” questions are difficult for respondents in non-traditional job settings to answer (e.g., teachers, contract workers, those in sales or who work shift work, etc.). RTI might consider either a different set of questions for people in non-traditional jobs, or it might include more instructions for how people in non-traditional jobs can appropriately answer these types of questions.

• Many feel uncomfortable answering the sexual orientation questions of the survey and say they would have “quit” or “abandoned” the survey if they had actually been taking it and had come across these questions. They don’t understand what the sexual orientation questions have to do with their employment and education. RTI should consider either eliminating these questions all together, or providing a brief explanation as to why this information is important to collect.

• In addition, the few who were asked the detailed questions about their dependents also seemed to feel uncomfortable providing this level of detail without knowing why and how the information would be used.

• For all questions using a scale, many suggest that the scale go from positive to negative as opposed to negative to positive.

2. SUMMARY OF FINAL DEBRIEFING QUESTIONS

• Most say the survey questions are not difficult to answer and that the survey is very user friendly.

• A few believe the survey is too long and too detailed.

• Some are offended by the sexual orientation/preference questions. Many agree that they do not seem to pertain to the overall subject matter of the survey – employment and education. This series would definitely cause some people to abandon the survey.

• Like the 2015 study, many agree that they would be more inclined to complete the survey if they knew more about the purpose of it and how their information will be used.

• Like the 2015 study, very few notice the navigation help aides.

• Those taking the survey on their mobile phone point out that some of the questions require a lot of scrolling, which can be frustrating. The “industry” and “occupation” questions also have some usability issues for those taking the survey on their mobile phones (see findings for Probes 13 and 14 for specifics).

“It’s not difficult, it’s just a lot of information to provide. Maybe they should tell us to have our resume nearby and our salary history before we begin. This would help people be more likely to finish it if you don’t have to stop numerous times and get up to search for the detailed employment information.” (Teacher, Tablet)

“I’m assuming people would be told the purpose of the survey. It’s very personal information, so I don’t know if I’d fill it out if I didn’t know the purpose.” (Teacher, Tablet)

“On my phone I had to scroll a lot. I preferred the questions that didn’t require as much scrolling.” (Other, Mobile Phone)

“It would be good to know the end goal of the survey. It would make me more open to answering some of the questions. And 30 minutes is a really long time to be taking a survey. Will there be an incentive?” (Other, Tablet)

“The sexual orientation questions could make people very uncomfortable, and I didn’t understand the correlation between my sexual preferences and education. How is this relevant to the survey?” (Other, Computer)

“I’d need to know more about why they’re requesting this personal information. Why do they need to know so much about my personal life? It seems a bit invasive.” (Other, Computer)

“The questions are easy to answer, but there was a lot of information I didn’t want to provide. I would be turned off by all these questions and would quit the survey.” (Other, Computer)

2. DETAILED FINDINGS BY PROBE

1. B18DSECURITY

• Many respondents say they’ve seen security questions like this before.

• However, in this case, many don’t understand why this information is needed for this survey and it makes some skeptical or nervous. In particular, providing the last four digits of their social security number makes some respondents uncomfortable.

• Some would rather the survey just “confirm” the last four digits of their social security number as opposed to asking respondents to provide it themselves.

• Verify the street name and phone number does not seem to be an issue or cause concern.

• Not everyone understands that the intent of these questions is to verify that the respondent is who they say they are before asking them to provide additional personal information during the survey. This concept seems to go right over their heads, perhaps because there’s too much introductory text. RTI probably should provide better clarification (perhaps using bold or underlined text) of the purpose of verifying this personal information at the start of the survey and assure respondents that their information is not going to be used for any other purposes.

“I am ok with the phone number part of the question because they can get that information easily. I am not really comfortable providing my social security number.” (Teacher, Tablet)

“I am fine with this section; I think it is typical to any background check.” (STEM, Computer)

“It takes away my anonymity so that my survey responses will be linked directly to me. I don’t like that.” (Other, Mobile Phone)

“I am not sure about providing my social security number. I don’t really like providing it. I would like something explaining why they need my social security number and assuring me that they will not use it for any other purposes.” (Other, Tablet)

2. B18LINKEDIN

• Of respondents who have a LinkedIn account, they seem to be somewhat split on whether or not their information is up-to-date and comprehensive.

• Of the respondents who would not provide access to their LinkedIn account, the majority say it’s because they’re not particularly active on LinkedIn or their information is not up-to-date. In fact, a few say they would rather provide their resume because it’s more up-to-date and comprehensive.

• Some respondents say they are unwilling to provide LinkedIn access due to security concerns. They do not want to provide their username and password to a third party resource.

• One respondent is skeptical that connecting to LinkedIn would actually save time in the survey because her experience with connecting to outside technology is that it can often slow things down.

“No, I wouldn’t link my account because I don’t update my LinkedIn page. I if I did link to it, it might save time, but I have also had experiences were it takes longer to import the information because of formatting.” (Teacher, Tablet)

“My LinkedIn is not exactly up-to-date. My job has been the same for 8 years and my job does not require networking. I would say my profile is pretty general.” (STEM, Computer)

“I do use LinkedIn. My profile is current but not very detailed; it is not anywhere near as detailed as my resume.” (Other, Computer)

“I wouldn’t want to allow you access to my profile because I wouldn’t know when you could cease having access to my account.” (Other, Computer)

“My profile is very current and up-to-date. It is as comprehensive as possible – employment history, other tasks, accomplishments and extracurriculars are all on it.” (Other, Computer)

“I don’t feel safe giving my passwords to external sites. I wouldn’t want to share my account with a third party.” (Other, Computer)

3. B18RESUME

• Respondents are mixed as to whether or not their resumes are up-to-date and comprehensive.

• Many of those whose resumes are not up-to-date say it’s because they’re not currently looking for new employment.

• Many also articulate that they’ve dropped internships and first jobs after college that are not related to their field of interest or their career in order to limit their resume to one page.

• There are many who did not upload their resume to the RTI site for the following reasons:

• They were uncomfortable providing it without further explanation about what RTI planned to do with it.

• Some don’t want personal information like their phone number and email address available in cyberspace.

• Some simply forgot or were too busy.

• Some say they did not see the instructions to upload their resume.

“My resume includes everything in education that I have done after college because that is my current field.” (Teacher, Tablet)

“My resume is not current because I am not looking for a job right now.” (Teacher, Tablet)

“No, I wasn’t comfortable with uploading it. If they needed something they could ask for it during the session. I don’t want too much information like my phone number and email out there in the world.” (Other, Tablet)

“It doesn’t’ have my entire history, but it’s the majority of it. Based on the type of jobs I’m targeting now, I only include the relevant history. I want my resume to be clear and concise and only one page, so I deleted some jobs during and right after college.” (Other, Mobile Phone)

4. B18DEMPZIP01

• Depending on a person’s employment scenario, this question is either easy or difficult to answer.

• Those who work offsite or in a different location than company headquarters tend to have a more difficult time providing the zip code.

• Also, those who tend to work in a different city than where they live seem to have a harder time remembering their company zip code.

• Some people who have multiple jobs (e.g., work for a company and are self-employed) are confused about how to answer this question.

• Many say that the question is easy to answer, but that they did have to take extra time to think about it because they often do not receive mail at the office.

• People who work in a traditional office setting have the easiest time answering this question.

“I gave the zip code of my school only. It was easy to provide because I write the address often. This question could be difficult to answer if you wanted me to provide the main office address of the school district and not my school.” (Teacher, Tablet)

“There’s only one office for my company, but I’m not there much. I work all around the city so I had to think about it for a minute before providing my answer.” (Teacher, Tablet)

“My company has one other location in Maryland. It was pretty easy for me to answer.” (STEM, Tablet)

“My company has multiple urgent care centers and I work at different ones so it’s hard to know which zip code to enter.” (STEM, Computer)

“It is not the only location; we have offices all over the country and I couldn’t remember the zip code.” (Other, Computer)

“I didn’t know it. If I was not doing the interview I would’ve Googled it. Not everyone is going to fit into this employment mold. My company is based in Wisconsin and I don’t to go the headquarters.” (Other, Tablet)

“It was relatively easy, but it did take me a minute since I don’t usually have to enter the zip code for my office. Also, I don’t live in the same city where I work. If I lived there too I’d remember more easily.” (Other, Tablet)

5. B18DSTART01

• In general, respondents find this question easy to answer and are very confident in their responses.

• A small number of respondents find this question slightly difficult to answer particularly if there was a significant difference between their hire date and their start date.

“I had to think about it. I was hired in July and processed, but I didn’t start teaching until August, so I didn’t know how to answer.” (Teacher, Tablet)

“I had no trouble answering the question and I am very confident.” (STEM, Computer)

“I don’t know what to answer here. I interviewed and got offered the job in one month, but signed the contract and started working in a different month.” (Other, Mobile Phone)

“No, I did not have any trouble. I just started in January.” (Other, Computer)

6. B18DEND01

• Respondents find the layout of this question to be somewhat confusing. Many do not notice the check box for “currently employed.” They would prefer the survey directly ask them “are you still currently employed by this employer?” and if “yes,” they would skip ending month and year, and if “no,” they would fill out ending month and year.

• This question is particularly confusing for consultants who are currently employed, but are not staffed on a project at the time of the survey. Many believe this question is designed for those with a traditional job and does not necessarily fit people in non-traditional jobs.

“They should know whether you’re currently employed there – didn’t I already say that? I saw this question and I was like, ‘Duh, I’m currently employed there.’” (Teacher, Tablet)

“It was straight forward. I liked that it had the checkbox, this makes it simpler.” (STEM, Computer)

“This question assumes you’re no longer employed, which is weird. First they should ask if you’re still employed there.” (Other, Computer)

7. B18DWKCONT01

• For the most part, respondents do not have trouble answering this question.

• Types of examples they provide for unpaid breaks include:

• Maternity leave

• Unpaid medical leave

• Family medical emergency

• Bereavement

• Sabbaticals

• Unpaid vacation

• Leave of absence

• Contractual employees and some teachers have trouble answering this question because they are not paid year-round. Again, they say the question is designed for people with traditional jobs versus non-traditional jobs.

“It is asking if I took any breaks from my employment. Some examples are sabbaticals, maternity leave or unpaid vacation.” (Teacher, Tablet)

“My job doesn’t go in the summer since I’m a teacher so I’m not sure how to respond to this question. It’s not really a break though because you know you’re going back. After the summer I always get a new contract. So, I’m not really sure how to answer.” (Teacher, Tablet)

“It means any undocumented breaks in your employment history. Examples are leave without pay, illness, family medical emergency or maternity leave.” (STEM, Computer)

“It is asking if I took any time off during my employment history…anything that wasn’t paid time off.” (Other, Computer)

“I think of taking time to care for a family member with medical issues, extended vacation without pay, the office shuts down and you get laid off or your contract ends.” (Other, Mobile Phone)

8. B18DWKMONX01

• More and more, people are employed in unique situations and this question seems designed for those in traditional employment settings.

• As such, this question is difficult to answer for contractual employees because their jobs/projects start and end continuously. They are also uncertain what to mark for “partial months.”

• It is also confusing for those who are self-employed and sometimes take breaks from their company.

• For those who took maternity leave, it’s easier to answer the question using Version 2. If they are answering for any other purpose, it is more difficult for them to remember the timing for the breaks in their employment (Version 2) than the months when they were employed and they prefer Version 1.

“I preferred selecting the months I was employed over selecting the months I was not employed. The question was challenging because I was an independent teaching contractor so I did projects at different times. I am 90% sure I have my months right, but I might be a month off in some places.” (Teacher, Tablet)

“If I go back to work on August 29^th, do I click on the month of August? How do we handle partial months?” (Teacher, Tablet)

“I liked the first version that asked which months I was employed. I did not have trouble answering this question. I am 100% confident about when my unpaid break was.” (STEM, Tablet)

“I prefer the first version, but the overall question is complicated because I haven’t worked continuously. I am in promotions so I cannot remember every break. My job is contractual so some months I work and others I don’t.” (Other, Tablet)

“I thought this was a little overwhelming. I had to read it through several times to be able to answer. Do I count partial months? If I was employed for half the month it’s not clear exactly how to fill it out and they should have an instruction about that.” (Other, Computer)

9. B18DCURL01

• People do not have trouble answering this question.

• They think of a “career” as long-term growth within connected fields as opposed to “filler” jobs that are just to pay the bills. They also define a career as something they’re interested in and passionate about.

“A career is long-term employment or pursuit of a particular goal or field.” (Teacher, Tablet)

“Where I am working now is helping further my career goals. A career is a profession within your occupational industry.” (STEM, Tablet)

“A career is your passion, something you enjoy doing with potential to progress.” (Other, Computer)

“A career is a professional choice in a field or industry that is long-term. It can evolve and change, but it’s very intentional.” (Other, Tablet)

“It’s asking whether I see this as a position I took just to have a job and pay the bills, or if it’s a stepping stone to my next position.” (Other, Mobile Phone)

10. B18DEMPLOY201

• Again, this question is difficult to answer for those in non-traditional employment settings (e.g., contract work, teaching, sales, event planning, etc.) whose average work hours and salary vary dramatically from week to week.

• Some contract workers would select their salary “per day” if that was an option.

• Many say they just provided their “base salary” because their bonuses, tips and commission vary greatly and they don’t know how to provide this information.

• The “starting job” and “ending job” language is somewhat confusing. Those who are still currently employed with this particular company do not understand why it says “ending job” instead of “current.”

• Many have feedback about the format of the question:

• Many are confused by the format of this question, particularly the “Same as starting job” middle column. They don’t understand why this is there/what its purpose is.

• Some don’t like that you can’t use a decimal for partial hours worked per week.

• Some get frustrated because you can’t use a comma when providing your salary. There should be formatting instructions for how the survey will accept salary numbers.

• Some say providing ranges would make the question easier to answer.

• A small number of people do not feel comfortable providing their current salary information. They feel this information is private and confidential.

• There are many instances where people work 30-37.5 hours per week, but they consider this full-time employment.

“I don’t understand ‘ending job.’ I think it means when you ended your employment. It is not clear, especially because I am still employed. I think you should use words like ‘current’ and ‘when employment ended’ instead of ‘starting job’ and ‘ending job.’” (Teacher, Tablet)

“The middle column is weird. I’m not really sure what it’s meant to do and the placement is weird. Would it auto fill the ‘ending job’? I’m not sure. I think they could just have salary and hourly ranges and that would work better.” (Teacher, Tablet)

“I wish they had a per day option because I was a contract worker and that is how my contract was set up. I work 30 hours per week - it is a full-time job. I am paid for a full 8 hours even if I only worked ½ days.” (STEM, Tablet)

“I work at an urgent care center and my hours vary so much per week that it’s very hard to answer. For people who do shift work, this would be confusing to answer.” (STEM, Computer)

“It was confusing. At first I thought it was asking about a previous company because of the language “starting job” and “ending job.” It needs to be clearer about which job it is asking about.” (Other, Computer)

“The question is intrusive and I don’t want to answer it.” (Other, Computer)

“You could make it easier to read by deleing ‘same as starting job’ because it’s throwing me off. I struggled with the average hours per week. I’m exempt from overtime pay so I didn’t know how to answer. I really work more than 40 hours per week, but I’m only supposed to report 40 hours, so what would I put here?” (Other, Computer)

“I don’t like the salary question and I don’t want to answer it.” (Other, Tablet)

“My salary was never set because I was in sales. So I knew my base salary, but my total salary ranged so much because of bonuses and commission. This is very hard to answer.” (Other, Tablet)

“For contractual workers or part-time workers, there’s a lot of variance for those people. The question seems to be developed for full-time employment, not contractual or part-time. They need a separate box saying ‘it varies’ that people can check and then have an open text field for people to enter details for their specific situations.” (Other, Mobile Phone)

11. B18DNREMP2

• Start dates, starting salary and zip code are all difficult pieces of information for people to provide for their first employer after college since it was 10-11 years ago. People are not confident in their answers to this question.

“It was hard. I had to reference my resume because I didn’t know the answers. I am confident in my answer only because I looked at my resume. The most difficult items to answer were the zip codes and dates of employment.” (STEM, Tablet)

“I had to look up the zip code. The salary I know is correct because the start and end salary were the same. The start and end dates are probably 75% accurate.” (STEM, Computer)

“I remembered the company name, but the dates were a little foggy. I’d say it was about 50% accurate.” (Other, Computer)

“I really had to think about it because it’s been over 10 years. The start date, starting salary and zip code were the most difficult.” (Other, Tablet)

12. INTJBLP01

• Most agree that providing this level of detail is “a lot” of information and they would prefer to have more information about WHY they are providing this type of information.

• Respondents agree that it would have been helpful for them to have access to their resume in order to fill out this level of detailed information for each employer since 2011.

• Recent employer information is much easier to report than historical employment information.

• Many say they would need to look up the zip codes for all former employers.

“I would’ve grabbed my resume or looked up the zip codes if we weren’t doing the interview. The hardest parts were the months worked. Because I am a contract teacher, it was so variable. I was fine providing the detail, it didn’t make me uncomfortable.” (Teacher, Tablet)

“It’s a lot of information to provide. I would want to know more about the purpose of why I had to provide this information in order to keep filling it out.” (Teacher, Tablet)

“It was difficult for me to remember my past employment dates since 2011. Listing the previous employers at the top is helpful. I had to look up the zip codes on my smartphone. I am not sure about my confidence regarding the information given, I like things to be accurate and it kind of made me feel like I was lying since I wasn’t sure.” (STEM, Tablet)

“Maybe the survey should ask about three jobs max since 2011. Some people work multiple jobs per year, so this is a lot of information to make people enter.” (STEM, Computer)

“For the more recent jobs it was easy, but the further back it goes the harder it gets.” (Other, Computer)

“It wasn’t complicated to provide the information, but it would’ve been helpful to have my resume in front of me. The most difficult information was the employment dates because I am contracted. The level of detail is a lot of information and I feel like at this point they should offer me an incentive to take the survey.” (Other, Tablet)

“It reminds me of filling out job applications. I don’t know what reason I have to provide all of this information. I’d be frustrated and irritated by this point. If I was getting something out of the survey I would be more open to doing it. I’m kind of pissed off now. I need an incentive that’s motivating or I’d just quit.” (Other, Tablet)

13. B18D1OCC01 or B18D20CC01 & 14. B18DINDUST01

• In general, many respondents have trouble with the occupation coding questions.

• People are confusing the terms “occupation” and “business/industry.” The questions need to clarify what is ultimately being asked.

• Most people are very unsatisfied with the selections provided and feel that none of the options truly match their occupation or industry. They feel they had to “settle” and just select something in order to move on to the next question. Most cannot find a “match.”

• Many get very frustrated with the survey at this point.

• Respondents want to be able to enter their own descriptions of occupation and industry versus trying to fit them into pre-coded categories. The current survey design makes some feel “pigeon-holed.”

• Some feel the industry categories are outdated and limited – Why only provide 4 at a time and how were these industries selected?

• There seems to be some functionality flaws for those answering these questions on their mobile devices. Many have significant trouble clicking on options at the bottom of the scroll list. They can see and read these options, but they are unable to click on them. Also, one respondent did not even see/notice that there were options he had to scroll through and select because they were not visible on his phone screen.

• Respondents are split on their preference for the different versions.

• Occupation Version 1

• Way too much to read and scroll through. Some don’t understand why so many options unrelated to what they originally typed in are shown.

• Choices do not seem to be alphabetized.

• Some like that they can read the detailed descriptions and select the appropriate one.

• Occupation Version 2

• Allows more freedom to type in a custom response.

• Respondents feel some of the choices in the drop-down list are inaccurate (e.g., “Education services” is not an accurate description for teachers).

• Many cannot find a “match” in the dropdown options provided.

• Industry Version 1 (radio buttons)

• Quicker to select from radio button choices. However, the choices are very limited.

• How is it helpful to select “industry not listed”?

• Industry Version 2 (open end text box)

• Allows more freedom to type in a custom response.

• Many cannot find a “match” in the dropdown options provided.

Occupation Quotes

“I didn’t find an occupation that accurately matched. I am an art instructor for young kids. There was only one option for a postsecondary art teacher, not a more general art teacher. The first version was tedious; the list was several pages long. The second version didn’t have a long list, the drop down was easier.” (Teacher, Tablet)

“I can’t really find something that actually fits. The education ones don’t really apply. I don’t know how much to keep scrolling and reading. There are just so many options; it’s a lot of stuff. I don’t understand why all of these choices came up.” (Teacher, Mobile)

“I am a patent examiner, and my occupation was not in either version. When the first version didn’t have it, I tried to play with the words in the second version. I used key words like ‘intellectual property’ and it still wasn’t there. If both versions stay like this they will need to be more accurate and have more choices to reflect all of the jobs in the world that each person does. If not, they should have a text box so that someone like me could fill in my job title. If I wasn’t on the phone with you, I would feel frustrated because I would leave both blank and then they would think I just skipped the questions.” (STEM, Computer)

“I was able to find something similar to what I do in both versions. I didn’t have a preference for either version. The functionality is useful in that it provides options for you, but there should be a blank text box option if you can’t find your exact occupation.” (Other, Computer)

“I’m frustrated. I even went back and changed the search terms to something more basic. I’m surprised that what I entered is not the first option to populate on the list. It was a lot to scroll through.” (Other, Computer)

“Version one was easier because version two is so general. In version one I liked that it listed out detailed descriptions.” (Other, Computer)

“I felt overwhelmed and irritated by the first option. It took up a huge amount of space and had way too much text. And the order was not alphabetical.” (Other, Tablet)

“When I scrolled down on my phone I could see ‘job title not listed,’ but I physically could not click on that option. Also, sometimes on my phone it was hard to see that there were lots of job descriptions to scroll through and select so I might have missed this part.” (Other, Mobile Phone)

“The second version has too many questions and drop downs you have to go through if you can’t find your occupation.” (Other, Mobile Phone)

“Considering the layout on my phone, the second way was far easier, but the first way provided better results.” (Other, Mobile Phone)

Industry Quotes

“I was able to find it in both, but ‘educational services’ is a weird option for teachers. It is less clear in the second version, so I guess I prefer the first one.” (Teacher, Tablet)

“I couldn’t find as close an industry as I would like. ‘Educational services’ is a broad option; to me it sounds more like text book designers. The second version was easier.” (Teacher, Tablet)

“In the first version I was able to find it, but ‘healthcare’ isn’t totally accurate for a nurse. Both are equally as difficult.” (STEM, Tablet)

“There is an issue…I can’t click on ‘Industry not listed’ on my tablet so I have to click on something.” (Other, Tablet)

“I went with something that was close, but nothing was exact. In the second version I tried to type ‘non-profit,’ but that wasn’t one of the options.” (Other, Computer)

“I only found my industry in the first version. I don’t like the second version where you had to type it in. And how is that even helpful if you have to select ‘not listed’? There should be more options to choose from. These questions are not helpful. There are so many industries out there, how is this helpful to have such limited options?” (Other, Mobile Phone)

“I missed that the second version was a drop down and I had to select something. When the keyboard on the phone is up, it makes the viewing portion of the screen really short and it’s hard to see. This is definitely hard to navigate on your phone. When you close the keyboard it like automatically selects the one you were last on, even if that’s not the one you want to select.” (Other, Mobile Phone)

15. B18DDEVX01

• The majority prefer the 5-point scale (Version 2) question because it allows for more nuances in their responses.

• Note that several people would prefer that the scale be reversed and start with “strongly agree.” This is true for all future questions with a similar type of scale.

• A small number of respondents prefer the yes/no (Version 1) because it’s “quicker” – but it’s not the most accurate.

• Most interpret “environment of diversity and inclusion” to mean an environment that is inclusive of many different backgrounds (e.g., age, race, gender, disabilities, socioeconomic, level within the organization) and opinions/ideas. Most do not have trouble with the “diversity” part of the question, but some struggle to understand the meaning of “inclusion.”

“I prefer the 5-point scale. It is more exact. Some of the items on the list cannot be answered with ‘yes’ or ‘no’ except maybe for ‘adequate resources and supports’ – but the rest should be on a scale.” (Teacher, Tablet)

“An environment of diversity and inclusion means employees/students of a variety of different cultures, races, socioeconomic and educational backgrounds. It is a place that values differences in opinions.” (Teacher, Tablet)

“As a respondent I like ‘yes’ or ‘no’ because it is easier, but as a data collector I would think the scale version is better for gathering data.” (STEM, Tablet)

“It’s easier to answer the first one, but the second one is more detailed and is best for this survey because people can provide different degrees in their answer.” (STEM, Computer)

“An environment of diversity and inclusion is racial and socioeconomic diversity and inclusion. It a place where everyone in the company feels important.” (Other, Computer)

“The ‘yes/no’ feels too strong or too final. The 5-point scale gives more diversity to your answer.” (Other, Computer)

“It means diversity in terms of age, gender, sexual orientation, race, personality, etc. It’s keeping everybody included.” (Other, Tablet)

“I like the second version, but I think ‘strongly agree’ should be on the left. It should read from left to right. The first version doesn’t allow for any gray area, it has to be 100% either way.” (Other, Tablet)

“The first version is simple, but there really needs to be some more scale points like ‘don’t know’ or ‘not sure.’ The second version makes more sense because a binary ‘yes/no’ doesn’t always fit.” (Other, Tablet)

“An environment of diversity and inclusion means that diversity is encouraged and that everyone is included. No one is singled out.” (Other, Mobile Phone)

16. B18DJBRESP01

• All interpret “supervise the work of others” to mean that there are employees who report to them whose work they are responsible for reviewing.

• Some teachers aren’t sure how to answer this question since they technically supervise the work of students, but most do not supervise other teachers.

“It means supervising other employees who work under me or report directly to me.” (Teacher, Tablet)

“It’s sort of difficult to answer this as a teacher. I do supervise the work of students, so I’d say ‘yes,’ but I don’t supervise other teachers.” (Teacher, Tablet)

“Providing guidance for those who work under me and fostering their career growth.” (STEM, Computer)

“It is reviewing an employee’s work and giving that person feedback.” (Other, Computer)

“Overseeing employees’ duties throughout the day; like completion of projects, etc.” (Other, Computer)

“It means you’re responsible for delegating tasks and ensuring their completion. You make sure people under you uphold certain standards and are doing what they should be doing.” (Other, Tablet)

17. B18DAUTONM01

• The majority select option 2 – “Someone else decides what I do, but I decide how I do it.” They say that they’re given defined tasks or objectives, but that they often have the freedom to decide how to complete the tasks or what path to take to reach the objectives.

“The school decides the curriculum and the timing of the school year, but I decide how to present it to students.” (Chose option 2) (Teacher, Tablet)

“I am assigned cases, but the way I approach the cases is my choice. There is some level of supervision that watches over me.” (Chose option 2) (STEM, Computer)

“I was a nurse supervisor so I was given authority to make decisions and delegate tasks.” (Chose option 3) (STEM, Tablet)

“Based on the consulting aspects of my job, I gather input from employees and clients and build systems based on that input.” (Chose option 2) (Other, Computer)

“I don’t have complete autonomy, but my company does promote an entrepreneurial spirit.” (Chose option 3) (Other, Computer)

“With my company you are given instructions on how to do each job, but you can adjust various tactics to get a better result.” (Chose option 2) (Other, Tablet)

“This is a little tricky. I’m given a lot of freedom, but I do have certain things that I have to get done, but it’s up to me to get the tasks done however I see fit.” (Chose option 2) (Other, Mobile Phone)

18. B18DPROG01

• Most interpret “demands at home” to include family obligations: taking care of children, sick parents, and marital problems.

• Most interpret “slowed down progress” to mean that they are unable to dedicate the time needed to advance in their career/they’ve been inhibited in moving forward in their career.

• Some interpret “professional activities” to mean their day-to-day job responsibilities, while others interpret it to mean activities above and beyond the day-to-day like work dinners, training, continuing education.

• In order to respond, people think about their current personal situation and determine whether or not they feel it has slowed them down at work.

“’Professional activities is not your actual job; its extracurricular things (e.g. work dinners).” (Teacher, Tablet)

“Maybe you have children or parents to take care of who might have an illness and it has resulted in decreased productivity at work.” (Teacher, Tablet)

“These are things that pop up in your personal life that are time consuming. A wedding, pregnancy or caring for a family member are some examples.” (Teacher, Tablet)

“’Demands at home’ mean family obligations, transportation issues, parenting or spousal issues.” (STEM, Tablet)

“I’m single and have no children or pets, so nothing slows me down.” (STEM, Computer)

“I thought about my current situation and I have minimal demands at home right now.” (Other, Computer)

“Examples include having young kids at home, keeping the house running, parenting or emotional demands with a spouse. And these things may have kept somebody from pursuing development at work.” (Other, Tablet)

“Not being able to commit to extra work so you can’t go above and beyond. It could be looked at as a lack of one’s commitment to their job.” (Other, Tablet)

“Things at home are not allowing me to devote the time I need to devote to the job and I have to say ‘no’ to things that could help me advance.” (Other, Mobile Phone)

19. B18DCHNG01

• Not enough respondents received this probe to provide analysis.

20. B18DCARMLT

• There is some confusion about this question. Some interpret it to be asking whether or not they’ve had multiple “job titles.” Others interpret it to be asking whether or not they’ve changed “job fields,” “industry” or “career paths.”

• In deciding on their response, most think about whether all the jobs they’ve had have been in the same industry or field. If not, they answer “more than one.”

“It’s asking if I’ve made a career change. I haven’t changed; I’ve been a teacher for 10 years.” (Teacher, Tablet)

“The question is asking if I have pursued a different career path within my long-term career goal in the last 10 years. I thought about teaching art education (my field) and that has been my only career, even though I have had some other money-maker jobs.” (Teacher, Tablet)

“If you’ve been employed by more than one employer or if you had different career paths like finance and sales.” (STEM, Tablet)

“Have I pursued more than one field of employment? I answered by not including being a student as part of a career or a job that doesn’t count toward the career.” (STEM, Computer)

“Maybe it means ‘Have you had a job in more than one industry?’ or ‘Have you held jobs outside of your current career path?’” (Other, Tablet)

“I started out as an interior designer, but then re-evaluated what I wanted to do and so I started down a different track.” (Other, Mobile Phone)

“It’s asking if you have changed paths in different industries in the last 10 years.” (Other, Computer)

“Maybe it’s asking how many titles I’ve had in the last 10 years? I’m not really sure.” (Other, Computer)

21. B18DEXPWRK

• Many do not understand what this question is asking and feel the term “same type of work” is ambiguous.

• The language in the survey vacillates primarily between “career,” “job” or “industry.” Then this question asks about “same type of work” and respondents are unclear about what this term really means.

• Respondents are unsure whether “same type of work” is the same as career. Responses to this probe vary greatly.

“It is asking if I plan to be in a position similar to the one I hold now. It is slightly different than a career. Like if I stayed in education, but not art education, I would still consider myself in the same career, but not the same type of work.” (Teacher, Tablet)

“The wording was a little weird so I wasn’t sure. I guess it’s ‘Will I be doing work within the same industry?’” (Teacher, Tablet)

“It means either the same job or the same field. No, it is not the same as a career.” (STEM, Computer)

“It means similar responsibilities to what I am doing now. It could be the same as career, but not necessarily…I am not sure.” (Other, Computer)

“Will you be doing the same duties or in the same industry altogether? They are not necessarily the same. ‘Type of work’ may be talking about day-to-day duties.” (Other, Computer)

“No, it’s not the same thing. A career is long-term, but I could have the same job. I don’t know, this is confusing.” (Other, Computer)

22. B18DNEGOTIAT

• Teachers and government employees answer “no” or “not applicable” because they are not allowed to negotiate salary or benefits. They have to accept the offer that’s given to them.

• Many in other fields have at some point negotiated salary or benefits.

• Some negotiate through a temp agency

• Some use Google to find out what the going rates are for their industry

• Many say the best time to negotiate salary/benefits is when starting a new role

• Some use their current salary as leverage when interviewing with a new company and providing information about their salary expectations

“Not applicable. I can’t negotiate salary or benefits as a teacher for public schools. They just give me an offer and I say ‘yes’ or ‘no.’” (Teacher, Tablet)

“There has never been an opportunity in my field. In education you have set pay scales that are universal. It is based on cost of living and school board determining salary.” (Teacher, Tablet)

“I always feel when you start a position that’s the best time to negotiate. There is a threshold or range that is in their budget, but you should negotiate. I negotiate all the time.” (STEM, Tablet)

“I negotiated based on my experience and what the national salary average is for my industry. I used statistics to get the appropriate salary to ask for.” (STEM, Computer)

“No I haven’t negotiated; it is not applicable because it is a government job.” (STEM, Computer)

“I do contract work so I can’t negotiate. They just tell me, ‘This is your pay.’” (Other, Tablet)

“I researched what others were making in the industry plus factored in my degree and experience and used this to negotiate.” (Other, Mobile Phone)

“I give them a salary number when they ask what I’m looking for and I leverage what I’m currently making now to get a higher salary.” (Other, Computer)

23. B18DPROMOTE

• Respondents are split on whether or not they’ve asked for a raise or promotion.

• Teachers and government employees can apply for a promotion, but cannot ask for one.

• A few say that when they ask for a raise/promotion they prepare a list of talking points about their accomplishments and contributions to help them justify to their employer why they are deserving of a raise/promotion.

“N/A – same as the salary question. I can apply for a promotion, but cannot ask for one. I have to fill out an application, and yes I have applied.” (Teacher, Tablet)

“No, it is not the way the system works. Raises are obtained through higher education or getting your Master’s.” (Teacher, Tablet)

“No, because what I have to do is clearly delineated to get to the next pay level since I work for the government.” (STEM, Computer)

“Yes, I know what I am worth so when it is time for reviews I do ask for a raise or I switch jobs. I always think about how I have helped the company advance and explain it to them. I have asked at multiple jobs.” (STEM, Tablet)

“I’ve asked for a promotion after a certain amount of time at a company or if I’ve done a certain amount of work to achieve certain goals. I would tell my boss those accomplishments.” (Other, Computer)

“I asked at my annual review. I prepared a list of talking points looking at my accomplishments, contributions and salary comps to justify what I’m worth.” (Other, Computer)

“I was asked to take on an additional job or more responsibility in addition to my current role, so I asked for more money.” (Other, Mobile Phone)

24. B18DEVERLK

• Many consider the following types of activities as “looked for employment”:

• Searching job boards and various internet cites

• Submitting resumes

• Talking to recruiters

• Attending job fairs

• Networking with friends

• Looking for jobs in other departments

• Updating their resume and their LinkedIn profiles

• Taking interviews

• Contacting a temp agency

• Teachers went to school district websites looking for openings

“Looking at job postings, applying on different websites, looking at job fairs.” (Teacher, Tablet)

“Taking interviews, applying to multiple schools, looking at other school district websites.” (Teacher, Tablet)

“Looking at online job boards, Craigslist, looking at other places on the internet with the intent to change jobs.” (STEM, Computer)

“Word-of-mouth networking, career job search engines, online websites, posting my resume online, attending open houses/job fairs.” (STEM, Computer)

“Scouring job listings online, speaking to people in your professional network, networking events, talking to Career Services.” (Other, Computer)

“Participating in phone or face-to-face interviews, soliciting companies with my resume, any active searching.” (Other, Mobile Phone)

25. B18FHSCDR

• Most do not have a problem with this question.

• One respondent noticed that there are names of schools in the dropdown that are clearly not high schools, and they find it frustrating that they have to scroll through schools that will not qualify.

• Many like that when they start entering the name of their high school the rest of the response begins to auto fill.

• One respondent would have preferred to search by city and state first and then put his/her school name in to limit the relevant choices appearing in the dropdown.

• Respondents like that the address is shown in the dropdown menu so that they can verify they’re selecting the right school.

“I liked that the options popped up so I didn’t have to type the full name of the school.” (Teacher, Tablet)

“It had the information I was looking for and it was accurate.” (STEM, Tablet)

“The search function was good. I liked that it auto populated before I got halfway through typing the name of the school. ” (Other, Tablet)

“It worked nicely. My high school is not a common school, but I liked that it narrowed down the list to only two choices and then I confirmed by checking the address.” (Other, Computer)

“I like that it shows the address in case you’re not sure.” (Other, Mobile Phone)

26. B18AMARSMY

• Many respondents question what the purpose of this survey item is and they want to know why it is relevant. They don’t understand what it has to do with their employment or education.

• One respondent was asked to provide the month and year she was divorced, which she felt was totally inappropriate.

“I’m ok with it. It was very specific…surveys don’t usually ask how long you’ve been married.” (Teacher, Tablet)

“It is a random question. I didn’t really want to answer it. They should tell me why they need it.” (Other, Tablet)

“It’s sort of a weird question. What does this have to do with the survey? It makes me a little paranoid.” (Other, Computer)

“I guess it’s ok, but at first I was wondering if I really needed to share this information. What is the survey for?” (Other, Computer)

“‘What month and year were you divorced?’ This is too prying and inappropriate.” (Other, Phone)

27. B18FSEX

• Respondents do not have any problems understanding this question.

• Many assume that this question is designed for those who are transgender and have changed genders from the one they were born at birth.

“They are asking sex at birth not your gender. It doesn’t bother me because I am a woman and was born a woman, but it might bother someone else.” (Teacher, Tablet)

“It’s asking exactly what it says…what gender were you born as?” (Teacher, Tablet)

“The question would be for someone with a gender identity situation, whether you identify as a male or female.” (STEM, Computer)

“It is asking my birth gender and it provides clarity for those who have changed their gender.” (STEM, Tablet)

“It is 2016 so it clarifies what sexual orientation is.” (Other, Computer)

“It’s asking what’s on your birth certificate.” (Other, Computer)

“This is the first time I’ve seen this question phrased this way. It’s more gender sensitive.” (Other, Mobile Phone)

28. B18FGENDERID

• Respondents do not have any problems understanding this question.

• Several do not understand why this question is necessary for the survey and think it’s irrelevant. Some even get offended and say this question is “too personal” and “none of your business.”

• Many like the definition provided, although a few state that they do not agree with the definition. One respondent thinks the definition is a little condescending and could make people feel like they are being talked down to.

• Most believe the answer choices seem inclusive.

• No one has trouble understanding what “transgender” means.

• Several people are unsure what “Genderqueer” means.

“Yes, this definition is something I knew. The definition is pretty comprehensive and I agree with it. It’s inclusive.” (Teacher, Tablet)

“Transgender means you were born one sex but you identify with a different gender.” (Teacher, Tablet)

“I don’t necessarily agree with the definition.” (STEM, Computer)

“I haven’t heard of ‘genderqueer.’” (STEM, Tablet)

“This question is a bit extreme.” (Other, Computer)

“Transgender means the person identifies with a gender other than what they were born with.” (Other, Tablet)

“I disagree with the definition. Gender is not something we choose or how we feel.” (Other, Mobile Phone)

“How is this relevant? How is this important? What is this survey for anyway? This is no one’s business.” (Other, Computer)

29. B18FLGBTQ

• Many respondents question what the purpose of this survey item is and they want to know why it is relevant. They don’t understand what this question has to do with their employment or education. Some even get offended and believe this question is too personal.

• Respondents have no trouble understanding the question.

• Several people think “asexual” is missing from the list.

• One or two think “hermaphrodite” is missing from the list.

“I chose my answer because I am bisexual; I’ve considered myself bisexual for the past 11 years. ‘Another sexual orientation’ covers most everything that is not on the list, but I think ‘asexual’ is pretty big so it should be on the list too.” (Teacher, Tablet)

“None of the terms are unfamiliar to me. ‘Hermaphrodite’ may be missing from the list.” (STEM, Tablet)

“‘Asexual’ is missing.” (Other, Computer)

“This is a strange question to ask.” (Other, Computer)

“How does this pertain to the survey?” (Other, Computer)

“I can’t believe these questions are on here.” (Other, Computer)

“This is an interesting question. I don’t have a problem with it, but what do they need this for? It has nothing to do with my job.” (Other, Tablet)

“This is what I identify myself to be.” (Other, Tablet)

“I answered based on my desires. I’m married to a woman.” (Other, Mobile Phone)

“This is asking too much. The last two questions are too personal. You don’t need to ask people’s sexual orientation.” (Other, Mobile Phone)

30. B18FAWARE

• Many respondents question what the purpose of this survey item is and they want to know why it is relevant. They don’t understand what it has to do with their employment or education.

• At this question, several people mention that the gender preference series of questions seems intrusive. “It started to feel invasive three questions ago.”

• Some respondents do not personally have a problem answering the question because they feel they are in the majority demographic (straight), however they could see how this question could be upsetting or intimidating to others.

• Several people think “All” should be an option on the scale.

• Many people say they do not talk about their sexuality at work and feel this aspect of the question is inappropriate.

“I am a straight, white woman, so it is easy for me to answer. For me it is not a problem.” (Teacher, Tablet)

“To answer the question, I thought about a specific example of all of the people in my life. I don’t purposely hide my bisexuality, but I don’t necessarily talk about it to my coworkers or family.” (Teacher, Tablet)

“I am fine with it because I don’t have a situation that would be an issue, like I am not in the closet.” (STEM, Computer)

“I am fine to answer this, but I can see where this might be difficult for someone else depending on their background.” (Other, Computer)

“It’s weird that there’s not an ‘all’ answer option. If I was uncomfortable with my sexuality it might intimidate me to answer this, or I might just lie and you would get data that is not accurate.” (Other, Computer)

“These questions do not pertain to career and education. Sexual orientation has nothing to do with this subject matter.” (Other, Computer)

“This is pretty intrusive depending on the lifestyle you are choosing to live.” (Other, Computer)

“Why is this important for them to know? It started to feel invasive three questions ago. What are they questioning me about my lifestyle for? What does this have to do with anything?” (Other, Computer)

“Again, what do you need this for? You said this was an educational survey, how will you use this information?” (Other, Tablet)

“People know me and my dating background so I guess it’s fine. It’s easier for me to answer this compared to someone who has a different sexual orientation than what people perceive as ‘normal.’ It shouldn’t matter if employees and co-workers know my sexual orientation.” (Other, Tablet)

“I would totally shut down here. I don’t want to answer this stuff. This is no one’s business. There should be an ‘NA’ category so I can just move on to a different question.” (Other, Mobile Phone)

31. B18FDEPDOB

• Only a few respondents received this question, but they feel the layout is fine.

• Some who received this question would only feel ok providing the information if they knew it was an important study.

• Some feel uncomfortable providing this level of detail about their children.

“Providing this information is fine with me after all of the other information I’ve had to provide. I have no issues with the layout. ” (STEM, Computer)

“The layout is pretty clear.” (STEM, Tablet)

“I don’t know why the month is necessary. Why is that important?” (Other, Computer)

“I think this is overkill. Providing the number of depends is sufficient or ask for their ages, but providing the month and year they were born is overkill.” (Other, Tablet)

“This makes me uncomfortable. It’s too much detail.” (Other, Mobile Phone)

32. B18FDEPDAT

• Only a few respondents received this question. They don’t have any problems understanding what it’s asking.

• One respondent is unsure how to answer the question because she had to provide goods and services even before the child was born.

• An example provided is if someone fathered a child that they were not aware of.

• Again, some feel uncomfortable providing this level of detail about their children.

“The question is trying to find out when I became financially responsible for my kids. Some examples could include if someone didn’t know they were the father of a child.” (STEM, Computer)

“It is asking you to identify when the kids became dependents. For people with fostered or adopted children or blended families, these dates might be different from the child’s birthday.” (STEM, Tablet)

“I guess an example would be if the child didn’t live with you at first. I think this is too detailed and it makes me very uncomfortable. My suggestion is to not ask this question.” (Other, Mobile Phone)

“This is too invasive. Why do you have to know if I have a foster or adopted child? It’s really none of your business.” (Other, Computer)

33. B18DLVCHLD

• Only a few respondents received this question. They don’t have any problems understanding what it’s asking.

• Examples provided include:

• Maternity leave

• Medical leave for their child

“Taking your kids to the doctor, family medical appointments or family trips.” (STEM, Computer)

“I took time off after childbirth. I can’t think of any other situations.” (STEM, Tablet)

“Hospitalization for birthing a child.” (Other, Mobile Phone)

“If you had surgery, FMLA, time away for whatever reason. Again, why do they need to know this? It feels too personal – too nosy.” (Other, Computer)

34. B18DLVCT

• Only a few respondents received this question. They don’t have any problems understanding what it’s asking.

• Women who’ve taken maternity leave have an easy time answering this question.

• Others think it’s difficult to remember back to 2011.

• One respondent wasn’t sure how to answer the question because the total time he took off for the birth or adoption of his child was less than one month. For example, he wanted to choose two to three “weeks,” but the only options provided in the question are “months” or “years.”

“It was difficult remembering back that far; I just ball-parked it.” (STEM, Computer)

“It is easy because I went right back to work after the childbirth leave.” (STEM, Tablet)

“What if the time was less than one month? They don’t have enough options. I wanted to answer two to three weeks, but that wasn’t an option.” (Other, Mobile Phone)

35. B18FOTHER1

• Respondents do not have any problems understanding this question.

• Examples of expenses include:

• Healthcare

• Rent/mortgage

• Food/groceries

• Car payments

• Tuition

• Utilities/household bills

• Clothing

• Transportation

• Examples of who they would include:

• In-laws

• Parents

• Siblings

• Close friends

• Cousins

“It is asking if there is anyone you support financially beyond kids. It was easy for me, but the explanation was helpful.” (Teacher, Tablet)

“If people are dependent on you for their own livelihood. Expenses would be rent, mortgage, food, household bills, car payments, any type of insurance.” (Teacher, Tablet)

“It includes everything like food, clothing, housing, utilities and transportation.” (STEM, Tablet)

“It is asking if there are people who are not your children who you support financially. Financial support includes daycare, food and private school. I would include a family member who couldn’t work, siblings, parents or a spouse.” (Other, Computer)

“I think it would be a car payment, education, lodging, clothes, food, daycare, medical expenses and health insurance. I would include siblings, parents, aunts and uncles, significant others, grandparents, children of a significant other if they didn’t live with us. I would exclude anyone that lives with me.” (Other, Computer)

“Financial support would include private school, college, health insurance, car payments or a paying for a retirement home.” (Other, Tablet)

36. B18FOTHER2

• Most respondents do not have any problems understanding this question, but it is confusing to a small number of people.

• One respondent pointed out that she didn’t think she should have received this question since she’d previously answered that she had “zero” dependents.

• Most think of elderly relatives or parents. A few others think of children out of wedlock or a friend.

“This might include a parent who needs non-financial or physical support. It would exclude anyone you support financially.” (Teacher, Tablet)

“This would be someone who was dependent on your for your car, but not for finances.” (Teacher, Tablet)

“An example would be if we had any other children and we did not give money to support them, but we are the primary caregiver.” (STEM, Tablet)

“You’re taking care of someone – like maybe if you’re married to someone with a child that’s not yours and you don’t do anything financially, but you still help take care of them.” (STEM, Computer)

“The question is asking if you support someone, but not financially. Perhaps helping someone under 18 make decisions, like an orphaned child of a friend or relative.” (Other, Computer)

“It’s asking if there is anyone who you are a primary caregiver for…perhaps looking after them medically.” (Other, Computer)

“Maybe it would be relatives that depend on your time and help, but not your finances. An elder, or like a grandparent.” (Other, Computer)

“It’s a weird question. It almost seems like a double negative or something. And I don’t understand why I got it since I said ‘no’ to having dependents earlier, so I don’t feel like I should have to answer it again.” (Other, Computer)

37. B18FAFFCOST2

• Respondents do not have any problems understanding this question.

• Many are ok with answering the question and some even think it is good, thought-provoking question to ask.

• A few say it makes them feel sad or angry.

• A few think the question is too personal.

• In order to answer, respondents think about their own situations and describe which answer choices are pertinent to their situation.

“The question is referring to your student loans and how it has affected the post-graduate choices you have made.” (Teacher, Tablet)

“I think it’s asking if you have any debt from college, whether you have had to make sacrifices or changed your ideal path.” (Teacher, Tablet)

“I’m fine with answering this question. It is relevant to what is going on in today’s economy and social situations.” (STEM, Computer)

“I felt angry answering this question. You go to school to advance your career, but you are taking out loans to do it that you will have to pay back and that can impact things.” (STEM, Tablet)

“It is asking if due to financial obligations I have had to compromise in any way.” (Other, Computer)

“I am fine to answer it...it is a good question to ask.” (Other, Computer)

“It’s about the repercussions of paying student loans or having fewer resources as a result of paying for your education and its impact on your personal life.” (Other, Computer)

“It made me a little sad to answer this. I’m just lucky that I only checked off two answers, but I know people who would have to check off all of them.” (Other, Mobile Phone)

“I thought about how I’ve had multiple jobs to make more income to pay off my loans. And I had to take a job outside of my field during the depression. More school would have been too much of a financial commitment so I didn’t do that. And I did hold off on a house so that I can pay off my education debt first.” (Other, Computer)

38. B18FWORTH

• Many respondents do have to think hard about their answer to this question.

• They understand the word “worth” to mean:

• ROI – return on investment

• Value of their education compared to the cost of their education

• Several respondents do not think their education was worth the financial cost. They feel they could have gotten a similar education for a cheaper cost.

• Others say it is worth the cost because a degree is a necessity in today’s job world.

• Some think the question is difficult to answer because measuring the “worth” of education is difficult.

“‘Has it been financially worth it? Is the money I make worth what I paid for my education? I said ‘yes,’ but someone who went to a more expensive school may answer differently.” (Teacher, Tablet)

“Did the services and quality of the education match the price the school charged me?” (Teacher, Tablet)

“No I do not think it was worth it. Looking at my situation with undergraduate education, I might not be as far in debt as some, but I still have to think about my debt and whether I could’ve gotten to the same place having gotten a public education.” (STEM, Tablet)

“‘Worth’ is what I got out of my education. Was the money spent worth the return on the investment?” (STEM, Tablet)

“Did all the pain pan out into something you’re proud of and would you do it again?” (STEM, Computer)

“I have mixed thoughts. I work with people who do not have a bachelor’s. But overall I think my education helps with things like networking.” (Other, Computer)

“This is a difficult question to answer. It’s hard to measure if it’s worth it. Society values education, so you sort of have to suck it up and do it. But is it really worth all the debt? I don’t know. But because of societal pressures to get a degree, I answered ‘yes.’” (Other, Computer)

“This was a little challenging. Were the trade-offs worth the debt? Has my degree helped me get to where I am and would I do it again? Has my degree added value?” (Other, Tablet)

39. B18FSELLPO

• Most respondents find this question relatively easy to answer, although they do take a few minutes to think about their response.

• Many respondents choose the first response – “have something left over.”

• Many say their possessions are insignificant (few respondents currently own a house).

• The major possessions they consider include:

• Clothes

• Jewelry

• Car

• Furniture

• Major electronics

• Savings

“I gave a rough estimate. It is a little difficult because there is some math involved. I included my car, large furniture and TV.” (Teacher, Tablet)

“I don’t have too many major possessions. So…I guess my car, electronics, furniture and jewelry. I don’t own a home or anything.” (Teacher, Tablet)

“It would include a house, cars, electronics, jewelry, tools and anything over $500. I am 90% confident with my answer; it depends on how much I could sell my house for.” (STEM, Tablet)

“I didn’t have to think too much about it. I don’t have any major debts so it was easy to answer.” (Other, Computer)

“It was not difficult to answer. I included my car and investment accounts.” (Other, Tablet)

“I had to think about it at first, but then it was fine.” (Other, Computer)

40. B18FBIGFIVE

• Many respondents are frustrated by this question. They do not agree with the adjective pairings and feel they should not be connected (i.e., you can be conventional, but not uncreative or you can be disorganized, but not careless). Each adjective should be its own survey item.

• Some mobile phone respondents think it is difficult to navigate the screen and there’s too much scrolling involved.

• Some say the scale should be reversed (i.e., start with agree and end with disagree).

• Some say there are too many scale points and would prefer it be cut down from seven to five.

“The question was fairly easy. The only thing that was a little difficult to answer was ‘open to new experiences, complex’ because they don’t really go together.” (Teacher, Tablet)

“It doesn’t fit on my whole tablet screen so it’s kind of annoying. There are too many choices. Maybe make it only five choices.” (Teacher, Tablet)

“The question is fine, but I don’t get option two – ‘open to new experiences, complex’ – I don’t necessarily think the former means ‘complex.’” (STEM, Tablet)

“It was difficult to answer. I liked the scale choices, but I don’t think some of the pairs go together like ‘dependable, self-disciplined’ and ‘open to new experiences, complex.’” (Other, Computer)

“The scale is long so I can’t see the whole thing on my screen. Because I couldn’t see the whole scale I didn’t know the rest of it was there.” (Other, Tablet)

“This was difficult. The screen was hard to navigate on my phone. Also, the last two pairings were in conflict. They’re such different traits. I mean, I’m disorganized, but not careless. I’m conventional, but I’m also creative. These pairings just don’t make sense.” (Other, Mobile Phone)

“Being honest can be hard. And a lot of these pairings don’t go together. I wouldn’t want to answer this. What’s the purpose? Why am I going to tell someone I’m disorganized?” (Other, Mobile Phone)

Supporting Materials

Background Profile of Participants

RECRUITMENT SCREENER

Eligibility Screening Questions

The recruitment screener, a 3-5 minute phone script, will be used to determine if individuals qualify for study participation in order to test the items identified for cognitive and usability testing for the B&B:08/18 survey. Recruiters will reach out to respondents or potential respondents will call a toll-free number mentioned in an ad and an experienced recruiter will screen the caller for eligibility for the project.

Intro: Thank you for inquiring about this study. I need to ask you a few questions to determine if you are eligible to participate. The screening is expected to take approximately 3-5 minutes.

1. Were you enrolled for a bachelor’s degree at a postsecondary institution at any time between July 1, 2006 and June 30, 2007?

Yes

No (Ineligible)

2. When did you complete the requirements for your bachelor’s degree?

Month/Year

If between July 1, 2006 and June 30, 2007 (Eligible)

Some other time (Ineligible)

3. What is the name of the college or university where you received your bachelor’s degree?

4. Was your primary major for your bachelor’s degree in a STEM field, that is, science, technology, engineering or math?

Yes

No

5. (If yes to 4): What was the specific STEM field for your bachelor’s degree?

6. (If no to 4): What was the primary major for your bachelor’s degree?

7. Have you enrolled in any college or university for an additional degree or certificate, including enrollment for any graduate or additional undergraduate degree or certificate programs, since earning your bachelor's degree?

Yes

No

8. Are you currently employed?

Yes

No

9. (If no to 8): Have you ever been employed since completing your bachelor’s degree requirements?

Yes

No (Ineligible)

10. Since receiving your bachelor’s degree, have you worked full-time or part-time for at least a year as a teacher at the K-12 level?

Yes

No

11. What is your age? Are you…

18-24

25-29

30-34

35-39

40 or older

12. Are you of either Hispanic or Latino origin?

Yes

No

13. What is your race?

Choose one or more.

White

Black or African American

Asian

American Indian or Alaska Native

Native Hawaiian or Other Pacific Islander

14. What is your gender?

15. Now that you have been out of school for a few years, how has your education impacted your career? (WRITE VERBATIM. PROBE AND CLARIFY)

16. Would you be willing to provide us with your current resume?

Yes

No

17. On a scale of “1” to “10,” where a score of “10” means that the statement describes you extremely well and “1” means that the statement does not describe you at all, how well does the following statement describe you? (CIRCLE ONE NUMBER PER ROW)

	CIRCLE ONE
	EXTREMELY WELL									NOT AT ALL WELL
I feel comfortable expressing my opinion in a group or interview		10	9	8	7	6	5	4	3		2	1

NOTE TO INTERVIEWERS: IT IS EXTREMELY IMPORTANT THAT RESPONDENTS IN THIS STUDY BE ABLE TO PROVIDE AND CLEARLY EXPRESS INFORMATION ABOUT THEIR EXPERIENCE. IF RESPONDENT CANNOT OR WILL NOT EASILY GIVE A ONE TO TWO SENTENCE UNPROMPTED ANSWER IN WELL UNDERSTOOD ENGLISH, PLEASE TERMINATE.

If ineligible: Thank you for inquiring about this study. If you are eligible to participate, we will contact you at a later date.

If eligible (Invitation to participate):

We are conducting online video interviews with respondents like you to evaluate some of the questions that will be used in an education survey with special topics. This is not a sales meeting of any kind.

The interview will take about one and a half hours to complete and respondents will receive $40 after the interview as a thank you for their participation. The interview will be conducted using online video technology and you can participate from your computer.

Can I schedule an interview at your convenience?

	CIRCLE ONE
Yes	1	(ASSIGN TIME SLOT AND CONTINUE)
No	2	(THANK AND TERMINATE)

Does the computer at the location where you will be participating in the interview have a webcam?

	CIRCLE ONE
Yes	1	(GO TO NEXT QUESTION)
No	2	(CONTINUE)

If you don’t have a webcam on this computer, we will send you one. Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

______________________________________________ (MAILING ADDRESS)

Part of the interview will involve filling out a questionnaire using a smart phone or tablet. Do you have a smart phone or tablet that you would be willing to use for this purpose? (ABOUT HALF OF RESPONDENTS WILL COMPLETE THE SURVEY USING A MOBILE DEVICE)

	CIRCLE ONE
Yes, has a smart phone	1	(CONTINUE)
Yes, has a tablet	2	(CONTINUE)
No, have neither that can be used	3	 (RECRUIT FOR COMPUTER USABILITY, GO TO NEXT QUESTION)

In order to complete the survey using a smart phone or tablet, we will need to send you a document camera to use to record the screen on your device as you are using it.  Please provide your mailing address below. The address must not be a Post Office box, and the office/suite number or apartment number must be included, if appropriate.

____________________________________________ (MAILING ADDRESS)

If for any reason your plans change and you can't keep your appointment, please call and let me know in advance. My telephone number is (INSERT NUMBER).

Finally, we will be sending you a “consent to participate in research form” via email that we will need you to sign and scan back to us in order to confirm your participation.

INTERVIEWER: RECORD APPOINTMENT DATE AND TIME ON FRONT OF SCREENER. COMPLETE NAME AND CONTACT INFORMATION.

Obtain the phone number and email address of the respondent and confirm that the phone number is the best one to reach him/her when they participate in the interview.

(RECORD BELOW.)

_______________________________________ (RECORD E-MAIL ADDRESS)

_______________________________________ (RECORD TELEPHONE NUMBER)

THANK YOU FOR YOUR PARTICIPATION.

WE WILL SEND YOU LOGIN INSTRUCTIONS FOR THE SESSION AND ONE OF OUR TECHNICAL STAFF WILL CONTACT YOU TO SET UP AN APPOINTMENT BEFORE THE SESSION TO ENSURE YOUR WEBCAM, DOCUMENT CAMERA, AND THE ONLINE VIDEO SOFTWARE ARE WORKING PROPERLY FOR YOU.