High School Longitudinal Study of 2009 (HSLS:09) Second Follow-up
Cognitive and Usability Testing
OMB# 1850-0803 v.135
Volume I
April 15, 2015
The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB #1850-0803) that allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, focus groups, and cognitive interviews. The request for approval described in this memorandum is to conduct cognitive interviews and usability testing in preparation for the main-study data collection of the Second Follow-Up of the High School Longitudinal Study of 2009 (HSLS:09). The data collection for this study is being carried out for NCES by RTI International – with Research Support Services (RSS) as a subcontractor – under contract to the U.S. Department of Education (Contract number ED-IES-14-R-0335).
The HSLS:09 Second Follow-Up will focus on respondents who have been out of high school for 2 and 3 years (including both high school graduates and students who dropped out). The survey will help to inform researchers, policymakers, and other stakeholders about issues pertaining to transition from high school to postsecondary education and the labor force, with a particular emphasis on sample members who are pursuing postsecondary education and careers in science, technology, engineering, and mathematics (STEM) fields.
A previous round of cognitive testing of a subset of draft questions (OMB# 1850-0803 v.118) was conducted from December 2014 through February 2015. It did not test the programmed instrument due to timing constraints. This request is to conduct a second round of testing focusing on questionnaire content and understanding and usability of the self-administered instrument (web and mobile-friendly versions). This round will test the entire programmed survey instrument. Revisions based on the results of cognitive testing will be included in the main-study instrument design. The main-study data collection request will be submitted to OMB in September 2015 and will document these revisions.
Cognitive interviews will help us to better understand respondent perspectives on a variety of issues and will be used to refine the survey questions to maximize the quality of data collected during the main study. We are also interested in respondent perspectives on the mechanics of the instrument, for example how easy it is to navigate, view question and response option wording, and provide responses. To address these multiple components of the respondents’ experiences participating in the HSLS:09 survey, usability testing will be conducted on tablets, Androids, and iPhones in addition to desktop or laptop computers. All respondents will be screened for device use and ownership and will be assigned to a device with which they are familiar. To maximize efficiency and minimize burden, the cognitive testing and usability testing will be conducted simultaneously.
This request is to conduct cognitive interviews and usability testing begining in early May 2015. Results will be summarized in two reports which will become available by August 2015 in time to present the findings to a technical review panel (TRP) scheduled for August 18-19, 2015. The summary reports will also be included with the main-study data collection package.
RSS (located in Evanston, Illinois) will recruit up to 45 participants in the greater Chicago area, selected to ensure representation of racial, ethnic, and sexual minority groups. In the first round of cognitive interviews completed in February 2015, NCES tested three questions about birth sex, gender identity, and sexual orientation. RSS successfully recruited 10 sexual minority participants for the first round and NCES plans to further test these questions in the second round. Table 1 provides a distribution of the desired minimum number of respondents by type, although a respondent may fit more than one category.
Table 1: Desired minimum number of respondents, by respondent type
Respondent type |
Desired minimum number of respondents |
Never enrolled in postsecondary education |
7 |
Currently enrolled in postsecondary education
|
7 7 |
Working while enrolled |
7 |
Working but not enrolled |
7 |
Left high school without diploma or has GED |
7 |
Identifies as LGBTQ |
10 |
RSS will recruit cognitive/usability testing participants by advertising in the Chicago metropolitan area. To maximize exposure to a diverse group of young adults, a variety of locations and media will be used, including
community locations (e.g., coffee shops, community organizations serving youth and/or LGBT populations);
neighborhood and city newspapers;
internet listing services (e.g., Craigslist, Meet-up groups, Facebook); and
word of mouth (e.g. snowball sampling).
The advertisements will describe the purpose of the cognitive and usability testing as well as the details of participation, including the time commitment, incentive for participation, contact information, and the participation eligibility requirements. A sample recruiting flyer and online advertisement are included in Attachment I.
Individuals who are interested in participating in a cognitive interview and usability test will be asked to call the RSS office to complete a brief screening interview to determine eligibility. Eligible participants will be between the ages of 20 and 22. All participants will complete both cognitive and usability testing, which will be conducted on a computer or a mobile device depending on the preference of the participant. The eligibility screening questions are included in Attachment III. The scripts that will be used to schedule the testing and determine the device to be used for usability testing are shown in Attachment IV.
Trained RSS staff will conduct the cognitive interviews in rented conference rooms. Each interview will last a maximum of 90 minutes, and all participants will be offered an incentive of $40 to thank them for their time and participation. Audio recording of each interview will be available to NCES for review. All participants will be asked to sign consent forms prior to the start of the interview. A sample consent form is included in Attachment II.
Respondents will be asked to answer the questions as they normally would in the absence of an interviewer, and to voice any concerns or thoughts as they do so, either regarding issues with the questions or with navigating the instrument. Probing will follow the self-completion and will aim to elicit narrative about what prompted each response. This approach will allow us to gain insight into the processes required to answer survey questions. These concurrent comments and observations as well as retrospective probing can provide valuable insight into the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment, and other processes affecting the final answers to survey items, as well as issues in administration related to the implementation and interface of the computer-assisted survey. To elicit relevant responses, respondents may be asked how they arrived at their answer, what terms might have been unfamiliar or ambiguous, to paraphrase the question or its accompanying instructions, to define a term, and to voice any difficulties. The interviewer will use probing techniques (such as “Tell me more about your answer” or “What was difficult in answering this question?”) to verify respondent interpretations of question wording and response options, investigate the meaning of specific phrases, or identify concepts that the respondent thought were critically relevant to the question but absent from the wording. Interviewers will observe and make notes on how the respondent navigates through the instrument, any backtracking to prior screens, and any exhibited confusion.
Usability testing evaluates the design of the survey’s user interface for ease of navigation, providing responses, and finding and making use of supporting features (e.g., help text, progress bar). Respondents will be asked about the visual display, layout of the questions, ease of navigation, and other mechanisms important for completing the interview, particularly on different devices such as desktop or laptop computers, tablets, and mobile phones.
The cognitive interview protocol, which includes a list of sample probes, is included in Attachment V, and the facsimile of the survey instrument in Attachment VI. The cognitive and usability testing will be conducted on the programmed survey instrument that was approved for the field test data collection (OMB# 1850-0852 v. 15 and 16, approved in March 2015).
Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, highlight potential themes that may have arisen from the cognitive testing, and document any usability issues observed. Following each interview, the digital audio recording will be archived for qualitative analysis. RSS will organize their observations and summarize the common themes, insights, and recommendations emerging from each of the interviews into one report for cognitive testing and one for usability testing.
Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C. § 9573). Participants will be assigned a unique participant identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the participants’s name in any way. Participants will sign an informed consent form, which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study, and which will be destroyed after the final report is submitted.
RSS will begin recruiting for the cognitive testing upon receiving OMB clearance. The interviews are scheduled to begin in May 2015 and be completed by June 2015. This timeline allows the project team to present the results at an August TRP and to use them to refine the survey questions to be administered in the spring 2016 main-study data collection.
To yield up to 45 completed interviews, up to 200 individuals may need to be screened for eligibility and to ensure that we achieve the target goals for recruiting a mix of respondent charcteristics. The screening process, on average, is estimated to take about 5 minutes per person and each inyerview about 90 minutes.
Table 2: Estimate respondent burden
Activity |
Number of respondents |
Number of responses |
Minutes per respondent |
Maximum total burden hours |
Screening |
200 |
200 |
5 |
17 |
Cognitive interview |
45 |
45 |
90 |
68 |
Study Total |
200 |
245 |
|
85 |
In order to encourage participation by recent high school students in this study, and to thank them for their time and effort spent traveling to the interview facility and partaking in the interview, each respondent will be offered $40.
Participants in the cognitive interviews must bear direct costs associated with travel by car or use of Chicago Transit Authority services to RSS’s rented conference rooms in Chicago.
The cost to federal governent for conducting these cognitive interviews and usability testing is $50,741, under the RSS subcontract to RTI, including recruitment, interviewing, analysis, report writing, and participant incentives.
Attachment I: Recruitment Materials
Sample Recruitment Flyer
Are you 19-23 Years Old and [one of the characteristics of interest]1?
[LGBTQ? / In college? / In vocational, technical, or trade school? / Working on your GED?]
You may be eligible to earn $40 by participating in an interview (a discussion between you and an interviewer about how you understand certain questions concerning your experiences and plans since leaving high school). This interview will also ask about the mechanical aspects of the web survey, including questions about the visual display and ease of navigation.
About the interviews:
Research Support Services (RSS) and RTI International are collecting data for a study conducted by the National Center for Education Statistics (NCES) of the U.S. Department of Education. The purpose of the study is to obtain your feedback on some of the questions that will be used in an NCES national study. The interview will begin in May 2015 and will be held at a central location convenient to public transportation in the Chicago area. The interview will last about 90 minutes. A researcher will ask you to respond to questions, including what you think the questions mean. The researcher will also ask you questions about how easy the web survey is to complete. The survey will be about your experiences since high school and future plans.
Eligibility to participate:
Please call RSS between 9 a.m. and 5 p.m. weekdays to complete a brief telephone screening in order to determine if you are eligible. The telephone screening will take approximately 3–5 minutes to complete. You will not be compensated for completing the telephone screening.
Contact: Helen Solis, Research Support Services, 888-540-6770
Sample Online Ad (e.g., Craigslist)
Are you 19-23 Years Old and [one of the characteristics of interest]?
[LGBTQ? / In college? / In vocational, technical, or trade school? / Working on your GED?]
You may be eligible to earn $40 by participating in an interview (a discussion between you and an interviewer about how you understand certain questions concerning your experiences and plans since leaving high school). This interview will also ask about the mechanical aspects of the web survey, including questions about the visual display and ease of navigation.
Research Support Services (RSS) and RTI International are collecting data for a study conducted by the National Center for Education Statistics (NCES) of the U.S. Department of Education. The purpose of the study is to obtain your feedback on some of the questions that will be used in an NCES national study. The interview will begin in May 2015 and will be held at a central location convenient to public transportation in the Chicago area. The interview will last about 90 minutes. A researcher will ask you to respond to questions, including what you think the questions mean. The researcher will also ask you questions about how easy the web survey is to complete. The survey will be about your experiences since high school and future plans.
Please call RSS between 9 a.m. and 5 p.m. weekdays to complete a brief telephone screening in order to determine if you are eligible. The telephone screening will take approximately 3–5 minutes to complete. You will not be compensated for completing the telephone screening.
Contact: Helen Solis, Research Support Services, 888-540-6770
Attachment II: Sample Consent Form
HSLS:09 Cognitive Testing of Questionnaire Items
Participant Informed Consent
You are invited to participate voluntarily in this interview, which is being conducted by the National Center for Education Statistics (NCES), part of the U.S. Department of Education, with data collection being carried out by RTI International, a not-for-profit research firm, and Research Support Services (RSS), a contract research firm. The purpose of this interview is to help NCES review the High School Longitudinal Study of 2009 (HSLS:09) Second Follow-Up questionnaire to ensure that it can be well understood.
This interview has been approved by the Office of Management and Budget (OMB# 1850-0803). The interview and discussion will take approximately 90 minutes. There are no right or wrong answers to the questions we ask—we just want to ask your opinions about the questionnaire that has been developed. You will be asked how you understand specific terms and phrases in the questions as well as how you arrived at your answers. You will also be asked about how easy you find the web survey to navigate. Finally, we will ask you about your thoughts and feelings toward the questionnaire overall. You can help us by describing anything you find confusing or difficult to understand in the questions or any issues that come up as you answer them.
Again, your participation in this study is voluntary. There is no physical risk and only minimal risk associated with data confidentiality. We will respect your privacy. We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However, you may refuse to answer any question or take a break at any time. There are no direct benefits to you for participating in this study, but we hope that these interviews will help us develop and improve questions for a national survey about individuals’ experiences as they transition from high school to postsecondary education and to the labor force. Upon completing the interview, you will receive $40 in appreciation for your time and participation. The information you give us will be combined with the responses of others in a summary report that does not identify you as an individual. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (ESRA 2002, 20 U.S.C. § 9573).
If you have any questions about the study, you may call Helen Solis at Research Support Services (888-540-6770) or Melissa Cominole at RTI International (919-990-8456). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).
____________________________________________________________________________
The above document describing the voluntary nature, response confidentiality, and procedures for this research study has been explained to me. I agree to participate.
Signature of participant________________________________ Date ___/___/___
I certify that the purpose, the voluntary nature, and response confidentiality associated with participating in this research have been explained to the participant.
Signature of Person Who Obtained Consent_______________________________
Date ___/___/___
Audio-taping the interview
In order to make the best use of our findings, we also request that you allow the interview to be audio-taped, through the use of a tape recorder or laptop computer that will be on the table. The audio-tape will only be heard by people who are working on this project. The only purpose of audio-taping is to allow us to review the interview as we document our findings. We will destroy the tapes upon completion of the project. If you would rather that your interview not be audio-taped, or if at any time during the interview you decide that you would like the audio-taping to be stopped, please tell the interviewer and we will stop taping.
____________________________________________________________________________
I agree to allow the interview to be audio-taped and to be listened to by others who are working on this project:
Signature of participant ________________________________ Date ___/___/___
1 The target group for this study is individuals aged 20-22. However, the recruitment flyers specify a slightly wider range of ages (19-23) in order to maximize the accuracy of the screening process. That is, during the screening process a wider range allows RSS to screen out individuals who are close to the desired age but would claim to be the exact desired age if they knew it.
Second
Follow-up of the High School Longitudinal Study of 2009 (HSLS:09) –
Cognitive Interviews
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Chapter 2 |
| Author | spowell |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |