National Center for Education Statistics
Volume I
Supporting Statement
2012/17 Beginning Postsecondary Students Longitudinal Study (BPS:12/17) Cognitive Testing – First Round
OMB# 1850-0803 v.134
Attachments:
Attachment I – Respondent Recruitment Procedures and Materials
Attachment II – Consent to Participate in Research
Attachment III – Eligibility Screening Questions
Attachment IV – Cognitive Interview Protocol
Attachment V – Cognitive Interview Survey Items
April 2, 2015
The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB # 1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting field tests, pre-tests, focus groups, and cognitive interviews.
This request is to conduct cognitive interviews in preparation for the 2012/17 Beginning Postsecondary Students Longitudinal Study (BPS:12/17) with recruitment starting in April of 2015.The data collection for this study is being carried out for NCES by RTI International – with Research Support Services (RSS) as a subcontractor – under contract to the U.S. Department of Education (Contract # ED-IES-09-C-0039). The cognitive research report summarizing the results will be available by July 15, 2015, and the results from this first round of cognitive testing will be presented for discussion with a Technical Review Panel (TRP) scheduled to convene in September 2015. This submission includes recruitment and screening materials, the informed consent form, the questions to be tested through the initial cognitive interviews, and the protocol for conducting the cognitive interviews.
The BPS longitudinal study is designed to follow a cohort of students who enroll in postsecondary education for the first time during the same academic year, irrespective of the date of high school completion. The study collects data on student persistence in and completion of postsecondary education programs; their transition to employment; demographic characteristics; and changes over time in their goals, marital status, income, and debt, among other indicators. Data from BPS are used to help researchers and policymakers better understand how financial aid influences persistence and completion, what percentages of students complete various degree programs, what are the early employment and wage outcomes for certificate and degree attainers, and why students leave school.
Sampled from the 2011-12 National Postsecondary Student Aid Study (NPSAS:12), BPS:12/17 is the second follow-up interview with sample members who were first-time beginning college students (FTBs) during the 2011-12 academic year. The first contact was made in 2012 as part of the NPSAS:12 data collection, and the second in 2014 as part of BPS:12/14.
Topics for cognitive testing were considered against the overarching purpose of BPS to collect data on persistence, attainment, and labor market outcomes, as well as the human capital framework that informs data collection for this cohort.The items listed in this package will be included for cognitive testing because they are new to the BPS study, new for this BPS cohort, or present opportunities to improve existing items. The topics include online course taking; remedial course taking; familiarity with and participation in competency-based education and prior learning assessements; financial literacy; loan repayment and the impact of loans on employment; and the time it takes to earn a degree.
The requested cognitive testing will be used to refine the survey questions to maximize the quality of data collected, and provide information on issues with important implications for the survey design, such as the following:
The manner and degree to which terms in the questions and instructions are comprehended;
Memory search strategies commonly used, and appropriate cues to facilitate recall;
Memory demands of the questions and the answering strategies used;
The appropriate levels at which conceptual categories should be presented;
Respondents’ ability to make calculations and judgments;
Sources of burden and respondent stress; and
Cognitive testing of BPS:12/17 will be conducted for NCES by its contractor, RTI International, with RSS as a subcontractor. RSS – located in Evanston, Illinois – during recruitment anticipates screening up to 80 students in the Chicago metro area in order to ensure 20 interviews are completed during the first round of testing. Cognitive interview participants will be selected to provide representation of the specific populations of the BPS:12/17 study. This distribution includes students who were first enrolled between July 1, 2010 and June 20, 2012 at institutions with particular characteristics, including level of degree offered (less-than 2-year, 2-year, and 4-year) and institution control (public, private not-for-profit, private for-profit). The student sample will include both program/degree completers, those still enrolled, and those who left prior to completing a degree. In addition, the cognitive interview questions ask subjects about their experience with competency-based education and prior learning assessments. These programs are available at several institutions in the recruiting area, and subjects will be recruited from these institutions. A list of such institutions is included in Attachment I.
To recruit students, RSS will advertise on a variety of websites, such as Facebook and Craigslist, as well as use direct outreach, such as postings at area businesses. Attachment I presents the materials that will be used for recruitment of cognitive interview participants. Once a candidate expresses interest in participating in the cognitive interview, one of the designated recruiters will use a set of scripted screening questions (Attachment IV) to determine his/her eligibility for the cognitive interview.
Table 1, below, provides a distribution of the desired minimum number of respondents by type, although a respondent may fit more than one category.
Table 1: Desired minimum number of respondents, by characteristics
Characteristic |
Desired number of respondents |
Institutions with Competency-based Education and Prior Learning Assessments |
5 |
Institution control |
|
Public |
8 |
Private, for-profit |
8 |
Private, not-for-profit |
4 |
Institution level |
|
4-year |
8 |
2-year |
8 |
Less-than-2-year |
4 |
Student enrollment type |
|
Completer |
12 |
Leaver |
4 |
Still enrolled |
4 |
Trained RSS staff will conduct the cognitive interviews in rented conference rooms, with the goal of completing an average of five cases per day. The cognitive testing will involve intensive one-on-one interviews. The survey will be developed and tested as a paper-and-pencil instrument; the survey items to be tested are provided in Attachment V. Each interview will last a maximum of 60 minutes, and all participants will be offered an incentive of $40 to thank them for their time, effort, and participation. Audio recording of each interview will be available to NCES for review.
The organizing objective of the cognitive testing approach will be to identify the processes by which respondents answer draft survey questions and to pinpoint potential sources of question misinterpretation in their responses. For example, respondents will be asked to “think aloud” as they answer questions. Concurrent and retrospective protocols can provide a valuable source of evidence about the organization of information in memory, comprehension of the questions, strategies used in retrieving information, judgment processes that come into play, and other processes affecting the final answers to survey items. To elicit relevant responses, respondents may be asked to point out unfamiliar terms, to paraphrase the question or its accompanying instructions, to define a term, and to make judgments regarding the confidence they place in their answers. Typical probes – e.g., “How certain are you of your answer?” or “How easy or difficult was it to answer this question?” – seek to verify respondent interpretations, investigate the meaning of specific potentially ambiguous phrases, or elicit notions that the respondent thought were critically relevant to but absent from the question. A cognitive interviewing protocol (Attachment IV) will be used to guide the flow and content of the cognitive interviews. Instructions for cognitive interview respondents and probes inquiring about the meaning of key terms or constructs will be included in the protocol. Interviewers will refer to the protocol to guide the content of the interviews, but will also be free to deviate from the guide should participants have difficulty answering questions that do not have scripted probes or prompts.
Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, and highlight potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. RSS will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report.
Cognitive interview participants will be informed that their participation is voluntary and that their responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573). Participants will be assigned a unique student identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to the student’s name in any way. Participants will sign an informed consent form (Attachment II) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study, and which will be destroyed after the final report is completed.
RSS will begin recruiting for the cognitive testing round upon receiving OMB clearance, and the testing is scheduled to begin by early May 2015. Recommended changes will be made and the revised survey instrument presented to a panel of experts to be convened in September 2015. Based on their feedback, a second draft of the survey will undergo a second round of cognitive and usability testing, from October to December 2015. With feedback from the these two rounds of cognitive and usability testing, a final draft of the survey will be used in a pre-test with a sample of 1,200 students, beginning in February 2016. These future studies will be submitted to OMB for review and approval at a later date.
To yield 20 completed interviews, we anticipate screening up to 80 individuals for eligibility and to ensure that we are achieving the desired distribution of respondent types. The screening process, on average, is estimated to take about 5 minutes per person.
Table 2: Estimate respondent burden
Activity |
Number of respondents |
Number of responses |
Minutes per respondent |
Maximum total burden hours |
Screening |
80 |
80 |
5 |
7 |
Cognitive interview |
20* |
20 |
60 |
20 |
Study Total |
80 |
100 |
|
27 |
In order to encourage participation in this study, and to thank respondents for their time and effort spent traveling to the interview facility and completing the interview, each respondent will be offered $40 for their participation.
Participants in the cognitive interviews must bear direct costs associated with travel by car or use of Chicago Transit Authority services to RSS’s rented conference rooms in Chicago.
The cost to federal government for conducting these cognitive interviews will be $34,364, under the RSS subcontract to RTI, including recruitment, interviewing, analysis, report writing, and participant incentives.
Attachment I
Respondent Recruitment Procedures and Materials
Identification of Participants
RSS will focus recruitment efforts in the Chicago metropolitan area, particularly among postsecondary institutions. RSS uses a variety of recruiting methods to generate sufficient numbers of callers from potential respondents, including:
Outreach to educational institutions, organizations, community leaders, and affinity groups that can distribute flyers among targeted groups. Many of these organizations and venues have worked with RSS before. Because they are familiar with the populations they serve, these organizations are often able to help RSS target specific populations;
Flyers, print and online advertisements; and
Word of mouth.
Although not exclusively, particular efforts will be made to recruit respondents from institutions that offer competency-based education (CBE) and prior learning assessments in order to inform questions associated with these programs. A list of these institutions is presented below. In conjunction with RTI, RSS will identify institutional staff who oversee CBE programs for assistance in recruiting students. In addition, RTI has a relationship with College for America (CFA) at Southern New Hampshire University, which is a national CBE program built on project-based learning. RTI will request CFA’s assistance in recruiting participating students located in the Chicago area.
Chicago area institutions offering competency-based education and/or prior learning assessment programs
Institution Name |
Institution Type |
Argosy University-Chicago |
4-year, Private for-profit |
City Colleges of Chicago-Harold Washington College |
2-year, Public |
City Colleges of Chicago-Harry S Truman College |
2-year, Public |
City Colleges of Chicago-Kennedy-King College |
2-year, Public |
City Colleges of Chicago-Malcolm X College |
2-year, Public |
City Colleges of Chicago-Olive-Harvey College |
2-year, Public |
City Colleges of Chicago-Richard J Daley College |
2-year, Public |
City Colleges of Chicago-Wilbur Wright College |
2-year, Public |
DePaul University |
4-year, Private not-for-profit |
Moraine Valley Community College |
2-year, Public |
South Suburban College |
2-year, Public |
Triton College |
2-year, Public |
Advertisements will be targeted toward individuals qualified for this study, describing briefly the purpose of the cognitive interviews and the opportunity for respondents to contribute to the development of a survey. The advertisements will identify the requirement that participants began college between July 1, 2010 and June 30, 2012, as well as the time commitment of the interview, and the incentive amount for participation. Sample print and online advertisements are presented below.
Sample Recruitment Flyer
DID YOU START COLLEGE FOR THE FIRST TIME IN between 2010 and 2012?
You may be eligible to help us review a 60-minute education survey and receive $40 for your time!
We are interested in talking to individuals who first started college, university, or vocational/trade school at any time between July 1, 2010 and June 30, 2012, even if you are no longer enrolled in school.
We offer flexible appointment openings, including weekends and evenings, to meet your scheduling needs.
Research Support Services is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. Your responses will be used to refine survey questions.
Email us: xxx@ researchsupportservices.com
Call: 1-8XX-XXX-XXXX
Sample Online Ad
Title:
First started college between 2010 and 2012: receive $40 for a survey!
Ad text:
Did you start college for the first time between 2010 and 2012?
You may be eligible to help us review a 60-minute education survey and receive $40 for your time!
We are interested in talking to individuals who first started college, university, or vocational/trade school at any time between July 1, 2010 and June 30, 2012, even if you are no longer enrolled in school.
We offer flexible appointment openings, including weekends and evenings, to meet your scheduling needs.
Research Support Services is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. Your responses will be used to refine survey questions.
Email us:
xxx@researchsupportservices.com
Call: 1-8XX-XXX-XXXX
Recruitment
The recruitment screener, a 5 minute phone script (Attachment III), will be used to elicit both the educational status and eligibility criteria needed to test the items identified for cognitive testing. Potential respondents will call a toll-free number and an experienced recruiter will screen the caller for eligibility for the project. Screening includes an opportunity to provide the potential respondent with a brief description of the project, the purpose of the interview, and questions about the specific characteristics needed. RSS’s experience on past projects with recruitment suggests that it is important to elicit respondent characteristics without asking the same questions which are intended to test and thus priming the individual. RSS’s screening methods are designed to minimize the otherwise negative impact of the “professional respondents” on the quality of the study results.
Attachment II
Consent to Participate in Research
Title of Research: Beginning Postsecondary Students Longitudinal Study – Cognitive Interview
You, along with other students, are being asked to participate in in-depth interviews being conducted by RTI International and Research Support Services (RSS) for the National Center for Education Statistics (NCES), part of the U.S. Department of Education. The purpose of the interviews is to obtain your feedback on some of the questions that will be used in the Beginning Postsecondary Students Longitudinal Study.
You are one of 20 individuals participating in these cognitive interviews.
The interviews will be audio recorded to make sure we don’t miss anything that you say and to help us write a report summarizing the results of the interviews. Upon completion of the written report, the recordings will be destroyed. Your name will never be used in the report that we write.
Your participation in an interview will take about 1 hour.
We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.
You will receive $40 to thank you for your participation in this study.
These interviews will help us develop and improve the questionnaires used in the national study so that they are clear and can be easily understood and answered.
Research Support Services (RSS) and RTI International are conducting this research for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (20 U.S.C., § 9543). Your responses may be used only to help inform the survey design and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C. § 9573).
We will not contact you in the future as part of this study.
Your decision to take part in this research study is completely voluntary. You can refuse to answer any question and you can stop participating at any time.
If you have any questions about the study, you may call Dr. Alisú Schoua-Glusberg at Research Support Services (847-864-5677) or Jason Hill at RTI International (919-541-6425). If you have any questions about your rights as a study participant, you may call RTI’s Office of Research Protection at 1-866-214-2043 (a toll-free number).
YOU WILL BE GIVEN A COPY OF THIS CONSENT FORM TO KEEP.
Your signature below indicates that you have read the information provided above, have received answers to your questions, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.
|
|
|
Signature of Participant |
|
Date |
|
|
|
Printed Name of Participant |
|
|
I certify that the nature and purpose, the potential benefits, and possible risks associated with participating in this research have been explained to the above-named individual.
|
|
|
Signature of Person Obtaining Consent |
|
Date |
|
|
|
Printed Name of Person Obtaining Consent
|
|
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Wine, Jennifer S. |
| File Modified | 0000-00-00 |
| File Created | 2021-01-27 |