Justification

National Center for Education Statistics

Volume I

Supporting Statement

NCER- NPSAS Grant Study – Connecting Students with Financial Aid (CSFA) 2017 Cognitive Testing

OMB# 1850-0803 v.191

February 2017

Attachments:

Attachment I: Recruitment Procedures

Attachment II: Consent to Participate in Research

Attachment III: Sample Text for Interview Scheduling and Confirmation

Attachment IV: Interview Protocol

Attachment V: Interview Guide and Notes Form

Attachment VI: Survey

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) generic clearance agreement (OMB# 1850-0803), which provides NCES the capability to improve data collection instruments by conducting testing, such as usability tests, focus groups, and cognitive interviews, to improve methodologies, survey questions, and/or delivery methods. This request is to conduct cognitive interviews during March-April of 2017 with postsecondary college students in order to refine the wording of survey items for the Connecting Students with Financial Aid (CSFA) 2017: Testing the Effectiveness of FAFSA Interventions on College Outcomes study (OMB #-1850-0931). This submission describes the cognitive testing recruitment, screening, and interview procedures designed to ensure quality, performance, and reliability of the tested items.

Background

In 2010, the National Center for Education Research (NCER) and the National Center for Education Statistics (NCES), both within the U.S. Department of Education’s Institute of Education Sciences (IES), began collaborating on an education grant opportunity related to the cross-sectional National Postsecondary Student Aid Study (NPSAS). Under the NCER-NPSAS grant opportunity, researchers could submit applications to the Postsecondary and Adult Education topic within the Education Research Grants program (CFDA 84.305A), under either the Exploration or Efficacy and Replication research goal. Consistent with these two goals, NCER supports research projects using NPSAS to: 1) explore relationships between malleable factors (e.g., information on benefits of financial aid and FAFSA renewal) and postsecondary persistence and completion, as well as the mediators and moderators of those relationships; and 2) evaluate the efficacy of interventions aimed at improving persistence and completion of postsecondary education (e.g., financial aid and FAFSA renewal advice delivered via mail and email). Researchers approved for funding through this program can obtain indirect access to a subsample of the national NPSAS sample (after the study’s student interviews are completed) in order to conduct unique research projects that adhere to the guidelines set forth in the Request for Applications for the Education Research Grants Program, as well as guidelines set forth by NCES and the NPSAS program.

On July 1, 2016, two grants were awarded using a subsample of 2015-16 NPSAS sample members:

• Could Connecting Students with Financial Aid Lead to Better College Outcomes? A Proposal to Test the Effectiveness of FAFSA Interventions Using the NPSAS Sample (referred to as “Connecting Students with Financial Aid (CSFA) 2017”; http://ies.ed.gov/funding/grantsearch/details.asp?ID=1853), and

• Financial Aid Nudges: A National Experiment to Increase Retention of Financial Aid and College Persistence (http://ies.ed.gov/funding/grantsearch/details.asp?ID=1848).

The request to conduct the CSFA study was approved in January 2017 (OMB #1850-0931 v.1). This submission supports cognitive testing of the survey collection phase of the CSFA study, which investigates whether an intervention that provides financial aid information to students in the form of letters and emails increases completion of the Free Application for Federal Student Aid (FAFSA). In addition, information is provided on how taking more college credits can increase the amount of financial aid received to see if this information influences enrollment intensity (full- versus part-time status). The primary grantee is Bridget Long, Harvard University (Grant Award #R305A160388), and the co-principal investigator is Eric Bettinger, Stanford University. Data collection for the study will be led by the NCES contractor, Research Triangle Institute (RTI).

The cognitive interviews are designed to provide information that may help us refine the existing survey language and organization of the survey tool, by examining:

• the extent that language terms in questions are comprehended correctly;

• the extent to which students’ possible answers are adequately captured by the multiple-choice options;

• the thought processes used to arrive at answers to survey questions; and

• potential sources of burden and respondent stress.

Design and Context

The cognitive interviews will be conducted from March to April 2017 in preparation for the CSFA survey that will be administered from October to December 2017. Attachment VI presents the survey items to be tested. Appropriately trained and experienced researchers will conduct the cognitive interviews, including Dr. Long and Dr. Bettinger (co-PIs) and CSFA Project Manager Lena Eisen. The interview protocol contains both open-ended questions and target probes to explore participants’ opinions, decisions, and understanding of the survey questions and associated terminology.

The cognitive interviews will be conducted with current postsecondary students in the greater Boston, MA; Denver, CO; and Stanford, CA areas. Interview opportunities will be announced using a variety of methods including networking referrals and invitations distributed via email, public postings, and social media announcements. We expect to reach out to up to 150 people to achieve our goal of interviewing up to 40 cognitive interview participants.

Respondents will be scheduled to participate in individual interview sessions. The sample will include individuals who are currently enrolled at least part-time at a postsecondary institution as of the time of the interview, yielding respondents that are likely to have similar characteristics as those who will participate in the CSFA Survey collection in the fall 2017. In our outreach, we will especially target low-income and first generation college students. So that we may capture additional perspective and feedback, the sample may also include respondents who are not currently enrolled students, but have professional experience working with postsecondary students and so can reflect on the understanding and opinions of the target population. All respondents will be informed of the voluntary nature of their participation in the interview and complete an informed consent form (Attachment II) if they agree to participate.

The interviewer will then administer the survey tool with additional probes as needed (see Attachment IV) to better learn whether the respondent understands the questions and response options contained within the survey. If the respondent does not understand the information, they will be asked whether and how the questions and/or responses could be improved to increase comprehension or provide options of responses that are more appropriate to their desired answers.

The estimated length of the CSFA survey is up to 10 minutes, so we predict that each cognitive interview, held in-person or remotely, using the online survey tool and real-time telephone communication, will last approximately 30 minutes, and no more than 60 minutes. Detailed notes (Attachment V) will be documented during each interview. Immediately following the conclusion of an interview session, the interviewer will organize his/her observations and summarize the experiences and insights from each respondent.

Attachment I in this submission presents the process and sample language that will be used for the recruitment of cognitive interview participants; Attachment II contains the Consent To Participate and Assurance of Confidentiality; Attachment III the sample email for interested participants and sample confirmation email for scheduled participants; Attachment IV the informational interview protocols; Attachment V the form used by interviewers to aid documentation of each session; and Attachment VI the CSFA survey tool.

Estimated respondent burden

We anticipate a yield of up to 40 cognitive interviews. Prospective respondents that initiate contact with the research team will receive an informational email (Attachment III; Part A) that will take 1 minute to review. It will take an average of 3 minutes for eligible respondents to schedule an interview time via an online scheduling tool. It may take 1 minute to review the email sent by researchers to confirm the scheduled interview time (Attachment III; Part B). The interview process will last approximately 30 minutes, and no more than 60 minutes.

Table 1: Estimate respondent burden

Activity	Number of respondents	Number of responses	Minutes per respondent	Maximum total burden hours
Invitation and Scheduling	150	150	5	13
Survey with Interview	40	40	60	40
Study Total	150	190		53

Note: Number of respondents is an unduplicated count.

Estimate of Costs for Recruiting and Paying Respondents

No monetary incentive will be given to participants.

Estimate of Cost Burden

There are no direct costs to respondents.

Cost to Federal Government

There will be no cost to the federal government for conducting these interviews.

Assurance of Confidentiality

Interview respondents will be informed that their participation is voluntary and that :

This study is authorized by law under the Education Sciences Reform Act (20 U.S.C., § 9543). All responses that relate to or describe identifiable characteristics of individuals may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

No personally-identifiable information from respondents will be included in the summary of results. Respondents will be assigned a unique identifier (ID), which will be used soley for summarizing input. The ID will not be linked to the respondent’s name in anyway. The signed consent forms (Attachment II) will be kept separately from the interview response files for the duration of the study and will be destroyed after the survey is finalized.

Schedule for Cognitive Interviews

The interviewers will begin recruiting for interview sessions upon receiving OMB approval, and interviews may be scheduled to begin within one week of OMB clearance. The CSFA study survey is authorized and scheduled for use with approximately 14,000 sample members, beginning in October 2017.

Schedule of activities

Recruit participants	March 2017
Cognitive Interviews	March/April 2017
Finalize revisions to the survey tool (if needed)	April/May 2017