Qualitative Testing Sumary Student Records

This appendix summarizes the results of qualitative testing conducted in preparation for the 2019-20 National Postsecondary Student Aid Study (NPSAS:20) institution data collection. This testing included virtual focus groups with institution staff and usability testing of the Postsecondary Data Portal (PDP) website. Full details of the pretesting components were approved in December 2018 and March 2019 (OMB# 1850-0803 v.243 and 247). The study design is described first, including discussion of overall conclusions, sampling and recruitment procedures, methods for conducting the virtual sessions, and data processing. Next is discussion of detailed findings from the focus group sessions, followed by detailed findings from usability testing.

Qualitative Testing Summary

The National Postsecondary Student Aid Study (NPSAS:20), conducted by the National Center for Education Statistics (NCES), collects student data directly from postsecondary institutions. In order to improve the quality of the data collected as well as reduce the burden of completing the data request for institution staff, RTI International, on behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, contracted with EurekaFacts to conduct virtual focus group sessions and usability testing interviews with institution staff who are responsible for completing the NPSAS institution data request via the Postsecondary Data Portal (PDP).

In general, the focus groups and usability testing sessions addressed the following topics:

Timing of the data request;
Instructions and resources provided to institution staff;
Content of the data collection instrument and the PDP website;
Navigability of the PDP website;
Ease of finding needed information; and
Ease of providing data via the offered modes (Web, Excel, or CSV).

Participant Sample

Participants for focus group and usability testing sessions were drawn from a list of individuals who completed the most recent round of NPSAS data collection, the 2017-18 NPSAS Administrative Collection (NPSAS:18-AC). Participants were recruited to obtain feedback from a variety of institution sizes, institution sectors, and the participants’ roles or departments within the institution. Twenty individuals participated in one of seven focus groups, and eighteen individuals participated in one-on-one usability interviews. See table 1 for details about participant characteristics.

Table 1. Qualitative testing participants, by institution and individual characteristics, 2019
Participant Characteristics	Focus Groups (n = 20)	Usability Testing (n = 18)
Institution size
Small (< 100 students)	8	7
Medium (100 – 200 students)	3	8
Large (> 200 sample students)	9	3
Institution type
Private, for-profit, 4-year	1	2
Private, not-for-profit, 4-year	4	8
Public, 4-year	6	4
Public, 2-year	9	4
Participants’ university department
Institutional Research/Institutional Effectiveness/Institutional Planning	10	5
Financial Aid	5	6
Registrar/Enrollment Management/Student Services	4	2
Bursar/Student Accounts/Student Finance	1	1
Information Technology/Data Management	0	4
Participants’ years of experience working in postsecondary education
Less than 1 year	0	0
1 - 4 years	3	1
5 – 10 years	6	4
10 – 15 years	4	3
15 – 20 years	2	4
20+ years	5	6

Key Findings

This section summarized the key findings of focus groups with institution staff and usability testing of the PDP website.

Timing of the Request. Two common themes emerged from participants’ discussion of the timing of the NPSAS data collection request:

The introduction letter sitting in someone else’s office for a period of time before being forwarded to the appropriate staff, leaving a shorter actual period of time to perform the data collection; and
The request arriving during a busy time of year, when institutions have other data requests to fulfill.

Reference Period/Term Structure. The reference period for NPSAS:20 is July 1, 2019 through June 30, 2020 (i.e., “the NPSAS year”), which aligns with the federal financial aid year. For NPSAS, institutions are asked to report students’ enrollment status for each term in the NPSAS year and the tuition charged, budgeted costs of attendance, and financial aid received for this period. Participants noted that the July 1 through June 30 NPSAS year presented challenges to institution staff because the period tends to cross two different academic years for most institutions. This overlap led to confusion about which summer terms to include, depending on the institutions’ term structure. Most participants reported that their institution uses a traditional semester-based term structure, and some consider summer terms to be part of the same academic year as the following fall/winter term (referred to as a “leading summer” or “summer header”), while others consider summer terms to be part of the same academic year as the preceding spring term (referred to as “summer trailing” or “following summer”). As a result, the data reported for NPSAS, which requests data for both years of summer terms, did not align with the institutions’ defined academic years. Participants from these term-based institutions would prefer to base their reporting by term rather than the current NPSAS year reference period.

Instructions/Functionality. Participants were largely able to understand the different mode options for providing the data and the conditions on which the mode recommendations are based. Most participants selected either the Excel mode or the CSV mode and generally had minimal difficulty completing the task. For the most part, participants reported the instructions for completing both the Excel and CSV modes to upload data were clear and easy to understand. One area of confusion was the list of critical items and when conditionally critical items (those that are only critical based on responses to other items) should be reported. Some Excel mode participants reported usability issues related to the fact that the template is locked and some formatting functions (e.g., sorting and filtering) are controlled; these participants also reported understanding the need to lock the template.

Error Messages. A few issues related to error messages were identified by the participants, including:

Both Excel and CSV mode users reported issues with error messages when leaving fields blank because they did not have that data. This was a particular challenge for participants from large institutions who reported that searching through the “hundreds” of error messages looking for critical errors was difficult and time consuming.
Excel users expressed a desire for a way to accurately validate their data file before uploading it to the PDP website. One of the participants advocating for this change in the review process needed to upload the file “five or more times before it was finally accepted.”
Some participants mentioned having uploaded files that were “accepted with errors” which they found confusing.

Budget. Some participants reported difficulty providing data for the budget section, particularly for students who did not apply for aid and therefore did not have an individualized budget. Participants reported that preparing a budget for these students can be a manual, and therefore time-consuming, process.

Financial Aid. Participants reported some confusion regarding the NPSAS year reference period and which financial aid awards should be reported. Participants reacted positively to the proposed change to (1) collect financial aid based on the term(s), rather than the NPSAS year reference period; and (2) to collect financial aid disbursed. Some participants noted that reporting individual awards is time-consuming, preferring to report a lump sum of aid received by the student.

Ease of Finding Needed Information. The instructions provided for completing the different pages and submitting the data appear to satisfy the needs of most participants. Participants appreciated the Contact Materials and Resources links in the Help Menu and consistently reported the various links were useful and helpful to them. Some participants felt that the Announcements link should be more pronounced or indicate that new announcements have been added.

Content and Navigability of the Website. Participants were able to move through the PDP easily and reported that the overall organization of the website was logical. Participants especially appreciated that the Task Menu page shows which tasks are completed and which are remaining. In addition, although many participants reported they would not usually watch instructional videos unless they had questions, several indicated the videos presented useful information and they liked the short length.

Student Enrollment List. Participants reported that the Student Enrollment List instructions were straightforward, easy to follow, and unproblematic; however some participants reported some areas of confusion on the Student Enrollment List page on the PDP.

Study Design

Recruitment and Screening

Participants for both focus groups and usability testing sessions were recruited from a list of institution staff who participated in NPSAS:18-AC. To be eligible to participate, participants had to be employed at an institution that participated in NPSAS:18-AC, responsible for providing some or all of the data for NPSAS:18-AC, and comfortable speaking in front of a team of researchers or other postsecondary educational professionals.

For focus groups, institution staff were divided into four categories based on their department at the institution and/or the sections of the NPSAS:18-AC student records collection they were responsible for completing:

Bursar/Student Accounts/Student Finance
Financial Aid
Institutional Research, Institutional Effectiveness, Institutional Planning
Registrar/Enrollment Management/Student Services

For usability testing, the mode used to provide data on the Postsecondary Portal (PDP) during NPSAS:18-AC was also considered in order to capture a variety of experiences using the three different modes (CSV mode, Excel mode, and Web mode).

All recruitment materials, including but not limited to initial outreach communications, Frequently Asked Questions, reminder and confirmation emails, and informed consent forms, underwent OMB approval (OMB# 1850-0803 v. 243). Recruitment materials were distributed through email using the contacts provided by RTI. Respondents first received a recruitment email before receiving a follow-up phone call from EurekaFacts staff.

In order to ensure that respondents met the qualification criteria, all potential participants completed a screening survey and then were screened by EurekaFacts staff over the phone. During screening, all participants were provided with a clear description of the research, including its burden, confidentiality, and an explanation of any potential risks associated with their participation in the study. Eligible participants who were screened by EurekaFacts staff over-the-phone were scheduled at the time of screening.

Participants were recruited to vary institution size, institution sector, and years in post-secondary education as much as possible. All participants were offered a $50 Amazon e-gift card as a token of appreciation for their efforts, although not all participants accepted an incentive.

To ensure maximum “show rates,” participants received a confirmation email that included the date, time, a copy of the consent form, and directions for participating in the virtual session. All participants received a follow-up email confirmation and a reminder telephone call at least 24 hours prior to their session to confirm participation and respond to any questions.

Data Collection Procedure

All focus group and usability testing sessions were conducted virtually using Adobe Connect software. Data collection followed standardized policies and procedures to protect participants’ privacy, security, and confidentiality. Upon their arrival to the virtual Adobe Connect meeting participants were welcomed and instructed to set up their microphones and speakers. Written consent was obtained via email prior to the session. The consent forms (which include participants’ names) were stored separately from their data and were secured for the duration of the study.

Shape1 Prior to each session, a EurekaFacts employee created an Adobe Connect meeting with a unique URL. All participants were sent a confirmation email that included the unique link for the Adobe Connect meeting (see figure 1). In order to allow enough time for technological set-up and troubleshooting, participants were requested to enter the Adobe Connect meeting room fifteen minutes prior to the start time of the session. When this time arrived, the moderator adjusted the security of the virtual room to allow participants to enter. Participants were greeted by a welcome slide that instructed them to connect their speaker, microphone, and webcam to the Adobe Connect software. However, participants were directed to keep their microphones muted and webcams off until it was time for the session to begin.

At the scheduled start time of the session, participants were instructed to turn on their microphones and webcams before being formally introduced to the moderator. Participants were then reminded they were providing feedback on the NPSAS:18-AC student records data collection and reassured that their participation was voluntary and that their answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law. Participants were also informed that the session would be recorded.

At the end each session, participants were thanked and informed when to expect the virtual $50 Amazon e-gift card. The recording of the session was then terminated and the Adobe Connect meeting ended to prevent further access to the virtual room.

Focus Group Procedure

EurekaFacts conducted seven, 90-minute, virtual focus groups between April 30 and May 30, 2019. Six focus groups were arranged by category so that each group was only administered probes relevant to the sections of data collection they completed. The final “make-up” group included participants from multiple categories. Financial Aid and Bursar/Student Accounts/Student Finance categories were combined into the same sessions because of the similarities between the two roles in NPSAS:18-AC. The three focus group session categories were as follows:

Financial Aid and Bursar/Student Accounts/Student Finance (FA) (3 sessions)
Institutional Research, Institutional Effectiveness, Institutional Planning (IR/IE) (2 sessions)
Registrar/Enrollment Management/Student Services (R) (1 Session)
Make-up session (multiple categories) (1 Session)

Focus group sessions were conducted using the OMB-approved moderator guide. The moderator guide was used to prompt discussion of staff members’ experiences completing the different data collection tasks in addition to focusing on the NPSAS Help Desk, methods of communication, and proposed data feedback report. While the moderator relied heavily on the guide, focus group structure remained fluid and participants were encouraged to speak openly and freely. The moderator used a flexible approach in guiding focus group discussion because each group of participants was different and required different strategies to produce productive conversation.

Topics for each session varied based on participants’ department at the institution. The following topics were discussed in the focus groups:

Topic 1: Enrollment List Collection (R and IR/IE groups)
Topic 2: Student Records Role Assignments (all groups)
Topic 3: Institution Information (R and IR/IE groups)
Topic 4: Student Records Modes (all groups)
Topic 5: Enrollment (R and IR/IE groups)
Topic 6: Budget (FA groups)
Topic 7: Financial Aid (FA groups)
Topic 8: Help Desk & Other Resources (all groups)
Topic 9: Giving Back (all groups)
Topic 10: Communication (all groups)
Topic 11: Closing (all groups)

The goals of the discussion were (1) to gain insight about the participants’ experiences in order to identify problematic processes, specifications, questions, terms, and/or resources; determine how respondents are able to coordinate data collection at their respective institutions; (2) to gauge how easily participants are able to enter student records data across the PDP modes; (3) to explore the interest of participants in receiving a data feedback report and/or follow-up “thank-you” for participating in data collection; and (4) to review the methods used by NCES to communicate information and instructions about the study.

Usability Testing Procedure

EurekaFacts conducted eighteen, 90-minute, virtual usability testing interviews between May 7 and June 17, 2019. Participants were asked to work with an interviewer to complete several tasks on the Postsecondary Data Portal (PDP) website, share their personal experiences with the website, and provide their impressions and suggestions on the usability of the PDP website and its tools.

Shape2 Interview sessions were conducting using an OMB-approved interview protocol. When prompted by the interview protocol, a trained EurekaFacts interviewer requested the participant share their screen before using the web link feature within Adobe Connect to launch the PDP website on the participant’s browser (see figure 2). The participant was also given a moment to clear their screen of any personal information prior to accepting the screen share request. After launching the PDP website, the interviewer shared a unique login with the participant to access the PDP prototype for the individual’s institution.

The interviewer protocol was used to prompt discussion of participants’ experiences with

using the PDP website to complete tasks;
using the PDP website’s data entry features to provide student records data in either CSV, Excel or Web mode;
utilizing and understanding the resources provided by the PDP website; and
navigating the PDP website.

During the interview, the interviewer asked the participant to complete several tasks, such as registering their institution and reviewing various features of the website. The interviewer also asked the participant which sections of the PDP tasks they have previously completed in order to focus on sections most relevant to the participant and their personal responsibilities. During the interview, participants were asked to select a PDP mode to evaluate and practice providing student record data. Each mode (CSV, Excel, or Web) had a unique set of questions to evaluate the usability of the data entry features. While the interviewer relied heavily on the protocol, the interviewer took a flexible approach to guide the interview because each participant had different experiences using and navigating the PDP which required different strategies to produce productive dialog and encourage the participant to speak openly and freely.

Some topics were not administered for all interviews based on the personal responsibilities of individual participants. The following topics and sections were included in the interview protocol:

Topic 1: Introductions & Homepage
Topic 2: Study Icons
Topic 3: Task Menu
Topic 4: Resources
Topic 5: Institution Registration Page
Topic 6: Student Enrollment List
Topic 7: Student Records – Mode Selection
- 7a: Excel Mode
- 7b: CSV Mode
- 7c: Web Mode
Topic 8: Debriefing

The goals of the interview were (1) to gain insight about participants’ perceptions of the appearance, functionality, and understanding of the homepage and task menu; (2) to assess the usability and functionality of the help menu resources; (3) to review how easily participants can determine how to complete institution registration, the student enrollment list, and select a mode for student records; (4) to gauge how easily participants are able to enter student records data across the PDP modes; (5) to identify problematic processes, specifications, questions, terms, and/or resources; and (5) to evaluate which features throughout the website are most helpful for institution staff.

Coding and Analysis

The focus group and usability testing sessions were audio and video recorded using Adobe Connect’s record meeting function. After each session, a coder utilized standardized data-cleaning guidelines to review the recording and produce a datafile containing a high-quality transcription of each participant’s commentary and behaviors. Completely anonymized transcriptions tracked each participant’s contributions from the beginning of the session to its close. As the first step in data analysis, coders’ documentation of focus group sessions into the datafile included only records of verbal reports and behaviors, without any interpretation.

Following the completion of the datafile, it was reviewed by two reviewers. One reviewer cleaned the datafile by reviewing the audio/video recording to ensure that all relevant contributions, verbal or otherwise, were captured. In cases where differences emerged, the reviewer and coder discussed the participants’ narratives and their interpretations thereof, after which any discrepancies were resolved. The second reviewer conducted a spot check of the datafile to ensure quality and final validation of the data captured.

Once all the data was cleaned and reviewed, research analysts began the formal process of data analysis. In doing so, these staff looked for major themes, trends, and patterns in the data and took note of key participant behaviors. Specifically, analysts were tasked with identifying patterns within and associations among participants’ ideas in addition to documenting how participants justified and explained their actions, beliefs, and impressions. Analysts considered both the individual responses and the group interaction, evaluating participants’ responses for consensus, dissensus, and resonance. Once all the data was cleaned and reviewed, it was analyzed by topic area using the following steps:

Getting to know the data – Several analysts read through the datafile and listened to the audio/video recordings to become extremely familiar with the data. Analysts recorded impressions, considered the usefulness of the presented data, and evaluated any potential biases of the moderator.

Focusing on the analysis – The analysts reviewed the purpose of the session and research questions, documented key information needs, focused the analysis by question or topic, and focused the analysis by group.
Categorizing information – The analysts identified themes, trends, or patterns.
Developing codes – The analysts developed codes based on the emerging themes to organize the data. Differences and similarities between emerging codes were discussed and addressed in efforts to clarify and confirm the research findings.
Identifying patterns and connections within and between categories – Multiple analysts coded and analyzed the data. They summarized each category, identified similarities and differences, and combined related categories into larger ideas/concepts. Additionally, analysts assessed each theme’s importance based on its severity and frequency of reoccurrence.
Interpreting the data – The analysts used the themes and connections to explain findings and answer the research questions. Credibility was established through analyst triangulation, as multiple analysts cooperated to identify themes and to address differences in interpretation.

Limitations

Qualitative research seeks to develop insight and direction, rather than obtain quantitatively precise measures. The value of qualitative testing is demonstrated in its ability to provide unfiltered comments from a segment of the targeted population. While this research cannot provide definitive answers, the sessions can play a large role in gauging the usability and functionality of the online survey, as well as identifying any consistently problematic survey items and response options.

The key findings of this report were based solely on notes taken during and following the focus group and usability testing sessions. Note that not all probes were administered to every participant, and even when probes were administered, every participant may not have responded to every probe due to time constraints and the voluntary nature of participation. Moreover, focus groups are prone to the possibility of social desirability bias, in which case some participants agree with others simply to “be accepted” or “appear favorable” to others. While it is impossible to prevent this, the EurekaFacts moderator instructed participants that consensus was not the goal and encouraged participants to offer different ideas and opinions throughout the sessions.

Findings: Focus Groups with Institution Staff

This section presents detailed findings from the focus groups with institution staff.

Topic 1: Enrollment List Collection

Participants were asked to discuss their experiences providing their institution’s student enrollment list for NPSAS:18-AC (a list of eligible students enrolled at their institution between July 1, 2017, and April 30, 2018). This topic was presented to focus group sessions that included Institutional Research and Registrar staff who would be involved in preparation of the enrollment list, totaling 9 participants.

Participants did not experience major difficulties preparing and submitting their student enrollment list but noted that the process was “time consuming.” When asked to elaborate further as to approximately how much time it took to complete the task, responses ranged from “a few hours” to roughly “a week or so.” Common explanations included having to pull separate information from the database and then combine together, ensuring it was the correct information needed, and certifying that the information was correctly recoded and formatted. One participant noted that they were fairly new to their role as an Institutional Research Analyst, so it took time to become familiar with the process.

One participant noted that it is “tricky” to mark high school completion status because their institution, a public two-year technical school, does not track this information as some programs do not require it.

Topic 2: Student Records Role Assignments

This section addresses how roles and responsibilities are assigned to complete the student records section at each institution. Participants were asked to describe their role in providing data for the components of the student records data request: institution term-structure and student-level data in four content sections (General Student information, Enrollment, Budget, and Financial Aid).

Section Completion

Participants were first asked whether they completed all sections by themselves or if they received additional help to complete sections of NPSAS:18-AC. Of those participants who responded (n=12), ten participants indicated that the sections were completed by multiple people, while only two participants reported being the only person to complete all sections. In cases where the sections were completed by multiple people, other departments included the Financial Aid and Registrar’s offices. All those who were asked to elaborate about their experience working with different departments and staff members described having a good working relationship with other departments in general and reported the same positivity for NPSAS:18-AC data collection too. When asked about providing an estimated amount of time involved to complete the student records request, times ranged from more than a day to three weeks.

Challenges with Retrieving Data

Participants were asked to report any challenges they experienced retrieving data. The most common challenges mentioned by participants were as follows:

Miscommunication or delayed communication among different departments and staff,
Lack of access to requested data, and
Timing of when the request is received.

Shape3

"The Introduction Letter sat in the President’s office for many, many weeks before it was even brought to anybody’s attention…Once we heard of it, we were already behind schedule.”

Miscommunication or delayed communication among staff was the most frequently referenced challenge among participants. For example, one participant who noted experiencing communication issues explained that the NPSAS introduction letter was received by their institution’s President rather than those who complete the data request. They explained, “the introduction letter sat in the President’s office for many, many weeks before it was even brought to anybody’s attention…Even though the President does need to be aware of it, maybe the original letter should go to the actual people that do the survey so that they can be prepared.” The participant further explained, “Once we heard of it, we were already behind schedule.” Another participant experienced communication issues because the Introduction letter went to the Institutional Research department, but they were not informed that the data request had come in.

Of those who encountered the challenge of missing or insufficient tracking of data, one participant who works for a large 4-year university explained that although her office has access to their data system, budget data is not kept in that system and not to the level of detail requested by NPSAS. As a result, retrieving and accessing the data was especially difficult. Additionally, participants who indicated experiencing challenges with timing identified that when they do receive the request in September or October, institutions are already busy with other projects or commitments, so being provided a short turnaround time to submit all necessary information to NPSAS has been an issue. One participant, an Institution Research and Assessment Coordinator at a 4-year private not-for-profit university, suggested receiving the survey earlier in the year or later in the year such as late October for a November submission.

Assigning Responsibilities

Participants were then asked about how their institution decides on who is responsible for completing different components of the student records section and whether it was something easy or difficult to determine. Five participants responded to these probes. Two participants stated that they “take it upon themselves” to work through what is needed. Meanwhile, the remaining three participants discussed receiving assignments from someone else, two participants received their assignments from someone above them (i.e., Director or President), while the third participant reported getting the request from the Business Office. Generally, the participants who answered these probes did not experience issues with determining who is responsible for completing the sections. However, one participant did note that because their institution’s President received the introduction letter and not her, this caused some confusion as to who was responsible as “no one knew where to begin.”

Help Desk

Lastly, participants were asked whether in the future they would prefer to coordinate with other institutional data providers themselves or if they would prefer Help Desk staff to contact other individuals directly instead. All participants who responded to this item (n = 7) identified that they would rather coordinate on their own than have the Help Desk staff contact and prompt those individuals directly. Participants all found that having Help Desk staff involved would cause more confusion. Two participants also noted that based on their past experiences, those calls would end up re-routing to them anyways whether Help Desk staff was there or not.

Topic 3: Institution Information Page

The discussion for this topic centered around users’ experiences with completing the Institution Information section of the PDP, which collects the terms occurring at the institution in the NPSAS year. Participants were asked whether they would report students’ enrollment status (e.g., full-time, half-time, etc.) by month or by term. Furthermore, users were then asked to report their preferences and reactions to NCES pre-loading term data on the PDP based on previous rounds of the study, and users revising the data rather than manually entering this information. This topic was administered to the four focus groups that included Institutional Research staff, for a total of ten participants.

Term Structure

Participants were asked about their institution’s term structure (e.g. semesters, trimesters, continuous enrollment, etc.) and how they reported the summer terms in NPSAS:18 AC. One participant represented a continuous enrollment institution that does not fit into a traditional semester-based term structure, and the remaining nine participants represented institutions that operate on a traditional semester-based calendar system (e.g., fall/winter, spring, and summer semesters).

Reporting Summer Terms

For institutions that offer summer terms, the NPSAS year spans two academic years of summer sessions, and the instructions direct institutions to report both years. However, for administration and internal reporting, institutions associate summer terms with either the prior academic year (a summer “trailer” term) or the following academic year (a summer “header” term). Six out of ten institutions represented treat the summer term as a header.

When asked how they reported their summer terms for NSAS:18-AC, half (5 out of 10) of the participants reported including two years of summer terms because the instructions directed them to do so. Four participants reported some level of confusion about how to report the terms. As one participant explained, “July 1st would have been in the middle of our summer session for [2017] and June 30th would have been in the middle of our summer session [2018].” Another area of confusion is related to differences within the institution department. For example, one participant mentioned that their institution’s fiscal year treats the summer as a header term, while their financial aid department treats the summer as a trailer term and expressed concern about the data not being consistent.

Preloading Terms

All participants from term-based institutions reported that their term calendar is consistent from year to year and expressed a preference for the terms to be pre-loaded and reviewed rather than having to enter the terms for each round of NPSAS. None of these participants identified any potential challenges to pre-loading the terms, especially considering that they would be given the opportunity to make adjustments as needed. One participant from a continuous enrollment institution noted that it is a challenge for them to provide term data because they have many programs that have unique term schedules. The participant acknowledged, however, that entering the data by month (e.g., enrollment status for July 2017) is easy.

Topic 4: Student Records Modes

This section focuses on how participants determined which mode (Web mode, Excel mode, or CSV mode) to use to provide student records data for NPSAS:18-AC. Focus group participants also discussed their understanding of the instructions and resources provided for using Excel mode and CSV mode. This topic was administered to all seven focus groups for a total of 20 participants, but not all focus groups discussed every mode.

Selecting a Mode

Fourteen participants described how they determined which mode to use to enter student records data. Participants most frequently reported selecting the mode that was recommended by the PDP instructions as optimal for an institution of their size (n=8). Four participants explained they chose the mode with which they were most familiar, either because it had historically been used at their institution or because they had preexisting familiarity with it. The final two participants participated in previous rounds of NPSAS data collection prior to NPSAS:18-AC and had the opportunity to try both the Excel and CSV modes, reporting they selected the easier one (Excel) based on their personal experience with each.

Generally, participants felt satisfied with the mode options that are made available to them for student records data submission. In fact, given the opportunity to freely suggest alternative formats for submitting data, participants could not think of a format they would find more convenient than the current mode options.

Mode Selection Instructional Video

Participants were asked about whether they viewed the Mode Selection instructional video, and eleven participants provided responses. Five participants reported that they did not watch any videos on the pages of the PDP website, three participants could not remember whether they had watched them, and three participants confirmed that they had indeed watched the instructional videos. The participants who had watched the videos each reported that they were helpful and informative, with two confirming that they returned to the videos as an additional resource multiple times during the process of attempting the data request. One participant reported that they would be unlikely to watch an instructional video under any circumstance. Five of the eleven indicated a general appreciation for instructional videos, reporting they are likely to watch them, and six participants felt they would watch an instructional video if they were having issues accomplishing the task at hand.

Topic 5: Excel Mode

This section focuses on participants’ experiences providing student records data using the Excel template file. This topic was relevant to ten participants who provided data for NPSAS:18-AC using the Excel template, and was administered in five of the seven focus group sessions. Due to two participants joining their sessions in the middle of the discussion, eight participants were probed on template-specific items, while the full ten were probed on uploading and reviewing.

Excel Template

Data Entry. Seven of eight participants provided an explicit report on the method they used to input data into the Excel template, as one participant could not recall this information. While all seven participants reported copy/pasting data, only four relied exclusively on this input method. The other three participants copy/pasted most of their data but utilized an additional method for special cases: two reported needing to hand-key certain fields and one used the template’s drop-down menus wherever they existed.

Shape4

"For me, those types of help and information are invaluable.”

Help Text. All participants reported referencing the help text provided in the template. Generally, participants described the help text as helpful with one participant adding, “for me those types of help and information are invaluable.” However, two of eight participants felt that the help text did not provide enough information for some specific situations and still needed to call the Help Desk.

Participant Edits. Participants were asked to identify what they would change about the Excel template. Participants generally understood why the template was locked but noted that this limitation made data entry more difficult. Participants made recommendations to improve their experience providing data:

Adjust column widths to make more efficient use of the space on the screen.
Add the ability to filter data or freeze a reference column.
Add data validation to only accept certain characters (e.g., numerals in numeric data fields).

Uploading Data and Reviewing Error Messages

Most participants reported that they did not encounter any problems when uploading their Excel templates. Four participants reported that they received error messages on the completed files they uploaded. These error messages occurred because participants did not include “critical” data elements in their Excel template file; multiple participants mentioned the Ethnicity and Race data elements.

Participants who received error messages tended to upload a revised file as opposed to utilizing Web mode to edit the necessary fields. Even though all participants reported that they would not change anything about the way the error messages are displayed, reporting them as “straightforward” and easy to understand, participants did express a desire for a way to accurately validate their data file before uploading it to the PDP website. One of the participants advocating for this change in the review process needed to upload the file “five or more times before it was finally accepted.”

Topic 6: CSV Mode

Four of the seven focus groups discussed CSV mode, for a total of eight participants.

File Specifications

This section addresses ease of understanding the data requested within the file specifications and problems with data formats (i.e. decimal places, field lengths). Also discussed was how participants handled recoding the data to meet file specifications as well as any changes participants would make to the specifications to make them easier to use.

Understanding the file specifications. All eight participants reported having a general understanding of the file specifications. However, a few participants (2 out of 8) credited their ease in understanding the file to past participation in NPSAS or IPEDS [the Integrated Postsecondary Education Data System data collection effort]. Despite recalling some challenges with mapping or programming their data to the file, two participants agreed that the file was “fine” or “straightforward”.

Data formats. A few participants (3 out of 8) discussed issues regarding the file formats. Two Institutional Research (IR) staff mentioned that it would be helpful if the file specifications included item labels (rather than just item descriptions) to include in their data queries. One participant initially expressed confusion about how to code missing values, but then acknowledged that the specifications file included an instruction about leaving fields blank. Two participants also expressed confusion about whether they should include a “header” row in the uploaded file; they also noted that including a header row would aid in validating the information they were entering.

Another participant expressed confusion about the “Critical Items Y/N” field in the file specifications for items that are conditionally critical (i.e., only critical based on responses to other data elements). When probed about how to improve the instructions or error messages, the participant suggested adding a comment stating, "This element is not required, but if you have responded to [X] element with [Y] response. Then it is required."

Recoding data. Four out of eight participants stated they did not have to recode data. Two reported that they made sure their queries would format the results as indicated in the file specifications, one stated it was more of a problem solving exercise to assure they were pulling the right data, and the fourth reported they had their “database programmer” develop a template that would format the data correctly to meet the file specifications. Two participants added that when it was not possible to format the output correctly using the database queries, they would do additional recoding in Excel or SPSS. Another two participants mentioned they used SAS for coding and pulling data from various sources within their systems. One participant stated that getting data from other offices requires an additional step to re-name fields because they didn’t know how the other office labeled those fields.

Uploading Data Files

This next section focuses on uploading data, including discussion of any problems encountered while uploading, ease of understanding and resolving error messages, and editing the data.

Problems encountered uploading the CSV file. Participants reported no issues with uploading the CSV file, other than those noted below related to error messages.

Error messages. Four of the seven participants specifically discussed receiving error messages after uploading the CSV file. Two participants reported that the process went well and that the error messages clearly identified what the problem was and what to fix. The other two participants, both from large public institutions, reported encountering many error messages. One stated their institution has a large international population for whom race/ethnicity are not reported, thus they received numerous error messages for all these students which made finding the more critical errors difficult. The other participant explained that the error messages were difficult to understand, and they were not able to resolve all the errors due to the lack of clarification. They expressed confusion because, after reviewing their NPSAS:18 AC data, three of the four files uploaded with errors but were accepted.

One participant added, "It would be nice to have some clarification and to know the difference of a 'fatal error,' meaning something that is not going to work, versus a kind of a 'warning error,'" similar to the way they report this value in IPEDS. Participants also expressed confusion about whether errors prevented files from uploading.

All three of the participants who were asked whether they revised their files within the Web Mode or if they uploaded a revised file, reported they upload a revised file. Although not explicitly stated, two other participants indicated they also revised their files to fix the errors before uploading it again.

Topic 7: Enrollment

This section asked participants for feedback on data elements collected in the Enrollment section, including remedial course-taking and tuition charged. We asked staff to report on the status of remedial course-taking at their respective institutions before investigating how staff would report (or not report) such courses when completing the NPSAS data request. Additionally, this section reviews institution staffs’ impressions of how data on tuition charged is collected. This topic was included in five focus groups, totaling 16 participants.

Remedial Course-Taking

Shape5
Participants were asked whether their institution offers co-requisite courses (i.e., “gateway courses”) wherein students are taught both developmental and post-developmental content. Fourteen participants responded to this question, and while most (11 of 14) participants reported that their institution does offer co-requisite developmental courses, staff were divided on whether or not they would count those courses when responding to the remedial course-taking item (see figure 3).

Only a few participants confidently reported “yes,” they would consider co-requisite developmental courses to be remedial courses for the purposes of the item in figure 3 (3 out of 11). These participants felt this was the correct way to procced because a portion of the course content is below college or program-level. Participants were most likely to report that they would not consider the courses they identified as “gateway” or co-requisite developmental to be remedial (5 out of 11), noting the following reasons:

Two participants reported the co-requisite developmental course they had been considering was a program wherein students of international background take beginners-level English for college credit. Both participants felt they would not think of such courses as remedial because “they are English-as-a-foreign-language” which “is different than not meeting [native] English standards in terms of high school or college-level.”
One participant would not report her institution’s “gateway classes” on this item because at their institution the phrase describes a different type of course completely ―one in which all freshmen students are grouped together to learn “basic life skills.”
Another participant would not include such classes in response to the item in Figure 1 simply because they are too difficult to “tease out” of their data systems, despite recognizing that her 2-year-private institution does offer remedial courses that are co-taught with college-level classes.

Finally, three of 11 participants were not sure about how to correctly report co-requisite developmental courses on the item in question.

Staff noted the following areas of confusion:

Can a course be gateway/remedial/developmental while also satisfying a degree requirement?
Does the co-requisite developmental course have to be an actual course or would NCES also like data on students who are required to take part in remedial workshops, support programs, or other non-course-based developmental/remedial options?

Tuition Charged

Shape6
School staff were asked about challenges encountered in determining how much to report for each student’s tuition amount charged in the NPSAS year (see figure 4).

Five participants, all term-based institutions, reported challenges related to the NPSAS year reference period.

One such participant had students enrolled in summer term courses that were billed in June (before the reference period) but the courses ran into August (within the reference period).
Another participant reported needing to include a preceding summer which had a tuition amount different than the rest of the terms included, because as a summer-trailer institution the summer term was a part of a different academic year.

Shape7

"I basically left that field blank because it was very difficult to identify what they were charged based on the criteria of the student.”

Two participants reported difficulty generating tuition charged because of challenges with their respective institutions’ data systems. One did not clarify the system but felt their data may not be reliable and doubted whether the individual responsible for pulling the data would accurately interpret the many potential data elements in their system. The other participant reported using Ellucian Banner products and explained “I basically left that field blank because it was very difficult to identify what they were charged based on the criteria of the student… So, when you go into the system it is very hard to identify what the tuition was because a student will have multiple rows of the amount being charged.” The participant was also unable to get assistance from the institution’s Financial Aid Department due to them being overwhelmed.

Fifteen participants felt they would have an easier time submitting the NPSAS data request if the item wording requested tuition charged for specific terms (i.e., Fall 2019, Spring 2020) instead of the July to June NPSAS year reference period.

Other Discussion of Enrollment Data

Institution staff were also given the opportunity to discuss any additional challenges they encountered in the Enrollment section. A variety of challenges were discussed and raised by staff, including:

Two participants reported that providing enrollment information can be challenging for students that transfer into the institution (e.g., high school graduation dates and ACT/SAT scores, which are not known for transfer students).
Two participants reported that because graduate students are considered full-time with fewer credits/credit hours than undergraduate students, it became very time-consuming to collect data requested by half-time, three-quarter-time, less-than-half-time enrollment categories as they have to manually confirm the student’s type of enrollment before being able to proceed.

Topic 8: Budget

This section addresses participants’ understanding of the date elements collected in the Budget section; preferences regarding reporting a full-time/full-year budget versus other budget period options; whether or not participants include summer terms within the full-time/full-year budget; how budgets are adjusted when students change their enrollment pattern; how participants determine an individualized budget; and any additional challenges completing the Budget section. This topic was administered in four focus groups, totaling 11 participants.

Reporting Budget Data

Most participants reported challenges providing budget data. While one participant from a career technology institution explained that the budget section was easy for them because they have a calculated budget for every program, several participants found it challenging to report budgets for students, especially for those not receiving financial aid. Two main themes emerged during this discussion: degree of manual labor required to produce the data and concerns about consistency. Two participants noted that completing this section was difficult because of having different programs with different tuition rates. As a result, completing the section was a manual process of reviewing each individual student and determining whether they should use their standard cost of attendance or if they should prorate the student’s budget. Another participant also expressed concern about the consistency of the data being reported because they used a mix of standard Pell budgets and individualized budgets.

One participant noted that the similarity between the budgeted tuition data element (in the Budget section) and the tuition charged item (in the Enrollment section) and expressed confusion about why it was necessary to ask for both data elements. Some participants noted that it would help them interpret the instructions for the Budget section if they received more information about how the budget data will be used.

Budget Period

Institution staff were asked if they would prefer that the Budget section request only a full-time/full-year budget, which may require them to prorate students’ budgets, and whether their institutions’ full-time/full-year budgets would include summer sessions (i.e., a 9-month vs. 12-month budget). Just over half of the participants (6 out of 11) expressed a preference for this section to request a full-time/full-year budget, with some suggesting that requesting their Pell budget would be a common answer for all institutions. Six out of 11 staff described their full-time/full-year budget as a nine-month year. One participant suggested that the drop-down option for the budget period include a “full-time/full-year, 12 months” option and a “full-time/full-year, 9 months” option, rather than just full-time/full-year. One participant from a career technology institution suggested that the instructions should clarify the meaning of “full-time” in terms of the student’s clock hours (e.g. less than 24 hours, more than 12 hours). One participant noted that their Financial Aid office only codes students as "full time or part time,” but the budget period response options classify part-time students as “3/4 time,” “half-time,” or “less than half-time.”

Adjusting Student Budgets

Participants were asked what would happen to a student’s budget when a student’s enrollment pattern changes after the budget is prepared (e.g., the budget is prepared in the fall term for fall and spring enrollment, and then later the student decides to enroll in summer session). Three out of 11 participants reported they would re-budget the student to add the summer session to the student’s budget and increase their cost of attendance because the summer falls within the same academic year. Four additional participants reported that the situation does not apply to them because they are either a “summer header” institution and the new summer term would be included in the next academic year period or that they have “rolling terms” in that some programs start at different times of the year beyond the traditional September or January start.

Estimated Budgets

Institution staff were also asked how they would estimate a budget if an individualized budget was not available for a student. While two participants reported they always have an individualized budget, more than half the participants (6 out of 11) reported they would use a generic institution budget. Of these, one participant explained their estimated cost for attendance is based on residency status as they do not offer campus housing; the other five participants explained that they would use a generic budget as an estimated budget for students who did not complete a FASFA. The remaining participants were unable to respond as they were not responsible for student budgets.

Other Discussion of Budget Data

Shape8

“In financial aid speak, the actual term is ‘cost of attendance’ [rather than budget]… The law requires them to provide the cost of attendance for all students receiving aid for that term. If the section is referring to it as something different it can cause very different answers about awards and cost of attendance.”

Institution staff were also given the opportunity to discuss any additional challenges they encountered in the Budget section. One participant mentioned that it is difficult to include non-credit students and, similarly, another participant reported that if a student did not complete a FASFA, they were hard to include. Two participants mentioned that some of the data requested within the budget section is not typically used for developing budgets, with one wondering why they had to provide data points like “health insurance” and “commuter/technology” because it was “tedious.” One participant, a Director of Financial Aid from a large public two-year institution, noted that “in financial aid speak, the actual term is ‘cost of attendance’” rather than budget. They further explained that “The law requires them to provide the cost of attendance for all students receiving aid for that term. If the section is referring to it as something different it can cause very different answers about awards and cost of attendance.”

Findings: Usability Testing

This section presents detailed findings from usability testing of the Postsecondary Data Portal (PDP) website.

Topic 1: Home Page & Study Icons

Participants were reacquainted with the PDP, asked to review the PDP homepage, and share their experiences using the PDP during NPSAS:18-AC data collection. All 18 participants were administered this section.

Over half of the participants (11 out of 18) described their overall experience of using the PDP website or homepage positively, such as recalling their previous experiences as “straight forward,” “user-friendly,” and “fine.” Negative comments related to the secure procedures for accessing the PDP, specifically that the two-factor authentication to enter the user ID slowed things down for them. One participant described the PDP login process as “unnecessarily cumbersome.”

Study Icons

Participants were nearly evenly split between those who noticed the study icons at the bottom of the screen (9 out of 17) and those who did not (8 out of 17). A few participants noted appreciating the icons and links to resources located on the homepage. Other participants described the home page as “busy,” reported that they were “a bit confused about where I wanted to be.” Users most commonly reported not clicking on the icons because they were focused on the task at hand.

Datalab website

Due to time constraints, only fourteen participants were asked if they had ever visited the NCES Datalab website; nearly all participants (13 out of 14) stated that they had not noticed the Datalab link (see figure 5). Furthermore, six participants expressed that they were unsure of what the Datalab was or how to find it on the PDP log-in page. Almost all users (12 out of 14) stated that they would be interested in receiving materials from NCES informing them how to utilize the Datalab website.

Shape9

Topic 2: Task Menu

This section focuses on assessing participants’ understanding of the Task Manu page, the tasks necessary to complete NPSAS, and the order in which they should be completed. A total of 18 participants were administered this topic.

General Understanding and Usability

Shape10 Shape11

"Right at the beginning it shows you where to start, where you can register your institution, and then it gives a status showing the progress stage where you left off."

All participants expressed positive opinions regarding the overall layout and usability of the Task Menu and that the tasks seemed to be presented in the chronological order in which they should be completed. As one explained, "I think it's pretty good because it's not thrown at you all at once. Right at the beginning it shows you where to start, where you can register your institution, and then it gives a status showing the progress stage where you left off." Five participants mentioned that the side bar Help Menu was especially useful, as it put many useful links in one location. Participants generally understood what they would need to do on this page in order to proceed.

Some participants expressed confusion over seeing “NPSAS:20” on this page and were not sure if they should interpret the “20” as the year 2020. One participant stated that the distinction between the "Student Enrollment List" and "Student Record Data" sections was confusing.

Task Menu and Next Steps

Nearly all participants were able to correctly describe the different tasks in the Task Menu (see figure 6). Two participants, however, explained that they were not sure exactly what type of documents would be included in 'Archive Notes or Other Documentation,’ but did understand that they could find additional documents under this section. All but one participant were able to identify that 'Provide Student Record Data' was ‘In progress’ and that none of the tasks were completed. One participant mistakenly conflated ‘In Progress’ with ‘Completed,’ and is the only participant who did not explain that their first step would be to ‘Register Your Institution.’

Topic 3: Help Menu

This section focuses on assessing participants’ usage of various resources found on the right-hand side bar Help Menu (see figure 7). Seventeen participants were administered this topic.

Using the Help Menu

Ten participants recalled using some of the resources provided in the Help Menu. FAQs and Resources were most commonly reported by participants. When asked whether they found the information helpful, all 10 participants indicated “yes.”

Contact Materials Link

Shape13

“[Contact Materials] summarizes exactly what [the study] is… what the survey is about and the instructions.”

A majority of participants (16 out of 17) found the Contact Materials link very useful, useful, or somewhat useful. Most participants acknowledged receiving physical contact materials in the mail but found having a digital copy helpful if participants “misplaced those materials” or if someone else received the hard copy and was not in the same office as the participant completing NPSAS. More specifically, participants found the Welcome Letter, the Student Enrollment List Instructions, and NPSAS:20 Brochure especially helpful. Other participants noted its usefulness if they needed to confirm information such as the purpose or dates. A participant explained, “[Contact Materials] summarizes exactly what [the study] is… what the survey is about and the instructions."

Resources Link

Shape14

“[The Resources link] lists all the data that I need.”

Most participants (14 of 17) found the Resources link useful as “it lists all the data that I need” and it was described as “… a resource in my pocket that I can use.” Participants found this link especially helpful for those who may need additional assistance in understanding each section better or for any new staff who may need to familiarize themselves. The Student Enrollment List Instructions and Overview Video, the Quick Guide to Providing Student Records Data, the Confidentiality and Data Security Fact Sheet, and the FERPA Fact Sheet were most commonly cited as being the most helpful resources.

Announcements Link

Participants were evenly split when asked whether or not they found the information in the Announcements link to be useful, with eight participants reporting some level of usefulness and another eight participants reporting the link was not at all useful. The participants who did not find the link useful generally based their responses on the sample announcements that were posted on the test website rather than the general inclusion of announcements. Three participants further explained that they are unclear as to what information would be announced and why, while another participant offered several suggestions about the types of information that would be useful (e.g., upcoming deadlines, changes to the file layout). Those who found the Announcement link useful noted it had all announcements in one location so that it “is easy to find.”

Participants overall indicated they would like to receive alerts regarding any “urgent items” such as upcoming deadlines as well as any “new information or changes made.” Participants suggested having some sort of indication that a new announcement is available, such as a flag or number representing how many new announcements are pending their review.

What to Expect Link

Most participants (16 out of 17) found the timeline within the What to Expect link useful, specifically noting the visual display of when items are due as the most enjoyable. One participant exclaimed, “This is cool! … This is great for planning!” Although the timeline was considered useful, some participants wanted more specific information, like the exact dates that may show when “hovering” the mouse over the line, rather than just an overview.

NPSAS:20 FAQs Link

Shape15

"Most answers were very short and even for the longer answers, [its] listed in bullets so it was easier to read.”

Most participants (16 out of 17) found the information in the FAQs page very useful, useful, or somewhat useful. When asked to elaborate, participants generally agreed that it provides more information on questions that may occur during the process. One participant especially liked that “most answers were very short and even for the longer answers, [it’s] listed in bullets so it was easier to read.” Participants most commonly indicated the Student Enrollment List: Which Students to Include section as being the most helpful followed by the Background and Purpose of NPSAS.

Contacting the Help Desk

Almost all of the participants who reviewed this resource (9 out of 10) recalled reaching out to the Help Desk when completing NPSAS:18-AC. Five participants indicated using both phone and email while the remaining four participants used only one of the two methods to contact the help desk. Participants who were asked about whether it was easy or difficult to find the Help Desk all found it very easy or easy.

Live Chat Option

Ten participants were asked how likely they would be to use a live chat option if it was offered by NCES. Most participants (9 out of 10) expressed that they would likely use this option as it would allow for “immediate help” and would not be distracting to others around them, unlike if they were on the phone. Only one participant noted they would not utilize this option due to personal preference of speaking to a representative as it may bring up other questions they may have in mind.

Manage PDP Users

Three participants acknowledged using information found on the website to get access to the PDP for someone else. When asked how they would accomplish this task, other participants indicated they would either use the Manage PDP Users link or contact the Help Desk for assistance. Two participants explained they would just give their information to the potential user if they needed access to the PDP.

When asked to demonstrate adding a PDP user, most participants (8 out of 9) described it as very easy or easy to add a user. A participant explained, “All information was laid out in a way that was easy to understand.”

Topic 4: Registration Page

This discussion focused on the “Register Your Institution” section of the PDP. Participants were asked to indicate whether they needed input from other offices at their institution when providing the information collected in this section during NPSAS:18-AC. The Registration Page collects detailed information about institutions’ term structures, which was formerly collected as part of the student records phase of NPSAS. This topic was administered to a total of 13 participants.

Six participants explained that there is no additional information about their institutions that NCES should know about to improve their experience completing the study. Although there was no consensus on additional information that NCES should know about, staff mentioned collecting information about credit-to-clock hour conversions for institutions that do not use credits and using institutions’ Carnegie classifications to filter questions.

Completing the Registration Forms

While the NPSAS:18-AC Registration Page was organized on a single vertical-scrolling webpage, the Registration Page tested for NPSAS:20 was organized as a “form wizard,” in which participants viewed one question at a time on a separate form and advanced from form to form. No participants had difficulty navigating through the form wizard. Most participants (8 out of 12) did not find any words or parts of the Registration Page confusing, and among those who raised points of confusion, there were no consistent patterns of issues identified. For example, there was some minor confusion relating to the enrollment list cutoff date of April 30, as "April 30th probably would have happened long after a drop/add period had taken place.” One institution questioned whether the Registration Page referred to enrollment status for graduate or undergraduate students.

Assistance from Other Offices

A majority of participants (9 out of 12) reported not needing input from other offices at their institution to complete the Registration Page. Those who reported needing input from other institution offices indicated that they would receive input from staff in the Registrar, Academic Dean, Provost, Financial Aid, and President’s Offices. When asked how long it took to receive information from other offices and complete this step, reports ranged from a day to one week.