Justification

National Center for Education Statistics

Volume I

Supporting Statement

2017 Integrated Postsecondary Education Data System (IPEDS)

Time Use and Burden Cognitive Interviews Round 2

OMB# 1850-0803 v.192

March 2017

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB # 1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting developmental studies such as pre-tests, focus groups, and cognitive interviews.

This request is to conduct a second round of cognitive interviews to better understand institutional burden and time use for the Integrated Postsecondary Education Data System (IPEDS). The data collection for this study is being carried out for NCES by the American Institutes for Research (AIR) under contract to the U.S. Department of Education beginning in January 1, 2017. For this second round of the IPEDS time and burden study, a series of questions regarding time and burden will be tested on target participants through cognitive interviews in March and April 2017 in order to determine if they work as intended and uncover any comprehension issues and measure participants’ overall understanding of the content. These questions are based on findings from the first round of cognitive interviews. A summary of results from both rounds of this cognitive research and the recommendations for a future time and burden question are anticipated to be developed by mid April 2017. This submission includes recruitment procedures and materials, the informed consent form, the cognitive interview protocol, and survey items for round 2 of testing.

Background

IPEDS is a data collection system designed to collect basic data from postsecondary institutions. IPEDS enables NCES to report on key dimensions of postsecondary education such as enrollments, degrees and other awards earned, tuition and fees, average net price, student financial aid, graduation rates, student outcomes, revenues and expenditures, faculty salaries, and staff employed. The IPEDS web-based data collection system was implemented in 2000-01, and it collects basic data from the approximately 7,300 postsecondary institutions in the United States and its other jurisdictions that are eligible to participate in Title IV Federal financial aid programs. All Title IV institutions are required to respond to IPEDS (Section 490 of the Higher Education Amendments of 1992 [P.L. 102-325]). IPEDS allows other (non-title IV) institutions to participate on a voluntary basis. Approximately 200 non-Title IV institutions elect to respond. IPEDS data are available to the public through the College Navigator and IPEDS Data Center websites.

Data collected in IPEDS is used in numerous public facing data tools that facilitate the identification and comparison of postsecondary institutions for prospective students, policy makers, and researchers. As such, over time, the data in IPEDS has become increasingly more important and ubiquitous in the postsecondary industry. This may have resulted in institutions taking additional time to review and consider their IPEDS data before submitting them. In order to provide accurate burden time estimates for IPEDS data collections, NCES would like to develop a question or a short set of questions to collect accurate estimates of the time and burden involved in reporting IPEDS data. NCES proposes to use two rounds of cognitive testing to develop these questions. The cognitive testing will attempt to discern the different types of time use and burden that are related to responding to IPEDS. For example, IPEDS respondents may take time to:

1. Extract and tabulate information,

2. Compare completed IPEDS forms with previous submissions,

3. Brief institutional leaders regarding the information in the new submission, and

4. Enter the information into the IPEDS web based collection system.

Presently, IPEDS estimates that the collection annually burdens the industry approximately 999,060 hours based on 77,600 responses. These responses are based on the organization of IPEDS into 11 different survey components:

1. Institutional Characteristics

2. Completions

3. Fall Enrollment

4. 12-Month Enrollment

5. Student Financial Aid

6. Graduation Rates

7. Finance

8. Human Resources

9. Admissions

10. Academic Libraries

11. Outcome Measures

The first round of the IPEDS time and burden cognitive interviews (approved in November 2016; OMB# 1850-0803 v.177) determined that respondents are not consistent in how they report time and burden related to IPEDS. Based on the first round of interview data, a series of time and burden questions have been drafted to be tested in round 2. These include questions regarding the initial collection of the data that are reported in IPEDS, the number of people involved in the data collection and reporting process, the steps in the data collection and reporting process, and the initial purpose of the data collection.

Design and Context

The exploratory 2017 IPEDS Time Use and Burden Cognitive Interviews study is designed to better understand respondents’ time use and burden for the 11 IPEDS surveys. Based on cognitive interviews with administrators from 24 institutions, the study will recommend time use and burden interview questions to replace the existing time and burden questions in order to more accurately report how much time respondents spend annually completing the 11 IPEDS surveys.

During cognitive testing, interviewer(s) will use a structured protocol drawing on methods from cognitive science to investigate the process respondents use to answer burden related questions. In particular, these interviews will identify problems of ambiguity or misunderstanding in the time use and burden question that currently is used as part of the IPEDS surveys. The main goal of this study is to explore respondents’ understanding of the current time use and burden question, particularly what respondents include and exclude from their calculation when answering it, in order to determine whether respondents are providing the intended information. The results will drive possible recommendations for a revised question (or set of questions) that would more accurately reflect each institution’s time use and burden associated with annual IPEDS reporting.

The cognitive interviewing methods will consist of two key components: think-aloud interviewing and verbal probing techniques (these methods are also known as concurrent and retrospective recall probing, respectively). With think-aloud interviews, respondents are explicitly instructed to think aloud (i.e., describe what they are thinking) as they work through question items presented by the interviewer. With verbal probing techniques, the interviewers ask questions, as necessary, to clarify points that are not evident from the “think-aloud” process, or to explore additional issues identified as being of particular interest. Cognitive interview studies produce qualitative data in the form of verbalizations made by participants during the think-aloud phase and in response to the interviewer probes. Both the think-aloud approach and probing techniques will be applied to all participants during the cognitive interviews.

NCES has contracted AIR to administer the 2017 IPEDS Time Use and Burden Cognitive Interviews in January 2017 through April 2017. On behalf of NCES, AIR will recruit 24 institutions representing a range of characteristics (including institutional level and size). Table 1 provides a distribution of the desired minimum number of respondents by characteristics, although a respondent may fit more than one category. Please note that although the sample will include a mix of characteristics, the results will not explicitly measure differences by these characteristics. Participants will include administrators and staff from institutions that have experience in compiling and submitting data for IPEDS surveys. The interviews will be conducted via conference call, web conferencing, at institutions, or at AIR Washington D.C. headquarters. The availability of remote interviews will serve to encourage participation by offering flexible options. Institutions for in-person interviews will be recruited primarily from the District of Columbia metropolitan area to maximize scheduling and interviewing efficiency and flexibility.

Table 1: Desired minimum number of respondents, by characteristics

Characteristic	Desired number of respondents
Institution size
Small (less than 800 enrollment)	5-7
Medium (between 800 and 3000 enrollment)	11-13
Large (greater than 3000 enrollment)	5-7
Institution level
4-year	7-9
2-year	7-9
Less-than-2-year	7-9

NCES will provide AIR a list of possible institutions and contact information that meet the desired specifications (institutional level and size) to recruit for participation in the study. Emails and phone calls will be used to contact potential participants during recruitment. AIR’s experiences with recruiting administrators and staff at postsecondary institutions for cognitive interviews indicate that these individuals are a hard-to-reach population; thus, increased time and effort will be dedicated to meeting recruitment targets. AIR anticipates contacting up to 200 institutions during recruitment in order to ensure 24 interviews are completed representing the specific sub-populations of the IPEDS study. Attachment I presents the materials that will be used for recruitment of cognitive interview participants. Once a candidate expresses interest in participating in the cognitive interview, logistics for the interview will be set-up with the institution. Trained AIR staff will administer the cognitive interviews at the selected institutions, AIR Washington D.C. headquarters, web conferencing, or conference call with the goal of completing an average of 2 institutional interviews per day.

A cognitive interviewing protocol (Attachment III) and handout (Attachment IV) will be used to guide the flow and content of the cognitive interviews. Instructions for cognitive interview respondents and probes inquiring about the meaning of key terms or constructs will be included in the protocol. Interviewers will refer to the protocol to guide the content of the interviews, but will also be free to deviate from the guide should participants have difficulty answering questions that do not have scripted probes or prompts.

Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, and highlight potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. AIR will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that:

“American Institutes for Research is carrying out this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act (20 U.S.C., § 9543). Your participation is voluntary. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).”

Participants will be assigned a unique identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to their name. If interview is in person, participants will sign an informed consent form (Attachment II) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study, and which will be destroyed after the final report is completed. If interview is remote, participants will provide verbal consent which will be included in the digital audio recording.

Schedule for 2017 IPEDS Time Use and Burden Cognitive Interviews Study Round 2

The cognitive interviews are anticipated to begin in March 2017. A summary of results from this study will be prepared 7 to 8 weeks (mid-April 2017) after receiving OMB clearance for Round 2.

Estimate of Respondent Burden

To yield 24 completed interviews, we anticipate contacting up to 200 institutions for eligibility and to ensure that we are achieving the desired distribution of respondent types. NCES and AIR expect the cognitive interviews to last approximately 60 minutes in length. The initial contact and recruitment of each potential participant is estimated at 15 minutes, or 0.25 hours.

Table 2: Estimation of respondent burden for IPEDS Cognitive Interview

Activity	Number of respondents	Number of responses	Hours per respondent	Maximum total burden hours
Recruitment	200	200	0.25	50
Cognitive interviews	24	24	1.00	24
Study Total	200	224		74

Estimate of Costs for Recruiting and Paying Respondents

Respondents will not receive any monetary compensation for their participation in the study.

Estimate of Cost Burden

Participants in the cognitive interviews must bear direct costs associated with travel to AIR headquarters, if the headquarters is selected by the institution as the desired location for the interview.

Cost to Federal Government

The cost to the federal government for conducting these cognitive interviews will be $30,000, under the AIR ESSIN Task 25 contract, which includes recruitment, interviewing, analysis, and developing burden and time use questions to be considered for inclusion in future IPEDS surveys.

5