Justification

National Center for Education Statistics

Volume I

Supporting Statement

2017 Integrated Postsecondary Education Data System (IPEDS)

Time Use and Burden Cognitive Interviews Round 1

OMB# 1850-0803 v.177

October 2016

Submittal-Related Information

The following material is being submitted under the National Center for Education Statistics (NCES) clearance agreement (OMB # 1850-0803) which allows NCES to improve the methodologies, question types, and/or delivery methods of its survey and assessment instruments by conducting developmental studies such as pre-tests, focus groups, and cognitive interviews.

This request is to conduct one round of cognitive interviews to better undertand institutional burden and time use for the Integrated Postsecondary Education Data System (IPEDS). The data collection for this study is being carried out for NCES by the American Institutes for Research (AIR) under contract to the U.S. Department of Education beginning in January 1, 2016. The findings from this round of cognitive interviews will provide recommendations for burden and time use interview questions and a revised cognitive interview protocol to be used in a second round of cognitive interviews on time use and burden estimates for IPEDS. A summary of results from round 1 of this cognitive research and the recommendations for future interview questions and protocol will be developed by late February 2017. This submission includes recruitment procedures and materials, the informed consent form, the cognitive interview protocol, and survey items for round 1 of testing.

Background

IPEDS is a data collection system designed to collect basic data from postsecondary institutions. IPEDS enables NCES to report on key dimensions of postsecondary education such as enrollments, degrees and other awards earned, tuition and fees, average net price, student financial aid, graduation rates, student outcomes, revenues and expenditures, faculty salaries, and staff employed. The IPEDS web-based data collection system was implemented in 2000-01, and it collects basic data from the approximately 7,300 postsecondary institutions in the United States and its other jurisdictions that are eligible to participate in Title IV Federal financial aid programs. All Title IV institutions are required to respond to IPEDS (Section 490 of the Higher Education Amendments of 1992 [P.L. 102-325]). IPEDS allows other (non-title IV) institutions to participate on a voluntary basis. Approximately 200 non-Title IV institutions elect to respond. IPEDS data are available to the public through the College Navigator and IPEDS Data Center websites.

Data collected in IPEDS is used in numerous public facing data tools that facilitate the identification and comparison of postsecondary institutions for prospective students, policy makers, and researchers. As such, over time, the data in IPEDS has become increasingly more important and ubiquitous in the postsecondary industry. This may have resulted in institutions taking additional time to review and consider their IPEDS data before submitting them. In order to provide accurate burden time estimates for IPEDS data collections, NCES would like to develop a question or a short set of questions to collect accurate estimates of the time and burden involved in reporting IPEDS data. NCES proposes to use two rounds of cognitive testing to develop these questions. The cognitive testing will attempt to discern the different types of time use and burden that are related to responding to IPEDS. For example, IPEDS respondents may take time to:

1. Extract and tabulate information,

2. Compare completed IPEDS forms with previous submissions,

3. Brief institutional leaders regarding the information in the new submission, and

4. Enter the information into the IPEDS web based collection system.

Presently, IPEDS estimates that the collection annually burdens the industry approximately 999,060 hours based on 77,600 responses. These responses are based on the organization of IPEDS into 11 different survey components:

1. Institutional Characteristics

2. Completions

3. Fall Enrollment

4. 12-Month Enrollment

5. Student Financial Aid

6. Graduation Rates

7. Finance

8. Human Resources

9. Admissions

10. Academic Libraries

11. Outcome Measures

Design and Context

The exploratory 2017 IPEDS Time Use and Burden Cognitive Interviews study is designed to better understand respondents’ time use and burden for the 11 IPEDS surveys. Based on cognitive interviews with administrators from 24 institutions, the study will recommend time use and burden interview questions and a cognitive interview protocol to be used in a future second round of cognitive interviews examining how much time respondents spend annually completing the 11 IPEDS surveys.

During cognitive testing, interviewer(s) will use a structured protocol drawing on methods from cognitive science to investigate the process respondents use to answer burden related questions. In particular, these interviews will identify problems of ambiguity or misunderstanding in the time use and burden question that currently is used as part of the IPEDS surveys. The main goal of this study is to explore respondents’ understanding of the current time use and burden question, particularly what respondents include and exclude from their calculation when answering it, in order to determine whether respondents are providing the intended information. The results will drive possible recommendations for a revised question (or set of questions) that would more accurately reflect each institution’s time use and burden associated with annual IPEDS reporting. A request for the second set of cognitive interviews will be submitted to OMB for review in 2017. The 2^nd round of interviews will be conducted to determine if the revised question(s) work as intended.

The cognitive interviewing methods will consist of two key components: think-aloud interviewing and verbal probing techniques (these methods are also known as concurrent and retrospective recall probing, respectively). With think-aloud interviews, respondents are explicitly instructed to think aloud (i.e., describe what they are thinking) as they work through question items presented by the interviewer. With verbal probing techniques, the interviewers ask questions, as necessary, to clarify points that are not evident from the “think-aloud” process, or to explore additional issues identified as being of particular interest. Cognitive interview studies produce qualitative data in the form of verbalizations made by participants during the think-aloud phase and in response to the interviewer probes. Both the think-aloud approach and probing techniques will be applied to all participants during the cognitive interviews.

NCES has contracted AIR to administer the 2017 IPEDS Time Use and Burden Cognitive Interviews in January 2017 through March 2017. On behalf of NCES, AIR will recruit 24 institutions representing a range of characteristics (including institutional level and size). Table 1 provides a distribution of the desired minimum number of respondents by characteristics, although a respondent may fit more than one category. Please note that although the sample will include a mix of characteristics, the results will not explicitly measure differences by these characteristics. Participants will include administrators and staff from institutions that have experience in compiling and submitting data for IPEDS surveys. The interviews will be conducted via conference call, web conferencing, at institutions, or at AIR Washington D.C headquarters or one its satellite offices in the United States. The availability of remote interviews will serve to encourage participation by offering flexible options. Institutions for in-person interviews will be recruited primarily from the District of Columbia, Chicago, San Mateo, Boston, and Austin metropolitan areas (near AIR offices) to maximize scheduling and interviewing efficiency and flexibility.

Table 1: Desired minimum number of respondents, by characteristics

Characteristic	Desired number of respondents
Institution size
Small (less than 800 enrollment)	5-7
Medium (between 800 and 3000 enrollment)	11-13
Large (greater than 3000 enrollment)	5-7
Institution level
4-year	7-9
2-year	7-9
Less-than-2-year	7-9

NCES will provide AIR a list of possible institutions and contact information that meet the desired specifications (institutional level, size, and proximity to AIR offices) to recruit for participation in the study. Emails and phone calls will be used to contact potential participants during recruitment. AIR’s experiences with recruiting administrators and staff at postsecondary institutions for cognitive interviews indicate that these individuals are a hard-to-reach population; thus, increased time and effort will be dedicated to meeting recruitment targets. AIR anticipates screening up to 80 institutions during recruitment in order to ensure 24 interviews are completed representing the specific sub-populations of the IPEDS study. Attachment I presents the materials that will be used for recruitment of cognitive interview participants. Once a candidate expresses interest in participating in the cognitive interview, logistics for the interview will be set-up with the institution. Trained AIR staff will administer the cognitive interviews at the selected institutions, AIR offices, web conferencing, or conference call with the goal of completing an average of 2 institutional interviews per day.

A cognitive interviewing protocol (Attachment III) will be used to guide the flow and content of the cognitive interviews. Instructions for cognitive interview respondents and probes inquiring about the meaning of key terms or constructs will be included in the protocol. Interviewers will refer to the protocol to guide the content of the interviews, but will also be free to deviate from the guide should participants have difficulty answering questions that do not have scripted probes or prompts.

Immediately following the conclusion of each interview, methodologists will review the interview recordings and notes, and highlight potential themes that may have arisen. Following each interview, the digital audio recording will be archived for qualitative analysis. AIR will organize their observations and summarize the common themes, insights, and ideas emerging from each of the interviews into a report, which will also include the survey item development and cognitive interview protocol for a future additional round of interviews.

Assurance of Confidentiality

Cognitive interview participants will be informed that their participation is voluntary and that:

“American Institutes for Research is carrying out this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. NCES is authorized to conduct this study by the Education Sciences Reform Act (20 U.S.C., § 9543). Your participation is voluntary. All responses that relate to or describe identifiable characteristics of individuals or institutions may be used only for resreach purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).”

Participants will be assigned a unique identifier (ID) that will be created solely for file management and used to keep all materials together. The participant ID will not be linked to their name. Participants will sign an informed consent form (Attachment II) which will be kept separate from the interview files and notes in a locked cabinet in a secure room for the duration of the study, and which will be destroyed after the final report is completed.

Schedule for 2017 IPEDS Time Use and Burden Cognitive Interviews Study Round 1

AIR will begin recruiting for the cognitive testing upon receiving OMB clearance. The cognitive interviews are scheduled to begin in early January 2017. A summary of results from this study and recommendations for Round 2 interview questions and protocol will be developed by late February 2017.

Estimate of Respondent Burden

To yield 24 completed interviews, we anticipate screening up to 80 institutions for eligibility and to ensure that we are achieving the desired distribution of respondent types. NCES and AIR expect the cognitive interviews to last approximately 90 minutes in length. The initial contact and recruitment of each potential participant is estimated at 15 minutes, or 0.25 hours.

Table 2: Estimation of respondent burden for IPEDS Cognitive Interview

Activity	Number of respondents	Number of responses	Hours per respondent	Maximum total burden hours
Recruitment	80	80	0.25	20
Cognitive interviews	24	24	1.50	36
Study Total	80	104		56

Estimate of Costs for Recruiting and Paying Respondents

Respondents will not receive any monetary compensation for their participation in the study.

Estimate of Cost Burden

Participants in the cognitive interviews must bear direct costs associated with travel to AIR offices, if an AIR office is selected by the institution as the desired location for the interview.

Cost to Federal Government

The cost to the federal government for conducting these cognitive interviews will be $30,000, under the AIR ESSIN Task 25 contract, which includes recruitment, interviewing, analysis, and developing burden and time use questions and interview protocol for a future second round of cognitive interviews.

5