National Postsecondary Student Aid Study (NPSAS:2012) Computer-Aided Data Extraction Usability Interviews 2013

Attachment 1

Recruiting and Screening Protocol

1. Hi, my name is ___________ and I’m calling on behalf of the U.S. Department of Education’s National Center for Education Statistics, also referred to as NCES. Have I reached {target}?

If yes, proceed to 2.

If no, follow-up to identify whether the target remains employed at the same institution and, if so, to gather his or her contact information. Update contacting information and attempt new contact.

If the target is no longer employed at the institution, close the case and mark the respondent as ineligible.

2. Great. I work with Westat in Rockville, Maryland. NCES has hired Westat to conduct a follow-up interview with a select group of individuals who helped the Department by participating in the 2011-12 National Postsecondary Student Aid Study, or NPSAS. The goal of the interview is to get your feedback on how we could have improved the process of providing your student data to the Department. Our records show that {institution} participated in NPSAS and that you were the NPSAS contact. Is that correct?

If yes, proceed to 3.

If no, identify whether the target doesn’t remember participating or whether work was delegated.

If delegated. If possible, I’d like to follow up with {new target} about their experiences with NPSAS. Can you tell me how to reach them?

If yes, thank respondent and initiate new contact.

If no, thank respondent and mark case as refusal.

If target doesn’t remember participating. You might remember that, as part of NPSAS, you were contacted by RTI International, a research firm working with NCES. As part of NPSAS, you or colleagues at your institution would have first provided a list of students enrolled during the 2011-2012 school year and then, for a subset of those students, provided information about those students’ enrollment histories, financial aid awards, and other student data. Does that sound familiar?

If yes, proceed to 3.

If no, thank respondent for their time and mark case as ineligible.

3. Wonderful. NCES is making plans for how it can improve the next round of the NPSAS data collection, and getting information about what worked well for schools and what didn’t work so well is an important part of that effort. Would you be willing to participate? It should take about an hour, and can be done whenever it is most convenient for you.

If yes:

1. identify an hour long block in the next three business weeks that works best for the respondent and place on interviewer schedule;

2. identify contact number at that date and time;

3. advise: Great. In advance of your interview, you will be receiving an email from the interviewer reminding you of the interview’s date and time. Included with that email will be our Assurance of Confidentiality, which outlines how Westat and NCES will protect your responses, and provides more information about the research and your rights as a participant. What email address should we send that information to?

4. Reconfirm date, time, and all contacting information, and thank respondent for their willingness to participate.

If no, use best judgment about refusal conversion. If concerns can be resolved, do so and schedule as above. If not, mark case as refusal.

Attachment 2

Informed Consent/Assurance of Confidentiality

Consent to Participate in Research

Title of Research:

2011-2012 National Postsecondary Student Aid Study Computer-Aided Data Extraction Usability Interviews, 2013

Introduction

You are being asked to participate in an interview which is part of a larger research study sponsored by the National Center for Education Statistics (NCES) of the U.S. Department of Education. This interview is designed to gather data about your experiences as an institutional representative during the 2011-2012 National Postsecondary Student Aid Study (NPSAS), in particular how easy or difficult you found it to provide student records data to NCES.

Before you decide if you want to take part in this study, you need to read this Informed Consent form so that you understand what the interview is about and what you will be asked to do. This form also tells you who can participate in the interviews, the risks and benefits of participating, how we will protect your information, and who you can call if you have questions. Please ask the researcher to explain anything you don’t understand before you make your decision.

Purpose

The interview is being conducted by Westat, Inc., a research organization located in Rockville, Maryland.

Participation will involve answering questions about your experiences providing data to NCES and its data collection contractor, RTI International, about students enrolled in the 2011-2012 school year. You will be asked about how easy or how difficult it was to provide lists of enrolled students to RTI, and then how easy or difficult it was to provide additional detailed data about students sampled from your institution to participate. Your answers will help us improve data collection techniques for the 2015-2016 National Postsecondary Student Aid Study, which is currently being designed.

Procedures

You are one of approximately 30 individuals participating in these interviews. The interview will be audio recorded to make sure we don’t miss anything that you say and to help us write a report summarizing the results of the interviews. Upon completion of the written report, the recording will be destroyed. Your name will never be used in the report that we write.

Study Duration

Your participation in the interview will take approximately 60 minutes.

Possible Risks or Discomforts

We do not anticipate that any of the discussion topics will make you uncomfortable or upset. However you may refuse to answer any question or take a break at any time.

Benefits

Your Benefits There are no direct benefits to you from participating in this study, and you will not be paid for your participation in this study.

Benefits for Other People However, we hope that this study will identify issues that can inform the U.S. Department of Education as it seeks to improve its future NPSAS data collection protocols.

Confidentiality

Westat, Inc. is conducting this study for the National Center for Education Statistics (NCES) of the U.S. Department of Education. This study is authorized by law under the Education Sciences Reform Act (ESRA 2002). Your participation is voluntary. Your responses are protected by federal statute (20 U.S.C., § 9573) and may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law.

Future Contacts

We will not contact you in the future in relation to these usability interviews.

Your Rights

Your decision to take part in this research study is completely voluntary. You can refuse any part of the study and you can stop participating at any time. You can refuse to answer any question.

Your Questions

If you have any questions about the study, you may call Denise Glover at Westat (301-251-2269) or Matthew Soldner at NCES (202-219-7025).

YOU SHOULD KEEP A COPY OF THIS FORM FOR YOUR RECORDS.

Your participation in the telephone interview indicates that you have read the information provided above, have received answers to any questions that you may have, and have freely decided to participate in this research. By agreeing to participate in this research, you are not giving up any of your legal rights.

Attachment 3

Portal Screen Captures

(Reduced Size)

Attachment 4

Preliminary Interview Protocol

1. Thank you again for agreeing to speak with me. Earlier this week I sent you an email entitled Assurance of Confidentiality. Have you had a chance to read it?

If yes, proceed to 2.

If no:

1. Ask: Do you still have a copy of the email, or can I send it to you again? If needed, resend informed consent email after verifying email.

2. Ask: I just sent the email again. Can you go ahead and check your inbox to see if it has been received. Wait as needed. Once received:

3. Ask: Great. Have you had a chance to read it?

2. Do you have any questions?

If yes, address questions.

If no, proceed to 3.

3. Great. As a reminder, this interview should last no more than sixty minutes. So that I can pay full attention to your responses now and have a record for quality assurance purposes, I’d like your permission to digitally tape our interview. Is that OK?

If no, attempt to identify source of concern and attempt to resolve. If firm refusal, take notes to the best of your ability.

If yes:

1. Turn on recording system.

2. Thanks, {target} for agreeing to allow me to record. Now that we’ve addressed the Assurance of Confidentiality {and answered your questions}, let’s get started.

4. Alright. As we mentioned when you were first contacted, the purpose of the interview is to provide information to the U.S. Department of Education’s National Center for Education Statistics, or NCES, about your experiences providing data for the 2011-2012 National Postsecondary Student Aid Study, or NPSAS.

I would like to ask you about your experience with NPSAS in two phases. First, I want to address what we call “list collection.” As you might remember, list collection was the first activity you completed for us, where you provided us a list of all students enrolled at your institution between July of 2011 and June of 2012, along with certain student characteristics.

The second phase is what we call computer-aided data extraction, or CADE. CADE is where you provided us very detailed information about individual students on your campus who were sampled to participate in NPSAS. You completed CADE either by extracting data from your campus data systems in to a CSV file and uploading it to our web portal, filling in data in to an Excel template and then uploading that file to our web portal, or filling out an on-line web form for each student.

Let’s begin with the enrollment list process.

INTERVIEWER: All participants respond to the questions below related to the enrollment list. Probe with follow-ups as appropriate to elicit complete responses.

1. Do you keep your enrollment information in an electronic student information system, or in some other way?

If electronic:

1. What is the name of the system you use?

2. If not Peoplesoft, Datatel, Banner, or Jenzabar: Is that a commercial product, or a system developed for your institution?

If some other way: Can you describe how those records are stored?

2. When you provided your enrollment list, you were asked to provide the name, social security number, date of birth, class level, and major of each student who enrolled at your institution between June 2011 and July 2012. How easy or hard was that activity for you?

3. If you had been asked to indicate which of the students on that list you believed would earn a bachelor’s degree by July 2012, how easy or hard would that have been for you?

4. If you had been given a list of majors or Classification of Instructional Program (CIP) codes that NCES defined as being a science, technology, engineering, or math field—a so-called “STEM field”—and then we asked you to indicate which of the students on that list had begun a doctoral program in a STEM field that year, how easy or hard would that have been for you?

5. Is there anything else you would like to tell me about the enrollment list process?

INTERVIEWER: Each participant will respond to only ONE section below, based on the method they used to complete NPSAS:12 CADE.

OK, let’s talk about how easy or how hard it was for you to provide the detailed student records we asked you to provide about students sampled for NPSAS:12. According to your records, you completed CADE by {method}, which means you {insert description here}.

INTERVIEWER: The questions below are meant to guide your conversation. As a reminder, the goal is to get a clear picture of how the respondent completed the NPSAS CADE request and what made it harder or easier. Please probe as appropriate.

1. From beginning to end, about how many hours did you spend providing us the detailed student information we requested?

2. As part of our request, we asked for student demographic information and student financial aid data. In your role at {institution}, did you have access to all of the information we requested, some of the information, or none of the information?

3. Given that you had access to {all/some/none} of the information, can you describe how you fulfilled the request?

INTERVIEWER: Based on response to #3, probe as appropriate to get a complete sense of the parties involved in fulfilling our request and the steps taken by each. Possible probes include:

1. What other campus offices did you collaborate with to fulfill the NPSAS data request?

2. Can you describe the processes undertaken by others on campus to fulfill the request, as you recall them?

3. Did any collaborators express concerns about the request, and, if so, how were they resolved?

4. How easy or hard was it for you to fulfill the NPSAS data request?

INTERVIEWER: Based on response to #4, probe as appropriate to understand what made the process easy or difficult. Possible probes include:

1. What made it {easy/hard} for you?

2. Did you seek out the assistance of others on campus, and, if so, how did they help you?

3. Did you need to contact the NPSAS School Help Desk for assistance, and, if so, were they able to assist you in resolving your concern?

5. As I mentioned earlier, our records show you used {method} to provide us the detailed student data we requested. If you were sampled for NPSAS again, would you choose the same method? Why is that?

6. In retrospect, what information could NCES have provided you at the beginning of the NPSAS data collection that would have made it easier for you?

7. If a colleague at an institution like yours called you and said their institution had been sampled for NPSAS and asked your advice in how they should best proceed in fulfilling the Department’s request, what advice would you give them?

8. Is there any additional feedback you would like to offer NCES as they prepare for NPSAS:2016, about this or any other part of the study?

9. Is there anything else you would like me to know before we wrap up?

Thank you for your time. The next step in our research process is for my colleagues and me to synthesize responses from all of our respondents and prepare a report for NCES and its data collection contractor. At no time will your responses be specifically attributed to you. Do you have any final questions?

If yes, address remaining questions.

Thank you again. This concludes our interview.