Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) Eye-tracking Cognitive Laboratory

National Center for Education Statistics

Middle Grades Longitudinal Study of 2017-2018 (MGLS:2017) Eye-tracking Cognitive laboratory

Volume III

Cognitive Laboratory Protocols

for Student Assessments and Surveys

OMB# 1850-0803 v. 104

November 2015

MGLS: 2017 Cognitive Laboratory Protocol:
Student Assessment and Survey

I. Interviewer Welcome Script

The following welcome script should not be read verbatim. You, as the interviewer, need to be familiar enough with the scripts to ask the item probes in a conversational manner. Text written in the box is suggested content for you to be thoroughly familiar with in advance. You should project a warm and reassuring manner toward the participant to develop a friendly rapport and should use conversational language throughout. While the text in this introduction does not need to be read verbatim, all text in bold (notifying student of confidentiality and item-specific probes) is read verbatim to ensure we cover these important topics and ask students these questions in the same way.

Hello, my name is [NAME] and I am working with the U.S. Department of Education on a research study called the Middle Grades Longitudinal Study of 2017-2018 (MGLS:2017).

Thank you for coming here today to help us. You may have already talked with your parents about what we are going to do, but just in case you still aren’t completely sure ,we want to take a second and talk you through what’s going on.

You are here because we asked your [mom/dad] if you might be able to work with us to learn about the questions we are using for the MGLS:2017. This is the first national study to learn about the experiences of students in the middle grades. The MGLS:2017 study will ask questions of middle-grade students like you from many different schools, and help us better understand you and what you know, as well as your experiences with your peers, your teachers, your school, and your family. Your participation in this study today will help us learn about the questions we are using so that we can improve them for students who will take part in the study in the future.

You will be helping us by answering questions on a computer. We are going to learn about how you move your eyes as you read what is on the computer. Therefore, before we begin we will have you look at dots in different areas on the screen; this will help the computer learn where you are looking while you read. Then you will go through and read and answer the questions. Please go through the questions and try to answer them the best that you can. After you complete each section, we will ask you some general questions about what it was like to answer the questions. For example, whether everything made sense, and if there’s something about the questions that makes them confusing, tricky to understand, or especially hard to answer. It will take about two hours to go through all the questions and talk about them. If you need to, we can take short breaks between each section. As a thank you for helping us, we will give you a $[30/45] gift card after your session is complete.

Your information and the answers you give today will be protected. Only people working on the research study will see the answers and ideas, but they won’t know who gave those answers and ideas. We will not talk about any of your answers or comments with your parents, teachers, or school, but please feel free to tell your parents about anything we talk about today. Your help today will only be used for research purposes and may not be used for any other purpose except as required by law.

While you are working, I will be taking notes and the computer will be recording where you look on the screen. I would also like to record what we are doing today so I can listen to it later when I write up my notes. No one outside our research team will hear the recording. Is it okay with you if I record our interview?

Just to remind you, your participation is completely voluntary, and if at any point you decide not to continue, that is okay.

Do you have any questions before we get started?

After answering questions and giving further explanation, have the student begin the assessment. If, for any reason, the student is no longer interested in participating, thank the student for his/her time and end the interview.

In addition to the notes you will take when talking with the student after they finish the assessment, make sure to observe the student while taking the assessment, noting signs of restlessness and frustration (such as fidgeting and time spent with eyes wandering around the room instead of on the task at hand). Note also the items for which the student uses a calculator or works on paper (and the location on the paper that has the work for each item). If a student returns to a previous item or instruction screen please note which item or screen they were on, what item or screen they returned to it and if they corrected or completed the item that they returned to.

II. Description of the Eyetracking Protocol

Eyetracking is designed to provide data about where and for how long students place their visual focus while taking an assessment. Therefore, as the students complete the assessments and survey, the eye tracker camera monitors where they look on the computer screen.

Apparatus

An eyetracking system is a video-based system utilizing a high speed camera and an LED infrared light that illuminates the face (no more powerful than typical sunlight). An Infrared camera is used to capture eye movements. The remote system requires no physical contact with the student: an infrared video camera acquires an image of the eyes and calculates gaze location in real time. Fors Marsh Group (FMG) eyetracking experts have used this particular system in a wide variety of studies. It takes a very short time (less than five minutes) to set up and calibrate the equipment for each participant.

Procedure

First, the eyetracking system is calibrated to the student. To calibrate the eye tracker, the cognitive laboratory interviewer will ask participants to follow circles, that will appear in different positions on the screen, with their eyes. Once the system is calibrated, students are asked to work through the assessments and the survey in a normal way, without interruption. During this phase their eye movements are unobtrusively recorded and all events on the screen are captured in real time.

A primary focus of this cognitive laboratory work is the information gained from the eye-tracking system. However, as the participants will have also worked through the assessments and the survey, we will collect general information about the assessments and the survey. No particular assessment items or survey items have been identified for cognitive laboratory work. However, we are interested in any additional information on clarity of instructions, clarity of items, and navigability of the assessments and surveys. The observer will also remark on any items where the student visibly struggled. Therefore, using a retrospective verbal protocol, students are asked to think about the tasks and attempt to reflect on, and describe out loud, what they were thinking as they went through the task, and will be prompted to think about clarity of instructions, clarity of items, and navigability of the assessments and surveys. These verbalizations are recorded.

The sequence to be followed for eyetracking and the subsequent retrospective think-aloud is described below.

Setup and Calibration

The participating student is greeted by a trained cognitive laboratory interviewer. The interviewer will be trained in the eyetracking system and be an expert in calibration of the system. The interviewer then describes the study procedure and responds to questions or concerns the student may have. The student is then seated in front of the eyetracker. The system is then calibrated. The student is asked to watch a circle appearing at a number of different locations on the blank screen for approximately 10 seconds. The setup and calibration typically take about five minutes.

Eyetracking Session

After a successful calibration, the student starts the session, in which he or she goes through the assessments and survey on the tablet computer. Students are asked to avoid any large head movements, to ensure proper collection of the eyetracking data. These movements will result in a temporary loss of data, but once the student resumes normal activity the eyetracker will resume recording data. If the eyetracker loses the participants gaze due to a change in seating position, the interviewer will prompt the student to change how they are seated. It is not necessary to recalibrate the student in either instance.

Along with the eyetracking data, the system also records participant’s computer activity (for example, cursor movement and clicks), screen changes (as a video file), and audio. The cognitive laboratory interviewer takes notes during the session about noticeable behaviors or events. The interviewer will indicate items where the student visibly struggled, taking notice of amount of time on task, changes in student facial expression, and erratic eye movement, as seen by the interviewer. This information will be recorded in a simple organizational table (such as the example below) listing the assessment section and item information, as well as allowing ample space for notes:

Component	Item	Observation	Additional Notes

This will allow the interviewer to ask the students about their experience with these particular items during the retrospective interview.

Cued Retrospective Verbal Report

After the student completes the assessments and surveys, eyetracking is stopped. As previously stated while a primary focus of this cognitive laboratory work is the information from the eyetracking system, a retrospective interview will be used to gain additional information about the assessments and the survey. This interview will include general questions about the experience as well as probes about any specific items students struggled with. For example, the interview may include questions such as the following.

Please think about when you were working on math problems. Did you understand the instructions for how to use the computer? If not, what did you find confusing? Did you understand how to use the computer to enter your answers? If not, what did you find confusing? Were there specific problems that you found confusing?If so, could you give me an example and talk about of one of those items?

Please think about when you were asked to read and answer questions about reading. Did you understand the instructions for how to use the computer? If not, what did you find confusing? Did you understand how to use the computer to enter your answers? If not, what did you find confusing? Were there specific questions that you found confusing? If so, could you give me an example and talk about one of those questions?

Please think about when you were asked to play a game with [DESCRIBE EXECUTIVE FUNCTION TASK]. Did you understand the instructions for how to play? If not, what did you find confusing?

Please think about when you were answering questions about yourself, your family, and school. Were there any items that you found confusing? If so, could you give me an example and talk about one of those questions.

I am going to show you some of the questions you just answered, and ask you to tell me what you were thinking as you were answering it. We are interested in what was on your mind at that moment in the task, where you were looking while completing it, and what you might have found tricky or confusing.

Try not to just reread the directions or your answers. Instead, talk about your goals and plans at the time you were working on that part of the task. Tell us what was in your mind as you read, answered questions, and thought about the task.

To help students with the verbal protocol task, they are informed that they will be asked to speak continuously and that if they stop talking, they will be prompted with terms such as “please keep talking” and “please tell me more about that.” Students’ verbal reports are recorded for later analysis.

III. Example Probes if Specific Items are Identified as Confusing

When probing, interviewers are instructed to think about the following kinds of details that capture how the student reads and understands each question:

1. Understanding of the Item: We are interested in learning if students have never seen the word before or if they forgot its meaning, and if they were still able to answer the question. Reassure them that it is okay if they did not know all the words and that their feedback is important – we want to make sure the questions are straight-forward for all students.

   1. What questions and/or words were confusing or difficult to read?

      1. Have the student circle or highlight problem words and talk more about them – are they completely unfamiliar?

      2. Ask student to identify a word that may have been less confusing or easier. Ask how their teacher or textbook usually asks these questions, and what words they use.

   2. Probe about word problems with more than two sentences in particular. Can they restate in their own words what the question is asking? If not, why?:

      1. Can you tell me, in your own words, what the question is asking?

      2. Are there any words, phrases, or (if applicable) pictures/tables/figures/graphs that are difficult for you? If so, which ones?

      3. Is there anything about this question that confuses you? If yes, do you have a suggestion for how to make it easier to understand?

      4. (If applicable) How many answers are there for this question? How did you go about selecting those answers?

2. Presentation of the Item:

   1. Certain items will have graphs or displays associated with the question. How does this affect the student’s understanding? Do they see questions like these in their math books or math class?

   2. Probe the student about their familiarity with the way concepts are presented, especially with fractions or equations that can be represented in multiple ways.

      1. Are some conventions easier for the student to understand than others? Why?

   3. Use Probes appropriate for item content:

      1. Ask the student to read the question to you (including the numbers). Note words that are difficult. Ask: Can you tell me, in your own words, what the question is asking?

      2. How did you answer this question?

      3. Are there any words, phrases, or (if applicable) pictures/tables/figures/graphs that are difficult for you? If so, which ones?

      4. Did you recognize the symbols in this question? Have you seen your math teacher or textbook use these symbols? [point to symbols in the item]

      5. [If the item was skipped] Did the symbols confuse you at all? Is there anything that would make it easier to understand the question?

3. Language:

   1. If an item has a proper noun, such as a person or place’s name, does the student recognize that it is a name or are they spending time trying to decode it?

   2. For sentences with item clauses (i.e., IF this, THEN that), does the student understand what the question is asking? Ask the student to put the question in their own words. Students can also show in writing what the question is asking, using the paper you provide.

   3. Probes:

      1. Can you tell me, in your own words, what the question is asking?

      2. How did you answer this question?

      3. Can you please read the word in these parentheses out loud [point to difficult word(s)]?

      4. Are there any words, phrases, or (if applicable) pictures/tables/figures/graphs that are difficult for you? If so, which ones?

         1. Did it affect how you answered the question? How?

         2. What can you tell me about these words [Point to words mentioned above)]? Did you need to read these words [proper names] in order to answer the question?

III. Debriefing and Thank You for Participation

Thank student for his/her time. Collect all hard copy forms and scratch paper the student used.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing this morning/afternoon.

Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to improve our questions.