Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) Student Assessment Usability Study

MEMORANDUM OMB # 1850-0803 v.173

DATE: September 29, 2016

TO: Robert Sivinski

Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Carolyn Fidelman

National Center for Education Statistics

THROUGH: Kashka Kubzdela

National Center for Education Statistics

SUBJECT: Middle Grades Longitudinal Study of 2017–18 (MGLS:2017) Student Assessment Usability Study

The Middle Grades Longitudinal Study of 2017-18 (MGLS:2017) will be the first study sponsored by the National Center for Education Statistics (NCES), within the Institute of Education Sciences (IES) of the U.S. Department of Education, to follow a nationally representative sample of students as they enter and move through the middle grades (grades 6-8). An eye-tracking cognitive laboratory study was conducted in 2016 (OMB# 1850-0803 v.148), concurrent with the MGLS:2017 Item Validation Field Test (OMB Control Number 1850-0911).

In the eye-tracking cognitive laboratory study, students completed the MGLS:2017 assessments and student surveys, which were administered on tablets that included eye-tracking technology. This technology provided information on the point of gaze (where one is looking) and the motion of the eyes as information is processed. The analysis of the eye-tracking movements provided information on assessment item functioning, clarity of instructions, and navigability, and on the clarity of the surveys. The final report recommended that changes be made to the student assessments and the instructions for them.

As part of the instrumentation development process for the MGLS:2017, the screen layout and format of the assessments and instructions are being revised based on the findings from the eye tracking cognitive laboratory study and the Item Validation Field Test (IVFT). The instructions are being streamlined and the layout is being made consistent with the web survey layout. RTI, under contract with NCES, will program the revisions to the assessments and instructions in the fall preceding the MGLS:2017 Operational Field Test, which will begin in January 2017, to allow time to thoroughly test the system.

This request is to test the programmed assessments and instructions with a convenience sample of up to 10 middle school age children to confirm that the revisions have improved understanding, usability, and efficiency of the assessments. These tests will be similar to the eye tracking cognitive laboratory study in that the students will complete the assessments under the observation of a project methodologist. At the end of the assessments, the project methodologist will administer the same probes and debriefing items that were administered in the eye tracking cognitive laboratory study. These tests will be different from the eye tracking cognitive laboratory study in that they will not be conducted with eye tracking software, and will not involve children recruited with from the general population. The tests will rely on a convenience set of children local to RTI’s North Carolina offices, who will be identified by RTI staff. No incentive will be offered to the participants, and no explicit recruitment is necessary.

The expected burden for this activity is 90 minutes per student for up to 10 students, for a total of 15 burden hours for this study.

Attachment 1 provides the parent informed consent for this activity. There will be no personally-identifying information (PII) collected from participants other than the parental consent form. There will be no PII recorded by the methodologist or by the assessments.

The activities will be conducted and concluded in early October 2016.

These activities do not represent any additional costs beyond those already included in the MGLS:2017 Eye-tracking Cognitive Laboratory study (OMB# 1850-0803 v.148). In essence, this testing is a follow-up to the original study.

Verbal Report after Completing Assessments

After the student completes the assessments, a brief retrospective interview will be used to gain additional information about the student’s experience with survey. This interview will include general questions about the experience as well as probes about any specific items students struggled with. For example, the interview may include questions such as the following.

Please think about when you were working on math problems. Did you understand the instructions for how to use the computer? If not, what did you find confusing? Did you understand how to use the computer to enter your answers? If not, what did you find confusing? Were there specific problems that you found confusing? If so, could you give me an example and talk about of one of those items?

Please think about when you were asked to read and answer questions about reading. Did you understand the instructions for how to use the computer? If not, what did you find confusing? Did you understand how to use the computer to enter your answers? If not, what did you find confusing? Were there specific questions that you found confusing? If so, could you give me an example and talk about one of those questions?

Please think about when you were asked to play a game with [DESCRIBE EXECUTIVE FUNCTION TASK]. Did you understand the instructions for how to play? If not, what did you find confusing?

Before we finish, I’d like to hear any other thoughts you have about the tasks you completed on the computer. Is there anything else you would like to tell me about working on the task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Attachment 1. Parent Informed Consent (for minor child)

On behalf of the National Center for Education Statistics (NCES), part of the U.S. Department of Education, RTI is conducting voluntary interviews with students in grades 6, 7, and 8 to support the development of items and survey questions for the Middle Grades Longitudinal Study of 2017-18.

Your child will be asked to participate in a 90 minute session at one of our corporate offices. Participating students will be asked to answer questions related to math, reading, and to play a memory game a tablet computer. A researcher will observe students one-on-one while they complete the task and the computer will record how students interact with the screen. The researcher will also conduct a brief interview at the end. Students will not be graded on their responses.

Participation is entirely voluntary and student answers will be used only for research purposes and will not be disclosed or used, in identifiable form, for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C., § 9573].

The feedback received from students will provide NCES with valuable information to improve the items that will be included in the national research study.

By signing below, you agree that your child may participate in this study. Your signature below will indicate that your questions have been answered satisfactorily, and that you have read and understood the information provided above.

Respectfully,

I hereby accept the terms described above and give permission for my child _____________________ to participate voluntarily in the MGLS:2017 student interviews.

­­­­­­­­­­­­­­­­­­­­­­­­­

__________________________________

Print Name

____________________________________ _______________________

Signature Date

4