Volume II - NAEP Sail Test Assembly

National Center for Education Statistics

National Assessment of Educational Progress

National Assessment of Educational Progress (NAEP)

Survey Assessments Innovations Lab (SAIL)

Test Assembly Experimental Study

Volume II

Assessment and Survey Items

OMB# 1850-0803 v.272

September 2020

Table of Contents

Paperwork Burden Statement 3

Authorization and Confidentiality Assurance 3

Directions for Administration 4

Math Assessment Items 6

Effort Likert Item 7

Post-Test Survey Items 8

End of Study Acknowledgment and Thank You 9

Paperwork Burden Statement

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is OMB# 1850-0803. The time required to complete this information collection is estimated at a maximum of 50-minutes, including the time to review instructions and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission, please write to: National Assessment of Educational Progress (NAEP), National Center for Education Statistics (NCES), Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

Authorization and Confidentiality Assurance

National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) and to collect students’ education records from education agencies or institutions for the purposes of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C.§151).

OMB No. 1850-0803 Approval Expires 7/31/2023

Directions for Administration

Below is a set of directions that will be provided to parents and legal guardians to provide guidance for how to administer the study, tips for troubleshooting typical problems, a guide for technology requirements, and contact information for further support.

Prior to Activity Administration

• Plan for 50-minutes of uninterrupted time to complete the math activity.

DAY of Activity Administration

• Provide your child with their Login ID and the link to the math activity. Both of these were provided in an email.

• Direct your child to the link [link to experiment materials]

• Help your child log in as needed.

• If your child appears to be using the wrong ID, please recheck the ID provided in the email.

• If it is necessary for your child to log off in the middle of a session, or if their computer crashes, when they log back in they will be taken back to where they left off. Your child should log back in with the same ID assigned to them. Please let us know if your child runs into any issues during completion of the study by contacting Blair Lehman (blehman@ets.org or 847-942-3789).

• Have your child independently complete the activity. Please maintain a quiet atmosphere during the administration. Your child must answer all questions on their own. However, it is OK to provide assistance navigating through the interface, explaining directions or vocabulary.

• If an unforeseen interruption occurs instruct your child to close the browser, re-open and re-login. Please let us know if your child runs into any issues during completion of the study by contacting Blair Lehman (blehman@ets.org or 847-942-3789).

• When your child finishes the activities please have them close the web browser window used for the task.

• Approximately 2 weeks after the study is completed, you will receive an email with information to access your child’s $20 virtual gift card. If you have not received that email or if you have any questions about the study, please feel free to contact Blair Lehman (blehman@ets.org or 847-942-3789).

Thank you for assisting us! We greatly appreciate your willingness to participate in this research effort.

Troubleshooting

Some common technical issues are very easy to solve. The following is a list of issues you may encounter and their quick fixes:

• Most issues can be solved simply by re-starting the browser and logging in again. When a test is re-started, your child will be returned to the screen they were previously working on.

• Screen is frozen: Try to reload the page. If this doesn’t work, close the browser, re-open it, and re-login.

• Activities won’t load: Make sure Pop-Up Blocker is Off. Close browser, re-open it, and re-login.

• Images appear distorted: Set browser to 100% zoom. Close browser, re-open and re-login.

• No scroll bar: Close Browser. Check screen resolution is set to 1024x768. Re-open browser and re-login.

If you have any questions, please contact:

Blair Lehman ------ blehman@ets.org ----- 847-942-3789

System Requirements

Operating System & Browser:

Windows 7, 8, or 10 with the current version of Chrome or Firefox

OS X 10.8 ("Mountain Lion") or later with Safari 6.2 or higher

Chromebook:

All models. If activity window does not open in full screen, please press the Immersive Mode button (located where the F4 button would be on a PC).

Display:

Screen resolution of 1024 x 768 or greater.

Note: JavaScript & cookies need to be enabled (which is the default). Popups should be allowed, in order to launch the assessment in full screen. Set browser zoom level to 100%.

Math Assessment Items

Participants will be presented with 30 math assessment items that are distributed over two blocks (15 items each). An example item is included below. The math items were developed for 8^th grade students and span a variety of item formats (e.g., multiple-choice, constructed-response, drag-and-drop).

Effort Likert Item

A self-report question about effort similar to the one shown below will be administered after the first time a participant views each of the math assessment items. The purpose of this Likert item is to have an additional source of information about the amount of effort that participants invest in answering each math item. The responses to this Likert item will be compared to process data (e.g., item view time, clickstream data).

Post-Test Survey Items

After participants complete the two blocks of math assessment items, they will be asked to complete a short post-test survey (see example questions below). The purpose of this post-test survey is to gather information about participants’ overall experience with the math assessment items. Participants will be asked to (1) estimate the proportion of questions answered correctly on the math assessment, (2) rate the overall difficulty of the math assessment, and (3) complete the Effort Scale (Eklöf, Pavešicˇ, & Grønmo, 2011) to provide an overall estimate of the amount of effort invested into the math assessment.

End of Study Acknowledgment and Thank You

Thank you for participating in our study! You have now completed all parts of the study. You will be sent a virtual gift card two weeks after the completion of the study for your participation.