NAEP Technology Based Assessments (TBA) Tools and Item Types Usability Study

National Center for Education Statistics

National Assessment of Educational Progress

Appendices

NAEP Technology Based Assessments (TBA)

Tools and Item Types Usability Study

OMB# 1850-0803 v.112

August 6, 2014

Volume I: Appendices

Contents

Appendix A: Sample Recruiting Flyer for Use at NCES-Sponsored Events 1

Appendix B: Usability Study Sample Email Contact with Participating Schools 2

Appendix C: Usability Study Sample School Contact Script 3

Appendix D: Usability Study Parent/Guardian Notification Letter 6

Appendix E: Thank You Email to Participating School 7

Appendix A: Sample Recruiting Flyer for Use at NCES-Sponsored Events

NAEP 2014-2015 Usability Study

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by the National Center for Education Statistics (NCES), part of the Institute for Education Sciences, within the U.S. Department of Education. As part of the effort to remain the gold standard in assessment, NAEP, through its contractor Fulcrum IT, regularly conducts usability studies on new assessment tools. These studies are conducted to see how students interact with new tools and item types being considered for future assessments. The study will not be testing finished items and will not be scoring students' performance. Screen activity will be recorded for analysis, but no personal identifying information will be recorded or retained. NAEP will be conducting these usability studies in up to 20 schools over the course of the 2014-2015 school year.

Student Participation

We are looking to include students from grades 4, 8, and 12 for the study. Each student will work one-on-one with a study facilitator for 1 hour and 15 minutes. We will have three study facilitators, allowing us to work with three students simultaneously.

Time

Each school visit will last two days. The study plan is to test 18 students over a two-day period in each school. Students will be seen in groups of three. Fulcrum IT will work with each school to determine when each session takes place and how many sessions will be conducted each day.

Equipment

The Fulcrum IT staff will provide all equipment used in the study, requiring only a quiet place to work and electrical outlets.

Schedule

We are scheduling schools continuously throughout the 2014-2015 school year. Schools may participate in multiple two-day sessions if they would like. We are approaching this with maximum flexibility so that we can accommodate any school’s schedule.

Incentives

Students will get to keep the earbuds they used for the usability study. Each school participating in the usability study will be offered a $50 gift card to a major office/school supply store as a token of appreciation.

For more information, or to participate, please contact the user experience team at Fulcrum IT.

Gregory Anderson, Ph.D.

Senior Usability Expert / Study Coordinator

uxteam@naepims.org

703-543-2877

Appendix B: Usability Study Sample Email Contact with Participating Schools

Email Subject Line: Arrangements for NAEP Usability Study Date:

Dear [NAME of School Coordinator],

My name is <NAME>, and I am calling from Fulcrum IT. [NAME of NAEP State Coordinator] forwarded your name to me to work out arrangements for my colleagues and me to visit [Name of School] in the next few weeks. We are requesting to work with [number of students] as part of a usability study to test tools and item types being considered for NAEP assessments. This study will be conducted by Fulcrum IT, on behalf of the National Center for Education Statistics in regards to the National Assessment of Educational Progress, or NAEP.

In the study, students will be asked to complete tasks on a tablet and answer questions about how easy or hard it is to accomplish computer functions, answer questions in different formats, and understand what the system expects them to do next. Before working on the tablets, students will be asked about their familiarity and frequency of use of laptops, smart phones, and tablets. Student information may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Any student may decline participation for any reason.

There are a few logistical matters to address before we conduct the study. I will summarize them in this email, and then follow-up with a phone call to you to finalize these arrangements, unless you prefer to finalize the arrangements over email.

Study Date(s)

We expect each user testing session to take approximately 75 minutes to complete. We could come to your school the week of [Date]; or we can arrange different dates with you.

Student Selection for Study

Please identify up to [number of students] [appropriate grade] students to participate in the study - approximately half boys and half girls. The students should be generally representative of your school makeup (e.g., race, SES, familiarity with computers). The purpose of this study is to examine the usability of assessment tools and item types for a wide range of students.

Space Requirements

When we visit to conduct the study, we will need to use a conference room, classroom, or space in your media center or library, if private, with room for up to three students with tablets and four research associates with laptops. We will bring the tablets and related equipment, and set up everything that we need. We would like to meet with up to three students at a time for up to 75 minutes each, but we are flexible depending on your class schedules and space considerations.

Parent/Guardian Notification

Parents/guardians of the students participating in the study should be notified by letter. A sample letter is attached to this email; please provide this letter to the parents/guardians.

Other Logistics

It would be helpful to know your school day schedule so that we can plan to arrive 45 minutes before the first testing session. In addition, please let me know where we should park, and where we should check in with school staff upon arrival.

I will call you within the next few days to determine which dates might work for you and to answer any questions you might have. Feel free also to contact me by email or phone, as indicated below. We greatly appreciate your school’s cooperation, and will offer a $50 gift card to a school supply store as a token of our appreciation. Thank you for your help.

Sincerely,

Gregory Anderson, Ph.D.

Senior Usability Expert / Study Coordinator

uxteam@naepims.org

703-543-2877

Appendix C: Usability Study Sample School Contact Script

NAEP 2014-2015 Usability Study

School Contact Script

Note: Information in grey boxes is not to be read; rather, these are instructions to the Fulcrum IT staff.

Hello, _________________________ (name of school coordinator). My name is __________________ and I am calling from Fulcrum IT, which has been contracted to conduct a study on behalf of the National Center for Education Statistics about the National Assessment of Educational Progress, or NAEP. I am following up on an email I sent to you a few days ago describing the study. This call should only take a few minutes. Would now be a good time to finalize arrangements for the study?

	Yes	 Go to question 1.
	No	 Continue

If not, when would be a good time? ___________________________

1. In my email, I proposed the week of [Date] for us to conduct the study. Is there a day during that week that works for you?

	Yes	 Date 1: ________________
	No	 Arrange new date: _____________________________________

2. Now let’s discuss the study logistics. We’ll need a quiet space with desks or tables, in which to set up tablets and laptop computers for the students and research associates. We will be working with the students in groups of three, and we need enough room for four tables on which to set up our equipment and work with the students. Three of the tables will be for student-research associate pairs and their equipment, while the fourth will be for the study supervisor and the server equipment. In the past we have worked in the library, an empty classroom or a portable. We will also need access to power for the laptops and tablets. We have our own extension cords, so just need a place to plug them in. If you know where you’d like us to work, I can write that down now, or you can just show us when we arrive.

 Testing Location: ___________________________ (if known)

3. We usually schedule three groups, for a total of nine students, per day. Even though the students will be working for no more than 75 minutes, sessions will need to be spaced at least 90 minutes apart to allow for transition time between groups of students. Sessions should be scheduled taking into account lunch times and other daily routines that a student might need to participate in. I’d like to write down the start times for the three sessions now, and then you can give us a list of which students are in which sessions when we arrive.

	Date:_____________________________
Students Group #1	Time: ___________	Location: ______________
Students Group #2	Time: ___________	Location: ______________
Students Group #3	Time: ___________	Location: ______________

My team and I will need about 45 minutes before the first group is scheduled to set up and prepare. What time should we arrive? ______ (time), is that OK? May we have access to __________________ (the testing location) at that time?

4. Now, let’s review the status of parent/guardian notification. Have the parents and guardians been notified?

	Yes	 Continue
	No	 Skip to question 5.

Please keep a copy of the letter for me that I’ll collect on the morning of the usability study.

Were notifications sent to only the parents and guardians of the selected students or to the parents and guardians of all the students in the [specify grade] grade?

	Parents/Guardians of selected students only	 Skip to question 6.
	Parents/guardians of all students in the [specify grade] grade.	On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible.
		 Skip to question 6.

5. As a reminder, the parents and guardians of the selected students must be notified before the usability study can take place. I’d like to call you back to confirm the letter has been sent. When is a good time for me to make that call?

 Callback date: _______________________ Time: ________________

Also, I’ll need a copy of the letter for my records, which I’ll collect from you on the morning of the usability study.

Do you plan to notify the parents/guardians of only the selected students or the parents and guardians of all the students in the [specify grade] grade?

	Parents/Guardians of selected students only	 Skip to question 6.
	Parents/guardians of all students in the [specify grade] grade.	On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible.

6. I have just a few other logistical questions to ask you, so that the study runs as smoothly as possible.

When my team and I arrive at your school on the morning of the study, where should we park?

_______________________________________________________________________________________

We will check in at the main office. Are there other procedures or protocols specific to your school that we need to follow?

_______________________________________________________________________________________

Is there anything scheduled that might interrupt the usability session, such as a fire drill or assembly?

_______________________________________________________________________________________

How should we contact the office from the study location if we need assistance should there be an emergency?

_______________________________________________________________________________________

How will we get the students for each session? Should we notify the office when we are ready for the next group of students?

_______________________________________________________________________________________

How do you announce school closings or delays, such as a snow day or weather delay? By what time are these decisions made?

_______________________________________________________________________________________

Our team lead will want to let you know when we are leaving, and will have a gift card for the school as a thank you for participating in our study. Do you know which store you’d like the gift card for?

_______________________________________________________________________________________

7. Those are all of the questions I have for you at this time. Do you have any questions for me?

If you have any questions before the usability study date, I can be reached at ________________________ (telephone number) or this email address _________________________ (email address).

Thank you for your time.

NOTES:

Appendix D: Usability Study Parent/Guardian Notification Letter

NAEP PARENT/GUARDIAN NOTIFICATION LETTER

Usability Evaluation of NAEP Assessment Tools & Item Types

[School Letterhead] [Date]

Dear Parent or Guardian:

We are pleased to inform you that [school name] has been selected to participate in the Usability Study for tools and item types for the National Assessment of Educational Progress (NAEP). This study will be used to guide development of future NAEP assessments.

NAEP is the largest continuing and nationally representative assessment of what our nation’s students know and can do. The results of NAEP are released as The Nation’s Report Card, which provides information about student achievement to educators, parents, policymakers, and the public.

In our school, the usability study will be conducted on [date]. Your child may be selected to participate. The study involves observing individual students working for 75 minutes on touch screen tablets and asking them questions about how easy or hard it is to perform tasks, answer questions in different formats, and understand what the system expects them to do next. Their comments and actions will be recorded for later analysis. Before working on the tablets, students will be asked about their familiarity and frequency of use of laptops, smart phones, and tablets.

It will take about 75 minutes for most students to complete the study. The screens of the tablets will be recorded, along with the audio from the session. No cameras will be used, and no images or video of your child or their surroundings will be captured. The researchers will collect no identifying information on the students. Each student will be identified only by a participant number in the study. All the information your child provides may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

Your child will represent many other students, so participation is very important. However, your child’s participation in the study is voluntary. In addition, your child may decide not to participate or to end his or her participation before the study is complete. If you do not want your child to participate, please notify me in writing by [date].

There is no need to prepare for this study. As a token of appreciation for participating in the study, your child will get to keep a set of earbuds given to them to use during the study.

Students are given an opportunity to ask questions before, during, and after the study. If you have any questions before or after the study, contact the lead field researcher, Gregory Anderson, Fulcrum IT Services, LLC, 703-543-2877 (ganderson@fulcrumit.com). If you would like additional information about the NAEP assessments, please visit http://nces.ed.gov/nationsreportcard/. If you have any other questions, please contact me at [telephone number] or via email at [email address].

Sincerely,

School Principal

Appendix E: Thank You Email to Participating School

Usability Evaluation of Touch-Screen Tablets

[Date]

Dear [Head of School]:

I wanted to write you personally to thank you for participating in our NAEP usability study. The results of this study will be used to improve the delivery of NAEP assessments, and make participation in these assessments a positive experience for everyone involved. If you have any feedback for us about your experience participating in this usability study, don’t hesitate to contact me or the project manager at Fulcrum IT.

Sincerely,

Gregory Anderson, Ph.D.,

Senior Usability Expert / Study Coordinator

Fulcrum IT

(703) 543-2877

Project Manager: Harlan Messinger

(703) 543-2949

2