NAEP Tablet Usability Study 2013 Appendices

NAEP Tablet Usability Study 2013 Appendices.doc

NCES Cognitive, Pilot, and Field Test Studies System

NAEP Tablet Usability Study 2013 Appendices

OMB: 1850-0803

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1850-0803 can be found here:

Document [doc]

Download: doc | pdf

National Center for Education Statistics

National Assessment of Educational Progress

Appendices

Usability Study for Use of Touch-Screen Tablets

in NAEP Assessments

OMB# 1850-0803 v.87

October 30, 2013

Volume I: Appendices

Appendix A: Usability Study Sample E-mail Contact with Participating Schools 1

Appendix B: Usability Study Sample School Contact Script 2

Appendix C: Usability Study Sample School Contact Script-Subsequent Sessions 5

Appendix D: Usability Study Parent/Guardian Notification Letter 8

Appendix E: Thank You Email to Participating School 9

Appendix A: Usability Study Sample E-mail Contact with Participating Schools

E-mail Subject Line: Arrangements for NAEP Usability Study Date:

Dear [NAME of School Coordinator],

[NAME of NAEP State Coordinator] forwarded your name to me to work out arrangements for my colleagues and me to visit [Name of School] in the next few weeks. We are requesting to work with [number of students] as part of a usability study to test our system on new touch-screen tablets. This study will be conducted by Fulcrum IT, on behalf of the National Center for Education Statistics in regards to the National Assessment of Educational Progress, or NAEP.

In the study, students will be asked to complete tasks on a tablet and answer questions about how easy or hard it is to accomplish computer functions, answer questions in different formats, and understand what the system expects them to do next. Before working on the tablets, students will be asked about their familiarity and frequency of use of laptops, smart phones, and tablets. Student information may only be used for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Education Sciences Reform Act of 2002, 20 U.S.C §9573). Any student may decline participation for any reason.

There are a few logistical matters to address before we conduct the study. I will summarize them in this email, and then follow-up with a phone call to you to finalize these arrangements, unless you prefer to finalize the arrangements over email.

Study Date(s)

We expect each user testing session to take approximately 75 minutes to complete. We could come to your school the week of [Date]; or we can arrange different dates with you.

Student Selection for Study

Please identify up to [number of students] [appropriate grade] students to participate in the study - approximately half boys and half girls. The students should be generally representative of your school makeup (e.g., race, SES, familiarity with computers). The purpose of this study is to examine the usability of tablets for a wide range of students. Therefore, if possible, it is important to include some students that have limited familiarity with tablets and/or computers.

Space Requirements

When we visit to conduct the study, we will need to use a conference room, classroom, or space in your media center or library, if private, with room for up to three students with tablets and 4 research associates with laptops. We will bring the tablets and related equipment, and set up everything that we need. We would like to meet with up to three students at a time for up to 75 minutes each, but we are flexible depending on your class schedules and space considerations.

Parent/Guardian Notification

Parents/guardians of the students participating in the study should be notified by letter. A sample letter is attached to this email; please provide this letter to the parents/guardians.

Other Logistics

It would be helpful to know your school day schedule so that we can plan to arrive 45 minutes before the first testing session. In addition, please let me know where we should park, and where we should check in with school staff upon arrival.

I will call you within the next few days to determine which dates might work for you and to answer any questions you might have. Feel free also to contact me by email or phone, as indicated below. We greatly appreciate your school’s cooperation, and will offer a $50 gift card to a school supply store as a token of our appreciation. Thank you for your help.

Sincerely,

Gregory Anderson, Ph.D.

Senior Usability Expert

Fulcrum IT Services, LLC

ganderson@fulcrumit.com

703-543-2877

Appendix B: Usability Study Sample School Contact Script

NAEP 2013 Tablet Usability Study

School Contact Script

Note: Information in grey boxes is not to be read; rather, these are instructions to the Fulcrum IT staff.

Hello, _________________________ (name of school coordinator). My name is __________________ and I am calling from Fulcrum IT, which has been contracted to conduct a study on behalf of the National Center for Education Statistics about the National Assessment of Educational Progress, or NAEP. I am following up on an email I sent to you a few days ago describing the study. This call should only take a few minutes. Would now be a good time to finalize arrangements for the study?

	Yes
	No

If not, when would be a good time? ___________________________

1. In my email, I proposed the week of [Date] for us to conduct the study. Is there a day during that week that works for you?

	Yes	 Date 1: ________________
	No	 Arrange new date: _____________________________________

2. Now let’s discuss the study logistics. We would like to work with [number of students] total, divided into 2-3 groups, for 75 minutes each. Students will be paired with a staff member from my company who will facilitate the study session. We’ll need a quiet space with desks or tables, in which to set up tablets and laptop computers for the students and facilitators.

	Date:_____________________________
Students Group #1	Time: ___________	Location: ______________
Students Group #2	Time: ___________	Location: ______________
Students Group #3	Time: ___________	Location: ______________

My team and I will need about 45 minutes before the first group is scheduled to set up and prepare. What time should we arrive? ______ (time), is that OK? May we have access to __________________ (the testing location) at that time?

3. Were you able to identify [number of students] [specify grade] students, generally representative of your school makeup (e.g., race, familiarity with computers)?

	Yes	 Continue
	No	 Provide assistance in selecting as diverse a group of students as possible.

As mentioned earlier, we’ll be working with the selected students a few at a time. When I arrive at your school on the morning of the usability study, please provide me with a list of the selected students, along with information about when the students are available.

4. Now, let’s review the status of parent/guardian notification. Have the parents and guardians been notified?

	Yes	 Continue
	No	 Skip to question 5.

Please keep a copy of the letter for me that I’ll collect on the morning of the usability study.

Were notifications sent to only the parents and guardians of the selected students or to the parents and guardians of all the students in the [specify grade] grade?

	Parents/Guardians of selected students only	 Skip to question 6.
	Parents/guardians of all students in the [specify grade] grade.	On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible.
		 Skip to question 6.

5. As a reminder, the parents and guardians of the selected students must be notified before the usability study can take place. I’d like to call you back to confirm the letter has been sent. When is a good time for me to make that call?

 Callback date _______________________ Time ________________

Also, I’ll need a copy of the letter for my records, which I’ll collect from you on the morning of the usability study.

Do you plan to notify the parents/guardians of only the selected students or the parents and guardians of all the students in the [specify grade] grade?

	Parents/Guardians of selected students only	 Skip to question 6.
	Parents/guardians of all students in the [specify grade] grade.	On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible.

6. I have just a few other logistical questions to ask you, so that the study runs as smoothly as possible.

When my team and I arrive at your school on the morning of the study, where should we park?

_______________________________________________________________________________________

We will check in at the main office. Are there other procedures or protocols specific to your school that we need to follow?

_______________________________________________________________________________________

Is there anything scheduled that might interrupt the usability session, such as a fire drill or assembly?

_______________________________________________________________________________________

How should we contact the office from the study location if we need assistance should there be an emergency?

_______________________________________________________________________________________

How do you announce school closings or delays, such as a snow day or weather delay? By what time are these decisions made?

_______________________________________________________________________________________

Our team lead will want to let you know when we are leaving, and will have a gift card for the school as a thank you for participating in our study. With whom should he speak?

_______________________________________________________________________________________

7. Those are all of the questions I have for you at this time. Do you have any questions for me?

If you have any questions before the usability study date, I can be reached at ________________________ (telephone number) or this e-mail address _________________________ (e-mail address).

Thank you for your time.

NOTES:

Appendix C: Usability Study Sample School Contact Script- Subsequent Sessions

NAEP 2013 Tablet Usability Study

School Contact Script

Note: Information in grey boxes is not to be read; rather, these are instructions to the Fulcrum IT staff.

Hello, _________________________ (name of school coordinator). My name is __________________ and I am calling from Fulcrum IT, which has been contracted to conduct a study on behalf of the National Center for Education Statistics about the National Assessment of Educational Progress, or NAEP. I am calling to make arrangements for our next study session at your school. This call should only take a few minutes. Would now be a good time to finalize arrangements for the study?

	Yes
	No

If not, when would be a good time? ___________________________

1. Earlier, we had talked about the week of [Date] for us to conduct this study session. Is there a day during that week that works for you?

	Yes	 Date 1: ________________
	No	 Arrange new date: _____________________________________

2. Now let’s discuss the study logistics. We would like to work with [number of students] total, divided into 2-3 groups, for 75 minutes each. Students will be paired with a staff member from my company who will facilitate the study session. Last time we were here, we worked in (location). Would you like us to use the same space this time?

	Yes
	No

If not, where would you like us to work? ___________________________

	Date:_____________________________
Students Group #1	Time: ___________	Location: ______________
Students Group #2	Time: ___________	Location: ______________
Students Group #3	Time: ___________	Location: ______________

3. Were you able to identify [number of students] [specify grade] students, generally representative of your school makeup (e.g., race, familiarity with computers)?

	Yes	 Continue
	No	 Provide assistance in selecting as diverse a group of students as possible.

4. Now, let’s review the status of parent/guardian notification. Have the parents and guardians been notified?

	Yes	 Continue
	No	 Skip to question 5.

Please keep a copy of the letter for me that I’ll collect on the morning of the usability study.

Were notifications sent to only the parents and guardians of the selected students or to the parents and guardians of all the students in the [specify grade] grade?

	Parents/Guardians of selected students only	 Skip to question 6.
	Parents/guardians of all students in the [specify grade] grade.	On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible.
		 Skip to question 6.

 Callback date _______________________ Time ________________

Also, I’ll need a copy of the letter for my records, which I’ll collect from you on the morning of the usability study.

Do you plan to notify the parents/guardians of only the selected students or the parents and guardians of all the students in the [specify grade] grade?

	Parents/Guardians of selected students only	 Skip to question 6.
	Parents/guardians of all students in the [specify grade] grade.	On the morning of the usability study, if any of the selected students are absent, we’d like to substitute another [specify grade] grade student in their place so that we can test as many students as possible.