National Assessment of Educational Progress (NAEP) Survey Assessments Innovations Lab (SAIL) English Language Arts (ELA) Collaboration and Inquiry Study 2017

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850–0803. The time required to complete this information collection is estimated to average 150 minutes, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this form, please write to: National Assessment of Educational Progress, National Center for Education Statistics, PCP, 550 12^th St. SW, 4^th Floor, Washington, DC 20024.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your participation is voluntary and without the permission of your parent or guardian your answers may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)]. With your parent or guardian’s permission some of your responses or clips from videos may be selected for use in research reports or presentations¹.

OMB No. 1850-0803 Approval Expires 07/31/2019

PART B. PLAY TESTING

I. Script for Introduction to Study, Goals, and Approach

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the play testing sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of the students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending the meaning of instructions or questions. Text placed in brackets is generic text that will be tailored to suit this specific research project (Virtual World for ELA Collaborative Inquiry).

Hi, our names are <NAMES>and we are from <INSTITUTION>. We are conducting research on how students work together to gather, process, and evaluate information from multiple sources, within a virtual, computer-based environment. We are doing this research for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. The system you’ll be seeing today isn’t part of the test right now, but we are doing this research to see how it works. Your input will help us make the system work as well as it can for all students. We are going to ask for your opinions based on some activities that we will ask you to do using the system. You are here to give us your thoughts about your experiences of interacting with the virtual world for information gathering. We’d like to hear about what you think is good, what you think is not good, and if you think anything is difficult or confusing.

My colleagues and I will take notes on what you are saying—please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything—you are helping us by giving us your opinions and thoughts and by interacting with the system and with each other so we can see how it works when it is being used in a collaborative fashion. There are no right or wrong thoughts or opinions or actions.

This process is being recorded so that we can review it later. Without the permission of your parent or guardian, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. § 9573). We will look at what you say later so that we can understand how our system is working and how we can make it better. With parental/guardian permission, some responses or clips from videos may be selected for use in research reports or presentations.

Overall, this session should take about <say length> minutes. If at any time you decide you don’t want to go on, that is your choice, and you may stop. [If the student is no longer interested in participating, thank the student for his/her time and end the interview].

II. Play Testing Method

II A. Pre-Task Questionnaire Instructions

At the beginning of the play testing session, students will individually complete a pre-task questionnaire about their relevant prior experiences, beliefs, and knowledge (see Part D Section I). Please provide the following instructions:

First, you will each complete a questionnaire to provide some information about relevant prior experiences, perspectives, and knowledge. Please answer all the questions to the best of your ability, and let me know when you are ready to move on to the task.

When both students have submitted their pre-test questionnaire responses, they can begin the task.

II B. Collaborative Task Play Testing: Starting the Tasks

In play testing sessions, students will work together in one of two modes, depending on the phase of play testing. Students will work either (1) in-person using a single, shared computer interface, or (2) remotely using the collaborative features of the digital platform to connect two, separate computer interfaces and to facilitate interaction with remotely-located partners (i.e., either in separate rooms or on opposite sides of a large room, so that partners cannot directly interact without using the digital collaborative platform). For each session, student pairs will be supervised by a facilitator (plus an additional observer). The facilitator and observer are responsible for setting up the task interface in the appropriate delivery mode (e.g., connecting computers or tablets to the Internet, setting up video cameras and other recording devices, ensuring data capture is enabled).

Students will be encouraged to freely verbalize their thoughts as they interact with their partner, using an approach that will be less formalized than is typical for cognitive interviews, so it doesn’t require a formalized practice session. There will be ad hoc questioning by the facilitator as needed (see below). The primary instructions for the two collaborative modes are as follows.

[FOR STUDENTS WORKING TOGETHER IN PERSON (Face-to-Face Collaboration)]
You will now work together to complete a collaborative task. As you work through these activities, you will talk with each other as you go along, working to complete the inquiry task together. For some parts of the activity, such as reading a text, we will provide a second screen so that you can both complete the reading task. Otherwise, you will share this computer and work together to input your responses and to complete the task as a team. We might also ask you a few questions now and then.
[FOR STUDENTS WORKING TOGETHER VIRTUALLY (Remote Collaboration)]
You will now work together to complete a collaborative task. As you work through these activities, you will use the features of the system to work together and communicate with each other, even though you will be sitting [in separate rooms/on opposite sides of this room]. Although you will use separate computers, you will follow the prompts displayed on the collaborative screen to help you work together to input your responses and to complete the task as a team. We might also ask you a few questions now and then.

For the Virtual World for Collaborative Inquiry study, students will be asked to complete tasks created around the manipulation of tools in the Virtual World. A facilitator will work with pairs of students and describe what students should do. Tasks will be based on principles of conducting research and inquiry from multiple information sources. Examples of prompts are:

[FOR STUDENTS WORKING TOGETHER IN PERSON (Face-to-Face Collaboration)]
You will help us to test how easy it is to use the interactive tools in the virtual world to accomplish some basic information-gathering and evaluation tasks, and how quickly you can perform these tasks while working together. We would like you to talk with each other during your process of completing the tasks, which means we’d like you to say out loud all the thoughts that come into in your head as you work together to complete the inquiry task. At certain points, we will give you instructions for how we’d like you to work together on a part of the task—please be sure to follow these instructions whenever they are introduced.

[FOR STUDENTS WORKING TOGETHER VIRTUALLY (Remote Collaboration)]
You will help us to test how easy it is to use the digital collaborative platform and the interactive tools in the virtual world to accomplish some basic information-gathering and evaluation tasks, and how quickly you can perform these tasks while working together. Although you are seated separately, we would like you to communicate with each other during your process of completing the tasks, which means we’d like you to use the communication tools to express all the thoughts that come into in your head as you work together to complete the inquiry task. At certain points, instructions to collaborate will appear on the screen—please be sure to follow these instructions whenever they appear.

[ALL STUDENTS] In this task you will work together to conduct a research project. Can you work together to figure out where to find and select information sources that you think are relevant to this task; read and summarize what you learned; use available tools to analyze; evaluate, and compare sources; and then use those sources to answer a research question?

Once the facilitators have explained the activity and asked the students to begin, they will generally observe them with little interruption or minimal reactive influences on their thinking, listening to the partners’ conversation for insights into what the students are thinking about the items and tasks. However, in the Face-to-Face version, facilitators will verbally pose specific collaborative prompts to students at pre-determined points (e.g., “Make sure you both agree before submitting your answer”). These prompts will be evaluated during the Face-to-Face implementation, and will be programmed into the digital delivery system for the Remote version.

Facilitators will observe students, take notes, and will pay attention to the types of interactions between students as well as verbal answers to the questions above and write them down. They will also look for evidence of engagement, boredom, frustration, misunderstanding, or confusion in what students say to each other and in their facial expressions or body language (see Part B, section II D for further details on making observations of student behaviors and actions).

If appropriate, staff can use these moments to probe for more information. Intermittently during the activity, or following a given activity as appropriate, the facilitators can ask targeted questions such as those in the following list:

Is this difficult? Why? / Why not?

Do you find the problem in this task interesting – why or why not?

Are there any things that seem confusing here?

Do you understand what you need to do?

Do you think you can do what is needed here?

Do you both agree on what you need to do to achieve <part of the task>?

Was it difficult to work together on this task? What activities are easier, versus more difficult?

Do the prompt(s) used in the system help you to work with your partner?

Have you done anything like this before?

How could this activity / the system be improved? Could it be clearer? Easier to use?

Were there aspects of what you just did that you didn’t like? Why?

This activity asked you to solve a problem about <insert focus of activity>. Have you learned about this before, either in school or anywhere outside of school?

Occasionally staff observing play testing sessions will introduce a question to students talking about the activity to get more information from them, particularly in cases when students talk about issues related to their interest (or non-interest) in the activities, or confusing aspects of an activity or of the system. In such cases, observers might ask something like, “Can you tell me more about what you said?” If students do not provide sufficient comments on targeted parts, a staff member may ask pairs of students if they had any thoughts about those aspects, using questions such as those described above, but focused on specific aspects or issues (e.g., the comprehensibility of the instructions).

When students have completed the task, they can begin the post-task questionnaire.

II C. Post-Task Questionnaire

After completing the collaborative inquiry task, students will each work individually to complete a brief questionnaire about their experiences with the collaborative activity, and with the virtual world and digital collaborative platform (see Part D Section II). Please provide the following instructions:

Now, you will each complete a questionnaire to provide your impressions of the collaborative task and the system you just used. Please answer all the questions to the best of your ability, and let me know when you are finished. If you’re not sure how to answer any of the questions, please let me know.

Answer any student questions about the items and make a note of the problematic questions; these items may be dropped prior to the small-scale tryouts. When both students have submitted their questionnaires, move on to the verbal questioning (time permitting).

II D. Notes on Student Actions and Behaviors

As students are proceeding through each activity they will be video recorded. In addition, a screen-capture system and/or video camera focusing on the screen, as well as digital logfile capture will record evidence of students’ interactions with the system or task. During the session the interviewer should take notes about any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task or system. The following are examples of such behaviors for interviewers to note:

Do the students express signs of confusion, boredom, or excitement?
How do the students use the tools provided?
Do the students rapidly move through the activity or take their time?
Do the students spend a lot of time on a particular aspect?

Interviewers should also make a note of any places in the task that appear valuable for follow-up with some additional questions after the task is completed (see section III).

III. Post-Play Testing Follow-Up Questions (Verbal Probing)

After completing the play testing process for a task, interviewers will follow up with a brief period of focused retrospective questioning. The post-task questions will comprise of the following:

One set of standardized post-task questions that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Up to three additional targeted questions. These will be generated by research staff prior to testing.

Standardized Questions for All Tasks: Perceptions of Collaborative Activity

What did you think about doing this kind of collaboration with your partner? Did you find it challenging to work together to complete the tasks, why or why not? Did you feel your collaboration was effective, why or why not? What changes would you make to either the task, or the system that allowed you to collaborate?

Additional Questions: Task-Specific Issues

The purpose of the additional post-task questions is to capture more information on issues such as students’ actions during the activities, particular aspects of knowledge or skills targeted in the activities, and general reflections about the activities.

The following list shows some sample question frames that research staff will use to build questions. The specifics of each question will be generated via an informal task analysis in which key points for understanding student thinking will be identified.

Let’s think about the part where you [describe key action or aspect]. Can you tell me more about what you both were thinking when you were doing that?

This [button or other tool] was on the screen to allow you to [describe action]. Tell me about how and when you used that, or if you didn’t really use it, why was that?

When you saw [information/text/graph/image/prompt to collaborate] at this point, what were you thinking? What was in your mind?

If students’ answers are unclear or not very explicit, the interviewer should use prompts to encourage them to say more, for example:

Can you say a bit more about that?

What else were you thinking?

Anything else?

That’s interesting. Tell me more about that.

Optional: Interviewer-Generated Questions

In addition, if the interviewer noted some especially interesting behavior during the task, he or she can ask additional ad hoc questions about these specific instances, referring to notes made during the task. Interviewers should use their judgment about the need for and value of additional questions, based on the student’s behavior during the task, and based on time constraints.

Some examples of potential interviewer-generated ad hoc questions are:

I noticed on this part of the task that you paused for a while. Can you tell me more about why you paused and what you were thinking at this point?

I noticed on this part of the task that you did not use [describe tool or action]. Can you tell me more about why you didn’t use that?

I noticed that on this part of the task you spent some time looking at/going back-and-forth between [describe images, text, or actions]. Can you tell me more about what was going on at that point?

IV. Debriefing and Thank You for Play Testing

At the end, students will be thanked for their participation and for helping us with our research. Students will be given gift cards for their participation at the conclusion of each play testing session.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing today.

Is there anything else you would like to tell me about working on these tasks?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to study these processes and improve our syste

PART C. TRYOUTS

I. Introduction to Study Script

Text written in italics is to be read aloud by the staff facilitator. The facilitator will describe this information to the pair(s) of participants, in an afterschool setting, community site, or a research facility (e.g., the ETS usability lab, or the EurekaFacts facility in Rockville). One facilitator can give instructions while another ensures that students are prepared to view the tasks (e.g., setting up computers or tablets appropriately, ensuring that recording devices and microphones are working properly, etc.).

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to introduce the tasks in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending the meaning of instructions or questions. Text placed in brackets is generic text that will be tailored to suit this specific research project (Virtual World for ELA Collaborative Inquiry).

Introduction Script:

Hello, our name(s) are ______ and we work for <INSTITUTION>. It’s nice to meet you and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out a new virtual world for collaborative information gathering. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is conducted by the U.S. Department of Education. We are doing research on how students work together to gather, process, and evaluate information from multiple sources, within a virtual, computer-based environment. The system you’ll be seeing today isn’t part of the test right now, but we are doing this research to see how the system works. Today we’ll be asking you to work in pairs to try out a new task that is being developed and studied for 9^th/10^th graders. You will be asked to use features of the system to communicate with each other as you work to complete the task.

My colleagues and I will take notes on what you are saying to each other—please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything—you are helping us by giving us your opinions and thoughts and by interacting with the system and with each other so we can see how it works when it is being used in a collaborative fashion. There are no right or wrong thoughts or opinions or actions.

This tryout session is being recorded so researchers can review it later. Without the permission of your parent or guardian, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). We will look at what you say later so that we can understand how our system is working and how we can make it better. With parental/guardian permission, some responses or clips from videos may be selected for use in research reports or presentations.

Overall, this session should take about 150 minutes. If at any time you decide you do not want to go on, that is your choice and you may stop. Do you have any questions?

Ask the students if they have any questions. If a student is no longer interested in participating, thank the student for his/her time and the student can end the tryout session.

By the tryout phase, tasks or activities will be self-contained and students will be able to work through them at their normal pace, without interruption and without needing support from the facilitator. As far as possible, tryouts should represent naturalistic test-taking conditions. While the facilitator will still be responsible for delivering pre-test, task, and post-test instructions to students, in the tryout phase, the facilitator will not need to deliver verbal instructions during the collaborative task, since they will be presented to the students via the system (the Virtual World for Collaborative Inquiry). The tasks will be presented in a way that is sufficiently self-contained that students can work through them at their own pace, without additional input from the facilitator.

II. Tryout Study Method

II A. Pre-Task Questionnaire

At the beginning of the tryout session, students will individually complete a pre-task questionnaire about their relevant prior experiences, beliefs, and knowledge (see Part D Section I). Students should be instructed as follows:

Please work on your own to complete the pre-task questionnaire. Answer all questions to the best of your ability.

After both students have completed the pre-task questionnaire, they can begin the collaborative task.

II B. Collaborative Task Tryout

In small-scale tryout sessions, students will work together remotely using the collaborative features of the digital platform to connect two, separate computer interfaces and to facilitate interaction with remotely-located partners (i.e., either in separate rooms or on opposite sides of a large room, so that partners cannot directly interact without using the digital collaborative platform). For each session, student pairs will be supervised by a facilitator (plus an additional observer). Students will be encouraged to freely verbalize their thoughts as they interact with their partner, using an approach that will be less formalized than is typical for cognitive interviews, so it doesn’t require a formalized practice session. Students will work uninterrupted without interference from the facilitators.

The primary instructions for the tryouts are as follows:

You will now work together to complete a collaborative task. As you work through these activities, you will use the features of the system to work together and communicate with each other, even though you will be sitting [in separate rooms/on opposite sides of this room]. Although you will use separate computers, you will follow the prompts displayed on the collaborative screen to help you work together to input your responses and to complete the task as a team.

In this task you will work together to conduct a research project. Can you work together to figure out where to find and select information sources that you think are relevant to this task; read and summarize what you learned; use available tools to analyze; evaluate, and compare sources; and then use those sources to answer a research question?

You aren’t being graded on your work today, so it’s fine if you don’t know the answer or don’t know how to go about any of the activities we are asking you to do. But try to do your best, and try to complete all of the activities, or as much as you can. I won’t be interrupting, I’ll just be here observing, so you and your partner can work through the tasks at your own pace.

You will have [indicate correct number of minutes] minutes to do this. Again, you will not be graded on what you do during the task, but please try to work through this as if it was a real test. Okay, you can go ahead and begin with the task.

When students have completed the collaborative inquiry task, they should move on to the post-test.

II C. Post-Task Questionnaire

Now, you will each complete a questionnaire to provide your impressions of the collaborative task and the system you just used. Please answer all the questions to the best of your ability, and let me know when you are finished.

When both students have submitted their questionnaires, move on to the debriefing (see III).

III. Debriefing and Thank You for Tryouts

After submitting their post-task questionnaire, students will be thanked for their participation and for helping us with our research. Students will be given gift cards for their participation at the conclusion of each tryout session. If time allows, researchers may ask students the following debriefing questions orally and record the students’ responses.

Before we finish, I’d like to hear any other thoughts you have about the collaborative inquiry task you both worked on today.

Is there anything you would like to tell me about working on this collaborative task?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to study these processes and improve our system.

PART D. QUESTIONNAIRE ITEMS FOR PLAY TESTING AND TRYOUTS

I. Pre-Task Questionnaire Items

In order to support analysis of data obtained from the tryout study, participants in the tryouts will be asked to complete a brief individual questionnaire prior to completion of the collaborative assessment tasks designed in the virtual world environment. These questionnaire items will provide information about students’ relevant prior experiences, perspectives, and knowledge. The questionnaire items of this instrument appear below. The measures to be collected include students’ age and gender; types/patterns of access to computers and the Internet (Sections 1 and 2); self-efficacy of online reading and research activities (Section 3²). frequency of experiences with the Internet, research, and collaboration (Section 4); epistemic opinions regarding social issues (Section 5³); dispositions to disagreement (Section 6⁴; modified from Infante & Rancer); preferences for working collaboratively (Section 7); and prior knowledge of several key concepts related to the scenario context used in the Collaborative Inquiry task (Section 8). If we make any changes to the specific questions that will be asked during the tryout study, we will provide to OMB an updated version of this document. The questionnaires will be pre-tested with play testing participants to evaluate their comprehensibility to the students, the variability of students’ responses, and the time it will take to administer the items. A final pool of questionnaire items will be selected based on the results of the play testing sessions.

Age __________

Gender: MALE FEMALE

1. How do you access the Internet?

	Yes	No
I have access to the Internet at home from a computer.
I can use a tablet from to access to the Internet.
I have a smart phone to access to the Internet.

2. On a typical day, for how long do you use the computer, tablet or smart phone, on average, for each of the following activities?

	Not at all	Less than 1 hour	1-2 hours	More than 2 hours
Entertainment (e.g., playing games, watching videos, listening music)
Communicating with friends (e.g., chat, WhatsApp, Facebook, Skype)
Searching for information (e.g., about hobbies, schoolwork, music, fashion etc.)
Sharing texts, pictures or videos made by myself on the Internet (e.g., in a blog, Twitter, YouTube)

3. How confident you are when you do the following activities?

I am confident that I can	Not at all confident	A little confident	Somewhat confident	Quite confident	Very confident
formulate questions that help me to search for information on the Internet.
formulate queries that help me to search for information in a library database.
pose questions to another person to help me gather useful information
choose the best link from the search results (e.g. a list provided by a Google).
skim a source to decide whether or not the information is useful for me.
evaluate whether information in a source is reliable.
combine information from more than one source in a way that makes sense to other people.
write about my research and give reasons to support my thinking.

4. How often do you use the Internet for searching for information?

	Almost never	Seldom (1-2 times a month)	1-2 times a week	Almost every day	Every day
I search for information on the Internet at school for assignments given by the teacher.
I search for information for my homework.
I search for information on the Internet in my free time on the issues that I am interested in (e.g. hobbies, music).
I work with a partner or small group at school to search for information on the Internet.
I work with others in my free time to search for information on the Internet.

5. Read the following six cases and choose one of the alternatives that best describes the situation.

CASE 1.

Robin has one view of why criminals keep going back to crime.

Chris has a different view of why criminals keep going back to crime.

Only one is right.

Both could have some rightness, but one could be more right.