NAEP SAIL Science Development 2014-15

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850–0803. The time required to complete this information collection is estimated to average 90 minutes, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any comments or concerns regarding the status of your individual submission of this form, please write to: National Assessment of Educational Progress, National Center for Education Statistics, 1990 K Street, NW, Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 09/30/2016

Part B. Play Testing Protocol

Script for Introduction to Study, Goals, and Approach

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the play testing sessions in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of the students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending the meaning of instructions or questions. Text placed in brackets is generic text that should be tailored to suit each of the research projects, as needed (Virtual Science Lab or Interactive Virtual Models).

Hi, our names are __ and we are from the Educational Testing Service. We are running a research project on a virtual, computer-based version of [a science lab/the solar system]. We are doing this research for a program funded by the federal government called the National Assessment of Educational Progress (NAEP). NAEP is a test taken by students in grades 4, 8, and 12 in the United States. The system you’ll be seeing today isn’t part of the test right now, but we are doing this research to see how it works. Your input will help us make the system work as well as it can for all students. We are going to ask for your opinions based on some activities that we will ask you to do using the system. You are here to give us your thoughts about your experiences of interacting with the [virtual science lab/virtual solar system]. We’d like to hear about what you think is good, what you think is not good, and if you think anything is difficult or confusing.

My colleagues and I will take notes on what you are saying—please be as honest as possible, talk freely with each other, and don’t worry if you find something confusing. You are not being graded on anything—you are helping us by giving us your opinions and thoughts and by interacting with the system so we can see how it works when it is being used. There are no right or wrong thoughts or opinions or actions.

This process is being recorded so that we can review it later. Unless your parent/guardian gave permission, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). We will look at what you say later, but only so that we can understand how our system is working and how we can make it better. Overall, this session should take about <say length> minutes. If at any time you decide you don’t want to go on, that is your choice, and you may stop.[If the student is no longer interested in participating, thank the student for his/her time and end the interview].

As you work through these activities, [FOR STUDENTS WORKING IN GROUPS] we want you to talk with each other as you go along [or FOR INDIVIDUAL SESSIONS] we want you to think aloud as you go along, so I’ll be asking you questions like “what are you thinking?” so you can tell me what’s in your head as you are thinking about what you are doing. We might also ask you a few questions now and then.

In Play Testing sessions, students will work either individually or in pairs/small groups with a facilitator (plus an additional observer in the case of individual sessions). The think aloud approach will be less formalized than in the cognitive interviews, and there will be more ad hoc questioning by the facilitator as needed (see below), so it doesn’t require a formalized practice session.

For the Virtual Models study, students will work with the Virtual Solar System. A facilitator will start the students off on each activity by verbally describing what they should do. The verbal prompts will ask the students to reason about relevant problems of demonstrate their knowledge. Examples of prompts are:

Imagine you’re out in space, looking at the earth, like this – that dot is where we are. Now use the controllers to figure out or show me why it’s hotter here in the summer than in the winter.

You know how the moon looks like it changes shape throughout the month? Can you figure out or show me why we see it looking like that from the earth? Use the controllers [or FOR STUDENTS WORKING IN PAIRS/GROUPS TELL HIM/HER HOW TO MOVE THE CONTROLLERS,] to show what makes the moon look different at different times.

For the Virtual Science Lab study, students will be asked to complete tasks created around the manipulation of tools in the Virtual Lab. A facilitator will work with individual students or small groups and describe what students should do. Tasks will be based on principles of science inquiry. Examples of prompts are:

You will help us to test how easy it is to use the interactive tools in the virtual lab to accomplish some basic lab tasks, and how quickly you can perform tasks. We would like to you think aloud during your process of completing the tasks, which means we’d like you to say out loud all the thoughts that come into in your head as you work.

The first task you need to complete is to measure 5 grams of salt. Can you figure out where to find and select tools that you think are relevant to this measurement task, move the selected tools into the experiment environment, and then carry out the measurement task?

Once the facilitators have explained the activity and asked the students to begin, they will generally observe them with little interruption or minimal reactive influences on their thinking and will listen to individuals’ verbalized thoughts or small groups’ conversation for insights into what the students are thinking about the items and tasks. Intermittently during the activity, or following a given activity as appropriate, the facilitators can ask targeted questions such as those in this list:

Is this difficult? Why?/Why not?

Are there any things that seem confusing here?

Do you understand what you need to do?

Do you think you can do what is needed here?

Have you done anything like this before?

How could this activity/the system be improved?

Were there aspects of what you just did that you didn’t like? Why?

This activity asked you to solve a problem about <insert focus of activity>. Have you learned about this before, either in school or anywhere outside of school?

Facilitators will observe students, take notes, and will pay attention to verbal answers to the questions above and write them down, but also will look for evidence of engagement, boredom, frustration, misunderstanding, or confusion in what students say to each other and in their facial expressions or body language. If appropriate, staff can use these moments to probe for more information.

Occasionally staff observing play testing sessions will introduce a question to individual students or groups of students talking about the activity to get more information from them, particularly in cases when students talk about issues related to their interest (or non-interest) in the activities, or confusing aspects of an activity or of the system. In such cases, observers might ask something like, “Can you tell me more about what you said?” If students do not provide sufficient comments on targeted parts, a staff member may ask individual students or a group of students if they had any thoughts about those aspects, using questions such as those described above, but focused on specific aspects or issues.

At the end, students will be thanked for their participation and for helping us with our research. Students will be given gift cards for their participation at the conclusion of each play testing session.

Part C. Cognitive Interviews

I. Introduction to Study Script

NOTE: The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of the students and adapt the wording or repeat with different wording if they detect that students are having difficulty comprehending meaning. Text placed in brackets is generic text that should be tailored to suit each of the research projects, as needed (Virtual Science Lab or Interactive Virtual Models). In general, protocols will be adjusted and customized in light of how individual students respond.

Introduction:

Hello, my name is ______ and I work for Educational Testing Service. It’s nice to meet you and thank you very much for helping us out today.

Create small talk to build rapport with the student by asking a question, such as:

What is your favorite subject in school?

When the student responds, follow up with 2 or 3 questions to get the student used to talking, such as:

That’s interesting—why do you enjoy <subject> so much?
What are you studying in <subject> at the moment?
And what’s the best thing you have studied in <subject> so far?

Good. Well, I think you’ll enjoy what we are going to be doing today. First, let me begin by explaining why I am here and what you are going to be doing. You are taking part in a special study looking at a new type of activity to find out how students interact with a [virtual science lab/virtual interactive model of the solar system]. It is related to the National Assessment of Educational Progress, or NAEP for short, a test that is funded by the U.S. Federal Government and run by a center in the Department of Education. NAEP is a test given to students in grades 4, 8, and 12 in the United States. You will help us test out these new systems and try out new activities that in the future may be used with other students. Overall, this session should take about [indicate correct length] minutes.

It’s okay if you don’t know how to do any parts of these activities. I will not be grading your work today, and no one will know that it was you who did the task. My goal is to learn how you react to the activities and to the system you will be using, so please just try to do your best.

If at any time you decide you don’t want to go on, that is your choice, and you may stop.

This interview is being recorded so we can review it later. Unless your parent/guardian gave permission, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573). We will look at what you say later, but only so that we can understand how our test is working and how we can make it better.

We will work together on a few different activities today.

Do you have any questions?

If the student is no longer interested in participating, thank the student for his/her time and end the interview. After answering any questions and giving further explanation, the facilitator will conduct the think-aloud training.

Okay, let’s move on. I want to make sure you have a good idea of what we’re going to be doing. So I am going to give you some practice activities. The practice activities should help you get used to what we will be doing during the real tasks. They should help you understand how we want you to respond. Do you have any other questions before we start?

II. Concurrent Think-Aloud method

II a. Concurrent Think-Aloud: Instructions and Modeling Example Script

NOTE: Text written in italics is to be spoken aloud by the interviewer. The interviewer should not read the script word for word, but should be familiar enough with its contents to conduct the interview in a natural and conversational manner, paraphrasing or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of students. Text placed in brackets is generic text that should be tailored to suit each of the research projects, as needed (Virtual Science Lab or Interactive Virtual Models).

To help us develop and improve our system, we will ask you to complete some activities. We will be video recording you as you do the activities, and a separate recording will capture all of your responses and actions with the system.

I’m going to ask you to do these activities in a way that may be different from what you are used to. Instead of working quietly, I want you to say aloud whatever is in your head as you work through them. We call this “thinking aloud,” because we are asking you to say everything you are thinking out loud.

In a moment, I will give you an example of what it’s like to think aloud. Then I will give you a chance to practice it. You won’t be graded on anything you say while you are thinking aloud. There are no incorrect thoughts, and everything you think and say is important to us.

Okay, now I’m going to show you how to think out loud—this will help you see how I want you to describe what is in your head as you are working on the activities. When I am finished I’ll ask you to try it, so you can see how it works.

Sample Think-Aloud Demonstration:

The example below shows how the facilitator should model the think aloud method for students. The key is to help them see that what is required is a fluid, naturalistic, continuous verbalization of all thoughts as they occur. Research staff may choose alternative tasks or activities as the basis for the demonstration, as appropriate. For example, since the Virtual Models study uses manipulatives in the form of hand-held controllers, a physical manipulation-while-thinking task may provide a suitable example; since the Virtual Science Lab study uses an interactive tablet-based task, a tablet-based activity may provide a suitable example. ETS research staff will select an appropriate task or activity to demonstrate think aloud for each study, and facilitators will be trained to model and elicit this kind of naturalistic think-aloud data using the example.

Since we can’t tell what is going in your head, we need you to “Think Aloud.”

Let me give you an example. [Place example in front of student.] Look at this question. It asks me to look at the five animals and choose two that are the most similar. So I am going to do this task and I will think aloud while I work on it.

Question:

Which two animals below do you think are similar? Circle the two that you think are most similar:

Beetle
Mouse
Crab
Dolphin
Cow

Okay, so, um…. two animals… most similar... Hmm…. just wondering… what is “similar”? Could mean anything…. not sure how to... Well, ok…. So we have beetle, mouse, crab, dolphin, cow. Well…. Hmm… beetle and crab… kind of similar—well, small, lots of legs… move around fast. But crabs live in water… beetle…? No. Mouse… also fast moving and small, but otherwise, not really—it’s a mammal! Hmm, dolphin… that’s in water… but so not like a crab! So… mouse and cow… both mammals… also dolphins… but really so different from mice and cows… Hmm, seems like a hard question… wish I knew what they want… Okay, decision time…umm…okay, I think… let’s see … mouse and cow. Because they both have fur and four legs and walk around on land. Hmm, that seems weird—they’re not really similar! But, well, all the others are too different… so, okay, that’s it… seems like the best choice to me.

Can you see that as I was thinking I was saying all of my thoughts out loud? That is what I want you to try to do today. The point of the think-aloud is to get at whatever is in your head as you are working. Just say aloud the words and the thoughts that come into in your head, as you go along. Don’t try to have a conversation with me or explain anything to me—just have a conversation with yourself, saying out loud anything that pops into your head as you go along.

II b. Concurrent Think-Aloud: Student Practice Script I

Interviewers should place the practice question in front of the student so that he/she can read it. Some students will be silent after reading the question. Students should be immediately encouraged to say whatever they are thinking.

It may be necessary to remind the student to talk aloud as he/she works through the questions and tasks. If necessary, interviewers should use the “Think-Aloud Hints” shown below to prompt the student, being careful not to lead the student and using the hints as reminders to verbalize but in a non-intrusive way. Since the goal is to gain insights into student cognition as it occurs naturally during these activities, it is important to avoid questioning students or otherwise leading or pushing them in any particular direction. It is also important to help students understand that they should not attempt to explain their thinking to the interviewer, in fact they should not even think of the process as describing their thinking for someone else, but rather they should simply say aloud any words that come into their heads, even if they result in broken sentences and unfinished thoughts, so that the process of thinking aloud does not alter or interrupt or the natural flow of the internal thinking process (Ericsson & Simon, 1989; see also Ericsson, 2006; Ericsson & Simon, 1984; Newell & Simon, 1972).

Now I want you to try a think-aloud. You can use this example. Like last time, you have to pick two of these things that you think are the most similar. As you are reading the question and the choices, and as you are figuring out your decision, I want you to say out loud all of the thoughts that come into your head at each moment.

If I don’t hear you speaking, I’ll ask you to keep talking. I’m telling you that now so you won’t think I am criticizing the way you are working. I’ll be reminding you to think aloud if you get quiet because we need to hear all of your thoughts.

Okay, now you try. Go ahead and start working on this question and remember to think aloud as you are doing it.

Which two living things below do you think are most similar? Circle the two that you think are most similar.

Apple tree
Grass
Wheat
Pear tree
Cherry tree

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels okay and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II c. Concurrent Think-Aloud: Student Practice Script II (Optional)

(Use only if you feel the student would benefit from another think-aloud practice before moving on to the actual questions. It may be beneficial to instead model thinking aloud again for the student rather than doing another practice [see think-aloud demonstration in IIa])

If the student struggles to think aloud, or if the verbalizations are not of the right kind, the interviewer should give the student another opportunity to practice and help to provide guidance in terms of how to make their verbalizations more appropriate. The interviewer should praise the student for the first attempt regardless of how good it was, for example: “Very good—let’s do another one before we start the real tasks. Are you ready? Here is the next practice question. Remember to think out loud as you begin to think about this question and all the way through—say whatever pops into your head as you work through it.” During the practice item, the interviewer should prompt the student to think out loud at any point when there are more than a few seconds of silence (see suggested prompts, below).

Which two of the following objects have the most similar properties? Circle the two that are most similar.

Silver coin
Chocolate coin
Gold coin
Blue plastic coin
Brown plastic coin

After the student has finished:

Now that you have practiced, how do you feel about thinking aloud while you are doing the tasks? What questions would you like to ask me? [If the student says he or she feels ok and doesn’t have any questions: Good, then let’s begin our study.] [If the student expresses concerns, says he/she has questions, or appears to be hesitant or reluctant, ask him/her to say more about the concerns or questions, and try to address his/her concerns or uncertainties in a supportive way. If the student indicates he/she does not wish to continue or does not feel comfortable continuing, allow him/her to stop.]

II d. Concurrent Think-Aloud: Starting the Tasks

Now we will move on to the actual task. Remember, as you are working I’d like you to say aloud everything that you’re thinking, and I may remind you to do that if you are quiet. This part should take about [indicate correct length] minutes. Remember, you will not be graded on what you do and there is no right or wrong way to think aloud, as long as you keep saying any thoughts that you are having. Your thoughts will help us make the activities and the system better. I will also have a few questions after you have finished working.

Do you have any questions before we go on? (Answer any questions the student may ask.)

Because the information you provide is so important to us, I am going to be taking notes while you think aloud and answer the questions.

Ready? OK, let’s begin.

The examples below illustrate the kinds of tasks and activities that students will be asked to work on during the cognitive interviews as they think aloud. During a cognitive interview session, a number of these kinds of activities will be given to each student, as time allows. If students become stuck or are visibly having difficulty, the facilitator will help them out with hints, or if a student is having a lot of difficulty, the facilitator can tell them that they have done what is needed and ask them to move on to the next activity or task.

Sample Virtual Science Lab Task Script:

To help us evaluate the Virtual Science Lab, we created a simple task that we would like you to carry out in the virtual lab. Your task is to measure 5 grams of salt. To complete this task in the Virtual Lab, you need to first select and find the relevant lab tools and materials. Then you will use the tool cart to deliver the tools and materials to the workstation to carry out the measurement. Remember to think aloud while doing this and pretend as if I’m not even here. If you fall silent for a while, I will remind you to keep thinking aloud. Are you ready? Then you can go ahead and start.

Sample Virtual Models Activity Script:

To help us evaluate the Virtual Solar System Model, we created a simple task that we would like you to carry out. To complete this task, first imagine you’re out in space, looking at the earth, and use the model to go to that view. Now use the controllers to figure out or show me why it’s hotter here in the summer than in the winter. Move the controllers to move the objects on the screen in a way that shows or explains what goes on in those different seasons. Remember to think aloud while doing this and pretend as if I’m not even here. If you fall silent for a while, I will remind you to keep thinking aloud. Are you ready? Then you can go ahead and start.

II e. Concurrent Think-Aloud: Hints for the Interviewer

The goal of the think-aloud method is to capture all the student’s mental processes while working through activities. Interviewers must strive to have the student speak aloud all of his or her thoughts during tasks. If a student is not talking, interviewers should use “continuers” to encourage them to talk. The goal is to get students to verbalize thoughts without putting words in students’ mouths. Care should be taken not to ask questions that lead students’ responses in particular directions or make them rush or change their approach. Interviewers should be as objective and unbiased as possible.

In general, if the student is silent for approximately 5 to 10 seconds, interviewers should use the following as a guide for encouraging the student to describe his or her thoughts, or to help the student elaborate his/her responses.

If the student is not verbalizing enough, interviewers should offer a verbal “nudge” to remind the students to keep talking, such as:

So what’s in your head?
What are you thinking?
What’s in your mind?
What are you thinking right now?
Go on….
Okay….keep going, keep talking

It is important to be responsive and sensitive to each student’s behavior. Students should be encouraged to verbalize as continuously as they can, but their reactions should be closely observed and pushing students should be avoided, especially if a student seems frustrated or uncomfortable. While it is desirable that students articulate as continuously as they can, sometimes students will simply be unable to say what is in their mind. Interviewers must be sensitive to nonverbal signals, if students cannot say any more than they have done.

In addition, during observations if it seems a student is hung up on something, interviewers should note when this occurred.

II f. Notes on Student Actions and Behaviors

As students are proceeding through each activity they will be video recorded. In addition, a screen-capture system and a video camera focusing on the tablet screen (in the case of the Virtual Lab studies) or digital log file capture (in the case of the Virtual Model studies) will record evidence of his or her interactions with the system or task. During the session the interviewer should take notes about any of the student’s expressions or behaviors that may reflect the status of his/her understanding, engagement, or use of the task or system. The following are examples of such behaviors for interviewers to note:

Does the student express signs of confusion, boredom, or excitement?
How does the student use the tools provided?
Does the student rapidly move through the activity or take his/her time?
Does the student spend a lot of time on a particular aspect?

Interviewers should also make a note of any places in the task that appear valuable for follow-up with some additional questions after the task is completed (see section IV).

III. Post-Think-Aloud Follow-Up Questions (Verbal Probing)

After completing the think-aloud process for a task, interviewers will follow up with a brief period of focused retrospective questioning. The post-task questions will comprise:

One standardized post-task question that all students will be asked following all tasks, which is designed to discover whether the student has prior knowledge of the content.

Up to three additional targeted questions. These will be generated by research staff prior to testing.

Standardized Question for All Tasks: Task-Specific Prior Knowledge

Have you studied anything related to this task in school, or have you learned about or come across these things in your own life? [If yes:] Tell me about what you have learned or studied or experienced that is related to what you did today.

Additional Questions: Task-Specific Issues

The purpose of the additional post-task questions is to capture more information on issues such as student actions during the activities, particular aspects of knowledge or skills targeted in the activities, and general reflections about the activities.

The following list shows some sample question frames that research staff will use to build questions. The specifics of each question will be generated via an informal task analysis in which key points for understanding student thinking will be identified.

Let’s think about the part where you [describe key action or aspect]. Can you tell me more about what you were thinking when you were doing that?

This [button or other tool] was on the screen to allow you to [describe action]. Tell me about how and when you used that, or if you didn’t really use it, why was that?

When you saw [information/graph/image] at this point, what were you thinking? What was in your mind?

If students’ answers are unclear or not very explicit, the interviewer should use prompts to encourage them to say more, for example:

Can you say a bit more about that?

What else were you thinking?

Anything else?

That’s interesting. Tell me more about that.

Optional: Interviewer-Generated Questions

In addition, if the interviewer noted some especially interesting behavior during the task, he or she can ask additional ad hoc questions about these specific instances, referring to notes made during the task. Interviewers should use their judgment about the need for and value of additional questions, based on the student’s behavior during the task, and based on time constraints.

Some examples of potential interviewer-generated ad hoc questions are:

I noticed on this part of the task that you paused for a while. Can you tell me more about why you paused and what you were thinking at this point?

I noticed on this part of the task that you did not use [describe tool or action]. Can you tell me more about why you didn’t use that?

I noticed that on this part of the task you spent some time looking at/going back-and-forth between [describe images, text, or actions]. Can you tell me more about what was going on at that point?

IV. Debriefing Questions and Thank You for Cognitive Interview

Thank student for his/her time. Provide a gift card.

Before we finish, I’d like to hear any other thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on these tasks?

Is there anything you would like to ask me about what we did today? [Answer student questions]

Thank you for helping us to study these processes and improve our system.

Part D. Small-Scale TryoutS

I. Introduction to Study Script

Text written in italics is to be read aloud by the interviewer.

Introduction Script:

Hello, my name is ______ and I work for ETS. It’s nice to meet you and thank you very much for helping us out today.

Let me begin by explaining why you’re here and what you’re going to be doing today. You are participating in a special study to try out new <insert name of task or system>. This is part of something called the National Assessment of Educational Progress, or NAEP for short, and is sponsored by the U.S. Department of Education. Today we’ll be asking you to try out a small group of new tasks that are being developed and studied for 8th graders.

If at any time you decide you do not want to go on, that is your choice and you may stop.

This tryout session is being recorded so researchers can review it later. Unless your parent/guardian gave permission, your information and responses may be used only for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C., § 9573).

Today we will aim to work through <say number of tasks>.

Do you have any questions?

Interviewer: Ask the student if he/she has any questions. If the student is no longer interested in participating, thank the student for his/her time and end the tryout session.

Ok, now I’m going to give you [number] questions or tasks to complete. I want you to work through them one at a time. Try to finish each task. You aren’t being graded on your work today, so it’s fine if you don’t know the answer or don’t know how to go about any of the activities we are asking you to do. But try to do your best, and try to complete all of the activities, or as much as you can. I won’t be interrupting, I’ll just be here observing, so you can work through the tasks at your own pace.

You will have [indicate correct number of minutes] minutes to do this. You will not be graded on what you do during the task, but please try to work through this as if it was a real test.

Okay, you can go ahead and begin with the first task.

II. EXAMPLES OF TASKS OR ACTIVITIES IN THE TRYOUT PHASE

By the Tryout phase, tasks or activities will be self-contained and students will be able to work through them at their normal pace, without interruption and without needing support from the facilitator. As far as possible, Tryouts should represent naturalistic test-taking conditions.

The kinds of activities that students will be asked to complete will be similar to those described for earlier phases. However, in the Tryout phase, the facilitator will not need to deliver the activities or tasks using verbal instructions, since they will be presented to the students via the system (the Virtual Science Lab or the Virtual Solar System Model). The exact look and feel of the presentation of these will be determined during the later development phases of each project. However, all of the tasks or activities will be presented in a way that is sufficiently self-contained that students can work through them at their own pace, without additional input from the facilitator. Examples of tasks or activities that students will complete in the Tryout phase are shown below.

Virtual Models Sample Tryout Task:

Students will use the virtual solar system in order to:

Show why it is warmer in the summer than in the winter, from earth view and space view.

Virtual Science Lab Sample Tryout task

Students will use the lab tools in order to:

Run an experiment to test the effect of salt concentration on floatation of objects in water.

III. POST-TRYOUT TASKS FOR VIRTUAL MODEL

Following the tasks, students in the Virtual Models sessions will be given three short post-task measures to complete. Refer to sections III Tasks 1-3 for the entire versions of the instruments.

– Task I: Astronomy and Space Science Concept Inventory (Sadler, 2011):

Facilitators should administer these one at a time, using the following script:

Ok, now I’m going to give you a few other things that I’d like you to complete. This first sheet has some questions about space science. Work through these questions at your own pace, and circle the answer you think is correct for each one. Try to answer them all, and if you’re not sure, just circle the answer that you think is correct. Ready? Ok, you can begin, and let me know when you are done.

– Task II: Spatial Imagery Questionnaire (Blajenkova et al., 2006)

This second task has some questions asking you to tell us about how you usually think and learn. For each question, circle the number that represents your own thinking or learning style, from 1, which means totally disagree to 5, which means totally agree. There are no right or wrong answers here, you should answer the way that reflects how you usually think. If there is a question about something you’ve never done before, or if there is something that you really can’t answer, just leave it blank. Ready? Ok, you can begin, and let me know when you are done.

This self-report questionnaire is about people’s preferences to use mental imagery (seeing in your mind’s eye) or verbal thinking (words in your head). Mental imagery is the ability to imagine things in the mind (especially visible objects). Mental images are not what we see directly, they are the pictures that we often have in mind while we think about things.

There are no right or wrong answers for this questionnaire so please just be as honest about yourself as possible.

In order to complete this questionnaire, please read the following statements and rate each of them on a 5-point scale. Use rating “5” if you totally agree with the statement that describes you and use rating “1” if you totally disagree with the statement. Use numbers in between 1 and 5 to show in-between degrees of how the statements describe you.

It is very important that you answer all the items but if you can’t answer a question because you have never done that thing before, then please leave it blank.

Totally disagree Totally agree

1.	I am very good at geometry.	1	2	3	4	5
2.	I have difficulty expressing myself in writing.	1	2	3	4	5
3.	If I were asked to choose between engineering professions and visual arts, I would prefer engineering.	1	2	3	4	5
4.	Architecture interests me more than painting.	1	2	3	4	5
5.	The images in my head are very colorful and bright.	1	2	3	4	5
6.	I prefer diagrams and sketches when reading a textbook instead of colorful and detailed pictures or illustrations.	1	2	3	4	5
7.	I tell jokes and stories better than most people.	1	2	3	4	5
8.	Essay writing is difficult for me and I do not enjoy doing it at all.	1	2	3	4	5
9.	My mental images are more like non-detailed outlines of things and events rather than like detailed pictures.	1	2	3	4	5
10.	When reading a book, I usually form a clear and detailed mental picture of a scene or room that has been described.	1	2	3	4	5
11.	If I had to choose between being an engineer, a writer, or an artist, I would most likely choose to be a writer.	1	2	3	4	5
12.	I have a photographic memory.	1	2	3	4	5
13.	I can easily imagine and mentally rotate three-dimensional geometric shapes.	1	2	3	4	5
14.	My verbal skills are excellent.	1	2	3	4	5

15.	I normally don’t have many vivid mental images or pictures in my head; I only use mental images when I am doing things like solving math problems.	1	2	3	4	5
16.	When entering a familiar store to get a specific item, I can easily picture the exact location of the item, the shelf it stands on, how it is arranged and the surrounding articles.	1	2	3	4	5
17.	Putting together furniture kits or models (e.g., a TV stand or a chair) is much easier for me when I have detailed verbal instructions than when I only have a diagram or picture.	1	2	3	4	5
18.	My images are very vivid and photographic.	1	2	3	4	5
19.	When explaining something, I would rather give an explanation in words than make drawings or sketches.	1	2	3	4	5
20.	I can easily sketch a plan or layout for a building that I am familiar with.	1	2	3	4	5
21.	I am a good Tetris player.	1	2	3	4	5
22.	If I were asked to choose between studying architecture and visual arts, I would choose visual arts.	1	2	3	4	5
23.	If someone were to give me two-digit numbers to add (e.g., 43 and 32) I would simply do the adding without seeing the numbers in my mind’s eye.	1	2	3	4	5
24.	My mental images of different objects really look like the size, shape and color of actual objects that I have seen.	1	2	3	4	5
25.	I usually do not try to create a mental image or sketch a diagram when I am reading a textbook.	1	2	3	4	5
26.	When I imagine the face of a friend, I have a perfectly clear and bright image.	1	2	3	4	5
27.	I have excellent skills for creating technical drawings and diagrams.	1	2	3	4	5
28.	When remembering a scene, I use verbal descriptions rather than mental pictures.	1	2	3	4	5
29.	I can easily remember a great deal of visual details that someone else might never notice. For example, I would just automatically take some things in, like what color is a shirt someone wears or what color are his/her shoes…	1	2	3	4	5
30.	I find geometry easier than art.	1	2	3	4	5
31.	I enjoy pictures with bright colors and unusual shapes like the ones in modern art paintings.	1	2	3	4	5
32.	Sometimes my images are so vivid and strong that it is difficult to ignore them.	1	2	3	4	5
33.	When I am thinking about an abstract concept (e.g., “buildings”) I imagine an abstract imaginary building in my mind or its plan or outline, rather than a specific real building.	1	2	3	4	5
34.	I have better than average fluency in using words.	1	2	3	4	5
35.	I would rather have a verbal description of an object or person than a picture.	1	2	3	4	5
36.	I am always aware of grammar in sentences.	1	2	3	4	5
37.	My images are more like diagrams or sketches than like real colorful pictures or photographs.	1	2	3	4	5
38.	I can close my eyes and easily picture a scene that I have experienced.	1	2	3	4	5
39.	I enjoy being able to rephrase my thoughts in many ways in both writing and speaking.	1	2	3	4	5
40.	I remember everything visually. I can recount what people wore to a party and I can talk about the way they sat and the way they looked probably in more detail than I could remember what they said.	1	2	3	4	5
41.	I sometimes have a problem expressing exactly what I want to say.	1	2	3	4	5
42.	I find it difficult to imagine how a three-dimensional geometric figure would exactly look if it was rotated.	1	2	3	4	5
43.	My visual images are in my head all the time. They are just right there.	1	2	3	4	5
44.	My graphic abilities would make a career in architecture fairly easy for me.	1	2	3	4	5
45.	When I hear a radio announcer or a DJ I’ve never actually seen, I usually find myself picturing what he or she might look like.	1	2	3	4	5

– Task III: Paper Folding Test (Ekstrom et al., 1976)

This last task asks you to imagine a square piece of paper that has been folded – the dotted line shows where the outline of the paper was and in this practice item it has been folded in half from the bottom, upward. Then some holes have been punched all the way through all of the layers. When the paper is unfolded, you have to decide where the holes will be. The answer here is C, so for this one you would circle C. Does that make sense? OK, before you start, look over this example and read the instructions, to make sure you understand what you have to do.

OK, do you understand what you have to do, or would you like me to explain it one more time? <Repeat explanation with sample item if needed> OK, good, so now you are going to do Part 1, and I’ll give you 3 minutes to do an many as you can. Don’t worry if you don’t finish all of the items, it is timed so that you might not finish. I will give you 3 minutes and I’ll tell you when the time is up. Don’t go on to the last part until I tell you to. Ready? OK, you can start Part 1 now.

OK, time’s up. Ready to do Part 2? Same as before, do as many as you can and I will tell you when the time is up. Ready? Ok, start Part 2 now.

OK, time’s up. Thanks very much for doing those extra things for us, it’s an important part of the study.

IV. Debriefing QUESTIONS

Before we finish, I’d like to hear [any/other] thoughts you have about what you’ve been doing.

Is there anything else you would like to tell me about working on the tasks?

Did you have any problems understanding [other/any of] what you did?

Did you have any problems understanding what you were supposed to do to answer or finish the tasks?

Is there anything that you think could make things clearer?

Thank student for his/her time. Provide a gift card.

References

Blajenkova, O., Kozhevnikov, M., & Motes, M. A. (2006). Object‐spatial imagery: a new self‐report imagery questionnaire. Applied Cognitive Psychology, 20(2), 239-263.

Ericsson, K. A. (2006). Protocol analysis and expert thought: Concurrent verbalizations of thinking during experts’ performance on representative tasks. The Cambridge handbook of expertise and expert performance, 223-242.

Ericsson, K. A., & Simon, H. A. (1984). Verbal reports as data. MIT Press.

Ericsson, K. A., & Simon, H. A. (1998). How to study thinking in everyday life: Contrasting think-aloud protocols with descriptions and explanations of thinking. Mind, Culture, and Activity, 5(3), 178-186.

Ekstrom, R. B., French, J. W., Harman, H. H., & Dermen, D. (1976). Kit of factor-referenced cognitive tests. Princeton, NJ: Educational Testing Service.

Newell, A., & Simon, H. A. (1972). Human problem solving (Vol. 104, No. 9). Englewood Cliffs, NJ: Prentice-Hall.

Sadler, P. M. (2011). The Astronomy and Space Science Concept Inventory: Assessment Instruments Aligned with the K-12 National Science Standards.Bulletin of the American Astronomical Society. Vol. 43.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Science Volume II Cog Lab Protocols
Author	BlairJ
File Modified	0000-00-00
File Created	2021-01-28