NAEP Reading and Math Technology Based Assessment Survey Focus Groups 2014

National Center for Education Statistics

National Assessment of Educational Progress

Volume II

Protocols

Focus Groups for Transitioning Reading and Math Survey Questionnaires to Technology Based Assessments

OMB# 1850-0803 v.96

February 7, 2014

Contents

1) Paperwork Burden Statement 2

2) Focus Group Protocol 3

Part I. Interviewer Welcome Scripts and Assent/Consent 3

Part II. Focus Group Questions 8

Attachment A 13

Attachment B 14

1. Paperwork Burden Statement

The Paperwork Reduction Act and the NCES confidentiality statement are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to average 90 minutes including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, D.C. 20202-4537. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: NAEP/NCES, U.S. Department of Education, 1990 K Street, N.W., 9^th floor, Washington, DC 20006.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

Your feedback may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

OMB No. 1850-0803 Approval Expires 9/30/2016

2. Focus Group Protocol

Part I. Interviewer Welcome Scripts and Assent/Consent

Student Participant Welcome Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the focus groups in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored by the interviewer.

[Welcome participants and thank them for participating in this focus group.]

[Introduce yourself and note taker.]

[If observers are present, acknowledge their presence and state that they will not be participating.]

Your ideas are extremely important to us, and we want you to feel free to tell us exactly what you think. We hope your thoughts will create discussion that will help us understand how students use technology in your everyday life, and the kind of things you learn about mathematics and reading. We want you to feel comfortable answering our questions and sharing in a discussion. So, I want to explain what focus groups are and then I will describe the project that brought us all here for your comments and opinions.

You may not be familiar with focus groups. A focus group is a group discussion about your opinions. There are no right or wrong answers and there is no need for us to agree with each other. If anybody says something you have a question about or want to comment on, please do so. Don't feel you have to wait for me to call on you. My job is to focus the discussion of the issues. I encourage all of you to participate in the discussion--everyone's ideas and opinions are important.

Some things are common to almost all focus groups and they apply here today, too.

1. People in focus groups usually have something in common. In our case, you were selected because you’re [4^th] / [8^th] grade students.

2. [Identify note taker] is taking notes and the session is being tape-recorded. The microphone is there. The recording is an additional note-taking system so that we are sure to collect everyone’s opinions accurately.

3. Even though we are taking notes and recording our discussion, your identity will be kept anonymous; this means that others will know what was said but not who said it. Comments you make during this discussion may be used in reports, but you will not be identified by name. All study materials will be kept safely in a locked office, and once we are done with our work, they will be destroyed.

4. Don’t be afraid to say what you think and be willing to explain why you think as you do. The group does not need to agree with each other. And, you don’t have to answer a question if you don’t want to.

5. In a focus group, it is important to have only one person speak at a time. We want to be able to hear each person's comments. So please speak one at a time. Listen to what others say, and be respectful of everyone’s contributions.

6. We will end around [End Time].

7. Please remember participating in this focus group is a choice, and that you do not have to answer any questions that you don’t want to answer.

8. Finally, if at any point you do not understand something that I have said, please stop me and I will clarify.

So, by now I am sure someone has talked with you about some of the work that NCES does. We create and administer assessments, which are more commonly referred to as tests. We are currently working on some survey questions that will ask students like you about their experiences with and feelings about using technology, and learning about mathematics and reading. It is really helpful for us to know more about the kinds of activities you are doing and technologies you are using both in and outside of school.

So, that is why we are all here. Does anyone have any questions? [Answer questions.] So, let’s get started.

Teacher Participant Welcome Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the focus groups in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored by the interviewer.

[Welcome participants and thank them for participating in this focus group.]

[Introduce yourself and note taker.]

[If observers are present, acknowledge their presence and state that they will not be participating.]

[Briefly describe how to use the technology associated with the WebEx]

Your ideas are extremely important to us, and we want you to feel free to tell us exactly what you think. We hope your thoughts will generate a lively and informative discussion that will help us understand the role of technology in your classrooms, teacher resources available to you, and student learning in mathematics and reading. We want you to feel comfortable answering our questions and sharing in a discussion. So, I want to explain what focus groups are and then I will describe the project that brought us all here for your comments and opinions.

You may not be familiar with focus groups. A focus group is a group discussion about your opinions. There are no right or wrong answers and there is no need for us to agree with each other or reach consensus. If anybody says something you have a question about or want to comment on, please do so. Don't feel you have to wait for me to call on you. My job is to focus the discussion of the issues. I encourage all of you to participate in the discussion--everyone's ideas and opinions are important.

Some things are common to almost all focus groups and they apply here today, too.

1. People in focus groups usually have something in common. In our case, you were selected because you are teachers with experience in [4^th /8^th] grade mathematics or reading content and/or curriculum.

2. [Identify note taker] is taking notes and the session is being recorded. The recording is an additional note-taking system so that we are sure to collect everyone’s opinions accurately.

3. Even though we are taking notes and recording our discussion, your identity will be kept anonymous. Comments from this meeting may be quoted directly or in summary in reports, but you will not be identified by name. All study materials will be kept secure in a locked office, and upon completion of our work, they will be destroyed.

4. Don’t be afraid to say what you think and be willing to explain why you think as you do. The group does not need to agree or have consensus. And, you don’t have to answer a question if you don’t want to.

5. In a focus group, it is important to have only one person speak at a time. We want to be able to hear each person's comments. So please speak one at a time. Listen to what others say, and be respectful of everyone’s contributions.

6. We will end promptly at [End Time].

7. You are doing this voluntarily and are not required to answer any questions.

8. Finally, if at any point you do not understand something that I have said, please stop me and I will clarify.

[Thank participants again and have them introduce themselves (first name only), where they teach, and at what grade levels.]

You may be familiar with some of the work that NCES does. We create and administer assessments, which are more commonly referred to as tests. In addition to tests, we do research to improve quality and fairness in education, and we are working on developing new ways to better assess and understand student knowledge. As part of many large national assessments, survey questions are information collected from students, parents, teachers, and/or schools to understand what may be contributing factors to student success.

We are currently transitioning from paper-and-pencil assessments to technology-based assessments. As part of this, we would like to have a better understanding of the role of technology in your classrooms, teacher resources available to you, and student learning in mathematics and reading. The goal of this discussion is for you to provide us with observations and opinions that can assist us in developing the survey questions.

So, that is why we are all here. Does anyone have any questions? [Answer questions]. So, let’s get started.

School Administrator Welcome Script

Text written in italics is to be read aloud by the staff facilitator.

NOTE: The facilitator should not read the script word for word, but should be familiar enough with its contents to conduct the focus groups in a natural and conversational manner, paraphrasing, or giving further explanation as appropriate. For example, facilitators should be attentive to the language comprehension of younger students when delivering scripts. Text placed in brackets is generic text that should be tailored by the interviewer.

[Welcome participants and thank them for participating in this focus group.]

[Introduce yourself and note taker.]

[If observers are present, acknowledge their presence and state that they will not be participating.]

[Briefly describe how to use the technology associated with the WebEx]

Your ideas are extremely important to us, and we want you to feel free to tell us exactly what you think. We hope your thoughts will generate a lively and informative discussion that will help us understand the role of technology in your schools, resources available to teachers and students, and how you monitor student academic progress. We want you to feel comfortable answering our questions and sharing in a discussion. So, I want to explain what focus groups are and then I will describe the project that brought us all here for your comments and opinions.

You may not be familiar with focus groups. A focus group is a group discussion about your opinions. There are no right or wrong answers and there is no need for us to agree with each other or reach consensus. If anybody says something you have a question about or want to comment on, please do so. Don't feel you have to wait for me to call on you. My job is to focus the discussion of the issues. I encourage all of you to participate in the discussion--everyone's ideas and opinions are important.

Some things are common to almost all focus groups and they apply here today, too.

1. People in focus groups usually have something in common. In our case, you were selected because you are school administrators to 4^th and 8^th grade students.

2. [Identify note taker] is taking notes and the session is being tape-recorded. The recording is an additional note-taking system so that we are sure to collect everyone’s opinions accurately.

3. Even though we are taking notes and recording our discussion, your identity will be kept anonymous. Comments from this meeting may be quoted directly or in summary in reports, but you will not be identified by name. All study materials will be kept secure in a locked office, and upon completion of our work, they will be destroyed.

4. Don’t be afraid to say what you think and be willing to explain why you think as you do. The group does not need to agree or have consensus. And, you don’t have to answer a question if you don’t want to.

5. In a focus group, it is important to have only one person speak at a time. We want to be able to hear each person's comments. So please speak one at a time. Listen to what others say, and be respectful of everyone’s contributions.

6. We will end promptly at [End Time].

7. You are doing this voluntarily and are not required to answer any questions.

8. Finally, if at any point you do not understand something that I have said, please stop me and I will clarify.

[Thank participants again and have them introduce themselves, where they work, and at what grade levels.]

So, by now I am sure someone has talked with you about some of the work that NCES does. We create and administer assessments, which are more commonly referred to as tests. In addition to tests, we do research to improve quality and fairness in education, and we are working on developing new ways to better assess and understand student knowledge. As part of many large national assessments, survey questions are information collected from students, parents, teachers, and/or schools to understand what may be contributing factors to student success.

We are currently transitioning from paper-and-pencil assessments to technology-based assessments. As part of this, we would like to have a better understanding of the role of technology in your schools, resources available to teachers and students, and how you monitor student academic progress. The goal of this discussion is for you to provide us with observations and opinions that can assist us in developing the survey questions.

So, that is why we are all here. Does anyone have any questions? [Field questions]. So, let’s get started.

[Conclusion. Thank participants. Ask participants if they have any additional questions or comments. Thank participants again and end meeting.]

Part II. Focus Group Questions

The purpose of the focus groups is to inform the development of the Mathematics and Reading student, teacher, and school administrator questionnaires prior to cognitive interviews and pilot testing. Below are the probes that students, teachers, and school administrators will be asked during the 90-minute focus group session. Please note that some probes may not be addressed if time does not permit.

Student

General

1. When someone uses the word “computer” what do you think they are talking about?

2. When someone uses the phrase “digital technology” what do you think they are talking about?

Mathematics¹

1. When someone uses the word “mathematics” what do you think they are talking about?

   1. Do you think solving mathematics problems “in your head” is the same as solving mathematics problems with a calculator?

      1. How are they the same?

      2. How are they different?

2. In general, what kinds of digital technologies (such as computers/laptops, tablets, or smart phones) do you use?

1. What types of digital technologies, if any, does your school provide to students?

2. In what ways do you use computers or other digital technologies for mathematics assignments or tests in mathematics class? How did you learn to use these technologies?

3. Do you work with tablets or laptops in your mathematics class? Do you have to bring your own devices or does your teacher give them to you? Do you work alone on one device or together with other students?

4. In what ways do you use computers or other digital technologies outside of the classroom to make calculations or for other mathematics tasks?

5. In what ways do you use computers or other digital technologies for calculations not directly related to schoolwork?

3. Do you enjoy your teacher’s mathematics lesson more when technology is used in part of the lesson?

   1. What are some reasons why you enjoy the lesson more?

   2. What are some reasons why you might enjoy the lesson less?

4. [Present students with select 2015 Operational Mathematics Survey Questionnaire items, see Volume II, Appendices A and B]. Please take a few moments to look over these questions.

1. Which of these questions can you relate to?

2. Do you think there are other questions or activities that should be included?

3. When the word “calculator” is used in these questions what do you think the question is talking about?

   1. When you need to use a calculator, do you only use a traditional calculator or do you also use your cell phone, tablet, or laptop?

Reading

1. When someone says the word “reading” what do you think they are talking about?

   1. Do you think reading printed text (e.g., a hard copy book) is the same as reading on a computer or other digital technologies?

      1. How are they the same?

      2. How are they different?

2. In general, what kinds of digital technologies (such as, computers/laptops, tablets or smart phones) do you use?

1. What types of digital technologies, if any, does your school provide to students?

2. In what ways do you use computers or other digital technologies for reading assignments or tests in reading class? How did you learn to use these technologies?

3. What types of technology are you most likely to use to read in the classroom (e.g., print books, laptops, tablets, printed packets/handouts, etc.)?

4. Do you work with tablets or laptops in your reading class? Do you have to bring your own devices or does your teacher give them to you? Do you work alone on one device or together with other students?

5. What types of technology are you most likely to use to read outside of school (e.g., at home, in the car or train, at the mall)?

6. Which of these technologies do you use for reading homework?

7. In what ways do you use these technologies for reading not related to schoolwork?

3. Do you enjoy your teacher’s reading lesson more when technology is used in part of the lesson?

   1. What are some reasons why you enjoy the lesson more?

   2. What are some reasons why you might enjoy the lesson less?

4. How often do you use digital devices such as e-readers or tablets for reading?

5. Where in your daily activities do you read?

   1. What activities do you consider reading, for example, does reading on a cell phone count as reading to you?

Teacher

General

1. When someone uses the word “computer” what do you think they are talking about?

2. When someone uses the phrase “digital technology” what do you think they are talking about?

Mathematics

1. When someone uses the word “mathematics” what do you think they are talking about?

   1. Do you think solving mathematics problems “in your head” is the same as solving mathematics problems with a calculator?

      1. How are they the same?

      2. How are they different?

2. What kinds of technologies, if any, does the school provide to students?

   1. Are these technologies only available to students in school or can students take them home?

3. What types of technology products are available to help you prepare for and conduct your mathematics lessons?

4. What types of technology, if any, do you incorporate into your mathematics lessons?

   1. What are some reasons why you use technology in your mathematics lessons? How do you incorporate technology in your mathematics lessons? Do students react differently to your mathematics lessons when technology is incorporated, as opposed to mathematics lessons where technology is not incorporated?

   2. What are some reasons why you do not use technology in your mathematics lessons?

5. What training, if any, have you received regarding integrating technology into your mathematics classroom?

6. What challenges, if any, have you encountered when incorporating technology into your mathematics classroom?

7. What kinds of technologies are students allowed to use for mathematics classroom activities (e.g., smart phones, laptop, and tablet)?

   1. Are students able to use these technologies for all or only select classroom activities?

   2. Overall, do you think that your students have access to various technologies in and outside of school (e.g., smart phones, laptop, and tablet)?

      1. What are some ways that you accommodate students who cannot complete homework because they do not have access to technology at home?

8. In the classroom, how often are students asked to solve mathematics problems “in their head”?

9. Is a student’s ability to solve mathematics problems “in their head” more important for some mathematics tasks than others?

1. If yes, which tasks?

10. In the classroom, how often are students asked to use a calculator to solve mathematics problems?

11. Is a student’s ability to use a calculator more important for some mathematics tasks than others?

    1. If yes, which tasks?

12. [Present teachers with select 2015 Operational Student Mathematics items focusing on calculators and computer use, see Volume II, Appendices A and B]. Please take a few moments to look over these questions.

    1. Which of these questions regarding calculator and computer use are relevant to your students, and which are not?

    2. Are there other digital technologies that are not captured in these questions that students are expected to use to complete their mathematics schoolwork?

Reading

1. When someone says the word “reading” what do you think they are talking about?

   1. Do you think reading printed text (e.g., a hard copy book) is the same as reading on a computer or other digital technologies?

      1. How are they the same?

      2. How are they different?

2. What kinds of technologies, if any, does the school provide to students?

   1. Are these technologies only available to students in school or can children take them home?

3. What types of technology products are available to help you prepare for and conduct your reading lessons?

4. What types of technology, if any, do you incorporate into your reading lessons?

   1. What are some reasons why you use technology in your reading lessons? How do you incorporate technology into your reading lessons? Do students react differently to your reading lessons when technology is incorporated, as opposed to reading lessons where technology is not incorporated?

   2. What are some reasons why you do not use technology in your reading lessons?

5. What training, if any, have you received regarding integrating technology into your reading classroom?

6. What challenges, if any, have you encountered when incorporating technology into your reading classroom?

7. What kinds of technologies are students allowed to use for reading classroom activities (e.g., smart phones, laptop, and tablet)?

   1. Are students able to use these technologies for all or only select classroom activities?

   2. For what types of reading schoolwork are students most likely to be encouraged to use technology and or multimedia?

   3. What technology or multimedia, if any, are students expected to use to complete their reading homework?

   4. Overall, do you think that your students have access to various technologies in and outside of school (e.g., smart phones, laptop, and tablet)?

      1. What are some ways that you accommodate students who cannot complete homework because they do not have access to technology at home?

School Administrator

1. When someone uses the word “mathematics” what do you think they are talking about?

   1. Do you think solving mathematics problems “in your head” is the same as solving mathematics problems with a calculator?

      1. How are they the same?

      2. How are they different?

2. When someone uses the word “reading” what do you think they are talking about?

   1. Do you think reading printed text (e.g., a hard copy book) is the same as reading on a computer or other digital technologies?

      1. How are they the same?

      2. How are they different?

3. When someone uses the word “computer” what do you think they are talking about?

4. When someone uses the phrase “digital technology” what do you think they are talking about?

5. What kinds of technologies, if any, does the school provide to students?

   1. Are these technologies only available to students in school or can children take them home?

   2. Who funds or pays for these technologies?

   3. Are students able to use a variety of technologies during school, for example would students be able to use smart phones for classroom activities?

6. What types of technology resources are available to students both in and outside of school?

   1. Overall, do you think that your students have access to various devices in and outside of school (e.g., smart phones, laptop, and tablet)?

7. What types of technology resources are available to teachers both in and outside of the school?

8. What challenges, if any, have teachers in your school encountered when integrating technology into their mathematics classroom?

9. What challenges, if any, have teachers in your school encountered when integrating technology into their reading classroom?

10. What challenges, if any, would you anticipate if students no longer took paper-and- pencil tests, and instead took technology-based assessments?

Attachment A

Selected 2015 Operational Mathematics Survey Questionnaire Items for Grade 4 Students

Attachment B

Selected 2015 Operational Mathematics Survey Questionnaire Items for Grade 8 Students

_____________________________________________________________________________________

1 Please note for fourth-grade students the word “math” will be used instead of mathematics.