Progress in International Reading Literacy Study (PIRLS) 2021 Field Test Pretest

Progress in International Reading Literacy Study (PIRLS) 2021 Field Test Pretest

OMB# 1850-0803 v.257

Volume 2

Submitted by:

National Center for Education Statistics (NCES)

Institute of Education Sciences (IES)

U.S. Department of Education

Washington, DC

October 2019

PAPERWORK BURDEN STATEMENT

The Paperwork Reduction Act and the NCES authorization and confidentiality statements are indicated below. Appropriate sections of this information are included in the consent forms and letters. The statements will be included in the materials used in the study.

Paperwork Burden Statement, OMB Information

The National Center for Education Statistics (NCES) is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543), and to collect students’ education records from educational agencies or institutions for the purpose of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35).

All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0803. The time required to complete this information collection is estimated to be an average of 150 minutes, including the time to review instructions and participate in the information collection.

If you have any comments concerning the accuracy of the time estimate, suggestions for improving this collection, or any other concerns, please write to: Progress in International Reading Literacy Study (PIRLS), National Center for Education Statistics, Potomac Center Plaza, 550 12th St., SW, 4th floor, Washington, DC 20202.

This is a project of the National Center for Education Statistics (NCES), part of the Institute of Education Sciences, within the U.S. Department of Education.

OMB No. 1850-0803 Approval Expires xx/xx/xxxx

INTRODUCTION TO THE ASSESSMENT

For the pretesting sessions described in Volume 1, the administrators will provide the following additional instructions to the students:

For this part of the study, we would like you to take your time exploring the system and interacting with the questions and system tools in different ways. While we would like you to provide answers to questions, we would like you to focus mainly on using the system tools and responding to the questions. Any response is acceptable for the questions as we really want you to experiment with how you can respond to questions and move from question to question. Have a little fun with things, and explore the system!

Please take the assessment in this way until your time is up.

STUDENT QUESTIONNAIRE

At the end of the cognitive assessments, students will take a brief questionnaire about themselves and their educational experience. This questionnaire will be administered on the tablets.

Question Number	Question
About you
1a	Which of these describes you? Click one circle only. 1. Girl 2. Boy 3. Other
1b	Are you Hispanic or Latino? Click one circle only. 1. Yes, I am Hispanic or Latino 2. No, I am not Hispanic or Latino
1c	Which of the following best describes you? Click one or more circles. 1. White 2. Black or African American 3. Asian 4. American Indian or Alaska Native 5. Native Hawaiian or other Pacific Islander
2	When were you born? Click the circles next to the month and year you were born.
2a	Month 1. January 2. February 3. March 4. April 5. May 6. June 7. July 8. August 9. September 10. October 11. November 12. December
2b	Year 1. 2008 2. 2009 3. 2010 4. 2011 5. 2012 6. 2013 7. 2014 8. Other
3a	How often do you speak English at home? Click one circle only. 1. I always speak English at home. 2. I almost always speak English at home. 3. I sometimes speak English and sometimes speak another language at home. 4. I never speak English at home. If Always, please go to question 4 If Almost always, Sometimes, or Never, please go to question 3B
3b	What language do you speak at home (other than English)? Click one circle only. 1. Spanish 2. Other (Please specify) __________
4	About how many books are there in your home? (Do not count magazines, newspapers, or your school books.) Click one circle only. 1. None or very few (0–10 books) This shows 10 books 2. Enough to fill one shelf (11–25 books) This shows 25 books 3. Enough to fill one bookcase (26–100 books) This shows 100 books 4. Enough to fill two bookcases (101–200 books) This shows 200 books 5. Enough to fill three or more bookcases (more than 200) This shows more than 200 books
5a	About how often are you absent from school? Click one circle only. 1. Once a week 2. Once every two weeks 3. Once a month 4. Once every two months 5. Never or almost never
5b	How many days were you absent from school in the last month? Click one circle only. 1. None 2. 1 or 2 days 3. 3 or 4 days 4. 5 to 10 days 5. More than 10 days
6	Have you ever repeated a grade in elementary school? Click one circle only. 1. Yes 2. No
7	How often do you feel this way when you arrive at school? Click one circle for each row. 1. Every day 2. Almost every day 3. Sometimes 4. Never
7a	I feel tired.
7b	I feel hungry.
8	How much time do you spend using a computer or tablet to do these activities for your schoolwork on a normal school day? Click one circle for each row. 1. No time 2. 30 minutes or less 3. More than 30 minutes
8a	Finding and reading information
8b	Preparing reports and presentations
9	How much do you agree with these statements about using computers or tablets? Click one circle for each row. 1. Agree a lot 2. Agree a little 3. Disagree a little 4. Disagree a lot
9a	I am good at using a computer or tablet.
9b	I am good at typing.
9c	It is easy for me to find information on the Internet.
9d	I know how to create text documents using a computer or tablet.
9e	I know how to create presentations.
9f	I know how to use keywords to search for information on the internet.
9g	I know how to use a link to go to a website.
9h	I know how to find things on a website.
9i	I can move from one website to another.
Your school
10	What do you think about your school? Click to show how much you agree with these statements. Click one circle for each row. 1. Agree a lot 2. Agree a little 3. Disagree a little 4. Disagree a lot
10a	I like being in school.
10b	I feel safe when I am at school.
10c	I feel like I belong at this school.
10d	Teachers at my school are fair to me.
10e	I am proud to go to this school.
10f	I have friends at this school.
11	During this year, how often have other students from your school done any of the following things to you, including through texting or the Internet? Click one circle for each row. 1. At least once a week 2. Once or twice a month 3. A few times a year 4. Never
11a	Made fun of me or called me names
11b	Left me out of their games or activities
11c	Spread lies about me
11d	Stole something from me
11e	Damaged something of mine on purpose
11f	Hit or hurt me (e.g., shoving, hitting, kicking)
11g	Made me do things I didn’t want to do
11h	Sent me nasty or hurtful messages online
11i	Shared nasty or hurtful information about me online
11j	Threatened me
Reading in school
12	Think about the reading you do for school. How much do you agree with these statements about your reading lessons? Click one circle for each row. 1. Agree a lot 2. Agree a little 3. Disagree a little 4. Disagree a lot
12a	I like what I read about in school.
12b	My teacher gives me interesting things to read.
12c	I know what my teacher expects me to do.
12d	My teacher is easy to understand.
12e	I am interested in what my teacher says.
12f	My teacher encourages me to say what I think about what I have read.
12g	My teacher lets me show what I have learned.
12h	My teacher does a variety of things to help us learn.
12i	My teacher tells me how to do better when I make a mistake.
13	How often do these things happen during your reading lessons? Click one circle for each row. 1. Every or almost every lesson 2. About half the lessons 3. Some lessons 4. Never
13a	Students don’t listen to what the teacher says.
13b	There is too much noise for students to work well.
13c	My teacher has to wait a long time for students to be quiet.
13d	Students interrupt the teacher.
13e	My teacher has to keep telling us to follow the classroom rules.
14	In school, how often do these things happen? Click one circle for each row. 1. Every day or almost every day 2. Once or twice a week 3. Once or twice a month 4. Never or almost never
14a	I read silently on my own.
14b	I read things that I choose myself.
14c	My teacher asks us in class to talk about what we have read.
Using the library
15	How often do you borrow books (including ebooks) from your school or local library? Click one circle only. 1. At least once a week 2. Once or twice a month 3. A few times a year 4. Never or almost never
Reading outside of school
16	How much time do you spend reading outside of school on a normal school day? Click one circle only. 1. Less than 30 minutes 2. 30 minutes up to 1 hour 3. From 1 hour up to 2 hours 4. 2 hours or more
17	How often do you do these things outside of school? Click one circle for each row. 1. Every day or almost every day 2. Once or twice a week 3. Once or twice a month 4. Never or almost never
17a	I read for fun.
17b	I read to find out about things I want to learn.
16	How much time do you spend reading outside of school on a normal school day? Click one circle only. 1. Less than 30 minutes 2. 30 minutes up to 1 hour 3. From 1 hour up to 2 hours 4. 2 hours or more
17	How often do you do these things outside of school? Click one circle for each row. 1. Every day or almost every day 2. Once or twice a week 3. Once or twice a month 4. Never or almost never
17a	I read for fun.
17b	I read to find out about things I want to learn.
What you think about reading
18	What do you think about reading? Click to show how much you agree with each of these statements. Click one circle for each row. 1. Agree a lot 2. Agree a little 3. Disagree a little 4. Disagree a lot
18a	I like talking about what I read with other people.
18b	I would be happy if someone gave me a book as a present.
18c	I think reading is boring.
18d	I would like to have more time for reading.
18e	I enjoy reading.
18f	I learn a lot from reading.
18g	I like to read things that make me think.
18h	I like it when a book helps me imagine other worlds.
19	How well do you read? Click to show how much you agree with each of these statements. Click one circle for each row. 1. Agree a lot 2. Agree a little 3. Disagree a little 4. Disagree a lot
19a	I usually do well in reading.
19b	Reading is easy for me.
19c	I have trouble reading stories with difficult words.
19d	Reading is harder for me than for many of my classmates.
19e	Reading is harder for me than any other subject.
19f	I am just not good at reading.
20	How hard was this test compared to most other tests you have taken this year in school? Click one circle only. 1. Easier than other tests 2. About as hard as other tests 3. Harder than other tests 4. Much harder than other tests
21	How hard did you try on this test compared to how hard you tried on most other tests you have taken this year in school? Click one circle only. 1. Not as hard as on other tests 2. About as hard as on other tests 3. Harder than on other tests 4. Much harder than on other tests
22	How important was it to you to do well on this test? Click one circle only. 1. Not very important 2. Somewhat important 3. Important 4. Very important
Activities outside of school
23	The following questions ask about activities you do outside of school. Click one circle for each row. 1. Yes 2. No
23a	Do you play on a sports team outside of school?
23b	Do you often play a musical instrument outside of school?
23c	Are you studying something in a class outside of school?
23d	Do you belong to a club outside of school (like Boy/Girl Scouts, 4-H, or Boys and Girls Club)?

STUDENT DEBRIEFING QUESTIONS

After all students have finished the assessment in each session, field staff will conduct a debriefing activity with the students. Field staff will ask the debriefing questions out loud to the group of students. The questions may be paraphrased, and additional follow-up questions, such as those indicated in the bulleted lists below, may be asked. Students will respond by raising their hands and by calling out responses, as appropriate. Field staff will use their discretion about whether some or all of the questions will be asked, depending on the amount of prompting the students require, and on the amount of time remaining. The debriefing session should not exceed 10 minutes.

Field staff will write down the students’ responses to the questions.

1. What did you like the most about the assessment?

[Follow-up questions may include:]

• Can you explain that more?

• What did you like the least about the assessment?

• What was missing from the assessment?

• If you had to take a test, would you rather take it on a tablet, or on paper? Why?

• What do you like most about taking a test on a tablet? What do you like the least about taking a test on a tablet?

2. Did you see anything you thought was strange while taking the assessment?

[Follow-up questions may include:]

• Can you explain that more?

• Did questions or images seem slow to load at all to you?

• Was the system slow to move from one page/question to the next at all?

• Did you notice any freezes of your tablet or any error messages?

• Were you able to make these errors occur when you wanted to? How?

3. Did you find the system easy to use and/or easy to learn to use?

[Follow-up questions may include:]

• How did you like the tutorial?

• Did you use the Help screens? Were the Help screens useful?

• What did you think of the tools available? Did you use them?

• Which tool or tools did you like the most? Why?

• Did you have any difficulty using any of the tools? What did you find difficult?

• How did you find the mouse, stylus, and keypad worked with the assessment?

• Did you have any difficulty responding to questions using the available controls? If so, what made it difficult?

4. Is there anything you think would make the assessment better? If so, what?

5. Is there anything else you want to tell us about taking assessments on tablets?

11