PISA Main Study Changes Memo

_MEMORANDUM NATIONAL CENTER FOR EDUCATION STATISTICS

Institute of Education Sciences

United States Department of Education

Date: February 15, 2017

To: Robert Sivinski, OMB

Through: Kashka Kubzdela, NCES

From: Patrick Gonzales, NCES

Re: PISA 2018 Main Study (OMB# 1850-0755 v.21) - changes from last approved study materials for PISA 2018 Main Study Recruitment and Field Test (OMB# 1850-0755 v.20)

The Program for International Student Assessment (PISA) 2018 Main Study Recruitment and Field Test materials and activities were approved by OMB in September 2016 with the latest change request approved in September 2017 (OMB# 1850-0755 v.20). This request is to conduct the PISA2018 Main Study in fall 2018. This memo summarizes changes between the activities and materials approved for PISA 2018 main study recruitment and field test (OMB# 1850-0755 v.18-20) and those that will be used in the PISA 2018 main study as described in the Supporting Statements Part A and B and Appendices A and B of this submission. The final U.S. versions of the 2018 main study questionnaires, as approved by the PISA Governing Board and PISA international consortium, are provided in Appendix B.

Part A

Information about the study design has been added.

The PISA 2018 Main Study Recruitment and Field Test Part A described the assessment design for the field test that included an examination of variable and fixed unit positioning. The result from the field test indicated that variable unit position shown no significant differences in item performance relative to the fixed unit positioning. This allowed for the implantation of a multiple adaptive testing model for the main study. Sections A.2, A.9, A12, and A.14 have been revised throughout to reflect the final plans for the main study.

Part B

Information about the concluded field test has been added, and sections B.2 through B.4 were revised to align with edits made to Part A. Descriptions of main study recruitment and data collection procedures, and a mention of a newly made recruitment video have been added in sections B.2 and B.3.

Appendix A – Recruitment Materials

Several items were added beginning on page 32-that include communications to school coordinators explaining specific activities, the PISA 2018 E-file instructions (for how to submit a teacher list) and PISA 2018 School Coordinator Handbook (alerting school coordinators to what they need to do and where to find more information). Also added were the email invitations to school principals and teachers, with information for accessing the school and teacher questionnaires, and follow-up reminder emails to complete the questionnaires. Login screens are provided for the online school and teacher questionnaires with a description of the screen content.

A placeholder has been added describing a student video that will be used to show students in the study sample what PISA is and what they need to do. A change memo currently scheduled for submission by April 2018 will provide the final video script and link.

Appendix C - Questionnaires

Two types of changes were made to the field test versions of PISA 2018 questionnaires in preparing them for the main study: (1) wording or structure of some items was revised and (2) items were deleted. All changes are listed in the table below. The field test questionnaire item numbers correspond to those in the OMB approved Appendix B Field Test Questionnaires and are provided below to facilitate cross referencing the noted changes.

School Questionnaire

1. Items with revised wording or structure

2018 Field Test		2018 Main Study
Item #	Field Test Item	Item#	Main Study Item
SC002	As of February 1, 2017, what was the total school enrolment (number of students)?	SC002	As of February 1, 2018, what was the total school enrolment (number of students)?
SC064	During 2015-2016 academic year, what proportion of students' parents participated in the following school-related activities?	SC064	During 2017-2018, what proportion of students' parents participated in the following school-related activities?

2. Deleted school questionnaire items from field test

SC157	To what extent do the following statements reflect an opinion shared by your teaching staff?
SC165	Do the following statements reflect teachers’ practices for multicultural learning in your school?
SC166	To what extent do the following statements reflect an opinion shared by your teaching staff?
SC167	Is there any formal curriculum for the following skills in 10th grade?
SC158	Is there any formal curriculum for the following topics in 10th grade?

Teacher Questionnaire

1. Items with revised wording or structure

2018 Field Test		2018 Main Study
Item #	Field Test Item	Item#	Main Study Item
	NA

2. Deleted teacher questionnaire items from field test

	2018 Field Test
TC206	Do the following statements reflect your education and training as a teacher?
TC198 – Q03	We would like to know how you generally feel about your job. How strongly do you agree or disagree with the following statements? I would like to change to another school if that were possible.
TC198 – Q08	We would like to know how you generally feel about your job. How strongly do you agree or disagree with the following statements? I think that the teaching profession is valued in society.
TC198 – Q09	We would like to know how you generally feel about your job. How strongly do you agree or disagree with the following statements? I am satisfied with my performance in this school.
TC046 – Q01	On average, how often do you do the following in this school? Teach jointly as a team in the same class
TC046 – Q02	On average, how often do you do the following in this school? Observe other teachers' classes and provide feedback
TC046 – Q03	On average, how often do you do the following in this school? Engage in joint activities across different classes and age groups (e.g. projects)
TC046 – Q04	On average, how often do you do the following in this school? Exchange teaching materials with colleagues
TC199-Q06	In your teaching, to what extent can you do the following? Make my expectations about student behaviour clear
TC199-Q07	In your teaching, to what extent can you do the following? Help students think critically
TC166-Q04	In your lessons, have you ever taught any of the following things? To understand the consequences of making information publicly available online on Facebook©, Instagram©, etc.
TC195- Q03	If at least one book was assigned, which choices did the students have? The students could freely choose the book(s) they read.
TC195- Q04	If at least one book was assigned, which choices did the students have? The students could choose books on a list provided by you or freely, but the choice had to be approved by you.
TC176	How often are you involved in the following reading activities?
TC175	About how much time per week do you spend reading for your work (e.g. articles, magazines, books, manuals and websites) out of your classes?
TC172	Which of the following statements best describes how you read books (on any topic)?
TC173	Which of the following statements best describes how you read the news (e.g. politics, culture, sport, local news)?
TC187	Have you ever taught in a country other than the United States?
TC188	Have you studied in a country other than the United States?
TC189	Do you hold a university degree from a country other than the United States?
TC200	To what extent do the following statements reflect an opinion shared by the teachers of your school?
TC208	To what extent do the following statements reflect an opinion shared by the teachers of your school?
TC207	In your lessons, do you include opportunities to promote the following skills?
TC178	In your lessons, do you include the following global challenges and trends?
TC196	People are increasingly moving from one country to another. How much do you agree with the following statements about immigrants?
TC209	How do you judge your own competence to teach in a class with a high degree of cultural and ethnic diversity?
TC181	How often do you read in a language other than English?
TC055-Q07	To what extent do you use the following approaches to assign final semester grades to students in 10th grade? I compare a student's performance to that of other students in the course.
TC055-Q011	To what extent do you use the following approaches to assign final semester grades to students in 10th grade? I consider the degree to which the student participates in the class.

Student Questionnaire

1. Items with revised wording or structure

2018 Field Test		2018 Main Study
Item #	Field Test Item	Item#	Main Study Item
Intro1	In this questionnaire you will find questions about the following topics: - You, your family, and your home - How you think about your life - Your school - Your school schedule and learning time - Language learning in school - Your view on reading	Intro1	Note: The bullet point order has changed in MS In this questionnaire you will find questions about the following topics: - You, your family, and your home - Language learning in school - Your view on reading - How you think about your life - Your school - Your school schedule and learning time
ST003	On what date were you born?	ST003	On what date were you born?
01	1998	01	2000
02	1999	02	2001
03	2000	03	2002
04	2001	04	2003

2. Deleted student questionnaire items from field test

	2018 Field Test
ST224	In how many languages are you able to do the following things?
ST189	How many foreign languages do you learn at your school this school year?
ST224	How well does each of the following statements below describe you?
ST215	How well does each of the following statements below describe you?
ST216	How well does each of the following statements below describe you?
ST217	How well does each of the following statements below describe you?
ST218	Imagine you are talking in your native language to people whose native language is different from yours. To what extent do you agree with the following statements?
ST219	To what extent do you agree with the following statements?
ST196	How easy do you think it would be for you to perform the following tasks on your own?
ST197	How informed are you about the following topics?
ST221	Do you learn the following at school?
ST220	Do you have contact with people from other countries?
ST200	How many of your close friends come from a country different from yours?
ST222	Are you involved in the following activities?
ST223	Thinking about teachers in your school: to how many of them do the following statements apply?
ST204	People are increasingly moving from one country to another. How much do you agree with the following statements about immigrants?
ST036-Q07	Thinking about your school: to what extent do you agree with the following statements? I enjoy receiving good grades.
ST181-Q01	How much do you agree with the following statements about yourself? I think that succeeding both at work and when playing games is important.
ST181-Q05	How much do you agree with the following statements about yourself? It annoys me when other people do better than me.
ST182-Q02	How much do you agree with the following statements about yourself? I more often attempt tasks that I am not sure I can do than tasks that I believe I can do.
ST183-Q04	How much do you agree with the following statements? When I am failing, people are less interested in me.
ST183-Q05	How much do you agree with the following statements about yourself? When I am failing, it matters to me that others who are important to me are disappointed.
ST187	Think about your goals in school: how true are the following statements for you?
ST188-Q04	How much do you agree with the following statements? I can get through difficult times because I’ve experienced difficulty before.
ST188-Q05	How much do you agree with the following statements? I can usually find something to laugh about.
ST123-Q01	Thinking about this school year, to what extent do you agree or disagree with the following statements? My parents are interested in my school activities.
ST209	During the past 12 months, how often did you have the following experiences at school?
ST210	During the past 12 months, how often have you had the following experiences in school?
ST177	How many languages, including the language(s) you speak at home, do you and your parents speak well enough to converse with others?
ST100-Q05	How often do these things happen in your English/Language Arts classes? The teacher gives students an opportunity to express opinions.
ST104-Q01	How often do these things happen in your English/Language Arts classes? The teacher tells me how I am performing in this course.
ST104-Q05	How often do these things happen in your English/Language Arts classes? The teacher advises me on how to reach my learning goals.
ST152-Q1	In your English/Language Arts classes, how often does the following occur? The teacher asks students to explain the meaning of a text.
ST152-Q2	In your English/Language Arts classes, how often does the following occur? The teacher asks questions that challenge students to get a better understanding of a text.
ST152-Q3	In your English/Language Arts classes, how often does the following occur? The teacher gives students enough time to think about their answers.
ST152-Q4	In your English/Language Arts classes, how often does the following occur? The teacher recommends a book or author to read.
ST157-Q03	If at least one book was assigned, which choices did you have? The students could freely choose the book(s) they read.
ST157-Q04	If at least one book was assigned, which choices did you have? The students could choose books on a list provided by the teacher or freely, but the choice had to be approved by the teacher.
ST152-Q07	During this academic year, when you have to read a book or a chapter from a book for your English/Language Arts class, does the teacher usually ask you to do the following tasks in the classroom or for homework? Evaluate the writing style of the author
ST160-Q06	How much do you agree or disagree with these statements about reading? When I read something I really like, I forget about time.
ST161-Q04	How much do you agree with the following statements? I am able to identify the main ideas in an informational or documentary text.
ST161-05	How much do you agree with the following statements? After having read a story, I am able to tell what the story was about.
ST163-Q05	How do you feel about these reading tasks? I could answer most of the reading comprehension questions.
ST169	Which of the following statements best describes how you read the news (e.g. politics, culture, sport, local news)?
ST170	Do you read in the following situations?
ST173	How often do you read in a language other than English outside of school?
ST176-Q3	How often are you involved in the following reading activities? Reading online news
ST176-Q6	How often are you involved in the following reading activities? Taking part in online group discussions or forums
ST176-Q08	How often are you involved in the following reading activities? Participating in social networks (e.g., Facebook, Instagram)
ST039-Q1	During the past 12 months, how often did you have the following experiences at school? Teachers called on me less often than they called on other students.
ST039-Q2	During the past 12 months, how often did you have the following experiences at school? Teachers graded me harder than they graded other students.
ST039-Q3	During the past 12 months, how often did you have the following experiences at school? Teachers gave me the impression that they think I am less smart than I really am.
ST038-Q1	During the past 12 months, how often have you had the following experiences in school? I got called names by other students.

Student Financial Literacy Questionnaire

1. Items with revised wording or structure

2018 Field Test		2018 Main Study
Item #	Field Test Item	Item#	Main Study Item
	NA

2. Deleted items from field test

	2018 Field Test
FL166-Q04	How often have you encountered the following types of tasks or activities in a school lesson in the last 12 months? Making calculations such as the following: if you put 100 dollars in a savings account for three years at the annual interest rate of 1%, at the end of the third year - without withdrawing or adding any money - how much money will there be?