Program for International Study Assessment (PISA 2015) Interviews and Focus Group with School Principals

Appendix A
Principal Focus Group Materials

A – 1. Principal In-depth Interview Recruitment Script ............................................ 1
A – 2. Principal Refusal Focus Group Recruitment Script ....................................... 2
A – 3. Principal Focus Group Screener ................................................................... 3
A – 4. Principal In-depth Interview and Focus Group Consent Form ..................... 5
A – 5. Facilitator’s Guide for In-Depth Interviews .................................................... 6
A – 6. Facilitator’s Guide for In-Person and Webinar Focus Groups .......................10

Exhibit A - 1. Principal In-depth Interview Recruitment Script

PROGRAM FOR INTERNATIONAL STUDENT ASSESSMENT (PISA)
Principal In-depth Interview Recruitment Script
(will be shared via email, letter, or phone)
Dear _____________,
My name is ______________ and I am reaching out to you on behalf of the National Center for Education
Statistics, also known as NCES. NCES supports international efforts to collect and report data on education. In
2012, your school was selected for PISA or the Program for International Student Assessment. As you may
recall, PISA is a system of international assessments that focuses on 15-year-olds' performance in reading,
mathematics and science literacy every three years. At present, we are preparing for PISA 2015 and would like
to better understand the factors that motivated you and your students to participate in PISA 2012.
To that end, we would like to conduct an interview with you to discuss your experiences with PISA and garner
advice to guide the development of communication materials for PISA 2015. Based on your recommendations,
we will create or adapt new materials for PISA 2015 and revise our procedures for communication with
schools. We will schedule this at the most convenient time for you as possible between and .
So what’s in it for you? If you agree to participate in the interview in you will receive a $50 stipend.
NCES is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary. Your
answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for
any other purpose except as required by law (Section 9573, 20 US Code).
While we recognize that this is a busy time for schools, we would very much appreciate the opportunity to
hear your insights, as well as those of your students, in order to more effectively reach out to schools during
our PISA 2015 recruitment efforts. Please let us know at your earliest convenience if you are able to
participate.
Sincerely, ….

A-1

Exhibit A - 2. Principal Refusal Focus Group Recruitment Script

PROGRAM FOR INTERNATIONAL STUDENT ASSESSMENT (PISA)
Principal Refusal Focus Group Recruitment Script
(will be either email, letter, or phone call)
Dear _____________,
My name is ______________ and I am writing on behalf of NCES, the National Center for Education Statistics.
NCES supports international efforts to collect and report data on education. In 2012, your school was selected
for PISA or the Program for International Student Assessment. As you may recall, PISA is a system of
international assessments that focus on 15-year-olds' performance in reading, math and science literacy every
three years. At present, we are preparing for PISA 2015 and would like to better understand the factors that led
you and your school not to participate in PISA 2012.
To that end, we would like to meet via WebEx between <> on <> with 8 to 10 principals from
schools that refused to participate in PISA 2012. During the meeting, we will discuss your experiences with
PISA and garner advice to guide the development of communication materials for PISA 2015. Based on your
recommendations, we will create or adapt new materials for PISA 2015. We also want to know about the
decision making that is involved in deciding to participate in studies like PISA and how we might better target
our materials and discussions when approaching schools to participate.
So what’s in it for you? If you agree to participate in the meeting in <> you will receive a $50
stipend.
NCES is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary.
Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form
for any other purpose except as required by law (Section 9573, 20 US Code).
While we recognize that this is a busy time for schools, we would very much appreciate the opportunity to hear
your insights, as well as those of your students, in order to more effectively reach out to schools during our
PISA 2015 recruitment efforts. Please let us know at your earliest convenience if you are able to participate.
Sincerely, ….

A-2

Exhibit A - 3. Principal Focus Group Screener

Sample Screening Questionnaire for Principals (if necessary)
Hello, my name is from . We are conducting a short
survey among principals and would like to ask you a few questions. We are not selling anything.
We are simply interested in your opinions.
1. In the past 6 months, have you participated in a market research group discussion?
[ ]
[ ]

Yes -- THANK AND TERMINATE
No -- CONTINUE

2. Are you currently an active high school principal?
[ ]
[ ]

Yes -- CONTINUE WITH QUESTION 3
No -- THANK AND TERMINATE

3. Are you familiar with the Program for International Student Assessment, also known as PISA? It
is an international assessment that measures 15-year-old students' reading, mathematics, and
science literacy.
[ ]
Yes -- CONTINUE WITH QUESTION 4
[ ]
No -- CONTINUE WITH QUESTION 6
4. Have you ever been at a school that was asked to participate in PISA?
[ ]
[ ]

Yes -- CONTINUE WITH QUESTION 5
No -- CONTINUE WITH QUESTION 7

5. Did your school participate in PISA, or not?
[ ]
[ ]

Yes -- THANK AND TERMINATE
No -- CONTINUE WITH QUESTION 6

6. Were you the one who decided whether your school would participate in PISA?
[ ]
[ ]

Yes -- THANK AND TERMINATE
No -- CONTINUE WITH QUESTION 7

7. Which of these best categorizes your school? → RECRUIT A MIX
[ ]
[ ]
[ ]

Rural
Urban
Suburban

A-3

Exhibit A - 3. Principal Focus Group Screener - Continued

8. How would you categorize your school’s enrollment?
[ ]
[ ]
[ ]

Small
Medium
Large

9. Are you of Hispanic, Latino, or Spanish origin? → RECRUIT A MIX
[ ]
Yes
[ ]
No
10. What is your race? (You may select one or more.) → RECRUIT A MIX
[
[
[
[
[
[
[
[
[

]
]
]
]
]
]
]
]
]

White
Black or African American
American Indian or Alaska Native
Asian (Chinese, Filipino, Japanese, Korean, Vietnamese)
Other Asian (Please specify)

Native Hawaiian
Guamanian or Chamorro
Samoan

Other Pacific Islander

11. RECORD GENDER. DO NOT ASK → RECRUIT A MIX
[ ]
Female
[ ]
Male

We would like to invite you to a group discussion among principals such as you. The discussion will
last approximately 1½ to 2 hours and will be held at on at
. Your opinions are very important to us and to compensate you for your time, we will give
you [$75 IN PERSON / $50 WEBEX] as a token of our appreciation. Are you able to join us?
RESPONDENT NAME: ___________________________________
SCHOOL NAME: _________________________________________
TELEPHONE/EMAIL: ____________________________________

[ ] GROUP A: Principals who have refused to participate in PISA
[ ] GROUP B: Principals who have never been asked to participate in PISA

A-4

Exhibit A - 4. Principal In-depth Interview and Focus Group Consent Form

PROGRAM FOR INTERNATIONAL STUDENT ASSESSMENT (PISA)
Principal In-depth Interview and Focus Group Consent Form

CONSENT FORM
PARTICIPATION IN A QUALITATIVE RESEARCH PROJECT

FOCUS GROUP PURPOSE
A series of telephone interviews and focus groups will be held with high school principals about participation in
voluntary educational assessments, and communication and motivation strategies for school and student
recruitment in those assessments. The information from these groups will guide school and student
recruitment in the effort to improve participation rates and collect valid data on educational achievement
across the U.S.
SPONSORING AGENCY
This project is on behalf of the National Center for Education Statistics, which is part of U.S. Department of
Education. Representatives from Hager Sharp, an independent communications firm, will be conducting the
interviews and focus groups.
CONFIDENTIALITY
Participation in this study is voluntary. Although notes will be taken during the conversation and telephone
interviews will be recorded, no information will be used to identify participants and participant names will not
be used in any report.
QUESTIONS
If you have any questions about the discussion group, please call Alexandra Roose of Hager Sharp at 202-XXXXXXX.

CONSENT
I, __________________________________, agree to participate in this focus group.

SIGNATURE

DATE

A-5

Exhibit A - 5. Facilitator’s Guide for In-Depth Interviews

In-Depth Interview Guide:
Principals who have previously participated in PISA
INTRODUCTION


My name is [INSERT] and I work for Hager Sharp, a communications firm, working on
behalf of the National Center for Education Statistics (NCES) within the Department of
Education. Is this still a good time to talk? It should take about 30-45 minutes.



I’m hoping to get your honest opinions during our phone call today. There are no right
or wrong answers to anything that we will be discussing. I encourage you to express
your opinion even if you think it might be ‘harsh’ or critical. Our conversation is
confidential; what you and others say will be summarized in a report for NCES and will
not identify you.



We will be discussing some materials that you’ve been shown, which were used when
you participated in the 2012 administration of PISA. Do you have these materials
available/in front of you? I just want to point out that these are not materials that I have
developed, so you will not flatter me or insult me by anything that you say. My job is to
ensure that we cover everything that we need to cover and that I keep the discussion
going. I have no stake in the outcome of the discussion.



We will be audio recording the session so that I have a record of the discussion to write
my report. Once the report is written, we will destroy the recording.



Any questions before we get started?

1
A-6

Exhibit A - . Facilitator’s Guide for In-Depth Interviews - Continued

EXPERIENCE WITH PISA
I’d like to start off with some questions about your experience with the 2012 administration of
PISA in your school.


How did you first hear about PISA? PROBE:
o Was it when you were asked to participate, or before that?
o Heard about it in the news?
o Read/saw materials about it?



What were your initial reactions to PISA? (e.g., likes, dislikes)



What influenced your decision to allow your school to participate? PROBE:
o Did you seek additional information/advice?
o Did anyone offer (without you asking) advice or recommendations to you on
participation?
o Who did you get advice from? Someone from your district? A fellow principal? A
teacher or staff member? PROBE
 Would it have been beneficial if we had shared materials with other
contacts? (Who? What role do they play?)
o How much of a role did the materials you were given play in your decision?
o How much of a role did the financial incentives play in your decision?
o What concerns did you have about participating?



What were your expectations when you agreed to participate? PROBE:
o How did you expect your teachers to react?
o How did you think your students would react to being selected?
o How did you expect parents of selected students to respond?



How did the administration of PISA compare to what you expected?
o Was it better, worse, or about what you expected?
o Looking back, is there anything your school—or the PISA team—could have done
differently?
o How burdensome was participating in PISA for your school?
o How did students and teachers feel about participating in PISA?
o How did they feel about the classroom time they missed due to the
administration?



What was the value of participating? (for you, your school, your students, your district,
your state, the nation) PROBE:
o What was your reaction to the school-level report you received?
o Was your school recognized in any way for having participated?
o How did your participation help inform educational decision-making?
2
A-7

Exhibit A - 5. Facilitator’s Guide for In-Depth Interviews - Continued

o Were there any other benefits?


Were there any drawbacks or challenges that came as a result of participating that you
didn’t anticipate? PROBE:
o How did you manage them?



Would you recommend participation again for your school?
o Why or why not?
o If you were talking with another principal whose school was asked to participate,
what advice would you give?

REACTION TO MATERIALS
We’ve talked about your experience with PISA, and now I’d like to shift to the materials I shared
with you. Have you had an opportunity to review them? These are the same materials that you
should have received when you participated in 2012.


Do you remember receiving these materials when your school participated in 2012?
o Was the timing appropriate, or would it have been better to have received them
earlier or later than you did?



What is your overall reaction to these materials?



Did anything in particular grab your attention or surprise you about them?



Was anything confusing or difficult to understand?



What questions, if any, came to mind as you reviewed them?



How do you feel about the quantity of materials in the packet?



Were any of the materials useful? If so, which ones? What made them useful?



What wasn’t useful to you? Why?



Of the information here, what is most critical for principals to know as they consider
participating in PISA?



Is there anything that might be missing from these materials? Anything else that would
have been more informative for you and your school?



How does this compare to the kind of information that you receive for other types of
tests?
3
A-8

Exhibit A - 5. Facilitator’s Guide for In-Depth Interviews - Continued



How did/would you use this information? In what way did/would it help you?



Did you share these materials with anyone else when you received them in 2012?
o Who did you share them with?
o Which ones did you share?
o Were they helpful?

CONCLUSIONS


That’s all my questions.



Do you have any additional comments about these materials, PISA, or assessments in
general?



Feel free to call or email me if you think of anything else that we didn’t discuss today.

 Thank you.

4
A-9

Exhibit A - 6. Facilitator’s Guide for In-Person and Webinar Focus Groups

PISA 2015
Facilitator’s Guide:
Principals Who Refused PISA Participation
Principals Who Have Not Been Approached to Participate
Research Questions








What value do international assessments and comparisons have in education?
What barriers did principals associate with participation in PISA 2012/do they associate with
participation in PISA?
What would minimize the burden in the principals’ school? (Briefly)
Given the structure of PISA 2015 (CBA and the associated assessment time), what factors might
motivate principals to agree to participate in voluntary assessments such as PISA?
What are principals’ reactions to the benefits articulated by principals who did participate?
What are their reactions to the current PISA 2015 materials?
o What materials convey useful information?
o What materials are not useful or informative?
What are their suggestions for recruitment strategies and materials?

INTRODUCTION (5 MINUTES)


My name is Maria Ivancin and I work for the Market Research Bureau, a research
company, working on behalf of the National Center for Education Statistics (NCES)
within the Department of Education. We do research to help clients make decisions
about many different types of issues.



This particular method that we’re using today is what we call a focus group discussion.
It allows us to talk with people in a bit more depth than other methods such as surveys.
We are looking to get your honest opinions. I will be asking questions, but those
questions are meant to generate discussion. There are no right or wrong answers to
anything that we will be discussing. I encourage you all to express your opinion even if it
might be different from everyone else’s. We want all types of viewpoints – positive,
negative, and everything in between. We do not have to come to any consensus on
what we’ll be talking about. Everyone’s opinion is important so I would like to hear
from everyone.



You’ve been shown/will be seeing some information and reports. I just want to point
out that these are not materials that I have developed, so you will not flatter me or
insult me by anything that you say. My job is to ensure that we cover everything that
we need to cover and that I keep the discussion going. I have no stake in the outcome
of the discussion.
1
A - 10

Exhibit A - . Facilitator’s Guide for In-Person and Webinar Focus Groups - Continued



We will be audio recording the session so that I have a record of the discussion to write
my report. Once the report is written we will destroy the recording.



(IF APPLICABLE) Does everyone have the packet of materials that you received
available—and reading glasses if you need them? [Refusors will receive materials in
advance; unapproached will receive materials during the focus group]



(IF APPLICABLE) Are there any questions about using the WebEx technology? ANSWER
ANY QUESTIONS ABOUT WEBEX



I’d like everyone to introduce themselves. Let me start (MODERATOR INTRO). Now, tell
me about yourself. When you do, please be sure to tell me your name (first name is
okay), a little bit about your school (size, grades, region of country), how long you’ve
been in education and anything else you’d like to tell us about yourself.

VALUE OF INTERNATIONAL ASSESSMENTS (10 MINUTES)


Let’s start by talking broadly about assessments that allow for comparison across
different types of schools or different geographies/regions. What assessments are you
aware of? What comes to mind?



Have you ever participated in any of these kinds of assessments? Do you know anyone
whose school has participated?



What is your overall reaction to such assessments?



To whom are they valuable?



Let’s talk specifically about international assessments – where we can compare the
education of students across different countries. How important are international
comparisons? PROBE:
o
o
o
o



For our nation?
Your state?
Your school?
District- or building-level administrators in general?

How much value would there be to your school to participate in an international
assessment?

2

A - 11

Exhibit A - . Facilitator’s Guide for In-Person and Webinar Focus Groups - Continued

BARRIERS TO PISA--REFUSORS (10 MINUTES)


Let’s talk more specifically about PISA. Your school was asked to participate in PISA and
decided not to.
o What factors led to that decision?
o How was the decision made not to participate? Tell me the process. Who was
involved in the decision? PROBE:
 Just you?
 District officials?
 Principal colleagues?
 Teachers?



What was their reaction?
o Was it something about the test or something about your school’s circumstances
at the time?
o Was there a particular “deal-breaker?”

BARRIERS TO PISA--UNAPPROACHED (10 MINUTES)



Let’s talk more specifically about an assessment called PISA.
I’m going to give you a packet of information about PISA that you would receive if your
school was asked to participate. The first thing I’d like you to do with it is glance through
it as you usually would, identify the piece you would read first, and look it over. [HAND
OUT PACKETS.]



[GIVE PARTICIPANTS A MOMENT TO REVIEW PACKETS, THEN GO AROUND TABLE AND
HAVE EACH SHOW THE PIECE THEY WOULD READ FIRST.]



At this point, what are your thoughts on PISA?
o What, if anything, is appealing about participating?
o What, if anything, are you concerned about?



Please take a few minutes to look through the rest of the packet. Based on the materials
that you read and anything else that you may know about PISA how would you react if
your school was asked to participate in PISA?
o What factors would you consider?
o What questions would you have?
o How would the decision to participate or not be made? Who would be involved
in the decision? What would be the best way to reach them?
o What specific information would you need or want in order to decide whether to
participate?

3

A - 12

Exhibit A - . Facilitator’s Guide for In-Person and Webinar Focus Groups - Continued

o Would the support—or resistance—of any particular person or group be
important in your decision? Who? (District officials? Principal colleagues?
Teachers? Parents?)
BARRIERS TO PISA--ALL (10 MINUTES)


Let’s take a few minutes to talk about what it would be like to administer PISA in your
school [BRIEFLY—use as a way to introduce details about PISA]
o How do you feel about PISA, or other assessments, being completely computeradministered?
o How do you feel about an assessment taking place over two days, rather than
one?
o How do you feel about having a group of students of a particular age
participating in an assessment, versus a group of students in a particular grade?



How likely would you be to participate in PISA in the future? NOTE: Nothing you say in
the group will commit you either way. We want to gauge your reaction to the concept,
but are not asking you to commit to participation in any way.
o What makes you say that?
o What aspects of PISA are you concerned about?
o What aspects are you interested in or do you find attractive?



What, if anything, might change your mind about participating in PISA?
o Would something about your school need to change? What?
o Would something about the test need to change? What? (PROBE FOR TIMING,
ETC.)



If one thing could change about PISA, what would be most important to increase your
likelihood of participation?



Of the assessments your school participates in:
o Which is the easiest? What makes it easy?
o Which is the most difficult? What makes it difficult?



Are there any other barriers that you see in participating in PISA? Are there any other
concerns or negative reactions that you have?

BENEFITS OF PISA (10 MINUTES)


Let’s talk a little more about the benefits to participating in PISA. What do you think
would be the benefits of having your school participate in PISA? (PROBE FOR BENEFITS
TO STAFF, STUDENTS, COMMUNITY)



Which of these is most important to you?
4
A - 13

Exhibit A - . Facilitator’s Guide for In-Person and Webinar Focus Groups - Continued



Here are some benefits mentioned by principals whose schools have participated in PISA
[READ LIST, NOTING ANY ALREADY MENTIONED BY PARTICIPANTS] What do you think
about those?



Do the benefits of participation outweigh the drawbacks? How could we change that?

REACTIONS TO PISA 2015 MATERIALS (30 MINUTES)


You’ve all had an opportunity to review the draft materials for PISA 2015.



What was your overall reaction to the packet of materials? Everyone, please give me
one sentence.



Please go through the folder and locate the two pieces that were most useful to you. In
other words, if you were going to tell another principal about PISA, which one or two of
these materials would you share? [HAVE EACH PARTICIPANT WRITE DOWN THEIR TOP 2;
TALLY]



Let’s talk about [HOLD UP/NAME A MATERIAL MENTIONED; REPEAT SEQUENCE FOR
EACH MATERIAL MENTIONED:]
o What made it useful?
o What did you like about it?
o What didn’t you like?
o Was anything confusing or hard to understand?
o Was anything missing?
o Is there anything in here that doesn’t need to be here? Anything that shouldn’t
be here?
o How believable was this information?
o How relevant was the information to you? Did it speak to you (as opposed to
someone in a different position or role)?
o What did you think about the length?
o How does this compare to the information that you receive on other types of
assessments?



We’ve been talking about what materials you found useful. Were there any materials in
this packet that were unnecessary? Which ones? What made them unnecessary?

RECRUITMENT STRATEGIES AND SOURCES OF INFORMATION (10 MINUTES)


This packet represents one way to communicate about PISA with you.

5

A - 1

Exhibit A - . Facilitator’s Guide for In-Person and Webinar Focus Groups - Continued



Where does a packet of materials rank on the list of best ways to communicate with
you?



What would work better? PROBE:
o Telephone call—from whom?
o Personal visit?
o Email?)



Where do you get information about what’s happening in education? About what’s
happening in your school system?



What sources would you trust the most in getting information about assessments?



Where would you go to get additional information about PISA?

CONCLUSIONS


Do you have any additional comments about these materials, PISA, or assessments in
general?



Thank you.

6

A - 1