1) NPSAS:12 Cognitive Testing 2010; and 2) Program for International Student Assessment (PISA) 2012 Participation Improvement Focus Groups

Program for International Student Assessment (PISA)

Student Focus Group Recruitment Letter

(to student and parent)

Dear _____________,

My name is ______________ and I am writing on behalf of NCES, the National Center for Education Statistics. NCES supports international efforts to collect and report data on education. In 2009 your school was selected for PISA or the Program for International Student Assessment. PISA is a system of international assessments that focus on 15-year-olds' performance in reading, math and science literacy every three years. At present, we are preparing for PISA 2012 and would like to better understand the factors that will motivate students to participate in the assessment and put in their best effort. Your principal recommended that we speak to you and we would like you to participate in a focus group about this topic.

We would like to meet in (Montgomery County, MD or Arlington County, VA) from ___ to ___ (90 minutes) on October ___ with 8 students from high schools in MD, DC, and VA. During the meeting, we will discuss your experiences with voluntary educational assessments and gather your feedback on how to improve existing messages and materials related to PISA. Based on your recommendations, we will create or adapt new materials for PISA 2012.

So what’s in it for you? If you agree to participate in both the focus group discussion you will receive a $35 stipend and your parent(s) will receive $20 to cover travel/transportation costs to bring you to the focus group discussion location.

NCES is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Section 9573, 20 US Code).

While we recognize that this is a busy time for students, we would very much appreciate the opportunity to hear your insights in order to more effectively reach out to students during our PISA 2012 recruitment efforts. In order to participate in the focus group or ask any questions, please contact Shattuck & Associates, the contractor who will be conducting the study. Their office line is 301-829-5737. They will also be collecting the attached consent forms, which should be signed by you and a parent or guardian.

Please let us know at your earliest convenience if you are able to participate.

Sincerely, ….

Program for International Student Assessment (PISA)

Student Focus Group Screening Form

SCREENING QUESTIONNAIRE

Hello, my name is (INTERVIEWER’S NAME) from (RESEARCH COMPANY). We are conducting a short survey and would like to ask you some questions.

1. Are you currently 15 years old?

[ ] Yes ------ CONTINUE

[ ] No ------ THANK AND TERMINATE

2. What do you plan to do after high school?

[ ] Go to 4-year college/university |

[ ] Go to junior/community college |

[ ] Work/get a job |

[ ] Go to vocational school/job training |-- RECRUIT A MIX

[ ] Travel |

[ ] Join the military/military service |

[ ] Not sure/Nothing |

3. Thinking in terms of your grades, where would you put yourself as compared to other seniors in your school? Would you say that you are in the top 10%, the top 25%, top half or bottom half of your class?

[ ] Top 10% |

[ ] Top 25% |

[ ] Top half | -- RECRUIT A MIX

[ ] Bottom half |

[ ] Don’t know/refused |

4. Are you of Latino or Hispanic origin?

[ ] Yes |

[ ] No | -- RECRUIT A MIX

5. Which of the following do you consider yourself to be? Choose one or more. READ LIST

[ ] Asian |

[ ] White |

[ ] Black or African American |--- RECRUIT A MIX

[ ] Native Hawaiian or other Pacific Islander |

[ ] American Indian or Alaskan Native |

6. RECORD GENDER

[ ] Male | --- RECRUIT A MIX

[ ] Female |

Program for International Student Assessment (PISA)

Student Focus Group Consent Form

CONSENT FORM

YOUTH PARTICIPATION IN A QUALITATIVE RESEARCH PROJECT

Focus Group Purpose

A series of focus groups will be held with teens in the Washington, DC area about participation in voluntary educational assessments and what might motivate them to put in their best effort on those assessments. The information from these groups will guide student recruitment in the effort to improve participation rates and collect valid data on educational achievement across the U.S.

Sponsoring Agency

This project is the National Center for Education Statistics, which is part of U.S. Department of Education. Representatives from Shattuck & Associates, Inc., an independent research consulting contractor, will be conducting the focus groups.

Confidentiality

Participation in this study is voluntary. Although notes will be taken during the conversation, no information will be used to identify participants and participant names will not be used in any report.

Questions

If you have any questions about the discussion group, please call Teresa Shattuck of Shattuck & Associates at 301-829-5737.

Consent

I agree to allow __________________________________(SON/DAUGHTER’S NAME) to participate in this focus group.

PARENT/GUARDIAN NAME

PARENT/GUARDIAN SIGNATURE DATE

YOUTH SIGNATURE DATE

Program for International Student Assessment (PISA)

Student Focus Group Moderator’s Guide

Focus Group Prep Sheet

Focus Group Objectives

By the end of today’s discussion, stakeholders will have an increased understanding of:

1. How students view standardized testing.

2. How students might make decisions about whether or not to participate in PISA.

3. Factors that might motivate 15-year-old students to participate and put forth their best effort on an assessment that has no personal consequences.

4. Recommendations for improving existing messages and materials related to PISA.

Date of Event: Start Time: End Time:

Location:

Facilitator: Note Taker:

Total # of participants: # of Men: ___ # of Women: ______

Materials:
Facilitator’s Guide/Script Note Taker’s Guide/Pad/Laptop Markers Name Tags Index Cards Sticky Dots Pens/Pencils Audio Recorder/extra batteries	Registration Form Handout: Participant Survey Handout: Flip Charts Food/Drink Paper Goods Other: Other:

Contact:

Room Set Up: Registration will be set up by host organization outside of the room. Inside the room, a welcome table will be set up with participant surveys, name tags, pens/pencils, markers, and sticky dots. Also set up will be flip chart(s) with markers and the note taker’s computer station.

Flip Chart Posters:

Flip Chart for Brainstorm/Discussion Notes:

Student Focus Group Facilitator’s Guide

Program for International Student Assessment or PISA

I. Welcome Table/Participant Survey (5 minutes)

Set up welcome table name tags, markers, and surveys. Include a welcome sign with the following instructions:

WELCOME!

After making yourself comfortable and enjoying some food/drink, please:

• Fill out a name tag

• Complete a participant survey

II. Introduction

1. Welcome, Thank You, Logistical Info

• Welcome the group.

• Thank participants for coming and for completing the participant survey.

• Point out where restrooms are located and discuss other necessary logistics.

2. Introductions

• Facilitator & Note-taker Introductions- First name and tell group a little about yourselves

• Participant Introductions – First name and single favorite thing about school

3. Background/Purpose/Sponsorship

• The purpose of today’s conversation is to discuss strategies for securing participation of students in PISA 2012.

• More specifically we will seek to better understand:

1. How students view standardized testing.

2. How students might make decisions about whether or not to participate in PISA.

3. Factors that might motivate 15-year-old students to participate and put forth their best effort on an assessment that has no personal consequences.

4. Recommendations for improving existing messages and materials related to PISA.

• These discussion groups are sponsored by the National Center for Education Statistics (NCES) which supports international efforts to collect and report data on education. NCES is housed in the Institute of Education Sciences at the US Department of Education.

• The information you share today is very important. Your opinions will guide NCES in its efforts to collect data that enables comparisons of US students to students around the world.

4. Confidentiality

The National Center for Education Statistics is authorized to conduct this study under Section 9543, 20 US Code. Your participation is voluntary. Your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (Section 9573, 20 US Code). The things you say may be put in a summary of this discussion, but your names will not be included in a summary report.

5. Note taking and Recording Session

We will be taking notes and audio recording the discussion so that we can accurately report the important information that you will be sharing. Again, this information will be kept confidential and will be destroyed once the final report is created. Is this OK with everyone?

Because we are using an audio recorder, please speak one at a time so that all opinions can be clearly heard.

6. Ground Rules

I would like to review some basic guidelines that will be used during our discussion:

• There are no right or wrong answers to the questions being asked today; all ideas are good ideas.

• We also welcome and respect different points of view. Please answer what you truly think, regardless of the opinions of the other group members.

• If you are uncomfortable with a question, feel free to pass. You are under no obligation to answer any question that makes you feel uncomfortable in any way.

• We ask that you share your honest feedback so that the greater community can benefit from your opinions.

• Although you may tell friends and family that you have participated in a focus group around this topic, we ask that you please not share any specific information you hear in this group with anyone outside the group. In this way each person’s perspectives can remain confidential.

• Finally, during today’s discussion, there are quite a few things to talk about; at times we may need to stop the discussion in order to move on. I apologize in advance for that.

7. PISA Overview – post information on wall and leave up throughout the discussion; (note: the information below was pulled from the PISA website)

   • PISA - Program for International Assessment

   • PISA is a system of international testing that focuses on 15-year-olds' performance in reading literacy, mathematics literacy, and science literacy every three years. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling.

   • PISA is organized by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries, and was administered for the first time in 2000.

   • PISA 2012 will focus primarily on math with minor emphasis on reading and science (2006 focused primarily on science; 2003 was math).

   • Administered every 3^rd year since 2000; 66 countries participated in 2009.

   • School and students are randomly selected to participate.

   • The assessment takes approximately 2.5 hours to complete.

3. Focus Group Questions

Objective 1: Perceptions of Standardized Testing

1. Standardized Testing -

• What are your views about standardized tests? (e.g., likes, dislikes)

• How much standardized testing is already going on in your school? (e.g., too little, reasonable amount, too much)

• To what extent are standardized tests valuable to your school? (to administrators, teachers, students, parents)

B. Voluntary & International Assessments

• Are you familiar with any ‘voluntary assessments’? If so, what do you know about them?

• What comes to mind when you hear the term ‘voluntary assessment’?

• Has you ever participated in a ‘voluntary assessment’?

  • If yes, please tell us about the test and your experiences (Probes: What was the test? Was it a valuable experience? If so, why? Are you glad you participated? Would you like to participate in future ‘voluntary assessments’?

  • If no, do you think you would be willing to sit for a ‘voluntary assessment’? Why or why not?

• Are you familiar with international assessments?  If so, what do you know about them?

• What comes to mind when you hear the term ‘international assessment’?

• Has you ever participated in an ‘international assessment’?

  • If yes, please tell us about the test and your experiences (Probes: What was the test? Was it a valuable experience? If so, why? Are you glad you participated? Would you like to participate in future ‘international assessments’?

  • If no, do you think you would be willing to sit for a ‘international assessment’? Why or why not?

C. Perceptions of PISA – refer to posted info about PISA

• What are your initial reactions to PISA? (e.g., likes, dislikes)

• What value do you see in PISA? (to participating students, schools, districts, states, US)

• Do you think it is important for the US to know how its students compare internationally?

Objective 2: How students might make decisions about whether or not to participate in PISA

1. Let’s suppose you were selected for PISA. How would feel about that?

2. Knowing what you know now, as a first instinct, would you be inclined to say ‘yes’ or ‘no’ to participation? Why?

3. With more time to think about participating in PISA, how would you go about making a decision?

• Who would you talk to?

• What would you be curious about? What more would you want to learn?

• Would the length of the test (e.g., 2 vs. 3 hours) influence your decision to participate?

• Would the type of test administration (paper/pencil vs. computer based assessment) influence your decision to participate?

Objective 3: Incentives for Student Participation

1. What do you think students would see as the primary obstacle to participation in PISA? (list & prioritize)

2. What incentives might motivate students to participate in PISA? (list & prioritize)

3. On newsprint share PISA incentives that were used for students in 2009:

• For Students: $20 cash, Certificate of Volunteer Service for 2.5 hours of community Service, and school specific incentives such as student recognition, volunteer hours, tickets to sporting events

2. What are your thoughts about these incentives?

Objective 4: Recommendations for improving existing messages and materials related to PISA

Note: The items below will be selected and adapted for the various existing PISA materials under review (e.g., study brochure, guide, video); not all questions will be asked on all PISA materials.

1. In terms of format for the materials, what are your preferences? (Probe using items below.)

• Size - What size should material be? Small/pocket-sized? Medium? Large?

• Colors - What colors are most appealing to you? Bold/bright? Muted/pastel?

• Graphics - What type of graphics are most appealing to you? Simple or elaborate? Black and white or colorful? Real photos?

• Balance – Does the guide strike a good balance between text/writing and pictures/graphics? Please explain

• Length - About how long should the materials be? Is there a maximum length that would appeal to you? How long should the chapters be? What’s a reasonable range?

• Organization - How should the materials be organized? Can NCES improve the way the existing materials are organized? If yes, how?

• Final comments about format:

• Is there one thing (in terms of format) that really stood out to you / that you really liked?

• Is there anything (in terms of format) that bothered you / that you really didn’t like?

2. How would you describe the quality of the information in the materials? Probes:

• What information did you find helpful?

• What did you learn? Please describe.

• What surprised you?

• Was the information clear? Easy/hard to read? Easy/hard to understand?

• Do the materials present the right amount of information (too much/too little)?

2. Capacity of PISA Materials to Motivate

• What, if anything, about the materials made you WANT to read them?

• What, if anything, about the materials made you NOT want to read them?

• How would you describe how relevant the materials were for you? (Probe: Did you feel the materials were created for you?)

• To what extent does the design of the materials motivate you to continue reading or learning about PISA?

• To what extent does the design of the materials motivate you to participate in PISA?

IV. Closure

1. Summarize/review the main objectives/points that arose during the group discussion.

2. Ask for comments or questions.

3. THANK the group very much for their time and for sharing their opinions and suggestions; the information they shared today was very VALUABLE.

4. Distribute incentives and get signatures for ‘Receipt Sheet.’

Program for International Student Assessment (PISA)

Student Focus Group Survey

Target Audience: 15-Year Old High School Students from the Washington Metro Area

[Perceptions about PISA]

1. After learning about PISA today, how inclined would you be to participate?

 Not at all inclined  Somewhat inclined  Very inclined

Please explain your answer:

2. What is the #1 thing that could be done to increase the chance that you would participate?

3. After learning about PISA today, how inclined do you think other 15-year-old students would be to participate? Explain

 Not at all inclined  Somewhat inclined  Very inclined

Please explain your answer:

4. What is the #1 thing that could be done to increase the chance that other 15-year-old students would participate? Explain.

5. In your view, are some students more or less likely to participate in PISA based on how well they do in school? Explain.

6. Any closing thoughts?

[Experiences with and Attitudes about Standardized Tests]

1. In our school we regularly take standardized tests.

 Not at all true  Somewhat true  True

2. I like taking standardized tests.

 Not at all true  Somewhat true  True

3. Standardized tests are a waste of time.

 Not at all true  Somewhat true  True

4. In our school, standardized tests help teachers make important decision about students.

 Not at all true  Somewhat true  True

5. Standardized tests do not accurately show how well as student knows the material.

 Not at all true  Somewhat true  True

[Participant Demographics]

1. What is your month and year of birth?

Month__________________ Year_______

2. I am:  Male  Female

4. I am a student in a(n) ___________ setting.

 Urban/city  Rural/country  Suburban

4. In what state to you live?

• Maryland  Virginia  Washington, DC

22