PISA 2009 Incentive Change Memo

To: Shelly Martinez, OMB

From: Daniel McGrath, NCES

Date: July 14, 2009

Subject: Change request for PISA 2009 incentive plan (1850-0755 v.7)

This memorandum describes issues encountered with securing school participation in the PISA 2009 assessment and requests changes to the approved PISA 2009 incentive plan (1850-0755 v.7) to encourage higher response rates so that the United States can meet international sampling requirements and be included in the reported PISA results, and to improve comparability of incentives and stipends across schools. The proposed changes are consistent with the final incentive plan used for PISA 2006. NCES is requesting two changes: (1) increase the incentive for refusing schools unable to attend the recent PISA summer conference for participating schools to the average amount spent on schools that attended the conference ($1500); and (2) increase the stipends for school coordinators (to $300) and students (to $75) in schools conducting the assessment out of school hours to better reflect the opportunity cost of their participation.

Background

The minimum allowable response rates for schools and students in PISA are as follows:

• 65 percent of originally sampled schools (if replacement schools are also used to supplement the sample; if replacement schools are not used, the minimum response rate is 85 percent of originally sampled schools—the United States includes replacement schools and so our minimum allowable rate is 65 percent of originally sampled schools).

• 80% student response rate (additionally, with respect to student response rates, schools with student response rates below 50% are not regarded as participating schools).

The data of countries that do not attain the minimum response rates are not included in international PISA results or in the international PISA database. In each of the past PISA administrations, the United States has barely achieved the minimum school response rate.¹ Other countries have not been so fortunate, including the United Kingdom in 2003 and the Netherlands in 2000, which failed to reach minimum school response rates and were not included in the international reports or databases. The United States will spend approximately $10 million² on PISA 2009, and so the potential loss if we fail to attain the minimum response rates is large.

Historically, meeting the minimum original school response rate has been difficult for the United States.  In 2000 and 2003, the United States barely met the international requirements for school response rates. As a result, in 2006, we introduced extra measures not implemented in PISA 2000 or 2003 that were intended to increase school response rates: holding a conference in Washington, DC for participating schools in the summer before PISA’s administration to provide them the opportunity to learn more about how PISA data are used to inform education policy and how the PISA assessment is administered in schools, and offering the option of administering the PISA assessment outside of school hours. Although we believe these measures were very useful in securing school participation, when it became clear that the United States was in danger of not reaching the minimum school response rate even with these inducements, the incentive plan was modified. It was modified in two ways: (1) to allow NCES to offer refusing schools that did not attend the summer conference an additional incentive equal to the average amount spent on schools that participated in the conference and (2) to allow NCES to offer larger stipends to school coordinators and students in schools participating outside of school hours to reflect the higher opportunity costs associated with outside school hours administration and facilitate high student participation. These measures were intended to increase response rates and to enhance fairness in the incentive plan.

In 2009, we used a similar incentive plan as the original plan used in 2006. The modifications to the 2006 incentive plan were not included in the original 2009 incentive plan in the hopes that the school response rate at the time of the summer conference would be sufficiently high that additional recruitment of original schools would not be required after the summer conference and that a high percentage of original schools would opt for administration within school hours (the overall burden to the school is thought to be lower for administration within school hours than outside school hours and it can be difficult to secure sufficiently high student participation outside school hours). Indeed, recruitment began a full year before data collection and included extraordinary measures such as the direct involvement of state officials in district and school recruitment.

However, NCES has just held the summer conference and school response rates are low and in danger of falling below the minimum allowable rate. To date, 66 percent (137 of 209) of the original sample schools have agreed to participate, which means that the United States is just barely meeting the required international minimum response rate for original schools. In the past month, 10 schools that had previously agreed to participate have notified the U.S. PISA contractor that they will now not be able to participate. NCES is concerned that more schools will back out in the coming months. Losing even 2 additional schools will bring the U.S. school response rate below the required international standard.

As in the past years, schools are reluctant to participate because PISA is not mandatory, schools do not want to lose instructional time, already feel overburdened by assessments, and see relatively little immediate benefit for themselves. This year, difficulties in gaining school participation have been especially acute. We have heard from field staff and from participants at the summer conference that the economic downturn is creating heightened concern with the burden and the uncertainty of having staff in place to coordinate the assessment.

As described in the PISA 2009 OMB clearance package, NCES and its contractors have already taken a number of steps to encourage school participation in PISA 2009, including the following.

• Burden on schools in terms of the information and actions requested prior to the assessment are kept to a minimum.

• The assessment is being administered in the fall, rather than the spring.

• After school or Saturday administration of PISA is allowed to provide greater flexibility to schools.

• Schools receive $200 as compensation for participating.

• The PISA school coordinators are provided a $100 reimbursement for carrying out administrative tasks in preparation for the test administration that will be carried out by the PISA contractor (coordinate logistics with the data collection contractor; supply a list of eligible students for sampling to the data collection contractor; communicate with teachers, students, and parents about the study to encourage participation; assist the test administrator in ensuring the sampled students attend the testing session; and assist the test administrator in arranging for make-up sessions as needed).

• Each student is offered $20 if they complete PISA assessment during school hours and $35 if they complete it during after school hours or on Saturday.

• NCES hosted a conference—PISA Comes to Your School—in Washington, DC, June 25-26, 2009, for participating schools. The purpose of the conference was to inform school representatives about PISA, to explain how PISA results are used, and to motivate schools’ participation in the fall 2009 data collection. The PISA 2006 summer conference was a big success in terms of keeping the schools' promises of their participation in the fall. We believe that PISA 2009 summer conference will serve the same purpose. Representative from 87 schools attended the conference.

• At NCES’s request, Arne Duncan, the U.S. Secretary of Education, sent a letter in June 2009 to superintendents of districts that oversee schools in the original PISA 2009 sample to encourage their participation. This extraordinary measure was taken because the United States is in jeopardy of not meeting the international standards. Secretary Duncan also addressed the school representatives at the PISA Comes to Your School conference, another effort to encourage schools to stay engaged in PISA assessment.

Unfortunately, these measures designed to encourage school participation in PISA are proving insufficient in 2009, and the United States is only two schools away from falling below the minimum original school response rate. During the 2009 summer conference, one school representative in the public question and answer session stated that schools consider monetary incentives when faced with budgeting uncertainty. This statement was echoed in informal discussions with other school representatives during the conference and during recruitment when field staff talked with school representatives about participating. Revising the incentive plan provides the best chance for increasing the participation rate of the original schools from the PISA sample, in particular among schools considered to be final refusals. Not taking strong steps to increase the participation of these schools will very likely result in the United States being excluded from the international reporting, which would result in a loss of $10 million invested by the United States in PISA 2009, a loss in the time invested by participating schools and students, as well as the loss of the comparative data the United States is seeking through the project.

Changes to Incentive Plan

The specific changes proposed by NCES are listed below. The goal in proposing these changes is to be flexible with regards to the needs of the refusing schools in order to encourage them to reconsider participating in PISA 2009 and to enhance comparability in incentives. The proposed changes would enhance comparability in incentives by offering late accepting schools the cash equivalent of the summer conference which was offered to schools that accepted in time for the conference, and by offering school coordinators and students that participate outside school hours stipends that reflect the added opportunity cost of participating outside the school day.

1. Offer refusing original sample schools up to $1,500. This amount is equivalent to the amount that refusing schools would have received if they had participated in the June 2009 PISA Comes to Your School conference, held for the precise purpose of encouraging schools’ participation in the study. In 2006, we also offered refusing schools $1,500 (the amount spent on each school that attended the conference) and were able to get 11 additional original schools to agree to participate, bringing our original schools response rate (and, in turn, final school response rate) above the minimum required. Schools could choose to use these funds in the manner they consider appropriate for supporting their school program and/or the implementation of PISA.
   
   

2. Increase school coordinator and student incentives. As the PISA data collection proceeds, there may also be a great difficulty in obtaining high enough student response rates in individual schools (and, in turn, a sufficient school response rate given that the student response rate must be 50 percent or higher for a school to be considered a participating school). This was the case in 2006 when the incentives for PISA school coordinators and students had to be revised in the midst of data collection. In 2006, in the final days of the administration period, the incentive for school coordinators in schools conducting Saturday assessments was increased to $300 to compensate for additional time spent getting students to attend the testing session, and the incentive for students was increased to $75 to encourage their participation. NCES proposes to offer similar incentives should the student response rate jeopardize our school response rate, as we approach the end of data collection. Requesting a change to incentives in the middle of data collection creates an emergency that is risky with regards to the final outcomes. It would be more prudent to have approval to increase the PISA school coordinators and students incentives in the event that the appropriate student response rate is not being achieved.

In sum, NCES is extremely concerned that the United States will not meet the international standards for response rates. If these are not met, PISA results for the United States will be considered invalid and will not be published. For this reason, as well as interest in improving fairness in the incentive plan by providing all participating schools the same compensation (under the existing plan, currently refusing schools that ultimately agree to participate will not have participated in the conference), NCES is proposing to revise the approved incentive plan to encourage greater school and student participation, as outlined above.

At the conclusion of the data collection period NCES will prepare a report to OMB that explicates which incentives were offered to schools, school coordinators and students (including how many entities received which incentives) and the outcomes in terms of participation in PISA.

Table A2.3 - Response rates

	Initial sample - before school replacement
	(1)	(2)	(3)	(4)	(5)
	Weighted school participation rate before replacement (%)	Weighted number of responding schools (weighted also by enrolment)	Weighted number of schools sampled (responding and non-responding) (weighted also by enrolment)	Number of responding schools (unweighted)	Number of responding and non-responding schools (unweighted)
OECD
Australia	98.40	247 212	251 222	349	356
Austria	98.77	91 471	92 606	197	203
Belgium	81.54	100 785	123 597	236	288
Canada	83.20	348 248	418 565	850	941
Czech Republic	72.87	91 281	125 259	198	264
Denmark	87.24	49 865	57 156	189	218
Finland	100.00	65 086	65 086	155	155
France	96.68	732 366	757 512	179	187
Germany	98.15	932 815	950 350	223	227
Greece	92.51	96 973	104 827	176	192
Hungary	94.70	108 354	114 425	180	189
Iceland	98.35	4 819	4 900	135	151
Ireland	100.00	57 245	57 245	164	164
Italy	90.53	564 533	623 570	753	874
Japan	87.27	1 032 152	1 182 688	171	196
Korea	99.24	572 256	576 637	153	155
Luxembourg	100.00	4 955	4 955	31	31
Mexico	95.46	1 281 867	1 342 898	1 115	1 184
Netherlands	75.70	151 039	199 533	146	194
New Zealand	91.69	54 182	59 090	162	179
Norway	90.47	54 613	60 369	193	213
Poland	95.41	507 651	532 061	209	222
Portugal	94.87	94 835	99 961	165	174
Slovak Republic	92.42	70 860	76 671	170	190
Spain	98.2626	416 539	423 904	682	686
Sweden	99.59	126 611	127 133	197	199
Switzerland	95.44	77 940	81 660	496	512
Turkey	97.16	773 777	796 371	155	160
United Kingdom	76.05	569 438	748 796	439	587
United States	68.95	2 689 741	3 901 131	145	209
Partners
Argentina	95.08	547 775	576 125	168	179
Azerbaijan	94.86	123 718	130 423	163	172
Brazil	98.01	2 300 530	2 347 346	606	629
Bulgaria	98.76	82 248	83 281	178	180
Chile	83.08	207 183	249 370	161	196
Chinese Taipei	98.03	420 165	428 630	235	240
Colombia	93.53	500 567	535 166	154	167
Croatia	98.59	48 081	48 768	159	163
Estonia	98.98	19 071	19 267	167	169
Hong Kong-China	68.57	52 768	76 956	106	156
Indonesia	99.72	2 249 728	2 256 019	349	352
Israel	89.89	95 231	105 941	139	167
Jordan	100.00	99 088	99 088	210	210
Kyrgyzstan	99.58	89 863	90 240	200	201
Latvia	97.57	31 740	32 532	171	175
Liechtenstein	100.00	362	362	12	12
Lithuania	96.85	48 989	50 584	190	197
Macao-China	100.00	6 608	6 608	43	43
Montenegro	94.64	7 363	7 780	49	51
Qatar	98.02	7 260	7 407	128	137
Romania	100.00	231 533	231 533	174	174
Russian Federation	100.00	1 848 221	1 848 221	209	209
Serbia	98.67	76 534	77 568	160	163
Slovenia	97.42	21 983	22 565	355	365
Thailand	97.70	705 353	721 963	208	212
Tunisia	100.00	153 009	153 009	152	152
Uruguay	96.30	38 378	39 854	270	280

Source: OECD (2007). PISA 2006 Science Competencies for Tomorrow's World, Volume I. Paris: Author.

Table A2.3 - Response rates (cont.)

	Final sample - after school replacement
	(6)	(7)	(8)	(9)	(10)
	Weighted school participation rate after replacement (%)	Weighted number of responding schools (weighted also by enrolment)	Weighted number of schools sampled (responding and non-responding) (weighted also by enrolment)	Number of responding schools (unweighted)	Number of responding and non-responding schools (unweighted)
OECD
Australia	98.85	248 321	251 222	350	356
Austria	98.77	91 471	92 606	197	203
Belgium	93.59	115 646	123 563	269	288
Canada	86.23	360 867	418 514	861	941
Czech Republic	93.87	117 526	125 202	244	264
Denmark	96.47	55 068	57 085	209	218
Finland	100.00	65 086	65 086	155	155
France	96.68	732 366	757 512	179	187
Germany	99.05	941 356	950 350	225	227
Greece	99.35	104 124	104 810	189	192
Hungary	100.00	114 266	114 266	189	189
Iceland	98.35	4 819	4 900	135	151
Ireland	100.00	57 245	57 245	164	164
Italy	97.47	607 860	623 619	796	874
Japan	92.38	1 092 616	1 182 688	181	196
Korea	99.89	575 984	576 637	154	155
Luxembourg	100.00	4 955	4 955	31	31
Mexico	96.20	1 291 872	1 342 898	1 128	1 184
Netherlands	94.25	187 953	199 423	183	194
New Zealand	96.06	56 762	59 090	170	179
Norway	95.40	57 582	60 359	203	213
Poland	99.99	532 150	532 197	221	222
Portugal	98.73	98 593	99 863	172	174
Slovak Republic	99.93	76 865	76 920	188	190
Spain	100	424 621	424 621	686	686
Sweden	99.59	126 611	127 133	197	199
Switzerland	99.09	81 345	82 095	509	512
Turkey	100.00	794 826	794 826	160	160
United Kingdom	88.15	660 503	749 270	494	587
United States	79.09	3 085 548	3 901 521	166	209
Partners
Argentina	96.19	554 186	576 125	171	179
Azerbaijan	99.37	129 952	130 775	171	172
Brazil	99.24	2 329 154	2 346 988	617	629
Bulgaria	99.35	82 548	83 092	179	180
Chile	87.89	219 082	249 283	173	196
Chinese Taipei	98.10	420 394	428 529	236	240
Colombia	99.22	530 585	534 764	165	167
Croatia	99.80	48 727	48 823	161	163
Estonia	100.00	19 261	19 261	169	169
Hong Kong-China	93.76	72 564	77 392	146	156
Indonesia	100.00	2 256 019	2 256 019	352	352
Israel	93.45	99 541	106 520	149	167
Jordan	100.00	99 088	99 088	210	210
Kyrgyzstan	100.00	90 240	90 240	201	201
Latvia	100.00	32 532	32 532	175	175
Liechtenstein	100.00	362	362	12	12
Lithuania	100.00	50 584	50 584	197	197
Macao-China	100.00	6 608	6 608	43	43
Montenegro	94.64	7 363	7 780	49	51
Qatar	98.02	7 260	7 407	128	137
Romania	100.00	231 533	231 533	174	174
Russian Federation	100.00	1 848 221	1 848 221	209	209
Serbia	99.96	77 539	77 568	162	163
Slovenia	97.71	22 049	22 565	356	365
Thailand	100.00	721 552	721 552	212	212
Tunisia	100.00	153 009	153 009	152	152
Uruguay	96.30	38 378	39 854	270	280

Source: OECD (2007). PISA 2006 Science Competencies for Tomorrow's World, Volume I. Paris: Author.

Table A2.3 - Response rates (cont.)

	Final sample - students within schools after school replacement
	(11)	(12)	(13)	(14)	(15)
	Weighted student participation rate after replacement (%)	Number of students assessed (weighted)	Number of students sampled (assessed and absent) (weighted)	Number of students assessed (unweighted)	Number of students sampled (assessed and absent) (unweighted)
OECD
Australia	86.30	200 410	232 221	14 071	16 590
Austria	90.81	80 765	88 942	4 925	5 542
Belgium	92.98	107 247	115 343	8 857	9 492
Canada	81.43	258 789	317 822	22 201	26 329
Czech Republic	90.62	110 435	121 869	5 927	6 560
Denmark	89.51	49 249	55 018	4 510	5 035
Finland	92.78	56 954	61 387	4 714	5 082
France	89.78	641 681	714 695	4 684	5 218
Germany	92.26	825 350	894 612	4 884	5 294
Greece	95.24	91 494	96 070	4 871	5 116
Hungary	93.12	98 716	106 010	4 490	4 823
Iceland	83.32	3 781	4 538	3 781	4 538
Ireland	83.75	46 160	55 114	4 585	5 469
Italy	92.30	467 291	506 270	21 753	23 465
Japan	99.55	1 028 039	1 032 727	5 952	5 971
Korea	99.04	570 786	576 314	5 176	5 229
Luxembourg	96.49	4 567	4 733	4 567	4 733
Mexico	96.40	1 101 670	1 142 760	30 885	32 119
Netherlands	90.15	161 900	179 592	4 848	5 375
New Zealand	87.03	44 638	51 291	4 823	5 535
Norway	87.81	50 232	57 205	4 692	5 345
Poland	91.70	473 144	515 945	5 547	6 074
Portugal	86.74	77 053	88 828	5 092	5 862
Slovak Republic	93.19	70 837	76 011	4 729	5 095
Spain	88.48	337 710	381 686	19 604	21 328
Sweden	91.37	115 210	126 095	4 443	4 851
Switzerland	94.94	84 366	88 861	12 191	12 778
Turkey	97.59	649 451	665 477	4 942	5 057
United Kingdom	87.65	565 955	645 688	13 050	15 182
United States	91.00	2 589 680	2 845 841	5 611	6 179
Partners
Argentina	89.31	447 966	501 589	4 297	4 854
Azerbaijan	98.02	119 024	121 433	5 184	5 284
Brazil	90.83	1 692 354	1 863 114	9 246	10 408
Bulgaria	94.47	69 821	73 907	4 498	4 768
Chile	93.72	192 205	205 089	5 233	5 585
Chinese Taipei	97.75	283 168	289 675	8 815	8 988
Colombia	93.89	500 459	533 020	4 478	4 787
Croatia	95.63	44 400	46 431	5 213	5 455
Estonia	94.89	17 708	18 662	4 865	5 119
Hong Kong-China	91.51	64 124	70 071	4 645	5 073
Indonesia	97.81	2 199 184	2 248 313	10 647	10 918
Israel	90.57	79 246	87 498	4 584	5 058
Jordan	96.26	86 890	90 267	6 509	6 791
Kyrgyzstan	97.08	78 319	80 674	5 904	6 074
Latvia	96.66	28 255	29 232	4 719	4 885
Liechtenstein	96.03	339	353	339	353
Lithuania	93.76	47 189	50 329	4 744	5 061
Macao-China	97.57	6 261	6 417	4 760	4 882
Montenegro	93.23	6 821	7 317	4 367	4 681
Qatar	87.34	6 224	7 126	6 224	7 126
Romania	99.83	223 503	223 887	5 118	5 129
Russian Federation	96.02	1 738 842	1 810 856	5 799	6 036
Serbia	93.91	69 375	73 877	4 798	5 112
Slovenia	91.50	18 489	20 206	6 576	7 194
Thailand	98.74	636 028	644 125	6 192	6 266
Tunisia	94.53	130 922	138 491	4 640	4 905
Uruguay	88.24	30 693	34 784	4 779	5 380

Source: OECD (2007). PISA 2006 Science Competencies for Tomorrow's World, Volume I. Paris: Author.

1 In PISA 2006, the original school response rate in the United States was 69 percent – the lowest original school response rate of all the OECD countries, followed by the Czech Republic at 73 percent, the Netherlands at 75 percent, and the United Kingdom at 76 percent. Of the remaining OECD countries, 14 had an original school response rate of between 80 and 95 percent, 9 of above 95 percent but below 100 percent, and 3 of a 100 percent. Including replacement schools, the final school response rate for the United States in PISA 2006 was 79 percent - again the lowest among the OECD countries, which all achieved a final school response rate of above 86 percent (including the United Kingdom at 88 percent). Among the OECD countries, 7 achieved a final school response rate of between 86 and 95 percent, 16 of above 95 percent but below 100 percent, and 6 of 100 percent. See the attached table from the OECD’s PISA 2006 report for all response rates.

2 As of July 14, 2009, the United States has obligated $7.3 million and spent $5 million. By the end of the data collection effort, at least $8.1 million of the $10 million will be spent. By the end of the project in 2011, all $10 million will be spent regardless of the final school response rate.

1