OMB83C Change Memo v8

DATE: April 11, 2018

TO: Robert Sivinski, OMB

THROUGH: Kashka Kubzdela, NCES

FROM: Lydia Malley, NCES

Re: International Computer and Information Literacy Study (ICILS 2018) Main Study Incentives Change Request (OMB# 1850-0929 v.8)

The International Computer and Information Literacy Study (ICILS) is a computer-based international assessment of eighth-grade students’ computer and information literacy (CIL) skills that will provide a comparison of U.S. student performance and technology access and use with those of their international peers. ICILS collects data on eighth-grade students’ abilities to collect, manage, evaluate, and share digital information; their understanding of issues related to the safe and responsible use of electronic information; on student access to, use of, and engagement with ICT at school and at home; school environments for teaching and learning CIL; and teacher practices and experiences with ICT. The data collected through ICILS will also provide information about the nature and extent of the possible “digital divide” and has the potential to inform understanding of the relationship between technology skills and experience and student performance in other core subject areas. ICILS is coordinated by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that creates the assessment framework, assessment, and background questionnaires and provides procedures and technical standards which all countries must follow. In the U.S., the National Center for Education Statistics (NCES) conducts ICILS. In preparation for the ICILS 2018 main study, NCES conducted a field test from May through June 2017 to evaluate new assessment items and background questions, to ensure practices that promote low exclusion rates, and to ensure that classroom and student sampling procedures proposed for the main study are successful. Recruitment for the main study began in May of 2017. The request to conduct the ICILS main study data collection in the United States from March through May 2018 was approved in December 2017 with a change request that provided the final versions of the questionnaires approved in January 2018 (OMB# 1850-0929 v.6-7). This request is to implement the strategy used in other international studies of a second-tier school incentive plus a new second-tier teacher incentive designed to help meet the response rates required for inclusion in international comparisons.

ICILS 2018 data collection will end in six weeks, on May 25th, 2018. The current school and teacher participation rates are not on target to meeting the levels required for inclusion in international comparisons:

International Response Rate Requirements for a country’s data to be included in international reporting:

Category 1: Satisfactory sampling participation rate without the use of replacement schools.

The U.S. does not meet category 1 because we have less than 85% of original schools participating.

Category 2: Satisfactory sampling participation rate only when replacement schools were included. A country will be placed in this category if:

1. It fails to meet the requirements for Category 1 but has either an un-weighted or weighted original school response rate without replacement of at least 50% (after rounding to the nearest percent)

The U.S. meets this requirement with an unweighted 69% of original schools participating (at present).

AND HAS

2. An un-weighted school response rate with replacement of at least 85% (after rounding to the nearest whole percent) AND an un-weighted overall student/teacher response rate (after rounding) of at least 85%

The U.S. is not yet meeting this criterion. Currently, with replacement schools, we have only 245 firmly cooperating schools, which will give us a 71% unweighted response rate with replacements [i.e., 245/343=71% given the denominator of 343, once the 9 ineligible original schools are excluded.]. To meet the required school rate of 85% with replacement schools, we need at least 292 schools, or 47 more schools.

Our current student/teacher response rates are

• Student: 90%

• Teachers: 60% of currently assessed schools

Countries that cannot meet either of these two categories are considered to have “unacceptable sampling response rates” and are usually not included in the tables with results for participating countries.

In order for the United States to be able to determine how its eighth-grade students’ computer and information literacy (CIL) skills and technology access and use compare with those of their international peers, we have to increase school and teacher response rates in these last six weeks of data collection.

To achieve this, we propose to implement a second-tier incentive strategy that offers the remaining 50 schools (see table below) an $800 incentive for participation, as is done successfully in PISA and PIRLS, and to mitigate the teacher response rate by offering teachers in the second-tier schools a $40 incentive for participation. The table below shows the current recruitment status of ICILS 2018 schools and which schools will be offered the second-tier $800 incentive.

ICILS 2018 School Recruitment Status as of April 2^nd, 2018 and Second-Tier Designation

Category of School	Number of Schools		Schools to be Designated Second-Tier and Offered the $800 Incentive
	Initial Sample	Replacement Schools
Total sample drawn	352	85
Never to be contacted	36	5
deemed ineligible (closed, wrong grade levels, etc.)	9	5
not contacted due to state request	1	0
not contacted due to district refusal	26	0
Hard refusal by the school (will not be contacted again)	59	50
Contacted & pending	20	5	25 schools
soft refusal	2	0
nonresponse	18	5
Accepted participation	237	25
the subset of schools that accepted but are not following through	17	0	17 schools
7. New replacement schools to be activated in April 2018	NA	8	8 schools

To reflect the use of second-tier incentives in ICILS 2018 and to recruit for ICILS 2018 in the second-tier schools, the following edits were made in the approved ICILS 2018 clearance documents:

Revisions to Supporting Statement Part A

Incentives: The following text shown in blue font has been added in Section A.9 (Payment or Gifts to Respondents; pp.7-8):

State education agencies will be contacted in April 2017 and school districts in May 2017 to let them know that ICILS 2018 sample includes schools in their jurisdiction and to ask for their support in achieving 100% school participation rate. School recruitment will begin in May 2017 and continue until the end of data collection in May 2018. Data collection will begin in March 2018 and end in May 2018. If by April 2018, a month into the 3-months data collection window, we find that we are not able to secure school cooperation at a level required by IEA for U.S. results to be included in international comparisons, we propose to initiate a second-tier school incentive of $800, as has been effective in PISA and PIRLS. The second-tier incentive would be offered only after all other refusal conversion methods have been exhausted to schools that: (a) have accepted participation but are not following through, (b) are pending or have not responded, and (c) are not final refusals. In addition, if by April 2018, the un-weighted school response rate with replacement schools is below 85%, a sufficient number of additional replacement schools will be selected to meet the minimum response rate requirements for U.S. data to be reported (i.e., an un-weighted school response rate with replacement of at least 85%, after at least 50% of original schools response rate). Given that these newly selected replacement schools will be approached late in the short data collection window, they will be offered the second-tier $800 incentive.

(…)

A check will be mailed to each school in the amount of $200 (or $800) and to each school coordinator in the amount of $100, once the ICILS assessment has been conducted in their schools.

(…)

If teacher response rates are below 85% when second-tier incentives are being activated, in the schools offered the second-tier incentive described above, where teacher recruitment will begin late in the data collection window, we will offer teachers $40 for participation. The 85% teacher response rate follows international guidelines for sufficient response rates for U.S. data to be reported. These teachers will have less time in the school year to complete the survey, given that they will be recruited later than other teachers, and at this point we will need them to participate at a high rate to meet the 85% teacher response rate. Because at the time the second-tier school incentives will be initiated, in April 2018, teachers in the second-tier schools will not have been contacted by their schools to complete the survey, the $40 incentive offer will be the first offer they will receive. As such, although not a randomized experiment, this will allow us to compare response rates under the offer of the $40 vs. the $20 incentive and thus inform the efficacy of potential teacher incentive experiments in future ICILS.

Burden: The following text shown in blue font has been added in Section A.12 (Estimates of Burden; p.10):

Table A.1. Burden estimates for ICILS 2018 Main Study.

Data collection instrument	Sample size	Expected response rate	Number of respondents	Number of responses	Minutes Per respondent	Total burden Hours
School Recruitment (Original Schools)	353	0.85 0.76*	300 270	300 270	90	450 405
School Recruitment (Replacement Schools)	120	0.25*	30	30	90	45
School Coordinator	353	0.85	300	300	240	1,200
District IRB Staff Study Approval	40	1	40	40	120	80
District IRB Panel Study Approval	240	1	240	240	60	240
Student Directions	11,250	0.85	9,562	9,562	15	2,391
Student Assessment	11,250	0.85	9,562	9,562	120	19,124
Student Questionnaire	11,250	0.85	9,562	9,562	30	4,781
School Administrator Questionnaire	300	1	300	300	30	150
Teacher Questionnaire (20 teachers / school)	6000	0.85	5,100	5,100	30	2,550
Total Burden Main Study	--	--	15,842	15,842	--	9,451
Note: Total Burden Requested in this Submission includes the already approved burden associated with ICILS 2018 recruitment and pre-assessment activities, as well as the data collection burden (items in black font). Total student burden does not include the time for the cognitive assessment and its associated instructions (in gray font). Minutes per respondent for School Recruitment include time for the second-tier recruitment effort. * Satisfactory sampling participation rate includes a final unweighted school response rate of at least 50% of original schools and at least 85% of original plus replacement schools, with original school sample as the denominator.

When an originally sampled school decides in final stages of recruitment not to participate, recruitment for a replacement school for that school may begin. Replacement schools are contacted only after an original school’s final refusal, and only in order to meet satisfactory participation rates.

The burden table has been updated to account for replacement schools. Our expected response rate of original schools has declined and we are planning to activate up to 120 replacement schools, now shown in the burden table. These revisions do not affect the total estimated response burden.

Cost to Federal Government: The following text shown in blue font has been added in Section A.14 (Annualized Cost to Federal Government; p.11):

Components with breakdown	Estimated costs
FIELD TEST (2017)
Recruitment	125,000
Preparations (e.g., adapting instruments, sampling)	307,650
Data collection, scoring, and coding	712,643
MAIN STUDY (2018)
Recruitment	149,885
Preparations (e.g., adapting instruments, sampling)	451,217
Second-tier recruitment	31,000
Data collection, scoring, and coding	2,837,231
Current package components	2,837,231 2,868,231
Grand total	$4,583,626 $4,614,626

The increased second-tier school recruitment incentives ($600 more per school for 50 schools plus additional recruitment materials) add $31,000 to the total cost to the federal government for ICILS 2018.

Schedule Plans: The following text shown in blue font has been added in Section A.16 (Plans for Tabulation and Publication; p.11) to reflect greater detail of the ICILS 2018 main study recruitment schedule:

Table A.2. Schedule of Activities for ICILS 2018 Field Test and Main Study.

Dates	Activity
April—December 2016	Prepare data collection manuals, forms, assessment materials, questionnaires
October 2016—December 2016	Contact and gain cooperation of states, districts, and schools for field test
December 2016—March 2017	Select student samples
May/June 2017 ( and through January 2018 for operations testing only)	Collect field test data
July 2017—October 2017	Deliver raw data to international sponsoring organization
August 2017—November 2017	Review field test results
March 2017—May 2018	Prepare for the main study/recruit schools
April 2017	Begin to contact and gain cooperation of state education agencies with sampled schools
May 2017	Begin to contact and gain cooperation of school districts with sampled schools
October 2017	Resume school contacts
March 2018—May 2018	Collect main study data
June 2019	Receive final data files from international sponsors
June 2019—November 2019	Produce report

Revisions to Appendix A (Recruitment Materials)

To reflect the heightened need for their participation and the higher incentives being offered, the School District ICILS Main Study Letter [First-Tier/Second-Tier] (p.4), the School ICILS Main Study Letter [First-Tier/Second-Tier] (p.5-6), and the Teacher Questionnaire ICILS Main Study Letter [First-Tier/Second-Tier] (p.7) were amended with text in maroon font to reflect how the letters that will be sent to: new replacement schools to be activated in April 2018; their respective school districts (if these districts have not been already contacted for other ICILS 2018 schools); and to all teachers in all second-tier schools will differ from the letters sent thus far (during the first-tier of ICILS 2018 recruitment) to school districts, schools, and teachers.

Additionally, we added the Second-Tier School ICILS Main Study Recruitment Letter [to participating schools with no follow-through/to non-participating schools with no final refusal] (p.37) that will be sent to: (a) participating schools with no follow-through (i.e., schools that have accepted participation but are still not following through) and (b) non-participating schools with no final refusal (i.e., schools that are pending, have not responded, or are not final refusals).

The second-tier letters will be sent starting in April 2018 and through the end of ICILS 2018 data collection in May 2018.