Justification

International Early Learning Study (IELS 2018) Cognitive Items Trial

Volume 1

Supporting Information

OMB 1850-0803 v.190

National Center for Education Statistics

Institute of Education Sciences

U.S. Department of Education

Washington, D.C.

February 2017

Appendixes:

Appendix A: Recruitment Phone Script

Appendix B: Parent Materials

Appendix C: Sample Testing Protocols

Appendix D: Affidavit of Nondisclosure

1 Background and Study Rationale

The International Early Learning Study (IELS), scheduled to be conducted in 2018, is a new study coordinated by the Organization for Economic Cooperation and Development (OECD), an intergovernmental organization of industrialized countries, which is conducted in the United States by the National Center for Education Statistics (NCES). In 2018, in the participating countries, including the United States, the IELS will assess nationally-representative samples of children ages 5.0-5.5 years (in kindergarten in the United States) through direct and indirect measures, and will collect contextual data about their home learning environments, early childhood education and care (ECEC) histories, and demographic characteristics. The IELS will measure young children’s knowledge, skills, and competencies in both cognitive and non-cognitive domains, including language and literacy, mathematics and numeracy, executive function/self-regulation, and social emotional skills. This assessment will take place as children are transitioning to primary school and will provide data on how U.S. children entering kindergarten compare with their international peers on skills deemed important for later success. A request for clearance for recruiting for the IELS field test, to be conducted in fall 2017, has undergone a 60-day public comment period announced in the Federal Register on December 13, 2016 (Vol. 81, No. 239, p. 89923), and will be submitted to OMB for review in mid-February 2017, upon publication in the Federal Register of the 30-day public comment period. A request for the field test data collection will begin a 60-day public comment period in March 2017 and be submitted to OMB for review in May 2017. This request is to conduct a small-scale trial to test new IELS assessment items, by gathering information about their technical qualities in order to ensure that they accurately measure children’s knowledge and skills and are unbiased toward U.S. students.

The procedures, instruments, and schedule for the item trial are provided to countries by the IEA Data Processing and Research Center (DPC) and the Australian Council for Educational Research (ACER), the OECD’s international contractor consortium. Because the IELS is a collaborative effort among many parties, U.S. must adhere to the procedures and schedule set forth by the OECD and its contractors.

2 Design and Recruitment

The purpose of the IELS cognitive item trial is to ascertain how well the direct assessment items assess five-year-old children on their development and skills in four domains (language and literacy, mathematics and numeracy, executive function/self-regulation, and social emotional competency). Additionally, the items will be evaluated for their clarity of language and cultural applicability. Other aspects of the direct assessment will also be examined, such as the functionality of the electronic delivery tablets on which the items are administered, the length of the assessment and the ease of administration. To gather these data, children five years old and starting kindergarten in the fall 2017 will be recruited to participate in a one-on-one direct assessment that will be presented using an electronic tablet. The children recruited will come from both urban and suburban areas, have different racial and ethnic backgrounds, and vary in their early childhood education and care experiences. All study children will be native speakers of English and an equal number of boys and girls will be included.

2.1 Research Questions

For children participating in the cognitive item trial, we will address the following questions:

• Do U.S. children find the proposed assessment items to be clear and easy to understand?

• Are the assessment items culturally applicable to U.S. children?

• How well do the tablets work with U.S. children this age? Are there any issues with using the tablets and running the assessment off the tablets?

• How long do the assessment tasks take for children to complete?

• How well does the assessment engage and interest the study children?

• Are there any unanticipated challenges in administering this assessment to children this age?

Data collected from U.S. children will be combined with data from the other participating countries and analyzed in order to address the above research questions and to evaluate items for bias. The results of these analyses will be used to inform the construction of the assessment instruments to be tested in the field test in fall 2017.

2.2 Procedures

The American Institutes for Research (AIR) will administer the items trial on behalf of NCES. AIR will recruit approximately 30 five-year-old children for participation in the IELS cognitive item trial. Children will be recruited from a database of over 100,000 people in the Washington, DC area that has been cultivated since the mid-1970’s. The database was developed originally through cold-calling and recruiting from mall-intercept locations. The database has continued to grow both through continued cold-calling from directory listings as well as through referrals, word of mouth, and social media. Approximately 400 families in the greater DC metro area will be called and screened for eligibility using a phone script (see appendix A). Parents of participating children will be asked to bring their children into AIR’s offices in Rockville, Maryland, for the assessment session. Upon arriving at AIR, parents will be asked to sign an informed consent form before the child is assessed (see appendix B).

2.3 Session Activities

Each child will participate in a one hour assessment session. During this session, children will work one-on-one with the assessor. The assessor will use a tablet to present items to each study child and to capture responses. Each child will be asked to answer items addressing two of the four IELS domains: language and literacy, mathematics and numeracy, executive function/self-regulation, and social emotional competency.

Each session will begin with a few warm-up items to help the study child become comfortable with the assessment situation. Next, items from two of the four IELS domains will be presented to the child. There will be a short break half way through the session, after the first domain. The assessment activities are designed to be interesting, varied, and fun for five-year-olds (see appendix C).

2.4 Analysis Plans

Children’s responses will be compiled and submitted to IEA DPC for analysis immediately following the item trial. Staff at IEA DPC will compile the data from all participating countries and produce classical item statistics (e.g., item difficulty, discrimination) to evaluate the item characteristics and to identify problematic items. Items that have poor discrimination, demonstrate bias or may vary by cultural experience, or are otherwise not functioning as expected will be revised or considered for elimination from the IELS field test assessment.

3 Consultations Outside NCES

Consultations outside NCES include contractors with expertise in designing and administering direct assessment of this kind. The direct assessment is being developed and coordinated by three organizations:

Australian Council for Educational Research (ACER)

Sacha DeVelle

PO Box 1321 West Perth WA 6872 Australia Tel: +61 8 9235 4820

IEA DPC

Julianne Henke

Überseering 27 22297 Hamburg Germany Tel: +49 40 48 500 500

OECD

Rowena Phair, Secretariat

2, rue André Pascal 75775 Paris Cedex 16 France Tel.: +33 1 45 24 82 004

4 Assurances of Confidentiality

All AIR staff working on the IELS cognitive item trial will sign Affidavits of Nondisclosure (appendix D). Each study child will be assigned a unique study ID. All the children’s responses will be entered and stored under their unique study ID. At no point will the files containing the children’s personal information (e.g., name, address) be linked to their assessment data.

All personally identifiable information will be destroyed once the cognitive item trial is completed. All presentations of data in reports will be in aggregate form, with no links to individuals.

5 Estimates of Burden

Each participating child will be assessed for about one hour. Parents will be asked a short screener during the recruitment process to ensure that their child is eligible for the study. There is no cost to participants. Table 2 provides the burden estimates for each respondent group.

Table 1. Burden estimates for IELS Cognitive Item Trial

Respondent group	Number of respondents*	Number of responses	Hours per respondent	Total burden hours
Parent – Recruitment	400	400	0.5	200
Child ‒ Assessment	30	30	1	30
Total Burden	400	430		230

* Respondent totals do not include duplicative counts of individuals.

6 Estimate of Paying Respondents

As part of recruitment efforts, children and their parents will be given a thank you gift for the time spent participating. Participating children will receive a gift of a sticker set (valued at less than $5) upon completion of the assessment. Parents will receive $50 in cash to thank them for their time and for bringing their child to and from the testing facility. A monetary gift for parents is deemed necessary both to thank individuals for their time and trouble to ensure participation by a representative sample. Parking will be validated.

7 Estimate of Cost to Federal Government

The overall estimated cost to prepare for and conduct the IELS cognitive item trial is approximately $15,390. This cost includes salaried labor for AIR staff and other direct costs associated with both participant recruitment and conducting the one-on-one assessment sessions, as well as the cost of the incentives.

8. Schedule

Table 2. Schedule of activities for IELS Cognitive Item Trial

Tasks	Date ranges
Recruit participants	February – March, 2017
Cognitive items trial sessions	March, 2017
Summary report	April 15, 2017

5