Appendix to ED Response

Assumptions and Activities for the development of the Technological Literacy Assessment

The assessment will be entirely delivered by computer and development and implementation involves a comprehensive range of development activities for background variables and cognitive items as well as systems development.

Framework and Specifications

• The framework was finalized and approved by the Governing Board in March 2010.

• The specifications will be finalized and approved by the Governing Board in May 2010.

Assessment Delivery

• The entire technological literacy assessment will be delivered via computer for both the pilot and probe.

• Fulcrum IT will develop the delivery system based on the system developed for the 2010 and 2011 computer-delivered writing assessment.

• Westat will rent laptops for the assessment administrations.

• The end-to-end delivery system will be available and used in the pilot.

• The Technological Literacy pilot will NOT be spiraled with the 2011 operational Writing assessment.

• The following accommodations will be available for the probe:

• Bilingual (Grades 4 and 8 only), selected blocks only

• Read aloud, selected blocks only

• Text-to-speech

• Magnification for low-vision students

• Braille reader and writer (on a select number of laptops only)

• High contrast display option

• A screen reader will NOT be available.

Assessment Structure and Timing

• Each student will take 2 cognitive blocks, followed by 2 background variable blocks for both the 2011 pilot and the 2012 probe.

• All cognitive blocks will be either 25 or 30 minutes long.

Cognitive Block Parameters for the Pilot:

• Items will be developed and piloted at a 2:1 ratio between pilot and probe.

• A total of 6 blocks will be created, forming 3 test forms. (If both Types A and B items are piloted, there will be a total of 8 blocks in 4 test forms.)

• 500 students will be assessed per block.

• It is possible that theme blocks may be developed for use in the probe.

Item Development activities:

• Background Variable Development:

• Developing overall background variable plans,

• Conducting a systematic review of existing questionnaires, and

• Developing new student, teacher, and school questionnaires for the appropriate grades including any tryouts/cog labs that are needed,

• Uploading items on the Item Tracking and Development System (ITDS).

• Cognitive Item and Scoring Rubric Development:

• Item development

• Student focus groups

• Item tryouts

• Scoring Rubric Development and Collaboration (expanded)

• Ensuring a High Standard of Assessment Materials,

• Translation for bilingual items, and

• Cognitive Item Material Preparation

• Uploading items on the Item Tracking and Development System (ITDS).

• Review of items after pilot for recommendations to carry forward to probe.

• Standing Committees:

• The Background Variables Standing Committee and Technological Literacy Standing Committee will participate in online or mail reviews to evaluate the relevance and importance of items in other NCES-sponsored or supported questionnaires, such as PATT.

• The Background Variables Standing Committee will review new background variables items at a committee meeting prior to piloting.

• The Technological Literacy Standing Committee will hold four item review meetings: two to review pilot items and two to review scenario based tasks and all items for the probe assessment, and two scoring review meetings (one for pilot items and one for probe items). Additional web-based meetings may be scheduled as needed. Face-to-face meetings will be held.

System Development:

• Computer programming of the :

  • System layout

  • System content (including, but not limited to: text, graphics and multimedia functionality, navigation, and data collection)

• System tryouts

• System QC and review by Alliance partners and NCES

• System testing and deployment

Page 3 of 3