Revisions Memo

MEMORANDUM OMB # 1850-0803 v. 210

DATE: September 25, 2017

TO: Kashka Kubzdela

National Center for Education Statistics

FROM: Linda Hamilton

National Center for Education Statistics

SUBJECT: National Assessment of Educational Progress (NAEP) 2018 Assessment Delivery Study - Revised [Revision to a previously approved package (1850-0803 v.203)]

The NAEP 2018 Assessment Delivery Study was approved on September 18, 2017 (OMB #1850-0803, v. 203). This special study is designed to test three different administration models in a sample of schools in Virginia. The purpose of the three different models is to explore their feasibility and to begin to determine the pros and cons of each model. Model 1 will allow NAEP to gain logistical experience with preparing for and using the wide range of devices currently in schools. Model 2 will allow NAEP to make a first step in running the current delivery system (eNAEP) on non-NAEP-provided devices. Model 3 will allow NAEP to explore the possibility of continuing to provide digital devices to schools using a less expensive device than the current tablets NAEP provides. Results from this study will be used to inform future decisions about the NAEP administration model.

Since OMB approval of this study, we learned that during recruitment we will need to contact a larger number of schools to reach the final target of 69 schools to participate in the study. This is because information about the equipment and infrastructure currently existing in Virginia schools that is necessary to support the use of school based equipment for the NAEP 2018 Assessment Delivery Study is not centrally available.

To reflect this reality and the adjustments necessary to the study procedures and materials, we revised the following parts of Volume I and the Appendices (the original, approved text is shown in red font, and the revised text in blue font) followed by an explanation of each revision (the revisions require 60 additional respondents/responses and 25 additional burden hours):

Revisions to Volume 1

1. Cover page:

Updated version number and changed “August 2017” to “Revised September 2017”

2. Section 3: Recruitment and Sample Characteristics

Original text:

A total of 69 schools will participate in the study. Three schools will be recruited to participate in each, round 1 and round 2 testing (which do not include students), and 63 schools will be recruited to participate in round 3 testing (which will include 50 students per school).

Revised Text:

A total of 99 schools will be contacted during recruitment about participating in the study. The schools will take a technology survey (see Volume 2) to gather data about the equipment and infrastructure in their school. Based on the results of the survey, a final sample of 69 schools will be selected to participate in the study (see Appendices F and G). Of the 69 schools selected to participate in the study, 3 schools will be recruited to participate in each, round 1 and round 2 testing (which do not include students), and 63 schools will be recruited to participate in round 3 testing (which will include 50 students per school).

Explanation of revision:

The technology survey (included in Volume 2) has not changed. We were originally planning to use the survey only to prepare for the assessment, but will now also use it to determine which schools meet the minimum equipment and infrastructure specifications to participate in the study.

3. Section 5: Paying respondents:

Original Text:

Schools will receive a $200 gift card to an office supply store (e.g., Staples or Office Depot) to encourage participation and to thank them for their time and effort.

Revised Text:

Schools will be given a $50 gift card to an office supply store (e.g., Staples or Office Depot) for completing the school technology survey. In addition, schools selected to participate in the assessment will receive an additional $150 gift card to encourage participation and to thank them for their time and effort.

Explanation of revision:

We want to offer incentives to schools to encourage them to complete the technology survey, given that it is now necessary for selecting which schools can participate in the full study. The total offered to each participating schools remains the same, it is just split into first $50 and then $150.

4. Section 9: Estimate of Hourly Burden:

A sentence and a table added:

For schools not selected for the study, the school personnel burden is estimated at 30 minutes to complete the technology survey.

Table 2. Estimate of Hourly Burden for Schools that Complete the School Technology Survey but are not Selected to Participate in the Study

Person	Task	Number of Respondents	Number of Responses	Hours per Respondent	Total Burden (in hours)
School administrator	Initial Contact by Westat	30	30	0.33	10
School personnel	Completing School Technology Survey	30	30	0.50	15
Total		60	60	​N/A	25

Note: Numbers have been rounded which may affect totals

Explanation of revision:

These account for the burden in schools that will complete the technology survey but not participate in the study.

Original Table of Total Hourly Burden (the revised and added row are highlighted in yellow)

Table 5. Total Hourly Burden

Person	Task	Number of Respondents	Number of Responses	Hours per Respondent	Total Burden (in hours)
School administrator	Initial Contact by Westat	69	69	0.33	23
School personnel - Model 2 Round 1 & 2	Scheduling, Logistics, and Debriefing Interview	6	6	1.5	9
School personnel - Model 1 & 2 Round 3	Scheduling, Logistics, and Debriefing Interview	42	42	3	126
School Personnel -Model 3 Round 3	Scheduling, Logistics, and Debriefing Interview	21	21	1.5	32
Parents	Initial Notification	3,150	3,150	0.05	158
Parents*	Refusals or Additional Research	60*	60	0.25	15
Students	NAEP Assessment Delivery Study	3,150	3,150	0.58	1,827
Total		6,438	6,498	​ N/A	2,190

Revised Table of Total Hourly Burden

Table 6. Total Hourly Burden

Person	Task	Number of Respondents	Number of Responses	Hours per Respondent	Total Burden (in hours)
School administrator	Initial Contact by Westat	99	99	0.33	33
School personnel in schools not included in final selection	Completing School Technology Survey	30	30	0.50	15
School personnel - Model 2 Round 1 & 2	Scheduling, Logistics, and Debriefing Interview	6	6	1.50	9
School personnel - Model 1 & 2 Round 3	Scheduling, Logistics, and Debriefing Interview	42	42	3.00	126
School Personnel -Model 3 Round 3	Scheduling, Logistics, and Debriefing Interview	21	21	1.50	32
Parents	Initial Notification	3,150	3,150	0.05	158
Parents*	Refusals or Additional Research	60*	60	0.25	15
Students	NAEP Assessment Delivery Study	3,150	3,150	0.58	1,827
Total		6,498	6,558	​ N/A	2,215

* These parents are a subset of those who will be initially notified.

Note: Numbers have been rounded which may affect totals

5. Section 10: Cost to Federal Government:

Original Table of Estimate of Costs (the revised row is highlighted in yellow)

Table 6: Estimate of Costs

Activity	Provider	Estimated Cost
Data collection activities	Westat	$605,000
Development and support of the test delivery system used for Models 2 and 3	Fulcrum	$250,000
Development and support of the test delivery system used for Model 1	Pearson	$900,000
Total		$1,755,000

Revised Table of Estimate of Costs

Table 7: Estimate of Costs

Activity	Provider	Estimated Cost
Data collection activities	Westat	$607,000
Development and support of the test delivery system used for Models 2 and 3	Fulcrum	$250,000
Development and support of the test delivery system used for Model 1	Pearson	$900,000
Total		$1,757,000

Explanation of revision:

The data collection costs increased by $2,000 to account for the additional recruitment of and payment to the schools that complete the technology survey but are not eligible participate in the assessment.

Revisions to Volume 2

No changes were made to Volume 2.

Revisions to Appendices

1. Appendix B: School Notification Letter:

Original third paragraph:

You are being notified of the selection now so the assessment can be included on the school calendar for the 2017-2018 school year. Your scheduled assessment date is Day, Month XX, 2018. Please let me know as soon as possible if this date needs to be revised to an alternate date within the testing window of January 29, 2018, through March 9, 2018. Please note that grade X schools should consider avoiding conflicts with the Virginia English: Writing Standards of Learning grade X assessment when requesting date changes.

Revised third paragraph:

You are being notified of the selection now so the assessment window can be included on the school calendar for the 2017-2018 school year. Your assessment date will be scheduled by the NAEP field representative within the testing window of January 29, 2018 through March 9, 2018. Please note that grade X schools should consider avoiding conflicts with the Virginia English: Writing Standards of Learning grade X assessment when scheduling this assessment.

Explanation of revision:

Given that not all schools will participate in the study, the assessment date will not be set at this point but at a later point (see new Appendix G, copied below).

Original set of bullets:

• provides the school a $200 gift card to an office supply store (e.g., Staples or Office Depot);

• protects the anonymity of the participating students and school staff;

• will not produce individual state, school division, school, or student data;

• is administered on NAEP provided computers OR on school-based equipment using TestNav OR on school-based equipment using eNAEP;

• is administered by trained contractor staff (Westat) who work with the designated school coordinator;

• requires one to two testing sessions of 120 minutes (will be determined in December); and

• is administered to approximately 50 randomly selected grade X students.

Revised set of bullets (revised text is highlighted in yellow):

• provides the school a $50 gift card to an office supply store (e.g., Staples or Office Depot) for completing the technology survey for the NAEP 2018 Assessment Device Study that will be emailed to the school coordinator;

• provides the school a $150 gift card to an office supply store (e.g., Staples or Office Depot) for completing the NAEP 2018 Assessment Device Study;

• protects the anonymity of the participating students and school staff;

• will not produce individual state, school division, school, or student data;

• is administered on NAEP provided computers OR on school-based equipment using TestNav OR on school-based equipment using eNAEP;

• is administered by trained contractor staff (Westat) who work with the designated school coordinator;

• requires one to two testing sessions of 110 minutes (will be determined in December); and

• is administered to approximately 50 randomly selected grade X students.

Explanation of revision:

The payments to the school are divided into the two payments ($50 for completing the technology survey and $150 for participating in the study), to account for the schools that will only be asked to complete the survey.

Added text and attachment:

Please designate a school coordinator that will serve as a liaison for all NAEP activities in your school. Attached is a school coordinator contact information form. Please complete the form and send it to me by Day, Month Day, 2017.

National Assessment of Educational Progress (NAEP) Assessment Delivery Study

School Coordinator Information and Contact Form

Purpose of study: To investigate the feasibility of using school-based equipment versus NAEP provided equipment

The Requirements for the NAEP School Coordinator are that he/she:

• Be familiar with how students participate in statewide assessments;

• Knows how to collect student information, such as birth dates, demographic information;

• Has time in October, December, and January to complete preparation activities; and

• Is available on the day of testing to meet with the NAEP field staff that will administer the assessments to review logistics, show the NAEP field staff testing location(s), get students to the testing locations, debrief with the NAEP field staff, etc.

The designated NAEP School Coordinator will need to:

• Visit the NAEP website at http://nationsreportcard.gov for sample questions, previous results, and other NAEP publications; and

• Beginning in October, work with the NAEP contractor staff to complete assessment planning activities such as completing a technology survey, setting an assessment date, planning for assessment locations, reviewing the selected student list, and indicating how the parents/guardians will be informed about the study, etc.

A representative from Westat, the contractor responsible for administering the assessment, will contact your NAEP School Coordinator in December to discuss preparations for the assessment.

Please indicate your designated school coordinator, his or her position at your school, and his or her contact information for this study:

School:

School Division:

Name:

Position:

Email:

Phone:

Return the form via email or fax to

Dr. Wendy M. Geiger, Virginia NAEP State Coordinator

Email: Wendy.Geiger@doe.virginia.gov

FAX: (804) 371-8978

Please return this form by Month, Day 2017

Explanation of revision:

Provides a clarification about when the school principal should designate the school coordinator and a description of the school coordinator’s role for the study.

2. Appendix F: Notification that School Was Not Selected to Participate in Study Email

New appendix added:

NOTE: This will be sent as an email to the school coordinators of the NAEP Assessment Delivery Study.

<Insert school coordinator name>, <insert school name>, <insert school division name>

Thank you for agreeing to participate in the NAEP 2018 Assessment Delivery Study and completing the technology survey.

Based on the results of the technology survey, your school has not been selected to participate in the NAEP 2018 Assessment Delivery Study. To thank you for your participation to-date, NAEP will send you the $50 gift card to an office supply store (e.g., Staples or Office Depot) soon.

If you have questions, please contact me Wendy.Geiger@doe.virginia.gov. I look forward to working with you and your staff on future administrations of NAEP.

CC: Principal

DDOT, Division Director of Testing

Wendy Geiger, Ed.D.

National Assessment of Educational Progress (NAEP) State Coordinator

Office of Test Administration, Scoring, and Reporting

Division of Student Assessment and School Improvement

Virginia Department of Education

Mailing Address:

P.O. Box 2120

Richmond, VA 23218-2120

Physical Address

101 N. 14th Street

James Monroe Bldg. Floor 18

Richmond, VA 23219

Phone:

(804) 786-1939 Direct Number

(804) 371-8978 Fax

(804) 225-2102 Office Receptionist

Wendy.Geiger@doe.virginia.gov

Explanation of revision:

Given that not all schools will participate in the study, this appendix was added to notify schools that were not selected to participate in the full study.

3. Appendix G: Notification of Assessment Date Email

New appendix added:

NOTE: This will be sent as an email to the school coordinators of the NAEP Assessment Delivery Study.

<Insert school coordinator name>, <insert school name>, <insert school division name>

Thank you for agreeing to participate in the NAEP 2018 Assessment Delivery Study and completing the technology survey.

Your school’s grade (4 or 8 or 12) students will take the assessment for about 110 minutes on Month, Day, 2018. Please place the assessment date on your 2017-2018 school calendar. NAEP representatives will provide significant support to your school, bring necessary materials and equipment, and administer the assessment.

If you have questions or a conflict with the scheduled assessment date, please contact me at Wendy.Geiger@doe.virginia.gov or (804) 786-1939 by Month, Day Year. NAEP representatives will be in touch with you soon with more information about preparing for the assessment.

I look forward to working with you and your staff to ensure the success of the NAEP 2018 administration. Please let me know, if you have any questions or concerns.

CC: Principal

DDOT, Division Director of Testing

Wendy Geiger, Ed.D.

National Assessment of Educational Progress (NAEP) State Coordinator

Office of Test Administration, Scoring, and Reporting

Division of Student Assessment and School Improvement

Virginia Department of Education

Mailing Address:

P.O. Box 2120

Richmond, VA 23218-2120

Physical Address

101 N. 14th Street

James Monroe Bldg. Floor 18

Richmond, VA 23219

Phone:

(804) 786-1939 Direct Number

(804) 371-8978 Fax

(804) 225-2102 Office Receptionist

Wendy.Geiger@doe.virginia.gov

Explanation of revision:

This appendix was added in order to set the assessment date for the schools that will be selected to participate in the study.

Page 11