Memorandum United States Department of Education
Institute of Education Sciences
National Center for Education Statistics
DATE: June 17, 2010
TO: Shelly Martinez, OMB
FROM: Patricia Etienne, NCES
THROUGH: Kashka Kubzdela, NCES
RE: Responses to 6-16-10 OMB Passback for NAEP 2011 System Clearance Burden Change Request; OMB# 1850-0790 v.28
OMB passback:
Your email of June 10th indicated that the burden increase "is primarily due to the fact that NAEP will collect data on more students with disabilities (SD) and English language learners (ELL), but instead of using one worksheet per students, school staff will now fill out a single worksheet for a number of SD and ELL students. This leads to a decrease in the total number of respondents/responses, but an increase in total burden hours."
Therefore, we wanted to ask how this change in procedure relates to the difficulties and associated promised pretesting discussed in 2009.
Specifically, we want to be sure that this "solution" is the right one so we want to understand pretesting plans and results and see the revised forms before approving this burden change.
2009 Wave II OMB's Passback Question:
Also, on NAEP, please ask NCES to describe for us what pretesting it did of the revised SD and ELL instructions sheets. We understand that they were designed to address deficiencies with the old way of collecting the data, but the new instructions look pretty daunting to us.
2009 Wave II NCES Response:
These materials were developed in response to numerous complaints from states and schools about the burden of completing questionnaires for each student. Even though, over the years, NCES has reduced the number of questions in the questionnaires, the number of students for which questionnaires must be completed continues to grow, so complaints continue.
Our intent is to pretest the worksheets and accompanying instructions in 2010, at which time the number of students and schools is smaller than the large NAEP years. While these materials have not been pretested in the field yet, they have been vetted by advisory groups including NAEP state coordinators, field directors and a select group of other field staff, who work directly with schools. The response to these materials has been very favorable.
2010 NCES Response:
Results of Pretesting the SD and ELL Worksheets in NAEP 2010
Introduction
Since the early 1990’s, NAEP has asked school staff to complete multi-page questionnaires for students with disabilities (SD) and English language learners (ELL). The information from these questionnaires was used to determine how these special needs students should be included on the NAEP assessment, as well as other qualitative information used for full population estimates. Over the years, the number of SD and ELL students has increased, placing additional burden on school staff to complete these questionnaires. In high minority areas, some schools had to complete an ELL questionnaire for each student sampled for NAEP (60 – 120 students). In 2009, the sampled schools in Los Angeles Unified School district had to complete approximately 4000 ELL questionnaires. In order to maximize the time the NAEP representative spends with the school during the pre-assessment to make the assessment arrangements, including how SD and ELL students should be included on NAEP, the SD and ELL questionnaires were to be completed by the time the NAEP representative conducted the visit.
The questionnaires were sent to schools in large FedEx boxes. Often the school coordinator responsible for getting the questionnaires filled out by school staff did not open the boxes until the NAEP staff arrived for the visit. This meant that either the school coordinator had to get the questionnaires completed while the NAEP representative was at the school or, more frequently the NAEP representative had to contact the school coordinator after the visit to obtain the information for the SD and ELL students. This approach usually required several callbacks or subsequent visits, taking up even more of the school’s time, and increasing the possibility of low response rates because of the perceived burden associated with responding to these questionnaires.
Development of the SD and ELL Worksheets
After the 2009 NAEP administration, NAEP began to brainstorm about how to reduce the perceived burden and the actual burden for schools completing the SD and ELL questionnaires. Working with a variety of stakeholders, including NAEP state coordinators, NAEP field staff, and other advisors, the multi-page questionnaires were converted to worksheets. The number of questions was reduced, while the information to make inclusion decisions and the research questions used for the full population estimates were retained. Instead of sending out one questionnaire per student, 10 students could be printed on a worksheet. Instead of sending out 100 ELL questionnaires to a school in Los Angeles Unified School District in 2 large FedEx boxes, 100 students could be printed on 10 worksheets and sent to the school in a medium FedEx envelope.
Along with the worksheets, NAEP developed a concise instruction brochure to assist the schools in completing the worksheets and the states provided state-specific information on how NAEP accommodations aligned with accommodations provided on the state assessments.
Results
Overall, the feedback from the states, schools, and field staff was that the SD and ELL Worksheets were a big improvement over the questionnaires. State Coordinators reported that the worksheets were “much improved and more user-friendly than the SD and ELL questionnaires used in previous assessments.”1 In response to the question, “Please comment on the Student with Disabilities Worksheet and the English Language Learner Worksheet, including content, ease of completion, and other aspects of the worksheets,” 26 of 30 state coordinators made positive comments about the worksheets, including:
A great and much appreciated improvement over the questionnaire.
A tremendous improvement for the schools and field staff; much more accurate and time efficient.
Again, the design has been improved from prior formats.
All feedback was very positive about the use of these new worksheets.
Content Appropriate.
User Friendly.
Directions much easier and less paperwork.
Ease of completion much better and received no negative comments from SC or school staff.
Ease of completion was excellent. It was concise and much more effective than ever before.
Every school loved it better than the old version.
Everyone liked that they were one page.
Experienced coordinators were very positive about this change.
I received many comments from the SCs that the worksheets were easy to complete. For those who previously had to complete the past individual student questionnaires they definitely agreed this format was much less labor intensive and a welcome change. I did not receive any feedback on the instruction sheets for these.
Much better than the cumbersome questionnaires.
Much improved! Schools liked them better than before.
The experienced School Coordinators liked the worksheets much more that the booklets of the past.
The worksheet was very user friendly.
The worksheet were well received in my state and made the task easier for the SCs and SVs.
The worksheets simplified the process for the schools considerably.
These 2-sided documents are much better than the booklets, though the booklets served a valuable intermediary purpose for design and decision-making. The one form would not have been possible or as easily developed without the booklet prototype.
They are user friendly.
This made the process much smoother.
This was a wonderful improvement! Thank you very much. It will especially "bear fruit" during next year's "big NAEP." In 2009, there were schools that had as many as 140 separate accommodations booklets to complete. This year's form was an excellent reform.
What an improvement this year!! From my standpoint as a NSC I did not hear one complaint this year about the amount of paperwork to fill out regarding SD/ELL accommodation assignments.
Worksheet was a MAJOR improvement, success...
The worksheet was much better than the booklet with skip patterns.
Nearly ¾ of the field staff who responded to debriefing questions about the SD and ELL Worksheets reported that the worksheets were completed accurately in 75 – 100% of their schools when they arrived for the preassessment visit and that the schools had no problems completing the worksheets. 63% reported that they had to contact none of their schools after the preassessment visit to obtain missing information from the worksheets. This was a marked improvement from the years when the SD and ELL questionnaires were used.
An analysis of the 2009 and 2010 data for SD and ELL students also points to a higher completion rate in 2010 using the worksheets. In 2010 the completion rate for SD and ELL was 98-99% while in 2009 the completion rate was 96%. For instances where a student is identified as both SD and ELL the improvement rate was in the 12-13% range (79-82% in 2009 and 92-95% in 2010).
1 NAEP 2010 Data Collection Activities Debriefing and Needs Assessment Summary Report, prepared by the NAEP State Service Center
| File Type | application/msword |
| Author | #Administrator |
| Last Modified By | #Administrator |
| File Modified | 2010-06-17 |
| File Created | 2010-06-17 |