Qs and As on NAEP questionnaires

Responses to OMB Questions

Regarding NAEP Wave 1 Clearance Package for the 2009 Assessment

May 30th, 2008

1. What specific testing was done for the new question items provided to OMB?

 The new items for the 2009 NAEP assessment were pilot tested with national samples of students in 2007.

2. Please provide a specific rationale for the following questions as well as what testing was done to ensure validity:

a. When reading a story in class, how often does your teacher ask you to talk about what the characters do and feel?

b. When reading a story in class, how often does your teacher ask you to talk about how you feel about the story?

 Both questions relate to classroom instructional practices that address important components of reading comprehension. The first question relates to the students’ ability to infer a character’s affect as well as action in a story. The second question relates to the student’s connection to the story. Reading research supports the benefit of a reader’s personal connection with a story in comprehension. Both questions were pilot tested prior to their first operational use on NAEP and have been used previously.

3. Did you specifically test these items and did the testing indicate that the italicized concepts were clear to 4th graders?

1. About how many pages a day do you have to read in school and for homework? (Is this "on average?" Can 4th graders answer this accurately?)

 The item is a trend item that appears in the core section at all three grade levels. The same item wording has been used in 2002, 2003, 2005 and 2007 in the fourth-grade questionnaires. The concept is to get at reading per day, or more precisely, reading time per day on average. However, we agree that students might not accurately measure their reading time on average. Thus, the question has the wording of “how many pages a day.” The pattern of student responses revealed in the data (see the table below) shows that fourth-graders can understand this question as well as eighth- and twelfth-graders.

About how many pages a day do you have to read in school and for homework?	Grade 4		Grade 8		Grade 12
	N	Pct	N	Pct	N	Pct
5 or fewer	41,425	21.6	44,917	29.8	8,762	32.9
6 - 10	37,124	19.7	35,084	23.8	6,167	22.4
11 - 15	27,563	14.5	22,623	15.4	4,075	14.4
16-20	28,179	14.6	17,257	11.8	3,138	11.2
More than 20	59,412	29.7	29,577	19.2	5,473	19.2
Non-responses	3,891	0.7	3,455	0.9	232	0.9
Note. Data come from the 2007 Grades 4 and 8 Mathematics (operational) and Grade 12 Writing (pilot) Assessments.

2. How often do you use either the school library or the public library to get information for your own use? (Does this include checking out a book for free/fun reading)

 Yes, this item is intended to ask about students’ library use including checking out books for reading for all purposes. This item taps two constructs: student engagement with reading and availability and use of instructional resources.

3. Do you study or do work for reading at an after-school or tutoring program?

 This item was piloted at grade 4 in 2007. The pattern of fourth-grade responses in relation to eighth-grade responses shown in the table below suggests reasonable responses on the part of fourth-grade students.

Do you study or do work for reading at an after-school or tutoring program? (Grade 4)	Grade 4		Grade 8
	N	Pct	N	Pct
Yes	642	22.0	186	6.9
No	2,094	71.8	2,486	91.6
Non-responses	180	6.2	43	1.6
Note. Data come from the Reading Assessments at grades 4 and 8, which were piloted in 2007.

4. How often does your teacher ask you to read aloud? (Does this mean "in class"?)

5. How often does your teacher ask you to read silently? (Does this mean "in class"?)

6. How often does your teacher ask you to read a book you have chosen yourself? (Does this mean "in class"?)

 Yes, these three questions ask about various instructional strategies that teachers employ in the classroom to help students improve their reading skills. The format of these items (i.e., matrix-format) could be one plausible explanation for a rather high non-response rate for these items in the past, and thus we have changed these items into a single-response format. We believe the pilot testing results do not indicate problems in using these items at grade 4 (see the table below).

How often does your teacher ask you to read aloud?	Grade 4
	N	Pct
Never or hardly ever	557	19.6
Sometimes	1,498	52.7
Almost always	550	19.3
Non-responses	240	8.4
Note. Data come from the Reading Assessment at grade 4, which was piloted in 2007.

How often does your teacher ask you to read silently?	Grade 4
	N	Pct
Never or hardly ever	278	9.8
Sometimes	1,028	36.1
Almost always	1,267	44.5
Non-responses	272	9.6
Note. Data come from the Reading Assessment at grade 4, which was piloted in 2007.

How often does your teacher ask you to read a book you have chosen yourself?	Grade 4
	N	Pct
Never or hardly ever	457	16.1
Sometimes	873	30.7
Almost always	1,156	40.6
Non-responses	359	12.6
Note. Data come from the Reading Assessment at grade 4, which was piloted in 2007.

7. This school year, did your teacher ask you to use a computer to practice spelling and grammar?

 Information from the pilot testing does not indicate this wording will be problematic for fourth-graders. The concern that was expressed from the pilot testing is not to use the matrix-format at grade 4, and thus we have changed this item format into a single-response format.

8. How often do you use e-mail, instant messages, blogs, chat rooms, or text messages to ask your teacher questions about your math homework? (What evidence suggests that 4th graders are using these tools for any purpose? At what prevalence?)

 The pilot testing suggests that about half of fourth-graders use these internet tools more than “sometimes” in order to ask teachers about homework. The format of this item (i.e., matrix-format) could be one plausible explanation for a rather high non-response rate for this item in the past, and thus we have changed this item into a single-response format.

How often do you use e-mail, instant messages, blogs, chat rooms, or text messages to ask your teacher questions about your math homework?	Grade 4
	N	Pct
Never or hardly ever	738	31.0
Sometimes	914	38.4
Every day or almost every day	191	8.0
Non-responses	538	22.6
Note. Data come from the Mathematics Assessment at grade 4, which was piloted in 2007.

4. The following appears to be a typo:

a. 8th grade questionnaire core item 10d - "our" should be "your."

 Using the word “our” in this item was intentional, not a typo. However, to avoid possible ambiguity, we suggest changing this item from “Explain our understanding of what we have read” to “Explain what we have read.”