Response to OMB Comments

Questions for NAEP: Response to public comment:

1.       Explain what statistical method was used to estimate the interview time.    

Two methods are used to estimate the time it takes to complete survey questions. One is a review of the survey literature in various fields. The results of that review are attached. The review suggests that most Likert question types, which are predominant on the Teacher and School surveys, require approximately 7 to 13 seconds per item. This puts both surveys, which ask up to 45 and 58 questions, respectively, well within the burden guidelines of 20 and 30 minutes, respectively.  The second method is a cognitive lab timing study. Cognitive labs are conducted whenever a new framework is implemented for a survey. The most recent new framework implemented was for the 2011 Writing Assessment. A cognitive lab was conducted for that study in December, 2008. However, timing results were only determined for new items; although all items were new for the student questionnaire, only a few items were new for the Teacher and School questionnaires.

2.       What procedures will be used in FY2009 and future years to validate the accuracy of these estimates?

In the future we will conduct additional cognitive lab studies for a subset of complete Teacher and School Questionnaires administered each year, even when such questionnaires do not reflect new frameworks. The general method will involve having participants first complete the survey for timing purposes, and then have participants go back through the survey question by question, to help clarify ambiguous wording, identify misunderstandings, and serve the primary purpose of the cognitive labs, which is to improve the quality of the survey and the data obtained from it.

Completion Time in Adult Surveys/Questionnaires

• Literature crosses psychology, sociology, politics (particularly polling), and public policy

• Table below shows representative set of studies, where completion time is indicated in the study

• Because target is teacher surveys, the topic search was restricted to adult samples

• Format, administration type, question length, and item readability appear to account for the range of reported values (see also, Fricker, Galesic, Tourangeau, & Yan, 2005, Public Opinion Quarterly, 69, 370-392)

• Studies where response time alone has been recorded, suggest the following: “Questions about facts that are less salient or that require a simple frequency estimate take between 1 and 1.4 sec. Simple attitude questions take between 1.4 and 2 sec., whereas more complex attitude questions take between 2 and 2.6 sec” (Bassili & Fletcher, 1991, Public Opinion Quarterly, 55, 339; see also Johnson, 2004, Political Psychology, 25, 679-702).

Authors	Sample	Sample Item	Response Scale	Format	Number of Items	Mean Time (minutes)	SD Time (minutes)	Average Time Per Item (sec)	Source
Naemi, Kyllonen, & Roberts (2009)	122 ETS Customer Service Representatives	I work hard	6-point: Very true of me to Very Untrue of Me	Web, proctored	127	12.86	3.41	6.10	Unpublished Study
Jackson, Paunonen, & Tremblay	1067 community dwelling volunteers	Personality measure	Likert scale (Not specified)	Paper and pencil, proctored	108	20 (max. time)	--	11.11	Technical Manual for Six Factor Personality Questionnaire
Montag & Reuter (2008)	47 university students	Affect measures (sample items not given)	4-point: Strongly Agree to Strongly Disagree	Paper and pencil, proctored	110	12.94	2.76	7.05	Cyber-psychology and Behavior, 11, 719-721.
Couper, Traugott, & Lamias (2001)	327 university students	For each of these categories, do you approve or disapprove of this aspect of the admissions policy? Applicants from economically disadvantaged backgrounds receive additional points:	6-point: Strongly Approve to Strongly Disapprove	Web, un-proctored	11	2.14	-	11.65	Public Opinion Quarterly, 65, 230-253.
Montag & Reuter (2008)	575 university students	Affect measures (sample items not given)	4-point: Strongly Agree to Strongly Disagree	Web, un-proctored	110	22.48	42.69	12.27	Cyber-psychology and Behavior, 11, 719-721.
Tourangeau, Couper, & Conrad (2004)	2,568 community dwelling respondents	I have to be very ill before I go to the doctor	5-point: Strongly Agree to Strongly Disagree	Web, un-proctored	1			10.8	Public Opinion Quarterly, 68, 368-393.

Questions for NAEP: General pass-back questions

1.       For student questionnaires, prepare a chart showing burden for each of the questionnaire components as in the example table below.

Example:

Questionnaire              Grade  Wave   Subject               Background (burden hours)                 Assessment (burden hours)

                                                                                    Core     Subject Specific    Total                                                           

Civics 4,8,12 1 5 10 15 65 mins. (inc bq and cognitive)

Geography 4,8,12 1 5 10 15 65 mins. (inc bq and cognitive)

History 4,8,12 1 5 10 15 65 mins. (inc bq and cognitive)

Writing 4 1 paper/pencil 5 10 15 65 mins. (inc bq and cognitive)

Writing 8,12 1 computer-deliv. 5 10 15 75 mins. (inc bq and cognitive)

Math 4, 8 1 5 10 15 65 mins. (inc bq and cognitive)

* As part of the Wave 1 supporting statement there is a burden chart with time per respondent for each survey (page 6) and another chart (page 7) that shows the total burden hours for all respondents by grade and by subject.

2.       Can a student be selected to complete more than one questionnaire per year? 

No, a student will only be selected for one subject.

3. What is the maximum annual burden possible per student? 

15 minutes for background questions

50-60 minutes for cognitive sections

---------------------------------------------

65-75 minutes total