ED Response to OMB Qs

February 2007

OMB Comments on National Study of Alternate Assessments

1850-NEW

1. The statute specifically discusses alternate achievement standards; will this study only cover the 1% flexibility for students with the most significant cognitive disabilities, or will it include other students with disabilities who take adapted assessments (e.g. students who take alternate assessments based on modified achievement standards under the current subregulatory 2% flexibility)? What does ED believe the statute intended the study to cover?

ED Response: Section 664 (c) of PL 108–446 explicitly refers to a “study on ensuring accountability for students who are held to alternative achievement standards.” ED interprets this to include those students with the most significant cognitive disabilities who can be held to alternate achievement standards under the final regulation published on December 9, 2003. ED has thus focused the study on those students and on alternate assessments based on alternate achievement standards.

2. The first draft of the document analysis report is due to be completed by the end of this month. How much data was ED able to obtain from reviewing existing ED documents and State websites?

ED Response: The due date for the draft report has been extended because of delays in obtaining documents from NCLB peer review submissions for some states. As of February 14, all required peer review and web-based documents had been collected and reviewed for 45 states, and summaries had been prepared for 31 states. It is anticipated that all such documents will be reviewed and summaries will be ready for state verification by March 31.

3. When will ED submit the next OMB package for this study? Will it only cover the case studies or are there other components of the study that it will address?

ED Response: An OMB package covering the case studies is scheduled to be submitted by the end of June 2007.

4. From where is ED obtaining the contact information for the States for the telephone interviews?

ED Response: A listing of state directors of assessment and accountability was obtained from CCSSO.

5. On average, how many people do you expect to have to contact in each State? The burden estimate in ROCIS shows that you expect 2 responses per State, but the supporting statement seemed to imply that there could be multiple contacts within each State.

ED Response: On average 2 per state. The maximum is 4 per state, but this number will rarely if ever be required. We expect most commonly 1 respondent will suffice.

6. The objectives of the project are multi-faceted and there are many tasks that are planned to work together to make it successful. Please elaborate on which tasks address each specific objective.

ED Response: There are three major objectives: (1) to produce state profiles for all 50 states and the District of Columbia, plus a national summary profile; (2) to describe and explicate in a selected sample of states (a) the characteristics of alternate assessments, processes of student placement, alignment with content standards, and uses of data; (b) the state and local processes that facilitate or impede the implementation of alternate assessments, alternate achievement standards, and modified academic achievement standards; and (c) consequences for students with disabilities; and (3) to conduct a quantitative analysis of the relationships between variables in alternate assessment systems and student outcomes. Objective (1) will be accomplished by means of the document analysis and state telephone survey. Objective (2) will be accomplished by means of the case studies. Objective (3) will be accomplished by means of the quantitative analysis.

7. Since this request is specific to the telephone interview survey, please explain the specific analytical techniques that will be used to review, tabulate, and analyze these data.

ED Response: The purpose of this phase of the study is descriptive. Answers to specific items will provide the descriptive data to be presented in the individual state profiles. Some data will be discrete, for example, the type of alternate assessment or the number of performance tasks included in an alternate assessment approach. Other information will be narrative in form, for example, a description of the process used to develop alternate achievement standards, the type of training provided to assessment administrators, the activities conducted by the state to validate technical quality of the alternate assessment. Coding schemes will be developed from the range of narrative data collected to categorize responses to items as appropriate. The national profile will present comparisons across states. Standard statistical techniques will be used for summarizing discrete and recoded data, including frequencies and cross-tabulations, consistent with the descriptive purposes of this phase of the study.

8. The timeline shows that a pilot test occurred for the telephone interview survey. Please provide more information on this pilot test. For example, was it for operational feasibility? For content? How was the feedback incorporated?

ED Response: The pilot test was for operational feasibility. The COR and the technical work group provided input on content, which was then pared to the essential elements. Four states were identified for pilot testing: Massachusetts, Maryland, South Carolina, New York. Only the first three were able to complete the verification and the interview during the comment period. New York is currently verifying the Data Summary/Interview. Data are presented on the following table:

State	Number of persons involved in verification/interview	Total number of hours for verification for all state persons involved	Number of hours for interview
MA	1	2	2
MD	1	4.5	2
SC	2	3.5	2.5 (1 person for 1.5 hours, 2 persons together for 1 hour)

As a result of the greater amount of time reported by respondents to verify the data summary than initially proposed, we increased the average estimated number of hours for verification from 2 to 4 hours. Also, based on feedback from respondents, we revised the format of the Data Summary/Interview document and eliminated the step of sending them a second version prior to the interview as originally proposed.

9. Since the respondent universe will be developed from the state data collection activity and the state data collection activity is scheduled to have been completed in January, please provide the number of individuals that will be contacted for the telephone interview survey in the initial round of contacts.

ED Response: The initial round of contacts will be made to the 51 state directors of assessment and accountability.

10. Please indicate the expected response rate to the telephone interview survey. What was the response rate in the prior studies that used this data collection approach?

ED Response: We are expecting 100% response rates from the states and the District of Columbia. This estimate is consistent with previous studies conducted by SRI, the contractor for NSAA. For example, in both the Special Education Elementary Longitudinal Study (SEELS) and the National Longitudinal Transition Study-2 (NLTS2), every state agreed to support district participation in the study and provided SRI the district level information they were requesting. The Evaluation of Title I Accountability Systems and School Improvement Efforts (TASSIE) interviewed individuals from 48 states and the District of Columbia.

11. Will all states have an opportunity to validate the information compiled in the document review process? Or will only the states that have outstanding questions?

ED Response: All states will review the Data Summary/Interview and decide if the information is accurate for 2005/06 and if changes are expected for 2006/07. They will also see which items SRI was unable to answer from the documents analysis.

12. Are the “in-depth case studies” a component of the telephone interview survey or is this a separate task?

ED Response: A separate task.

13. In instances when the respondent would like the Data Summary/Interview documents sent in hard copy, what measures will be taken to keep this information secure?

ED Response: SRI will send the documents via Federal Express with a signature required, and will provide a pre-paid Federal Express envelope for responses to be returned to SRI.

14. The language in the second letter on the description of the two-phase process is a bit unclear. The language provided in the “Interview Example” (second paragraph) is cleaner and easier to understand.

ED Response: The second letter will be revised in accordance with this comment.