National Study on Alternate Assessments (NSAA)
Respondents will include one or more individuals from the state departments of education for each state and the District of Columbia. Because the entire universe of states will be included in the state data collection activity, no sampling will be conducted. During the document analysis process, key individuals responsible for the development and implementation of the alternate assessment system in each state and the District of Columbia will be identified.
Contractor will address the research questions to produce the state and national profiles and case study and quantitative reports through a variety of data collection activities. A review of pertinent state documents and a follow-up telephone interview will confirm and fill in information not obtained from document review in the 50 states and the District of Columbia, and provide a map of the current terrain of state policy and practice in alternate assessment of students with significant cognitive disabilities. A description of the technical adequacy of the alternate assessment instruments and procedures states use will be central to the document review process, as will collection and description of the participation and performance of students taking alternate assessments based on alternate achievement standards. Together, the document analyses and telephone interview will provide the raw materials for state profiles, which will use a common template to report the most important features of state policy, assessments, related practices, and available participation and performance data. A cross-state analysis will produce a national profile of the alternate assessment policy arena. In-depth case studies will be conducted in 16 states (3 districts per state and 9 students per district) selected to represent key dimensions of variation identified through existing research and informed by emerging results of the document analyses and telephone interview. Case studies will involve interviews with administrators and members of IEP teams for sample students, focus groups with parents, and administration of a uniform alternate assessment for all sample students. Extant data along with the uniform assessment measure of students in the case study sample will be examined for the quantitative analysis.
Our approach to conducting the telephone interviews is designed to elicit a high response rate and includes a comprehensive notification process to achieve “buy-in” prior to data collection (see description under “Gaining State Cooperation”). Contractors anticipate that the primary respondent for the telephone interview will vary by state and may be an administrator for accountability, assessment, or special education. During the document analysis process, key individuals responsible for the development and implementation of the alternate assessment system in each state and the District of Columbia will be identified. Initial contact will be made with the accountability and assessment directors—those responsible for state standards, assessments, and accountability, as well as for alternate assessments, and the data associated with them. If there are questions that cannot be answered by the initial respondents, interviewers will ask them to identify who the best respondents would be. Interviewers will follow up with additional respondents as needed.
The focus of the telephone interview will be to fill in the information that was not found during the document analysis, to update information on changes to the alternate assessment system, and to identify areas where further changes are planned for the 2006-07 school year. A copy of all the potential items to be addressed in the telephone interview is included in this pacakage. However, each state’s interview will be customized, based on the need for clarification and updates to the summary of findings and the items for which information was not located from the document analysis. The development of the items in this instrument is based on the guidance ED provided to states for submitting peer review materials for approval of state assessment systems under NCLB. Thus, the bulk of information will be obtained from the document review, and the actual telephone interview will not address all items.
During the telephone interview, the interviewer will use the NSAA database system to record responses and note other pertinent information, such as sources of data provided by the respondent. The NSAA database system has the capability to schedule and track multiple data collection activities, in years 1 (2005-06) and 2 (2006-07) (the first phase) for the document analysis and telephone interview components, and in years 3 (2007-08) and 4 (2008-09) (the second phase), for the case studies and quantitative data collection components. The first phase will allow data entry, as well as data analysis and reporting. The framework is organized by state, including the District of Columbia. Each record contains a number of screens linked to that state.
The application resting on this database framework facilitates data entry, analysis, and reporting. The system has the capacity to function in an “online” mode for all data functions (at SRI or PSA when connected to the Internet) and “offline” mode (on laptops at the U.S. Department of Education for document analysis or at case study sites). Data collected in offline mode are uploaded into the online system daily to maintain security and confidentiality of the data. Read-only or read-write access will be given to researchers, depending on their tasks. A comprehensive security system, requiring user name and password login, limits access to data to authorized personnel. The NSAA database system which is maintained through a secure channel (SSL).
Primary functions of the application include
data entry for document review and telephone interview;
contact management (contact information and logs);
project progress updating and reporting;
supporting documentation entry and lookup; and
data tabulation.
Each function is implemented in one or more screens, with links between screens. Data entry functions allow input of data to an individual record or to all records. Contact management functions allow entry of contact information and updating of scheduling linked to an individual state or across states if a global change is implemented to an activity. Progress on data collection can be viewed both within and across states, and reports can be generated. Results of individual responses to questions can be tabulated and reported.
Gaining the cooperation of state education officials can be a formidable task. Our efforts will be guided by three key strategies to achieve adequate participation: (1) an introductory letter on ED letterhead and signed by ED, (2) preparation of high-quality informational materials, and (3) personal contacts with state staff.
U.S. Department of Education Letter. A letter from the Department will be requested that describes the purpose, objectives, and importance of the study, as well as the steps that will be taken to minimize respondent burden. An example of text of such a letter is provided with this submission. In addition, notification materials will include a reference to the Education Department General Administrative Regulations (EDGAR) and Elementary and Secondary Education Act (ESEA) participation requirements, stating that the law requires grantees to cooperate with evaluations of ESEA-supported programs. Section 9306(a)(4) of ESEA requires grantees to cooperate with any evaluation conducted by or for the Secretary. Section 76.591 of the U.S. Department of Education General Administrative Regulations also has a similar requirement.
High-Quality Informational Materials. Preparing relevant, accessible, and persuasive informational materials is critical to gaining cooperation. The primary component of the project’s informational materials will be a letter that addresses the following points:
the study’s purpose;
information about the design of the sample and the schedule for data collection; and
the organizations involved in designing and conducting the study.
Instructions for completing the review of the Data Summary/Interview document will be attached to the document for easy reference.
Contacting States. Potential interview respondents for each state and the District of Columbia will be identified by contacting the state director of the office for accountability and assessment (actual names of the office vary across states)—the individual with oversight responsibility for state standards, assessments, and accountability, as well as for alternate assessments, and the data associated with them. This person may identify others within the state department of education who may also be appropriate to interview. The number of individuals to be interviewed and their titles will vary; in some states, where roles and responsibilities are narrowly defined, more than one individual may need to provide the information that a single individual can provide in states where responsibilities are broader. State-level respondents will be asked to suggest the individuals who can respond to items related to aspects of the interview they are unfamiliar with. Interviewers will follow up with calls to reach identified respondents. Interviewers will contact multiple respondents until all required information is obtained for each state.
One month before the state telephone interview period is to begin, the identified initial respondent will be sent a letter from the Department of Education by first class mail explaining the purpose of the study and the importance of having information about each state’s alternate assessment adequately and accurately represented (see first letter).
Approximately 1 week after the ED mailing, interviewers will telephone the identified respondents to discuss the study, the process of verification of the document analysis findings, and the interview they will be asked to complete (including the estimated burden); request their participation; discuss alternate or additional respondents; discuss possible times and dates for communicating about the document analysis verification process and dates for the interview; and answer any questions they might have. Researchers familiar with the state’s alternate assessment system who have been involved in conducting the document analysis of that state will make the telephone contacts and conduct the interviews. Contractors believe that having these researchers conduct the recruitment calls will prove important in establishing rapport that should maximize both recruitment and interview response rates. Researchers also will be able to obtain names of alternate respondents to replace respondents unable to participate in the interview or additional respondents when multiple respondents are necessary to complete the information needed.
The Data Summary/Interview packet will be sent each the appropriate respondent. The packet will include a cover letter from SRI (see second letter) and the Data Summary/Interview document, with instructions on how to complete the verification process. The Data Summary/Interview document will include the NSAA’s findings for each interview item or an indication that information could not be located when reviewing the state’s peer review documents and department of education website about the state’s alternate assessment based on alternate achievement standards. The Data Summary/Interview packet will be sent in hard copy or electronic form (depending on the respondent’s preference) to the appropriate state respondent(s) for verification. Respondents will indicate whether the findings for 20050-06 are “accurate and complete” or “not accurate and/or not complete,” or “information has changed for 2006-07 school year.” From this information, the interview will be customized for each state, reflecting the questions for which information, clarification, or updating is still needed.
Every effort is being made to minimize the burden on state staff. As described above, analyses of the peer review submissions and state departments of education websites are being used to collect the information that is currently available, and primary data collection will involve data that can not be obtained in a timely manner from any source other than interviews with individuals responsible for the development and implementation of the state’s alternate assessment system. Experience from data collection activities that used a similar approach suggests that when respondents are made aware of the efforts made to obtain information from existing sources in order to reduce respondent burden, they are more willing to participate.
Steps will be taken to maximize response rates. Interviewers will have reviewed the peer review submissions and the state background data available on state department of education websites to be familiar with the state’s alternate assessment system prior to contacting state officials. The interviewer for a particular state will have been one of the researchers conducting the document analyses for that state and have been involved in preparing the summary of findings that will be sent to the interviewee prior to the interview, and therefore will be thoroughly familiar with the available information about the state’s alternate assessment system and aware of the information needed to complete the state profile, as well as areas where future changes and development can be expected. Contractors believe that having these researchers conduct the recruitment calls will prove important in establishing rapport that should maximize both recruitment and interview response rates. The interviewer will make personal contacts with state interviewees to respond to questions, to schedule interviews at times convenient for respondents, and to follow up as needed.
The telephone interview was field tested during fall 2006, during the first OMB comment period. Field test respondents were state-level officials involved in the development of their state’s alternate assessment system. Each interview was conducted according to the procedures outlined in this document. Feedback was requested from each participant. There were fewer than 10 respondents for the field testing; therefore, prior approval from OMB was not required. The results of the field testing have been incorporated into the revised instruments that are part of this final OMB clearance package.
Dr. Jose Blackorby is the Principal Investigator for the study. His mailing address is SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025. Dr. Blackorby also can be reached at 650-859-4210 or via e-mail at jose.blackorby@sri.com.
Dr. Renée Cameto is the Project Director for the study. Her mailing address is SRI International, 333 Ravenswood Avenue, Menlo Park, CA 94025. Dr. Cameto also can be reached at 650-859-6451 or via e-mail at renee.cameto@sri.com.
Page
| File Type | application/msword |
| File Title | EVALUATION OF TITLE I ACCOUNTABILITY SYSTEMS AND SCHOOL IMPROVEMENT EFFORTS (TASSIE) |
| Author | Christine Padilla |
| Last Modified By | david.malouf |
| File Modified | 2007-01-12 |
| File Created | 2007-01-12 |