Grantee Screening Interview (Module B)

Appendix C

Grantee Screening Protocol—
Module B: District Assessment Representative Interview

Conversion Magnet Schools Evaluation

Grantee Screening Protocol—
Module B: District Assessment Representative Interview (2004 Grantee Cohort)

April 2007

Introduction

Interviewer: Begin each module with this introduction. If the official has responded to an earlier module, just indicate the goal and expected duration of the module now being administered.

Good morning [or afternoon], I am _____, with [the American Institutes for Research/Berkeley Policy Associates], a research firm in [Palo Alto/Oakland, California]. We are doing preliminary work for a Department of Education research study on the relationship between magnet programs and the academic achievement of the students who attend them. The study will focus on elementary schools that used funds from Magnet Schools Assistance Program (MSAP) grants to introduce new magnet programs beginning in 2004 or 2007. As a first step, we are contacting all of the MSAP grantees with new magnet elementary schools to determine whether they meet certain criteria required for the study. If these interviews show that a scientifically rigorous study is possible, we will begin a major evaluation study late this year.

Today, we need to learn how the school choice system in your district works and how many children gain admission to magnet schools through lotteries.

You have been identified to us by [the MSAP Office/ the magnet project director in this district] as the person who can answer our questions about [topic]. A few days ago, we scheduled this appointment with you and sent you a brochure about the study as well as a copy of the questions we’d like to discuss with you. As indicated in the brochure, the answers you give us are confidential, and responses to the study’s data collections will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district, school, or individual. We will not provide information that identifies respondents, students, or districts to anyone outside the study team, except as required by law. [Interviewer: Verify that the official has received these documents and has them available to read during the interview.]

Your participation in this interview is voluntary, though of course we hope you will be willing to help us in this important evaluation for the Department of Education.

It will take about 30 minutes to go through the questions I have for you. Are you able to talk with me now? Or should we set another appointment?

Module B: Assessment Representative Interview—Questions About Academic Achievement Tests

A major purpose of the Conversion Magnet Evaluation is to examine trends in the academic achievement of students enrolled in magnet schools before and after the schools were converted into magnet schools. This study will require individual students’ standardized test scores in English language arts/reading and mathematics for at least the three years prior to and the three years following the conversion date. The following questions ask you to identify the tests that your district has used over this period and to provide some details about the nature, continuity, and availability of the test data.

Achievement Tests Used in District 2001-2002 to the Present

1. Beginning with the tests you use now and going back to the 2001-2002 school year, what standardized English language arts/reading and mathematics assessments have been administered to the students in grades K-8 in this district? Please give the name and version/form of each test and the years that it was administered. (Include the statewide achievement tests and any other tests administered district-wide.) Please be sure to mention any tests administered to grades K-2.

Test Name and Version	State Test (yes/no)	District Test (yes/no)	English Language Arts (yes/no)	Mathematics (yes/no)	Years Administered (e.g., 1998-2004)
1.
2.
3.
4.
5.
6.
7.
8.

2. For each of the English language arts/reading tests listed above, please provide the following information:

Test Characteristic (ELA)	Test 1	Test 2	Test 3	Test 4
a. Test name
b. Administration time (fall, spring)
c. Grades tested each year:	2001-02:	2001-02:	2001-02:	2001-02:
	2002-03:	2002-03:	2002-03:	2002-03:
	2003-04:	2003-04:	2003-04:	2003-04:
	2004-05:	2004-05:	2004-05:	2004-05:
	2005-06:	2005-06:	2005-06:	2005-06:
	2006-07	2006-07:	2006-07:	2006-07:
	2007-08	2007-08	2007-08	2007-08
d. Are all students at a grade level tested?
e. If not, approximately what percent of students take the test, and what students do not take the test?
f. Test type (norm-referenced or criterion referenced?)1

(continued on next page)

2. For each of the English language arts/reading tests listed above, please provide the following information (continued):

Test Characteristic (ELA)	Test 1	Test 2	Test 3	Test 4
Test name
g. In what metrics are the scores available in individual student records? National percentile rank
Scale scores
National percentile rank
Performance levels (e.g., basic, proficient, advanced)
Other (e.g., grade equivalents, normal score equivalents, raw scores?) Please specify:

3. For each of the mathematics tests listed above, please provide the following information:

Test Characteristic (Mathematics)	Test 1	Test 2	Test 3	Test 4
a. Test name
b. Administration time (fall or spring)
c. Grades tested each year:	2001-02:	2001-02:	2001-02:	2001-02:
	2002-03:	2002-03:	2002-03:	2002-03:
	2003-04:	2003-04:	2003-04:	2003-04:
	2004-05:	2004-05:	2004-05:	2004-05:
	2005-06:	2005-06:	2005-06:	2005-06:
	2006-07	2006-07:	2006-07:	2006-07:
	2007-08	2007-08	2007-08	2007-08
d. Are all students at a grade level tested?
e. If not, approximately what percent of students take the test, and what students do not take the test?
f. Test type (norm-referenced or criterion referenced?)2

3. For each of the mathematics tests listed above, please provide the following information (continued):

Test Characteristic (Mathematics)	Test 1	Test 2	Test 3	Test 4
Test name
g. In what metrics are the scores available in individual student records? National percentile rank
Scale scores
National percentile rank
Performance levels (e.g., basic, proficient, advanced)
Other (e.g., grade equivalents, normal score equivalents, raw scores?) Please specify:

Availability of Data Files

4. For each of the tests listed above, are student-level data available, and if so in what formats could they be provided for research purposes? By formats we mean (1) whether the data are in paper or electronic form; and (2) if data are in electronic form, what is the data format (e.g., Excel; SAS; SPSS, one or many flat files (ASCII); or relational databases such as Access or DBase)? If formats are different for different years, please specify what formats are available in what years.

Interviewer: Be sure to obtain an answer for each test mentioned in response to question 1.

Test Name	Availability	Formats
1.	Years: _________________________ Years: _________________________
2.	Years: _________________________ Years: _________________________
3.
4.
5.
6.
7.
8.

Student ID Codes

5. Please describe the student ID codes your district uses.

1. When were ID codes instituted in this district?

2. When are students assigned their ID codes?

3. Does each student have only one ID code, or are there different ID codes for different types of information (e.g., test scores vs. demographic information)?

Linkability of Data Files

5. For which tests do the score files include a student ID code that allows test scores to be linked to:

1. Other information about a student in the same school year (e.g., demographic information such as race, gender, and free lunch status; address; and neighborhood attendance area); and

2. The student’s scores from previous years?

(Complete just the first row if linkages are the same for all tests.)

Test Name	Linkable to other types of student data within the year (yes/no))	Linkable to student’s scores from previous years (yes/no)
True for all of the tests:
1.
2.
3.
4.
5.
6.
7.
8.

Changes in Tests Used by the District

Interviewer: If the district changed tests between 2001 and 2007, ask the following:

5. Do you know if there are any studies that provide a crosswalk between scores on the older and newer tests?

____ YES—Who did the study, and can you provide us with a copy?

____ NO—Can you tell us who to contact at the state department of education to find this information?

5. Do you anticipate that the current district-wide tests for English language arts or mathematics will be replaced soon?

____ YES

____ NO

Interviewer: If yes, ask for a description of the new test—grades affected, name of test, whether it will be norm-referenced or criterion referenced, and when the test will be administered for the first time.

1 Norm-referenced tests assign numeric scores that rank test takers’ performance relative to each other. Criterion-referenced tests categorize students’ performances relative to absolute standards (for example, basic, proficient, and advanced).

2 Norm-referenced tests assign numeric scores that rank test takers’ performance relative to each other. Criterion-referenced tests categorize students’ performances relative to absolute standards (for example, basic, proficient, and advanced).