Purpose of Grantee Screening Variables

Appendix H

Purposes of Variables in
Grantee Screening Protocol
for Modules A through D

Purposes of Variables in the Grantee Screening Protocol

Page No.		Item No.		Text of Item		Description of Use		Topic		Research Questions
										F-1: Eligible Conversion Magnet Schools, Comparison Schools		F-2: Consistent Test Data		F-3: District Capacity		F-4: Applicant Lottery Data		E-4: Program Evolution
B-1	A-1a		School Name				MSAP magnet school
B-1	A-1b		School has a whole school magnet program (not a program-within-a-school).		Determine magnet school eligibility—whole school program		MSAP magnet school		X
B-1	A-1c		Grades served (2006-07)		Determine magnet school eligibility—elementary grades		MSAP magnet school		X
B-1	A-1d		The school had no magnet program (federal or local) between 1998 and 2004).		Determine magnet school eligibility—new program		MSAP magnet school		X
B-1	A-1e		The school has been open continuously since at least 2001-2002.		Determine magnet school eligibility—operated 3 years before MSAP grant		MSAP magnet school		X
B-1	A-1f		The school has a clearly defined neighborhood attendance area (i.e., it does not overlap with another school’s attendance area).		Determine magnet school eligibility—clearly defined attendance area		MSAP magnet school		X
B-1	A-1g		Attendance area boundaries have not changed since 2001-2002.		Determine magnet school eligibility—stable attendance area preferable (Interrupted time-series analysis assumes stability of resident population pre/post conversion to magnet status. Assess potential for selection bias.)		MSAP magnet school attendance area		X
B-2	A-1h		If boundaries have changed: (1) Approximately what percent of students who were eligible to attend the school before the change are no longer eligible? (2) Approximately what percent of students currently eligible to attend the school were not eligible before the change?		Determine magnet school eligibility—stable attendance area preferable (Interrupted time-series analysis assumes stability of resident population pre/post conversion to magnet status. Assess potential for selection bias.)		MSAP magnet school attendance area		X
B-2	A-1i		Are all neighborhood students admitted to the school, or do they apply for admission for a limited number of seats?		Determine magnet school eligibility—limitations on resident student enrollments (Confirm assumption of interrupted time-series approach that attendance area residents are not selected for admission to the conversion magnet school. Assess potential for selection bias.)		MSAP magnet school enrollment composition		X						X
B-3	A-2a		School Name
B-3	A-2b		What is the approximate proportion of the school’s enrollment that is from the school’s attendance area?		Description of treatment and comparison groups—resident vs. non-resident composition		MSAP magnet school enrollment composition		X						X
B-3	A-2c		In what grades can non-resident students apply for admission?		Determine lottery study feasibility—identify non-resident student entry grades (If most students enter in K, the proportion of students with standardized achievement test data will be small.)		MSAP magnet school non-resident students								X
B-3	A-2d		In what grades are students typically accepted into the magnet?		Lottery study feasibility—identify non-resident student entry grades (If most students enter in K, the proportion of students with standardized achievement test data will be small.)		MSAP magnet school non-resident students								X
B-3	A-2e		Was the magnet program implemented in all grades in 2004-2005? If not, how were the grades phased in?		Describe magnet evolution; verify full implementation of “treatment” during all three years of the grant.		MSAP magnet school evolution										X
B-3	A-2f		Has this school been identified for Program Improvement under NCLB? If so, in what year of the process was the school in 2006-2007?		Inform magnet-comparison school matches—policy initiatives in school (Program improvement is an important contextual factor that cannot be inferred from publicly available demographic and assessment score data files.)		MSAP magnet school		X
B-3	A-2g		Will this school continue as a magnet beyond 2006-2007? If so, how will the program be funded?		Magnet evolution		MSAP magnet school evolution										X
B-4	A-3a		School Name				Potential comparison school		X
B-4	A-3b		Grades served (2006-2007)		Determine comparison school eligibility—elementary grades		Potential comparison school		X
B-4	A-3c		The school has had no magnet program (federal or local) since 1998.		Determine comparison school eligibility—no magnet program		Potential comparison school		X
B-4	A-3d		The school has been open since at least 2001-2002.		Determine comparison school eligibility—operated 3 years before the MSAP grant.		Potential comparison school		X
B-4	A-3e		The school has a clearly defined neighborhood attendance area.		Determine comparison school eligibility—clearly defined attendance area.		Potential comparison school		X
B-4	A-3f		Attendance area boundaries have not changed since 2001-2002.		Determine comparison school eligibility—stable attendance area preferable (Interrupted time-series analysis assumes stability of resident population pre/post conversion to magnet status. Assess potential for selection bias.)		Potential comparison school attendance area		X
B-4	A-3g		If boundaries have changed: (1) Approximately what percent of students who were eligible to attend the school before the change are no longer eligible? (2) Approximately what percent of students currently eligible to attend the school were not eligible before the change?		Determine comparison school eligibility—stable attendance area preferable (Interrupted time-series analysis assumes stability of resident population pre/post conversion to magnet status. Assess potential for selection bias.)		Potential comparison school attendance area		X
B-5	A-3h		What is the approximate proportion of the school’s enrollment that is from attendance area?		Description of treatment and comparison groups—resident vs. non-resident composition		Potential comparison school enrollment composition		X
B-5	A-3i		Has this school been identified for Program Improvement under NCLB? If so, in what year of the process was the school in 2006-2007?		Inform magnet-comparison school matches—policy initiatives in school (Program improvement is an important contextual factor that cannot be inferred from publicly available demographic and assessment score data files.)		Potential comparison school		X
B-5	A-3j		Does this school have any programs designed to attract non-resident students to enroll? (Please specify.)		Inform magnet-comparison school matching—special programs in non-magnet schools.		Potential comparison school		X
B-5	A-3k		Please give any reasons why this school would not be a good comparison school for these MSAP magnets (e.g., school will be closed, relocated, or become a magnet soon; has a low proportion of English learners, while the magnet has a high proportion; etc.)		Inform magnet-comparison school matching—avoid introducing unanticipated differences between schools.		Potential comparison school		X
C-1	B-1		Beginning with the test you use now and going back to the 2001-2002 school year, what standardized English language arts/reading and mathematics assessments have been administered to the students in grades K-8 in this district? Please give the name and version/form of each test and the years that it was administered. (Include statewide achievement tests and any other test administered district-wide). Please be sure to mention any tests administered to grades K-2.		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-2 C-4	B-2 B-3		For each of the English language arts/reading [mathematics] tests listed above, please provide the following information:		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-2 C-4	B-2a B-3a		Test name		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-2 C-4	B-2b B-3b		Administration time (fall or spring)		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-2 C-4	B-2c B-3c		Grades tested each year:		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-2 C-4	B-2d B-3d		Are all students at a grade level tested?		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-2 C-4	B-2e B-3e		If not, approximately what percent of students take the test, and what students do not take the test?		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-2 C-4	B-2f B-3f		Test type (norm-referenced or criterion referenced?)		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C3 C-5	B-2g B-3g		In what metrics are the scores available? National percentile rank Scale scores National percentile rank Performance levels (e.g., basic, proficient, advanced) Other (e.g., grade equivalents, normal score equivalents, raw scores?) Please specify:		Determine district eligibility—assess characteristics of standardized tests administered between 2001-2002 and 2006-2007.		Standardized assessments in use				X
C-6	B-4		For each of the tests listed above, are student-level data available, and if so in what formats could they be provided for research purposes? By formats we mean (1) whether the data are in paper or electronic form; and (2) if data are in electronic form, what is the data format (e.g., Excel; SAS; SPSS; one or many flat files (ASCII); or relational databases, such as Access or DBase)? If formats are different for different years, please specify what formats are available in what years.		Determine district eligibility—assess district capacity to provide usable data sets and identify software needed to process the data sets.		Format of standardized assessment data				X
C-7	B-5		Please describe the student ID codes your district uses.1		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics						X
C-7	B-5a		When were ID codes instituted in this district?		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics						X
C-7	B-5b		When are students assigned their ID codes?		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics				X		X
C-7	B-5c		Does each student have only one ID code, or are there different ID codes for different types of information (e.g., test scores vs. demographic information)?		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics				X		X
C-8	B-6		For which tests do the score files include a student ID code that allows test scores to be linked to: (a) Other information about a student in the same school year (e.g., demographic information such as race, gender, and free lunch status; address; and neighborhood attendance area); (b) The student’s scores from previous years?		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics				X		X
C-9	B-7		Do you know if there are any studies that provide a crosswalk between scores on the older and newer tests? (If yes—Who did the study and can you provide us with a copy; If no—Can you tell us who to contact at the state department of education to find this information?)		Determine district eligibility—identify crosswalk studies if district’s achievement tests are not consistent across time.		Standardized assessments in use				X
C-9	B-8		Do you anticipate that the current district-wide tests for English language arts and mathematics will be replaced soon? (If yes, interviewer prompts respondent for details.)		Determine district eligibility—identify districts with changing assessments.		Standardized assessments in use				X
D-1	C-1		Which of the following additional types of public school choice (beyond federally funded magnet schools) are available to elementary school students in this district?		Assess potential for substitution bias. (Assess the likelihood that lottery losers will attend their attendance area school vs. another special school.)		Non-resident applicants (feasibility of lottery study)								X
D-1	C-2		If a student wants to attend an elementary magnet school that is not in his or her local attendance area, how does he or she apply?		Assess centralization of lottery records		Non-resident applicants (feasibility of lottery study)								X
D-1	C-3		Is there a date by which the student must apply for admission to the elementary magnet school(s)?		Assess feasibility of regression discontinuity design for study of non-resident students (lottery procedures).		Student selection procedures								X
D-2	C-3a		If a student applies later than this date, is he or she placed later on a waiting list for admission? If so, are students admitted from this waiting list only after all the students who applied before the deadline are admitted?		Assess feasibility of regression discontinuity design for study of non-resident students (lottery procedures).		Student selection procedures								X
D-2	C-4		Can the student apply to multiple magnet schools?		Assess complexity of district choice system (participation in multiple lotteries)		Student selection procedures								X
D-2	C-4a		Must the student rank order the magnet schools to which he or she applied?		Assess complexity of district choice system (participation in multiple lotteries)		Student selection procedures								X
D-2	C-4b		Does the student then get preferences of some sort at his or her higher ranked schools?		Assess complexity of district choice system (participation in multiple lotteries)		Student selection procedures								X
D-3	C-5		Which of the following best describes the admissions process at elementary magnet schools in this district?		Assess centralization of lottery procedures.		Student selection procedures								X
D-3	C-6		Which of the following best describes who decides on the number of students who can be admitted to each magnet school and when this decision is made:		Assess centralization of lottery procedures.		Student selection procedures								X
D-4	C-7		Because there may sometimes be more students who apply to a school than there are slots, each district must decide how to admit students. Does the district use a lottery, random drawing, or a similar randomized method to decide upon the order in which applicants will be admitted?		Confirm use of randomizing procedures (lotteries) in selection of magnet applicants.		Student selection procedures								X
D-4	C-8		What method(s) other than lotteries are used to decide the order in which students will be admitted?		Assess incidence of “true” lotteries in the district.		Student selection procedures								X
D-4	C-9a		Does the district itself conduct this randomized ordering, or do individual magnet schools do this?		Assess centralization of lottery records.		Student selection procedures								X
D-4	C-9b		Which best describes whether and how records of the randomized admission order are kept after each year’s admissions decisions:		Assess centralization of lottery records.		Student selection procedures								X
D-4	C-9c		How is this information on applications and lotteries maintained? Who oversees it? Is it digital? How long is it kept?		Assess centralization, quality, and accessibility of lottery records.		Student selection procedures								X
D-5	C-9d		Do magnet schools in your district make separate admission decisions for each grade level?		Estimate total number of true lotteries in district.		Student selection procedures								X
D-5	C-9e		Are applicants to a particular grade in magnet elementary schools in your district divided into groups before students are randomly assigned to admission rankings?		Estimate total number of “true” lotteries in district.		Student selection procedures								X
D-5	C-9f		For each MSAP-funded elementary school, please describe the way in which applicants to a particular grade are prioritized. For example, you might say “Siblings are given top priority, followed by students who do not have siblings in the school, with preference given to students from certain geographic areas of the city before others, and finally, inter-district transfers.”		Estimate total number of “true” lotteries in district.		Student selection procedures								X
D-6	C-10		Please answer the following for each of the MSAP-funded elementary magnet schools. In 2006-2007, how many non-resident students in each grade (1) applied to the school; (2) were admitted and enrolled in the school; and (3) were admitted but did not enroll in the school?		Estimate number of lottery winners and losers. Estimate proportion of lottery winners who did not enroll in the magnet. (Assess difference between “intent to treat” and “effect of treatment” samples.)		Student selection procedures								X
D-6	C-11		How did these numbers change between 2004-05 and 2006-07? (That is, for each school, between the first and last year of the grant, did the number of applicants increase or decrease? How did the number of admissions for applicants change? How did the rate of no-shows change? Did the grades at which most students applied/were admitted change over time?)		Estimate total number of lotteries during three grant years.		Student selection procedures, program evolution								X
D-7	C-10 (alt.)		For the following MSAP-funded elementary magnet schools in 2006-2007, which of the following best describes the number of applications compared to the number of spaces available.		Estimate incidence of true lotteries in district.		Student selection procedures								X
D-7	C-11 (alt.)		How did these numbers change between 2004-05 and 2006-07? (That is, did the availability of slots for applicants increase or decrease over the 3 years of the grant, and by how much?)												X		X
D-8	C-12		Please describe the principal methods your district uses to recruit applicants for these magnet schools (e.g., newspapers, presentations at other elementary schools, internet site).		Contextual variable (describe programs, evolution of programs)		Student recruitment										X
D-8	C-13		Are recruitment materials and activities targeted to families in particular neighborhoods or with particular characteristics, or are they targeted to all families in the district? (Do recruitment methods differ by school?)		Contextual variable (describe programs, evolution of programs)		Student recruitment										X
D-8	C-14		Over the course of the past 3 years, has the district changed its recruitment strategies for any of the MSAP elementary schools? If yes, please describe.		Contextual variable (describe programs, evolution of programs)		Student recruitment										X
E-1	D-1		Please describe the student ID codes your district uses.2		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics						X
E-1	D-1a		When were ID codes instituted in this district?		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics						X
E-1	D-1b		When are students assigned their ID codes?		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics				X		X
E-1	D-1c		Does each student have only one ID code, or are there different ID codes for different types of information (e.g., test scores vs. demographic information)?		Determine district eligibility—assess linkability of student data through consistent ID codes across files and years.		Student identifier characteristics				X		X
E-2	D-2		For each type of information listed below, please indicate the years for which the information is available as well as the source from which it can be obtained (for instance, central student data system; student test data files other than main data system; district lottery database; average daily attendance file).		Determine district eligibility—confirm availability of multiple years of student-level data elements from district databases		Student records data						X
E-2	D-2a		Age/birthdate		Determine district eligibility—confirm availability of multiple years of student		Student demographic variables						X
E-2	D-2b		Gender		Determine district eligibility—confirm availability of multiple years of student data		Student demographic variables						X
E-2	D-2c		Race/ethnic group		Determine district eligibility—confirm availability of multiple years of student data		Student demographic variables						X
E-2	D-2d		Poverty indicators (e.g., free lunch status, parental education)—Please specify:		Determine district eligibility—confirm availability of multiple years of student data		Student demographic variables						X
E-2	D-2e		English language status/level		Determine district eligibility—confirm availability of multiple years of student data		Student demographic variables						X
E-2	D-2f		Home language		Determine district eligibility—confirm availability of multiple years of student data		Student demographic variables						X
E-2	D-2g		Special education status (e.g., IEP)		Determine district eligibility—confirm availability of multiple years of student data		Student demographic variables						X
E-2	D-2h		Eligibility for Title 1 services		Determine district eligibility—confirm availability of multiple years of student data		Student demographic variables						X
E-2	D-2i		School name/code		Determine district eligibility—confirm availability of multiple years of student data		Student attendance information						X
E-2	D-2j		School address		Determine district eligibility—confirm availability of multiple years of student data		Student attendance information						X
E-2	D-2k		Classroom assignment (e.g., teacher name or code number)		Determine district eligibility—confirm availability of multiple years of student data		Student classroom assignment information						X
E-2	D-2l		Date(s) enrolled in school		Determine district eligibility—confirm availability of multiple years of student data		Student attendance information						X
E-2	D-2m		Attendance/absence days		Determine district eligibility—confirm availability of multiple years of student data		Student attendance						X
E-2	D-2n		Student’s “home” attendance area and a mapping of those attendance areas		Determine district eligibility—confirm availability of multiple years of student data		Student residence information						X		X
E-2	D-2o		If no attendance area indicator: Is there any other indicator of residence, such as the student’s block or block group?		Determine district eligibility—confirm availability of multiple years of student data		Student residence information						X		X
E-3	D-2p		Indicator that the student is attending a school outside his attendance area		Determine district eligibility—confirm availability of multiple years of student data		Student residence information						X		X
E-3	D-2q		Indicator of why the student is enrolled in a school outside his attendance area (e.g., open enrollment choice, magnet)		Determine district eligibility—confirm availability of multiple years of student data		Student residence information						X		X
E-3	D-2r		Student’s zip code		Determine district eligibility—confirm availability of multiple years of student data		Student residence information						X		X
E-3	D-2s		Scores on English language arts and mathematics tests administered district-wide		Determine district eligibility—confirm availability of multiple years of student data		Student achievement variables				X		X
E-3	D-2t		To what out-of-attendance area school(s) the child applied		Determine district eligibility—confirm availability of multiple years of student data		Student lottery information						X		X
E-3	D-2u		Into which out-of-attendance area school(s) the child was admitted		Determine district eligibility—confirm availability of multiple years of student data		Student lottery information						X		X
E-4	D-3		When we went through the data elements, it appeared that these data are stored in ____ electronic data systems (and in paper records kept by (__________). How would the district combine information from different files for the same student?		Determine district eligibility--confirm linkability of data across files.		District data system characteristics (linkability)						X				X
E-4	D-4		In what format can data be extracted from each source for research purposes (e.g., flat (ASCII) file; Excel file; SAS; SPSS; relational database, such as Access or DBase)? Can the district provide a data dictionary?		Determine district eligibility—assess district capacity to provide usable data sets and identify software needed to process the data sets.		District data system characteristics (formats)						X
E-4	D-5		Are all of the data back to 2001-2002 readily accessible, or are data for some years archived and thus require more effort to retrieve? (If the latter, please explain.)		Determine district eligibility—assess burden to districts of retrieving data and accessibility of early-years of data.		District data system characteristics (accessibility of earlier years of data)						X
E-5	D-6		Does the district maintain information about the classroom to which each student in the school is assigned? (If yes, go to question 7a; If no, go to question 8.)		Determine feasibility of classroom composition analysis.		Aggregate classroom characteristics						X
E-5	D-7a		Please describe the type of data (e.g., classroom or teacher assigned to student IDs). Has the system for storing this information been the same since 2001-2002? (If no, please explain.)		Determine feasibility of classroom composition analysis.		Aggregate classroom characteristics						X
E-5	D-7b		How available are these data (e.g., part of the student record data or kept in separate files)?		Determine feasibility of classroom composition analysis.		Aggregate classroom characteristics						X
E-5	D-7c		For what years, from 2001-2002 forward, are these data available?		Determine feasibility of classroom composition analysis.		Aggregate classroom characteristics						X
E-6	D-8		Does the district maintain summary information about the composition of each classroom for each of the characteristics listed below? (If yes, go to question 8a; If no, the respondent is finished with the survey.)		Determine feasibility of classroom composition analysis.		Aggregate classroom characteristics						X
E-6	D-8a		If yes, for which school years from 2001-2002 to the present?										X
E-6	D-8b		How is this information maintained and linked?										X

1 The request for information about student IDs appears in both Module B (Questions about Academic Achievement Tests) and Module D (Questions about Data Management Systems). If the interviewer obtains a complete answer from the first respondent, the question will be skipped in the interview with the second respondent.

2 The request for information about student IDs appears in both Module B (Questions about Academic Achievement Tests) and Module D (Questions about Data Management Systems). If the interviewer obtains a complete answer from the first respondent, the question will be skipped in the interview with the second respondent.