Response to OMB Comments

We appreciate the phone call with ED and their responses to our original set of questions.  Here is the last of our passback on the Title I/II Implementation Study. We believe there are three remaining issues:

 

1.      We appreciate IES/ED’s responses to the questions we sent on 11/26 and the discussion on 12/2 to clarify how the research questions and surveys were developed.  However, the collection of data on the uses and targeting of Title I funds is of particular importance to us.  While we understand that it might fall out of the purview of this study, we would like a firm plan in place for collecting these data before providing clearance on this collection. IES Response: The Office of Planning, Evaluation and Policy Development (OPEPD) is working on a plan to address this issue and has been in communication with OMB as to the best way to accomplish the goal of collecting data on the uses of Title I funds.

2.      We understand that information on the uses of Title II funds is already being collected but we’d like to explore the option of adding a question on Title II funds that isn’t necessarily covered by the existing Title II survey and is related to the scope of the study.  We would suggest something along the lines of the item below.  Would it be possible to add such a question (and an analogous question on principals)?

For which of the following activities does your District’s Title II, Part A allocation fund?	Yes/No
a)     Professional development related to implementing [COMMON CORE STATE STANDARDS (CCSS) FOR ELA OR MATH/ CURRENT STATE CONTENT STANDARDS FOR ELA OR MATH]	____________
b)     Professional development on analyzing student assessment data to improve instruction	____________
c)      Professional development activities and supports informed by teacher evaluation systems including planning professional development of individual teachers and development of performance improvement plans for low-performing teachers.	____________
d)     Implementation of any part of a teacher evaluation system:
a.      Building the data infrastructure necessary to evaluate teachers using student achievement data including purchasing or developing data systems	____________
b.     Classroom observations conducted by the principal including training and supports for observers	____________
c.      Classroom observations conducted by someone other than a school administrator including training and supports for observers	____________
d.     Assessments by a peer or mentor teacher	____________
e.     Administration of student or parent surveys	____________
f.       Providing professional development on understanding teacher evaluation systems and resulting feedback	____________
e)      Mechanisms and strategies to help schools recruit and retain highly qualified teachers, principals, and specialists in core academic areas and address any inequities found in teacher quality or effectiveness (e.g., scholarships, loan forgiveness or repayment assistance , signing bonuses, differential pay for effective teachers, external recruitment activities)	____________
f)      Using external provider to prepare, recruit, or supply more effective teachers to high need schools.	____________

 

IES Response: We have added questions 4-40 through 4-42 to the district survey to address whether the district receives Title II, Part A funds, and if so, the uses of funds related to teachers and principals. Questions 4-41 and 4-42 (which address uses of funds related to teachers and principals, respectively) are similar to the question suggested above with a few modifications. Program office guidance materials (http://www2.ed.gov/programs/titleiparta/flexswp091313.pdf) indicate that Title II funds may not be used to purchase evaluation system-related data systems to manage linking student and teacher data, so the response option related to building the data infrastructure necessary for the implementation of evaluation systems has been removed. Some of the other response options have been re-grouped to fit into these content areas: professional development, implementation of evaluation systems, and other activities.

3.      We appreciate that IES conducted a thorough process to reduce the burden of these surveys for respondents.  We recommend adding some additional framing within the Introduction of the State survey to acknowledge the reporting burden placed on States in the coming year and clarify the need for collecting these data on this survey and the effort by the study team to reduce the burden of this survey as much as possible. IES Response: We have added the following text to the introduction of the state survey:

“We recognize the burden placed on states in the coming year. The study team has worked to reduce the burden on this survey as much as possible. The study team wants to reiterate the need for collecting this data.”

Also, with regards to two other items that we’ve gone back-and-forth on, we are ok with their sample augmentation of ELL teachers and the proposed survey item below (transmitted to us on 12/12).

District XX.  Which of the following did the district take into account when selecting the interventions to implement in these schools?

Our district considered:	YES	NO	DK
a.    Guidance or advice from the state education department or a technical assistance center funded by the state......................	1	0	d
b.    A list of vendors approved by the state...................................	1	0	d
c.     Information provided by the intervention’s developer or vendor...................................................................................	1	0	d
d.    Recommendations from colleagues in other school districts....	1	0	d
e.    Information from  a U.S. Department of Education Comprehensive Center	1	0	d
f.     Information from a U.S. Department of Education Regional Educational Laboratory	1	0	d
g.    Information from the What Works Clearinghouse...................	1	0	d
h.    School staff’s interest in specific interventions........................	1	0	d
i.      Parent and/or community input..............................................	1	0	d
j.     Grade level of the school (i.e., elementary, middle, or secondary).............................................................................	1	0	d
k.    Cost of interventions and amount of funding available......	1	0	d
l.      District and/or school capacity to implement the interventions	1	0	d
m.   Something else (SPECIFY).......................................................	1	0	d
(STRING (NUM))

IES Response: This question on the motivation of selecting interventions has been added to the district survey, following questions asking about interventions in various types of schools (e.g., focus and priority schools). The additional questions addressing the motivation of selecting interventions in various types of schools are questions 3-15, 3-20, 3-29, 3-34, 3-50, and 3-55 on the district survey. Some of these questions apply to districts in states with ESEA Flexibility and others to districts in states without ESEA Flexibility, so no single district would have to answer all six questions.

The Office of English Language Acquisition (OELA) has decided that they do not want to augment the sample with additional ELL teachers. The sample is therefore the same as the previous submission of the OMB package: all 50 states and DC; 570 school districts; 1,300 schools; and 9,100 teachers of core academic subjects and special education.

Sent in an email from Alex Hunt on 12/20:

We have one last request regarding this ICR:OMB would like to learn more information about how student growth is used in evaluation systems through the District survey.  Would it be possible for IES to add items to get at this? IES Response: We have added new questions and revised existing questions on the district survey to address the complex issue of weighting of student growth in evaluation systems for teachers (see 4-4 through 4-10 for all questions addressing student achievement growth in teacher evaluations) and principals (see 4-21 and 4-22).

Because not all districts use a simple weighting procedure, questions 4-9 and 4-21 first ask how districts combine student achievement growth with other measures of teacher or principal performance to determine an overall evaluation rating. If districts use a simple weighting procedure, then the survey asks what percentage weight is used for both teachers (question 4-10) and principals (question 4-22). Because districts may weight student achievement growth differently for different types of teachers, question 4-10 focuses on tenured teachers of ELA/math in grades 4-8 (i.e., those teachers most likely to have value added measures (VAM) or student growth percentiles (SGP) scores for their own students) and tenured teachers of core subjects (ELA, math, science, and social studies) where VAM or SGP scores are not calculated for their own students.

There are also some corresponding edits in the state, school, and teacher surveys to ensure that the language aligns with these changes.

In addition, the extant data form (included in the state survey document following the survey) lists information that the study team will collect about state policies from publicly available documents (e.g., state websites). If a state has a policy that addresses how student achievement growth is factored into a teacher’s or principal’s overall performance rating (and if so, what that weight is), we will collect that information on the extant data form (in 4E-2 through 4E-5 and 4E-7 through 4E-9).