ED Response to Public Comment

ED’s Response to the Public Comment

Study of Schools Targeted for Improvement Using Title I Section 1003(g) Funds Provided Under ARRA (Study of School Turnaround) (1850-New)

12/6/2010

Thank you for expressing interest and sharing your perspectives on this important research study. We first wish to point out that, although it may not have been clear in our presentation, we do plan to collect information from students at the high school level via focus groups. However, we do not plan to use student focus groups in our sample of elementary schools since it would be challenging to acquire detailed feedback from students at such a young age.

In regards to selecting students, choosing the brightest and most interested students (i.e., AP students) would almost surely provide a very different perspective than talking to average or struggling students. Given that school reform is primarily targeted towards helping those students most at risk, it seems like having a broader student perspective would be wiser. Our approach to selecting students for the high school focus groups will reflect this underlying philosophy.

In your suggestions you mentioned that “the students themselves carrying out the report would be under the impression that they are the only ones aware of the report.” Perhaps we have misunderstood what you mean by this, but we interpret this as a deceptive practice, which would not be permitted under the ethics code dictated by the Institutional Review Board that oversees human subjects research. All our respondents are entitled to full transparency in how the data are collected, reported, disseminated, and ultimately used.

One of the primary concerns you raised about collecting data from principals and teachers in particular is reporting dishonesty. In our opinion, it is not obvious that there is a strong incentive for principals and teachers to be dishonest in this study. Respondents are guaranteed confidentiality to the fullest extent of the law. Moreover, we are only looking at school level processes related to SIG; we are not monitoring schools or evaluating individuals’ compliance with SIG requirements. Rather, we seek to learn about the ways in which SIG funds are used to facilitate change or the ways in which SIG implementation has proven challenging – with the purpose of improving the program, not blaming schools or school personnel for any failures. In fact we will not even be analyzing student performance data to see if a school succeeded in turning around. In this way, the study’s objective is not to produce findings that would be job-threatening, and so regardless of what their responses are, there will be no direct bearing on the respondents’ job or whether the school receives more SIG money. This is particularly true in the case of the teacher survey which is anonymous and given to a large sample of teachers. It is not clear why they would be dishonest for example, about what they think about their principal’s leadership ability or what they think about the school’s climate given how they know this information will be used.

Of course, it is true that honest reporting of facts/opinions is always a potential concern. Thus, we hedge our risk to some extent by collecting data from a variety of sources. For instance, while a principal may have some incentives to distort, as possibly would a teacher, parent, district superintendent, or student, the nature of the distortions would not be perfectly identical across these different groups of respondents. So we wouldn’t be getting data that were overwhelmingly biased in a specific way. Moreover, we are not just interviewing a large and diverse group of people or conducting surveys, but rather we are also collecting data from existing records and documents to verify factual information, as well as conducting school visits and making our own observations. Our view is that gathering data from multiple sources, documenting multiple perspectives, and synthesizing/triangulating the sources whenever possible is the most informative approach for this study.

We also would like to point out that at the end of the day, school reform necessarily depends on the actions of school administrators, principals, and teachers (not just students or parents). To learn about what has or hasn’t changed, we have to secure the cooperation of and be able to communicate openly with the stakeholders who are directly responsible for the day-to-day activities of the school. If this were an impact evaluation, we of course would want to focus quite closely on student outcomes/opinions – but this is not in the scope of this study (there is a forthcoming study from the Department that looks more carefully at student achievement outcomes for the SIG program). Given the objectives of this study, it simply would be senseless to not collect data from principals and teachers.

Finally, you suggest in your response that “a phone call from a government official is intimidating to most people.” We would like to clarify that this study is not actually conducted by government officials, but by researchers from private, non-governmental organizations. The government sponsors and finances the research but does not actually do the day-to-day activities such as collect data and conduct interviews. One of the reasons for this setup is precisely to ensure that evaluations are as independent, objective, and non-threatening as possible.

We hope these comments collectively have addressed the noteworthy concerns you raised. Thank you once again for sharing your thoughts with us.

Subject: SPER interest group's opinion on The Study of School Turnaround (email rec’d 12/3/10)

I am a student attending Long Beach City College.  I am an interested person in this issue regarding Education.  I represent the Students Promoting Education Reform (SPER) interest group.

I wanted to comment on the Study of School Turnaround.  I think the study in and of itself is a good idea with good intentions.  However the methods of extracting and receiving education as reported in the Federal Register, such as the interviews, seem subject to dishonesty as school principals and teachers are weary of their job security.  I believe that a phone call from a government official is intimidating to most people and can easily cause a lapse in judgment leading to a lie regarding how the money from the SIG's were spent.   I am aware that I have very limited resources to prove that these methods of information extraction are not as good as they should be, however I have a suggestion for a better way to analyze the education system.

As a student who just went through America's Public High School system, I know its pros and cons.  I know what isn't done and what should be done better to better prepare America's youth for taking on our nation's burdens.  And I also know that I would be willing to carry out a request made by the Department of Education asking me to document my experience in the classroom and with staff members.  I firmly believe the best way to improve the Education system, is to listen to what the students themselves think.  

This is why I would propose hand selecting somewhere between 5-10 students from each school that the study is being done in and requesting a thorough report on their opinions of the curriculum and teacher performance.  The selection process could be done by classified nominations of students who appear to be bright and interested.  I would assume most nominations would be of students in AP (Advanced Placement) classes.  That would also be my recommendation for the type of students that would emit the best and most accurate reports.  The students themselves carrying out the report would be under the impression that they are the only ones aware of the report, so that they would not share it with anyone else ensuring the opinions are genuine.  They would sign a confidentiality document to provide the study with security.  

An incentive to carry out the task could be decided by the Department of Education.  Also, any improvements to my proposed idea would be welcome.  This is meant to be a skeleton and not anywhere near a full outline of the study.  I did not want to expend a bunch of energy proposing an idea that is not even guaranteed to get read or even produce a response.  

Thank you for your opportunity for me to express an opinion, even if it's impact is felt by none.