Comments and Responses to OMB Comments

Responses to “OMB Comments for Impact Evaluation of Teacher and Leader Evaluation Systems” from December 6, 2012

December 14, 2012

ED is happy to respond to the issues received on December 6, 2012 regarding the Impact Evaluation of Teacher and Leader Evaluation Systems (OMB Control No: 1850-0890; IRC Reference No: 201210-1850-001). Revisions in response to OMB comments are reflected in the two attached surveys. The surveys also contain other minor revisions made in response to pilot testing. The following are responses to OMB’s comments.

Teacher Survey

1.  We appreciate that these questions have been pilot-tested.  However, for some of the more standard questions, did you check with NCES on the formatting, structure, and specific language to use in the question? We only ask since NCES may ask some of these questions in their surveys (SASS, TFS) and their survey items have been through significant rounds of testing and there is probably less chance of misinterpretation.  One such question that comes to mind is Item #1.

ED: In developing the teacher survey, we reviewed all of the items on the Schools and Staffing Survey (SASS) Teacher Questionnaire and the Teacher Follow-up Survey (TFS). TLES survey items 1 and 2 contain identical question stems and response options to SASS items 33a and 33b (2). And TLES survey items 34 and 35 are identical to SASS items 58a and 58b.

There were some SASS items that overlapped with what we intended to collect, but these items were broader than what we needed. As a result, other items are informed by SASS items (see TLES teacher survey items 3-6). For instance, TLES survey item 5 is similar to SASS item 12 except that (1) the TLES survey instructions have been modified slightly to explicitly refer to the study year, and (2) grade levels have been restricted to only those grades served in the study schools.

2. For Items #24 and #29, we recommend adding “Behavior Management” to the list of areas.  Unless you think this is reflected in the current options, we think that this is an area that many teachers would select as an area of improvement.

ED: We believe that this would be a useful subitem to add and we have incorporated it among items 24 and 28. Whereas two of the existing subitems are broadly related to behavior management (i.e., e. student engagement, f. emotional support for students), to some teachers behavior management is associated with such issues as problem behavior and discipline. Therefore the following subitem has been added to both items: “Behavior management (e.g., resolving issues related to problem behavior).”

3. We recommend adding a question that gets at the connection between the feedback a teacher receives and the areas of improvement that a teacher has identified for him/herself.  For example, after Item #24 or #29, you could ask something like: “You have identified these specific areas of improvement because of a) feedback received from observation, b) feedback received based on student growth, c) self-reflection, d) other.”  It would be interesting to determine to what extent teachers are recognizing areas of improvement because of the feedback mechanisms in these evaluation systems. 

ED: From item 24, we will be able to see if there are impacts on the areas identified for improvement. And as the item is currently structured, we will also be able to see if any treatment-control differences could be driven by whether the particular area identified for improvement was discussed with someone who provided feedback on the teacher’s instruction.

We are concerned about the validity of asking a teacher to attribute his or her belief in the need to change in specific areas to a particular component of the intervention, or to self-reflection. For many people it may be the case that they are influenced to believe that they need to improve because of multiple sources of information that may or may not be related to the intervention. We are also concerned about the added burden required to complete such an item, especially given the cognitive load required for a respondent to reliably discern between different components of the intervention and self-reflection, which could be very time consuming.

4. For Item #42, we recommend adding an option that reflects whether a principal creates conditions to give teachers time to collaborate, observe other teachers, and implement strategies to improve their instruction – or something along those lines.

 

ED: We agree that it is important to ask teachers whether they thought that their principal creates conditions for teachers to engage in certain activities. Therefore, we have added two subitems to item 41. The first added subitem (41j) corresponds to principals actively creating opportunities for teachers to collaborate with other teachers; the second subitem (41k) corresponds to principals actively creating opportunities for teachers to observe other teachers’ instruction. A third subitem for “implement[ing] strategies to improve their instruction” was not added because we think that such an item might be duplicative to item 41f which already asks teachers to mark the extent to which they agree/disagree that their principal encourages teachers to implement what they have learned from their professional development.

Principal Survey

5. We recommend adding an “Other” option to Items #8, #29.

ED: We are concerned about adding an “Other” option to item 8 because several of these categories are specific to aspects of the VAL-ED intervention (i.e., 8a, b, c, and e). If we were to add an “Other” option we would be introducing a response outlet that would allow the respondent to invent a new category, as opposed to using the closest, existing category. Therefore if we add this option, our ability to see an impact on these items would be dependent on our ability to recode the “Other” responses correctly, as opposed to having the respondent select from among the existing categories.

Regarding the second item, we agree, and the “Other” option has been added to item 28 because we want to capture a comprehensive list of the different types of feedback that principals provide to teachers.

6. We recommend adding a similar item as we suggested for the Teacher Survey (3) to get at the connection between the feedback a principal receives and the areas of improvement that a principal has identified for him/herself. 

ED: Similar to the response provided to OMB’s comment #3, we are concerned about the validity of asking a principal to attribute his or her belief in the need to change in specific areas to a particular component of the intervention, or to self-reflection. We are also concerned about the added burden that would be required.

2