ED's Response to OMB Passback

ICR Questions: Integrated Evaluation of ARRA

Responses from IES/NCEE

1. Does Question #23 in the SEA survey (row 1) include alternative routes to certification?  If it is meant to, then please specify in the instrument.  If not, we recommend asking about alternative routes to certification and whether the State provides meaningful alternative routes to certification.  We also have the same question on the district questionnaire.

Our understanding is that SEAs don’t typically administer alternative certification pathway programs (recruit, train, etc.). Therefore, our initial inclination was to address this topic on the LEA survey. However, we understand that though virtually all states allow for provisional and/or emergency licensure (and so, based on one definition, offer an alternative pathway) there are differences in how meaningfully different these pathways are from traditional programs. Moreover, we know that State-level alternative certification reform criteria accounted for 21 points in the RTT proposal. In turn, on reflection, we agree that this topic should be included in the state survey.

We have modified Question 23 of the SEA survey to better assess alternative pathways in the following ways:

Changed overall row heading to read: SEA strategies to shape educator certification, licensure, and/or educator preparation programs (including university-based and alternative pathway preparation and certification programs)

Added rows as indicated below:

Row 1: Simplified and/or shortened process of obtaining full licensure/certification (e.g., require fewer credit hours):

Row 2: For state university based teacher preparation programs

Row 3: For alternative pathway teacher preparation programs

Row 15: Authorized independent providers (not associated with institutions of higher education) to provide teacher training

The LEA survey addresses the district role in relying on alternative teacher training programs in question 2 (row 6). Specifically we ask about “use of alternative teacher pipelines (e.g., Teach for America, local alternative program) as a source of teacher recruits”.

2. Do the SEA surveys include questions about the use of teacher evaluation systems for teacher placement and dismissal?  If so, please indicate the item number; if not, we recommend adding this (we have noted that you did so to the LEA questionnaire). 

We revised the SEA grids on teacher and principal evaluation to address this issue in items 16 (row 8) and items 18 (row 8). The specific rows are included below:

Question: Indicate whether your SEA included the components below in your statewide system or in your SEA’s standards, guidelines and/or technical assistance for teacher evaluation systems in 2009-2010 and 2010-2011, or plans to include them in 2011-2012.

Row heading: Components Included in Statewide System or Standards, Guidelines and/or Technical Assistance for Teacher Evaluation Systems (or Principal Evaluation Systems)

Q16, Row 8: Use of student achievement gains or growth in making decisions on teacher placement or dismissal

Q18, Row 8: Use of student achievement gains or growth in making decisions on principal placement or dismissal

3. Please provide any letters and all other supplemental materials for all of the data collections for which IES is currently seeking approval under this ICR.

It was an oversight that the letters that will accompany the surveys, in addition to the already approved recruitment letters, were not included in the OMB submission. Please see attached. In addition, the letter being signed by ED program staff to support response rates is attached.

4. “Polls”

a) Why is IES referring to the short web-based surveys as “polls?”  Is IES trying to differentiate a lower standard of quality that will be applied?

We used the term “polls” simply to reflect their similarity to political or marketing collections: (1) small number of question (taking no longer than 10 minutes), and (2) capturing primarily attitudinal information. They are intended to be low-burden, quick turnaround surveys. However, IES intends for the contractor to use the same standards for data collection administration, response follow up, and analysis as used in other part of the ARRA evaluation and in other IES studies.

b) What is the data collection plan and expected response rate for these polls?  They don’t seem to be detailed in Part B.

To minimize cost and time to report out on the data collected, we had intended to administer the polls as web-based surveys. However, we have changed our plans and now propose to administer them as telephone polls, for the following reasons:

• The sample for each is relatively small – 400 to 450 districts – so a web survey may not be as clearly cost effective (e.g., survey programming costs vs. data entry of questionnaires).

• While some number of districts will complete web surveys without prompting, it is likely that we would need to call a significant number of the poll sample to draw them to the web collection; we believe it will be easier to simply complete the surveys on the phone.

• We view the telephone contact as part of a strategy to stay in touch with respondents and to “hear them,” which we hope will ensure high response rates over time. The poll topics are specifically designed to capture district perceptions of their needs for guidance, technical assistance, levels of support – issues that we believe districts would appreciate having ED know about. We expect response rates will be similar, if not better, than response rates to the surveys.

c) Has IES done any pretesting of these “poll” questionnaires, especially in the web format in which they will be used?  If so, please provide any results.  If not, we’d like to see some pretesting, as it appears that in an attempt to appear brief, there is an inordinate amount of material packed into a tabular format which could likely be made cognitively less burdensome with some “unpacking.”

Unfortunately, given that in fall 2010 ED officials were interested in a variety of topics we were not able to pre-test the potential versions. We agree that it would be useful to do so. We also think ED officials may be better able now to predict what information would be useful for at least the fall 2011 poll.

Would OMB consider approving the collection – so that we can begin the baseline survey administration – with a requirement that ED will report back on pre-testing of all 4 poll versions as telephone surveys? We will submit an 83-C for any change in burden based on the pre-testing.

5) We read SS A10 as meaning that IES is planning to offer confidentiality to ALL “individuals and institutions” (including LEAs and SEAs) in the data collection process yet later we see that state level results will be reported.  Please clarify in A10 to which respondent entities confidentiality will be pledged and to which it will not. 

Yes, this was an error on our part. We do intend to present some summary funding and some implementation information by state, because we believe that will be of interest to a broad audience. This has been clarified in A10.

Does OMB want us to revise and upload a new version of the whole package? This will include the changes to SEA and LEA surveys as mentioned above and the change to Part A of the OMB package.

3