OMB-ED Q&A May 2009

OMB Questions: Transition to Teaching

1. Several questions ask the respondent to rank the component they believed was most successful/ useful. This seems impressionistic -- how reliable will this data be?

Grantees will be asked to identify the most successful strategies in recruiting, preparing, supporting, and retaining participants for their projects. Although there may be ways to more accurately capture this information without the reliance on self-report data (i.e., surveying individual project participants, or reviewing applicant profiles), these methods are not feasible for the scope of this evaluation, nor are there sufficient resources to collect original data from project participants. Therefore, self-report data provided by project directors are best suited to the task as grantees have the best understanding of what is working for their individual projects.

By reviewing grantee evaluation reports and APRs from previous years, it’s clear that grantees are proficient at identifying the most successful strategies for their projects, as the project progresses and more successful strategies become the most utilized. In addition, prior to completing these items, grantees are instructed to base their responses to these questions on data from their individual project evaluations, which in some cases include data from project participants.

We plan to verify the project directors’ self-reported survey data by examining how well these measures converge with other available data sources, including grantee evaluation reports and annual performance reports (APRs). (Unfortunately, these other data sources are not reliable for all items and all grantees, which necessitates the grantee survey.) In addition, other survey items will serve as a reliability check. For example, data on successful recruitment strategies can be verified by the number of enrolled participants who were actually recruited through each of the most successful recruitment strategies identified.

2. Will TTT program staff have access to the project specific survey information collected? If so, how do they plan to use it?

Yes, the expectation is that WestEd, the organization that will be collecting the survey data, will provide the TTT program staff with the survey information for each individual project following data collection Staff will use the information to complement the information provided by grantees in their individual performance and interim evaluation reports to better understand the demographics of the participants, effective recruitment strategies, the specific courses delivered, and types and amount of support. This information will also be used as a basis for technical assistance provided to individual grantees. It will help staff better target what areas and what kinds of assistance are needed to ensure an effective project.

In addition to informing TTT program staff of issues regarding project implementation and technical assistance, the TTT office will use survey data collected during the interim and final years to report to Congress on the status of the program.

3. What specific changes or improvements were made to the survey based on the data collected from the FY 2002 grantee cohort?

The TTT program staff and WestEd reviewed the FY 2002 grantee survey and the data provided from the first administration of the survey. It was decided to keep many of the same questions in the FY 2006 survey, such as items relating to participant demographics, recruitment strategies, incentives, coursework, types of support for teachers. In cases where the items did not yield the necessary information, or where the information was deemed unnecessary for later cohorts, the items were dropped. However, in the FY 2006 survey we have clarified the definitions of such terms as participants, teachers of record, and certified teachers to ensure more accurate reporting. We have also added items requesting the LEAs where grantees have actually placed participants and the number in each to gain information about which districts are actually being served. In addition, in the FY 2006 survey, we added items that address mentoring which were not included in the 2002 survey. The goal of the new survey is to collect data on current cohorts, we did not consider replicating the survey items that worked from the 2002 survey as duplicative.

4. How will the survey data be aggregated, since some grantees will be responding once and others twice? Will a distinction be made when aggregating the data based on whether the survey is administered during the project's interim or final year?

Grantee data will initially be aggregated by cohort, which will provide the distinction between data for the interim or final year of the project. In the future, WestEd will consider aggregating data across cohorts to identify best practices for those in the interim year and final year of the project. Further, as the FY 2006 cohort prepares to submit their final performance reports in 2011, some data from the interim survey will be made available to grantees (notably data on the number of participants) so that grantees can simply update these fields. In this way, the data collection burden will be kept to a minimum.