ED Response to OMB Qs

Social and Character Development Research Program

To: Rachel Potter

From: Amy Silverman

CC: Katrina Ingalls, James Hyler, Kim Rudolph

Date: February 7, 2006

Re: IES Response to OMB Comments on Social and Character Development Research Program National Evaluation 1850-0792

Dear Rachel:

Thank you for thoughtful comments on the Social and Character Development data collection package. We have reviewed your comments and responded to each in turn below.

1. We would like to set up a conference call to discuss how the study has been going generally. We are especially interested in any problems you have encountered (if any) and innovative solutions that you have implemented.

Response: Overall, the study has been going well, as we prepare for the third year of data collection activities. I will describe some of the challenges that we have encountered and the innovative solutions that we have implemented. First, the data that was collected via the SACD-Observation Instrument was not reliable (poor inter-rater reliability between independent coders). For this reason, we decided to drop this instrument from data collection during the final year of the project. Second, grantees expressed concern about enough power at the site-level to answer their local complimentary research questions. In response to this concern, data was collected in an additional 12 schools (cohort 2) to increase power at the individual site-level. A third challenge was the quality of the data that was received from the in-person Principal Interview (independent coders were not able to achieve adequate reliability) and the challenge of scheduling in-person interviews with principals at the end of a busy school year. In response to these concerns, the in-person format of the interview will be changed to a telephone format that will be easier to schedule with principals and which will provide better quality of data (see question #8 for more details).

2. Please explain why ED is not planning to follow-up with the second cohort in the spring of 2008 (when they will be in fifth grade).

Response: IES is not planning to follow-up with the second cohort in the spring of 2008 because the grantee’s cooperative agreements end on August 31, 2007. For this reason, IES decided to follow cohort 2 for two years ending in spring 2007 rather than extending the data collection into spring 2008. The main purpose of adding the second cohort of students was to improve power at the site-level, not at the multi-site level where power was adequate.

3. Data collection for this final follow-up is expected to be completed in spring 2007. Please explain why ED believes the collection will extend into the June/ July timeframe (summer 2007).

Response: The collection for this final follow-up will extend into July in order to complete the principal interviews and follow-up phone interviews with primary caregivers who have not returned their Primary Caregiver Reports. The additional time is needed to achieve a high response rate for these instruments. All other data collection will be completed by the end of the school year.

4. Is ED on track to issue the First Impact Report in March?

Response: The Year 1 Multi-program Impact Report has been delayed a few months. It is expected to be released in June 2007.

5. What have the response rates been for the various survey instruments? More specifically, what has the response rate been for the principal interview, which you are requesting to change from an in-person to a telephone interview?

Response: Response rates for the various survey instruments have been quite high. See attached table that describes both year 1 and year 2 response rates for each data collection instrument. The response rate for the principal interview is 100% at each data collection point.

6. Please identify the specific question that you are requesting to add to the Teacher Report on Students.

Response: The questions that were added to the Teacher Report on Students are listed below:

Thinking about the entire school year, was this child…(Response categories – yes or no)

1. Frequently late for school

2. Frequently absent from school

3. Sent to the office for behavior or discipline problems

4. Given an in-school suspension or detention

5. Suspended from school

7. Please explain how the modifications to existing questions will affect the comparability of data across the waves?

Response: The majority of questions (all but 1 question) stayed the same across the five data collection waves. The 1 question that will be dropped will thus have four data collection points rather than five. The five new items that we are proposing for spring 2007 will only be collected once.

8. What evaluations are planned to ensure that there isn’t bias by the switch in modes from an in-person to a telephone interview? Are there evaluations planned to ensure mode consistency across the waves for comparability?

Response: IES decided to switch modes from in-person to a telephone interview because we felt that improving the quality of the data collected was important and outweighed the costs that might be associated with changing modes from an in-person interview to a telephone interview. The items in the principal interview remain the same; it is just the format of the interview that has changed. To date, coders have not been able to achieve adequate reliability with the in-person interview data collected in fall 2004. An examination of the data revealed that the data collected via the in-person interviews was not detailed enough and displayed variability between sites (different data collection teams collected data at different sites). For this reason, we feel that changing to a telephone-format will improve the quality of the data for the following reasons: 1) fewer interviewers will be needed to collect data across sites which will reduce the variability in the data received (the plan is to use 2 or 3 interviewers across all sites), and 2) more consistent follow-up details can be obtained when a smaller number of experienced interviewers is utilized. There are no plans to ensure mode consistency across the waves for comparability. However, we will be sure to report this change in mode in any write-up of the data that reports on change in principal-interview items over time.

Overall, the main focus of the principal interview is to examine treatment and control school differences in SACD-related activities at each time-point (rather than changes over time), which is not influenced by changes in the mode of the interview. We feel that maintaining consistency in modes within each data collection year is more important than changing modes across data collection years.

Thank you again for taking the time to submit your thoughtful comments. If you have any remaining questions or concerns, please do not hesitate to contact me.

Sincerely,

Amy Silverman, Ph.D.

Research Associate

Teaching and Learning Division

National Center for Education Research

cc: Katrina Ingalls

James Hyler

Kim Rudolph

3