Response to OMB Comments

memorandum

to: Shelly Wilkie Martinez, OMB

from: Freddie Cross and Kathryn Chandler, NCES

subject: Responses to OMB Comments Received between September 30 and October 13, 2009

BTLS 2009-2012 clearance, OMB# 1850-NEW-v.1

date: October 16, 2009

CC: Kashka Kubzdela, NCES

1. Please consider adding items to the “Reasons for Returning” section that are similar to the items from the “Reasons for Leaving section,” except phrased positively.  Specifically:

Because I was offered a financial incentive to teach at the new school.

NCES response: We will add, “Because I was offered a financial incentive to teach.” We left off “new school” because the respondent may have returned to the original SASS school.

Because I was offered the opportunity to earn additional compensation, benefits, or rewards based on the performance of my students.

NCES response: We will add, “Because some of my compensation, benefits, or rewards are tied to the performance of my students.” so it more closely matches the items indicating reasons for leaving.

Because I was offered a position as a better performing school.

NCES response: We have the item “Because I obtained a position in a school with more desirable characteristics.” We agree that we prefer the OMB wording and will replace the item.

2. Please provide the following documents:

1. Burns et al document referenced on page 10 (in footnote).

NCES response: This is a 392 page document found on the NCES website: http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=200303

2. Cox et al, 2007 document referenced on page 21 of the SS.

NCES response: This is a 251 page document found on the NCES website: http://nces.ed.gov/pubSearch/pubsinfo.asp?pubid=2007349

3. Results of the cognitive interviewing by Census mentioned on page 21.

NCES response: See Attached (FinalMacroBTLSReport(2009))

4. Results of the bias study referenced on page 19.

NCES response: See Attached (New teacher bias analysis appendix_100509).

5. The Systems of Record Notice that covers this collection.

NCES response: The Systems of Record Notice (SORN) that covers this collection is (64 FR 30181-30182) 18-13-01 (copy attached: DeptED 1999 SORN pgs 30181-30186).

3. Please clarify whether responses to the web survey are actually received on an NCES server and if so, whether they are ever “at” the Census Bureau or other location.

NCES response: Data remains on the NCES server until data collection is closed. Once closed, the Census Bureau downloads data daily to its own internal secure server and processes the data before sending the raw and processed data back via secure server to NCES and ESSI. Census will have access to the raw data and ESSI will view the raw data for the purposes of quality assurance reviews. 

 

4. What was the response rate to the TFS for this cohort? 

NCES response: 84.8%

 

5. What is the basis for the 90% response rate estimate for this study? 

NCES response: We hope the incentive, along with emphasizing that the BTLS is about each teacher’s individual career choices over time, and more personal connections with NCES will result in a response rate of 90%.

6. Please clarify whether private school “first year” teachers are part of the universe for this survey.  In Part A, the answer seems to be no but in Part B, it appears more ambiguous.

Only “first year” public school teachers in SASS are included.

7. Will any of the teachers be contacted at their homes? If so, this constitutes a “household survey” and cannot occur during the Census moratorium (http://www.whitehouse.gov/omb/assets/information_and_regulatory_affairs/household_surveys_091608.pdf), which begins on March 1, 2010. We would like to know the size of the survey population that will be contacted at home (eg, the "leavers") and the necessity of collecting information from them during the moratorium, if it is in fact necessary.

The BTLS data collection will begin on January 4, 2010 and NCES will make every effort to secure responses from all subjects with only home contact information before the onset of the Census moratorium in March, 2010.

The total BTLS universe is 2,001 cases, for which NCES has the following contact information:

558 cases with both WORK e-mail and phone

1006 cases with WORK e-mail only

59 cases with WORK phone only

378 cases with no WORK information

NCES will contact by phone only those who have not completed the online questionnaire in response to the original letter and the follow up E-mail. In the TFS data collection, 60 percent of sampled teachers had completed the questionnaire. We expect a similar rate in this collection. Therefore, we expect to contact by phone about 650 subjects at WORK and about 150 at HOME. We will begin to contact these subjects two weeks after the onset of data collection, around mid- January 2010. We expect only a very small portion of these subjects will still need to be contacted by March 2010.

Of the 2,001 total cases, 242 are leavers, for whom NCES has the following contact information:

20 cases with both WORK e-mail and phone

103 cases with WORK e-mail only

3 cases with WORK phone only

116 cases with no WORK information

Therefore, of the approximately 150 non-respondents NCES expects to have to contact at home, about 46 are expected to be leavers.

8. We wanted to let NCES know that we plan to issue a term of clearance requiring NCES to investigate, and report back prior to the next round of data collection, how other longitudinal or panel surveys handle informed consent, and to make appropriate changes. We are concerned that the approach taken here may not be adequate. We have discussed with Marilyn Seastrom, NCES Chief Statistician, who is also chairing an interagency privacy subcommittee, that this analysis might be of interest to the larger group to examine.

In its first paragraph, the study brochure informs the panel members that they will be followed for a decade:

"The Beginning Teacher Longitudinal Study (BTLS) is an important new study of a group of public school teachers who began teaching in 2007 or 2008. The study will follow this cohort of teachers for a decade to create an unfolding “story.” You have been selected to be in this study as a representative of this cohort."

All study subjects will receive a letter informing them that they are in the study sample, followed by another letter and an E-mail with their password. Both letters will be accompanied by the study brochure, which will also be posted on the study’s website (the final version of the brochure is attached with this response).

Additionally, NCES will research informed consent language used in other longitudinal studies to determine the best informed consent language to be used in future waves of BTLS.

9. We reviewed the Burns et al document and found that teachers who were asked to report on 16 or more students (n=15) were provided with a $40 incentive and that no other teacher group was incentivized (pages 42-43). The "16 or more" group did have "one of the highest" (not "the highest") response rates per the report, but there was no range of incentives offered, as suggested in the SS. Rather, the contractor recommended such an incentive plan for the full scale study to follow. We also note that the TFS incentive study did not pay for itself at $10 per teacher, leading the authors to recommend testing a $5 incentive if the web survey option were pursued. Therefore, we are not sure that either of these provides relevant evidence to support an incentive. Please revise the SS accordingly.

10. Related, we do not see a justification for any incentive. Given that this study will continue for some years, we are open to an experiment, particularly given our concern that you are not being up front with respondents about the number of future contacts. If there is not time to propose one for this year, please drop the incentive and considering proposing one for next year.

This response applies to both OMB question 9 and 10. NCES would like to conduct an experiment on the effectiveness of different incentive levels in this current collection wave. The Supporting Statement Part A has been revised as follows:

The Teacher Follow Study achieved an 84.8 percent response rate. Because BTLS is a longitudinal study, and all longitudinal studies experience sample attrition between consecutive collection waves, BTLS has to achieve higher response rate - ideally of at least 90 percent. Achieving and maintaining a higher response rate also minimizes the likelihood of bias in the data collected.

During this wave, NCES proposes to test the effect of providing different levels of a monetary incentive to help motivate participation and improve the response rate, encourage the respondent to complete questionnaires in future waves, and try to re-engage and interview first follow-up non-respondents. NCES proposes to test the effectiveness of employing an advance cash reimbursement of $10 and $20 mailed out in a letter with the respondent’s username, password, and link to the questionnaire to alleviate the burden caused by taking 20 minutes of their limited time.

The cases would be distributed as follows:

Teacher Status	$10 incentive	$20 incentive
Total	997	997

Note that each cell would include 577 or 578 current teachers, 151 or 152 nonresponding teachers, and 88 former teachers.

The test of interest is the $10 incentive group vs. the $20 incentive group for the total sample (n1=997, n2=997).

With an alpha of .05 and a beta of .8, this design is expected to detect a difference of 5.3 percentage points. With this design, the power of this test to detect a 5 percentage point difference with an alpha of .05 is .75. Under the same testing conditions; their power to detect a 5.5 percentage point difference is .84. If the alpha is relaxed to .10, this design is expected to detect a difference of 4.7 percentage points. With this design, the power of the test to detect a 5 percentage point difference is .84. The power to detect a 5.5 percentage point difference is .91.

The total cost of the incentive under this scenario is $29,910; however the marginal cost of the increase in the incentive from $10 to $20 is $9,970. IF this experiment is successful, it will yield more respondents than would be expected with a $10 incentive to all cases, and will provide information that is likely to result in an even larger response rate in the next wave if the $20 incentive is implemented.

11. The contractor report on nonresponse bias has several recommendations including to develop a revised nonresponse bias plan for BTLS.

    1. How is NCES responding to these recommendations?

NCES will adopt all of the recommendations.

2. What is the nonresponse bias plan for BTLS?

NCES adopted the recommendations in the provided New Teacher Bias Analysis Appendix as its non-response bias plan.

3. Please clarify if the 90% is a conditional response rate (based on response to the TFS).

The 90 percent response rate is not conditional on the TFS response rate. NCES will incorporate additional efforts beyond what was employed by TFS in hopes of improving the TFS response rate (e.g. providing a study brochure, using more targeted E-mails, calling previous non-respondents earlier in the study, and testing the effect of incentives).

4