Ed Response to Second Round of OMB Questions

Response to OMB’s Concerns Regarding Student Consent Incentives for the

Evaluation of Secondary Math Teachers from Two Highly Selective Routes to Alternative Certification

This memo presents our response to OMB’s concerns about incentives offered to students in the Evaluation of Secondary Math Teachers from Two Highly Selective Routes to Alternative Certification (referred to below as the HSAC Study). We understand that OMB is concerned about the $5 incentive paid to students for the return of the parental consent forms in school districts that require active parental consent. This memo describes our concerns about potentially low rates of return of consent forms and our rationale for proposing student incentives. While we would strongly prefer to offer incentives to all students in the study in active consent districts, we would be willing to conduct an experiment on the effectiveness of incentives for obtaining consent if OMB so desires. The memo ends with a discussion of a potential incentive experiment.

A. Concern about Rate of Return of Consent Forms

For the HSAC study, we will collect district math assessment test scores and other school records for middle and high school students and administer a math assessment to high school students only. Of the 16 school districts participating in the study, four require active consent for both administering the math assessments and collecting the district school records, and two require active consent for administering the math assessments only. Hence, in two districts we will need to collect active consent forms in high schools only, and in four districts we will need to collect active consent forms in both middle and high schools.

The six districts that require active consent include some of the largest and, in our experience, the most difficult, school districts from which to obtain parental consent, including New York City, Washington DC, and Chicago. We expect that 45 to 60 percent of the student sample will be from these school districts.

In school districts that require active consent, the rates at which consent forms are returned can be as low as 33 or 40 percent (Lueptow et al. 1977; Thompson 1984; Eaton et al. 2004; Pokorny et al. 2001). And, the characteristics of the student population in the HSAC study make obtaining parental consent particularly challenging. Studies have shown that rates of parental consent are lowest for older youth, minority youth, students in Title I schools, schools with high percentages of students receiving free or reduced price lunches, and students in schools in northeastern cities (Anderman et al. 1995; Kearney et al. 1983; Ebsensen et al. 1999). Nearly all the students in the HSAC study will be in low-income schools, about 50 percent will be in high schools, many students will be minorities, and a substantial proportion will be from large northeastern cities.

A low rate of return of signed parental consent forms will reduce the sample size and our ability to generalize the study findings to the student population. The rate of return of signed consent forms sets the upper limit for the overall response rate so a high consent return rate is critical to achieving a high response rate for the study overall. A low rate of consent will reduce the size of the sample and compromise our ability to produce precise impact estimates. Furthermore, evidence that minorities, low achievers, and children with less educated parents or in less stable families are less likely to return the forms suggests that we may not be able to generalize our findings to all students in the study classes if the rate of return of consent forms is low (Ellickson and Hawes 1989; Esbensen et al. 1999).

B. Approaches to Increase Return of Consent Forms

Given the challenges of obtaining a high rate of return of consent forms and the implications of a low rate of return, we propose to use every tool at our disposal to increase the rate of return of active consent forms. Our approach involves providing incentives as well the following additional strategies for promoting high consent rates:

• Sending students home with a consent form that is clear and easy to read and understand. It will be accompanied by a cover letter that describes the study, stresses the confidentiality of the data we receive, and provides a toll-free number for parents to call to ask questions about the study. If possible, a letter from the school principal supporting the study will be sent along with the consent form. The letter and consent form will be provided in English, Spanish, and other languages, if necessary.

• The school principal will be called before the beginning of the school year to alert him or her to the need to distribute the consent forms. The forms will then be sent via Federal Express to the school, with clear instructions for their distribution.

• The school will be asked to send the parental consent forms in the “first-day” packages distributed to parents.

• Teachers will be asked to collect the signed forms, and the school will be provided with postage-paid Federal Express packages to return the completed forms.

• Schools will be asked if they would be willing to have the students hand address an envelope with the consent form so that the material can be directly mailed to the parent.

• Before the end of the first week of school, we will call the school and the teachers to remind them to encourage students to return the forms. Further calls will be made as needed.

• A member of the evaluator’s data collection team will visit the school to talk personally to the principal, teacher, and class.

• This study member may also attend school events that are frequented by parents, such as back-to-school nights or parent-teacher nights. At these events, study members can talk about the study and directly ask parents to complete the form.

• Additional consent packets will be distributed to students/parents as needed. We will discuss with the school the possibility of sending the consent forms with the students’ report cards or other school materials requiring parent signature (such as class syllabi). However, the school may not be willing to assume this additional burden.

We also proposed to offer a $5 gift to students as an incentive for students to return the consent form. The gift will be given on the return of the signed consent form to the teacher, irrespective of whether the parent provided consent. As compensation for the teacher’s role in ensuring that the students are reminded about the need to return the forms, distributing the forms, and collecting the forms, we had proposed that the class teacher be given $25 to be used for the benefit of the class, such as school supplies or books for the classroom. No student or class incentive will be paid in the districts that require only passive parental consent.

C. Evidence that Incentives Will Result in Higher Consent Rates

The most direct evidence of the efficacy of student incentives is provided by an experimental study of K-6 students (Thompson 1984). In this study, students were randomly assigned to five groups:

1. Control group: the consent form was sent home with the students, and the parents were to return the form via the student, but no further efforts were made to obtain parental consent.

2. Treatment group 1: the students who returned the form were promised an incentive (a photograph of themselves).

3. Treatment group 2: the parents who returned the form were given an incentive—the offer of copies of articles on previous research.

4. Treatment group 3: study staff directly encouraged the students to return the form.

5. Treatment group 4: study staff directly encouraged the parents to return the form.

The rate of return of the consent forms was lowest for the control group (41 percent) and highest for students in treatment group 4 in which study personnel directly spoke with parents (96 percent) (Table 1). However, the rate at which forms were returned in the group in which students were promised an incentive (treatment group 1) was also significantly higher than the return rate in the control group—83 percent compared with 41 percent. Treatment groups 2 and 3 had return rates of 76 and 71 percent, respectively.

While direct communication with parents may be effective in encouraging the return of consent forms, the study team is unlikely to be able to use this approach in the HSAC study because of the difficulties of obtaining student contact information. At least five of the study districts that require active consent, including the three large districts of New York City, Washington, DC, and Chicago, either require active consent before parents’ contact information can be released and/or specifically prohibit school staff from providing information on students to researchers. Since the study team cannot talk directly with parents in most active consent districts in the study, Thompson (1984) suggests that offering student incentives is the next best strategy.

The relatively high rates of return of consent forms in studies that have used student and/or class incentives are also very suggestive of the effectiveness of the incentives. In 10 of the 14 studies listed in Table 1, a small student incentive was provided.¹ In these studies, the rate of return of consent forms varied from 68 to 90 percent, with 5 of the 10 studies achieving rates of return of over 80 percent. The five studies that offered student incentives but obtained rates of return of consent forms of less than 80 percent were all asking for consent to participate in studies of very sensitive subjects, such as substance abuse or sexual behavior, in which the rate of return of consent forms is always particularly low.

Many studies that do not include student incentives receive very low rates of return of consent forms. Of the five studies in Table 1 that did not include student incentives, the rates of return of consent forms varied from 33 to 96 percent. Only three achieved a rate of 80 percent or higher and in the Pokorny et al. (2001) and Eaton et al. (2004) studies this rate was achieved only in some schools. The highest rates of return were obtained when the consent forms were sent home with report cards and could be returned with the report card (85 percent) or when study staff communicated directly with the parent (96 percent). Unfortunately, we expect that we will be able to send consent forms home with report cards and communicate directly with parents in only a small number of schools, if any.

Less evidence is available about the effectiveness of the class incentives. Class incentives were given in 4 of the 14 studies in Table 1. In each case, the study also included student incentives and other strategies to increase the consent rate. The rate of return of the consent forms varied from 68 to 87 percent in these studies.

D. Proposed Experiment

Our strong preference is to use all possible tools—including student and class incentives—to achieve a high consent rate. However, if OMB will not grant permission to use a $5 student incentive for all students in the active consent districts, we are willing to conduct an experiment of the effectiveness of student and class incentives. The experiment we propose involves randomly assigning participating schools in our sample to one of three groups. (We need to randomly assign schools rather than students because the schools are likely to object to offering some, but not all, students in a school with an incentive.) The three groups would be:

• Group 1: Class receives a $25 incentive if 95 percent or more of the forms are returned; individual students are not given a financial incentive.

• Group 2: Individual students are offered $5 for returning the form; there is no class incentive.

• Group 3: Class receives a $25 incentive if 95 percent or more of the forms are returned and individual students are offered $5 for returning the form.

Other procedures to encourage the return of the forms, not involving financial incentives, will be identical in all three groups and will be documented carefully. We will record the percentage of forms returned each week starting with the week the forms are sent home with the students. Comparisons of the number of forms returned each week across groups will provide estimates of whether individual or class incentives are more effective and whether offering both individual and class incentives is more effective than offering only individual incentives or only class incentives. The results of the study will be documented and presented to OMB.

Table 1. Rate of Return of Consent Forms in School Student Studies Requiring Active Consent

Study	Sample	Procedures to Obtain Consent	Return Rate of Consent Forms
Studies with Student Incentives
Thompson 1984a	1,314 K-6 students	Randomly assigned to five groups: All groups form sent home with students Control: no additional procedure Treatment 1: Incentive to child	Control 41% Child incentive: 84%
Botvin et al. 1989	608 junior high students	Sent form home with students Periodic reminders from teachers Class incentive for class with highest rate of return Student incentive (raffle) Study staff talked with students	87%
O’Donnell 1997	3,253 7th graders	Form sent home with child Student incentive ($5.75 gift) Teacher incentive ($25)	89%
Johnson et al. 1999	2,331 students 8, 10, & 11th graders	District superintendent letter with form Student incentive (lottery for $50 prize) Calls to parents Class incentive (pizza or ice-cream party)	74%
Ji et al. 2004	21,173 students 7, 8, 9, & 10th graders	Included some but not all of the following: Principal letter attached Class incentive (pizza party) Student incentives (raffle, food coupon, movie pass) Teacher incentive Study staff at school events Inclusion of form with school report cards	68% overall 82% for middle schools 57% for high schools
Ebsensen et al. 1999	2,496 middle school students	Mailing to parents Follow-up call to parents Student incentive (pencil) Class incentives	74%
Unger et al. 2004	4,427 6th grade students	Form sent home with student Duplicate form sent home Student incentive (small gift) Teacher incentive ($25)	85%
Ebsensen et al. 2008	4,653 6th & 7th grade students	Forms sent home with student Student incentive ($3) Teacher incentives ($10 or more)	90%
New Jersey Student Health Survey 2007	2,729 high school students	Form mailed home; student addressed envelopes b School incentive of up to $800 based on return rate b Student incentive ($5) b	72%
Mandatory Random Drug Testing (IES)	8,898 9–12 grade students	Form sent home with student Student incentive ($7 movie ticket) Study team called parents and visited school	73%
Studies with No Student Incentives
Thompson 1984a	1,314 K-6 students	Randomly assigned to five groups: All groups form sent home with students Control: no additional procedure Treatment 2: Incentive to parent Treatment 3: Communication with child Treatment 4: Communication with parent	Control 41% Parent incentive: 76% Communication with child: 71% Communication with parent: 96%
Enhanced Reading Opportunities (IES)	3,389 9th grade students in 2nd cohort	Form sent home with student If return form, could get intervention Study team called parents	65%
Lueptow et al. 1977	4,470 high school seniors	Sent form home with students	42% of seniors requiring consent
Pokorny et al. 2001	7,138 6-8th grade students	In one school, consent form sent with report card Other schools, consent form mailed directly to home	33% to 85% (highest when consent form sent with report card)
Eaton et al. 2004	13,195 6th ­–12th graders	Form sent home with student Second form sent home	Only 85% or more in 36% of schools

^a Listed twice as has some groups with student incentives and others without

^b Obtained from personal communication with staff at the Bloustein Center for Survey Research

References

Anderman, C., A, Cheadle, S. Curry, P. Diehr, L. Shultz, and E. Wagner. “Selection bias related to parental consent in school-based survey research.” Evaluation Review, vol. 19, no. 6, 1995.

Botvin, Gilbert J., Horace W. Batson, Sylvia Witts-Vitale 1989, Valerie Bess, Eli Baker, and Linda Dusenbury. Public Health Reports, vol. 104, no.6, 1989.

Eaton, Danice K., Richard Lowry, Nancy D. Brener, Jo Anne Grunbaum, and Laura Kann. “Passive versus active parental permission in school-based survey research: Does the type of permission affect prevalence estimates of risk behaviors.” Evaluation Review, vol. 28, 2004.

Ebsensen, Finn-Aage, Michelle Hughes Miller, Terrance Taylor, Ni He, and Adrienne Freng. “Differential attrition rates and active parental consent.” Evaluation Review, vol. 23, no. 316, 1999.

Ebsensen, Finn-Aage, Chris Melde, Terrance Taylor, and Dana Peterson. “Active parental consent in school-based research: how much is enough and how do we get it.” Evaluation Review, vol. 32, no. 335, 2008.

Ellickson, Phyllis and Jennifer Hawes. “An assessment of active versus passive methods of obtaining parental consent.” Evaluation Review, vol. 13, 1989, pp. 45-55.

Ji, Peter Y., Steven B. Pokorny, and Leonard A. Jason. “Factors influencing middle and high schools’ active parental consent return rates.” Evaluation Review, vol. 28, no. 578, 2004.

Johnson, Knowlton, Denise Bryant, Edward Rockwell, Mary Moore, Betty Waters Straub, Patricia Cummings, and Carole Wilson. “Obtaining active parental consent for evaluation research: A case study.” American Journal of Evaluation, vol. 20, no.2, 1999.

Kearney, Kathleen, Ronald Hopkins, Armand Mauss, and Ralph Weisheit. “Sample bias resulting from a requirement for written parental consent.” Public Opinion Quarterly, vol. 47, 1983.

Lueptow, L., S. Mueller, R. Hammes, and L. Master. “The impact of informed consent regulations on response rate and response bias.” Social Methods and Research, vol. 6, 1977.

New Jersey Department of Education 2007. New Jersey Student Health Survey.. http://www.nj.gov/education/students/yrbs/2007/full.pdf

O’Donnell, Lydia, Richard Duran, Alexi San Doval, Michael Breslin, Gregory Juhn, and Ann Stueve. “Obtaining written parent permission for school-based health surveys of urban young adolescents.” Journal of Adolescent Health, vol. 21, 1997

Pokorny, Steveb B., Leonard Jason, Michael Schoeny, Stephanie Townsend, and Carrie Curie. “Do participation rates change when active consent procedures replace passive consent?” Evaluation Review, vol. 25, no. 567, 2001.

Thompson, Teresa, “A comparison of methods of increasing parental consent rates in social research.” The Public Opinion Quarterly, vol. 48, no. 4, 1984.

Unger, Jennifer, Peggy Gallaher, Paula Palmer, Lourdes Baezconde-Garbanati, Dennis Trinidad, Steven Cen, and C. Anderson Johnson. “No news is bad news: Characteristics of adolescents who provide neither parental consent nor refusal for participation in school-based survey research.” Evaluation Review, vol. 28, no. 52, 2004.

1 The Thompson (1984) study is listed twice as it had some research groups with student consent incentives and some without.

7