Memo to OMB_NonSub Change_Online Form

370 L'Enfant Promenade, SW, Washington, DC 20447 www.acf.hhs.gov

DATE: June 11, 2015

TO: Josh Brammer

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

FROM: Hilary Forster

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

SUBJECT: Request for Non-Substantive Change to Health Profession Opportunity Grants (HPOG) Impact Study Data Collection Plan (OMB Control Number 0970-0394)

Background on the HPOG Impact Study

The Health Profession Opportunity Grants (HPOG) Impact Study data collection approved under OMB Control Number 0970-0394 includes baseline data collection for study participants at program enrollment and follow-up surveys conducted 15 months and 36 months after random assignment. To help ensure the study team can reach study participants for the follow-up surveys, the participants are contacted periodically to confirm or update the contact information provided at enrollment or through the interim tracking efforts.

Proposed Alternative Mode for Collecting Updated Contact Information from Study Respondents

To ensure contact information is up-to-date for the 15-month follow-up survey, study participants are sent letters with a form to complete and return by mail four, eight and twelve months after random assignment. The average response rate for the mailed contact update requests is approximately 20 percent and diminishes across updates: the 4 month mailing has a 25 percent return rate, the 8 month mailing has a 20 percent return rate, and the 12 month mailing has a 15 percent return rate. In order to increase the collection of updated contact information, Abt Associates and its partner, the Urban Institute, who are conducting the HPOG Impact Study, propose to offer a web-based update form as an alternative mode of collecting this information. OPRE requests approval to implement this new mode of collecting updated contact information as a non-substantive change to the data collection plan approved under OMB Control Number 0970-0394.

The information collected online will not differ in content from that collected through the mailing. The online form will impose no further burden on respondents. It simply offers recipients another option for responding and offers cost advantages including a reduction in costs for printing, postage, staff time for the mailing, and staff time for data entry for returned paper forms. The visual look of the online form, however, is somewhat different. In order to make it easier for the respondent to navigate through the online form, particularly from mobile devices, only a limited number of items will be displayed at one time. On the paper form respondents see their name, address and phone number as they appear in the study’s records, and they see spaces for them to provide updates if needed. The online form allows the respondent to indicate which piece(s) of information on records are correct. The next screen will allow them to update the piece(s) of information that were not marked as correct. To guide respondents, some additional question text and instructions have been added. In addition, the online form and the tracking letter have both been modified to ensure consistency in the tracking data collection components captured prior to the 15-month follow-up and for the 36-month follow-up data collection efforts. These changes for the 15-month survey include the addition of address data for secondary contacts, which the study team has found useful for locating efforts on other similar studies, as noted on the attached forms.

Expected Benefits

The benefits of a web-based contact information form are both financial and, potentially, with respect to return rates to contact information updates.

Costs for a web-based contact information form are primarily incurred at start-up, with per-unit costs being very low compared to per-unit costs for paper forms. Sending an update email does not incur printing and postage costs. Labor costs for sending updates are also lower, with programmer time required to send invitations being unaffected by the number of updates sent, as compared to the labor costs for stuffing envelopes which scale linearly by the number of pieces mailed. Labor costs are also lower for entry of returned forms. Web-based forms require a one-time expense of programmer time to write code to move the data from the web platform hosting the form to the study database and limited project staff labor to check data for quality. By contrast, paper updates must all be entered by hand with expenses scaling linearly by number of forms received.

A web-based contact information form may also yield increased return rates for contact information updates. In some cases, participants may have mode preferences for web-based update forms (e.g., ease of use, no need to put in the mail). In other cases, an email address may still be functional while the last known address may be inaccurate. Any improvement in return rates for contact information updates will increase contact rates for the 15 month survey, reducing the level of effort required to complete 15 month interviews.

Proposed Procedures for Online Contact Updates

Study participants who have provided an email address (roughly 80 percent of the sample) will be sent an email with a unique link to the online update form where they can review contact information on record and make updates as needed. As with the mailed update mode, participants who reply to the request for updated information will receive $5. Where the email address is undeliverable, a mailed update will be sent to ensure that the participant still has an opportunity to update her or his contact information.

To assess the effectiveness of the online approach, an experiment will be conducted. Study participants with email addresses will be randomized into mail and email/web treatments, stratified by HPOG treatment and control status and prior history of returning contact updates. The results will be assessed after the first month to determine the effectiveness of the email/web treatment.

In order to be deemed a success, the proportion of treatment group updates received from the control group must be significantly higher than the sum of the proportion of control group updates received and the proportion of control group USPS address forwarding notifications received. Address forwarding notifications are taken into account when assessing the control group because these notifications provide an updated mailing address and are used to send additional mailed contact information updates, improve matches on Accurint for telephone numbers, and—most importantly—by field personnel to obtain interviews. The experiment will be discontinued if the minimum detectable difference at appropriate power (i.e., 0.8) is greater than the observed difference. One tailed tests will be used because H_a: x_t > x_c rather than H_a: x_t ≠ x_c, where x is the proportion returned as described above for the treatment (t) and control (c) groups. If neither condition holds—the effect is neither significant nor below the minimum detectable difference—the study team proposes continuing the experiment in the following month. Based on the anticipated size of the July mailing (n=1,617), the study team anticipates the design will be able to detect a difference in return rates of 5.3 percentage points with 0.8 power (where the proportion returned for the control group is based on historical norms).

If deemed successful by the criteria described above, all cases with email addresses will receive the email/web treatment.

The current version of the mailed form (including revisions reflecting consistency with the 36-month form, identified by comment fields) and a draft of the format for the online form are included as Attachments 1 and 2, respectively, within the attached Appendix H. Contact Letter and Form. OPRE requests approval to implement this new mode of collecting updated contact information as a non-substantive change to the data collection plan approved under OMB Control Number 0970-0394.

References:

Medway, Rebecca L. and Jenna Fulton. 2012. “When More Gets You Less: A Meta-Analysis of the Effect of Concurrent Web Options on Mail Survey Response Rates.” Public Opinion Quarterly 76:733-46.