Supporting Statement Part B - NEW (ODRD) - Revised

Supporting Statement for Ohio Direct Referral Demonstration (ODRD)

OMB No. 0960-NEW

2. Collection of Information Employing Statistical Methods

1. Statistical Methodology

The respondent universe for the Ohio Direct Referral Demonstration (ODRD) is individuals in Ohio ages 18 and 19, who are either (1) applying for Social Security Disability Insurance (SSDI) or Supplemental Security Income (SSI) or (2) undergoing an age-18 redetermination of SSI eligibility. Recruitment will occur throughout the state of Ohio (OH). The current caseload in OH is about 3,000 age-18 SSI redeterminations and applications for SSDI benefits and SSI payments at ages 18 and 19.

Because our demonstration authorities require voluntary participation, we will continuously invite individuals meeting the above criteria to participate. We are expecting a 25 percent enrollment rate, which compares with the enrollment rate of the Promoting Readiness of Minors in SSI (PROMISE) demonstration (OMB # 0960‑0799) project’s rates of 15 percent (NY), 22 percent (ASPIRE and WI), 26 percent (AR), 29 percent (CA), and 43 percent (MD).  Thus, we expect to enroll approximately 750 individuals. We are not planning to stratify or sample the population.

2. Procedures for Collecting the Information

The Ohio Division of Disability Determination (DDD) will obtain consent as part of their pre-existing interview with the claimant. If the individual consents, then DDD will provide Ohio’s Bureau of Vocational Rehabilitation (BVR) with the consent forms for tracking purposes.

The DDD will use the terminal digit of the claimant’s Social Security number (SSN) to assign an individual to the treatment and control groups randomly, since the last four digits of the SSN are assigned randomly (unlike the first five, which used to be determined by geography). Over the years, we used this method within SSA for several random samples for auditing, research, and other purposes.

Ohio’s BVR will send letters to the treatment and control group explaining their study assignments. The DDD will send the treatment group’s contact information to the BVR for follow-up to begin any vocational rehabilitation assessments. The control group will receive no further information from the BVR.

All data, other consent, will come from either SSA or OH administrative records. As such, we consider it the most accurate with respect to program experiences. Specifically, the BVR will provide the following information for all participants:

• Date of application

• Date of individualized plan for employment

• The total number of services provided

• Cost of purchased services

• Closure date (if case closed)

• Type of closure (if case closed)

• Reason for closure (if case closed)

For the evaluation, SSA will use this information, matched to information in the following systems of records:

• SORN 60-0103-Supplemental Security Income Record and Veterans Benefits;

• SORN 60-0059-Earnings Recordings and Self-Employment Income System; and

• SORN 60-0090-Master Beneficiary Record

We will use this data to answer the following questions:

• What effect did the intervention have on receipt of Ohio’s BVR services?

• What effect did the intervention have employment outcomes, such as job placement and earnings?

• What is the length of time from application to eligibility decision?

• What general vocational rehabilitation outcomes did participants achieve?

• What was the number of closed cases resulting in employment and what was the number of cases closed for other reasons?

We will use an intent-to-treat design, including all participants in the analyses. This design will constitute an analysis of the results based on the treatment arm to which the participants are assigned, regardless of whether the participants receive VR services. This allows for a policy-relevant evaluation of the direct referral to OBVR services, which an individual may not choose to follow up on or utilize.

It is important for the demonstration to be able to detect impacts on these outcomes. For the purposes of planning, we chose receipt of Ohio’s BVR services as the primary outcome of interest. Table B.1 presents estimates of the minimum treatment-control differences we could detect for binary outcomes. We present the minimum detectable treatment-control differences for these outcomes under the assumption that we use a two-tailed test and 95 percent confidence levels to determine impacts. Table B.1 shows minimum detectable differences at 80 percent power (that is, the ability to detect true differences 80 percent of the time).

As shown in Table B.1, with an even split between treatment and control groups (375 per group), we will be able to detect impacts of about 8-9 percentage points if 90‑80 percent of the control group receives vocation rehabilitation (VR) services. We expect to detect between 10 and 15 percentage points if there are lower rates of VR participation in the control group.

Impacts of this size would be sufficient to warrant further study of this demonstration in other settings. The sunset date of our section 234 authority precludes us from targeting smaller impacts; however, if the enrollment rate is higher than expected we may be able to recruit additional participants. If that is the case, we will submit a change request to OMB to adjust the burden accordingly.

Table B.1: Sample Required for Selected Minimum Detectable Impacts

Group Proportion	Control Outcome	Minimum Detectable Impact
		0.05	0.07	0.1	0.15
T=C	0.9	726	401	219	113
T=C	0.8	1,134	603	313	151
T=C	0.7	1,417	741	376	176
T=C	0.6	1,574	815	408	186
T=C	0.5	1,605	825	408	183

Note: Calculations assume a 95 percent level of confidence for a two-tailed test and an 80 percent level of power and individual-level (non-clustered) randomization; calculations do not adjust for multiple comparisons.

There is no known measure of the baseline rate of VR participation for the study population. As a second-best reference, in the Youth Transition Demonstration (0960-0687, discontinued), between 21 and 87 percent of the participants (who were SSI or SSDI recipients, or potential recipients, between ages 14 and 25) participated in an educational or training program three years after random assignment. In the Promoting Readiness of Minors in SSI project (0960-0799), about 90 percent of participants (who were SSI recipients ages 14-16) received transition services after 18 months.

3. Methods to Maximize Response Rates

To maximize response rates, we will obtain written consent from all youth at the time of enrollment and random assignment into the project. Therefore, all ODRD participants must necessarily complete the demonstration forms as a first step in their enrollment in the project. We will mail the Invitation to Participate and Consent Form to all individuals ages 18 and 19 at the time of enrollment, who are either (1) applying for SSDI or SSI or (2) undergoing an age-18 redetermination of SSI eligibility. As participants sign and return these forms, we will randomly assign them to a control group or test group (randomization described below). Once we receive the appropriate number of responses (750 responses), the DDD will stop sending the Invitation to Participate and Consent form and will stop randomly assigning individuals to these groups.

We will train the DDD staff to use the terminal digit of the claimant’s Social Security number (SSN) to assign an individual to the treatment and control groups randomly, since the last four digits of the SSN are assigned randomly (unlike the first five, which used to be determined by geography). As we mentioned previously, SSA uses this method for several random samples for auditing, research, and other purposes. SSA will provide DDD staff with written guidelines on the terminal digit method and will ensure they receive in person training. We will limit the number of individuals who we will train, and who will be responsible for this task (no more than 4 DDD staff).

Finally, we will not limit contact to Ohio’s BVR. Those individuals who we randomly place in the control group may still contact OBVR on their own, through the normal front door process.

4. Tests of Procedures

SSA conducted usability testing on the Invitation to Participate and Consent Form with nine individuals between the ages of 18-19. This convenience sample was comprised of participants in a local program where the lead SSA staff member volunteers. Four of the nine individuals received SSI or SSDI at the time of the testing. As a result of the test, we did not see the need to make any changes to the form.

5. Statistical Agency Contact for Statistical Information

The SSA staff member responsible for managing this project is Joyanne Cobb, (202) 358-6509. Additional SSA staff members who will be involved in analyzing the data include Jeffrey Hemmeter, (410) 597-1815. We may add more staff members when we begin the evaluation process.