Supporting Statement Part B - 0687 (2012)

Supporting Statement Part B For
YOUTH TRANSITION PROCESS DEMONSTRATION EVALUATION
OMB CONTROL No. 0960-0687

1. B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

   1. 1. Respondent Universe and Sampling Methods

YTD projects intend to improve long-term employment outcomes for youth ages 14-25 with disabilities. Both current SSI beneficiaries and youth who are at risk of receiving benefits as adults comprise the respondent universe for YTD services. In April 2005, approximately 776,000 youth ages 14 to 25 years old received SSI benefits. In addition, 320,000 youth were at risk of receiving benefits as adults, even though they did not qualify to receive benefits as children.¹

YTD projects deliver services to youth with disabilities in their jurisdictions. On September 30, 2003, SSA awarded five-year cooperative agreements to seven state agencies and universities to implement YTD projects. We selected three of these projects for the national random assignment evaluation. In addition, we selected five new projects (out of 13 that applied) for a limited pilot phase. We selected three of the pilot projects in the fall of 2007 to join the national random assignment evaluation, for six random assignment projects. The respondent universe for this evaluation is youth who are willing and eligible to participate in the YTD services of the six random assignment projects.²

We began collecting baseline data collection in July 2006 and continued through 2010. We began the 12-month follow-up data collection in November 2007 and continued through 2011. Likewise, the 36-month data collection, for which we continue to request clearance, began in July 2009 and will continue through 2014. All randomly assigned youth who consented to participate in the demonstration and completed a baseline interview comprise the sample for the 36-month follow-up survey.

1. a. Selection of Youth

Each of the random assignment projects serve 400 treatment group youth. To allow for attrition, we generated a treatment group of 480 youth who we may serve, and a control group of 400 youth in all sites except Maryland. The Maryland CTP site had to generate a target treatment group of 440. This results in a total of 5,200 youth in the study (880 youth in five of the projects, and 800 in one of the projects) recruited between July 2006 and October 2010. We obtained baseline information and written consent to participate in the evaluation for all youth participating in the YTD demonstration. At 36 months, we expect to obtain responses for 3,962 youth: 82 percent of those who responded to the 12-month interview and 12.5 percent of those who were non-respondents at 12 months for a longitudinal response rate of 75 percent three years after random assignment.

Table B.1 provides descriptions of the populations and our best estimates of the numbers of youth who meet the eligibility criteria for each project.

TABLE B.1

CHARACTERISTICS OF YTD PROJECTS PARTICIPATING

IN THE RANDOM ASSIGNMENT EVALUATION

Project Name	Location	Description of Youth Served	Population Estimates
Colorado’s Youth Work Incentive Network of Support (WINS)	Boulder, El Paso, Larimer, and Pueblo counties, CO	14- to 25-year-old current SSI beneficiaries	2,750
New York’s Transition WORKS	Erie County, NY	16- to 25-year-old current SSI beneficiaries in Erie County	3,300
New York’s CUNY Youth Transition Demonstration	Bronx, NY	15- to 19-year-old current SSI beneficiaries	4,533
Abilities, Inc.	Miami-Dade County, FL	16- to 22-year old current SSI beneficiaries in Miami-Dade County	6,952
Career Transition Program (CTP)	Montgomery County, MD	High school juniors and seniors with with severe emotional disturbances—15 percent current SSI beneficiaries, 85 percent at risk of becoming beneficiaries	1,650a
Human Resources Development Foundation (HRDF)	WV (19 counties)b	16- to 25-year-old current SSI beneficiaries	4,702

^a Based on high school juniors and seniors residing in Montgomery County who are receiving special education services and have a code of severe emotional disturbance (SED).

^b Monongalia, Preston, Marion, Taylor, Harrison, Barbour, Lewis, Upshur, Randolph, Wood, Jackson, Mason, Cabell, Wayne, Putnam, Kanawha, Fayette, Raleigh, and Mercer counties.

2. 2. Procedures for the Collection of Information

a. Recruiting Study Participants at Baseline

Different recruitment strategies were necessary depending on whether a project serves only youth who are in SSA records or whether it also or only serves youth who we identified by other means, such as referrals. We first discussed recruiting procedures for youth with presence in SSA records. For projects serving current beneficiaries or at-risk youth who we identified through SSA records (for example, youth whose applications we denied), MPR conducted baseline interviews, gathered written informed consent, and randomly assigned consenting youth into the treatment or control group. After random assignment, we only shared the names of treatment group members with the YTD projects for enrollment and services. We list the specific steps in the recruitment process below:

1. Obtain a list of beneficiaries (or denied applicants) from SSA for the relevant catchment areas

2. Check the list to exclude ineligible youth based on age, place of residence, or disabling condition(s)

3. Randomly sort the list into batches of youth (also called replicates)

4. Send letters to a batch of youth informing them about YTD program services to recruit them into the study

5. Place telephone calls to determine interest in YTD services

6. Gather baseline and re-contact data by telephone, and obtain written informed consent from youth or parent by mail or in person. Appendix G contains the consent forms for the three existing sites. Consent forms for newly selected will be similar.

7. Randomly assign youth to the treatment or control group

8. Provide YTD project staffs with information on treatment youth so they can contact them and start providing program services

9. Continue to release cases in batches until we reach the desired enrollment for the project

We identified at-risk youth who had not applied for SSI benefits through referrals from local organizations, including schools and other agencies that work with youth with disabilities. Our design for the recruitment procedures is as follows:

1. The YTD project determines whether a youth who has been referred to it meets the project’s eligibility criteria.

2. If a youth meets the criteria, the project obtains a completed application form, collects baseline and re-contact data, and obtains written informed consent.

3. The project transmits this information to MPR.

4. MPR conducts random assignment and immediately provides the YTD project with information on the treatment/control status of the case.

5. YTD project staff informs the youth of his or her random assignment status and commences services to youth in the treatment group.

6. This process continues until we reach the desired enrollment target.

We designed hybrid procedures for projects that serve both youth who we could identify in SSA records as well as youth we needed to identify through other sources.

For either recruitment method, MPR (or the YTD project) contacted parents or legal guardians of youth under age 18 and gained consent to speak with youth. For youth over age 18 with legal guardians, MPR (or the YTD project) gained permission from the legal guardians before approaching the youth. Both the baseline and 12-month follow-up interviews contain a parent module consisting of questions that youth may not be able to answer reliably.

1. b. Study Procedures for 12- and 36-Month Follow-Up Interviewing

We use neither stratification nor sampling in conducting the 36-month follow-up survey. We attempt to contact all individuals who agreed to participate in the demonstration. We mail an advance letter to all sample members. We then attempt to contact sample members by telephone. We assign non-respondents to field interviewers who first attempt to locate and then interview study participants.

We mailed an advance letter to sample members advising them of the upcoming survey about one week prior to their 12-month anniversary and will mail an advance letter again one week prior to their 36-month anniversary. These letters contain a toll-free number the youth or parent may call if they have questions or wish to set an interview appointment. Next, MPR telephones the last known number for the youth and parent or guardian. If the number is disconnected, MPR attempts to locate an address or telephone number. MPR uses CATI as the primary mode of data collection for the follow-up survey. We interview in person all sample members who do not respond by telephone, or whose disabilities prevent them from being able to complete the interview via telephone. However, before conducting an in-person interview, we attempt to use TTY, computers, and other technologies that might enable an interview without field follow-up, similar to the procedures used for the 12-month data collection. It is important that we conduct follow-up interviews at the appropriate interval following random assignment, which is 36 months or shortly thereafter. Given that the sample intake period is over a long period for most projects, the number of in-person interviews required per month at a site may be too few to justify the cost of computer-assisted personal interviewing (CAPI) data collection. Thus, MPR uses more cost-effective in-person data collection methods, such as providing the field interviewers with cell phones they can use to call in to MPR. For youth who decline to participate, MPR identifies why they are reluctant to participate and sends a letter that addresses their concerns and encourages participation.

2. c. Study Procedures for Process Visits and Focus Groups

A major source of information for the process analysis consists of two comprehensive visits to each random assignment project. The exact timing of the process visits to a specific project depends on how long youth enroll in the project as well as the duration of intervention services. However, we made certain the first visit to most projects was within the first two years of demonstration startup (that is, the start of random assignment), and the second visit was approximately a year later.

Staff Interviews. During the site visits, the evaluation team conducted individual and group interviews with management and staff of various stakeholders in the local YTD project such as the following:

• Project directors and site managers offered insights into the history of each project’s sponsoring organization and its experience in serving youth with disabilities: an overview of the conception, development, and implementation of the program model and the organizational and management structure for the project, including the project budget and key project partners; and the roles and qualification of staff members, their caseloads, and the supervisory structure of the primary service providers. We designed interviews at this level to highlight some of the major challenges service providers have encountered.

• Project line staff, who are in direct contact with the youth being served, provided insight into how the youth were identified and recruited, the methods used to assess a youth’s needs and the project’s approach to serving them, the way appropriate services were selected and delivered, and the extent to which youths’ families were involved with project services. These staff also provided insight into how much structure or flexibility staff members had in performing their jobs, the extent to which clients’ experiences diverged from the program model, and the reasons behind such variation.

• Staffs of partner organizations provided information on linkages between the project and other services providers as well as on the successes or challenges of the collaborations. They provided perspectives on the nature of the agreements, how effectively they functioned, and the ways in which they complemented or integrated project services with the services of partner organizations. These might have included interviews with the staff of direct service partner organizations as well as with the staffs of schools, vocational rehabilitation agencies, mental retardation and developmental disabilities agencies, and other agencies that serve persons with disabilities.

• Staffs of local SSA field offices provided insights into the broad context of services available in the area and the local implementation of SSA’s waivers for YTD participants.

We developed separate protocols to provide structure for the each of the types of data we collected during the site visits. We created a master protocol that included the items covered during the visits and identified their relationships to the objectives or key questions for the process analysis. We selected, tailored, and used items from the master protocol with appropriate follow-up probing and elaboration depending on the specific project and the person we interviewed. Similarly, we created focus group guides, as well as structured protocols to record data from case reviews and observations of project activities.

Focus Groups with YTD Participants. To capture critical qualitative information about the experiences of YTD participants (and where relevant, their families), we conducted two focus groups in each project with participating youth and their families. We used these focus groups to discuss and gather information on participants’ experiences while participating in the project and their awareness and utilization of services. The focus group discussions covered the perceived quality of project services, perceptions of gaps in activities or services, and SSA’s explanation and offer of waivers to participants. Each group included 8 to 12 youth or parents. The focus groups complemented the information collected in the follow-up surveys, providing a more in-depth and qualitative understanding of their experiences. They helped the evaluation team assess whether and how the projects did or did not meet participants’ expectations. We also tried to conduct a focus group in each project with treatment group members who did not participate in services to understand their reasons for nonparticipation. Project staff members recruited youth and parents to participate in the focus groups. We held the discussions at project facilities that were well known in the community and accessible to persons with disabilities.

3. d. Statistical Power/Precision Estimates

For this evaluation to be useful to policymakers, it needed to have a sample large enough to allow us to detect policy-relevant impacts. The design of the YTD evaluation calls for the random assignment of 880 youth with disabilities to either a treatment or a control group for each of six projects. Table B.2 presents estimates of the minimum treatment-control differences that we could detect for three types of outcomes that the evaluation will examine. First, for outcomes that can be expressed in binary terms, such as the likelihood of becoming employed or of leaving the SSI rolls, we present estimates for outcomes centered on 50 percent (the most conservative assumption), as well as on 30 or 70 percent. Second, we examine annual earnings based on SER data. Third, we consider monthly SSA benefit amounts. The earnings and benefit outcomes will be critical in determining the cost effectiveness of YTD services. We presented the minimum detectable treatment-control differences for these outcomes under the assumption that we use a two-tailed test and 90 percent confidence levels to determine impacts. The table shows minimum detectable differences at 80 percent power (that is, the ability to detect true differences 80 percent of the time). We assume a reduction in variance of 10 percent owing to the use of regression models.

The numbers in the table indicate that, with sample sizes of 480 treatment group members and 400 control group members, we could detect impacts on employment and benefit receipt of 7 to 8 percentage points, impacts on earnings of $489 annually, and impacts on SSI benefits of $42 per month. For example, if the likelihood of being employed one year after random assignment were 30 percent in the absence of YTD services, and if YTD services raised this to 38 percent, then we would have an 80 percent chance of detecting this impact with our sample.

We confirm the adequacy of samples of 480 treatments and 400 controls by several studies of people with disabilities. For example, we use the evaluation of the Transition Employment Training Demonstration study, which used samples of about 375 recipients each in the treatment and control groups. The study estimated that transitional employment services for SSI recipients with mental retardation increased earnings during the second year after random assignment by $835 and the probability of employment increased at the end of that year by 12 percent. Similarly, the evaluation of the Structured Training and Employment Transitional Services demonstration, which targeted youth with mental illness, found an increase of more than 9 percentage points in employment for treatment group youth 15 months after random assignment.

TABLE B.2

MINIMUM DETECTABLE IMPACTS FOR THE YTD EVALUATION,

ASSUMING INDIVIDUALIZED RANDOM ASSIGNMENT

	Employment Rate or SSI Receipt Rate
Sample Size (Treatment/Control)	50 Percent	30 or 70 Percent	SER Annual Earnings (Mean = $1,213)	Monthly SSI Benefits (Mean = $588)
Full Sample
480/400	8.0	7.3	$489	$42
Subgroup Sample
240/200	11.7	10.3	$690	$60

Note: The calculations assume (1) a 90 percent level of confidence for a two-tailed test and an 80 percent level of power, (2) a standard deviation of $267 for the monthly SSI benefits amount and $3,069 for annual earnings, and (3) a reduction in variance of 10 percent owing to the use of regression models. We derived the standard deviations from Mathematica’s Ticket to Work Evaluation Summary Earnings Records data and SSI benefits data for youth ages 18 to 25 in 2001.

3. 3. Methods to Maximize Response Rates

We will continue to use the following procedures to maximize the response rates.

• Effective and targeted advance materials

• Collecting and updating of contact data for the sample member at 12 months for someone who would know how to reach the sample member at the 36-month interview.

• Multiple methods for tracking and locating beneficiaries, including the use of extracts from SSA administrative data to capture address updates during the course of the survey, the use of an independent vendor providing commercially available contact information, and MPR’s internal respondent tracking efforts

• At the 12- and 36-month follow-up interviews, the use of a combination of telephone and in-person interviewing to maximize our ability to contact sample members

• Interviewer training that includes instruction on motivational interviewing, that stresses the importance of respondent cooperation, and that develops interviewer skills for averting and converting refusals

• Interviewer training on when and how to select an appropriate proxy to conduct an interview

• A bilingual module to help bilingual interviewers assess whether to conduct an interview in Spanish or English and to cover differences in dialects

• Protocols for breaking off and then resuming interviews to accommodate beneficiaries who may become fatigued during the interview

The focus of all respondent materials (letters, brochures, and consent forms) is to secure cooperation through the clarity, simplicity, and thoroughness of the materials, which we write at a sixth grade reading level. Despite this, locating participants is still a challenge. While SSA has contact information for all current beneficiaries, that information is not always accurate, and at follow up some sample members no longer receive benefits from SSA. Telephone numbers are particularly problematic because there is no administrative reason to keep them updated in SSA records. Addresses are more reliable because they sometimes use them for mailing checks. However, these might be post office boxes, addresses of guardians, financial institutions, or other individuals and organizations that are of only limited use in locating a beneficiary. Further, since many beneficiaries now receive their checks via direct deposit, SSA address information is less accurate now than it once was.

To improve the contact information, we mail an advance letter to each sampled person prior to each survey, using the most recent address of record. The letter describes the survey, provides a toll-free number to contact Mathematica, and indicates that we will contact the beneficiary regarding it. We send the letter “address service requested,” which results in (1) the mail being forwarded to recipients who have a forwarding address and (2) a notice of the new address being sent to the sender. If the forwarding authorization expired, we return the letter to the sender with the new address attached.

When an address is available but a phone number is not, we conduct a directory search to obtain a number. For cases where neither SSA records nor the directory search yields a telephone number, MPR uses alternative locating strategies, including online nationwide databases to verify or update addresses and other information. During the 12-month interview, we request the name, address, and telephone number of two people who are likely to know how to contact the sample member in the future. If we lose contact with the sample member, we contact these individuals to obtain the sample member’s most recent information. At the 36-month follow-up, if locating contacts are exhausted and no current phone number is available, we conduct a field search, starting with any available information. This usually involves a contact with the addressee for the beneficiary’s monthly check, which may be the beneficiary or their representative payee. If the addressee is not the beneficiary, we would expect that individual to have the contact information that we are seeking. Some sources might be reluctant to provide that information, and in such an instance, we ask the source to pass on a written request to the beneficiary to send us the information on a postage-paid card, to call a toll-free number, or to contact us by email.

When a phone number is available or we obtain it, we attempt to contact the beneficiary by telephone to conduct the interview. We make attempts on different days and times. If we make successful contact and the beneficiary consents to the interview, we conduct the interview using CATI technology. As indicated above, we make multiple accommodations to increase response and encourage participation by sample members in the interview. For respondents who are deaf or hard of hearing we use amplified telephones, TTY, and Relay technologies. For respondents who speak Spanish, advance materials are available in Spanish, and we developed a Spanish-language version of the survey instruments that we administer by Spanish-speaking interviewers. We use interpretation services for other non-English speakers. For respondents who fatigue easily, we use structured checkpoints during the interview so that interviewers can assess whether a respondent is becoming too fatigued to continue with the interview and schedule a convenient time to complete the interview. A ten-dollar post-paid incentive at baseline and after each follow-up interview keeps sample members engaged over time.

4. 4. Tests of Procedures

The procedures for the 36-month follow-up are identical to those on the 12-month interview. Most questions in the 36-month follow-up questionnaire appear on other studies of youth or persons with disabilities. These include our 12-month interview National Longitudinal Transition Survey (NLTS), the National Beneficiary Survey (NBS), the Short Form 12 (SF12), the Canadian Youth in Transition Survey (YITS), Opening Doors, and the Job Corps Evaluation (JC).

In addition, we conducted nine telephone pretests among youth with disabilities selected for the pilot programs. We used the pretest to evaluate the clarity of the questions asked, identify possible modifications to either question wording, or question order that could improve the quality of the outcome data, and estimate respondent burden. We scrutinized new questions (not used in previous, similar surveys) especially closely during the pretest. The interviews took, on average, 50 minutes to administer.

5. 5. Statistical Consultants and Persons Collecting and Analyzing the Data

Mathematica Policy Research, Inc. (MPR) is conducting this study, including collecting and analyzing the survey data, under contract to SSA (Contract No. SS00-05-60084). MDRC is a subcontractor to MPR on this study. Thomas Fraker of MPR (202-484-4698) is the project director and has overall responsibility for the project. Anu Rangarajan (609-936-2765) and John Martinez of MDRC (212-340-8690) are the principal investigators. Karen CyBulski (609-936-2797) and Anne Ciemnecki (609-275-2323) direct the data collection effort. Joyanne Cobb of SSA (202-358-6509) is the technical Project Officer.

1 Our definition of at-risk youth includes denied child SSI applicants age 16 to 25 and youth with serious emotional disturbances age 14 to 17.

2 All seven of the existing projects and the three new ones selected for the random assignment study will be included in a process study of the implementation of YTD. That study will include discussions with project staff and service providers.