Supporting Statement--0726

This is a request for Office of Management and Budget (OMB) clearance to collect data for the Mental Health Treatment Study (MHTS). We are conducting this study for the Social Security Administration (SSA), under a contract with Westat and its subcontractors: Dartmouth Medical School; ValueOptions/Colorado Health Networks; and the University of Maryland (Baltimore County). SSA is seeking approval for the primary data collections for this intervention.

Purpose and Authority

Overview

The Mental Health Treatment Study (MHTS) is a randomized trial to test the extent to which eliminating programmatic work disincentives, establishing an accurate diagnosis, delivering appropriate mental health, and supporting employment (SE) will lead to improved functioning and competitive employment among Social Security Disability Insurance (SSDI) beneficiaries with a primary impairment of schizophrenia or affective disorder. Study outcomes assess the impact and cost effectiveness of the intervention, including identification of specific factors within the interventions that result in positive employment outcomes.

Section 234 of the Social Security Act (42 U.S.C. 434) gives the Commissioner of Social Security the authority to develop and carry out experiments and demonstration projects designed to determine the relative advantages and disadvantages of interventions that facilitate a beneficiary’s return to work. Part of the agency’s role involves finding ways to promote work and increase independence. Because of advances in medical treatment, assistive devices, changes in our views toward those with disabilities, and legislation designed to assure access to employment, SSA is taking on an increasingly active role in assisting beneficiaries who want to return to work.

In February 2001, SSA received additional support through President Bush's New Freedom Initiative -- a comprehensive program whose primary goal is to promote the full participation of individuals with disabilities in all areas of society. The aim of the Initiative is to help Americans with disabilities by increasing their access to effective technologies, expanding educational opportunities, increasing the ability of Americans with disabilities to integrate into the workforce, and promoting increased access into daily community life. This initiative provided SSA with the support necessary to address the need to expand educational and employment opportunities for beneficiaries in an effort to provide supports and services that will enable them to maximize their self-sufficiency and potentially enter or reenter the workforce.

There are at least three compelling reasons for conducting the MHTS. First, we demonstrate effective mental health treatments for persons with psychotic disorders (e.g., schizophrenia) and affective disorders, the target population for the MHTS. The evidence is strong that research-based, individualized treatment can alleviate symptoms and improve functioning in individuals with psychiatric disorders, allowing them to return to their premorbid status (US DHHS, 1999). Thus, there is every reason to expect that if we make high quality mental health treatments available to SSDI beneficiaries, they will experience symptom reduction, improved functioning, and increased work capacity.

Second, supported employment (SE), in particular the most researched model of SE, the Individual Placement and Support (IPS) model recommended by the MHTS Technical Advisory Panel, has been shown to be highly effective in returning individuals with psychiatric disabilities to competitive employment (Becker and Drake, 1993). Since beneficiaries with a primary impairment of mental illness often need long-term supports to maintain employment, providers of employment services may consider them “hard-to-serve”. However, most forms of mental illness are treatable, and there are promising findings from research on interventions that integrate treatment with rehabilitation services.

Third, growth in the disability rolls based on a primary impairment of mental disorder has been dramatic. This is a function of increases in applications and awards, and duration of disability. SSDI beneficiaries with primary mental impairments cost SSA more than any other group of SSDI beneficiaries. As the MHTS Technical Advisory Panel points out, this group’s low earnings and long tenure on SSA’s rolls mean, “any intervention that moves even a small share of people into employment could have a very large impact on the overall costs.”

Finally, the MHTS will ensure access to evidence-based treatment and follow-up, as well as comprehensive medical care and integration of all mental health services and employment supports. Notwithstanding the huge advances in the past decade in the knowledge base of effective treatments for persons with psychiatric disorders (Institute of Medicine, 2001), these effective treatments are not widely accessible, and thus are not part of routine care for most persons with psychiatric disorders (Drake, Goldman, Leff, et al., 2001; Lehman and Steinwachs, 1998). While “treatment as usual” has improved considerably in recent years, an enormous gap exists between the research-based best practices and the care accessible and provided to most persons with psychiatric disorders.

References:

Becker, D.R., and Drake, R.E. (1993). A working life: The Individual Placement and Support (IPS) Program. Concord, New Hampshire-Dartmouth Psychiatric Research Center.

Drake, R.E., Goldman, H.H., Leff, H.S., Lehman, A., Dixon, L., Mueser, K.T., and Torrey, W. (2001). Implementing evidence-based practices in routine mental health service settings. Psychiatric Services, 52(2), 179-182.

Institute of Medicine (2001). Crossing the Quality Chasm: A New Health System for the 21^st Century. Washington, DC: National Academy Press.

Lehman, A.F., and Steinwachs, D.M. (1998). Translating research into practice: The schizophrenia PORT treatment recommendations. Schizophrenia Bulletin, 24, 1-10.

Lehman, A.F., and Steinwachs, D.M. (1998). Patterns of usual care for schizophrenia: Initial results from the Schizophrenia Patients Outcomes Research Team (PORT) client survey. Schizophrenia Bulletin, 24, 11-23.

U.S. Department of Health and Human Services (1999). Mental Health: A Report of the Surgeon General. Rockville, MD: U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services, National Institutes of Health, National Institute of Mental Health.

Approach and Goals

The MHTS implemented a randomized trial study design to evaluate the impact of the intervention on employment and functional outcomes for SSDI beneficiaries with a primary mental impairment of schizophrenia or affective disorder. We are currently implementing the MHTS in 22 demonstration sites across the United States, with one site having two locations. The study participants are SSDI beneficiaries with varying clinical and demographic characteristics, employment histories, and sometimes-additional medical impairments. The study design has two arms: treatment (special services) and control (regular services) group. We randomly assigned study participants to the treatment or control group. Each treatment or control beneficiary will participate for a total of 24 months following enrollment. The treatment intervention activities include the following:

Diagnostic psychiatric assessment (using the Structured Clinical Interview for DSM-IV [SCID]) upon enrollment;
Comprehensive medical assessment upon enrollment;
Systematic medication management (for those prescribed medication);
Supported Employment (SE) according to the Individual Placement and Support (IPS) model;
Individualized clinical treatment (using evidence-based practices);
Supplemental health insurance, when needed by the beneficiary; and
Coordination and payment of beneficiaries’ claims (for both psychiatric conditions and non-psychiatric conditions that may affect the beneficiary’s ability to work).
The intervention plan will also include quality assurance mechanisms such as measuring fidelity to the SE model and other evidence-based clinical treatments and adherence to treatment guidelines overall, with subsequent training to improve fidelity when implementation deficiencies are identified.

The MHTS intervention activities will fulfill specific intervention process goals, as follows:

1. Diagnostic psychiatric assessment, comprehensive medical assessment, and systematic medication management will produce accurate psychiatric diagnosis and optimum medication regimen;

2. Quality assurance will produce adherence to treatment guidelines;

3. Supported Employment and individualized clinical treatment will lead to beneficiaries’ receiving high fidelity, integrated supported employment and other evidence-based mental health practices; and

4. Supplemental health insurance and claims coordination and payment will result in beneficiaries’ receiving full health care coverage.

Data Collection Summary

We collected initial data for purposes of solicitation, screening for eligibility, consent to participate in the study, and study enrollment. Outcome data will consist of data collected from baseline, quarterly, and follow-up interviews with study participants. Outcome data will include assessment of the impact and cost effectiveness of the intervention, as well as the identification of the specific factors within the intervention that result in positive employment outcomes. Finally, we collect quality assurance data to ensure fidelity of intervention implementation. In summary, we collect five types of data:

Initial Data: we used SSA administrative records to identify the target population and their contact information. Then used the information to contact beneficiaries and invite them to a research information group meeting where solicitation will take place. Interested beneficiaries were screened (using a 3-item screener) to confirm a) their ability to give consent; b) that they are not living in a nursing home or residential care facility; and c) that they do not have any medical conditions that preclude working.

Baseline Data: In-person interviews collecting baseline information for treatment and control groups including demographic characteristics, type of disability, benefits status, attitudes toward treatment and work, health care coverage, health care utilization, and provider contact information;

Quarterly Data:

1) Control Group: Periodic tracking of the control group to measure the nature and extent of “usual treatment” and to obtain information about employment outcomes, and use of services. We also use the quarterly interviews to update contact information for the control group. We administer the quarterly interviews for the control group by telephone.

2) Treatment Group: Intervention group data to determine the nature and extent of the intervention received, as well as repeated measures of outcomes to examine change over time. The Treatment Group quarterly report will include data from the Control Group Quarterly Questionnaire as well as more intensive information from a subset of questions from the Baseline Questionnaire. We administer the quarterly interviews for the treatment group by telephone.

Followup Data: In-person interviews on outcomes from both the treatment intervention and control group (including outcomes on medical recovery, functioning, employment, and receipt of benefits); and

^¹Quality Assurance Data: We collect quarterly measures of quality assurance from the demonstration sites to ensure adherence to principles of systematic medication management and the IPS model.

Westat developed a secure web-based study management system (SMS) to manage data collection, storage, and reporting, as well as collect the quality assurance data.

Appendix A presents an outline of the constructs measured, the instruments used for each construct and a brief description, and the timing of data collection (baseline, quarterly and/or follow-up). We do not use some instruments in their entirety; rather we only use a subset of items or sections for the MHTS data collection as the CAPI navigates the questionnaire in response to the beneficiary’s answers to questions. The actual questionnaires used for the MHTS data collection are included in Appendix B under the collections Instruments tab of the package. Each questionnaire section is a compilation of select items or scales taken from existing instruments. For each section, we list the items with a notation indicating the source instrument(s) according to the acronyms or abbreviations noted in Appendix A.

The measures that do not influence respondent burden, and are part of customary and usual business practices, are not included in Appendix A or B. They include quality assurance measures, data extracted from participant medical charts, data obtained from SSA administrative records, data collected through the psychiatric assessment as part of the intervention plan, and data collected or compiled by MHTS staff.

Use of Information

The information collected as part of data collection will be used by the SSA according to Section 234 of the Social Security Act (42 U.S.C. 434). We summarize the findings in a report that we submit to the SSA at the end of the study. The comprehensive assessment of the MHTS outcomes will identify which, if any, of the interventions resulted in successful employment and functioning outcomes, and identify the characteristics of the interventions that contributed to that success. This information will enable SSA to further develop ways to improve services to current and future beneficiaries. SSA will also use this information to guide any potential changes to program rules to allow for better coordination among other Federal and State programs.

3. Method of Collection

A nurse Care Coordinator (CC) and Research Assistant (RA) at each demonstration site (MHTS staff) will be the primary data collectors. In addition, a trained clinician associated with each demonstration site will administer the Structured Clinical Interview for DSM-IV (SCID) to confirm an accurate psychiatric diagnosis at baseline among the treatment group participants. The purpose of the accurate diagnosis is to develop an individual treatment regime.

We will collect data using multiple techniques: 1) computer-assisted person/telephone interviewing (CAPI/CATI); 2) web-based data entry interface, and 3) extraction of information from electronic files. In each case, the CC or RA will conduct an interview or extract data from participant charts to complete data collection.

Avoidance of Duplication

The instrumentation team has carefully reviewed the measures for the study to eliminate redundancy among the questionnaires. There is currently no in-depth data available on beneficiaries, from the demonstration sites or SSA, which would provide sufficient detail on baseline related variables. Moreover, since this study involves the implementation of a new intervention, data is collected quarterly and at the 24-month follow-up is unavailable until after the intervention has begun implementation.

Westat will obtain relevant existing information on beneficiaries’ employment outcomes (i.e., earning records and SSDI status) directly from SSA to avoid duplication and reduce data collection time and effort.

Small Business Impact

No small businesses will be involved as respondents in this data collection. Therefore, there will be no small business impacts.

Consequences of Not Collecting Information

The data collected will provide SSA with the scientific evidence it needs to assess the value of the intervention activities of the MHTS. Without this information, SSA would not be able to adeptly develop additional ways to improve services to current and future beneficiaries based on the outcomes of the MHTS, as well as other sources.

7. Special Circumstances

The proposed data collection is consistent with 5 CFR 1320.6 and therefore involves no special circumstances.

Consultation Outside the Agency

SSA published the first Federal Register (FR) Notice on February 17, 2009 at 74 FR 7506, and SSA has received no public comments. The second Notice published on May 20, 2009, at 74 FR 23764.

As a first step in designing the MHTS, an eleven-person technical advisory panel convened for a series of three meetings (total of four days). We submitted a report to SSA that made initial recommendations on the general parameters of the MHTS. This included the general focus, the target population, and actual interventions and treatment services.

The technical advisory panel consisted of expert researchers, clinicians, advocates, insurance executives, and rehabilitation and vocational experts. These individuals reflected expertise in mental health care financing, mental health treatment, research design, research ethics, state mental health systems, disability management, vocational rehabilitation, employment services, consumer perspectives, provider perspectives, employer perspectives, and disability benefits. We listed the members of the technical advisory panel in Exhibit 1.

Exhibit 1. Technical Advisory Panel Members

Name	Title & Affiliation
Deborah Becker	Assistant Research Professor and Director of Supported Employment Programs, New Hampshire-Dartmouth Psychiatric Research Center, Dartmouth Medical School
Dale Dutton	CEO, Noble Solutions, Inc. Former National Director of the Commission on the Accreditation of Rehabilitation Facilities (CARF) Parent of young adult currently receiving SSI and DI
Laurie Flynn	Senior Research and Policy Associate, Division of Child and Adolescent Psychiatry, Columbia University College of Physicians and Surgeons Former President of the National Alliance for the Mentally Ill (NAMI)
Kevin Hennessy	Science to Service Coordinator for the Substance Abuse and Mental Health Services Administration (SAMHSA), U.S. Department of Health and Human Services
David Mechanic	University Professor and René Dubos Professor of Behavioral Sciences, Institute for Health, Health Care Policy and Aging Research, Rutgers University
Daniel O’Brien	Trainer/Program Coordinator, University of North Texas, Region VI Community Rehabilitation Program, Rehabilitation Continuing Education Program
Thomas O’Conner	Disability Management Consultant, O’Connor Associates
Patricia Owens	Health and Disability Programs Consultant Board Member of the Disability Policy Panel on the National Academy of Social Insurance (NAMI) Former Associate Commissioner for Disability, SSA
Harold Pincus	Professor and Executive Vice Chairman of Psychiatry, Western Psychiatric Institute and Clinic, University of Pittsburgh Medical Center
Sally Rogers	Director of Research and Research Associate Professor, Center for Psychiatric Rehabilitation, Boston University
John Rush	Associate Professor and Director, Department of Psychiatry, University of Texas Southwestern Medical Center at Dallas

Reimbursement of Respondents

The beneficiaries in the treatment group will not receive any financial incentive payments or gifts for their participation in the intervention and/or data collection activity. The beneficiaries in the control group will receive a total of $100 each for their participation over the course of the study. Upon completion of the baseline interview, they received $20. They received $10 upon completion of each of the seven quarterly interviews. They receive $10 upon completion of the final interview.

Assurances of Confidentiality

During the recruitment phase of the study, we invited beneficiaries to a research information group meeting that explained the study. During this meeting, as well as on the consent form, we provided participants with the following assurance of confidentiality:

Information collected for this study will be kept private and confidential and is protected by law according to Section 1106 of the Social Security Act (42 U.S.C. 1306) and the Privacy Act (5 U.S.C. 552a). SSA will use my information solely for the purposes of this research. SSA will not share information about me with any other department at SSA.

SSA ensures data security and confidentiality of electronic information through proven standardized security configurations and methods that include state-of-the art certificate services and other infrastructure features.

Sensitive Items

The desired intervention outcomes are three-tier: improved clinical recovery, positive employment outcomes, and SSDI benefits reduction. In order to adequately assess these outcomes, we ask participants questions related to these outcomes, which include sensitive questions. One area related to improved clinical recovery is improved overall functioning, which includes reduction or elimination of alcohol and/or illegal substance use. Thus, we ask participants sensitive questions about their alcohol and/or illegal substance use at all of the data collection intervals (i.e., baseline, quarterly, and follow-up).

Section 234 of the Social Security Act gives the Commissioner of Social Security the authority to develop and carry out experiments and demonstration projects designed to determine the relative advantages and disadvantages of interventions that facilitate a beneficiary’s return to work. Consequently, the purpose of the MHTS is to test the extent to which the aforementioned intervention activities will lead to better employment outcomes and other benefits among SSDI beneficiaries with a primary impairment of schizophrenia or affective disorder. Reduction or elimination of alcohol and substance use is a significant benchmark for improved clinical recovery, which can invariably lead to improved employment outcomes. Consequently, it is necessary and appropriate to assess change in alcohol and illegal substance use to adequately measure the impact of the intervention activities with the goal of improved clinical recovery.

The informed consent form provided participants with the explanation for the study and explained that they may refuse to answer any question at any time. Moreover, participants attended a research information group meeting that provided details on the purpose of the study and its data collection activities, including the sensitive and personal questions.

Estimates of Burden

We reported estimates of the hour burden for the participants/respondents are below in Exhibits 2 through 5. Exhibit 2 provides the total burden hours for the screening process. Exhibit 3 provides an estimate of hour burden for the treatment group. Exhibit 4 provides an estimate of hour burden for the control group and Exhibit 5 are all survey totals. We developed the MHTS questionnaires using existing items from reliable and valid instruments, many of which we specifically developed for individuals with severe psychiatric disabilities. Based on previous experience with these instruments, we estimate that it will take approximately 30 minutes for respondents to complete the baseline interview and 20 minutes for respondents to complete the follow-up interview. For the quarterly interviews, we estimate it will take the treatment group approximately 25 minutes to complete, while only taking the control group 10 minutes to complete.

Exhibit 2. Screener Estimated Burden

Questionnaire

Number of Respondents

Frequency of Response

Total Number of Responses

Burden per Response

(minutes)

Total burden (hours)

Screener Survey

2,265

151

Exhibit 3. Estimated Burden for Treatment Group

Questionnaire	Number of Respondents	Frequency of Response	Total Number of Responses^²	Burden per Response (minutes)	Total Burden (hours)
Baseline	1,121	1	1,121	47	878
Quarterly	1,121	7	7,847	18	2,354
Follow-up	1,121	1	1,121	30	561
Total			10,089		3,793

Exhibit 4. Estimated Burden for Control Group

Questionnaire	Number of Respondents	Frequency of Response	Total Number of Responses	Burden per Response (minutes)	Total Burden (hours)
Baseline	1,117	1	1,117	47	875
Quarterly	1,117	7	7819	7	912
Follow-up	1,117	1	1,117	30	559
Total			10,053		2,346

Exhibit 5. Total Estimated Burden for All Study Activities

Questionnaire	Number of Respondents	Frequency of Response	Total Number of Responses	Burden Per Response (minutes)	Total Burden Hours
Screener Survey	2,265	1	2,265	4	151
Treatment Group	1,121	9	10,089	(see above)	3,793
Control Group	1,117	9	10,053	(see above)	2,346
Total			4,503		6,290

13. Estimated Annual Cost Burden to Respondents

There are no costs to respondents beyond those shown in item 12 above.

14. Estimated Annual Burden to Federal Government

The estimated cost of data collection efforts associated with the burden described in item 12 (estimate of burden) is comprised of 4 separate costs including (1) costs associated with actual data collection from study participants, (2) costs associated with training data collectors, (3) costs associated with developing the questionnaires, training materials, and conducting training, and (4) administrative costs.

The estimated total cost of data collection is $702,456. The estimated cost for Year 1 is $232,046, including $16,695 in direct data collection, $73,103 in administrative costs, $17,248 in training costs, and $125,000 in preparation costs. The estimated cost for Year 2 is $206,266, including $49,850 in direct data collection and $146,206 in administrative costs. The estimated cost for Year 3 is $177,916, including $26,319 in direct data collection costs and $146,206 in administrative costs. The estimated cost for Year 4 is $86,228, including $10,894 in direct data collection and $73,103 in administrative costs.

15. Program Changes in Burden/Cost Estimates

We modified the burden estimates due to a) reduction in the final sample size and b) revised hours per response. We reduced the final sample size from 3,000 to 2,238, and modified the hours per response to reflect actual averages. Therefore, we reduced the total burden hours by approximately 29%.

We have modified the estimate costs of data collection efforts due to the reduction in the sample size. This resulted in a change in costs for direct data collection in years 2, 3, and 4.

16. Plans/Schedules for Tabulation and Publication

The MHTS was originally planned as a 4 ½ year randomized trial of a demonstration. However, due to delays in recruitment startup and an extension of the recruitment period, we added a sixth phase to the contract. As a result, the contract will continue through January 31, 2011. The six phases of implementation are:

Phase I (months 1-12): Demonstration site recruitment, development of data collection instruments and data management system, and participant (initial) and intervention.
Phase II (months 13-24): Participant (final) and intervention.
Phase III (months 25-36): Intervention.
Phase IV (months 37-48): Intervention.
Phase V (months 49-54): Intervention and preliminary analysis.
Phase VI (months 58- 64): Analysis and report

Data Analysis

Data analyses will include the following five components: a) treatment intervention impact analysis; b) assessment of service costs and related intervention costs; c) implementation analysis; d) participation analysis; and e) within treatment intervention group analyses.

A key underlying principal in evaluating the impact of the MHTS intervention will be intent-to-treat analysis. In this analysis, all subjects enrolled in the treatment intervention and control groups will be included (except for those on whom the relevant outcome measure is unavailable).

Treatment Intervention Impact Analyses

The key element in this process is to compute differences in the mean values of the outcome variables between treatment and control groups without adjustment for covariates, and confidence intervals around these differences. In some cases, we will use transformations of outcome variables to allow application of parametric hypothesis tests. In view of the large size of the treatment and control groups, we expect that randomization will produce equivalence between these groups among potentially relevant covariates.

We use several different data structures to test for MHTS effects. For outcome variables based on SSA administrative data that are available at multiple time points in the study, we employ repeated measures of outcomes for each subject. For outcome measures derived from the quarterly and follow-up interviews, treatment intervention vs. control comparisons will use a cross-sectional data structure (i.e., one observation per subject). We use the cross-sectional design for outcome variables at multiple time points to test for MHTS effects that vary over time. As an example, we will examine MHTS effects on earnings at 12 months and at 24 months after enrollment in separate cross sections to allow for the possibility that these effects increase (or decrease) with longer follow-up periods. Confidence intervals for these effect estimates will take into account the clustered structure of the data.

The specific statistical tests used for estimated MHTS effects will depend on the distribution of the outcome variable under study. In some cases, where outcome measures are counts, we use discrete distributions such as the negative binomial to describe the data. For some measures, such as earnings amounts, one can view the outcome as having a truncated continuous distribution (such as a truncated normal). Appropriate distributional assumptions will be required for parametric statistical tests (e.g., tests based on t-distributions or likelihood ratios). We can also compute nonparametric tests via a bootstrapping procedure.

We test interaction effects by partitioning the study population into subgroups of interest (e.g., subjects not yet covered by Medicare at baseline, subjects in sites with low employment rates) and then estimate separate MHTS effects within each of these subgroups. We view results of these analyses with caution, however, because of their limited statistical power. In addition, for subgroups that are relatively small in size, there is a greater risk that randomization will not produce equivalent treatment and control groups within each subgroup. In this event, we will utilize adjustment (via regression or propensity score weighting) for increasing the precision of the estimated MHTS effects.

We calculate estimates of treatment and interaction effects both with and without statistical controls for covariates (such as the subject’s work history, gender, or age). We select the covariates included in regression or analysis of variance methods for estimating treatment and interaction effects based on evidence of their relevance from prior research. In addition, the extent of inter-subject variance of values for each particular covariate will be considered (since inclusion of covariates with very little inter-subject variance may substantially inflate the variances of our effect estimates).

Assessment of Service Costs and Related Intervention Costs

The cost analysis will (1) document the levels of costs for various types of services provided in the MHTS and (2) describe the relationships between beneficiary characteristics and variations in these cost levels. These analyses will provide information on the costs for providing services on a national level and how the costs would vary for different beneficiary subgroups. We draw data for the cost analysis from claims for payment submitted by service providers to the MHTS. We obtain additional data for costs of non-billed MHTS components that pertain to implementation from contractor and subcontractor accounting data.

Implementation Analysis

The principal objective of the implementation analysis is to document the degree to which service delivery to beneficiaries demonstrates a high degree of fidelity with the appropriate evidence-based models. While the ongoing quality assurance activities of the care coordinators will include consideration of conformance between evidence-based practices and actual service delivery for each individual beneficiary in the treatment intervention group, the implementation analysis will examine the same question at the level of the service provider. In the case of employment service providers, fidelity to the IPS-supported employment scale will be the standard against which we measure provider performance. We will construct scale ratings on a quarterly basis by the care coordinators in each of the sites. Demonstration site-level scores will be averages of client-level scores for all the clients of each program. In addition, we will present descriptive data on standard deviations, minimums, and maximums for each program of the client-level scores and we will compute the percentage of clients whose scores fall below a high-fidelity threshold (e.g., 70% used in previous studies).

We base descriptive data on mental health service provider implementation of evidence-based practices on the two related client-level scores: the Med-MAP fidelity scale and the Individualized Treatment Plan fidelity scale. In both cases, we will compute descriptive statistics similar to those described above for supported employment again at the program level based on the summaries of the client-level scores for each mental health provider.

Analysis of these quarterly data on fidelity scores will be descriptive. The descriptive analysis will examine whether we have maintained, or even increased, the high level as the study progresses. We do not anticipate any detailed statistical analysis of inter-provider variations in fidelity. Since the number of providers of employment services will be small and the inter-provider variations in scores will be small, complex modeling of the relationships between provider characteristics, site characteristics, and fidelity scores will not be feasible. We carry out descriptive cross-tabulations of fidelity by selected provider characteristics (such as organizational size and sponsorship).

One additional topic covered in the implementation analysis will be a descriptive report on the provider recruitment-contracting process. This will describe steps taken to develop and execute the provider MHTS contracts, along with issues that come up in the contract negotiation process.

Participation Analysis

The participation analysis will examine participation by beneficiaries and participation by employers. The principal objective of these analyses is to identify major obstacles that encountered in replicating the MHTS on a larger scale.

The beneficiary participation analysis will concern three different types of beneficiary decisions: (1) the decision of beneficiaries to express interest in participating and screening for eligibility, (2) the decision of those who are eligible to provide informed consent prior to randomization, and (3) the decision to continue participation following randomization. The objectives of the analysis areas follow:

(1) to provide information on take-up rates that are generalizable to the national level;

(2) to identify the groups of potentially eligible beneficiaries who are most difficult to enroll; and

(3) to document differential attrition between the treatment and control groups that may result in unbalanced average characteristics at the group level and that may result in bias in our estimates of intervention impacts.

We will use multiple profit or logistic regression models to identify the beneficiary characteristics that predict the decision to undertake the eligibility interview. We obtain available data on beneficiaries for this analysis from SSA administrative files, including the Master Beneficiary Record. Beneficiary characteristics available from SSA for this analysis include demographics (age, race, and gender), impairment (diagnosis) category (schizophrenia, bipolar disorder, etc.), history of SSDI and SSI benefit payments, countable earnings and non-earned other income (for SSI recipients), date of disability onset, date of application for benefits, evidence of secondary impairments and drug and alcohol dependence, and history of SSA-covered earnings. We will also test for site-specific differences in beneficiary decisions. We also compute bivariate tests comparing participants and non-participants for each of the variables included in the regression models.

We will develop descriptive information on employer contacts from the quarterly reports on each treatment group beneficiary filed by the employment service provider. This information will include descriptive data (for non-working or newly placed beneficiaries) on the numbers of employers contacted for placing a beneficiary and brief descriptions of the contacted employers (type of business, type of ownership). It will also include brief descriptions of the employers interviewing or receiving an application from the beneficiary, as well as indications of which employers offered employment to the beneficiary. In addition, in the case of involuntary terminations, employment services providers (with permission from the beneficiary) may contact the employer to identify the reasons and problems that led to the involuntary termination. We provide basic characteristics of these employers in the quarterly report on each beneficiary.

The outcome analysis team conducts descriptive information and appropriate statistical summaries and compiles them into a yearly report on service provider – employer interactions and beneficiary – employer interactions.

Within Treatment Intervention Group Analyses

We also conduct impact analyses on additional outcome measures that are only available on a repeated measures basis within the treatment intervention group. These outcome measures are the ones based on items contained in the quarterly treatment group interviews (except for the binary employment status measure that we also collect quarterly for control group subjects). These include employment-related items from the Vocational Update Form and other outcome measures that are only available from the quarterly reports on the treatment group. Auxiliary outcomes analyzed on a quarterly basis for treatment group subjects include educational attainment, alcohol use, and drug use.

Statistical Approach

We model repeated measures of the above outcomes for the treatment group using random-effects regressions that allow for clustering of random components at the level of the individual and the site. Explanatory variables will include subject characteristics measured at baseline (that are not time varying), time-varying site characteristics (such as unemployment rate or other measures of labor market conditions), and program characteristics (some of which will vary over time).

Each of the outcomes are measured repeatedly at baseline and then quarterly (the last includes the follow-up interview) over two years for 9 time points. We assumed each outcome is continuous. The main analysis will consist of repeated measures Analysis of Variance (ANOVA). We will use SAS PROC MIXED in the analysis for its repeated measures capability. We use contract statements in PROC MIXED to compare changes in these measures from baseline to each of the subsequent follow-up times or to test for trend in the mean outcome profiles. The type and amount of missing data affects the analysis.

Dissemination of Findings

We will prepare the following types of reports: annual reports, final report, general-use presentation materials, manuscripts in peer-reviewed journals, and brief project summaries. We will also present on findings at various meetings held by SSA.

17. Expiration Date Omission Approval

SSA does not request an exemption to printing the expiration date for OMB approval on the materials associated with this new collection of data.

18. Exceptions

SSA is not requesting an exception to the certification requirements at 5 CFR 1320.9 and related provisions at 5 CFR 1320.8(b)(3).

1 Please see attached Construct of Quality Assurance Activities.

2 The number of respondents may reduce over time due to study withdrawals.

File Type	application/msword
File Title	--PAPERWORK REDUCTION ACT SUBMISSION
Author	Annmarie Winkler
Last Modified By	177717
File Modified	2009-05-26
File Created	2009-05-26