Part 1 -- Supporting Statement

SAMHSA/CMHS Initiative to Reduce/Eliminate Seclusion and Restraint:

Data Collection and Analysis for the Alternatives to Restraint and Seclusion Grant Program

A. Justification

1. Circumstances of Information Collection

The Substance Abuse and Mental Health Services Administration’s (SAMHSA) Center for Mental Health Services (CMHS) is requesting OMB approval for a revision to the evaluation of the State Incentive Grants to Build Capacity for Alternatives to Restraint and Seclusion (short title: Alternatives to Restraint and Seclusion). The current approval is OMB No. is 0930-0271, which expires on November 30, 2008.

This evaluation will utilize three separate data collection instruments the: 1) Facility/Program Characteristics Inventory (FPCI) (Attachment A); 2) Inventory of Seclusion and Restraint Reduction Interventions (ISRRI) (Attachment B); and the 3) Seclusion and Restraint Event Data (SRED) Matrix (Attachment C). Both the Facility/Program Characteristics Inventory and the Inventory of Seclusion and Restraint Reduction Interventions are OMB approved. The Seclusion and Restraint Event Data Matrix is a new instrument.

The grants are designed to promote the implementation and evaluation of best practice approaches to reducing the use of restraint and seclusion in mental health facilities. Grantees consist of 8 sites (state mental health agencies), all of which will be implementing interventions in multiple facilities (a total of 21 facilities). These include facilities serving adults and those serving children and/or adolescents, with various subgroups such as forensic and sexual offender populations.

The SAMHSA Initiative to Reduce/Eliminate Seclusion and Restraint program authority is Title 4. Public Health and Welfare, Chapter 6A – Public Health Service, Subchapter III A – Substance Abuse and Mental Health Services Administration, Part A Organization and General Authority, as amended, 42 U.S.C. 290aa et seq., The Children’s Health Act of 2000, Part H, Requirement Relating to the Rights of Residents of Certain Facilities [42 U.S.C. at 290ii -290ii-2], and Part I, Requirement Relating to the Rights of Residents of Certain Non Medical, Community-based Facilities for Children and Youth [42 U.S.C. at 290jj-1 – 290jj-2], and the Protection and Advocacy for Individuals with Mental Illness Act of 1986, as amended, [42 U.S.C. 10801 et seq.]. SAMHSA evaluation studies are authorized by Section 501(d) (4) of the Public Health Service Act (42 USC 290aa).

SAMHSA is in the process of implementing National Outcome Measures (NOMs) in all of its programs. None of the current ten NOMs directly apply to this grant. However, this information collection request will be gathering data on specific performance measures that relate to the targeted focus.

2. Purpose and Use of Information

This evaluation study will collect data on the: 1) number of facilities and programs adopting best practices involving alternative approaches to restraint and seclusion, including staff training models and other multi-faceted approaches; and 2) the program’s impact of reducing restraint and seclusion use and adoption of alternative practices. The evaluation will utilize three instruments:

1. The Facility/Program Characteristics Inventory will be collected at baseline and provide information about the types of facilities/programs, characteristics of persons served, staffing patterns, and unit specific data (Attachment A). This information will be provided by designated staff of grantee states, who will complete the inventory on-line;

2. The Inventory of Seclusion and Restraint Reduction Interventions (ISRRI) will provide information about components of the interventions that are implemented (Attachment B). The ISRRI will be collected four times, once to capture baseline and three additional times approximately each year throughout the 3-year grant period. Designated respondents of the grantee states will review the ISRRI Reviewers’ Guide (Attachment I) which will be utilized to complete the ISSRI survey. The ISRRI survey can be completed on-line;

3. The Seclusion and Restraint Event Data (SRED) Matrix will be collected monthly to collect outcome data about the use of restraint and seclusion within each facility or program (Attachment C). This information will come from states management information systems. Moreover, 17 of the 21 mental health facilities currently compile and provide this data monthly for accreditation reporting. Designated respondents of grantee states complete the SRED Matrix and submit it on-line.

CMHS received comments on these three instruments in response to the 60-day Federal Register Notice (FR Doc: E8–5570) (see Attachment J for comments). In response, CMHS made a number of changes to these instruments (Exhibit 1).

CMHS will utilize the data collected from the three instruments to identify effective practices for the reduction of restraint and seclusion. The data will also be utilized in support of the Government Performance and Results Act (GPRA).

Exhibit 1. Changes to Evaluation Instruments in Response to Comments on the 60-Day Federal Register Notice
Comment Question Number	Instrument	Instrument Question Number	Change Implemented (new text in italics)
1	FPCI (Section 1)	1	Changed “Free Standing Psychiatric Facility” to “Free Standing Psychiatric Hospital”
4	FPCI (Section IV)	Policy	Added the definitions of “seclusion” and “restraint”, which are the same as those used in our other instrument, the Seclusion and Restraint Event Data Matrix, and are based on CMS definitions.
7	ISRRI (Worksheet 1)	L.1.5	Added the following: “(If the consumer is a child/adolescent, this treatment planning process includes parents or guardians).”
8	ISRRI (Worksheet 1)	L.1.6	Added imminent harm to self. The question now reads: “A policy restricting the use of S/R to emergencies that reach the level of imminent risk of harm to self, staff or other consumers only.”
9	ISRRI (Worksheet 1)	L.2.2	Added additional descriptive text which now reads: “A policy supporting the adoption of principles of recovery and/or resiliency.”
15	ISRRI (Worksheet 2)	Immediate Post-Event Debriefing Description	Added additional text to the Description: “An immediate post-event debriefing that is done onsite after each seclusion or restraint event and is led by the senior on-site supervisor who immediately responds to the unit or area.”
18	ISRRI (Worksheet 2)	Immediate Post-Event Debriefing	Added another survey item: “Consumer who experienced event provided personal comments on his/her experience in person or by proxy (peer or parent/guardian).”
19	ISRRI (Worksheet 2)	D.2.11	Added the following clarifications in italics: “Release criteria are reasonable with the burden on staff, not consumer.”
22	ISRRI (Worksheet 2)	Formal Debriefing	Added another survey item: “Consumer who experienced event provided personal comments on his/her experience in person or by proxy (peer or parent/guardian).”
24	ISRRI (Worksheet 4)	W.2.7	Revised “consumer” to now read “consumers/parents/guardians/peers.”
26	ISRRI (Worksheet 5)	Emergency Intervention Description	Added text to read: “Policies and procedures for emergency seclusion and restraint interventions including:”
27	ISRRI (Worksheet 6)	Consumer Roles Description	The term “consumer” in the description text has been changed to read: “The full and formal inclusion of consumers (or parents/guardians for child/adolescent facilities)…”

Comment Question Number	Instrument	Instrument Question Number	Change Implemented (new text in italics)
30	SRED (Page 1)	Service Recipient Referral Source	The referral source categories have been revised. The category, “Voluntary – Community Based Program,” has been eliminated because it overlaps with the category, “Voluntary – Outpatient Provider (mental health/general medical).” Instead “Voluntary – Social Services/Community Residences” has been created. “Voluntary – Inpatient Provider” has been subdivided into “Voluntary – Inpatient Hospital” and “Voluntary – Inpatient Residential Treatment Center” to distinguish between inpatient hospital and residential treatment centers. All referral sources should now be more mutually exclusive. Respondents are now provided with examples of each category.
31	SRED (Page 3)	Definition - Seclusion	Removed “time-out” from the “Other Exclusions” list and have added the text “voluntary time-out” to the text in the 3rd paragraph.

3. Use of Information Technology

To minimize the burden to the states and facilities/programs and to ensure data quality, data will be submitted by the states and facilities/programs electronically via a secured web-site.

4. Efforts to Identify Duplication

The data will be collected specifically for the purpose of this program and is not available elsewhere.

5. Involvement of Small Entities

The information collected will not have a significant impact on small entities.

6. Consequences if Information Collected Less Frequently

Data for the Facility/Program Characteristics Inventory will be collected at baseline to assess the impact of facility/program characteristics on the implementation of seclusion and restraint reduction interventions and on seclusion and restraint rates. Data for the Inventory of Seclusion and Restraint Reduction Interventions will be collected at baseline and three other times towards the end of each year of the 3-year grant period. Data from this instrument will allow the attribution of changes in the use of restraint and seclusion to the implementation of components of the interventions. The Seclusion and Restraint Event Data Matrix will be collected monthly in order to examine monthly trends in seclusion and restraint rates. Utilizing a less frequent schedule of data collection than what is proposed will significantly impair the ability to analyze the data in timely fashion to determine impact of various intervention strategies.

7. Consistency with the Guidelines in 5 CFR 1320.5(d)(2)

The proposed evaluation is consistent with all guidelines in the 5 CFR 1320.5(d)(2).

8. Consultation Outside the Agency

The 60-Day notice required in 5 CFR 1320.8(D) was published in the Federal Register on March 20, 2008 (Vol. 73, pg. 14992). CMHS received comments in response to this notice (Attachment J). These comments generally inquired about (1) the comparability of the measures used in the current evaluation with those used in previous SAMHSA grants, and (2) specific definitions of terms and measures, and how they might apply to children/adolescents. These comments were addressed (Attachment K). In response, a number of revisions were made to the three evaluation instruments (Section A2, Exhibit 1).

The Facility/Program Characteristics Inventory and the Inventory of Seclusion and Restraint Reduction Interventions instruments have been utilized previously in the evaluation of the Alternatives to Restraint and Seclusion SIGs. Both instruments have been reviewed by the SIG Steering Committee. This SIG Steering Committee was comprised of nationally recognized experts, state and local mental health professionals involved with the states and facility/programs, and federal staff. The consultation showed that the data collection instruments were written using plain, coherent, and unambiguous terminology and are understandable to those who are to respond, and collected in the least burdensome way possible. Based on the consultation, changes were made to the instruments. See Attachment D for a list of individuals who reviewed these two instruments.

9. Payments to Respondents

No payments or gifts are provided to the respondents (states) for their participation in this data collection.

9. Assurance of Confidentiality

No individual/client-level data will be collected for the proposed evaluation. Instead, CMHS will receive only facility-level, aggregated data from state grantees. Thus, there is no risk of a security breach for any individual/client.

With respect to facility-level data, policies and procedures will be followed to insure the security of all data. These include the use of a Web-based data entry and file transfer system that uses industry standard secure socket layer data (SSL) encryption; and firewall protection against unauthorized access to data; Web access that requires use of assigned user names and passwords; with passwords to be  changed at regular intervals; data files that are password protected with access limited to only those individuals who have a need to work on them using a secure decryption key; and the data will be stored on a secure partition of a Windows-based server and as such will be strongly encrypted. Access to data on this server is username and password protected. 

9.  Questions of a Sensitive Nature

There are no questions of sensitive nature being collected.

9. Estimates of Annualized Hour Burden

The estimated annualized burden for the project is based on an analysis of the amount and type of data requested. The results, as presented by grant year in Table 1, were validated by a number of respondents from the states and facilities/programs.

Estimated response time for the three instruments is 18 hours. The annualized burden is 364 responses and 2,870 hours. The burden is an average for the states and facilities/programs to complete the questionnaire manually, produce centralized electronic files, and to submit the electronic files. Variation in burden may occur due to the complexity of interventions being implemented, organization of records, the availability of informed staff to complete the instruments, and other unpredictable factors. The instruments will involve 21 facilities ranging from state mental health hospitals to child and adolescent residential programs. The individual hourly wage was calculated using the 1999 mean earnings ($42,707) (QT-P31 Earnings in 1999 of Full-time, Year Round Workers, Census 2000 Summary File 4 (SF4)-Sample Data).

Table 1. Estimates of Maximal Annualized Hour Burden, by Grant Year

Grant Year 1

Instrument	No. of respondents	Responses per respondent	Total responses	Average hours per response	Total annual burden (hours)	Hourly wage rate	Total hourly cost
Facility/Program Characteristic Inventory	21	1	21	2	42	$20.53	$862.26
Inventory Of Restraint and seclusion Reduction Interventions	21	1	21	8	168	$20.53	$3,449.04
Seclusion and Restraint Event Data Matrix	Not given during Year 1a
TOTAL ANNUAL	21				210		$4,311.30

^a This instrument may be given during Year 1 pending timely OMB approval. If this is the case, some of the responses allotted to Year 2 may be shifted to Year 1 in order to lessen the burden to respondent burden.

Table 1 (Continued). Estimates of Maximal Annualized Hour Burden, by Grant Year

Grant Year 2

Instrument	No. of respondents	Responses per respondent	Total responses	Average hours per response	Total annual burden (hours)	Hourly wage rate	Total hourly cost
Facility/Program Characteristic Inventory	Not given during Year 2
Inventory Of Restraint and seclusion Reduction Interventions	21	1	21	8	168	$20.53	$3,449.04
Seclusion and Restraint Event Data Matrix	21	29	609	8	4,872	$20.53	$100,022.16
TOTAL ANNUAL	21				5,040		$103,471.20

Table 1 (Continued). Estimates of Maximal Annualized Hour Burden, by Grant Year

Grant Year 3

Instrument	No. of respondents	Responses per respondent	Total responses	Average hours per response	Total annual burden (hours)	Hourly wage rate	Total hourly cost
Facility/Program Characteristic Inventory	Not given during Year 3
Inventory Of Restraint and seclusion Reduction Interventions	21	2	42	8	336	$20.53	$6,898.08
Seclusion and Restraint Event Data Matrix	21	18	378	8	3,024	$20.53	$62,082.72
TOTAL ANNUAL	21				3,360		$68,980.80

9. Estimates of Annualized Cost Burden to Respondents

There are no capital, startup costs, operations, or maintenance costs to the states and facilities/programs associated with this project.

9. Estimated of Annualized Cost to the Government

This data collection will be conducted over 3 years for which the total cost of the contract is $482,656 and the average annualized cost is $160,886. The estimated annual cost for Federal staff time is $36,667 which represents 1/3 FTE. Thus, the total government cost is $592,656 and the average annualized cost is $197,553. CMHS has planned and allocated resources for the efficient and effective management and use of the information to be collected, including the processing of the information in a manner which shall enhance, where appropriate, the utility of the information to agencies and the public.

9. Changes in Burden

Currently, the total burden was 9,708 hours. CMHS is requesting 2,870 hours for the current project. The decrease of 6,838 hours is due to a programmatic change in the number of mental health facilities participating in the evaluation from 49 to 21 facilities.

9. Time Schedule, Publication and Analysis Plans

Table 2 shows the time schedule for the evaluation of the best practice approaches to reducing and ultimately eliminating the use of restraint and seclusion in mental health facilities.

Table 2. Time Schedule

ACTIVITY	PLANNED START TIME
Train states and facilities/programs on submitting FPCI and ISSRI #1 data via secured website.	March 2008
Collect baseline data for FPCI and ISSRI #1.	April 2008
Develop quarterly data reports.	June 2008
Train states and facilities/programs on submitting SRED Matrix data via secured website.	OMB approval + 2 week
Collect data for SRED Matrix (after initial collection it will be collected from each site at a scheduled date on a monthly basis).	OMB approval + 4 weeks
Train states and facilities/programs on submitting ISSRI #2 data via secured website.	September 2008
Collect data for ISSRI #2.	October 2008
Train states and facilities/programs on submitting ISSRI #3 data via secured website.	September 2009
Collect baseline for ISSRI #3.	October 2009
Train states and facilities/programs on submitting ISSRI #4 data via secured website.	April 2010
Collect data for ISSRI #4.	April 2010
Report summarizing findings and recommendations.	June 2010
Deliver final files to SAMHSA.	September 2010
Send project documentation.	September 2010

Note: the use of the instruments noted in Activities 1–2 and 6–7 has been approved by OMB through November 30, 2008.

Analysis Plan

The research questions to be answered by the evaluation are:

1. Do the models (i.e., interventions based upon evidence described in the literature [“best practices”]) have a positive effect in reducing rates of restraint or seclusion (i.e., the outcome variables)?

2. Does the magnitude of this effect vary (be moderated) according to characteristics of facilities/programs (mission, size, ownership, etc.) and of the population they serve (e.g., demographic characteristics)?

3. Is the extent to which components of model interventions are implemented as planned influenced by organizational characteristics, staffing patterns, and service recipient characteristics?

4. Does the magnitude of the reduction effect vary (be mediated) according to the extent to which components of interventions were fully implemented (fidelity)?

5. Does the magnitude of the reduction effect vary according to a combination of site characteristics (moderating variables) and extent to which components of interventions (mediating variables) were fully implemented (fidelity)?

Prior to analysis of specific hypotheses, univariate, bivariate, and multivariate analyses of all variables (moderating variables, intervention components, mediating variables, and outcome components) will be examined overall, by state, by intervention type, and by facility/program prior to intervention implementation and at the last time point measured at the facility/program. Tables 3 and 4 are examples of how the data for these analyses will be displayed. Bivariate and multivariate analyses may include comparisons of baseline variables and demographics within and between states to identify important covariates and confounding variables prior to examining the primary hypotheses.

Hierarchical models may be used to examine the effect of intervention components on outcome variables. The number and type of best practices used will be incorporated into the models as well as the duration of the implementation. Fidelity to intervention components will be incorporated as possible confounders to the effect of the intervention. Moderating variables, including facility and program information, service recipient information, and staffing information, will be included as covariates in the models.

The examination of changing outcome results from initial implementation to final assessment will be addressed in several analyses. The significance of the slope of the outcome variables will be tested to assess trends in outcomes. Outcome variables will be tested pre and post implementation and to control for baseline levels. If possible, states and facilities/programs with late implementation will be used as a control group to compare to facilities/programs with interventions already implemented over the same time period.

Table 3. Mean Rates Pre (12 months prior to intervention), Post (24 months post intervention) and % Change

	Seclusion Hours			Seclusion Episodes			Restraint Hours			Restraint Episode
	Pre	Post	% Change	Pre	Post	% Change	Pre	Post	% Change	Pre	Post	% Change
State
State 1
State 2
State 3
State 4
State 5
State 6
State 7
State 8
Type of Facility/Unit
Hospital
Residential Treatment Facility
Other
Unit Mission
Adult Acute
Adult Continuing Care
Adult Combined
Child and/or Adolescent Acute
Child and/or Adolescent Continuing Care
Child and/or Adolescent Residential
Specialty of Facility/Unit
General
Forensic
Other Specialty
Number Served
Small
Medium
Large

Table 4. Mean Fidelity to Intervention Best Practice: Components

	Leadership	Use of Data	Workforce Development	Use of Tools	Consumer Involvement	Oversight	Debriefing
State
State 1
State 2
State 3
State 4
State 5
State 6
State 7
State 8
Type of Facility/Unit
Hospital
Residential Treatment Facility
Other
Unit Mission
Adult Acute
Adult Continuing Care
Adult Combined
Child and/or Adolescent Acute
Child and/or Adolescent Continuing Care
Child and/or Adolescent Residential
Specialty of Facility/Unit
General
Forensic
Other Specialty
Number Served
Small
Medium
Large

9. Display of Expiration Date

The expiration date will be displayed.

9. Exceptions to Certification Statement

This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions. The certifications are included in this submission.

B. Collection of Information Employing Statistical Methods

1. Respondent Sampling Frame and Sampling Methods

A full enumeration of all facilities/programs (21) of states (8) implementing the best practices to reduce restraint and seclusion is planned; therefore, there is not a need for a sampling plan. Table 5 notes the States that are part of the Alternatives to Restraint and Seclusion SIG Program and the number of facilities in which the best practices to reduce restraint and seclusion are planned.

Table 5

States and Number of Facilities Implementing Best Practices

Alternatives to Restraint and Seclusion

STATE	NUMBER OF FACILITIES/ PROGRAMS
Connecticut	2
Indiana	2
New York	3
New Jersey	4
Oklahoma	2
Texas	4
Vermont	2
Virginia	2

2. Information Collection Procedures

In February 2008, staff from the states and facilities/programs responsible for reporting data for the Facility/Program Characteristics Inventory and the Inventory of Seclusion and Restraint Reduction Interventions was mailed an introductory letter introducing the project (Attachment E) which will include copies of these OMB approved instruments. Instructions for Data Entry/Submission for these instruments were mailed to the Data Contact for each state and posted on the project website (Attachment G). In March 2008, staff from the states and facilities/programs responsible for reporting data for the two instruments were trained on submitting the data via a secured website. In April 2008, the states and facilities/programs submitted an online baseline survey of the Facility/Program Characteristics Inventory and the Inventory of Seclusion and Restraint Reduction Interventions. Immediately after OMB approval, facility/program staff responsible for data reporting will receive an introduction letter (Attachment F) notifying them of the data submission schedule for the Seclusion and Restraint Event Data Matrix. Instructions for Data Entry/Submission for this instrument will be mailed to the Data Contact for each state and posted on the project website (Attachment H). The Data Contact will then receive training on submitting these data and will begin submitting these data two and four weeks after OMB approval, respectively. After the initial submission, the Seclusion and Restraint Event Data Matrix will be submitted at a scheduled date on a monthly basis. The Inventory of Seclusion and Restraint Reduction Interventions will be submitted three additional times on October 2008, October 2009, and April 2010.

3. Methods to Maximize Response Rates

CMHS expects to receive data from all 8 states since it is a condition of their grant. It is projected that at least 80% of facilities/programs will submit the required data elements. Since the Seclusion and Restraint Event Data Matrix relies heavily on already existing data sources collected for accreditation reporting, CMHS is increasing the likelihood that data will be submitted by a vast majority of facilities/programs. Each state and facility/program has identified a data contact that will be emailed along with the project director a week prior to the data submission deadline (baseline, monthly and yearly). A week after the data submission deadline, reports will be generated to identify states and facilities/programs that did not submit data. The data contact and project directors will be contacted one more time by email, and if not responsive, will be contacted by phone.

4. Tests of Procedures

Data submission procedures for the Facility/Program Characteristics Inventory and the Inventory of Seclusion and Restraint Reduction Interventions have been used extensively during the evaluation of the previous SAMHSA Alternatives to Restraint and Seclusion grant, and the same procedures will be used for the evaluation of the current grant. As previously noted above, the Seclusion and Restraint Event Data Matrix relies heavily on data already collected for accreditation reporting, and therefore, the procedures are well established.

5. Statistical Consultants

CSR Incorporated will have overall responsibility for implementation and execution for the evaluation and for the preparation of all reports. The following individual is the director of the project and will conduct statistical analysis for the project:

Daniel Falk, Ph.D.

Senior Research Associate

CSR, Incorporated

2107 Wilson Boulevard, Suite 1000

Arlington, VA 22201,

Phone: (703) 741-7124

Email: dfalk@csrincorporated.com

The SAMHSA Project Officer is:

Nichole Washington, MHSA

Public Health Advisor

Division of State and Community Systems Development
Center for Mental Health Services

SAMHSA

1 Choke Cherry Road, Rm 2-1108
Rockville, MD 20852
Phone: (240) 276-2719

Email: nichole.washington@samhsa.hhs.gov

No outside statistical consultants will be utilized.

List of Attachments

Attachment A: Facility/Program Characteristic Inventory (FPCI)

Attachment B: Inventory of Seclusion and Restraint Reduction Interventions (ISRRI)

Attachment C: Seclusion and Restraint Even Data (SRED) Matrix

Attachment D: Consultation Outside the Agency for FPCI and ISRRI

Attachment E: Introductory Letter for FPCI and ISRRI

Attachment F: Introductory Letter for SRED Matrix

Attachment G: Instructions for Data Entry/Submission for FPCI and ISRRI

Attachment H: Instructions for Data Entry/Submission for SRED Matrix

Attachment I: Inventory of Seclusion and Restraint Reduction Interventions (ISRRI)

Reviewers’ Guide

Attachment J: 60-Day Federal Register Notice Comments

Attachment K: Responses to 60-Day Federal Register Notice Comments

2