Form Appendix II - Surv Appendix II - Surv Appendix II - Surveys/Interviews

National Evaluation the Protection and Advocacy for Individuals with Mental Illness (PAIMI) Program

Attachment II PAIMI Evaluation Plan with Appendices

Executive Director Interview

OMB: 0930-0287

Document [doc]

Download: doc | pdf

Attachment II

EVALUATION OF THE PROTECTION AND ADVOCACY FOR INDIVIDUALS WITH MENTAL ILLNESS (PAIMI) PROGRAM

PHASE II:

EVALUATION PLAN

SUBMITTED BY:

HUMAN SERVICES RESEARCH INSTITUTE

John Hornik, Ph.D.

Virginia Mulkern, Ph.D.

Elizabeth Pell, M.A., M.S.W.

Faith Sabel, M.P.H.

Deborah Potter, M.A.

Valerie Bradley, M.A.

Judi Chamberlin

Stephan Haimowitz, J.D.

Leslie Scallet, J.D.

SUBMITTED TO:

THE CENTER FOR MENTAL HEALTH SERVICES (CMHS)

April 19, 2006

Report for Contract # 280-2003-2100

Table of Contents

INTRODUCTION
Conclusions of the Evaluability Assessment	1
Evaluation Strategy and Goals	3
PAIMI Detailed Logic Model	5
CHAPTER I: EVALUATION QUESTIONS
Overview	6
Clusters of Evaluation Questions	7
A. Inputs	7
B. Effectiveness and Efficiency of the PAIMI Planning, Priority Setting, and Application Processes	7
C. Effectiveness and Efficiency of Federal Guidance and Review of Applications	9
D. Federal Monitoring	9
E. Federal Technical Assistance (TA) and Other Support	11
F. State PAIMI Implementation	11
G. Process Outcomes	13
H. Cross-Cutting Issues	14
CHAPTER II: EVALUATION MODULES
Available Sources of Data	15
A. PPR Review and Analysis: Goals & Objectives Section	17
B. PPR Review and Analysis: Data Tables	20
C. PPR Review and Analysis: Budget Information	23
D. Advisory Council Reports	23
E. PPR Peer Review Findings	23
F. CMHS Site Monitoring Reports & Tools	25
G. PAIMI Annual Reports to Congress	25
H. NDRN Technical Assistance Conference Evaluations	25
Additional Data Needed	26
A. Data Management	26
B. Online Survey Methodology	27
C. Surveys	28
1. P&A Executive Directors Survey	28
2. PAIMI Advisory Council Chairs Survey	29
3. PAIMI Client Survey & Focus Group Meetings	29
4. State Mental Health Authority (SMHA) Directors Survey	31
5. Other State Mental Health Organizations that Relate to PAIMI	32
D. Key Informant Interviews Related to Federal Oversight Activities	32
1. CMHS PAIMI Project Officers	33
2. NDRN Technical Assistance (TA) Staff	33
3. PAIMI Site Monitors	34
4. PAIMI Peer Reviewers	34
E. Site Visit to Pilot Evaluation Processes and Instruments	34
CHAPTER III: SCHEDULE OF WORK & ESTIMATED COSTS
Table 1: PAIMI Task & Timing Chart	37
Expected Cost of Evaluation	38
Table 2: Estimated Costs of PAIMI Evaluation Modules	38
CHAPTER IV: EVALUATION REPORT DISSEMINATION
Distribution of the Final PAIMI Evaluation Report	40
Briefings on the Final PAIMI Evaluation Report	40
AttachmentS
SURVEYS
Attachment A: P&A Executive Directors	42
Attachment B: PAIMI Advisory Council Chairs	64
Attachment C: PAIMI Client Survey	81
Attachment D: State Mental Health Directors	89
Attachment E: State Mental Health Advocacy Organization Director’s Survey	98
INTERVIEWS
Attachment F: CMHS PAIMI Project Officers	108
Attachment G: NDRN Technical Assistance Staff	111
Attachment H: PAIMI Site Monitors	112
Attachment I: PAIMI PPR Peer Reviewers	113

Introduction

Consistent with federal efforts to address issues of accountability, capacity, and effectiveness (such as the Program Assessment Rating Tool (or PART)), HSRI and our partners have undertaken a three-stage approach to independently evaluate the Protection and Advocacy for Individuals with Mental Illness (PAIMI) Program. First, an Evaluability Assessment (EA) has been conducted and is being followed by the proposed Evaluation Plan presented in this document. As a next step, the National Evaluation of the PAIMI Program will be completed. Together these efforts address the three objectives outlined in the RFTOP:

To assess the feasibility of evaluating the PAIMI Program, HSRI undertook an EA study and recommended a series of evaluation design modules (with specific performance measures) from which SAMHSA could undertake a National Evaluation of the PAIMI Program;
To evaluate various structures, processes, and outcomes associated with the PAIMI Program, HSRI first incorporated feedback from stakeholders to craft a Logic Model of how the PAIMI Program operates; then used that model to select among research questions and performance measures to use in the National Evaluation; and finally, used the model to guide the design and implementation of the National Evaluation; and
To write a comprehensive evaluation report which focuses on the program’s strengths, challenges, and opportunities for bolstering our country’s protection and advocacy system for individuals with psychiatric disabilities, including children with significant emotional impairments, HSRI focused on policy-specific questions that reflected the concerns of a variety of stakeholders and were amenable to change.

Conclusions of the Evaluability Assessment

The activities proposed for a National Evaluation of the PAIMI Program are based on the results of the Evaluability Assessment of the PAIMI Program. The Federal government, in particular the Public Health Service (PHS), has recognized that evaluation research is central to guiding program design, assessing performance, and supporting knowledge development and dissemination (Johnson, 1996; Crane and Ginsburg, 1996; GAO, 2000). PHS agencies and offices have supported a range of evaluation activities including process, impact, and outcome evaluations. In all of these endeavors, especially when large and/or complex programs with multiple stakeholders (such as the PAIMI Program) are being assessed, formal evaluability studies provide an important, initial activity and assess the degree to which the program is evaluable.

Therefore, as a first stage in assessing the PAIMI Program, HSRI and our partners completed an Evaluability Assessment (EA). Evaluability Assessment is a distinct and formal analytic endeavor which asks the fundamental question of whether or not a program may be formally evaluated in a meaningful way.^¹ Evaluability studies provide an opportunity to consider “ethical, methodological, and practical dilemmas” (Scheirer, 1998) and move stakeholders forward as they choose among several alternative designs for conducting a subsequent evaluation, should at least one design be offered that has the potential for producing meaningful, valid, and practical results at a reasonable cost. At their core, EA studies include intended users of the evaluation (policymakers, program staff, consumers and others) and contribute to meaningful and efficient evaluations by gaining consensus among these stakeholders in specifying evaluation criteria.

The EA was an important first step for the National Evaluation to assess the readiness of the PAIMI Program to undergo evaluation, according to the following criteria:

Level of agreement among key stakeholders on program goals and objectives;
Feasibility of achieving these goals and objectives, given current inputs; and
Adequacy of currently collected information (including reliability and validity of data, data completeness, comparability of data across states, and timeliness of information).

During the Evaluability Assessment we completed the following steps:

The intention of the program was clarified. Stakeholders identified and defined program goals, objectives, unintended effects, performance criteria, and other programmatic information needs.
As a result, both program models and performance indicators were identified.
Evaluators explored the reality of the program’s operations and compared these with the intended program design. As differences in stakeholders’ expectations were identified and the programmatic reality was clarified, the logic model being developed by HSRI and our partners was revised.
Not all aspects of a program were identified for evaluation (Strosberg and Wholey, 1983). Rather the following criteria were used to identify those aspects of the PAIMI Program which would be amenable to evaluation:
1. Existing performance measures are available and provide meaningful information about PAIMI;
2. The cost of using existing data and/or collecting new data is minimized;
3. New data collection strategies place minimal burden on SAMHSA, grantees and other stakeholders; and
4. Selected areas of interest are policy relevant and are “ready to be managed to achieve desired performance and outcomes” (Schmidt et al 1982).
Finally, using the above criteria, stakeholders agreed that the PAIMI Program was evaluable. The sources of data were found to be rich and diverse; strategies for minimizing costs by using available data were identified and cost effective methods for collecting new data were identified; and policy relevant/actionable facets of the PAIMI Program were highlighted. Guided by these criteria, the project team, with guidance from the Evaluation Advisory Workgroup, chose among several evaluation design options to recommend to SAMHSA. In making the selection, we reached agreement on evaluation priorities and on subsequent use of information collected in the evaluation.

In addition, the EA concluded that a National Evaluation of the PAIMI Program presents an important opportunity. The PAIMI Program affords the Federal government a distinct opportunity to monitor various protection and advocacy processes and trends in state mental health systems. Such an overview can provide vital information in guiding subsequent policy direction at the national level. For states, the PAIMI Program provides a distinct tool to bolster system change in mental health protection and advocacy, an over-riding policy concern in many state and federal mental health initiatives. An evaluation of the PAIMI Program has the potential to provide valuable feedback to a variety of stakeholders as our nation continues to solidify efforts to bolster public mental health services.

Evaluation Strategy and Goals

The National Evaluation of the PAIMI Program, as outlined in this document, is based on the outcomes of the EA and will entail a coordinated overall strategy to implement a mixed-model design, comprised of multiple components.

In the EA, the evaluation team designed a detailed PAIMI Logic Model (see model on page 5) which incorporated various components of the PAIMI Program that could be addressed through a series of evaluation modules, targeting specific evaluation questions. This Logic Model will continue to inform the activities of the National Evaluation of PAIMI.

The evaluation modules, outlined previously in the EA report, have been developed further and are being presented in Chapter III of this Report. The recommended modules reflect the following priorities:

Evaluation designs should be economical. They should be respectful of the burden of data collection on federal and state staff as well as provider organizations;
The designs should prioritize questions that reflect the concerns of a variety of stakeholders including CMHS/SAMHSA, states, providers, consumers, family members and other advocates;
The evaluation should focus on issues that are amenable to change within limitations of resources and funding;
The evaluation should be based on rigorous methods and sufficient power to support conclusions; and
Evaluation modules should capitalize on data that are already being collected.

Performance measures that met the following criteria were prioritized:

Consistency with program goals and stakeholder priorities;
Emphasize outcomes in addition to compliance with regulations;
Address issues that can be influenced by the program;
Are culturally sensitive;
Have strong psychometric properties (including reliability, validity, reactivity, sensitivity/specificity, etc.);
Have face validity and are easily interpretable;
Have clear directionality;
Can be portrayed as rates or sentinel events;
Are suitable for comparison (over time, across similar localities or against a priori targets/benchmarks);
Are simply implemented; and
Minimize cost and burden at all system levels.

Therefore, in designing and selecting the proposed evaluation modules, the evaluation team carefully mapped how each of the question-specific data elements was to be implemented. While some elements involve extant data sources, other elements require field efforts to collect new data. Although most data are quantitative, some qualitative data have been recommended (especially regarding some of the process-level variables). A central part of the work plan for the National Evaluation includes the planned linking of qualitative and quantitative data, as well as the incorporation of extant and new data, in ways consistent with producing policy-relevant analyses.

I. Evaluation questions

Overview

The Logic Model provides the foundation for developing the evaluation questions. If we understand that the Logic Model represents the program as it is intended to function, then the goal of the evaluation is to assess whether and how well the program is working according to plan. That very broad question is then broken down into more specific evaluation questions that are organized below into clusters following the structure of the Logic Model. The Logic Model focuses on four aspects of the program:

The context in which PAIMI operates (including distal and proximal inputs);
The processes established to implement the PAIMI Program (at the state and federal levels);
The actual activities and tasks undertaken as part of the PAIMI program; and,
The process and distal outcomes achieved.

Given the broad goals of the program, however, it is difficult to establish a causal connection between PAIMI and the intended distal program outcomes. For that reason, the distal outcomes are separated from the other components in the Logic Model (as depicted by the shaded arrow). Not only does the PAIMI Program not operate in a vacuum, but the relatively small amount of funding dilutes the effect of PAIMI activities on outcomes. Many factors affect the diffuse goals reported under the auspices of the PAIMI Program. For example, in addition to PAIMI funds, other forces influence the degree to which states pursue particular policy agendas such as reducing and/or eliminating seclusion and restraint. Some of these forces are related to the PAIMI Program (e.g., a grantee’s decision to focus on this issue and its capacity to influence it), but others bear no relationship to the program (e.g., general workforce capacity within a locality or political factors within the locality). System change at the state and federal levels is a function of multiple factors, and the PAIMI Program is but one of these forces. Thus, the ability of the PAIMI Program to influence system change has to be understood in light of its circumscribed scope and its limited ability to prescribe state activities. This notion of shared responsibility for outcomes is a central feature in designing the PAIMI evaluation and interpreting evaluation results.

For this reason, while the evaluation will examine the influence of the PAIMI Program on process outcomes, we have not included distal client and system outcomes in the evaluation plan. The National Evaluation will not attempt to assess the direct influence of the PAIMI Program on the distal client and system outcomes. The consensus of the PAIMI Evaluation Advisory Workgroup (EAW) was that measuring PAIMI Program performance based on the degree to which these distal outcomes change in the desired direction is not a fair test of PAIMI performance since other factors (e.g. judicial rejection of new legal arguments, better reporting systems giving the appearance of increased incidents of harm, or insufficient housing for people with low incomes) may play a significant role in the outcome of a PAIMI Program’s efforts. The EAW suggested that it is more appropriate to judge PAIMI Program performance based on the degree to which grantees take on significant issues facing people with serious mental illness and examine the effort invested in securing change given resources as opposed to achievement of particular outcomes. Therefore, among other relationships to be examined, the influence of the PAIMI Program on process outcomes will be included in the National Evaluation.

Clusters of Evaluation Questions

The Logic Model developed for the PAIMI Program presents the program as it is intended to function and provided the foundation for developing an extensive list of evaluation questions. The Evaluability Assessment Report listed a wealth of evaluation questions developed in collaboration with our Evaluation Advisory Workgroup. The list below reflects two types of changes from that presentation. First, we did some streamlining through reorganization and elimination of redundancy and rewrote some questions for clarity. Second, limited resources do not permit examination of all the relevant areas identified; thus, below are those questions that we have determined are most essential to examine at this juncture with the available resources.

A. Inputs

How does the funding, organization and delivery of state mental health services affect the PAIMI Programs?
Are the resources provided to PAIMI grantees sufficient to achieve the goals of the program?
How do federal or state laws and policies directly related to P&A activities facilitate or constrain the PAIMI grantees in their pursuit of program objectives (e.g. outpatient commitment laws, SSDI eligibility, etc.)?

B. Effectiveness and Efficiency of the PAIMI Planning, Priority Setting, and Application Processes

What methods do PAIMI Programs use to set priorities for the PAIMI Application?

What do PAIMI programs do to encourage and support the participation of clients and other constituencies in the priority setting and planning process (e.g., providing transportation, childcare, outreach to communities of color)?
Do PAIMI-eligible individuals think they are heard when local PAIMI Programs are setting priorities? How effectively do PAIMI Programs ensure meaningful involvement by clients and individuals eligible for PAIMI services in setting priorities?
In what ways do PAIMI Programs make use of quantitative information (e.g., PPR data, demographic/prevalence data) in their planning and priority-setting processes?
Do grantees use federal policy directives such as the New Freedom Commission Report in crafting their plan?
How are PAIMI systems providing opportunity for public comment (e.g. using public partners and collaborating organizations to provide input, etc.)? Who do they seek out for comment? Have effective methods been identified and adopted across states? To what extent do PAIMI Programs share outreach materials?

Applications:

Do PAIMI Applications focus on mental health protection and advocacy resources and needs across multiple sectors (e.g. various institutional and community settings, housing, health care)?
Does the application serve a useful purpose for the PAIMI program? For the federal government?
Do PAIMI Applications reflect learning from prior year’s implementation experience?

Advisory Councils:

How do Advisory Councils vary in terms of size, structure, membership and practices (e.g. number and length of meetings per year; who sets the agenda; whether PAIMI staff attend, etc.)?
How are Advisory Councils supported?
1. How do PAIMI Programs provide ongoing staff support, program information, orientation and education?
2. How are members of the Advisory Council supported to participate (e.g. transportation, translation, child care, stipends, etc.)?
3. In what ways do PAIMIs develop rules and expectations for the conduct of advisory committees (e.g. conflict of interest, recruitment, terms, handbooks explaining the roles of advisory committee members, etc)
What role do the Advisory Councils play in formulating program-level policy?
1. How well do Advisory Councils fulfill their required PAIMI Program activities: planning, monitoring and advocating?
2. To what extent are Councils influencing the work of P&As?
3. How collaborative is the relationship between the Advisory Council and staff? Between the Advisory Council and the Governing Board?
To what extent are recommendations of the Advisory Council adopted by the P&A Governing Board? How are differences resolved?
Do Advisory Council members believe that their input is respected? Do members believe that they are effective in moving the PAIMI in desired directions?
How do Advisory Councils consider the needs of people with the most unserved/underserved/complex needs as well as reflect cultural diversity in the development of priorities?

C. Effectiveness and Efficiency of Federal Guidance and Review of Applications

Does CMHS provide grantees clear and unambiguous guidance for the preparation of their Applications?
Are all Application components necessary and appropriate? In what ways do numerical goals assist or hinder the attainment of program goals? How is the information from Applications used by CMHS?

D. Federal Monitoring

General oversight issues:
1. Is federal oversight effective in assuring compliance with statute and regulations?
2. How is CMHS oversight viewed by the Office of Management and Budget (OMB)?
3. What is the relationship between oversight of the PAIMI Program and oversight of the Mental Health Block Grant Program, given that this monitoring is conducted by the same Project Officers?
4. Are there enough resources at CMHS to oversee the PAIMI Program?
5. Five separate governmental entities are responsible for administering the eight P&A programs. To what extent does the multiplicity of federal agencies involved in the administration of the P&A Program between CMHS, ADD, RSA and other federal agencies impact PAIMI performance?
Uses of PPR Information:
1. What is the quality of the information reported?
2. How is this information used to monitor programs?
3. How is this information used for identifying TA needs?
4. How do the grantees use this information? Is it helpful to the states/territories?
5. Does oversight identify important strengths and deficiencies at an individual program level? At a systemic level?
How effective is the CMHS PPR peer review process in monitoring grantees’ performance?
How effective are site monitoring visits?
How effective are the processes for reviewing fiscal reports in identifying and addressing problems related to the appropriate expenditure of PAIMI funds?
What are the different contributions of the application reviews, the PPR peer reviews, the site monitoring visits and fiscal reports in providing oversight? Taken together, is this an adequate set of monitoring tools? What changes might be necessary? Are improved or additional methods of assessing compliance needed? Are these methods, taken collectively, the appropriate ones for monitoring quality and helping states to improve performance?
Grantee perspectives on Federal oversight.
1. How do grantees view federal oversight, particularly the PPR process and the monitoring site visits?
2. Is the federal reporting a burden on grantees?
3. What are the rewards and sanctions available to CMHS to reinforce appropriate management of the PAIMI programs by grantees? How are these rewards and sanctions used?

E. Federal Technical Assistance (TA) and Other Support

What types of Technical Assistance (TA) are provided and in what amount? Is the TA that is available commensurate with need (i.e. does CMHS’ PAIMI program have adequate resources to provide TA?)
How are TA needs identified? Is the process for deciding how to allocate TA resources efficient and equitable?
Is there an alignment between PAIMI program deficiencies and the PAIMI availing itself of TA?
Do grantees find the TA useful?
Does the program use techniques to extend the reach of available TA resources (e.g. identifying clusters of states/territories with similar needs, producing materials for wide distribution, etc.)?
Does TA conform to principles of adult learning, communicating in multiple formats and in a variety of ways?
TA Conferences and Meetings:

Who attends the TA meetings?
How useful are the conferences/meetings for PAIMI representatives?

How effective are the various back-up centers and other TA efforts supported with PAIMI dollars?
How effective is the list serve? Other educational materials distributed by NDRN?
How effective is the TA provider’s system for evaluating the effectiveness of TA and modifying the TA program?

F. State PAIMI Implementation

1) Goal Achievement:

Are annual program goals the most effective timeframe by which to develop and measure PAIMI Program effectiveness? Are other time frames feasible and acceptable to the federal government?
To what extent do states/territories achieve goals set in their plans?
To what extent do PAIMIs track goals/objectives not pursued due to lack of resources (staff, funding, etc.)?

2) Program management:

How effective are grantees in securing additional resources?
How are grantees utilizing pro bono legal resources?
How do grantees organize their program to achieve objectives?
How do grantees make strategic decisions about which competing opportunities to pursue? What is the process for making decisions about competing priorities? To what extent are PAIMIs flexible in responding to emerging issues? To what extent is PAIMI work influenced by new local, state and federal policy directions/initiatives?
How do grantees assure the quality of their efforts? Do the organizations examine their performance in systematic and regular ways (e.g., client satisfaction surveys, case review, analysis of information and referral calls, performance reviews of staff, analysis of client grievances data, etc.)?
To what extent does the PAIMI Program have to capacity to influence public policy through:
1. Individual and class litigation where indicated?
2. Systems advocacy in the administrative arena?
3. Legislative advocacy?

How comprehensive are strategies for the identification of individual and system problems?

What processes are employed to outreach to populations-in-need?
Is there timely access to records and reports (e.g. deaths) that identify individual or systems problems?

How many information and referral (I&R) calls and contacts are handled annually?

a. What is the ratio of I&Rs to individual cases accepted?

How do P&As that are organize departments by issue, as distinct from those organized by disability/program funding stream, prevent diversion of PAIMI resources to persons with other disabilities?
How do grantees address the responsibility to serve persons residing in the community?

a. How are PAIMI Programs managing client caseloads with the increased mandate to serve clients living in the community?

b. Are PAIMI funds dedicated to services for persons with mental illness? How do staff determine whether clients meet eligibility standards (e.g., mental illness diagnosis)?

c. Is current regulatory clarification sufficient?

How do PAIMI Programs distribute resources across intervention strategies, and what factors constrain their choices?
Do PAIMI Programs’ grievance procedures offer complainants timely and objective review? Are clients and prospective clients and families advised of grievance procedures?

a. Does the internal PAIMI grievance process provide complainants satisfaction that their grievance was heard objectively?

b. Are PAIMI program governing authorities providing oversight of grievances?

c. Is there an external authority to which PAIMI clients can appeal the results of a grievance?

Do PAIMI programs provide access to services to a client population that reflects the age, gender and racial composition of the population in the geographic area they serve?
How does the program assure that the PAIMI Program is organized to reflect cultural competency and provide culturally competent services?

G. Process Outcomes

How are PAIMIs demonstrating effort via:

Number of clients served (includes intake and referral, individual and class action clients)
Types of representation
Number of cases served
Number of cases litigated
Number of referrals
Number and types of trainings
Number and types of conferences
Number and types of collaborations within the state
Number and types of collaborations outside of the state
Degree of satisfaction/dissatisfaction reported by clients
Degree of satisfaction/dissatisfaction reported by organizational partners
Degree of satisfaction/dissatisfaction reported by advocacy organizations (PAIMI reputation)
Giving clients of the PAIMI Program voice (e.g., opportunity to have issue(s) aired, representation of equivalent quality to paid legal representation, etc.)?
Providing quality representation?

H. Cross-Cutting Issues: To what extent do the following structural aspects of the program or environment affect many of the concerns stated above?

State operated vs. not for profit entity?
Staffing pattern (e.g., ratio of advocates to attorneys)?
Organizational structure (e.g., single, integrated program vs. separate subcontractor for PAIMI)?
Single agency with PAIMI as a separately identified staff group within?
Level of PAIMI funding (minimum grantee or larger grantee)?
Urban, rural, or mixed environment?
Economic conditions in state or territory?

II. Evaluation Modules

Available Sources of Data

Given the relatively higher costs of new data collection, one important goal of the EA was to determine the extent to which available data could be employed to address these questions in the evaluation itself. There are a number of sources of existing data that we propose to analyze for this evaluation. They include the following:

Major sections of the grantees’ annual PPRs
- Goals and Objectives
- Data Tables
- PAIMI Advisory Council Reports
- Budget Information
PPR Peer Review findings
CMHS PAIMI Program site monitoring tool and site visit reports
PAIMI Annual Reports to Congress, and
NDRN technical assistance and conference evaluations.

PAIMI Programs are required to submit Annual Program Performance Reports (PPR) which consists of both narrative and quantitative information. These data sources provide an important foundation on which to build the National Evaluation of the PAIMI Program. Below we present a discussion of PPR narrative and quantitative data and our approach to extracting relevant information for the national evaluation.

Narrative PPR Data

The narrative information includes program priorities (goals and objectives), strategies, and outcomes -- the latter organized by institutional and community objectives and including case examples. Although there is a template with some instruction for completing the PPR, grantees do not report this information in a uniform way or with the same level of descriptive detail. While the National Association of Protection and Advocacy Systems (NAPAS), recently renamed the National Disability Rights Network (NDRN), has provided training on how to write narratives, the quality of information varies from grantee to grantee. Moreover, some key informants surveyed for the EA indicated that the priorities included in the PPR are not necessarily complete, and asserted that a conversation would yield more information on the agency’s priorities than a review of the PPR. According to the informants, there is a tendency to view these reporting requirements as not being directly useful to day-to-day operations and taking time away from the main line of work for PAIMI programs. Despite these shortcomings, we note that grantees generally write very extensive narratives.

The narratives are the only existing source of information on activities other than representation of individual clients. The alternative would be for the evaluation team to construct surveys or interviews in which we would request this information from the PAIMI programs in another uniform format. While that may be attractive, it has two major drawbacks. First, it would represent a substantial duplication of effort for the PAIMI programs. Second, because we would be requesting information about periods that are now two or three years in the past, there is no reason to believe that the data would be more reliable.

Therefore, we believe that the most cost-effective approach is to rely upon the narrative information present in the PPRs. That will enable us to address questions about program goals and objectives, target populations, and types of activities (exclusive of individual representation) in a reasonable fashion. In the module proposed, a specific strategy for content analysis of this narrative information was constructed to yield useful results. We recommend that narrative PPR data be included in the Evaluation.

Quantitative PPR Data

The quantitative information consists of 11 tables that represent primarily aggregated reporting on the characteristics of individual persons served and the services they received, and two tables reporting on groups. There are additional data on program staff, members of the Advisory Council and the Governing Board by race, ethnicity and gender, and sources of program income. Finally there are estimates of the numbers of persons reached by efforts other than individual representation (e.g. persons trained).

Programs employ various methods of tracking their individual cases. At least half have an automated client information system that is purchased from and supported by NDRN. Others employ their own automated systems or keep only manual records. Furthermore, the aggregation of data is only as good as the information that is actually recorded. There is no process for auditing these data.

Again, there are clearly shortcomings with this information. As is common with such systems, a small percentage of fields are missing, as shown in the aggregate tables. Looking at the aggregated tables, questions may be raised about the completeness of the data in other ways. For example, most open cases appear to have only one problem associated with them. PAIMI PPR Peer Reviewers would expect that a significant number of cases would present two or more problems, so this likely represents evidence of underreporting. Finally, key outcome variables (e.g., whether the case was successfully resolved in the client’s favor) are dependent upon the judgment of the PAIMI advocate/attorney providing representation and any supervisory review. In summary, there are two problem areas: 1) PAIMI staff may underreport individual cases or issues addressed during the course of representation, and 2) grantee reports of “success” of their representation of individuals may lack reliability across grantee programs.

However, the PPR aggregate tables are the only existing source of information on activities that are focused on individual representation. We believe that by including this data source in the evaluation we will be able to address questions of who is served and what services they receive in a reasonable fashion.

The alternative to including these data would be for us to construct forms for reporting on individual cases and request that programs complete them for a significant sample of current clients. This could include information about the “success” of each case from the perspective of the individual client. While that may be attractive, it has three major drawbacks. First, it would represent a substantial duplication of effort for the PAIMI programs. Second, it would not be possible to request this information for past cases, and limiting our analysis to current cases would be too restrictive and would require a period for data collection and analysis that would take us well beyond the period in which this evaluation must be completed. Finally, there is still the risk of underreporting unless we were also to add procedures that would allow us to review each client file quickly for inconsistencies or to perform other audits, the cost of which would also challenge the available evaluation budget.

We believe that the most cost-effective approach will be for us to rely upon the tabular information present in the PPRs. Below we propose a specific strategy for the analysis of these data and in the final report we will note any shortcomings with the data, indicating what biases may be present and how they affect the interpretation of findings.

The purpose, methodology, and expected response rate for each existing source of data are discussed separately below. Chapter III contains a schedule of work and estimated costs.

A. PPR Review and Analysis: Goals & Objectives Section

Purpose: In the Goals & Objectives section, each PAIMI describes their goals for the prior year (set during the Application process), an indicator to measure achievement of that goal, and information on whether that indicator of success was “met,” “partially met/continuing,” or “not met.” This section of the PPR is particularly helpful for two reasons. It is one of the only sections of the PPR in which the grantee can report on activities that affected groups or classes of persons, rather than just the casework that assisted individual clients. Also, for each indicator, the grantee is asked to provide an example of their work in that area. These narrative examples provide a “flavor” of the PAIMI grantee’s work that goes beyond what we can learn from examining the data on their individual casework.

The purpose of our intended review of this section is, first, to understand the range of policy-relevant issues that PAIMIs tackle in their work. Second, we wish to understand the degree to which PAIMIs are working in both community and institutional settings. Finally, we will use the “partially met” and “not met” indicators to gain understanding of what barriers PAIMIs encounter in those instances when they fall short of their goals.

Specifically, our evaluation of the PPR Goals and Objectives section will address the following Evaluation Questions: A2, B2a, D2a, F1b, F1c, F2d, F2f, F3, F6, F7, G1b, G1f, G1g, G1m, G1n.

Methodology: To make use of the information in this section of the PPR, we plan to randomly sample 20 of the 57 grantees. For each of these 20, we will examine the Goals & Objectives from the 2004 PPR in detail, employing the categories shown below.

Abuse Issues

Inappropriate or involuntary treatment
Restraint & seclusion
Abuse by staff (including physical, sexual, financial exploitation, and threats)
Death

Neglect Issues

Access to/failure to provide mental health treatment and diagnosis
Access to/ failure to provide medical treatment and diagnosis
Failure to provide a safe environment or humane conditions
Inappropriate commitment, failure to discharge, failure to place in least restrictive environment (LRE)

Civil Rights Violations

Housing discrimination
Employment discrimination
Failure to provide confidentiality, informed consent
Access to public benefits and entitlements
Guardianship and other substituted decision-making issues
Access to legal counsel
Criminal justice issues
Other civil rights violations: advanced directives, family rights, immigration issues, financial issues, privacy rights, and any other civil rights issues that do not fit in another category

Other Advocacy Issues

Knowledge about individual rights
Consumer self-advocacy
Stigma
Access to facilities, including clients, non-clients, and all records
Access to appropriate education and issues related to the transition from child to adult mental health systems

Our intention is to record, for each of the 20 grantees in the sample, whether they do or do not have at least one goal in each of the 21 areas above. Reading through each PPR’s Goals & Objectives, we will acknowledge the occurrence of priorities and indicators that relate to the issues listed by checking off a box on an Excel spreadsheet formatted for this purpose. Additional mentions of issues that have already been checked off will not be checked for a second time.

We will also identify the breadth of settings in which PAIMIs work, in particular whether they are assisting clients both in institutions and in the community, as directed by the recent expansion of their mandate. To this end, we have created the following list of potential settings:

24-hour, supervised residences for adults
A personal or family home
Prison, jail, or other adult criminal justice setting
Homeless
Nursing homes
Psychiatric hospitals
The psychiatric or emergency units of general hospitals
Children’s residential placement
Juvenile justice facility
Foster care for children
Schools

As with the list of issues above, we will give a state a single check mark for doing any type of work in that setting, regardless of the number of instances in which that setting was mentioned in their goals and objectives. It is important to note that we do NOT expect that each grantee will cover every issue and every setting, nor are we “judging” grantees against any standards of how many issues/settings they should tackle. We simply wish to report on the breadth of issues and settings that a sample of PAIMIs across the country are dealing with in a given year. With this information, we intend to examine the extent to which states in our sample were working on a variety of issues in both community and institutional settings in 2004. We will also identify issues and settings that were infrequently identified.

The next section of our analysis of PPR goals and objectives will focus on all indicators that were listed as “not met.” Even cursory examination of the available data on percentage of goals met or partially met, indicates that it is very rare for a goal to be recorded as “not met”. Therefore, we will focus our analysis on these. For each goal or objective that is indicated as not met, we will briefly describe the content of the goal and the constraint that prevented it from being accomplished.

Analysis of the unmet indicators will help us to identify which issues may be problematic and the types of barriers that PAIMIs face when trying to meet their goals. We expect that these barriers will include lack of resources, lack of time, and difficulty collaborating with other organizations, among others. What other barriers exist and how often they tend to prevent PAIMIs from reaching their goals will be discovered through this analysis.

Fourth and finally, we will pull examples from the narrative that describe important PAIMI activities that were directed toward helping groups or classes of individuals. Because there is very little other data on PAIMI’s work with groups, we will use these examples to give a sense of the kinds of important, non-casework activities that PAIMI grantees complete each year. A sample of the sorts of activities that we are interested in documenting would include: class action lawsuits, regulatory changes that affects groups of people, and legislative advocacy. Relevant examples will be noted and briefly described. We may draw on these examples in future reports to provide real-world illustrations of typical PAIMI activities. We will also examine all of the examples to look for themes, similarities, and differences.

Estimated Response Rate: HSRI currently possesses copies of every PPR from FY 2004, and therefore we expect to have all the necessary data required to complete this activity. However, there are some other limitations. It should be noted that the method proposed here for examining the Goals and Objectives section of the PPR is the result of many rounds of revision. We have considered both more detailed and more simplistic methods for examining this section of the PPR, but found that the data did not support the former, while the latter would not give us enough relevant information to be a worthwhile undertaking. Also, our findings will not be a full enumeration of PAIMI activities because our sample is limited to twenty grantees in only one year and because PPRs list examples of activities, rather than full accounts. Additional information about the limitations of the data and the other methodology that we considered will be written up in the final evaluation report, as we consider our observations about the data’s limitations to be a significant finding.

B. PPR Review and Analysis: Data Tables

Purpose: The PPRs contain a wealth of numerical data in tables throughout the report. These data describe characteristics of clients, staff, and advisory council members, as well as provide information on the types of complaints received by PAIMIs and the activities that PAIMIs conducted in response to complaints.

The intention is to take these data and develop quantitative process and outcome indicators for each of the grantees and for the national PAIMI program in aggregate. After examining all the data that are available in the PPRs, we have narrowed down a list of potential indicators:

Total individuals served

Percent of individuals served by minority racial/ethnic categories (African American, Hispanic, Asian, Native American) and total percent minority

Percent of individuals served by age groups (<21, 21-59, >60)

Percent of individuals served by gender

Percent of clients that are new

Percent of individual clients in institutional vs. community settings

Total number of complaints

Ratio of number of complaints to clients served

Percent of individual complaints that are categorized as related to abuse, neglect, or rights violations

Rates of “No Merit” and “Withdrawn” for abuse, neglect, and rights cases

Estimated number of individual persons denied services by PAIMI

Percent of closed cases that were favorably resolved for client

Percent of total interventions that were of the following types: short term, administrative, negotiation, mediation, and litigation

Total number of information and referral (I&R) contacts

Ratio of I&R contacts to cases

Estimated number of individuals represented in class action suits

Estimated number of individuals attending education and training sessions

Percent of grantees meeting Advisory Council client and family membership requirements

Rate of Advisory Council attendance at meetings

Number of Advisory Council meetings per year

Total number of priorities and goals

Percent priorities and goals fully or substantially achieved

Annual grant amount

Budget carryover from previous fiscal year

Additional sources of revenue and relevant amounts

Staffing: number of attorneys and non-attorney caseworkers

These indicators will help us to answer a number of the Evaluation Questions: B3a, D2a, F1b, F2a, F4, F6, F7, F9, G1(a) through G1(f).

Methodology: Annual indicator trends: Some of these indicators at the aggregate national level have already been assembled across grantees and are included in tables presented in the CMHS Annual Reports on Activities Under the Protection and Advocacy for Individuals with Mental Illness Act of 1986. We have these reports going back to 1999. An example of an indicator trend graph, showing the percent of cases favorably resolved at the national level, can be seen below in Figure 1.

Figure 1: Sample Trend Graph

Grantee-level Analysis:

For each of the process and outcome indicators above, we will create frequency distributions to observe the range of the data. In consultation with the Evaluation Advisory Workgroup, we will consider identifying “thresholds” for some of the indicators. For example, CMHS had set an expectation that 80% of all cases closed should be favorably resolved for the client; it has been raised recently. Figure 2 below shows how these data might be presented.

Figure 2: Sample Frequency Distribution

Each grantee (de-identified) is represented as a vertical bar in Figure 2. The percentages reached are shown by the height of the bar. Looking at the figure, we can immediately see that most grantees are above the 80% threshold. The intention above is to show both the characteristics of the persons and cases served and the variation among grantees for each of the indicators. This approach may have the following benefits:

It shows what is normative among grantees
It suggests what might be considered as outliers at either end of the range of performance.
It may be useful to CMHS in considering additional GPRA indicators to adopt.

Estimated Response Rate: For the proposed indicators, we will be collecting data for all 57 grantees for multiple years, most likely for FFY 2002-2004. NDRN has already provided data for most of the tables from the PPRs for FFY 2002, 2003, and 2004. However, tables that are not already in these electronic files will have to be located in each of the 57 grantees’ individual PPR reports and entered manually into files. We currently have copies of all of the 2004 PPRs at HSRI, but are missing a great number from 2002 and 2003. The first step, therefore, in this process will be to send a staff member to CMHS to collect these data on-site.

As with any effort to collect data from multiple sources, in this case 57 distinct grantees, there will be some discrepancies in the data. Early examinations of the data showed some findings that did not always “add up.” For example, in 2002, the ratio of complaints to clients served was less than 1, indicating that not every client made a complaint. Given that such a situation is unlikely, it is possible that there were errors in the reporting of data. Our findings about the limitations of the data will be noted in the final evaluation report.

C. PPR Review and Analysis: Budget Information

Purpose: PAIMI programs vary in funding available to support PAIMI activities. Some grantees operate solely on their federal grants while other grantees expend considerable effort to secure additional funding. Evaluation Questions to be addressed include A2, F2.

Methodology: While not intended to be an exhaustive inquiry into budget variations, this evaluation will gather information from grantees PPR reports with respect to awards and additional funding, and analyze this with responses from questions from the Executive Director’s survey. Project staff anticipate comparing grantees along such factors as state operated vs. nonprofit, demographic data, and success in securing outside funding. Information available in the PPRs includes: total grant award, budget carryover from the previous year, and contributions secured from outside funding sources.

Estimated Response Rate: We anticipate securing all PPRs for the look back period of this evaluation. Thus a response rate of 100% is expected.

D. Advisory Council Reports

Purpose: The Annual PAIMI Advisory Council Report contains a section for the Council to assess PAIMI Programs’ efforts toward meeting planned goals and objectives. The purpose of this review is to understand the extent to which Advisory Councils’ evaluations of PAIMI activities toward meeting annual goals and objectives align with or are divergent from the PAIMI staff’s analysis of progress made toward these same goals and objectives. From this comparison we anticipate inferring the degree of independence with which Council’s report on PAIMI outcomes. Evaluation Questions to be addressed include: B3c, B3d.

Methodology: Utilizing the same randomly sampled 20 grantees from 2004 PPRs, we will compare the Annual PAIMI Advisory Council Report (Section C, Advisory Council Assessment of PAIMI Operations) with the PAIMI Program’s self assessment of progress toward meeting goals and objectives (Section II, PAIMI Program Goals and Objectives). Each goal and objective will be compared for consistency in reporting across the categories of ‘Met’, ‘Partially Met/Continuing’, and ‘Not Met’. In cases where there is a difference between what the PAIMI PPR notes and what the Council’s report notes, both the Council’s explanations along with the PAIMI program’s explanation will be noted. A ratio of consistency versus inconsistency of outcomes will be reported and analyzed.

Estimated Response Rate: HSRI possesses copies of every PPR from FY 2004; thus we anticipate having all necessary data required to complete analysis of the 20 PAIMI grantees selected for Advisory Council report.

E. PPR Peer Review Findings

Purpose: The PPR Peer Review Process was developed in 2004 to evaluate applications and annual reports for 2003. There has now been a second year of reviews, involving many of the same reviewers. These are unusual in that the resulting observations and comments generally represent the views either of persons with direct experience as a member of the P&A staff or the PAIMI Advisory Council or of persons who have considerable contact with the PAIMI program. Thus the evaluations are by “peers” who are in a position to understand the context in which the grantees operate and to know what is reasonable to expect. Thus through this process CMHS is seeking to add significant expertise to its monitoring of the PAIMI program. The process produces three main products:

The formal review that is sent to the grantees in which CMHS staff summarize the major points.
The PAIMI Review Report which contains the peer review team’s summary assessment of the strengths and challenges of each grantee’s report and application, as well as their recommendations.
The notes of the peer review teams, which follow a template with “review prompts” that cover the major areas of the PPR.

The first two contain the most significant observations and comments of the review teams that represent the consensus of the group. The third includes much more detail about the team’s thinking, reflecting the opinions of the primary and secondary reviewers, but not necessarily the entire team.

The process is structured so that the evaluation is intended to be standard across all grantees. The reports produced by this process represent the only feedback that all grantees receive annually on their work. We believe that these reports, if compared from one year to the next, should provide an excellent opportunity to determine the following:

The types of program strengths and challenges that are common across many grantees;
The extent to which there is change from one year to the next to determine if problems identified in one year reappear in the next year at the individual grantee level;
The extent to which there is change from one year to the next to determine if problems identified in one year reappear in the next year at the aggregated grantee level.

Evaluation Questions to be addressed include: D3.

Methodology: We propose to obtain at least the first two products above from CMHS for 2003 and 2004. (The notes may also be obtained should there be a need to clarify the meaning of an observation in the PAIMI Review Report.) Once these become available, we will undertake a content analysis of the data. The focus of the analysis will be on identifying and classifying the different types of strengths, challenges, and recommendations contained in each report. We will aggregate these across grantees. We will also examine whether the same strengths and particularly challenges do or do not recur from 2003 to 2004. We will report across grantees on the rate at which challenges noted in 2003 recur in 2004 and the types of challenges that are most frequently addressed, as well as those that apparently remain unaddressed.

F. CMHS Site Monitoring Reports & Tools

Purpose: To assess the efficacy of CMHS site monitoring visits for identifying PAIMI Program regulatory compliance, program strengths and areas for improvement, program needs for technical assistance, and program budget management. To review the monitoring tool to evaluate its fit for ascertaining federal regulatory compliance. To further evaluate the periodicity of monitoring visits, delivery timelines of monitoring reports and dissemination practices.

Methodology: To review all site monitoring reports since 2003 after the revised monitoring tool was adopted in 2002. Evaluation Questions to be addressed include: D4.

G. PAIMI Annual Reports to Congress

Purpose: Since its inception, the PAIMI program has provided annual reports to Congress. Within the past seven years, with the introduction of a standard PPR that has changed very little over that period, CMHS has developed a standard report format that also varies only slightly from one year to the next. As a result, it is possible to examine trends in the program across several years and for a number of program indicators. The following are examples of data that can be followed:

Total I&R contacts
Total individual complaints
Individual Complaints broken down by abuse, neglect, rights categories
Percent individual complaints by intervention type: short term, administrative, negotiation, mediation, litigation
Percent individual complaints broken down by institutional and community settings
Percent closed cases favorably resolved for client
Total individuals served
Percent individual persons served by minority racial/ethnic categories (African American, Hispanic, Asian, Native American) and total percent minority
Percent of individuals served by age groups (<21, 21-59, >60)
Percent of individuals served by gender
Estimated individuals denied services
Percent priorities and goals fully or substantially achieved

Evaluation Questions to be addressed include: F1b, F4, F6, F7, F9, G1a,b,c.

Methodology: We propose to examine these annual reports from 1997 to 2004 and to construct trend lines showing the direction of the program across all grantees in each of the areas noted above, as well as others that may be of value. The latter will be determined by review of the annual reports for these years.

H. NDRN Technical Assistance Conference Evaluations

Purpose: Of particular interest to this evaluation is: a) the extent to which technical assistance conferences address issues that are relevant to PAIMI programs, b) the degree to which conferences are considered relevant and helpful by participants, c) how PAIMI issues for technical assistance are identified, and d) how NDRN and the federal partners utilize conference feedback. Only a small portion of these evaluation issues can be addressed by review of NDRN conference evaluations; other aspects will be covered through key informant interviews, interviews with P&A Executive Directors, and surveys of Advisory Council chairs.

NDRN solicits feedback from members who attend technical assistance conferences. Conference evaluations typically ask participants to rate each conference session independently as well as an overall conference evaluation. Evaluations typically cover the following: 1) the quality of each speaker’s presentation, 2) the quality of training materials, 3) the strengths of the session, and 4) areas for improvement, and, 5) additional comments. Evaluation Questions to be addressed include: E1, E2, E4, E6, E7.

Methodology: NDRN has provided the evaluation team with evaluation data from the Skills Building conferences for 2003, 2004, and 2005, from the Fiscal Managers conferences for 2003 and 2004, and the P&A system Annual Conferences for 2003 and 2004. In addition, we anticipate that 2005 Annual Conference evaluations will be made available in a few months. We will examine the evaluation data collected by NDRN and assess the extent to which conference sessions address issues relevant to the PAIMI program, participants’ reported satisfaction with the speakers/content and materials provided, and participants’ suggestions for improvements related to PAIMI issues.

Estimated Response Rate: We plan to review all conference evaluations available during the timeline of this evaluation. At this time, NDRN data provides a count of the number of evaluations received per session but does not indicate how many people attended each session. Thus while a response rate per session cannot be determined from available data, all available extant conference feedback data will be reviewed.

Additional Data Needed

In the previous section, we reviewed the available data and considered whether and how they can best be employed to address the evaluation questions for the PAIMI Program. Although the existing data provide an important foundation for the Evaluation plan, they are not sufficient to allow us to rely upon them for the entire study. For many of the evaluation questions, there is no existing source of data. In the following section we describe the types of new data collection that are necessary and the evaluation questions they are designed to address. Drafts of data collection instruments are included in the Attachments section.

A. Data Management

Data will be treated with strict confidentiality. Data gathered from interviews and surveys will be coded to mask individuals’ identities. Findings and recommendations will be based on aggregate data analysis. Data will be entered into EXCEL and SPSS spreadsheets and maintained in password protected electronic files. As with the key informant interviews conducted for the EA, persons interviewed or surveyed will be assured that their responses will be kept strictly confidential and no quotes will be used that would identify a specific respondent.

B. On-line Survey Methodology

We plan to use an online survey tool to conduct several of our proposed surveys. We have researched, considered, and pilot tested multiple online survey providers, including Zoomerang, Question Pro, Advanced Survey, and Survey Monkey. Given the combination of pricing and available features, we believe that Survey Monkey best fits our needs.

The Survey Monkey software provides the user with a simple interface to transform a paper & pencil survey into a web-based survey. The software allows for multiple question types (yes/no, multiple choice, write-in responses, rating scales, etc.) and for skip patterns, also known as branching. The skip pattern option will be especially useful in the survey for related organizations as we can direct the user to answer entirely different questions depending on what type of related organization they belong to. Survey Monkey also features a “list management” tool to keep track of contacts and responses as well as the ability to download the collected data directly into Excel. In contrast to some other survey sites that we tested, the Survey Monkey site loads very quickly and appears to be very user friendly for survey respondents.

Three surveys, described further below, will be designed for completion on-line:

Chairs of PAIMI Advisory Councils
State/territorial mental health program directors
Other mental health organizations within each state (e.g., state chapters of the National Alliance for the Mentally Ill, the National Mental Health Association, a consumer advocacy organization)

Respondents will receive an email requesting their participation in the survey.

Email addresses for the mental health program directors will be obtained from the National Association of State Mental Health Program Directors (NASMHPD). Similarly, where the state organization to be surveyed is a chapter/affiliate of a national organization, we will obtain email contact information from the national organization. We will ask P&A Executive Directors to provide us with email addresses and other contact information for their Advisory Council Chairs, and further, to nominate the directors of two other state mental health advocacy organizations (one of which is to be a statewide consumer organization). Along with these two nominations, evaluators will request the directors of state chapters of the National Alliance for the Mentally Ill (NAMI) participate.

Each of these groups has a website that lists contact information and additional information can be obtained as needed by phoning offices. Once we have completed the lists of potential respondents, we will contact them by email and offer them the choice of completing the survey online or requesting a paper survey. The email will include a link to the website where the survey can be completed. For those who request a paper and pencil survey, we will send it to them through the mail. For those for whom we do not have an email address or do not respond to the initial email, we will attempt to contact them through other telephone and mail.

We hope that the online survey option will increase initial response rates. Once the period of data collection is over, we will download the resulting data into EXCEL for analysis.

The cost to use the Survey Monkey website for one year is $200 and the cost to analyze the data from these surveys will depend on the length of each survey.

C. Surveys

1. P&A Executive Directors Survey

Purpose: Although the annual PPRs provide a window into the composition of PAIMI Programs and note some outputs for clients and systems changes, there are many aspects of PAIMI Programs about which there is little or no information. For instance, no information exists regarding how P&A Executive Directors would assess Federal oversight activities, how they engage with their governing boards and Advisory Councils, and conduct quality management functions. Evaluation questions covered by this survey include governing board activities, priority setting processes, the relationship between the governing board and the PAIMI Advisory Council, the efficacy of federal support and federal oversight of PAIMI operations, and PAIMI program operations.

Evaluation Questions to be addressed include: B1, B3, C1, D1, D2, D4, D7, E1, E4, E8, E9, F1(c), F2(a), F2(d), F2(f), F3, F5, F6(b), F7, F8, F9, F10, G1(n).

Methodology: Because of the length of this survey instrument, rather than utilizing the on-line survey, a structured interview will be conducted by telephone as personal contact greatly improves response rates. P&A Executive Directors will receive copies of the interview questionnaire in advance of the interview, and will be offered interview times that accommodate their schedules. We anticipate that many of these interviews will take place during evenings and weekends. Directors will also have the option of asking other members of their staff to participate and respond to questions in areas where they believe that someone else is the most knowledgeable. Questions are both closed and open ended, and in many cases the closed ended questions ask for further explanation. Thus these interviews are expected to take approximately two hours per interview but will allow Directors time and space to fully respond and explain PAIMI operations.

We expect that preparation for an interview, scheduling and conducting the interview to take approximately three hours of project staff time. Another hour will be needed to clean up the interview for data entry. Thus, interviewing all 57 Directors would be cost prohibitive given current evaluation resources. We therefore plan to interview a sample of 20 selected P&A Executive Directors. (See Attachment A to review the interview protocol for P&A Executive Directors.)

Expected Response Rate: As Evaluation staff will be able to follow up with Directors via email and phone to schedule interviews, we anticipate a response rate approximating 90%.

2. PAIMI Advisory Council Chairs Survey

Purpose: The annual PPR contains limited descriptions of the members of the PAIMI Advisory Councils and their meetings (e.g., frequency of meetings). Because of the importance Congress envisioned for Councils in guiding the PAIMI Programs’ annual priorities and representing consumers and family members, more information is needed to determine how well these entities are fulfilling their large mandate. Moreover there is a lack of information about the satisfaction of members in performing their duties and the level of support to Councils received from grantees, CMHS, and the technical assistance contractors. Evaluation Questions to be addressed by this survey include: B1, B3, C2, F1c, F8.

Methodology: This survey will be available both on-line and as a mailed survey to Chairs of PAIMI Advisory Councils in the same states/territories identified in the sample drawn for surveying P&A Executive Directors. We selected Council Chairs because they should have sufficient experience and service on the Council to provide a longer term view of the Council’s functioning and interface with PAIMI operations. Surveys will be tracked to determine those outstanding, and Council Chairs who have not responded will receive telephone calls to request their participation and determine if any accommodations are necessary. (See Attachment B to review the Advisory Council Chair survey.)

Expected Response Rate: We anticipate a response rate of approximately 80% of the Advisory Council Chairs.

3. PAIMI Client Survey & Focus Group Meetings

Purpose: No national data exists regarding how clients of the PAIMI program perceive the quality and outcomes of program services. As quality management technique evolves in the disabilities field, it has moved to embrace client feedback for assessing program impact and identifying areas for service improvement. Because mail surveys have poor response rates and are biased to gather information from those who are most highly satisfied and also most dissatisfied, our team determined that a blended approach combining in-hand delivery of a written survey with a focus group meeting would better elicit meaningful feedback on client experience with the program.

According to conventions used in social research, focus groups are composed of 6-12 individuals who participate in a 1 ½ to 2 hour guided discussion facilitated by a researcher-moderator who is often assisted by another researcher taking notes and observing the discussion. While the researcher guides the discussion using a discussion guide with certain issues already identified, the format allows the participants to raise concerns and issues which the researcher had not previously identified.

Focus groups, in general, offer several advantages over other quantitative research methods:

Issues which are of concern to the participant but which have not yet been identified by the research team may be identified;

Focus groups may elicit spontaneous reactions and ideas that would not arise in a more structured format such as an interview or survey;

Focus groups permit the researchers to observe group dynamics and thus add another dimension to the study; and

The purpose of focus groups is not to build consensus, but to obtain a range of opinions from a wide range of participants. We will work with the grantees selected to participate in the focus group study and provide recruitment material (e.g. posters) to them encouraging a broad range of participants from among their PAIMI clients.

There are additional advantages to using the focus group format for eliciting information from PAIMI clients:

The focus group format permits in-depth exploration of sensitive topics such as those likely to be raised by PAIMI clients; and

The qualitative approach yields in-depth understanding of particularly complex needs and issues, including those surrounding protection and advocacy.

As with other studies of vulnerable populations and service recipients, it would likely be difficult to obtain a representative sample of PAIMI clients to participate in individual interviews or surveys. Response rates would likely be relatively low. While results from the focus groups may not be generalizable, they will yield richer information than traditional interviews often permit. Given the unstructured format of the focus groups, the research team may obtain a clear understanding of the complex and/or sensitive issues PAIMI clients raise. Specific examples of such issues may be elicited and the research team may prompt the participants to explain points which are unclear. In addition, as PAIMI clients raise issues which are salient for them, the research team will explore the range of attitudes, perceptions, and experiences among other participants.

Evaluation Questions that pertain to client evaluation of the efficacy of PAIMI services include: B1, F8, F9, F10, G1(j), G1(n).

Methodology: We plan to conduct four site visits to collect client data. Client data from the pilot site visit (discussed below in Section 5 of this chapter) adds a fifth source of client input data. Different regions of the country will be represented, and at each site visit, two client focus group meetings will be held in order to provide for client input from persons who reside in institutions and those who reside in the community. Focus group meetings will be co-facilitated by one HSRI evaluation team member and a consumer representative from either the EAW or the selected P&A’s PAIMI Advisory Council Chair or consumer member.

We plan to utilize a combined data gathering methodology of in-hand delivery of surveys to stimulate client thinking about their experience as a PAIMI client followed by an hour focus group meeting. Client participants will be reimbursed for travel expenses incurred to participate in the focus group meetings and offered $20.00 as a token payment for their time.

The protocol for client participation in a focus group will require that he or she have an open PAIMI case for at least 30 days. P&As will be directed to include clients that are community based and institution based. Project staff will prepare an announcement/flyer regarding the focus group meetings and will ask the participating P&A Executive Directors to nominate a coordinator for project staff communications. The client coordinator at the PAIMI program will be responsible for disseminating the flyer to clients, assist project staff with selecting the locations for both the community meeting and the institution-based meeting, and provide logistical support to evaluation team members for the site visits. Should it be too burdensome on the P&A to organize a meeting of clients at an institution, then the P&A will have the option of identifying clients who have been released from an institution within the past 45 days to participate in a focus group meeting. (See Attachment C to review the survey for PAIMI Clients.)

Expected Response Rate: Obviously visiting five sites and conducting 10 rounds of surveys and focus group meetings constitutes only a small portion of the more than 20,000 clients who receive individual representation each year. Nevertheless it should be sufficient to provide information on how the program is viewed by persons who are served directly. Depending on the size of the PAIMI Program, we anticipate meeting with between eight and 18 PAIMI clients per site visit.

4. State Mental Health Authority (SMHA) Directors Survey

Purpose: The PAIMI programs are responsible for protecting persons with mental illness against actual or potential abuse, neglect and violations of civil rights. At its inception, those problems were generally perceived to occur within institutions of the public mental health system, institutions that are the responsibility of state mental health authorities. While the mandate of the PAIMI program has broadened, state mental health authorities remain a major potential adversary of the P&A system, as well as a possible ally. The purpose of this survey is to determine how the PAIMI programs are perceived by state/territory mental health authorities. A major focus is on the relationship between the PAIMI program and the State Mental Health Authority. Parallel questions are also asked of the P&A Executive Directors. Evaluation questions addressed by this survey include: B1e, F2f, F3, G1h, ln.

Methodology: This survey will be available both on-line and as a mailed survey to state/territorial mental health program directors of states/territories selected to participate in the survey of P&A Executive Directors. SMHA directors may designate another member of their senior management team (e.g., counsel) to complete the survey if that is appropriate in their judgment. Surveys will be tracked to determine those that are outstanding, and SMHA directors who have not responded will receive first email requests and later telephoned requests to complete the survey. (See Attachment D to review the survey for State Mental Health Authority Directors.)

Expected Response Rate: Again although it is untested, we anticipated a much higher response rate with the option of on-line participation. We are hoping to secure a response rate of 70% from the 20 selected state/territorial mental health authority directors.

5. Other State Mental Health Organizations that Relate to PAIMI

Purpose: In order to meet their goals, PAIMI programs interface with other mental health organizations in their state. For example, they may collaborate with a family advocacy organization to push for additional funding for mental health treatment in jails or with a consumer advocacy organization to block passage of a proposed expansion of a state outpatient commitment law. At other times the PAIMI may take a position that is in opposition to those same organizations. Our draft survey of directors of other state/territory mental health advocacy organizations intends to ascertain the extent to which these players understand the role of PAIMI and their perception of PAIMI program effectiveness.

Methodology: For each grantee selected to participate in the survey for P&A Executive Directors, we will include review by three peer organizations. One of the three shall be the state/territory chapter of the National Alliance for the Mentally Ill (NAMI), a family advocacy organization. The other two organizations shall be nominated by the P&A Executive Director, with the caveat that one nomination be a statewide consumer-run advocacy organization (where available) or the state’s most active mental health consumer organization. The Evaluation Questions relevant to these surveys include: B1, G1k, G1L. This survey will be administered via the on-line survey. (See Attachment E to review the survey for directors of other state mental health organizations that relate to PAIMI.)

Expected Response Rate: Although survey responses tend to be low (typically below 20%) for most mailed surveys, we hope that the ease of on-line participation coupled with the automated prompting to non-respondents will boost response rates significantly to around 70%.

D. Key Informant Interviews Related to Federal Oversight Activities

Purpose: Key informant interviews are intended to address a subset of the evaluation questions designed to examine whether Federal oversight is effective in assuring compliance with statute and regulations. Such oversight includes design and review of annual applications, annual program performance reports, site monitoring visits, and fiscal plans and reports, as well as the extent to which CMHS-funded technical assistance is effective in supporting the needs of grantees. The key informant interviews will be used to address many of the evaluation goals, including: an understanding of the strengths and weaknesses of each of the key monitoring activities (oversight by the project officer, site monitoring visits, PPR peer reviews) and the relationship to technical assistance. More specific goals for each of the interviews are indicated in the separate sections describing the interview. Evaluation Questions to be addressed include: D1, D2, D3, D4, D7, E1, E2, E3, E4, E6, E8, E9, E10. (See Attachments F, G, H and I for Key Informant interview protocols.)

Methodology: Key informant interviews are a method for obtaining detailed information from individuals who have substantial knowledge about the PAIMI program and are key actors with respect to particular functions. These interviews are typically semi-structured and are conducted using an interview guide that helps keep the interview focused, but also allows flexibility to pursue other relevant areas and room for elaboration where appropriate.

We will work with the GPO to develop a list of key informants. Within each group we propose that five key informants be interviewed. These individuals will include the following:

CMHS PAIMI Project Officers;
NDRN Staff with responsibility for technical assistance and training activities;
PAIMI Site Monitors who have participated in the site monitoring visits; and
PAIMI Peer Reviewers who have participated in the annual reviews of the Program Performance Reports.

Separate interviews with individuals representing each group will be conducted either in person where convenient or via telephone. The content areas for these interviews will be different for each type of individual and are described below.

1. CMHS PAIMI Project Officers

PAIMI Project Officers play a key role as the interface between CMHS and the grantees. The assignment and development of this role is relatively new, representing an organizational change that has taken place within the State Systems and Planning Branch within the past two years. The purposes of these interviews is to understand the following:

Generally the role of the project officers

The following review processes from the project officers’ perspective

Application process
Site Monitoring visits
PPR Peer Reviews
Fiscal reviews

Project Officer’s role with respect to Technical Assistance

Relationship between PAIMI and the Mental Health Block Grant

Overall strengths and challenges of the PAIMI program

(See Attachment F to review interview questions for CMHS PAIMI Project Officers.)

2. NDRN Technical Assistance (TA) Staff

NDRN receives funds from ADD, RSA and CMHS as the prime contractor for provision of technical assistance to the PAIMI and other P&A programs. NDRN staff involved in the administration of PAIMI technical assistance shall be interviewed to determine how information is collected from the system regarding needs for technical assistance (both system wide as well as individual grantee), how requests for technical assistance are prioritized, and what feedback is received regarding the usefulness of the technical assistance. Evaluators are interested in learning how NDRN staff craft PAIMI resource allocation decisions and what guidance is provided by federal partners over this aspect of program support. In addition, evaluation staff will inquire about NDRN’ self assessment of the effectiveness of technical assistance and how program improvement is managed. (See Attachment G to review interview questions for NDRN PAIMI technical assistance staff.)

3. PAIMI Site Monitors

Site monitors are enlisted by CMHS to act as consultants in the on site PAIMI program monitoring visits to grantees. Monitors are selected generally for their experience with PAIMI programs and for their particular contribution within the monitoring team (e.g., fiscal review). They are placed in teams of four to conduct the site visit. The CMHS Project Officer is always a member of the site visit team. Six to seven site visits are conducted each year, but any single monitor will only be involved in two or three. The purposes of these interviews is to understand are as follows:

Role and contributions of the site monitors
Selection of site monitors
Orientation and preparation provided monitors

Site review process from the reviewers’ perspective

Representation of site visitors’ on site feedback in notes and reports

Strengths and challenges of the site visit process

Dissemination of findings

(See Attachment H to review interview questions for PAIMI Site Monitors.)

4. PAIMI Peer Reviewers

The PPR Peer Reviewers are enlisted by CMHS to act as consultants in the review of the annual PAIMI Program Performance Reports required of all grantees. They are selected generally because they have a role within a grantee program, either as staff or a member of the Advisory Council. They are placed in teams of four to conduct the review of PPRs from about ten grantees each. This process is still new; it has been in operation for the past two years. The purposes of these interviews are as follows:

Understand generally the role of the peer reviewers

Understand the peer review process from the peer reviewers’ perspective

Understand the background and orientation of the reviewers

Understand the representation of peer comments in notes and reports

Understand the strengths and difficulties of the peer review process and the PPR requirements

Overall strengths and difficulties of the PAIMI program

(See Attachment I to review interview questions for PAIMI PPR Peer Reviewers.)

E. Site Visit to Pilot Evaluation Processes and Instruments

Purpose: One on-site visit to a PAIMI grantee program was conducted to pilot the Executive Director survey, PAIMI Advisory Council Chair survey, the State Mental Health Association Director’s survey, PAIMI Client Survey and Focus Group meetings, and surveys with Other Mental Health Advocacy Organization Directors. Piloting these processes and instruments provided the evaluation team feedback regarding whether evaluation methods were on target to address the most poignant research questions, whether any critical areas for assessment had been overlooked and need incorporating, and whether there were methods that yielded little relevant information that could be eliminated. The grantee program selected for the pilot site visit, the New Jersey Protection & Advocacy, Inc., met the following criteria: a) an Executive Director with long term experience as a P&A Executive Director, b) a PAIMI program with sufficient complexity to test our instruments and methods, and c) important for cost considerations, close enough in proximity to HSRI to minimize the evaluation team’s travel expenses.

Methodology: The pilot test site visit was scheduled for October 10^th,11^th & 12^th, 2005, dates selected by the grantee. A conference call with the P&A Executive Director, CMHS, and project staff was scheduled to discuss logistics. A package of site visit materials was mailed to the Executive Director three weeks in advance of the site visit to allow adequate time for preparation, principally to allow sufficient time to give notice to PAIMI clients for participating in the focus group meetings. The site visit utilized three evaluation team members and took two full days. As planned, at least one of the on-site team members was a Co-Project Director.

The site visit tested out the following data collection methodologies:

Interview with P&A Executive Director
Advisory Council Chair Survey
PAIMI Client Survey & Focus Group Meetings
State Mental Health Association Director Survey
State Mental Health Advocacy Organizations Survey

In addition to the field testing of survey instruments, the site visit was useful for other purposes. They were opportunities for evaluation staff to share impressions of the PAIMI program performance indicators. Evaluation staff shared our analysis of the grantee’s activities gathered from analysis of PPR qualitative and quantitative data. We also planned to use the site visit to hold focus group meetings with PAIMI staff and PAIMI Advisory Council members. A focus group meeting held with PAIMI staff was instructive in evaluating the efficacy of many Executive Director interview questions.

At the end of the site visit an exit interview to debrief on the visit was held with the Executive Director, and other key PAIMI staff the Executive Director invited to attend. P&A staff provided instructive feedback; their recommendations and impressions were written up and distributed to CMHS and the EAW for review and comment.

Expected Costs: The pilot site visit expended two days for scheduling and preparation, one day of travel, two days on-site, and one day post-site visit to evaluate strategies, as well as hotel accommodations for three people for two nights, plus per diem and ground transportation expenses. See Section IV for costs associated with the site visit.

III. Schedule of Work & Estimated Costs

The schedule of work is highly dependent on the timing of OMB Clearance of this evaluation plan. Our preliminary Federal Register Notice (FRN) was submitted to CMHS for review. Upon receipt of comment, a revised version was delivered for posting on August 1st. We had planned that this federally required process, a process that provides for public comment on new data collection, to commence in August and be completed by the end of September 2005. Subsequent to the public comment period we will be able to submit our OMB Clearance package along with any public comment received. However, the FRN was published January 26, 2006, and thus the public comment period ended March 26, 2006. The evaluation project was essentially on hold from mid-October 2005 until the end of March 2006.

OMB Clearance is typically a 90 day turn around. Thus the period of time from April through June 2006 is expected to be dedicated to work on elements of the evaluation that do not require new data collection -- as well as those elements that are new data collection but do not exceed the OMB prohibition of more than nine new data elements, such as conducting nine of the 20 interviews with P&A Executive Directors.

We will present the proposed list of interviewees to the GPO in May 2006. Once the list is approved, we will initiate these interviews. (As fewer than nine individuals will be interviewed using the same interview schedule, OMB Clearance is not required.) We anticipate completing all key informant interviews by the end of July 2006.

Data collection for existing data is underway. In July 2006 we expect to commence the bulk of surveys and interviews. Data analysis will be an on-going task that we anticipate covering the period of time from August through November 2006. Beginning in the last quarter of 2006, a series of conference calls will be held with CMHS and the EAW to review data collection and preliminary findings. A month has been set aside for drafting the final report and another month for reviewing it with CMHS and the Evaluation Advisory Workgroup. Expected delivery of the final report is January 2, 2007, the current end date of the contract extension. Table 1 below lays out the principal evaluation tasks and timing.

Legend:

Draft

Completed

able 1. PAIMI Evaluation Plan Task and Timing Chart

Task

2005

Jul

Aug

Sep

Oct-Dec

2006

Jan

Feb

Mar

Apr

May

Jun

Jul

Aug-Dec

2007

Jan

Deliver draft Evaluation Plan to CMHS & EAW for review

Deliver final FRN to CMHS

CMHS & EAW conference calls to review draft Evaluation Plan

Deliver Final Evaluation Plan

Conduct Pilot Site Visit

Deliver OMB package including public comment from FRN

FRN ended 3/26/06

Data analysis of available data

Initiate Key Informant interviews & interviews with P&A EDs

OMB authorization expected

Initiate new data collection: all surveys

Analyze new data collected

Prepare draft sections of Evaluation Report & distribute to CMHS & EAW for review

Conference calls with CMHS & EAW to review draft Evaluation Report

Deliver Final Evaluation Report

Stakeholder briefings

Deliver evaluation materials to CMHS

Expected Cost of Evaluation

The overall costs of evaluation, including data collection, data analysis, writing the final evaluation report, and presentations to and for CMHS are estimated to be $150,000. (These are exclusive of the costs of the first two phases of work, the Evaluability Assessment and the Evaluation Plan.) The estimated costs for each evaluation module separately are shown in the second column of Table 2 below. A second set of costs which assume a supplemental budget of $45,000 is shown in the third column of Table 2. All of the instances in which costs differ between the two columns have been shaded.

Table 2. Estimated Costs of PAIMI Evaluation Modules

EVALUATION MODULE	Base Budget Costs	Base plus Supplemental Budget Costs
EXISTING DATA
PPR Narrative	$1,704	$1,704
PPR Data Tables	$2,498	$2,498
Advisory Council Reports	$426	$426
PPR Peer Review Findings	$3,264	$3,264
Site Monitoring Repts	$3,102	$3,102
Annual Repts to Congress	$984	$984
NDRN conference eval surveys	$2,418	$2,418
SURVEYS
P&A Exec Dir	$15,294	$35,384
PAIMI AC Chairs	$7,046	$14,664
PAIMI clients	$9,910	$9,910
SMHA directors	$6,406	$14,024
Other State MH orgs	$8,518	$18,248
KEY INFORMANTS
CMHS Project officers	$2,157	$2,157
NDRN TA staff	$1,905	$1,905
Site monitors	$2,220	$2,220
Peer Reviewers	$2,157	$2,157
OTHER COSTS
Site visit	$8,020	$8,020
OMB Clearance Package	$3,096	$3,096
Develop Analysis Plan	$2,892	$2,892
Organization of Data Files	$1,920	$1,920
Team meetings	$35,740	$35,740
Editing final report	$19,732	$19,732
CMHS Monthly Reports & Briefings	$3,816	$3,816
Report Briefings	$4,620	$4,620
Estimated costs	$149,845	$195,901

As our evaluation plan is presently constructed, we intend to sample 20 PAIMI Programs from among the 57 grantees for the interviews with the P&A Executive Directors, and corresponding surveys of PAIMI Advisory Council Chairs, State Mental Health Program Directors, and the directors of three other mental health advocacy organizations. With a supplemental budget we could anticipate conducting these data collection and analysis efforts with all 57 grantees. This is a much preferred alternative to sampling because it would provide a complete data set. With a complete data set more robust analysis of differences among types of grantees could be analyzed, and findings related to existing data (PPR Data Tables and PPR Peer Review findings) where we intend to review all grantees could be linked to grantees for richer analysis.

The PAIMI contract was amended to extend the evaluation from the original end date of March 2006 to January 2007. As no additional monies were awarded were expand the evaluation to all grantees, we plan to conduct the evaluation with a sample 20 grantees and affiliated organizations.

IV. Evaluation Report Dissemination

Distribution of the Final Evaluation Report

Once the TOO has approved the evaluation report as final, copies will be distributed to CMHS as per the contract. Electronic copies, and where requested hard copies, will also be distributed to members of the PAIMI Evaluation Advisory Workgroup. At the discretion of CMHS, project staff may offer electronic copies of the report to those who participated in the evaluation, e.g., P&A Executive Directors, PAIMI clients, and directors of State Mental Health Associations, NAMI, and consumer-run organizations. In addition HSRI would request that the report be made available to the pubic on-line through either CMHS and/or HSRI’s websites.

Briefings on the Final PAIMI Evaluation Report

Project staff plan to conduct at least three briefings on the final report. The first briefing will be for CMHS staff. The remaining two briefings will be for stakeholder groups. All persons/audiences to be briefed will be identified by the TOO. For each briefing, project staff will develop power point slides that provide a description of the PAIMI program, summarize the principal findings overall, as well as those findings most relevant to the stakeholder group. Recommendations for program change will also be incorporated into each briefing. Individuals attending these briefings will be informed about how to obtain a copy of the full Final National Evaluation of the PAIMI Program Report.

LIST OF ATTACHMENTS

Surveys

A. P&A Executive Directors

B. PAIMI Advisory Council Chairs

C. PAIMI Clients

D. State Mental Health Directors

E. Other State Mental Health Organizations that Relate to PAIMI

INTERVIEWS

F. CMHS PAIMI Project Officers

G. NDRN Training & Technical Assistance Staff

H. PAIMI Site Monitors

I. PAIMI Program Performance Reports Peer Reviewers

Appendix A

Form Approved

OMB No.: 0930-XXXX

Expiration Date: MM/DD/YY

See burden statement on last page.

National Evaluation of the Protection & Advocacy for

Persons with Mental Illness (PAIMI) Program

Protection & Advocacy Agency Executive Director Interview

This survey is being conducted by the Human Services Research Institute (HSRI) through a contract with the Substance Abuse and Mental Health Services Administration’s Center for Mental Health Services (CMHS) for the first National Evaluation of the PAIMI Program. Survey results will provide information from P&A Executive Directors across the states and territories to describe the a) P&A Governing Board activities, b) range and scope of PAIMI program management, and c) what factors influence the effective processes and the achievement of systemic objectives. HSRI’s evaluation team and the Evaluation Advisory Workgroup members recognize that all PAIMI programs are not alike and have no preconceived model of ‘correct’ responses. This survey attempts to gather information on the range of program configurations to assess what is typical and possible given very limited resources allocated to PAIMI grantees.

Responses are strictly confidential. While the identity of the 20 Protection and Advocacy agencies sampled for this evaluation will be noted in our report, the responses of the Executive Directors and other respondents will not be shared or revealed.

Questions are grouped into sections:

Section 1	P&A Governing Authority or Board structure and operations
Section 2	P&A Governing Authority’s relationship to the PAIMI Advisory Council
Section 3	PAIMI Advisory Council role and functioning
Section 4	PAIMI priority setting process
Section 5	Federal support for PAIMI operations
Section 6	Federal oversight of PAIMI operations
Section 7	PAIMI program operations
Section 8	Demographic information about Executive Director
Section 9	Reserved for comments the Executive Director would like to make

We expect that it will take approximately 2 hours to complete this telephone interview (1 ½ hours to conduct the call and ½ hour of preparation). Although most responses are pre-coded, we hope that you will elaborate and comment to fully address your experiences as managers of PAIMI programs.

INTERVIEWER PROTOCOL: Advise of strict confidentiality of responses. Ensure that the respondent has a copy of the interview guide to follow along. Please remember to thank the Executive Director for assisting with the first National Evaluation of the PAIMI.

SECTION 1: Structure and operations of your P&A Governing Authority/Board

How many Board members does your P&A have? ___
How many Board members reflect the following demographic make up?

People who are knowledgeable about mental health issues _____
People with disabilities _____
People who are family members of persons with disabilities _____
People who are knowledgeable about disability service systems ______
People who represent minority or underserved communities _____
People with mental illness _____
Family members of people with mental illness _____

Is there an orientation to new Governing Board members regarding PAIMI responsibilities?

 No

 Yes If YES, which of the following topics is covered? Check all that apply.

 Legal and ethical responsibilities to PAIMI clients

 History of the PAIMI Program including federal mandates & limitations in statute and regulation

 Federal oversight and reporting requirements

 Setting PAIMI priorities

 PAIMI operations & staff

 Current PAIMI casework & litigation

 Understanding abuse, neglect, civil rights violations

 Other, please list ________________________________________________

Do Board members have access to information from staff beyond what the executive director provides, for example….?

 Scheduled opportunities for the entire Board to meet with staff

 One on one meetings between staff & Board members

 Other, describe

_________________________________________________

Fiscal

Is there a written plan for securing resources for PAIMI operations beyond the federal grant?

 No

 Yes

If YES, who is responsible for implementing that plan? _________________________________________________________

In the last 2 federal fiscal years, from how many sources (other than CMHS) has the P&A sought additional funding for the PAIMI program? Please list the entities solicited & whether you were successful in securing the funds or not. Include any funds received from the state by funding source.

Funds solicited from:	Funds requested (insert $ amount)	Funds received (insert $ amount)

Monitoring

Has the Board developed a written strategic vision for P&A operations within the past 5 years?

 No

 Yes

If YES, how does it address PAIMI Program, e.g., how was the PAIMI Advisory Council involved in creating the vision? Please describe: ___________________________________________________________

What mechanisms are in place for the Board to monitor activities and performance of the Advisory Council? Check all that apply.

 Monitor nominations to PAC for any conflicts of interest

 Regular communications between Board & Council representatives

 Monitoring PAC outreach activities

 Monitoring P&A support to PAC orientation, training & ongoing supports

 Other, please describe ___________________________________________

SECTION 2: P&A Governing Board/Authority’s relationship with the PAIMI Advisory Council.

In the last 2 years approximately how many PAIMI Advisory Council recommendations for PAIMI annual priorities for goals and objectives were presented to and adopted by the Board?

2005 2004

# presented: ____ # presented: ____

# adopted: ____ # adopted: ____

If in the past 2 years the Advisory Council has recommended PAIMI priorities for goals and objectives to the Board that have not been adopted, please describe the rejected priority/priorities & rationale for not adopting:

How are differences resolved between what the Advisory Council recommends and what the Board adopts?

SECTION 3: PAIMI Advisory Council

Are policies in place that address Council member term limits?

 No

 Yes

If Yes, what is the term limit? _____________

Are policies or systems in place that address Council members having staggered terms?

 No

 Yes

Is there an orientation for new PAIMI Advisory Council members?

 No

 Yes

If YES, which of the following topics is covered? Check all that apply.

 Legal and ethical responsibilities to PAIMI clients

 History of the PAIMI Program

 Federal oversight and reporting requirements

 Setting PAIMI priorities

 Staffing PAIMI operations

 Current PAIMI casework & litigation

 Understanding abuse, neglect, civil rights violations

 Accommodations available to assist Council members to attend meetings and participate fully

 Other, please list ________________________________________________

In your opinion, how long does it typically take for a new Council member to become a fully contributing member?

 Less than 1 year

 1 to 2 years

 2 to 3 years

 More than 3 years

In the past 2 years was member development training provided to Council members in the following areas?

 leadership skills

 cultural competency

 participating in meetings

 self advocacy

 negotiation

 collaboration with other stakeholders in mental health system

Is there a designated staff member to provide on-going support to the PAIMI Advisory Council?

 No

 Yes

SECTION 4: PAIMI priority setting process

How does the level of effort vary from year to year, if any, in establishing your PAIMI Program priorities?

 Effort is substantially the same year to year.

 Every 2 or 3 years there is a substantially more effort expended when developing priorities.

If DIFFERENT describe why this strategy is in place.________________

Which of the following sources of information is utilized by the either the Board and/or Advisory Council to establish PAIMI Program priorities? [Interviewers should inquire about each source of information.]

	Board	Council
Performance on prior year’s PPR goals & objectives		
Asking PAIMI clients		
Outreach to individuals that speak on behalf of underserved & minority communities for opinion on greatest needs		
Outreach to organizations that speak on behalf of underserved & minority communities for opinion on greatest needs		
Advisory Council’s outreach to PAIMI-eligible citizens and family members for gathering information on greatest needs		
Outreach to community leaders who speak on behalf of underserved & minority communities for opinion on greatest needs		
Demographic data about citizenry (e.g., Census data)		
Epidemiological data (e.g., incidence of mental health diagnoses, impact of toxins in environment on health)		
Reports on problems such as deaths, abuse, neglect		
I&R (information & referral) calls to the P&A, including data from persons refused services		
Any federal government initiatives such as the New Freedom Initiative or Suicide Prevention		

Describe ways that the PAIMI Program is currently collaborating with other mental health advocacy groups.

Group/organization	Nature of the collaboration

What activities does your PAIMI Program not engage in as other organizations fill that role in your state/territory?

Are there disagreements between your PAIMI Program and other mental health advocacy groups?

 No

 Yes

IF YES, please describe:_______________________________________

Does the PAIMI Program have a representative on the state mental health planning council?

 No

 Yes

IF NOT, why not? ___________________________________________

Describe ways in which the PAIMI program is currently working with the state mental health authority (SMHA), for example jointly pursuing legislation, participating on a work group to address abuse, etc.

In the past 2 years of PPR reporting, has the PAIMI program added or dropped key goals?

Goal(s) added:____________________________________________

Goal(s) dropped:___________________________________________

What goals have not been pursued due to lack of resources such as staffing, funding, etc.?

What joint activities if any do the PAIMI Advisory Council and Governing Board engage in?

 No joint activities except for cross board membership

 Special invitations to attend key meetings

 Open invitation to attend meetings

 Fundraising

 Public educational activities

 Other, describe _________________________________________________

One of the responsibilities of a PAIMI Advisory Council is to author the section of the PPR. How does your Advisory Council manage this responsibility?

 P&A staff draft the report section for Council review

 Council Chair drafts the report section for Council member then P&A staff review

 Selected Council members draft with input from P&A staff

 Selected Council members draft without input from P&A staff

 Other (describe)

_____________________________________________________

How regularly does the PAIMI Program provide the Advisory Council with the following types of organizational and operations data?

Financial information  Often  Sometimes  Rarely

Info about client grievances  Often  Sometimes  Rarely

Intake & referral  Often  Sometimes  Rarely

Status of lawsuits  Often  Sometimes  Rarely

Advocacy casework  Often  Sometimes  Rarely

 Other, please describe ___________________________________________

Section 5. Federal support

What do you understand to be the role and responsibilities of your GPO (Government Project Officer)?

In the past 2 years, regarding what issues have you been in contact with the CMHS Project Officer assigned to your PAIMI Program?

In the past 2 years has your P&A made use of SAMHSA Grants Management consultation for fiscal reporting?

 No If NO, why?_____________________________________

 Yes If YES, for what purposes? ________________________

In the past 2 years have you experienced any difficulty with the PAIMI Application?

 No

 Yes

If YES, What guidance, if any, has been useful for completing the annual

PAIMI Program Application?

	Very Useful	Somewhat Useful	Not Useful	Not Applicable
Application instructions
Assistance from CMHS Project Director
Assistance from CMHS Project Officer
Assistance from NDRN (formerly NAPAS)
Other, please describe: ____________________________
Not availed of federal assistance for completing Application ____________________________

In the past 2 years have you been advised there were any problems with your PAIMI Application?

 No

 Yes

If YES, please describe ______________________________________

In the past 2 years have you had any difficulty with the PAIMI Program Performance Report (PPR)?

 No

 Yes

If YES, What guidance, if any, has been useful for completing the annual PPR?

	Very Useful	Somewhat Useful	Not Useful
PPR instructions
Assistance from CMHS Project Director
Assistance from CMHS Project Officer
Assistance from NDRN (formerly NAPAS)
Other, please describe _________________________________
Not availed of federal assistance for completing PPR ________________

Section 6. Federal oversight of PAIMI operations

In the last 2 years, has your PAIMI program been advised that you are not in compliance with federal statute or regulations?

 No

 Yes

If YES, please describe______________________________________

In the past 2 years has a SAMHSA fiscal review identified any problems related to PAIMI expenditures?

Are there different expectations (e.g., definition of what constitutes a case, etc.) for PAIMI Program performance or operations that arise from the involvement of federal agencies responsible for different aspects of administration of the P&A programs (CMHS, SAMHSA, ADD, RSA)?

 No

 Yes

If YES, please describe_______________________________________

Have there been circumstances in the past 2 years in which you have experienced conflict between administering the PAIMI program and other P&A programs? Examples of areas of conflict might be:

Areas of potential conflict:	No conflicts occurred	Yes conflicts occurred	If YES please describe conflict
Program organization
Funding accountability
Client selection
Reporting on activities
Other, please note:_______________________

PPR (PAIMI Program Performance Report)

How useful is the PPR for administering the PAIMI Program?

 Very useful
 Somewhat useful

 Minimally useful
 Not useful at all
 Don’t know/not sure

What changes, if any, would improve the usefulness of the PPR for PAIMI operations? Are there other PPRs that you would recommend as better reporting tools?

How do you rate the usefulness of the written reports resulting from the CMHS annual PPR peer review process initiated in 2004?

2004 2005

  Very useful
  Somewhat useful

  Minimally useful
  Not useful at all
  Don’t know/not sure

What of value have you learned from the peer review process reports?

What changes, if any, have you made based on the report findings generated by the peer review process?

What would make the peer review reports more useful (e.g., more detailed information on strengths and areas in need of improvement)?

CMHS Site Monitoring

Has your PAIMI Program experienced a site visit since the new protocol was initiated in 2002?

 No [If answer is ‘No’, skip to question 52.]

 Yes

Did the site visit team recommend technical assistance or make any other recommendation(s)?

 No

 Yes

If Yes, what was the recommendation(s) and/or technical assistance?

Was there specific information that the site visit team relayed in the exit interview that was helpful?

 No If No, what would improve that part of the process?

 Yes If Yes, please provide example of information that was helpful:

Did you consider the information conveyed by the site visit’s written report to be helpful?

 No If No, what would improve the process?

 Yes If Yes, please provide example of information that was helpful:

__________________________________________________________

Was information from the site monitoring used to make changes to your PAIMI program?

 No

 Yes,

If YES, What was the problem and how was it addressed?

___________________________________________________________

What changes would improve federal monitoring efficacy?

Technical Assistance

What needs for technical assistance related to PAIMI services have you identified in the past 2 years? Please list:_______________________________

What was the outcome of any identified TA need?

In the past 2 years, what needs for technical assistance has CMHS identified for improvement of your PAIMI Program?

Please indicate the usefulness of these methods of delivering technical assistance for your PAIMI program:

Technical Assistance Type	Not utilized	Very useful	Somewhat useful	Not useful
Skills Building conference
Fiscal Managers conference
P&A/CAP Annual Meeting
NAPAS Staff
List serves: Abuse & neglect
ADA
Board of Directors & CEO issues
Community integration
Housing
Resource advocacy
Return to work
Public policy
DAD support
Secure confinement
Voting
Legal Directors
Legal back up centers: Bazelon Center for Mental Health Law
Center for Public Representation
Disability Law Project
National Health Law Project
Neighborhood Legal Services in Buffalo, New York
NAPAS/NDRN resources & educational materials: Legal Director’s Q&A ADA-Olmstead MH Advanced Directives

Is the technical assistance (TA) available sufficiently addressing PAIMI issues?

 No, IF NO, please explain__________________________________

 Yes

Section 7. PAIMI Program Operations

Are the issues PAIMI has addressed in the past 2 years influenced by specific local, state, and/or federal policy directions/initiatives?

	Not influenced	Yes influenced	If YES, explain
Local issues
State issues
Federal issues

All PAIMI programs operate in political environments and this evaluation is interested in the pressure resulting from that environment. In the past 2 years have you experienced political pressure that has interfered with pursuing any mental health system advocacy effort? If YES, please explain.

In advocating for the state mental health appropriations over the past two years, the directions of the PAIMI and the State Mental Health Authority have been aligned

Always or most of the time
Often
Sometimes
Rarely or never
Not applicable (PAIMI does not advocate for the mental health appropriation.)
Don’t know

When the PAIMI program has advocated for changes in mental health regulations over the past 2 years, how often has the State Mental Health Authority been receptive to proposed changes?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No changes in regulation initiated in the past two years.)

 Don’t know

When the PAIMI program has advocated for changes in policies and practices in the public mental health system over the past 2 years, how often has the State Mental Health Authority been receptive to proposed changes?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No policy changes initiated in the past two years.)

 Don’t know

When the State Mental Health Authority has set up advisory committees on changes to the service system in the past 2 years, how frequently has the PAIMI been invited to participate?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No advisory committees have been set up in the past two years)

 Don’t know

During the past 2 years, when the PAIMI has addressed systemic problems of client abuse/neglect/rights violations that could lead to litigation, what has been the typical path(s) to resolution? Check all that apply.

 The problem has been dealt with through informal negotiations between the P&A and the state mental health authority.

 The problem has been dealt with through filing of a lawsuit followed by a negotiated but formal resolution (i.e., a settlement agreement or consent order).

 The problem has been dealt with through filing of a lawsuit followed by court proceedings that have (or appear likely to) end with a judicial ruling.

 The P&A has not dealt with systemic problems of client abuse or neglect in the public mental health system over the past two years.

How much contact does the P&A Executive Director or key PAIMI staff typically have with the State Mental Health Program director or other senior management staff?

 Meet with or talk to once per week

 Meet with or talk to 2-3 times per month

 Meet with or talk to about once per month

 Meet with or talk to at least four times per year

 Meet with or talk to once or twice per year

 Never or almost never

Generally, would you characterize those contacts as?

 Always adversarial

 Mostly adversarial

 Mixed

 Mostly cooperative

 Always cooperative

How many P&A staff members are dedicated solely to PAIMI program issues?

How many full time equivalent staff are working on PAIMI issues?

Attorney ____________ FTE

Paralegal ____________ FTE

Advocate ____________ FTE

Consumer-Advocate ____________ FTE

Volunteer staff ____________ FTE

Management staff _________ FTE

Administrative support staff _________ FTE

Other PAIMI Program services (such as education staff, etc.) ______FTE

Have we covered the ways in which your PAIMI utilizes consumers?

 Yes

 No, IF NO, please explain__________________________________

When you have staff vacancies in positions that have responsibility for PAIMI issues, how long on average does it take to fill the position?

What factors, if any, constrain program operations or interventions in the following activities?

Program Activity Area	Constraint
Intake & referral
Non legal casework
Individual litigation
Class litigation
Legislative advocacy
Community organizing

When an individual problem is identified whose resolution appears to necessitate litigation, does the PAIMI program have the capacity to initiate and follow through with individual litigation?

When a systemic problem is identified whose resolution appears to necessitate litigation, does the PAIMI program have the capacity to initiate and follow through with systemic reform litigation? What barriers present when determining capacity to follow through with systemic reform litigation?

What do you think the PAIMI Program has accomplished in this state/territory in the past 2 years?

Does your PAIMI have timely access to the following?

	Timely access	Not timely	Mixed: timely & not
Death reports from MH institutional settings			
Reports of abuse/neglect from institutional settings			
Unannounced visits in 24 hour facilities			
Announced walking around in 24 hour facilities			
Privacy to meet with clients & prospective clients			
Access to client records with client’s consent			

Which of the following does your staff visit and along what timelines?

PAIMI visits to:	Quarterly or less often	Monthly	Weekly	Daily
Jails/prisons				
Homeless Shelters				
Nursing Homes				
Psychiatric Hospitals/Veteran’s Administration facilities				
Board & care/licensed assisted living/residential care				

In the past 2 years, what has been the most challenging area of access for your PAIMI program? This question can cover access of any type be it records, information or physical presence.

In the past 2 years, has your P&A contracted out litigation representation of PAIMI clients?

 No

 Yes

If YES, please give a brief description____________________________

How does your PAIMI program ensure that volunteers assigned to work on PAIMI issues who perform the same functions of staff are provided equivalent training and supervision?

Technological Support

What types of information technology are in use for PAIMI program operations? Check all that apply.

 website

 email contact with clients

 database for client information

 information and referral tracking database

 database for PAIMI fund tracking

 subscription to legal analysis services such as LexisNexis, etc.

 electronic bulletin boards

 OTHER, please describe: __________________________________

What additional information technology or other technological support is needed?

Client Problem Identification

How does the organization assure that the PAIMI program activities are delivered in a culturally competent manner?

In the past 2 years, what strategies did the P&A utilize to identify systemic problems for people with serious mental illness?

Training and Education

Are there any areas in which more staff training is needed?

 No

 Yes If YES, what if any barriers exist to implementing the needed training? ________________________________________________________

Quality Management

How does your P&A provide supervision over the quality of work rendered to PAIMI clients?

How does the PAIMI program solicit input regarding satisfaction of people who have been clients or who have participated in program activities? What mechanism is used to utilize that information?

Does your P&A track the number of PAIMI clients dually diagnosed with mental illness and developmental disabilities?

 No

 Yes, IF YES, what percent of PAIMI clients are also clients of other P&A programs? __________%

How does your P&A ensure that PAIMI advocacy activities (outreach through representation) reach people who may have difficulty accessing the P&A?

A. People who live in remote or isolated areas

__________________________________________________________

B. People who have extremely limited access to phones

___________________________________________________________

C. People with cognitive limitations

___________________________________________________________

D. People who speak a language other than English or communicate through sign language

___________________________________________________________

E. People with different cultural background OR communities of color

___________________________________________________________

Do you track the numbers of people denied for PAIMI services?

 No

 Yes If YES, would you be able to provide estimate of the categories of reasons, such as does not meet eligibility criteria?

 Does not meet eligibility criteria

 Conflict of interest

 Prior client

 Lack of resources

 Does not meet priority areas for service

 Other ___________________________________________

Does the P&A have conflict of interest policies and procedures that provide effective guidance for avoiding potential areas for conflicts of interest?

 No

 Yes

IF YES, Have you had any instances of conflict of interest that were not resolved by these policies, and if so, what were the issue(s)?

__________________________________________________________

Grievance

How does your PAIMI program ensure clients are aware of and able to access the client grievance system?

Does the Board or its designee track or trend PAIMI client grievances over time to determine if there are issues to address both systemically and/or with individual employees?

 No

 Yes

SECTION 8: This section provides information regarding your background and experience.

How long have you been the P&A Executive Director (ED)? ___________

Prior to assuming the ED position, were you employed in a P&A staff position?

 No

 Yes at this P&A

 Yes at another P&A

What is the length of time you have been involved in either P&A or PAIMI work?

 less than 2 years

 2 to 5 years

 5 to 10 years

 10 to 20 years

 more than 20 years

94. Answering the following question is voluntary.

Are you of Hispanic or Latino ethnicity?  Yes  No

95. Answering the following question is voluntary. What is your race?

Select one or more.

 American Indian or Alaska Native

 Asian

 Black or African-American

 Native Hawaiian or Other Pacific Islander

 White

96. Answering the following question is voluntary. What is your experience with

people with mental illness? Check all that apply.

 I am or have been a recipient of mental health services

 I am or have been a client of PAIMI/P&A services

 I have a family member with mental illness:

 I have work experience representing people with serious mental illness

 I have work experience as a provider of mental health services

 I do not have personal or work experience but am interested because (please describe) ____________________________________________________

 Other (please describe) ___________________________________

97. Answering the following question is voluntary. Are you labeled with a

disability?

 No

 Yes

If YES, please check all that apply: (Remember this is a voluntary response.)

 Mental illness

 Autism

 Cerebral Palsy

 Brain injury

 Seizure disorder/neurological problem

 Chemical dependency

 Vision or hearing impairments

 Physical disability

 Communication disorder

 Alzheimer’s disease

 Down Syndrome

 Other disabilities not listed

98. What is your background and experience? Check all that apply.

 Attorney

 Mental health professional (psychiatric nursing, social work, psychology, psychiatry, etc.)

 Advocate

 Other, please describe ______________________________________

Section 9: Your Comments

Thank you for taking the time to answer these questions. Please feel free to provide us with comment or feedback on this survey process.

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-xxxx. Public reporting burden for this collection of information is estimated to average 2 hours per respondent per year, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

Appendix B

Form Approved

OMB No.: 0930-XXXX

Expiration Date: MM/DD/YY

See burden statement on last page.

National Evaluation of the Protection & Advocacy for

Individuals with Mental Illness (PAIMI) Program

PAIMI Advisory Council Chair Survey

Thank you for helping with the first National Evaluation of the PAIMI Program by completing the attached questionnaire. We expect that it will take about 30 minutes to complete this survey. This survey is being conducted by the Human Services Research Institute (HSRI) through a contract with the Substance Abuse and Mental Health Services Administration’s Center for Mental Health Services (CMHS) for the first National Evaluation of the PAIMI Program. These surveys will provide information from PAIMI Advisory Councils across the states and territories to determine a) to what extent PAIMI programs are supporting the work of PAIMI Advisory Councils, b) factors that influence Council performance, and c) Council member impressions of their PAIMI Program operations.

Please know that there is not a ‘right’ answer to these questions. Comparing programs with vast differences in structure, operations and funding is challenging. To address this we as evaluators have included a range of answers to try to address this program diversity. If you find yourself choosing a ‘no’ or ‘don’t know’ response, please do not feel as if your program is doing anything wrong. Although most questions have a list of responses from which to choose, please feel free to further explain your Council’s experience.

Responses are strictly confidential. While the identity of the 20 Protection and Advocacy agencies sampled for this evaluation will be noted in our report, the responses of the PAIMI Advisory Chairs and other respondents will not be shared or revealed.

INSTRUCTIONS:

	You have the option to complete this survey either on line or in printed form. Whether you are completing this survey on line or printed version, all responses are confidential. If you come to a question that you feel uncomfortable answering, skip it.
	On line surveys are automatically delivered into a database when you complete and sign off. If you are completing a printed copy of the survey, please return it to us in the enclosed envelope that is ready for mailing. Please complete your survey as soon as possible.
	If you would like assistance with completing this survey, or if you need an interpreter, please contact Elizabeth Pell at the Human Services Research Institute. Elizabeth’s phone # is 617-876-0426 x 2307 or email epell@hsri.org. Collect calls will be accepted.

SECTION 1: This section provides information about the structure of your PAIMI Advisory Council.

How long have you been a member of this PAIMI Advisory Council?

 Less than 1 year

 1 to 2 years

 2 to 3 years

 More than 3 years, please write in length of tenure _____________

How long have you served as PAIMI Advisory Council Chair?

 Less than 1 year

 1 to 2 years

 2 to 3 years

 More than 3 years, please write in length of tenure _____________

Did you have experience being on an advisory or governing board for any other organization before being selected to the PAIMI Advisory Council?

 No

 Yes

How were you nominated to serve on the Council? Check all that apply.

 I knew someone else on the Council

 I formerly used P&A/PAIMI services

 I was recruited by P&A staff

 The organization where I work/volunteer nominated me

 I don’t know

 Other (please describe) _____________________________________

Who sets the agenda for Advisory Council meetings?

 Council members set the agenda without input from P&A staff

 Council members set the agenda with input from P&A staff

 P&A staff set the agenda

Is the Advisory Council membership representative of the population demographics (race, income, recipients of mental health services, family members, professionals, etc.) present in your state/territory?

 No

 Yes

If NO, what type of representation is needed to be representative? Check all that apply.

 People who have used the mental health system, e.g., recipients

 People who are family members

 People who are generally knowledgeable about the mental health system

 People who represent minority or underserved communities

 Other (for example, race, age, geographic area, etc.), please describe

_____________________________________________________

Do you currently have any open or unfilled member positions on the Council?

 No

 Yes

If Yes, what reason(s) primarily account for the unfilled positions?

 Recent retirement/rotation & not enough time to replace

 Difficulty finding representatives with necessary ethnic/racial diversity

 Difficulty finding representatives with sufficient knowledge of mental health

 Difficulty finding someone to volunteer their time to Council

 Other, please describe _____________________________

Approximately how much turnover (both term expiration and unplanned leaving) occurs in the Council membership per year?

 1 or fewer members leave per year

 2-3 members leave per year

 More than 3 members

How many P&A staff attend Advisory Council meetings? __________

If staff attend, how do you perceive the presence of staff?

 Perceived as not helpful

 Perceived as helpful at times

 Perceived as very helpful

Does the Council have committees or other work groups?

 No

 Yes

SECTION 2: This section asks questions regarding P&A agency support to the PAIMI Advisory Council.

Whose role is it to take minutes of Advisory Council meetings?

 Council members keep minutes without assistance from P&A staff

 Council members keep minutes with assistance from P&A staff

 P&A staff keep minutes and distribute to Council members

Does the P&A provide new Advisory Council members with an orientation?

 No

 Yes

If Yes, whose responsibility is it to provide the new member orientation?

 Advisory Council members only

 P&A staff only

 Both Advisory Council & P&A staff share responsibility

In your opinion, how long does it typically take for a new Council member to become a fully contributing member?

 Less than 1 year

 1 to 2years

 2 to 3 years

 More than 3 years

If there is an orientation for new members to the Advisory Council, which of the following topics are covered? Check all that apply.

 History of the PAIMI Program

 Federal agencies responsible for oversight of the PAIMI Program

 Setting PAIMI priorities

 P&A agency operations & staff

 Current casework & litigation of the P&A

 Understanding abuse, neglect, civil rights violations

 Accommodations available to help you attend meetings and participate fully

 Trainings on topics such as leadership skills, cultural competency, participating in meetings, negotiation, etc.

 Federal reporting requirements (PAIMI Application, PPR, Advisory Council Annual Report)

 Other, please list ________________________________________________

In what areas are policies and procedures in place to guide Advisory Council members? Check all that apply.

 Remaining free of conflicts of interest

 Keeping information confidential

 Role & responsibilities of Governing Board

 Role & responsibilities of Advisory Council

 How to conduct outreach on priorities

If policies and procedures exist, do members receive a copy of these policies and procedures?

 No

 Yes

Is there a designated P&A staff member to provide on-going support to the PAIMI Advisory Council?

 No

 Yes

Does your PAIMI Program offer any of the following types of assistance so that Advisory Council members can attend Council meetings or conferences, such as that offered by NDRN (formerly NAPAS)? Check all that apply.

 Reimbursement for transportation after you submit receipts

 Payment for transportation up front (in advance)

 Payment for daycare or childcare for dependents with disabilities

 Parking

 Meals

 Motel

 Per diem expenses

 Stipends or allowance

 Other, please list__________________________________________

In the past 2 years, have you or any other Council members attended one of the national P&A conferences?

 No

 Yes

If Yes, did you find the experience helpful to your work as a Council member?

 No

 Yes

SECTION 3: This section asks questions about the Council’s relationship with the P&A Governing Board.

In addition to the Advisory Council Chair, does your Council have other representatives that sit on the P&A Governing Board?

 No

 Yes

If Yes, how are representative(s) nominated?

 Nominated by Council Chair

 Nominated by Council membership

 Nominated by P&A Executive Director and/or staff

 Person applies independently

 Not sure/don’t know

 Other process, please describe:_____________________

_________________________________________________

And then, how are representative(s) selected?

 Selected by Council Chair

 Selected by Council membership

 Selected by P&A Executive Director and/or staff

 Not sure/don’t know

 Other process, please describe:_____________________

_________________________________________________

In what ways does having Council representation on the Board facilitate communication or collaboration between the advisory and governing bodies?

 No collaboration exists

 Good to have a representative but it doesn’t seem to have much impact

 Provides useful forum for mutual exchange of ideas

 Don’t know or unable to determine

 Other _________________________________________________

Do you as the PAIMI Advisory Chair have a vote on the P&A Governing Board?

 Not sure/don’t know

 Yes

 No

Which of the following choices best represents the relationship between your Advisory Council and P&A Governing Board?

 Not collaborative

 Some interaction but only on required tasks such as priority setting & the annual report (PPR)

 Interaction is frequent and open

 Very collaborative and respectful relationship

Does the PAIMI Advisory Council send recommendations for change in P&A or PAIMI policies and procedures or other organizational matters to the P&A Governing Board?

 Don’t know

 No

 Yes, If YES, Is the P&A Governing Board receptive to the Council’s suggestions?

 Don’t know

 No

 Yes

SECTION 4: This section asks questions regarding the priority setting process.

Who is primarily responsible for coming up with the Advisory Council’s PAIMI Program priorities?

 P&A staff

 Advisory Council members

 Both P&A staff AND Advisory Council members

 Don’t know

Describe the method the PAIMI Advisory Council uses to set priorities. Check all that apply.

 Review prior year’s recommended PAIMI priorities to Governing Board

 Review prior year’s PPR (Program Performance Report) goals & objectives

 Review priorities recommended by P&A staff

 Outreach to leaders who speak on behalf of underserved & minority communities for opinion on greatest needs

 Outreach to PAIMI-eligible individuals and family members for opinion on greatest needs

 Examine demographic data about citizenry such as the rate of incarceration of people with mental illness in jails, US Census data for income information, the prevalence of mental illness in the state/territory, etc.

 Examine federal government initiatives such as the New Freedom Initiative or Suicide Prevention

 Looking at changes in state policies or services such as mental hospital closings or rates or a bill promoting involuntary commitment

 Other, please describe: ______________________________________

In what ways does the Council make use of quantitative information available for determining annual priorities?

 Review data on the PAIMI’s representation of individuals

 Review data on incidence of abuse, neglect, deaths

 No data is reviewed

 The Council reviews other data. Please describe:_________________

How does the PAIMI Advisory Council (PAC) provide opportunities for public input into their priority setting process? Check all that apply.

 Advisory Council does not engage in activities to gather public input

 PAIMI staff organize outreach meetings

 Advisory Council attend outreach meetings

 PAIMI contracts with other organization(s) to conduct outreach

 PAIMI staff collaborate with other organizations to develop recommended PAIMI priorities

 Review outreach strategies or information gathered from other PACs

 Surveys

 Focus groups

 Attend meetings of organizations such as NAMI, consumer empowerment

 Other, please describe ______________________________________

Are the opinions of PAIMI clients and PAIMI-eligible individuals solicited during the priority setting process?

 No

 Yes

 Don’t know

If Yes, how are these individuals’ recommendations solicited? _________

Currently PAIMI Advisory Councils are to establish priorities on a yearly basis. What is your opinion of the length of an annual priority setting cycle?

 Annual priority setting is sufficient

 At times annual priorities are sufficient and other times constraining

 Annual priority setting is too constraining

Does the P&A Governing Board generally adopt priorities recommended by the Advisory Council?

 Usually adopts all recommended priorities

 Adopts some but not all recommended priorities

 Usually does not adopt recommended priorities

When the Board does not adopt a recommended priority, how is this communicated and resolved with the Advisory Council?

How does the Advisory Council determine whether the PAIMI Program has met the annual goals and objectives reported in the annual PPR (Program Performance Report)? Check all that apply.

 P&A staff advise the Advisory Council on the extent to which annual goals and objectives were met

 Advisory Council has data and makes an independent determination as to whether goals and objectives were met by the PAIMI Program

 Don’t know/not sure

 Other process, please describe: _______________________________

One of the responsibilities of a PAIMI Advisory Council is to write a section of the PAIMI annual report. How does your Council manage this responsibility?

 P&A staff draft the report section for Council review

 Council Chair drafts the report section for Council member then P&A staff review

 Selected Council members draft with input from P&A staff

 Selected Council members draft without input from P&A staff

Section 5: This section addresses PAIMI Advisory Council activity regarding their role as monitors of PAIMI activity.

Are P&A staff available to meet and discuss issues of concern raised by the Advisory Council?

 Staff are available

 Sometimes available and sometimes no

 Staff are not available

Does the P&A regularly provide the Advisory Council with any of the following types of organizational and operations data? Check all that apply.

 Financial information about the PAIMI Program revenues & expenditures that you can understand

 PAIMI client grievances

 Intake & referral issues

 Status & outcomes of representation for individual lawsuits

 Status & outcomes of representation in class action lawsuits

 Status & outcomes of advocacy casework

 0ther, please describe ______________________________________

 None/No operational information is shared with the Council

Does the PAIMI Advisory Council provide recommendations to the P&A Executive Director or other staff regarding other aspects of performance such as reviewing policies and procedures, such as...? Please check all that apply.

 Staff to be hired

 Reviewing draft policies and procedures

 Participating in developing the agency strategic plan

 Fundraising

 Marketing

 No recommendations are made

 Other, please describe ____________________________________

What is your impression of the usefulness of the PAIMI Program Performance Report (PPR) as a report on the effectiveness of the PAIMI program’s work?

 Not familiar with this report

 Not useful

 Somewhat useful

 Very useful

Has information from the CMHS PPR Peer Reviews of either 2003 or 2004 been communicated to you as the Council Chair?

 No

 Yes

 Not sure what the PPR Peer Review process is

Are PAIMI clients able to talk to and meet with their advocate/attorney at times convenient to them?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

39. Do PAIMI clients think that their advocate/lawyer listens to their story and truly understands their circumstances?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

40. Do PAIMI clients receive training in self-advocacy skills?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

41. Are clients encouraged to speak for themselves?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

42. Who makes the decisions in individual PAIMI cases?

 Clients generally make their own decisions

 Advocates/attorneys generally make the decisions

 Decisions are generally made together between P&A staff & clients

 Don’t know

43. Do you think that PAIMI clients believe that their advocate/attorney helps them to manage their rights problems?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

44. Are P&A agency staff respectful of clients’ cultural background (for example your religion, race, language)?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

45. Are clients encouraged to connect with consumer-run programs or supports?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

Section 6. This section addresses Advisory Council member impressions of PAIMI Program quality.

47. Do the Advisory Council members feel that their input is respected by the P&A?

 No

 Yes

 Not sure

48. If in the past 2 years the Advisory Council has recommended PAIMI priorities to the Board which have not been adopted, please describe the rejected priority/priorities & rationale for not adopting:

49. Do the Advisory Council members think they are effective in moving the PAIMI in directions members wish to see it move?

 No

 Yes

 Not sure

50. Overall, how would you rate the quality of the representation that PAIMI clients receive?

 Excellent

 Good

 Both good and bad

 Needs a lot of improvement

 Terrible

What do you think the PAIMI Program in your state has accomplished in the past 2 years?

SECTION 7: This section provides information regarding your race, culture, and experience with mental illness. All of these questions are voluntary. Please feel free to skip over any you do not want to answer.

What is your age?

 under 35

 35 – 54

 55 - 74

 75 or older

52. What is your gender?

 Female

 Male

53. Answering the following question is voluntary.

Are you of Hispanic or Latino ethnicity?  Yes  No

54. Answering the following question is voluntary. What is your race?

Select one or more:

 American Indian or Alaska Native

 Asian

 Black or African-American

 Native Hawaiian or Other Pacific Islander

 White

55. What other types of diversity or perspective do you personally bring to the Council? Check all that apply. It is voluntary to answer this question.

Sexual orientation:

 Gay/lesbian

 Transgender identity

Geographic:

 Urban dweller

 Rural

I bring other diversity to the Advisory Council. Please describe: ______________________________________________________

56. What is your experience with people with mental illness? Check all that apply.

 I am or have been a recipient of mental health services

 I am or have been a client of PAIMI/P&A services

 I have a family member with mental illness

 I am, or have been, a peer advocate for persons with psychiatric disability

 I have work experience as a provider of mental health services

 I do not have personal or work experience but am interested because (please describe) ____________________________________________________

 Other (please describe) ____________________________________________________

57. Are you self-identified with a disability?  No  Yes

If Yes, Please know that this information will be kept strictly confidential and is VOLUNTARY to answer. Check all that apply.

 Mental illness

 Autism

 Cerebral Palsy

 Brain injury

 Seizure disorder/neurological problem

 Chemical dependency

 Vision or hearing impairments

 Physical disability

 Communication disorder

 Alzheimer’s disease

 Down Syndrome

 Other disabilities not listed

58. What was your personal income last year?

 Below $15,000

 $15,001 - $25,000

 $25,001 - $50,000

 $50,001 - $75,000

 Over $75,000

59. Please indicate your employment status.

 Volunteer

 Employed part time

 Employed full time

 Retired

 Unemployed

60. What level of education have you received?

 some schooling but not enough for a GED or high school diploma

 GED or high school diploma

 some college

 college degree

 graduate degree

Section 8: Your Comments

Thank you for taking the time to respond to these questions. Please use the space below for any other comments you would like to make about PAIMI or the P & A program in general.

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-xxxx. Public reporting burden for this collection of information is estimated to average less than one hour per respondent per year, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

Appendix C

Form Approved

OMB No.: 0930-XXXX

Expiration Date: MM/DD/YY

See burden statement on last page.

National Evaluation of the

Protection & Advocacy for Individuals with Mental Illness (PAIMI) Program

Client Survey

Thank you for helping with the National Evaluation of the PAIMI Program by completing this survey. This survey and follow up meeting is being conducted by the Human Services Research Institute (HSRI) through a contract with the Substance Abuse and Mental Health Services Administration’s Center for Mental Health Services (CMHS) for the first National Evaluation of the PAIMI Program. Your opinions will be compiled with other clients’ across the country to evaluate PAIMI client satisfaction with Protection and Advocacy services.

We expect that it will take about 20 minutes to complete this survey. Surveys will be handled with strict confidentiality and privacy. This means that while our report will identify the 20 Protection and Advocacy agencies we visited, the answers of all those surveyed will be kept private. You will not be identified, nor will your responses be shared with the Protection and Advocacy agency. Our report will combine all the PAIMI Program client information and represent the collective findings from all persons who participated anonymously.

The enclosed cash is an expression of our appreciation for the time you are taking to in complete this survey and participate in the meeting to follow.

INSTRUCTIONS:

For most questions, check the box that applies to you. For some questions you may be asked to describe your answer with more detail. If you do not feel comfortable answering a question, please skip over it.

If you have a question, please do not hesitate to ask one of the HSRI staff for clarification.

Your answers will be kept confidential. Confidential means that your answers can not be traced to you. HSRI will keep your identity private.

When you have completed the questionnaire, please hand it to us in the enclosed envelope.

THANK YOU!

SECTION 1: This section asks questions about how you learned about the Protection & Advocacy (P&A) agency and becoming a PAIMI program client.

How did you learn about the Protection & Advocacy (P&A) agency?

 I have used this P&A before

 I was referred by another agency to contact this P&A

 Friends advised me to contact the P&A

 Family advised me to contact the P&A

 Mental health professionals referred me

 I saw a flyer/brochure

 I heard a radio announcement

 I learned about it when attending a community meeting

 I found out about the P&A searching the internet for help

 P&A staff visited the place I am residing.

 Other: please explain _______________________________________

When you were first explaining your situation, were you advised that there are types of cases the P&A could help with as well as cases they could not provide help with?

 No

 Yes

Did you need services in a language other than English?

 No

 Yes

If Yes, were you provided services in your language?

 No

 Yes

Which of the following categories best describe the problem(s) for which you sought help from the P&A? Please check all that apply.

 lack of access to quality treatment

 involuntary admission to a facility

 involuntary treatment

 abuse or neglect in a facility

 abuse or neglect in a community setting

 housing

 employment

 securing public benefits

 other, please describe: ______________________________________

SECTION 2: This section asks questions regarding your satisfaction with PAIMI services.

Which type of staff is working with you?

 Advocate who is not an attorney

 Attorney

 I have both an advocate and an attorney

 I don’t know what kind of staff is working with me

Are you able to talk to and meet with your advocate/attorney at times and places that were convenient to you?

 No

 Yes

Are you able to meet with or talk to your attorney/advocate often enough so that you know the status of your case and when decisions have to be made?

 No

 Yes

Do you think your advocate/attorney listens to your story and truly understands your circumstance?

 No

 Yes

Who makes the decisions in your case?

 P&A staff make the decisions

 I make my own decisions

 The P&A staff and I make decisions together

Are Protection and Advocacy agency staff respectful of your cultural background (for example your religion, race, language)?

 No

 Yes

Are you given information about your rights?

 No

 Yes, IF YES, please describe whether this was adequate or not:

 I was given more information about my rights than I needed.

 I needed more information about my rights than I was given.

 I received just the right amount of information about my rights.

Have you been provided with self advocacy skills or resources?

 No

 Yes, IF YES, was the self advocacy information adequate?

 I was given more information on self advocacy than I needed.

 I needed more information on self advocacy than I was given.

 I received just the right amount of information on self advocacy.

If applicable, were you encouraged to connect with consumer-run programs or supports?

 No

 Yes

 Not applicable

Do you believe that your advocate/attorney is doing everything they can do to obtain the outcome that you want?

 No

 Yes

Has your advocate/attorney helped with the problem that brought you to the P&A?

 No

 Yes

What difference in your life has representation by the P&A made?

 No change, nothing is different

 Some change was made but not everything I wanted

 I understand what I can do to improve my situation

 My situation is much improved

Do you believe that you have received representation that is as good as representation given to people who pay for legal services?

 No

 Yes

 Don’t know

Overall, how would you rate the quality of the representation that you received?

 Excellent

 Good

 Both good and bad

 Needs a lot of improvement

 Terrible

Would you use this P&A again?

 No

 Yes

Would you recommend this P&A to others?

 No

 Yes

Have you been advised that you could complain about the Protection and Advocacy agency’s services should you have felt the need to do so?

 No

 Yes

Have you felt free to complain?

 No, If No, why not? ______________________________________

 Yes

Have you made a complaint about the service you received from the P&A?

 No

 Yes

If Yes, do you think that your complaint was heard fairly?

 No

 Yes

SECTION 3: This section asks questions about your personal characteristics. This information also will be kept strictly confidential. It is your choice to answer these questions, please skip any questions you do not feel comfortable answering.

What is your age?

 under 18

 18-34

 35 – 54

 55 - 74

 75 or older

What is your gender?

 Female

 Male

 Transgender

27. Answering the following question is voluntary.

Are you of Hispanic or Latino ethnicity?  Yes  No

28. Answering the following question is voluntary. What is your race?

Select one or more:

 American Indian or Alaska Native

 Asian

 Black or African-American

 Native Hawaiian or Other Pacific Islander

 White

 I do not wish to disclose

29. Where did you live when you became a P&A client?

 Home or apartment

 Group home

 Board and care facility

 Institution for mental health treatment

 Facility for substance abuse treatment

 Jail or prison

 Other, please describe: _________________________________________

30. Are you labeled with a disability?

 No

 Yes

IF YES, Which of the following labels could be used to describe your disability? Please know that this question is voluntary to answer. If you choose to answer, please check all that apply.

 Mental illness

 Autism

 Cerebral Palsy

 Brain injury

 Seizure disorder/neurological problem

 Chemical dependency

 Vision or hearing impairments

 Physical disability

 Communication disorder

 Alzheimer’s disease

 Down Syndrome

 Other disabilities not listed

31. Please indicate your employment status.

 Volunteer

 Employed part time

 Employed full time

 Not employed

 Retired

32. What level of education have you received?

 some schooling but not enough for a GED or high school diploma

 GED or high school diploma

 some college

 college degree

 graduate degree

33. What type of housing do you live in now?

 rent my own home or apartment

 own my own home or condo

 live with my family

 institution

 nursing home

 jail

 homeless shelter

 no home/homeless

34. What is your yearly personal income?

 Below $15,000

 $15,001 - $25,000

 $25,001 - $50,000

 $50,001 - $75,000

 Over $75,000

Section 4: Your Comments

Thank you for taking the time to complete these questions. Please use the space below for any other comments you would like to make about PAIMI or the P & A program in general.

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-xxxx. Public reporting burden for this collection of information is estimated to average 1.5 hours per respondent per year, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

Appendix D

Form Approved

OMB No.: 0930-XXXX

Expiration Date: MM/DD/YY

See burden statement on last page.

National Evaluation of the

Protection & Advocacy for Individuals with Mental Illness Program (PAIMI)

State Mental Health Program Directors’ Survey

We expect that it will take 20-30 minutes to complete this survey. Surveys will be handled with strict confidentiality and privacy. You will not be identified, nor will individual’s responses be shared with the P&A. Your opinions will be compiled with surveys from other State Mental Health Program Directors to evaluate the functions of the PAIMI program across the country.

INSTRUCTIONS:

This survey is available both on-line and in paper format. Please let us know if you would prefer to utilize a paper survey or have a telephone interview. For most questions, all you need to do is check the box that applies to you. All responses will remain confidential. If you come to a question that you feel uncomfortable answering, skip it.

On line surveys are automatically delivered into a database when you complete and sign off. If you are completing a printed copy of the survey, please return it to us in the enclosed envelope that is ready for mailing. Please complete your survey as soon as possible.

If you would like assistance with completing this survey, or if you need an interpreter, please contact Elizabeth Pell at the Human Services Research Institute. Elizabeth’s phone number is 617-876-0426 x 2307 or email epell@hsri.org. Collect calls will be accepted.

SECTION 1: This section asks questions about the interaction between the State Mental Health Authority and the Protection & Advocacy (P&A) agency’s PAIMI Program.

1. Is the state mental health authority involved in the following P&A/PAIMI activities?

a. PAIMI Advisory Council

 No

 Yes

 Don’t know

b. PAIMI annual planning and priority setting processes

 No

 Yes

 Don’t know

c. Contracting with the P&A to provide particular advocacy services to clients (e.g., rights training to clients in state hospitals)

 No

 Yes

 Don’t know

d. Providing reports of individual deaths of persons in mental health facilities

 No

 Yes

 Don’t know

e. Providing reports of abuse and neglect complaints in mental health facilities

 No

 Yes, aggregated reports

 Yes, individual reports

 Don’t know

f. Please describe ways in which the PAIMI program is currently working with the state mental health authority (SMHA), for example, jointly pursuing legislation, participating on a work group to address abuse, etc.

2. In addition to the public mental health system, with which other State-administered, public programs is the PAIMI actively involved in mental health-related issues?

Populations with Mental Health Needs Served within Other State Programs	PAIMI is not involved	PAIMI is involved with individuals in institutional settings	PAIMI is involved with individuals in community settings	PAIMI is involved in systems level advocacy	Don’t know.
a. social services for adults	 1	 2	 3	 4	 5
b. social services for children	 1	 2	 3	 4	 5
c. criminal justice	 1	 2	 3	 4	 5
d. juvenile justice	 1	 2	 3	 4	 5
e. rehabilitation	 1	 2	 3	 4	 5
f. special education	 1	 2	 3	 4	 5
g. nursing facilities	 1	 2	 3	 4	 5

3. In advocating for changes in mental health statutes over the past two years, the directions of the P&A and the State Mental Health Authority have been aligned

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No changes in legislation initiated in the past two years)

 Don’t know

4. In advocating for the state mental health appropriations over the past two years, the directions of the P&A and the State Mental Health Authority have been aligned

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (P&A does not advocate for the mental health appropriation.)

 Don’t know

5. When the P&A has advocated for changes in mental health regulations or policies over the past two years, how often has the State Mental Health Authority been receptive to proposed changes?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No changes in regulation initiated in the past two years)

 Don’t know

6. When the State Mental Health Authority has set up advisory committees on changes to the service system in the past two years, how frequently has the P&A been invited to participate?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No advisory committees have been set up in the past two years)

 Don’t know

7. In general, what are the strengths of the PAIMI program?

8. In general, what are the weaknesses of the PAIMI program?

9. What has the PAIMI Program in your state accomplished in the past 2 years?

SECTION II: This section asks questions regarding the nature of individual P&A/PAIMI services.

10. Are clients encouraged to connect with consumer-run programs or supports?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

11. Is the PAIMI effective in making themselves known to individuals with mental illness in your state/territory?

 Always effective

 Mostly effective

 Mixed

 Mostly ineffective

 Always ineffective

12. How effective is the PAIMI program in assuring the protection of individuals from abuse, neglect, and other rights violations in your state/territory?

 Always effective

 Mostly effective

 Mixed

 Mostly ineffective

 Always ineffective

SECTION III: This section asks questions regarding P&A systems level activities.

13. Are consumers of mental health services involved in the annual planning process of the P&A for its PAIMI program?

a. The P&A holds open forums to which consumers are invited	 Yes	 No	 Don’t know
b. The P&A makes a substantial effort (beyond holding open forums) to conduct outreach to consumers	 Yes	 No	 Don’t know
c. Individual consumers participate	 Yes	 No	 Don’t know
d. Consumer organizations participate	 Yes	 No	 Don’t know
e. Consumers are able to participate through other means--please describe:

14. Are family members of persons receiving mental health services involved in the annual planning process of the P&A for its PAIMI program?

a. The P&A holds open forums to which family members are invited	 Yes	 No	 Don’t know
b. The P&A makes a substantial effort (beyond holding open forums) to conduct outreach to family members	 Yes	 No	 Don’t know
c. Individual family members participate	 Yes	 No	 Don’t know
d. Family organizations participate	 Yes	 No	 Don’t know
e. Family members are able to participate through other means--please describe:

15. In your state/territory, how actively do each of the following organizations work with the PAIMI program? For each, please indicate whether the organization:

1= Does not exist in the state/territory

2= Exists but does work with the PAIMI program

3= Interacts occasionally with the PAIMI program, but does not work actively with it (two dimensions?)

4= Works actively with the PAIMI program

5= Participates by membership in PAIMI Advisory Council meetings or the P&A

Governing Board meetings (could do this and another option above?)

Other State Mental Health Advocacy Organizations	Does not exist in the state/territory	Exists but does work with the PAIMI program	Interacts occasionally with the PAIMI program, but does not work actively with them	Works actively with the PAIMI program	Participates by membership in PAIMI Advisory Council meetings or the P&A Governing Board meetings
a. State Mental Health Planning Council	 1	 2	 3	 4	 5
b. State Chapter of National Alliance for the Mentally Ill	 1	 2	 3	 4	 5
c. State Chapter of the National Mental Health Association	 1	 2	 3	 4	 5
d. State Chapter of the Federation of Families for Children	 1	 2	 3	 4	 5
e. State Chapter of the International Association of Psychosocial Rehabilitation Services	 1	 2	 3	 4	 5
f. Statewide provider/trade organization/s	 1	 2	 3	 4	 5
g. Statewide consumer organization/s	 1	 2	 3	 4	 5
h. State legal services corporation	 1	 2	 3	 4	 5
i. Other statewide advocacy organization (Please specify:_____________________)	 1	 2	 3	 4	 5
j. Other statewide advocacy organization (Please specify:_____________________)	 1	 2	 3	 4	 5

16. When mental health advocacy organizations have formed coalitions in your state to promote system changes in the past two years, how frequently has the P&A program participated in these efforts?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 No coalitions have been formed in the past two years

 Don’t know

When—during the past two years---the P&A has dealt with systemic problems of client abuse, neglect or other rights violation that could lead to litigation, what have been the paths to resolution of the problem? Check all that apply.

The problem has been dealt with through informal negotiations between the P&A and the state mental health authority
The problem has been dealt with through formal negotiations and a settlement agreement short of litigation
The problem has been dealt with through filing of a lawsuit followed by a negotiated but formal resolution (i.e., a settlement agreement or consent order)
The problem has been dealt with through filing of a lawsuit followed by a court proceedings that have (or appear likely to) end with a judicial ruling
The P&A has not dealt with systemic problems of client abuse or neglect in the public mental health system over the past two years

18. Does the PAIMI program have any pending litigation against the State? If so please describe ________________________________________________

_____________________________________________________________

19. How much contact does the State Mental Health Program director or other senior management staff typically have with the P&A Executive Director or key PAIMI staff?

Meet with or talk to once per week
Meet with or talk to 2-3 times per month
Meet with or talk to about once per month
Meet with or talk to at least four times per year
Meet with or talk to once or twice per year
Never or almost never

20. Generally, would you characterize the nature of that contact as…?

 Always adversarial

 Mostly adversarial

 Mixed

 Mostly cooperative

 Always cooperative

SECTION IV: This section asks questions about your personal characteristics. This information also will be kept strictly confidential.

21 What is your position in the State Mental Health Authority?

 State Mental Health Program Director/Commissioner

 Other (Please describe): ___________________________

22 How informed are you about the mental health-related work of the P&A?

 Very informed

 Somewhat informed

 Slightly informed

 Not at all informed

23. Are you labeled with a disability?

 No

 Yes

IF YES, please check all that apply. This question is VOLUNTARY to answer.

 Mental illness

 Autism

 Cerebral Palsy

 Brain injury

 Seizure disorder/neurological problem

 Chemical dependency

 Vision or hearing impairments

 Physical disability

 Communication disorder

 Alzheimer’s disease

 Down Syndrome

 Other disabilities not listed

Section VI: Your Comments

Thank you for taking the time to complete these questions. Please use the space below for any other comments you would like to make about PAIMI or the P & A program in general.

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-xxxx. Public reporting burden for this collection of information is estimated to average .5 hours per respondent per year, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

Appendix E

Form Approved

OMB No.: 0930-XXXX

Expiration Date: MM/DD/YY

See burden statement on last page.

National Evaluation of the

Protection & Advocacy for Individuals with Mental Illness Program (PAIMI)

State Mental Health Advocacy Organization Directors’ Survey

Thank you for helping with the National Evaluation of the PAIMI Program by completing this survey. The PAIMI program is supported through a grant from the Substance Abuse and Mental Health Services Administration’s Center for Mental Health Services (CMHS) to the Protection and Advocacy (P&A) organization. This survey is being conducted by the Human Services Research Institute (HSRI) through a contract with CMHS for the first National Evaluation of the PAIMI Program. Your opinions will be compiled with surveys from other directors of State Mental Health Advocacy Organizations (e.g., NAMI, the Federation of Families for Children’s Mental Health) to evaluate the functions of the PAIMI program across the country.

We expect that it will take 20 to 30 minutes to complete this survey. Surveys will be handled with strict confidentiality and privacy. You will not be identified, nor will individual’s responses be shared with the P&A.

INSTRUCTIONS:

	This survey is available both on-line and in paper format. Please let us know if you would prefer to utilize a paper survey or have a telephone interview. For most questions, all you need to do is check the box that applies to you. All responses will remain confidential. If you come to a question that you feel uncomfortable answering, skip it.
	On line surveys are automatically delivered into a database when you complete and sign off. If you are completing a printed copy of the survey, please return it to us in the enclosed envelope that is ready for mailing. Please complete your survey as soon as possible.
	If you would like assistance with completing this survey, or if you need an interpreter, please contact Elizabeth Pell at Human Services Research Institute. Elizabeth’s phone number is 617-876-0426 x 2307 or email epell@hsri.org. Collect calls will be accepted.

SECTION 1: This section asks questions about the interaction between your organization and the Protection & Advocacy (P&A) agency’s PAIMI Program .

1. What is your understanding of the role of the state PAIMI Program?

2. Is your organization involved in the following P&A/PAIMI activities?

a. P&A Governing Board

 No

 Yes

 Don’t know

b. PAIMI Advisory Council

 No

 Yes

 Don’t know

c. PAIMI annual planning and priority setting processes

 No

 Yes

 Don’t know

d. Reviewing PAIMI annual program performance report (PPR)

 No

 Yes

 Don’t know

e. Contracting with the P&A to provide particular advocacy services to clients

 No

 Yes

 Don’t know

f. Please describe ways that the P&A is currently collaborating with your organization and other advocacy groups.

3. In your state/territory, which of the following populations in the public mental health system does the PAIMI program serve? Please rate each population.

1= PAIMI Program does not target this group

2= PAIMI Program provides minimum services to this group

3= PAIMI Program provides moderate amount of services to this group

4= PAIMI Program focuses much of its services on this group

5= Don’t know

Population	PAIMI Program does not target this group	PAIMI Program provides minimum services to this group	PAIMI Program provides moderate amount of services to this group	PAIMI Program focuses much of its services on this group	Don’t know.
a. children/adolescents with serious emotional disturbances living in the community	 1	 2	 3	 4	 5
b. children/adolescents with serious emotional disturbances in mental institutions	 1	 2	 3	 4	 5
c. adults with serious mental illness living in the community	 1	 2	 3	 4	 5
d. adults with serious mental illness living in institutions	 1	 2	 3	 4	 5
e. older adults with serious mental illness living in the community	 1	 2	 3	 4	 5
f. older adults with serious mental illness living in institutions	 1	 2	 3	 4	 5

4. In addition to the public mental health system, with which other State-administered, public programs is the PAIMI actively involved in mental health-related issues? Check all that apply.

1=PAIMI is not involved

2=PAIMI is involved with individuals in institutional settings

3=PAIMI is involved with individuals in community settings

4=PAIMI is involved in systems level advocacy

5=Don’t know

Populations with Mental Health Needs Served within Other State Programs	PAIMI is not involved	PAIMI is involved with individuals in institutional settings	PAIMI is involved with individuals in community settings	PAIMI is involved in systems level advocacy	Don’t know.
a. social services for adults	 1	 2	 3	 4	 5
b. social services for children	 1	 2	 3	 4	 5
c. criminal justice	 1	 2	 3	 4	 5
d. juvenile justice	 1	 2	 3	 4	 5
e. rehabilitation	 1	 2	 3	 4	 5
f. special education	 1	 2	 3	 4	 5
g. nursing facilities	 1	 2	 3	 4	 5

5. In advocating for changes in mental health law and policy over the past two years, the directions of the P&A and your organization have been aligned…..

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No changes initiated in the past two years.)

 Don’t know

6. When your organization advances a new initiative to promote a change in the service system, how frequently has the P&A been invited to participate?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Not applicable (No new initiatives advanced in the past two years)

 Don’t know

7. To the best of your knowledge, what are the responsibilities of the PAIMI program?

8. Is the level of resources (funding and staff) available to the PAIMI program appropriate to fulfilling its responsibilities?

 Too few resources

 About the right level of resources

 Too many resources

 Don’t know

9. In general, what are the strengths of the PAIMI program?

10. In general, what are the weaknesses of the PAIMI program?

11. What has the PAIMI Program in your state accomplished in the past 2 years?

SECTION II: This section asks questions regarding the nature of individual P&A services.

12. Are clients encouraged to connect with consumer-run programs or supports?

 Always or most of the time

 Often

 Sometimes

 Rarely or never

 Don’t know

13. Is the PAIMI effective in making themselves known to individuals with mental illness in your state/territory?

 Always effective

 Mostly effective

 Mixed

 Mostly ineffective

 Always ineffective

14. In general, how effective is the PAIMI program in assuring the protection of individuals from abuse, neglect, and other rights violations in your state/territory?

 Always effective

 Mostly effective

 Mixed

 Mostly ineffective

 Always ineffective

15. In what areas is the PAIMI Program strongest in addressing individual level problems? Please check all that apply.

 lack of access to quality treatment

 involuntary admission to a facility

 involuntary treatment

 abuse or neglect in an institution or residential facility

 abuse or neglect in a community setting

 civil rights violation

 housing

 employment

 securing public benefits

 other (please describe): ______________________________

16. In what areas is the PAIMI Program weakest in addressing individual level problems? Please check all that apply.

 lack of access to quality treatment

 involuntary admission to a facility

 involuntary treatment

 abuse or neglect in an institution or residential facility

 abuse or neglect in a community setting

 civil rights violation

 housing

 employment

 securing public benefits

 other (please describe): ______________________________

SECTION III: This section asks questions regarding P&A systems level activities.

17. To what extent are consumers of mental health services involved in the annual planning process of the P&A for its PAIMI program?

a. The P&A holds open forums to which consumers are invited	 Yes	 No	 Don’t know
b. The P&A makes a substantial effort (beyond holding open forums) to conduct outreach to consumers	 Yes	 No	 Don’t know
c. Consumers participate	 Yes	 No	 Don’t know
d. Consumer organizations participate	 Yes	 No	 Don’t know
e. Consumers are able to participate through other means--please describe:

18. To what extent are family members of persons receiving mental health services involved in the annual planning process of the P&A for its PAIMI program?

a. The P&A holds open forums to which family members are invited	 Yes	 No	 Don’t know
b. The P&A makes a substantial effort (beyond holding open forums) to conduct outreach to family members	 Yes	 No	 Don’t know
c. Family members participate	 Yes	 No	 Don’t know
d. Family organizations participate	 Yes	 No	 Don’t know
e. Family members are able to participate through other means--please describe:

19. With regard to providing training for clients of mental health programs to inform them about protection of their rights, the PAIMI Program provides….?

 Too much training for clients

 About the right level of training for clients

 Too little training for clients

 Don’t know

20. With regard to providing training for staff of mental health programs to inform them about issues related to the protection of clients’ rights, the PAIMI Program provides…….?

 Too much training for staff

 About the right level of training for staff

 Too little training for staff

 Don’t know

21. With regard to providing public education about clients’ rights and the problems of abuse, neglect, and other rights violations, the PAIMI Program provides…….?

 Too much public education

 About the right level of public education

 Too little public education

 Don’t know

22. In your state/territory, how actively do the following organizations work with the PAIMI program? For each, please indicate whether the organization:

1= Does not exist in the state/territory

2= Exists but does not interact with the PAIMI program

3= Interacts with the PAIMI program, but does not actively work with it

4= Works actively with the PAIMI program

Other State Mental Health Advocacy Organizations	Does not exist in the state/territory	Exists but does not interact with the PAIMI program	Interacts with the PAIMI program, but does not actively work with it	Works actively with the PAIMI program
a. State Mental Health Planning Council	 1	 2	 3	 4
b. State Chapter of National Alliance for the Mentally Ill	 1	 2	 3	 4
c. State Chapter of the National Mental Health Association	 1	 2	 3	 4
d. State Chapter of the Federation of Families for Children	 1	 2	 3	 4
e. State Chapter of the International Association of Psychosocial Rehabilitation Services	 1	 2	 3	 4
f. Statewide provider/trade organization/s	 1	 2	 3	 4
g. Statewide consumer organization/s	 1	 2	 3	 4
h. State legal services corporation	 1	 2	 3	 4
i. Other statewide advocacy organization (Please specify:_____________________)	 1	 2	 3	 4
j. Other statewide advocacy organization (Please specify:_____________________)	 1	 2	 3	 4

23. When in the past two years mental health advocacy organizations have formed coalitions in your state to promote system changes, how frequently does the P&A program provide leadership in these efforts?

Always or most of the time
Often
Sometimes
Rarely or never
No coalitions among advocacy organizations have been set up in the past two years
Don’t know

24. When the PAIMI program identifies priorities for systems level change, how frequently are the problems they choose to address those that are most compelling?

Always or most of the time
Often
Sometimes
Rarely or never
The PAIMI program does not undertake efforts focused on systems change
Don’t know

25. When—during the past 2 years---the P&A has dealt with systemic problems of client abuse, neglect or other rights violation that could lead to litigation, what have been the paths to resolution of the problem? Check all that apply.

The problem has been dealt with through informal negotiations between the P&A and the state mental health authority that led to change in policy or practice
Publishing public reports authority that led to change in policy or practice
The problem has been dealt with through filing of a lawsuit followed by a negotiated but formal resolution (i.e., a settlement agreement or consent order)
The problem has been dealt with through filing of a lawsuit followed by court proceedings that have (or appear likely to) end with a judicial ruling
Regulatory change
Legislative change
The P&A has not dealt with systemic problems of client abuse or neglect in the public mental health system over the past two years

26. In what areas is the PAIMI Program strongest in identifying and addressing mental health system level problems? Please check all that apply.

 lack of access to quality treatment

 involuntary admission to a facility

 involuntary treatment

 abuse or neglect in an institution or residential facility

 abuse or neglect in a community setting

 discrimination

 housing

 employment

 getting Social Security or other benefits

 other (please describe): ______________________________

27. In what areas is the PAIMI Program weakest in identifying and addressing mental health system level problems? Please check all that apply.

 lack of access to quality treatment

 involuntary admission to a facility

 involuntary treatment

 abuse or neglect in an institution or residential facility

 abuse or neglect in a community setting

 discrimination

 housing

 employment

 getting Social Security or other benefits

 other (please describe): ______________________________

28. All P&A programs operate in political environments. In the past 2 years has the P&A experienced political pressure that has interfered with pursuing its mental health system advocacy efforts?

 Always or most of the time

 Often

 Sometimes

 Rarely

 Never

 Don’t know

SECTION IV: This section asks questions about your personal characteristics. This information also will be kept strictly confidential.

29. What is your position in this organization?

 Director

 Other (Please describe): ___________________________

30. For how many years has the director of this organization been involved in the public mental health system? _____years

31. How much contact does the director of this organization or other senior management staff typically have with the P&A Executive Director or key PAIMI staff?

Meet with or talk to once per week
Meet with or talk to 2-3 times per month
Meet with or talk to once per month
Meet with or talk to at least four times per year
Meet with or talk to once or twice per year
Never or almost never

32. How informed are you about the mental health-related work of the P&A?

 Very informed

 Somewhat informed

 Slightly informed

 Not at all informed

33. Answering the following question is voluntary. Are you labeled with a disability?

 No

 Yes, If yes, please check all that apply. Again this is a voluntary response.

 Mental illness

 Autism

 Cerebral Palsy

 Brain injury

 Seizure disorder/neurological problem

 Chemical dependency

 Vision or hearing impairments

 Physical disability

 Communication disorder

 Alzheimer’s disease

 Down Syndrome

 Other disabilities not listed

Section VI: Your Comments Thank you for taking the time to complete these questions. Please use the space below for any other comments you would like to make about PAIMI or the P & A program in general.

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-xxxx. Public reporting burden for this collection of information is estimated to average .5 hours per respondent per year, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 1 Choke Cherry Road, Room 7-1044, Rockville, Maryland, 20857.

ATTACHMENT F

KEY INFORMANT INTERVIEWS: CMHS PAIMI Project Officers

Which PAIMI states or territories are you responsible for?
When did you become the PAIMI Project Officer for these grantees?
What background did you have related to PAIMI before assuming these responsibilities?
What orientation did you receive for these new responsibilities?
How would you describe your responsibilities as the Project Officer for the PAIMI grantees?
When do you have contact with the PAIMI programs in the course of a year?

Before or after the annual application?
Before or after the submission of the PPR?
When they need technical assistance?
Other times?

What is the nature of this contact?

Application Reviews

What is your role in the review of annual applications?
Have any of your grantees experienced application difficulties within the past two years?
What was the nature of these problems?
What was your role in resolving them?
How were they resolved?
In general, do you think that the annual application process is appropriate for the PAIMI program? Are there changes that you would recommend?

Site Monitoring Visits

Have any of your grantees had a site monitoring visit in the last two years?

[Review each grantee visit.]

To what extent were you able to participate in that visit(s)? Did you consider yourself a member of the site visit team?
What were the strengths of the team that was responsible for the visit?
What were the problems associated with this team?
Did the visit run smoothly? What problems were encountered during the visit?
Did the site visit team identify significant strengths and problems of the PAIMI grantee’s program? To the best of your recollection, what were they?
What formal recommendations did the team make following the site visit?
What were the results of these recommendations?
Did the team offer informal comments and suggestions to the grantees during the visit or at the exit interview that were of value to the program? [Why were these excluded from the final report?]
What are the strengths of the design of these visits?
What problems are these in the design of these visits?
What changes would you recommend for future visits?

Annual PAIMI Program Performance Reports

What is your role in the review of annual PPRs?
Have any of your grantees experienced PPR difficulties within the past two years?
What was the nature of these problems?
What was your role in resolving them?
Are you aware of specific problems in the quality of information reported in the PPR?
Does the PPR adequately represent the activities and accomplishments of grantees?
How does CMHS make use of the information reported in the PPR?
Do you know if this information is useful to the grantees themselves?

PPR Peer Reviews

Were you able to sit in on the peer reviews for your grantees in 2004? 2003?
Did you review the notes from the reviewers and the draft review reports prior to their transmission to the grantees?
What significant strengths were identified through this review process?
What significant problems were identified through this review process?
Do you think that the reviewers missed either significant strengths or problems for your grantees?
Do you think that the grantees received information and recommendations that will assist them in improving their programs? Are these particular examples that are salient?
Were there specific recommendations that grantees were expected to respond to?
What were the results of these recommendations?
What are the strengths of the design of the peer review process?
What problems are these in the design of the peer review process?
What changes would you recommend for future peer reviews?

Fiscal Reviews

Do you have a role in the review of the grantee budget?
Do you have a role in the review of the grantee expenditure reports?
Have any of your grantees experience fiscal difficulties within the past two years?
What was the nature of these problems?
What was your role in resolving them?
How were they resolved?
In general, do you think that there are adequate fiscal oversight mechanisms for the PAIMI program? Are there changes that you would recommend?

Technical Assistance

Do you work with grantees when they require technical assistance?
How frequently do grantees take advantage of the technical assistance offered through the contract with NDRN?
What are the strengths of the technical assistance program?
Are you aware of difficulties that grantees have experience in obtaining necessary technical assistance? If yes, please explain.
Are there changes that you would recommend in the technical assistance program?

PAIMI and the Block Grant

Are these the same grantees for whom you also serve as the Mental Health Block Grant Project Officer?
Is it helpful to serve as a project officer for both types of grants for the same states?
Have there been conflicts between the state mental health authorities and the PAIMI programs that you supervise? Do you have a role to play in working out these conflicts?
Do project officers have sufficient time and other resources to meet their Block Grant, PAIMI, and other responsibilities?

Working with other P&A Programs

Do you have contact with the other Federal agencies that are administering the other P&A program (e.g., ADD)? What is the nature of that contact?
Are you aware of any circumstances in which different policies or practices of these agencies interfere with the pursuit of PAIMI goals and objectives? [If yes,please describe.]

PAIMI Program Overall

What are the strengths of this program?
Are there achievements by your grantees that you would point to as particular examples of the value of PAIMI?
What are the difficulties grantees experience in achieving program goals?
What changes would you recommend in the program overall?

ATTACHMENT G

KEY INFORMANT INTERVIEWS: NDRN Technical Assistance Staff

How does NDRN allocate funds for PAIMI training and technical assistance (T/TA)?
How is the contract managed with CMHS and its partner federal agencies for PAIMI T/TA? Are there management objectives for administration of the PAIMI T/TA funds?
What types of Technical Assistance (TA) are provided and in what amount?
How many FTE staff at NDRN are allocated to PAIMI T/TA? At other T/TA sites?
How are TA needs identified? Is the process for deciding how to allocate TA resources efficient and equitable?
How Is the TA that is available commensurate with need (i.e. does CMHS’ PAIMI program have adequate resources to provide TA?)
Is there an alignment between PAIMI program deficiencies and the PAIMI availing itself of TA?
Does NDRN have access to information regarding PAIMI grantee needs for T/TA identified in site monitoring visits? PPR Peer Reviews?
How does NDRN track the 2% allocation derived from grantee awards for delivering T/TA to the PAIMI Program?
How are decisions made to contract with back up centers for PAIMI T/TA? How is the quality and quantity of services provided through back up centers to PAIMIs monitored?
Given that much of the T/TA delivered through NDRN is devoted to expertise that crosses populations, how does NDRN demonstrate commitment to identifying and addressing issues for T/TA that are targeted to people with serious mental illness?
What is NDRN’ plan for conducting regular and on-going evaluation of T/TA activities rendered to the PAIMI Program?
Does the program use techniques to extend the reach of available TA resources (e.g. identifying clusters of states/territories with similar needs, producing materials for wide distribution, etc.)?
Does TA conform to principles of adult learning, communicating in multiple formats and in a variety of ways?
Do grantees find the TA for PAIMI Program useful?
- Conferences and Meetings: fiscal, skills building, annual, Executive Directors
- List Serves
- Legal Back Up Centers
- Publications
- Webcasts
- On-site NDRN consultation
How does NDRN evaluate the effectiveness of T/TA?
How does NDRN make changes to improve the T/TA to PAIMIs?
What do you consider most successful about NDRN’ deliver of T/TA to PAIMI Programs?
What are the greatest challenges for NDRN in administering T/TA to PAIMI Programs?
What changes are necessary to improve delivery of T/TA to PAIMI Programs?

ATTACHMENT H

KEY INFORMANT INTERVIEWS: PAIMI Site Monitors

What is your role in public mental health?
What reviews have you participated in during the past three years?
Which states or grantees were you responsible for?

Background and orientation

What background did you have related to PAIMI?
What orientation to the site visits did you receive?
What were the strengths of the orientation?
What else might have been helpful to include in the orientation?
What changes would you recommend for orienting new reviewers?

Review Process

How would you describe your responsibilities as a site visitor?
Did you have all the materials that you needed to conduct the visit?
Were your other team members well-prepared to undertake the visit?
Did the process run smoothly?
What problems were encountered in conducting the site visit?
As a participant in this process, did you learn things that are useful to you in your role outside the site visit process?

Representation of Team Comments in Notes and Reports

Did you review the entire set of notes from your site monitoring team? If yes, did they adequately reflect the consensus of your site monitoring team?
Did you have an opportunity to review and comment upon the draft monitoring reports prior to their transmission to the grantees?
Did you see the final review that was transmitted? Do you think that it adequately reflected the most important comments of your site monitoring team?
Do you think that the grantees received information and recommendations that will assist them in improving their programs? Are there particular examples that are salient?
Did the team offer informal comments and suggestions to the grantees during the visit or at the exit interview that were of value to the program? [Why were these excluded from the final report?]

Strengths and difficulties of the peer review process, the PPR, and the PAIMI program

What are the strengths of the design of the site monitoring process?
What problems are these in the design of the site monitoring process?
Are the resources available to conduct this process consistent with what is needed?
What changes would you recommend for future site monitoring visits?
Judging from the grantees that you have visited, what do you think are the major strengths of the PAIMI program overall in reaching its goals?
What do you think are the major difficulties of the PAIMI program overall in reaching its goals?
What changes would you recommend to the PAIMI program overall?

ATTACHMENT I

KEY INFORMANT INTERVIEWS: PAIMI PPR Peer Reviewers

What is your role in the PAIMI in your state? (If none, what role do you have in the public mental health system?)
Did you participate in reviews in 2004? In 2005?
Which states or grantees were you responsible for?

Background and orientation

What background did you have related to PAIMI?
What orientation to the peer review process did you receive?
What were the strengths of the orientation?
What else might have been helpful to include in the orientation?
What changes would you recommend for orienting new reviewers?

Review Process

How would you describe your responsibilities as a peer reviewer?
Did you have all the materials that you needed to conduct the review?
Were your other team members well-prepared to undertake the review?
Did the process run smoothly?
What problems were encountered in conducting the review?
As a participant in this process, did you learn things that are useful to you in your role?

Representation of Team Comments in Notes and Reports

Did you review the entire set of notes from your review team? If yes, did they adequately reflect the consensus of your review team?
Did you have an opportunity to review and comment upon the draft review reports prior to their transmission to the grantees?
Did you see the final review that was transmitted? Do you think that it adequately reflected the most important comments of your review team?
Do you think that the grantees received information and recommendations that will assist them in improving their programs? Are there particular examples that are salient?
In reviewing the same grantees a second year, did you see evidence of responsiveness to the recommendations from the prior year? [Specific examples?]
Did you also see areas in which you felt that the team’s recommendations were ignored? [Specific examples?]

Strengths and difficulties of the peer review process, the PPR, and the PAIMI program

What are the strengths of the design of the peer review process?
What problems are these in the design of the peer review process?
Are the resources available to conduct this process consistent with what is needed?
What changes would you recommend for future peer reviews?
What are the strengths of the design of the annual Program Performance Review document for purposes of program monitoring?
What are the problems in the design of the annual Program Performance Review document for purposes of program monitoring?
What changes would you recommend for the PPR?
Judging from the grantees that you have reviewed, what do you think are the major strengths of the PAIMI program overall in reaching its goals?
What do you think are the major difficulties of the PAIMI program overall in reaching its goals?
What changes would you recommend to the PAIMI program overall?

1 Meaningful evaluations meet three conditions (Strosberg and Wholey, 1983): program objectives are well defined; program objectives are plausible, and the intended use of the evaluation is well defined.

File Type	application/msword
File Title	Draft Outline: PAIMI Evaluation Plan Report
Author	jhornik
Last Modified By	jmorrow1
File Modified	2007-07-11
File Created	2006-04-20