HKNC OMB Supporting Statement Part B 07-07-10

U.S. Department of Education

Rehabilitation Services Administration (RSA)

EVALUATION OF THE HELEN KELLER NATIONAL CENTER FOR DEAF-BLIND YOUTHS AND ADULTS

Office of Management and Budget

Clearance Package Supporting Statement

And Data Collection Instruments

Part B

Collections of Information Using Statistical Methods

REVISED

June 18, 2010

Table of Contents

Page

B. Collections of Information Using Statistical Methods 28

B1. Respondent Universe and Sampling Methods 28

B2. Procedures for the Collection of Information 32

B3. Methods to Maximize Response Rates 34

B4. Tests of Procedures or Methods 35

B5. Consultations on Statistical Aspects of the Design 36

List of Tables

Table B-1 Data collection details 32

SUPPORTING STATEMENT FOR
PAPERWORK REDUCTION ACT SUBMISSION
INFORMATION COLLECTION PLAN FOR THE
EVALUATION OF THE HELEN KELLER NATIONAL CENTER

B. COLLECTIONS OF INFORMATION USING STATISTICAL METHODS

B1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, state and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The Helen Keller National Center for Deaf-Blind Youth and Adults (HKNC) Evaluation Study plan includes conducting:

• In–person interviews with deaf-blind former HKNC consumers (n=36deaf-blind individuals who received services at HKNC headquarters in New York );

• Email census survey of vocational rehabilitation (VR) agencies (N=80);

• Telephone interviews with state VR agency staff (n=20);

• Telephone interviews with service providers (n=50);

• Telephone interviews with national stakeholder group representatives (n=9);

• Telephone interviews with family members of individuals who received services from HKNC New York (n=9).

• In-person interviews with HKNC department heads and key staff (n-9);

• In-person group interview with HKNC regional representatives (n=9);

• Site visits with 5 HKNC regional representatives (n=5); and

• In-person interviews with OMB and Department of Education staff (n=9 or fewer).

Potential deaf-blind former HKNC consumers will be drawn from HKNC lists of consumers who received services between 2005 and 2008 from HKNC Headquarters in New York. HKNC has reported that 268 individuals received services from HKNC New York. Since interviews will be in-person, respondents are expected to come to the HKNC headquarters in New York or to a location near one of two other regional centers to participate in the interview (Los Angeles and Boston). Interviews will be conducted concurrently with contractor site visits to these locations. So as to reduce the burden for respondents, only individuals who live within 45 miles of HKNC Headquarters, Boston, and Los Angeles will be invited to participate in the survey. A total of 73 deaf-blind former consumers meet this criterion. HKNC will be asked to eliminate from consideration any former consumers whom they believe cannot communicate at least at a third- or fourth-grade level using certified interpreters, if needed. Since HKNC regional representatives actively maintain contacts with many of the former consumers in their areas, they have been asked by HKNC to review address information for former deaf-blind consumers to ensure that they still meet the distance criteria (i.e., that they have not moved out of the area). A random sample of 52 individuals will be drawn from the sample frame. These eligible former HKNC consumers will receive an advance email advising them that they may be asked to be interviewed. Individuals will be provided with an opportunity to withdraw from the study, although ED expects that at least 40 of the 52 individuals will agree to participate. HKNC will be asked to keep track of reasons given to ensure that self-selection bias is avoided to the extent possible. HKNC will schedule interviews, and ED anticipates that the 36 of the 40 potential interviewees will be able to keep scheduled appointments and complete in-person interviews (some interviewees may not be able to participate at the last minute due to illness or scheduling conflicts). Proxies will not be able to respond on behalf of the survey participants, since the study intends to examine the first-hand experience of deaf-blind consumers.

The contractor anticipates that 70 percent of eligible deaf-blind former consumers contacted will be willing and able to participate during the time frame for the site visits that will be concurrent with the interviews. Because random selection techniques will be used to obtain the sample, RSA and our contractor believe that 36 completed interviews will be sufficient to gather in-depth information about the wide variety of experiences of former deaf-blind consumers.

Westat will conduct a census email survey of all 80 state VR agencies. It is anticipated that at least 90 percent of the state VR agencies will respond, consistent with other surveys of state VR agencies. A census survey of all state VR agencies is necessary given the anticipated variety of models that state VR agencies may use to address the independent living and vocational needs of deaf-blind individuals. Response rates will be achieved through contractor follow-up activities including email contacts and phone calls to the VR staff completing the survey, or the VR agency directors, if necessary. If needed, the contractor will consult with RSA if a VR agency is nonresponsive. RSA is highly confident of a minimum 90 percent response rate because of the endorsement of the study by the Council of State Administrators of Vocational Rehabilitation and the National Council of State Agencies for the Blind.

A sample of 25 percent of the population of 80 state VR agencies (20 state VR agency personnel who were asked to complete the email survey will be selected for an in-depth telephone interview to review survey results. ED believes that the use of sampling procedures will minimize the burden on state VR agencies. The sample size selected is not based on a statistical analysis of key estimates, but rather a sample of state VR agencies large enough to allow a wide variety of deaf-blind individual service models to be included. The contractor will develop criteria for selection following analysis of the email survey results and submit the selection to RSA for approval. The contractor expects a high degree of cooperation for this portion of the survey. The contractor will identify additional agencies meeting the criteria to ensure that there are 20 completed interviews.

HKNC reports that an average of 1,025 professionals in state VR agencies and other organizations are served by HKNC annually. Of these, approximately 100 individuals per year receive formalized training at HKNC New York or elsewhere nationally, while other contacts include HKNC participation in conferences and meetings. There were a total of 404 training participants from 2005-2008 in approximately six service provider training classes. The names of the individuals who participated in these trainings were either maintained by HKNC, or will be obtained from individuals who coordinated trainings conducted for groups of individuals from specific organizations. A preliminary sample of 72 individuals, or 18 percent of the population, will be randomly selected and contacted using an advance letter, in anticipation of a 70 percent response rate that will result in 50 completed telephone interviews using a semi-structured interview instrument. The sample size and number of completed interviews were not developed using key estimates about the population data, since detailed population level data was not available. The six training classes will be used to stratify the sample. ED believes that the number of individuals in the sample will provide a diverse cross-section of service providers who received training from HKNC. As with deaf-blind former consumers, RSA anticipates a relatively strong response rate because of the nature of the training provided and because of the follow-up and other coordinating activities undertaken by regional representatives with the population of service providers in their regions.

RSA has identified nine national stakeholder groups that should have detailed knowledge of HKNC. A representative from each of these groups will be contacted to participate in interviews. RSA expects that all nine organizations will participate in interviews for a 100 percent response rate.

Family members of individuals who received training or services from HKNC New York and who are active in regional or local family groups will be identified through internet searches, contacting members of the National Family Association of Deaf-Blind, and by HKNC. Up to nine individuals will be contacted who meet the criteria. Population data are not available for this group because it is not known how many deaf-blind individuals had family members active in their HKNC training and who were active with groups of other family members.

In-person interviews with HKNC key staff and department heads are critical to obtaining a better understanding of HKNC operations. Nine staff were identified for in-person interviews. A group interview with nine HKNC regional representatives was conducted to get a better understanding of services and activities offered by these individuals. The population of 11 regional representatives was considered for the group interview. Two individuals were asked to not participate. One was a regional representative who was recently appointed, while the second actively participated in pilot testing of HKNC evaluation instruments, and was therefore exempted from participation in the group interview.

Site visits will be conducted during the same time period when interviews with deaf-blind individuals are being conducted in Boston, Massachusetts (HKNC Region 1), Sands Point, New York (HKNC Training Center, Region 2), and Los Angeles, California. Informal unstructured discussions with the HKNC regional representative will occur before and after interview time blocks and in between interviews as time allows. Subjects are expected to include services provided by the regional representatives, their working relationship with state VR agencies, services alternatives available regionally and locally to serve deaf-blind individuals who do not travel to HKNC New York for training, and opportunities and challenges in serving deaf-blind individuals in their regions.

Questionnaires for email surveys and semi-structured interviews are attached.

Table B-1 includes response rates from all data collection efforts included in this OMB clearance.

Table B-1. Data collection details

Who Is Interviewed/Surveyed	Mode of Data Collection	# of Completed Interview/Surveys	Expected Response Rates
Deaf-blind former HKNC consumers who received training and services from HKNC New York	In-person interviews	36	70% of eligible interviewees
VR agency staff	Telephone	20	100% of sample frame
Service providers	Telephone	50	70%
State VR agencies	Email survey	80	90%*
Stakeholder group representatives	Telephone interviews	9	100%
Family member interviews	Telephone interviews	9	100% of eligible individuals identified
HKNC department heads and key staff	In-person interviews	9	100%
HKNC regional representatives	Group in-person interview	9	100% of eligible regional representatives

*A response rate of 90 percent is expected from the state VR agencies because some of the 80 agencies may not respond if they do not serve any deaf-blind individuals. This response rate is consistent with other surveys of state VR agencies.

B2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

This data collection will not be using any statistical methodology for stratification and sample selection because the sampling universe is very small. As noted in B1, the contractor plans to interview 36 deaf-blind former consumers, conduct a census survey of all 80 state VR agencies, and interview 50 service providers. Preliminary analytic results from the email survey of state VR agencies will be used to draw a purposive stratified sample of 20 VR agency staff who will participate in more detailed interviews to examine more deeply HKNC support of state VR agencies and services to deaf-blind individuals. There are no unusual problems requiring specialized sampling procedures.

Questionnaire Design

The contractor developed questionnaires for each category of interviews or survey (see Appendix). Pretests of these instruments have been conducted, resulting in minor wording changes and reordering of interview or survey items. The VR telephone interview protocol is a more open-ended instrument than the other semi-structured instruments. It was pretested as a telephone survey following administration of the email survey. Prompts were pretested only as appropriate in the course of the interview.

Data Collection Methods and Procedures

The data collection procedures will be adapted to the needs of the deaf-blind population to enhance the response rate and address issues of data quality. These procedures must provide the ability to communicate with all members of the study population, and they must be sufficiently flexible to work around issues of both physical and mental fatigue.

The interview with deaf-blind former consumers will have the following attributes to address issues of accessibility and quality of response.

• There will be two interpreters at each interview. A family member or friend may be invited to observe the interview only if one is available and if the deaf-blind interviewee requests that the family or friend be present.

• Alternative phrasing prompts will be used if it becomes clear from the responses that the interviewee does not understand a question. Question wording was simplified, and alternatives and examples included following the pre-test of the instruments, although for the most part the instrument worked effectively.

• Interviewers will follow criteria to determine when an interview should be stopped or time taken between questions if the interviewee tires and to allow a change of interpreters if needed.

Interviewers for all data collection activities will receive training to prepare them for issues that will arise when interviewing deaf-blind individuals and others. In addition to general telephone interviewer training, nonresponse avoidance and conversion training, and training in the specific content of the interview, interviewers will receive sensitivity training. This training will focus on preparing staff for what to expect and techniques to use in different situations. It is anticipated that staff conducting the deaf-blind interviews will have had some prior experience working with deaf-blind individuals. The protocol for the interviews includes brief guidance on appropriate techniques for communicating with deaf-blind individuals who are using interpreters.

There is no use of periodic data collection cycles to reduce burden.

B3. Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Overall response rates were presented above. The following describes procedures to be followed to maximize the number of sample members who complete the survey:

• VR directors will be contacted using the advance notification (included with the electronic files uploaded to EDICS) and asked to select individuals who are qualified to complete the VR agency email survey. An advance email will be sent to identified VR staff to explain the purpose of the study and gain their cooperation in participating. The contractor will follow up with identified staff to clarify questions and ensure timely submission by the established deadline.

• Only those who agree to participate will be invited to come for the interview.

• Consumers will be sent advance notice letters explaining the study.

• HKNC has agreed to assist with identification of their former consumers, arrange interview times, and identify qualified certified interpreters for deaf-blind interviews. These activities are considered part of HKNC’s normal business activities as an educational institution with a network of regional representatives whose job responsibilities include follow up with consumers and coordination of services.

Deaf-blind individuals are being interviewed in an urban area with long commute times and high commuting costs. Deaf-blind individuals are required to travel to central locations to be interviewed because of requirements for interpreters who will assist with communication during interviews. Interviews are similar in scope and approach to cognitive interviews. While the interview instrument has been designed to take all levels of intellectual capacity into account, some aspects of the interview may present an unusually challenging burden for some interviewees. To encourage deaf-blind individual participation in the interviews, particularly given mobility difficulties experienced by this population, the study protocol includes an offer to pay $65 at the conclusion of the interview as an effective means of reducing interview nonparticipation. Inadequate resources will likely result in a lower than desired response rate.

Debrief Interviewers

Senior contractor staff will debrief the data collection staff after the interviews. The purpose will be to identify effective data collection techniques that can be shared among data collection staff, determine where training materials might be improved, and identify problems in the survey instrument. Notes of the debriefings will be reviewed by senior project staff who will decide where adjustments are needed. Telephone interviews will be selectively monitored by senior project staff. In such cases interviewees will be informed about the monitoring, and their consent will be sought. Substantive recommendations will be brought to the attention of the RSA Contracting Officer’s Representative (COR) and OMB for approval before extensive changes to study protocols are made.

B4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The in-person instrument for the deaf-blind former consumers was pretested with two deaf-blind consumers in Maryland using two certified interpreters. The pretest included an assessment of the effectiveness of communication protocols as well as the usefulness of the instrument. Communication protocols using interpreters to establish effective communications with deaf-blind interviewees were acceptable. The interview protocol was modified on the basis of pretest findings. Changes were made to the sequencing of questions, question wording, the way in which question clarifications are provided by interviewers, and inclusion of specific language to provide examples and simplified language if needed. The contractor conducted the interviews, which were observed by RSA staff.

The telephone surveys with service providers were pretested in person with two deaf-blind service providers in Maryland who received HKNC training or services at HKNC New York or from regional representatives. Minor changes were made to the wording and content based on the pre-test. The contractor conducted the interviews, which were observed by RSA staff.

The email survey was pretested twice. One pretest was conducted as an in-person interview with a VR counselor responsible for services for deaf-blind VR consumers in southern Maryland. RSA staff observed this interview. The other pretest was conducted via a telephone interview with the VR staff person in Illinois responsible for coordinating VR services with HKNC for deaf-blind individuals in that state. Minor adjustments in the content and wording of the protocol were made. The VR staff interview was pretested by telephone in a separate telephone call. Probes were adjusted based on responses provided by the interviewee.

B5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Name	Affiliation	Telephone Number	Responsibility
Frank Bennici	Westat	301-738-3608	Consult and analyze data
David Bernstein	Westat	301-738-3520	Consult, collect, and analyze data

APPENDICES

APPENDIX A

draft advance letters containing assurances

APPENDIX B

QUESTIONNAIRES