The applicable population (universe) and sampling method varies for each data collection effort but are the same as those used during the Round I evaluation effort. Further detail for each data collection activity is provided below.
CCOE Center Director and Program Coordinator Survey
The respondent universe includes designated leadership at each CCOE. This leadership consists of a Center Director and Program Coordinator who administer the CCOE program within their organization. These individuals will be responsible for completing the CCOE Center Director and Program Coordinator Surveys. All fourteen CCOEs will be asked to participate because data on each CCOE is integral in conducting a meaningful evaluation, especially because each CCOE deals with a very different patient population. Baseline information will be compared to Round II evaluation data to obtain a comprehensive picture of the CCOEs progress towards goals and growth over time. Sampling is not appropriate and there are too few CCOEs to employ a psychometrically sound sampling strategy. The anticipated response rate is 100% as participation is a requirement for receiving the CCOE designation and associated funding from OWH (please see B.2 for further justification of this expected response rate). This collection has not been conducted previously.
CCOE Community Partner Survey
The respondent universe includes designated points of contacts at all CCOE partner organizations that have been in a partnership with a CCOE for 3 months or more. Due to the wide variety in the number of and type of services offered by partners at each CCOE, the sampling strategy for the partner survey will be developed in detail after a review of Phase I data. The sampling strategy chosen will be based on the CCOE’s current number and mix of partners to ensure a consistent, comprehensive, and accurate picture of partner activities is developed. One potential sampling strategy consists of sampling all partners for each CCOE. This was the methodology that was used during Round I after a review of the volume and service-mix of CCOE partners. If the number or mix of partners are considerably higher than during Round I , the Evaluation Team will consider alternative sampling strategies. For example, the Evaluation Team could sample one partner organization for each service type offered. Alternatively, another potential strategy could entail surveying a random stratified sample of partners that provide services in each one of the CCOE core components in which partners organizations are involved at each CCOE.
The Evaluation Team anticipates sampling no more than 12 to 15 partner organizations for each CCOE based on existing information regarding the numbers and service mix of current CCOE partners. Because the CCOE program is continuously evolving and is continually adding, replacing, or losing partner organizations, the sampling strategy will be determined based on the number of CCOE partners and the service mix offered at the time of the evaluation. This strategy is cost-effective, does not overburden the CCOE network with an undue amount of data collection, and still enables OWH to gather a comprehensive and up-to-date picture of the CCOE program and its services. Overall response rates are estimated at 85-90%.
CCOE Client Survey
The respondent universe is all individuals who are registered clients of the CCOE program who are 18 years of age or older. Every CCOE client will have an equal probability of selection and will not be re-sampled during the six-month data collection effort. Participation is voluntary; any patient may decline to participate without penalization. Respondents will be chosen using a random sampling strategy. Each CCOE will be asked to administer the survey to 400 patients. Therefore, the number of patients sampled weekly at each CCOE will differ based on patient volume. The survey effort will target an 85% response rate and sample the CCOE patients accordingly. This amount depends on the number of patients who decline to participate, these numbers will be monitored throughout the survey period. Results from the survey can be generalized to the CCOE’s patient population because a random sampling scheme will be used.
CCOE Site Visit Protocol
The respondent universe includes CCOE leadership (at each CCOE), such as Center Directors, CCOE staff, and partner organization leadership and staff. All CCOE leadership will participate in the Center Director introductory brief and debriefing meetings. The eleven CCOEs and three Ambassadors for Change (AFC)1 will be asked to participate because additional information from each CCOE is necessary to clarify data gathered during the CCOE Center Director and Program Coordinator Survey and in order to understand some of the unique processes and tools developed at each CCOE. Therefore, sampling is not appropriate and there are too few CCOEs to employ a psychometrically sound sampling strategy.
However, not all CCOE staff will be invited to participate in the focus groups. Focus groups will consist of no more than six to eight staff. Staff will be chosen so as to minimize any disruptions in service at the CCOE. Focus groups will consist of a representative sample of CCOE staff; however, this will be limited to who is working at the time of the site visit and their availability to meet.
In choosing which partner facilities to visit, Booz Allen will first consider the number of community partners each CCOE has (a key factor influencing this decision include the physical distance between facilities). Booz Allen, with feedback from the CCOE, will select partners based on frequency of interaction with the CCOE and volume of CCOE patients. For each partner facility chosen, a senior leader or liaison partner will be asked to participate in an interview. The senior leader or liaison partner will choose employees to participate in one of two focus groups. Similar to the CCOE staff focus group, employees will be chosen so as to minimize any disruptions in service at the partner facility. A representative sample of employees will be chosen to participate.
The procedure for collecting data varies for each data collection instrument, however the collection procedures will be similar or the same as those used for the Round I evaluation. Any differences will incorporate lessons learned during Round I. The procedures to collect data are described below. Each data collection instrument will be administered one time only, thus, data will be collected only once.
CCOE Center Director and Program Coordinator Survey
The Evaluation Team will send each CCOE Center Director and Program Coordinator an email inviting him or her to complete a survey for his or her CCOE. The email will include detailed instructions on how to access the online survey, what data are needed, and the time frame provided for completion. A point of contact on the Evaluation Team will be provided to answer questions regarding the survey or the CCOE evaluation. The CCOEs will have approximately one month to gather requested information, fill out and submit the completed survey. Based on pilot testing, estimated time for completing the survey is on average approximately 30 minutes. (This time may vary by CCOE depending on how many partners and services are offered and how easy it is to generate utilization information from their information technology system.) Survey respondents will have one month to collect the necessary information and complete the survey at their convenience. Throughout this time, the Evaluation Team will be actively involved, providing survey related support and monitoring survey response rates.
CCOE Community Partner Survey
The Evaluation Team will send an email invitation to each partner organization chosen to participate. The email will include detailed instructions on how to access the online survey, what data are needed, and the time frame provided for completion. A point of contact on the Evaluation Team will be provided to answer questions regarding the survey or the CCOE evaluation. The partners will have approximately one month to gather requested information, fill out and submit the completed survey. Based on pilot testing, it will take approximately 20-30 minutes to complete the survey. This will vary according to the type of services the partner offers CCOE patients. The amount of time it takes to complete the survey will decrease as the number of services offered decreases. The Evaluation Team will enlist CCOE leadership to actively encourage partner participation in the evaluation and to follow up with their partners to ensure timely completion of the survey, a strategy that worked well during Round I. Partners will have a month to collect the necessary information and complete the survey, allowing them to complete the survey at their convenience. Throughout this time, the Evaluation Team will be actively involved, with the survey process, providing survey related support and monitoring survey response rates. No unusual circumstances that would require specialized sampling procedures are anticipated.
CCOE Client Survey
Data will be collected from CCOE clients via a point-of-service, pen, and paper instrument. Four hundred patients from each CCOE will be surveyed over a six-month time frame. Participants will be randomly selected2 to ensure that every patient who visits the CCOE during the survey administration period have an equal probability of selection. The survey has been designed to allow completion within 15 minutes, and survey administrators will be available to answer patient questions regarding the survey or to translate from English to Spanish, if necessary. Prior to CCOE staff administering the survey, they will review the Client Survey Administration Training Guide and participate in an hour-long training session. This will acquaint the survey administrators with the methodology and overall sampling strategy to be used at their CCOE.3 Additionally, Booz Allen will remain in close contact with CCOEs so that they can effectively modify the sampling strategy when necessary and monitor data quality.
The purpose of this survey is twofold. Gathering information on the current patient population will validate the findings from Round I. The surveys will also provide a current understanding of this population’s perception of the services offered or coordinated through the CCOE. There are no a priori hypotheses regarding group differences; therefore, the sample will not be stratified by demographic characteristic. The sampling plan considers not only the desired number of survey participants (400 per CCOE) but also the client volume at each CCOE during the survey administration period. For example, if CCOE #1 provides services to 400 patients a month, they will see 2400 patients during the 6-month administration period. If we estimate that 10% of those asked to participate will decline, then our sampling ratio becomes 1:5. In other words, at CCOE #1 the survey administrator will ask every 5th client if she would like to participate in the survey. The brevity of the survey, anonymity, and the in-person survey administration should reduce client nonresponse. However, when a client elects not to participate or has been previously surveyed, the survey administrator will revert back to the original sampling plan, and invite the next identified client to participate.
CCOE Site Visit Protocol
Each site visit will take approximately two to three days, including travel time. Four to five interviews and focus groups will take place at each CCOE. Interviews will be conducted with leadership and staff. Two to three interviews and focus groups will occur with each partner organization selected during the CCOE site visit. Each of these focus groups/interviews will last no more than 1.5 hours. This protocol mimics the one successfully used during Round I.
The Booz Allen Evaluation Team will send two staff members to conduct each site visit. This ensures that there is always one individual available to record information during all interviews, discussions, and demonstrations that take place during the site visit. One of the team members conducting the site visit will be a certified clinician, public health expert or other Subject Matter Expert (SME) in the medical and/or public health fields. The other team member will be skilled in facilitation and data collection to ensure that all topics of interest are addressed appropriately during the site visit.
An interview protocol will be used to guide the site visits. The Booz Allen Evaluation Team members will interview staff and gather documentation to gain clarification, validate, and obtain more information on previously submitted data gathered in the CCOE and/or partner surveys, as needed. Information gathered for each protocol question will be entered into a template (either Word or Excel) to facilitate the data interpretation process.
One of the requirements for receiving a CCOE designation and accompanying OWH funding is participation in the CCOE program evaluation. This requirement served as an incentive for ensuring the Center Directors and Program Coordinators responded to this survey effort during Round I, for which there was 100% participation. Each of the CCOEs in existence at the time was actively involved in the development of the evaluation methodology and the survey instruments for Round I and had multiple opportunities to provide input to these tools for that evaluation. The CCOEs have all been informed of the upcoming Round II evaluation and are aware that this is an effort that they will be asked to participate in as a requirement related to their grant funding. We anticipate that the implementation of all of these strategies will help to ensure that all CCOEs complete a survey, providing a response rate of 100% percent.
CCOE partners were very responsive to the CCOE Community Partner Survey during Round I. CCOEs have developed cooperative relationships with their partners in order to offer integrated services under the seamless model of care that each of the CCOEs is working to develop. In many cases, these relationships are defined by memoranda of understanding or actual contractual arrangements. During Round I each CCOE reached out to their partners to ascertain their willingness and ability to participate in a data collection effort. We anticipate that a high response rate will be achieved for Round II with the use of the same methods used in Round I and described here.
The use of an Internet-based survey that allows respondents to directly key in their responses, will help to maximize response rates for both the CCOE Center Director and Program Coordinator and CCOE Community Partner Surveys. The survey was constructed for ease of use. Clear and concise instructions, similar to those used in Round I with updates made to reflect reference Round II appropriately, will be included with the surveys. The Booz Allen Evaluation Team will be available to provide instructions and guidance on how to complete the survey and to answer general questions as needed. The Evaluation Team will monitor response rates and work with CCOEs and their partners to ensure completion. These methods were utilized successfully during Round I and we anticipate their successful use during Round II as well.
Non-response is not anticipated to be a factor in the CCOE Site Visits since meeting times with CCOE staff and partner organization will be planned prior to the site visit. There were no issues with participation/response during Round I, so this method has already proven successful. In addition, the CCOEs have experience with hosting site visits through their regularly scheduled OWH site visits for program review and technical assistance. The Booz Allen Evaluation Team will work with each CCOE and partner organization to determine which staff will be chosen to include in the interviews/focus groups for Round II. As with the previous evaluation effort, the team will work with the CCOEs to structure the site visit so as to minimize any disruptions in service.
OWH is planning to provide additional funding to each of the CCOEs for one of their staff members to support the client survey data collection efforts for this evaluation. This method was used successfully during Round I. Just as with Round I, a point-of-service (or in-person) survey administration will maximize response rates for the CCOE Client Survey. Prior to taking the survey, the potential respondent will be informed of the purpose of the survey and how the collected information will be used. They will also be informed that their responses will be kept anonymous. The survey is designed to be administered in a short period of time (10-15 minutes), and it will be administered immediately after a patient agrees to take the survey. The survey administrator(s) will be available to answer questions and translate to primary language spoken in the local community, if necessary. Each of these techniques was utilized during Round I, and it is anticipated that they will help to increase the likelihood of client participation in Round II as well.
The CCOE Center Director and Program Coordinator Survey was pilot tested with multiple CCOEs prior to the Round I evaluation. The specific CCOEs that participated in the pilot testing of this instrument include Northeast Missouri Health Council, Inc, Kirksville, Missouri; St. Barnabas Hospital and Healthcare System, New York, New York; NorthEast Ohio Neighborhood Health Services, Inc., Cleveland, Ohio; and Hennepin County Department of Primary Care, Minneapolis, Minnesota. During the pilot test, length of time to complete the survey was determined, as well as individual CCOE leadership reaction to survey content, clarity, and design. Feedback from this pilot test was used to make refinements to the survey instrument. Additionally, this instrument was utilized during Round I of the National Evaluation on each of the 12 CCOEs in existence at the time. It was administered using the same methods and approach planned for Round II.
The CCOE Community Partner survey was pilot tested with 9 CCOE partners who were partnering with three of the CCOE programs (Northeast Missouri Health Council, Inc, Kirksville, Missouri; Hennepin County Department of Primary Care, Minneapolis, Minnesota; Northeastern Vermont Area Health Education Center, St. Johnsbury, Vermont) prior to the National Evaluation. During the pilot test, length of time to complete the survey was determined, as well as partner reaction to survey content, clarity, and design. Feedback from this pilot test was used to make refinements to the survey instrument. This instrument was also utilized during the Round I National Evaluation, using the methodology for Round II as described in this supporting statement.
The CCOE Client Survey was pilot tested with 9 CCOE patients at an existing CCOE (NorthEast Ohio Neighborhood Health Services, Inc., Cleveland, Ohio). During this pilot test, length of time to complete the survey was determined, as well as patient reaction to survey content, clarity, and design. Feedback from this pilot test was used to make refinements to the survey instrument. This instrument was also utilized during Round I, administered using the methods described.
No pretest had been conducted for the CCOE site visits prior to Round I of the National Evaluation, however OWH’s previous experiences with conducting site visit reviews for the CCOE program and other similar programs for grant program management purposes were taken into consideration when determining the structure and methods for conducting the CCOE site visits. These methods were implemented successfully during Round I and will be repeated for Round II.
Statistical Aspects Contact
Barbara F. James
Director, National Community Centers of Excellence in Women’s Health Program
Office on Women’s Health
(301) 443-1402
Data Collection/Analysis and Statistical Contact
Fatima Riaz
Associate
Booz Allen Hamilton
240-314-5675
CCOE Center Director and Program Coordinator Survey
CCOE Community Partner Survey
CCOE Client Survey
Site Visit Protocol
1 As of FY 2007, seven AFCs are in existence, however four of them will have transitioned to AFC status in October 2006 and will not have sufficient experience functioning as an AFC to respond to the evaluation questions from an AFC perspective at the time data collection is anticipated to begin. Thus, they will be treated as full CCOEs for the purposes of this evaluation and will be asked to respond to evaluation questions from a CCOE perspective.
2 As stated in B.1, stratified random sampling will be employed for CCOEs where the survey will be administered at more than one clinical care site.
3 A customized sampling strategy will be developed for each CCOE based on the monthly volume of patients and type of CCOE services offered.
CCOE
National Evaluation: Round II OMB Supporting Statement
| File Type | application/msword |
| Author | DHHS |
| Last Modified By | DHHS |
| File Modified | 2006-12-20 |
| File Created | 2006-12-20 |