Office of Women’s Health
The Heart Truth Professional Materials
Program Evaluation
1. Circumstances that make collection of information necessary
The Heart Truth Campaign was launched by the National Heart, Lung, and Blood Institute (NHLBI) in September 2002 to increase women's awareness of heart disease. The Heart Truth joins together leaders in women's health—along with corporate and media partners—to create a national movement aimed at delivering an urgent wake-up call to women about heart disease.
As part of NHLBI's The Heart Truth Campaign, The Office on Women’s Health’s (OWH) National Centers of Excellence in Women's Health (CoEs) and National Community Centers of Excellence in Women's Health (CCOEs) worked jointly to develop professional education materials for health care professionals. Using these materials, the CoEs mad CCOEs are involved in dissemination activities that will educate health professionals on the prevention of heart disease in women.
The campaign materials include:
Training/curriculum materials for medical students and allied health professional students,
Slide presentations for cardiologists, primary care physicians, and allied health professionals,
Web-based interactive multiple-unit, case-based learning modules for training and self study, and
An online location The Heart Truth Professional Education Web Site housing all the above materials (www.womenshealth.gov/hearttruth).
OWH is interested in evaluating the effectiveness of these materials in terms of the following two outcomes:
To what extent were the CoE/CCOE teams successful in persuading healthcare institutions and practitioners to obtain and actively use the education and training materials; and
To what extent did the use of professional education/training materials lead to changes in practice by healthcare providers who were exposed to them?
This collection is authorized by 42 USC 241 Public Health Service Act (see attached). The section authorizes the collection of information for research of diseases significantly occurring in the U.S.
How, by whom, and for what purpose is the information used
HOW: This information will be used by the Office on Women’s Health in meeting the requirements of PART.
BY WHOM: OWH and NHLBI will use the evaluation of material dissemination to inform future dissemination efforts undertaken by both OWH and NHLBI.
FOR WHAT PURPOSE:
OWH and NHLBI will use the results of this evaluation to:
A. inform future dissemination efforts undertaken by both institutions, and
B. guide future distribution of the program and it’s materials to health care facilities
across the country in an effort to continue to heighten awareness of women’s heart health issues.
Extent of automated information collection
Respondents will be asked to take the survey on-line. In total, 100% of respondents will use the on-line option. The Heart Truth participants will receive an e-mail from their professor or instructor with a link to the on-line survey. It is estimated that the on-line option will reduce the burden for those taking the survey (in contrast to a paper or phone survey).
Efforts to avoid duplication
Part of the Program Coordinating Center (PCC - the Center which administered The Heart Truth program) evaluation was to assess pretest and posttest knowledge for an online presentation of these materials for Continuing Medical Education/Continuing Education units (CME/CEU). OMB clearance has been obtained for this evaluation (OMB No. 0990-0288). However, the Center’s evaluation was geared toward understanding whether individuals were learning the materials.
This survey differs from the PCC’s because it is testing outcomes of the program itself. Specifically, the outcome of interest is to what extent did the use of professional education/training materials lead to changes in practice by healthcare providers who were exposed to them.
Efforts to minimize the burden on small businesses
It is not anticipated that small businesses will participate extensively in this survey (less than 10% of total). The majority of respondents will be from medical schools. However, the survey is open for medical professionals and therefore, a physician may be eligible. To minimize the burden to these physician’s the following steps were taken:
Respondents were asked to opt-in or out of this survey process when they download the materials. Gallup will only contact those physicians who agree to participate.
The time estimated to complete the online survey is only 5 minutes.
All contact will be through electronic mediums so the physician will be able to participate at a time that is most convenient for them.
The survey is 10 questions long so there is no reason to shorten it further for small businesses.
Impact of less frequent collection of information
This data collection is a one time assessment specifically designed for The Heart Truth materials.
Special Circumstances
There are no special circumstances for this data collection.
Compliance with 5 CFR 1320.8
A 60-day Federal Register Notice was published in the Federal Register on September 21, 2006 Vol 71, No 183, pp 55202. There were no public comments received (see attached)
The study methodology for this program was reviewed in 2006 by:
Ana Nunez, MD, Director of CoE Drexel University, 212-991-8450
Karen Freund MD, Director of CoE, Boston University, (617) 638-8035, karen.freund@bmc.edu
Wilma Tilson, PhD, Senior Health Policy Analyst, ASPE, 202-205-8841, Wilma.Tilson@hhs.gov
No problems occurred during the consultation that could not be resolved and no other public contact was received.
Payments or Gifts to Respondents
There are neither payments nor gifts to respondents.
Assurance of Confidentiality
There is no promise of confidentiality given to respondents because there is no personal identifying information asked for or given by respondents. The contractor will also not know the e-mail addresses of respondents
Justification for data collection of a sensitive nature
There is no sensitive data being collected in this effort.
Burden hours were calculated based on test surveys taken by professional Gallup staff. Five different individuals took both the screener and the survey to test the timing of them and all five completed the survey in between 5 and 6 minutes and the screener in just under 3 minutes
Type of Respondent
|
Form Name
|
No. of Respon- dents |
No. Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden Hours |
Physicians |
Screener |
200 |
1 |
3/60 |
10 |
Professional Presentees (primarily medical students) |
On-Line Survey |
400 |
1 |
6/60 |
40
|
Total |
|
600 |
|
9/60 |
50
|
Estimate of cost burden
The number of respondents estimated in each category:
Physicians (200): The number of physicians who have already responded that they would participate in the survey, plus an additional 50 people.
Professional Presentees (400): Response rate and eligibility among physicians assumed at 30%, assuming each physician can recruit 10 students (professional presentees) to participate.
Type of Respondent |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Physicians* |
10 |
$83.21 |
$830.21 |
Respondents (medical school students)++ |
40 |
$41.61 |
$1664.40 |
Total |
50 |
|
$2494.61 |
*salary estimate taken from Bureau.
*salary estimate taken from Bureau of Labor Statistics-General Internists salary.
++Medical student’s salary conservatively considered to be 50% of physician’s salary
13. Capital Costs (Maintenance of Capital Costs)
Not applicable
14. Estimate of cost to the Federal Government
OWH has contracted for development of the evaluation design, creation of instruments, data collection, analysis and reporting. The total cost of data collection will be $85,000 for contractor time and $6,256 for OWH staff time for a total of $91,256.
Explanation of program changes or adjustments
This is a new collection of information. There are no program changes or adjustments.
Publication of results of data collection
PLANS FOR PUBLICATION
The contractor will prepare a final report based upon the requirements of the contract. The report will contain a background section and scope, design, and methodology sections. The main body of the report will describe and interpret the key findings, which will include final analytical tables. A chapter will focus on the relevance of the evaluation to PART. The final section of the report will have a conclusion and report recommendations based on the survey outcomes.
The study report will be available upon request. In addition, the study results will be reported to all institutions that request a copy.
PLANS FOR TABULATION
The analysis of this data will be focused on the three outcome questions asked in the survey (the remainder are demographic questions). Tables will be by the type of professional exposed to the materials (i.e. medical students vs. practitioners) or by type of professional exposed (i.e. cardiologist vs. internal medicine).
Once thoroughly reviewed and revised as necessary, the analytical tables will be converted from raw statistical output to a format that will be easy for reviewers to read and interpret. This will include incorporating long titles for tables and full descriptive names for all variables and classification categories. Variable construction and table generation will be done with SPSS software.
Exhibit 1: Scheduling of Tasks
ITEM |
DUE DATE |
Develop draft Survey Instrument |
Completed |
Finalize survey instrument |
Completed |
OMB Clearance Package submitted |
Completed |
IRB approval from each participating hospital |
As needed |
Receive OMB clearance |
In progress |
Begin data collection (of those who have been exposed w/in the past 3 months) |
January, 07 |
Analyze data |
March, 07 |
Produce data tables |
April, 07 |
Draft Report on Study Results |
April 30, 07 |
Final Report |
May 15, 07
|
Conduct First Briefing |
May 30, 07 |
OMB approval for not displaying OMB date
The OMB approval expiration date will be displayed on all survey materials.
Exceptions to the certificate statement
There are no exceptions to the certification statement.
B: COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Describe potential respondent universe and any sampling selection method to be used
This research project employs a survey instrument, in Web mode, administered 2-3 months after the respondent was exposed to The Heart Truth materials. The population will be medical students and practitioners who are exposed to The Heart Truth materials (i.e., participants who were presented this information).
Respondents will be drawn from three locations:
Practitioners or others who download The Heart Truth materials from the OWH website are asked at that time if they would be willing to participate in a follow-up survey. If they respond ‘yes’ the contractor will work with these practitioners to understand when they will be presenting the information. The follow-up will occur 2 to 3 months following that date, the contractor will forward an e-mail to the practitioner to forward on to their audience members. The e-mail will contain a link to a survey that will ask the outcome questions of interest. Because the practitioner will be forwarding the e-mail to the audience, the contractor will not offer any confidentiality pledges.
All audience members of a given practitioner will be given the same access code in order to analyze data by practitioner to understand if there are outcome differences between presentation styles or presentation manners (i.e. grand rounds vs. classroom).
Participating CoEs and CCOEs will be asked about their dissemination efforts. For those CoEs and CCOEs who intend to disseminate via classroom, professors/practitioners will be asked for their assistance with student follow-up. For those who intend to disseminate materials through grand rounds, lecturers/practitioners will be asked for assistance in sending e-mails to those who attended the grand round.
The follow-up will proceed as in #1 above.
MEDSCAPE is a continuing medical education provider. Practitioners can go on-line and get continuing credit for completing modules on various medical subjects. MEDSCAPE will have 5 The Heart Truth modules which will enable users to earn Continuing Medical Education/Continuing Education units (CME/CEU) at no cost through the University of Wisconsin. At the completion of the module, practitioners will be asked if they would be willing to participate in a follow-up survey and the contractor will be forwarded their name and e-mail address. The follow-up will be to send those respondents a link to the survey directly.
Because the contractor will be recruiting the practitioner to participate in the survey and it will be the practitioner who asks exposed students and other practitioners to participate directly at the time of exposure, respondents will be more likely to participate. Response rates are expected to be approximately 65% (of those who the practitioner opted into the study). However, final response rates may not be known as the initial group of exposed presenters will not be known nor will the complete population of exposed professionals. The only response rate to be calculated will be the response rate of the number of requests sent out and the number of returns received by the group of exposed professionals.
2. Describe procedures for collecting information, including statistical methodology for stratification and sample selection, estimation procedures, degree of accuracy needed, and less than annual periodic data cycles.
Statistical methodology for stratification and sample selection: There is no stratification of the data. There is no “selection” of sample either. The contractor will attempt to contact anyone who agrees to participate in the assessment of The Heart Truth materials and agrees to an evaluation of their audience 3-4 months after being exposed to the materials. However, it is hoped that the evaluation will be able to get responses from at least 400 exposed professionals to complete this evaluation process.
Estimation procedures: There is no estimation in this process as the population of exposed professionals is unknown.
Degree of accuracy: There is no degree of accuracy with this survey as the population of exposed professionals is unknown.
Unusual problems requiring specialized sampling: Sampling for this survey requires permissions at two steps of the dissemination process: The process requires that The Heart Truth materials be disseminated to organizations that will host presentations, and that the materials then be presented to an audience of professionals. This survey is focused on the professionals who will be presented the materials in the second step. However, a representative of the host organization must first agree to participate in the study (the practitioner). Although we will know the population that decides to download the materials from the website, and know which CoEs and CCOEs obtained the materials, it is unknown who actually then receives the materials in a lecture. Because the universe of exposed professionals is unknown, no statistical sampling will occur. Those students/practitioners who participate in the MEDSCAPE module will have to agree to participate in our study. We will not know how many students in total completed the MEDSCAPE module on The Heart Truth information (therefore, no response rate will be known among those who participate in our survey vs. those who are eligible to participate).
Type I and Type II error: Statistical testing of this type is not permitted for this type of data collection as the population is unknown.
Use of periodic data collection: This is a one time data collection effort.
3. Methods to Maximize Response Rates
The survey will use several methods to get as many respondents as possible (response rates are not the focus here, rather getting 400 completes is the focus).
The contractor inserted a note at the end of the MEDSCAPE module asking those completing the posttest to participate. As these names are collected, participants will be contacted within two weeks to set up a process to again contact them 2-3 months later to complete the survey.
OWH hosted a National Heart Truth dissemination conference with representatives from the CoEs and CCOEs. At the conference, we formally asked the CoEs and CCOEs to participate in the study. The contractor will follow-up with each CCOE and CoE to get their information on who is presenting the materials and set up a process for including them.
Before materials can be downloaded from the OWH website, every downloader is asked if they would be willing to participate in The Heart Truth follow-up study. Those who agree will be contacted within two weeks to set up a process for following up with their exposed practitioners within 2-3 months.
Therefore, each of the three potential inputs into the data collection process will be touched at least once before the actual survey is sent out.
4. Test of Procedures or Methods
The only testing of procedures will be between those who are exposed to the materials through the OWH affiliated CCOEs or CoEs compared to those who are exposed to the materials via downloading from the website (i.e. no specific affiliation with OWH). No other tests of procedures or methods will be undertaken.
5. Contact Names
Dr. Alison Simon, Project Director, The Gallup Organization, 202-715-3030, alison_simon@gallup.com
Dr. Manas Chattopadhyay, Chief Statistician, The Gallup Organization, 202-715-3030, manas_chattopadhyay@gallup.com
Dr. James Wells, Methodologist, The Gallup Organization, 202.715.3030, jim_wells@gallup.com
| File Type | application/msword |
| Author | Preferred Customer |
| Last Modified By | DHHS |
| File Modified | 2006-12-15 |
| File Created | 2006-12-15 |