SUPPORTING STATEMENT FOR
EVALUATION OF OHRP EDUCATIONAL ACTIVITIES
Submitted by
Division of Education and Development
Office for Human Research Protections
Rockville, MD
Contact person:
Irene Stith-Coleman
Director
Division of Policy and
Assurances
Office for Human Research Protections
1101
Wootton Parkway, Ste. 200
Rockville, MD 20852
ph.240.453.8138
fax.301.402.2071
Supporting Statement
The Office for Human Research Protection (OHRP) conducts a broad range of educational and outreach activities related to protecting humans involved in research. These activities include conferences, research community forums, and quality improvement consultations, among other projects and programs. All of OHRP’s educational and outreach activities aim to meet the needs of the research community, based on feedback and input from multiple sources. The ‘research community’ in this context includes institutions conducting human subject research, institutional review boards, investigators, and research staff.
OHRP’s educational activities are developed and implemented by the Division of Education and Development within OHRP. The project design is intended to serve OHRP’s multi-faceted needs for evaluation information, including reports to internal and external audiences about program accomplishments; emerging educational and outreach needs; and Program development opportunities.
The function of OHRP is to provide leadership and oversight on all matters related to the protection of human subjects participating in research conducted or supported by the U.S. Department of Health and Human Services (HHS). OHRP helps ensure that such research is carried out in accordance with the highest ethical standards and in an environment where all who are involved in the conduct or oversight of human subjects’ research understand their primary responsibility for protecting the rights, welfare, and well-being of subjects.
OHRP:
establishes criteria for and approves assurances of compliance for the protection of human subjects with institutions engaged in HHS-conducted or -supported human subject research,
develops, monitors, and exercises compliance oversight of HHS regulations for protection of human subjects;
provides clarification and guidance on involving humans in research,
develops and implements educational programs and resource materials, and
promotes the development of approaches to enhance human subject protections.
The Division of Education and Development is responsible for the Educational Program for the OHRP. As such, it maintains educational and guidance materials and provides workshops, technical assistance, and quality improvement programs to ensure that institutions have up-to-date information regarding human research protections. The specific objectives of the Division are to:
Produce and coordinate conferences focusing on issues in human subjects protection;
Develop and conduct quality improvement activities to improve human research protection programs;
Promote cooperative education and development efforts among external groups and consortia to improve human subjects protections and related processes;
Respond to requests for clarification and guidance regarding ethical issues in biomedical and behavioral research involving human subjects;
Provide technical assistance to institutions engaged in HHS-conducted or -sponsored research involving human subjects; and,
Maintain, promulgate, and update educational and institutional review guidance materials
This evaluation study is not mandated, but addresses the Office of Public Health and Science’s (OPHS) Government Performance and Results Act (GPRA) priority of strengthening the public health infrastructure. An educated and informed staff in organizations involved in the research enterprise can best protect the rights and welfare of research subjects, and thereby enhance subjects’ trust in the system and their willingness to participate in research studies that may lead to improved human health.
The relevant legislation authorizing collection of this type of information is found in 42 USX 289(b)(1); a copy of this statute is contained in Attachment 1.
The study directly impacts the program objective of updating educational and institutional review guidance materials. OHRP believes this evaluation would be especially timely since OHRP was recently reorganized.
The collection of information discussed in this application addresses evaluation of the first three of the six types of educational activities conducted by the Educational Division:
1. National Conferences-These regional events, conducted in collaboration with a host institution, usually draw about 175-200 attendees, last two days, and cover the broad scope of the field of human research protection. Approximately three conferences per year are anticipated.
2. Research Community Forums-These local/regional events, formerly called Town Hall Meetings, usually last one day, and are anticipated to occur nine times per Fiscal Year, based on the FY 2005 schedule.
3. Presentations-These local events are in response to a request from institutions, professional associations, private industry, patient advocacy groups, or the federal government. They take place in formal or informal gatherings of staff. The OHRP invests heavily in invited presentations, with six people who travel regularly to conduct between 80 and 100 presentations per year.
4. Staff Training Workshops-These institution-specific events, generally longer than presentations, involve 3-4 hours of intensive staff training, including a question-and-answer session.
5. Quality Assurance and Quality Improvement Site Visits -These intensive 1 ½ -2 day on-site activities are conducted in response to a request from an institution by a team usually comprised of two OHRP staff members.
6. Train-the-Trainer Site Visits- During these visits, OHRP assesses the human research protections system of the institution, as they do on all Quality Assurance and Quality Improvement Site Visits. In addition, a group of employees from the host institution (“trainers”) use the visit as a learning opportunity. These trainers accompany the OHRP staff during the two day visit in order to learn how conduct Quality Assurance and Quality Improvement Site Visits of other components of institution’s human research protection system in the future using the OHRP techniques.
The outcome of the evaluation will be used immediately and directly by the OHRP education division staff to revise educational activities, including but not limited to modification of the approach to providing education, addition or deletion of topic areas, and increased or decreased frequency of providing educational activities. This information will also help guide the determination of the OHRP Division of Education resource needs.
The specific programmatic questions to be answered from this Assessment include the following: (1) what are the outcomes of the educational (and outreach) activities conducted by OHRP; 2) do the OHRP educational activities affect research institutions’ human subjects protections programs; 3) if programs are affected, what determines the scope and magnitude of those outcomes; and, 4) how could OHRP’s educational activities be more responsive to the needs of the research community?
Data will be gathered on perceived amount of knowledge gain in the topic area; self-reported pre-disposition to change behavior related to the human protections area; satisfaction with the training; actual use of the knowledge gained in the workplace setting; and perceived need at an organizational and individual level for additional training.
The aim of this evaluation is to identify the individual components of each educational activity that should be modified and any current or new components that deserve serious consideration when further efforts are planned. Since the many OHRP educational activities are not duplicates of each other—but, instead, differ in specific ways—examination by each component will ensure an appropriate focus. The outcomes anticipated from the evaluation effort will help to address questions such as:
What types and numbers of educational activities were planned for a given period?
What types and numbers of educational activities were carried out in a given period?
How many individuals participated in those activities?
How were participants in those activities distributed across subgroups targeted for educational support?
How much resource outlay (estimated dollars/person-time) was associated with conducting those activities, including resources expended by OHRP/educators and participants?
How did participants rate their satisfaction with educational activities?
How did participants rate the type/level of knowledge gained from educational activities?
Three to six months after the educational activity, what proportion of attendees confirm that they use, in their workplace, the knowledge gained?
What features of educational activities, including type of activity and characteristics of participants, were associated with positive outcomes?
Which subgroups and topics need to be addressed in subsequent educational efforts?
Automated response collection will be utilized whenever possible. However, the initial educational activity participant survey must be administered using paper questionnaires since it will be administered immediately following an education activity in a conference or training room. The follow-up questionnaire, and the assessment of training needs, will be available as web-based modules.
There have been numerous limited or peripherally related reports by the Office of the Inspector General, Institute of Medicine, General Accounting Office, and National Bioethics Advisory Commission; however, none of these reports contained specific information on the effectiveness of OHRP’s educational and outreach activities. In fact, several of these reports encouraged OHRP to collect such information as a means of enhancing human subject protections.
No small businesses will be involved in this study.
Participants will be re-contacted approximately three to six months after attendance at the educational activity and asked to self-report changes in level of knowledge, impact of training on actual practice and overall perception of training activity. This time period will enable assessment of changes over a time period that is sufficient to enable opportunity for behavior change to occur while ensuring attendance at the activity can be recalled.
This project fully complies with all guidelines of 5 CFR 1320.5.
The Federal Register notice published December 13, 2005 (70 FR 73747) invited comments from the public regarding this information collection. No comments were received on this information collection from the public in response to the Federal Register notice during the 60 day comment period.
The protocol, methods of study design and questionnaires have been evaluated by the investigators in the agency, by a review panel specifically developed for this study (the Technical Advisory Panel; membership is listed in Appendix A), and by an expert survey review board. The questionnaires (Attachment 2) have been tested on a small group (less than nine for each questionnaire) of potential responders.
There will be no remuneration offered to participants.
Completion of the feedback form is voluntary. No identifying information is collected or stored on individuals. Individuals may opt out of provision of email address for follow-up purposes. Subjects are informed of the measures taken to protect their confidentiality in the pre-introductory letter (Attachment 3). All contractor staff sign a pledge agreeing that all information provided by the respondent will be accorded the highest degree of confidentiality allowable (Attachment 4). All data will be destroyed after completion of the evaluation.
The Privacy Act does not apply to the information contained in the Evaluation of OHRP Education Program database. OHRP will not be retrieving information about individuals from this internet site by name or other individual identifier. Therefore, this internet site will not be a “system of records” that would be subject to the requirements of the Privacy Act of 1974.
No sensitive questions are asked.
The following tables provide estimates of hour burden for each proposed survey and annualized cost to respondents.
A12-1. Estimates of Hour Burden
Type of Respondent
|
Number of Respondents |
Frequency of Response |
Average Time per Response |
Annual Hour Burden |
Initial |
2,400 |
1 |
6 min |
240 |
Follow-Up |
1,200 |
1 |
6 min |
120 |
IRB members |
2,998 |
1 |
6 min |
300 |
TOTAL |
6,598 |
|
|
660 |
A12-2. Annualized Cost to Respondents
Type of Respondent
|
Annual Hour Burden |
Hourly Wage Rate |
Annual Hour Burden |
Members of Research Community |
660 |
$42 |
$27,720 |
Total |
|
|
$27,720 |
There are no additional costs to respondents or recordkeepers.
The total remaining cost to the Federal Government is estimated to be $1.2 million. The total annualized cost is estimated to be approximately $ 364,850. The total annual costs include approximately $14,850 in Federal personnel costs, $250,000 in questionnaire administration, data entry and database creation, and $100,000 for study management.
This is a new collection of information.
Data analysis will address questions about associations between educational activities and outcomes, and will examine areas in need of educational intervention by target audience, topic, or educational modality. Reports will be created on a quarterly basis, to ensure timely use of information for revising content or addressing additional topic areas.
The most basic level of analytic output will be univariate statistical reports, such as the percentage of participants who reported high satisfaction immediately after an educational event (Initial Assessment Survey)--measurement information that could be useful for demonstrating performance and planning educational activities for ensuing periods of time. Separate tabulations of collected data for each type of educational activity will enable simple comparisons of common measures, such as satisfaction, across activity types.
Multivariate statistical analysis techniques (e.g., regression analysis) will identify correlates of a dependent variable of interest, e.g., Do respondents with different characteristics, such as a greater or fewer number of years of prior experience with human subjects research, or with a different role in human research protection, report higher or lower levels of satisfaction with a particular type of educational activity than other respondents?
A.16-1 Project Time Schedule |
|
Activity |
Time Schedule |
Administration of Initial Assessment Surveys |
First education session after OMB Approval (approximately 1-2 months) |
Administration of Follow-up Surveys
|
3 to 6 months subsequent to first education session |
Administration of Questionnaires to IRB members |
1 month after OMB approval |
Analyses |
Within 1 month of first data collection; ongoing thereafter |
Publication |
Internal reports to OHRP on quarterly basis |
The expiration date will be displayed on all forms.
No exceptions to 5 CFR 1320.9 are being requested.
All participants in an OHRP Training activity will be asked to complete an evaluation form. Sampling is not appropriate in these circumstances, due to the immediacy of the data collection activity. All Institutional Review Board (IRB) chairs identified by the OHRP will be sent a letter inviting participation in the organizational assessment. IRB chairs will select IRB members to participate within their organization.
The protocol for administration of the Initial Assessment Survey questionnaire is as follows:
Immediately following the completion of each educational activity sponsored by OHRP, questionnaires will be distributed to all participants. All participants will be asked to complete the survey, and return it to a designated place that is convenient to the room exit.
Approximately three to six months subsequent to the educational event, a list of attendees will be provided to the survey coordinator. A letter will be emailed to each attendee inviting attendees to participate in the Follow-up Assessment Survey; the respondent will have the option to complete an online survey immediately via an embedded link to the web site containing the survey form, or by going later to the designated web site.
All IRB chairs in the OHRP database will be mailed a request for completion of the Survey of Organization Representatives regarding training needs, and will be asked to solicit participation by others in their IRB. Completion of the survey will be web-based.
An individual dedicated to survey distribution and retrieval will be present at each conference. An embedded link to the web survey will be included in each emailed request for participation.
Respondents were recruited using a recent attendee list from an OHRP National Conference and Regional Community Forum and the OHRP IRB Registration List. Participants were interviewed by telephone. Each respondent completed one of three forms: Initial Assessment Survey (3 respondents); Follow-up Assessment Survey (3 respondents); or Survey of Organization Representatives (5 respondents). The instruments were pre-tested for content, wording and time needed for administration. This pre-test was considered adequate for the purposes of this study.
The statistical aspects of this study design were developed by James Bell Associates, 1001 19th Street North, Arlington VA 22209, 703-528-3230, in consultation with OHRP. The organization responsible for data collection activities and analysis during the first two years of the evaluation process is James Bell Associates. In subsequent years the assessment activities will be transitioned to the OHRP.
Attachment
Legislative Authority
From the U.S. Code Online via GPO Access
[wais.access.gpo.gov]
[Laws in effect as of January 7, 2003]
[Document not affected by Public Laws enacted between
January 7, 2003 and February 12, 2003]
[CITE: 42USC289]
TITLE 42--THE PUBLIC HEALTH AND WELFARE
CHAPTER 6A--PUBLIC HEALTH SERVICE
SUBCHAPTER III--NATIONAL RESEARCH INSTITUTES
Part H--General Provisions
Sec. 289. Institutional review boards; ethics guidance program
(a) The Secretary shall by regulation require that each entity which applies for a grant, contract, or cooperative agreement under this chapter for any project or program which involves the conduct of
biomedical or behavioral research involving human subjects submit in or
with its application for such grant, contract, or cooperative agreement
assurances satisfactory to the Secretary that it has established (in
accordance with regulations which the Secretary shall prescribe) a board (to be known as an ``Institutional Review Board'') to review biomedical and behavioral research involving human subjects conducted at or supported by such entity in order to protect the rights of the human subjects of such research.
(b)(1) The Secretary shall establish a program within the Department of Health and Human Services under which requests for clarification and guidance with respect to ethical issues raised in connection with biomedical or behavioral research involving human subjects are responded to promptly and appropriately.
(2) The Secretary shall establish a process for the prompt and
appropriate response to information provided to the Director of NIH
respecting incidences of violations of the rights of human subjects of
research for which funds have been made available under this chapter.
The process shall include procedures for the receiving of reports of
such information from recipients of funds under this chapter and taking
appropriate action with respect to such violations.
(July 1, 1944, ch. 373, title IV, Sec. 491, as added Pub. L. 99-158,
Sec. 2, Nov. 20, 1985, 99 Stat. 873.)
Section Referred to in Other Sections`
This section is referred to in sections 280e, 283f, 287c, 289a-1 of
this title.
| File Type | application/msword |
| File Title | Outline of Request for OMB Review |
| Author | mcdonnell |
| Last Modified By | DHHS |
| File Modified | 2006-12-12 |
| File Created | 2006-12-12 |