Evaluating a Structured Reporting Template to Increase Transparency and Reduce Review Time for Healthcare Database Studies
BACKGROUND
Real world evidence (RWE) generated from real world data (RWD) increasingly informs important decision making in healthcare. However, generating high quality evidence from secondary data that is not collected for research purposes can involve numerous complex design and analytic decisions.1,2,3 Because of this, reviewing evidence from studies conducted with RWD, such as administrative or clinical healthcare databases, can require substantial staff time within regulatory and other organizations.
Status Quo
Currently, there is wide variation regarding which study parameters are specified in protocols or reported in publications for database studies. Research groups use varied terminology to describe the same design and analysis concepts.4,5 Attempts to replicate evidence from database studies have been hampered by a lack of transparency in reporting on critical study implementation details.4,5,6,7,8,9
Transparent reporting on key design and technical implementation parameters would streamline and increase efficiency for stakeholders who need to report or review such evidence. Recognizing this, the International Society for Pharmacoepidemiology (ISPE) and International Society for Pharmacoeconomics and Outcomes Research (ISPOR), the two leading professional societies with representation from stakeholders focused on generating evidence from RWD, created a joint task force to address transparency, reproducibility, and ability to assess validity of database research. This task force produced a catalogue of specific parameters that represent scientific decisions made when implementing database studies.4 Clear reporting of these parameters would facilitate replication of the evidence as well as the ability to evaluate validity and relevance to a given population.5 However, full transparency must be balanced against the burdens it would impose on those reporting or reviewing supporting materials.
To facilitate transparency and replicability of evidence, greater detail on study implementation or programming code to enable replication could be provided in appendices. However, it can be extremely time-consuming to parse pages of convoluted prose and unrealistic to expect decision makers to deconstruct the thousands of lines of programming code involved in implementing an RWD study. Study implementation decisions would remain obfuscated for decision makers without clear annotation regarding which design and analysis elements were actually implemented.
Building on the ISPE/ISPOR consensus document, this project will design and field test a structured reporting template that presents key study parameters clearly and concisely in tabular and visual formats. This will reduce confusion and misinterpretation of prose that lacks specificity or uses uncommon terminology and will provide annotation for study decisions implemented in programming code.
The structured reporting template will be based on a principle of “maximum efficiency,” meaning the template will be limited to information required to communicate the key study implementation decisions outlined in the catalogue endorsed by ISPE/ISPOR, such as temporal anchors in the patient timeline, ordering of exclusions relative to selection of study entry date, and complete code algorithms.4 Further, the level of detail in the template will be readily tailored for different use cases. Prose will be limited although investigators will have the flexibility to add details in their own words. A glossary will be provided for terminology used in the template.
A structured reporting template would increase efficiency for those who generate or review evidence from databases by creating clear expectations regarding which details should be reported in publications, study protocols, or reports as well as how and where such information will appear. Engaging researchers, journals, regulators, health technology assessment, payers, and other stakeholders will be critical to maximize the impact of the structured reporting template.10 Therefore, this project will engage a consortium of stakeholders and seek endorsement and cultural acceptance of providing structured reporting of study implementation decisions as a part of standard research practice.
SPECIFIC AIMS
Evaluate gains in clarity and efficiency of review of healthcare database studies with structured reporting of specific study design and implementation parameters.
Gain stakeholder input to optimize a proposed template with the goal of endorsement/adoption by stakeholders.
METHODS
The Brigham and Women’s Hospital (BWH) team will fill out a structured reporting template for one selected publication with feedback from the Food and Drug Administration (FDA) (the sponsor). We will select a paper that is missing some key information on study design or implementation. The reporting template will have sections for details about specific study design and analysis parameters for the published study. We will extract information that is included in the publication and highlight where that information may be missing from the published manuscript or appendices.
BWH and FDA will compile a list of stakeholders or stakeholder organizations whose input we would like to invite (see list in Appendix A).
BWH will reach out to contacts at each organization, asking if they would be interested in participating in a conference call focused on evaluating and offering feedback for a structured reporting template designed for non-randomized database studies (see letter in Appendix B).
Consent for offering feedback in focus groups will be documented with signed consent forms (Appendix C).
Participation will involve:
BWH will schedule focus groups via WebEx conferences to collect qualitative feedback on the usability, clarity, and efficiency of the structured report.
The 90-minute focus groups will include a brief introduction to the structured reporting template with one example from the literature and a structured discussion about risks, benefits, and areas that could be modified to improve clarity, efficiency, and usability.
At least two weeks prior to the call, BWH will provide participating staff members with the structured template + user Guide, a list of discussion questions, and two optional background publications to review (ISPE/ISPOR joint task force paper and the design visualization paper).
IV. REFERENCES
1. Guidance for Industry and FDA Staff: Best Practices for Conducting and Reporting Pharmacoepidemiologic Safety Studies Using Electronic Healthcare Data, May 2013. https://www.fda.gov/downloads/drugs/guidances/ucm243537.pdf
2 Public Policy Committee, International Society of Pharmacoepidemiology. Guidelines for good pharmacoepidemiology practice (GPP). Pharmacoepidemiol Drug Saf. 2016;25(1):2-10. https://www.ncbi.nlm.nih.gov/pubmed/26537534
3. Langan SM, Benchimol EI, Guttmann A, Moher D, Petersen I, Smeeth L, Sørensen HT, Stanley F, Von Elm E. Setting the RECORD straight: developing a guideline for the REporting of studies Conducted using Observational Routinely collected Data. Clin Epidemiol. 2013;5:29-31.
4. Wang SV, Schneeweiss S, Berger ML, Brown J, de Vries F, Douglas I, Gagne JJ, Gini R, Klungel O, Mullens CD, Nguyen MD, Rassen JA, Smeeth L, Sturkenboom M, joint ISPE-ISPOR Special Task Force on Real World Evidence in Health Care Decision Making. Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0. Pharmacoepidemiol Drug Saf. 2017;26(9):1018-1032.
5. Wang SV, Verpillat P, Rassen JA, Patrick A, Garry EM, Bartels DB. Transparency and Reproducibility of Observational Cohort Studies Using Large Healthcare Databases. Clin Pharmacol Ther. 2016;99(3):325-332.
6. Reproducible Evidence: Practices to Enhance and Achieve Transparency (REPEAT) Initiative. https://www.repeatinitiative.org/
7. Matcho A, Ryan P, Fife D, Reich C. Fidelity assessment of a clinical practice research datalink conversion to the OMOP common data model. Drug Saf. 2014;37(11):945-959.
8. Lalmohamed A, van Staa TP, Vestergaard P, Leufkens HGM, de Boer A, Emans P, Cooper C, de Vries F. Statins and Risk of Lower Limb Revision Surgery: The Influence of Differences in Study Design Using Electronic Health Records From the United Kingdom and Denmark. Am J Epidemiol. 2016;184(1):58-66.
9. de Vries F, de Vries C, Cooper C, Leufkens B, van Staa TP. Reanalysis of two studies with contrasting results on the association between statin use and fracture risk: the General Practice Research Database. Int J Epidemiol. 2006;35(5):1301-1308.
10.Nosek BA, Alter G, Banks GC, Borsboom D, Bowman SD, Breckler SJ, Buck S, Chambers CD, Chin G, Christensen G, Contestabile M, Dafoe A, Eich E, Freese J, Glennerster R\, Goroff D, Green DP, Hesse B, Humphreys M, Ishiyama J, Karlan D, Kraut A, Lupia A, Mabry P, Madon TA, Malhotra N, Mayo-Wilson E, McNutt M, Miguel E, Paluck EL, Simonsohn U, Soderberg C, Spellman BA, Turitto J, VandenBos G, Vazire S, Wagenmakers EJ, Wilson R, Yarkoni T. SCIENTIFIC STANDARDS. Promoting an open research culture. Science. 2015;348(6242):1422-1425.
https://www.ncbi.nlm.nih.gov/pubmed/26113702
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Wang, Shirley,Ph.D. |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |