What Every Reader Should Know About Studies Using Electronic Health Record Data but May Be Afraid to Ask (Preprint)

Isaac S. Kohane, Bruce J. Aronow, Paul Avillach, Brett K. Beaulieu‐Jones, Riccardo Bellazzi, Robert L. Bradford, Gabriel A. Brat, Mario Cannataro, James J. Cimino, Noelia García Barrio, Nils Gehlenborg, Marzyeh Ghassemi, Alba Gutiérrez‐Sacristán, David A. Hanauer, John H. Holmes, Chuan Hong, Jeffrey G. Klann, Ne Hooi Will Loh, Yuan Luo, Kenneth D. Mandl, Mohamad Daniar, Jason H. Moore, Shawn N. Murphy, Antoine Neuraz, Kee Yuan Ngiam, Gilbert S. Omenn, Nathan Palmer, Lav P. Patel, Miguel Pedrera‐Jiménez, Piotr Sliz, Andrew M. South, Amelia L.m. Tan, Deanne Taylor, Bradley Taylor, Carlo Torti, Andrew Vallejos, Kavishwar B. Wagholikar, Griffin M. Weber, Tianxi Cai2020 · DOI

TL;DR

Recent retractions of high-profile COVID-19 studies using Electronic Health Record (EHR) data highlight a critical gap in how these studies are appraised. This paper provides a practical, six-point framework for researchers, clinicians, and editors to critically evaluate EHR-derived research, emphasizing data quality, transparency, and methodological rigor.

What Problem Does This Paper Address?

The proliferation of studies leveraging Electronic Health Record (EHR) data, particularly in the context of the COVID-19 pandemic, has unfortunately been accompanied by a concerning number of retractions due to issues with data quality and study methodology. While EHR data offers a rich source of real-world information, its inherent complexities and variability make it challenging for those outside of specialized informatics fields to critically assess the validity and reliability of research findings. This paper aims to bridge that gap by demystifying the appraisal process for EHR-derived studies, offering a clear set of considerations to ensure more robust and trustworthy research.

Methods

The authors, a consortium of 96 biomedical informaticians (the 4CE consortium), synthesized their collective expertise to develop a practical checklist for evaluating EHR-derived studies. They focused on six key areas crucial for appraising data quality, validity, and interpretability, drawing from the broader informatics literature but specifically tailoring it to EHR-specific challenges. The methodology involved distilling common concerns and best practices into a less technical, broadly applicable framework, excluding general biostatistical best practices and recommendations already covered by existing reporting guidelines like RECORD.

Datasets

Not discussed in the paper. The paper focuses on the methodology of appraising studies that use EHR data, rather than presenting new research with specific datasets.

Key Insights and Ideas

  • EHR data is not a monolithic entity: The paper stresses that EHR data is highly variable, with different data types, collection methods, and institutional practices significantly impacting its quality and interpretation.
  • Transparency is paramount: A lack of transparency regarding data collection, handling, transformation, and analytic code is a major red flag. Readers should expect detailed descriptions of these processes.
  • "Garbage in, garbage out" applies strongly: The quality of EHR-derived research is fundamentally limited by the quality of the underlying data. Simple statistical analyses cannot compensate for poor data practices.
  • Multidisciplinarity is essential: Effective EHR research requires collaboration between informaticians, clinicians, statisticians, and other domain experts to navigate the complexities of the data and its interpretation.
  • Retractions are a symptom, not the disease: The paper argues that the recent retractions are indicative of a broader need for better critical appraisal skills among researchers and editors when dealing with EHR data.

Key Results

This paper does not present quantitative results from a specific study. Instead, its key "results" are the six core considerations for appraising EHR-derived studies, presented as a framework for critical evaluation:

  1. Data Completeness: How much of the expected data is actually present? Are specific data sources clearly identified?
  2. Data Collection and Handling: How was the data extracted, transformed, and managed? Were potential biases addressed?
  3. Data Type: What specific data types were used (e.g., codified codes vs. free-text clinical notes)? How were they processed?
  4. Robustness Against EHR Variability: Did the study account for differences in data collection and practice across institutions, countries, or time?
  5. Transparency: Is the data extraction and analytic code shared or clearly described?
  6. Multidisciplinary Approach: Was the study conducted with input from a diverse team of experts?

A 12-item checklist summarizing these points is provided in Table 1.

Limitations and Future Work

The authors acknowledge that this viewpoint is not focused on reviewing specific methodological options for investigators using EHR-derived data, which has been covered in detail elsewhere. They purposefully limited the discussion to issues most relevant to a general audience, particularly for pandemic-related studies. The paper implicitly suggests that future work should focus on developing more standardized and transparent methods for EHR data extraction and analysis, as well as fostering greater collaboration and education across disciplines to improve the quality and trustworthiness of EHR-derived research. The authors also note that while patient-level data sharing is challenging due to privacy concerns, sharing intermediate results and aggregate distributions can enhance transparency.

Summary updated

Abstract

<sec> <title>UNSTRUCTURED</title> Coincident with the tsunami of COVID-19–related publications, there has been a surge of studies using real-world data, including those obtained from the electronic health record (EHR). Unfortunately, several of these high-profile publications were retracted because of concerns regarding the soundness and quality of the studies and the EHR data they purported to analyze. These retractions highlight that although a small community of EHR informatics experts can readily identify strengths and flaws in EHR-derived studies, many medical editorial teams and otherwise sophisticated medical readers lack the framework to fully critically appraise these studies. In addition, conventional statistical analyses cannot overcome the need for an understanding of the opportunities and limitations of EHR-derived studies. We distill here from the broader informatics literature six key considerations that are crucial for appraising studies utilizing EHR data: data completeness, data collection and handling (eg, transformation), data type (ie, codified, textual), robustness of methods against EHR variability (within and across institutions, countries, and time), transparency of data and analytic code, and the multidisciplinary approach. These considerations will inform researchers, clinicians, and other stakeholders as to the recommended best practices in reviewing manuscripts, grants, and other outputs from EHR-data derived studies, and thereby promote and foster rigor, quality, and reliability of this rapidly growing field. </sec>
BibTeX
@misc{kohane2020what,
  author = {Isaac S. Kohane and Bruce J. Aronow and Paul Avillach and Brett K. Beaulieu‐Jones and Riccardo Bellazzi and Robert L. Bradford and Gabriel A. Brat and Mario Cannataro and James J. Cimino and Noelia García Barrio and Nils Gehlenborg and Marzyeh Ghassemi and Alba Gutiérrez‐Sacristán and David A. Hanauer and John H. Holmes and Chuan Hong and Jeffrey G. Klann and Ne Hooi Will Loh and Yuan Luo and Kenneth D. Mandl and Mohamad Daniar and Jason H. Moore and Shawn N. Murphy and Antoine Neuraz and Kee Yuan Ngiam and Gilbert S. Omenn and Nathan Palmer and Lav P. Patel and Miguel Pedrera‐Jiménez and Piotr Sliz and Andrew M. South and Amelia L.M. Tan and Deanne Taylor and Bradley Taylor and Carlo Torti and Andrew Vallejos and Kavishwar B. Wagholikar and Griffin M. Weber and Tianxi Cai},
  title = {{What Every Reader Should Know About Studies Using Electronic Health Record Data but May Be Afraid to Ask (Preprint)}},
  year = {2020},
  doi = {10.2196/preprints.22219},
}
Figure