TL;DR
Recent retractions of high-profile COVID-19 studies using Electronic Health Record (EHR) data highlight a critical gap in how these studies are appraised. This paper provides a practical, six-point framework for researchers, clinicians, and editors to critically evaluate EHR-derived research, emphasizing data quality, transparency, and methodological rigor.
What Problem Does This Paper Address?
The proliferation of studies leveraging Electronic Health Record (EHR) data, particularly in the context of the COVID-19 pandemic, has unfortunately been accompanied by a concerning number of retractions due to issues with data quality and study methodology. While EHR data offers a rich source of real-world information, its inherent complexities and variability make it challenging for those outside of specialized informatics fields to critically assess the validity and reliability of research findings. This paper aims to bridge that gap by demystifying the appraisal process for EHR-derived studies, offering a clear set of considerations to ensure more robust and trustworthy research.
Methods
The authors, a consortium of 96 biomedical informaticians (the 4CE consortium), synthesized their collective expertise to develop a practical checklist for evaluating EHR-derived studies. They focused on six key areas crucial for appraising data quality, validity, and interpretability, drawing from the broader informatics literature but specifically tailoring it to EHR-specific challenges. The methodology involved distilling common concerns and best practices into a less technical, broadly applicable framework, excluding general biostatistical best practices and recommendations already covered by existing reporting guidelines like RECORD.
Datasets
Not discussed in the paper. The paper focuses on the methodology of appraising studies that use EHR data, rather than presenting new research with specific datasets.
Key Insights and Ideas
- EHR data is not a monolithic entity: The paper stresses that EHR data is highly variable, with different data types, collection methods, and institutional practices significantly impacting its quality and interpretation.
- Transparency is paramount: A lack of transparency regarding data collection, handling, transformation, and analytic code is a major red flag. Readers should expect detailed descriptions of these processes.
- "Garbage in, garbage out" applies strongly: The quality of EHR-derived research is fundamentally limited by the quality of the underlying data. Simple statistical analyses cannot compensate for poor data practices.
- Multidisciplinarity is essential: Effective EHR research requires collaboration between informaticians, clinicians, statisticians, and other domain experts to navigate the complexities of the data and its interpretation.
- Retractions are a symptom, not the disease: The paper argues that the recent retractions are indicative of a broader need for better critical appraisal skills among researchers and editors when dealing with EHR data.
Key Results
This paper does not present quantitative results from a specific study. Instead, its key "results" are the six core considerations for appraising EHR-derived studies, presented as a framework for critical evaluation:
- Data Completeness: How much of the expected data is actually present? Are specific data sources clearly identified?
- Data Collection and Handling: How was the data extracted, transformed, and managed? Were potential biases addressed?
- Data Type: What specific data types were used (e.g., codified codes vs. free-text clinical notes)? How were they processed?
- Robustness Against EHR Variability: Did the study account for differences in data collection and practice across institutions, countries, or time?
- Transparency: Is the data extraction and analytic code shared or clearly described?
- Multidisciplinary Approach: Was the study conducted with input from a diverse team of experts?
A 12-item checklist summarizing these points is provided in Table 1.
Limitations and Future Work
The authors acknowledge that this viewpoint is not focused on reviewing specific methodological options for investigators using EHR-derived data, which has been covered in detail elsewhere. They purposefully limited the discussion to issues most relevant to a general audience, particularly for pandemic-related studies. The paper implicitly suggests that future work should focus on developing more standardized and transparent methods for EHR data extraction and analysis, as well as fostering greater collaboration and education across disciplines to improve the quality and trustworthiness of EHR-derived research. The authors also note that while patient-level data sharing is challenging due to privacy concerns, sharing intermediate results and aggregate distributions can enhance transparency.
Summary updated