Abstract
Medical imaging datasets are frequently treated as static benchmarks, yet our understanding of their inherent biases, shortcuts, and errors evolves long after their initial publication. This paper introduces a "living review" framework and an interactive SQL database designed to track "research artifacts"—supplementary findings such as label noise, demographic biases, or spurious correlations. By mapping the relationships between datasets and subsequent research, the authors aim to provide the AI community with a transparent, up-to-date foundation for model development, moving beyond the limitations of static data snapshots.
Introduction: The Problem of Static Benchmarks
In the current machine learning (ML) landscape, medical imaging datasets are often utilized as "as-is" benchmarks. However, these datasets are rarely perfect; they are frequently plagued by research artifacts discovered years after release. These issues include label noise, demographic biases, and "shortcuts"—spurious correlations where a model learns to identify a hospital's specific scanner type or clinical markings rather than the actual pathology.
Traditional literature reviews are static and quickly become obsolete. A researcher today might train a model on a dataset from 2017, unaware that a 2021 study identified a significant shortcut that invalidates their results. This disconnect highlights a critical lack of "data provenance." To bridge this gap, the authors propose a dynamic system to monitor and link evolving findings to their original data sources.
The Living Review Framework
The authors introduce a three-part framework designed to maintain the relevance and transparency of medical imaging data:
- Overarching Publication: A continuously updated "living" paper that serves as a central hub, linking to new evidence and findings as they emerge.
- Zenodo Documentation: A dedicated repository for research artifacts, such as new segmentation masks or bias reports. These are assigned unique DOIs to ensure they are properly cited and tracked.
- Interactive SQL Database: A searchable database (built using PostgreSQL, Python, and Streamlit) that maps the complex relationships between datasets and papers. It utilizes specific "citation functions" to clarify whether a paper produces a new dataset, extends an existing one with new labels, or identifies a flaw.
This framework is supported by a synthesized set of best practices for the Dataset Lifecycle, covering everything from the initial clinical task translation to long-term data governance and the implementation of persistent identifiers.
Meta-Analysis of Datasets and Artifacts
While the paper does not introduce a new image repository, it provides a comprehensive meta-analysis and database of 16 major public datasets and 24 associated research artifacts. The scope includes:
- Chest X-rays: NIH-CXR14, CheXpert, MIMIC-CXR, and PadChest.
- Skin Lesions: ISIC Archive, Fitzpatrick17k, and PAD-UFES-20.
- Fetal Ultrasound: Fetal Planes DB and HC18.
- Neuroimaging: OASIS, BraTS, and ISLES 2022.
The authors highlight how critical artifacts—such as the CheXmask (additional segmentation masks) or reports on gender bias in CheXpert—often remain unlinked to the original data in traditional repositories, a gap their database specifically addresses.
Key Insights: Artifacts and Shortcuts
The "Artifact" as a First-Class Citizen
The authors define a "research artifact" as any produced object—derived datasets, bias reports, or embeddings—that provides necessary context for a dataset. Their analysis reveals that papers associated with shared artifacts receive approximately 75% more citations, suggesting that the community highly values these supplementary contributions.
The MICCAT Taxonomy of Shortcuts
Shortcuts are pervasive in medical imaging. The paper utilizes the "MICCAT" taxonomy to categorize these confounders. For instance, models often learn to detect pneumonia by identifying chest drains (a treatment for the condition) or identify skin cancer by spotting ink markings left by surgeons. These environment-level confounders lead to models that perform well on specific benchmarks but fail in real-world clinical settings.
The Demographic Data Gap
A significant finding is the lack of demographic metadata. In neuroimaging, while 77% of studies report sex, only 10% report race and a mere 4% report ethnicity. Without this data, auditing models for algorithmic fairness is impossible, leading to "blind" bias that can harm marginalized patient populations.
Results and Proof-of-Concept
The authors have released a public demo at inthepicture.itu.dk, allowing researchers to visualize the "citation graph" of a dataset. This tool enables users to see at a glance all papers that have added value to or found issues with a specific dataset. Additionally, through four clinical case studies, they demonstrate that even "balanced" datasets can suffer from performance disparities due to undocumented physiological differences, such as variations in breast tissue in X-ray imaging.
Limitations and Future Directions
The primary challenge for this framework is community buy-in. Maintaining a living review requires active, ongoing contributions, which often conflicts with academic incentives that prioritize "new" research over "maintenance" work. Furthermore, the current proposal lacks a formal moderation mechanism to verify community-submitted artifacts.
Future work suggests that major conferences (e.g., NeurIPS, MICCAI) could adopt "rolling" review styles for data artifacts. This would better support the evolving ecosystem of medical imaging and ensure that data work—often dismissed as "invisible labor"—is properly recognized and rewarded.
Summary updated