TL;DR
This paper presents a rigorous, "leakage-aware" benchmark of six deep learning architectures for brain tumor MRI classification, revealing that nearly 36% of common test sets consist of near-duplicates from training data. While models appear indistinguishable on internal benchmarks, external validation exposes massive performance gaps, showing that medical foundation models can paradoxically generalize worse than standard CNNs due to source-specific shortcut learning.
What Problem Does This Paper Address?
In the race to achieve state-of-the-art (SOTA) results in medical imaging, researchers often rely on public datasets where the independence of training and testing partitions is rarely scrutinized. Many papers report accuracies between 95% and 99% for brain tumor classification, but these figures may be inflated by "data leakage"—where identical images or different slices from the same patient appear in both the training and test sets.
Furthermore, models are often compared solely on accuracy, neglecting critical clinical factors like calibration (does a 90% confidence score actually mean 90% accuracy?), explainability (is the model looking at the tumor or the background?), and external validation (does it work on data from a different hospital?). This paper aims to strip away these biases to see which architectures actually hold up under rigorous evaluation.
Methods
The author benchmarked six models across three families:
- CNNs: EfficientNetV2-M and ConvNeXt-V2-Tiny.
- Vision Transformers (ViTs): Swin-V2-S and ViT-L/16.
- Foundation Models: DINOv2-L (natural image self-supervised) and MedSigLIP-448 (medical domain-specific).
The study employed a "leakage-aware" protocol:
- Pixel-Verified Screening: Using perceptual hashing and cross-correlation to find and remove near-duplicates.
- Patient-Grouped Splits: Ensuring all slices from a single patient stay within one partition.
- Rigorous Evaluation: Assessing models using temperature scaling for calibration, decision curve analysis for clinical utility, and five different XAI methods (like Grad-CAM and LIME) evaluated against expert-annotated tumor masks.
Datasets
- Primary: The Kaggle Brain Tumor MRI Dataset (7,216 images), a composite of the Figshare, SARTAJ, and Br35H collections.
- External: BDNeuro-MRI (5,941 images). Crucially, the author discovered that 72% of this "external" dataset actually duplicated the Kaggle training data, requiring a massive filtering effort to create a truly independent test set of 1,675 images.
The paper highlights how previous studies (summarized in Figure 1) often failed to account for these overlaps.
Key Insights and Ideas
- The Illusion of SOTA: On the standard public test set, all models performed similarly (94.5%–95.6%). However, once duplicates were removed, accuracy dropped across the board. Internal benchmarks on these datasets are essentially "uninformative" for distinguishing model quality.
- The Foundation Model Paradox: MedSigLIP-448, a model pretrained specifically on medical images and text, achieved the highest validation accuracy but the worst external generalization (falling to 77.6%). It appears to have learned "shortcuts"—identifying the source dataset's specific scanner characteristics rather than the actual pathology.
- Explainability is Often "Borderline": Quantitative analysis of attribution maps showed that models rarely "look" at the tumor. ViT-based models, in particular, placed 33–42% of their attention on the outer 10% border of the image. Heatmaps that look plausible to the human eye often fail to overlap with expert masks.
- Robustness > Pretraining: The standard ViT-L/16 and the CNNs (EfficientNet/ConvNeXt) were the most robust, losing the least accuracy when moving to external data.
Key Results
- Data Audit: 35.6% of the test images had a verified near-duplicate in the training folder.
- Internal Performance: Accuracies were statistically indistinguishable (94.5%–95.6%).
- External Validation: Performance diverged sharply. ViT-L/16 led with 95.0%, while the medical foundation model (MedSigLIP) crashed to 77.6%.
- XAI Failure: The "pointing-game" hit rate (how often the model's most important pixel is inside the tumor) was abysmal, peaking at only 25.5% for the best model-method pair.
Limitations and Future Work
The author acknowledges several constraints:
- 2D vs. 3D: The study uses 2D slices rather than 3D volumetric scans, which is how radiologists actually diagnose tumors.
- Metadata Gap: There was no access to patient demographics or scanner metadata, which could have helped further investigate shortcut learning.
- Single External Source: While external validation was performed, it relied on only one additional dataset.
Future work should prioritize prospective, multicenter validation on volumetric MRI with realistic tumor prevalence and develop training strategies specifically designed to suppress source-specific shortcuts.
Summary updated