Benchmarking CNNs, Vision Transformers, and Foundation Models for Explainable Brain Tumor MRI Classification: Calibration, Decision Analysis, and External Validation

Akwasi AsarearXiv 2026 · arXiv

TL;DR

This paper presents a rigorous, "leakage-aware" benchmark of six deep learning architectures for brain tumor MRI classification, revealing that nearly 36% of common test sets consist of near-duplicates from training data. While models appear indistinguishable on internal benchmarks, external validation exposes massive performance gaps, showing that medical foundation models can paradoxically generalize worse than standard CNNs due to source-specific shortcut learning.

What Problem Does This Paper Address?

In the race to achieve state-of-the-art (SOTA) results in medical imaging, researchers often rely on public datasets where the independence of training and testing partitions is rarely scrutinized. Many papers report accuracies between 95% and 99% for brain tumor classification, but these figures may be inflated by "data leakage"—where identical images or different slices from the same patient appear in both the training and test sets.

Furthermore, models are often compared solely on accuracy, neglecting critical clinical factors like calibration (does a 90% confidence score actually mean 90% accuracy?), explainability (is the model looking at the tumor or the background?), and external validation (does it work on data from a different hospital?). This paper aims to strip away these biases to see which architectures actually hold up under rigorous evaluation.

Methods

The author benchmarked six models across three families:

  • CNNs: EfficientNetV2-M and ConvNeXt-V2-Tiny.
  • Vision Transformers (ViTs): Swin-V2-S and ViT-L/16.
  • Foundation Models: DINOv2-L (natural image self-supervised) and MedSigLIP-448 (medical domain-specific).

The study employed a "leakage-aware" protocol:

  1. Pixel-Verified Screening: Using perceptual hashing and cross-correlation to find and remove near-duplicates.
  2. Patient-Grouped Splits: Ensuring all slices from a single patient stay within one partition.
  3. Rigorous Evaluation: Assessing models using temperature scaling for calibration, decision curve analysis for clinical utility, and five different XAI methods (like Grad-CAM and LIME) evaluated against expert-annotated tumor masks.

Datasets

  • Primary: The Kaggle Brain Tumor MRI Dataset (7,216 images), a composite of the Figshare, SARTAJ, and Br35H collections.
  • External: BDNeuro-MRI (5,941 images). Crucially, the author discovered that 72% of this "external" dataset actually duplicated the Kaggle training data, requiring a massive filtering effort to create a truly independent test set of 1,675 images.

The paper highlights how previous studies (summarized in Figure 1) often failed to account for these overlaps.

Table 1: Representative brain tumor MRI classification studies as described by their authors. The studies used different datasets, class definitions and validation protocols, so the reported results are not directly comparable.

Key Insights and Ideas

  • The Illusion of SOTA: On the standard public test set, all models performed similarly (94.5%–95.6%). However, once duplicates were removed, accuracy dropped across the board. Internal benchmarks on these datasets are essentially "uninformative" for distinguishing model quality.
  • The Foundation Model Paradox: MedSigLIP-448, a model pretrained specifically on medical images and text, achieved the highest validation accuracy but the worst external generalization (falling to 77.6%). It appears to have learned "shortcuts"—identifying the source dataset's specific scanner characteristics rather than the actual pathology.
  • Explainability is Often "Borderline": Quantitative analysis of attribution maps showed that models rarely "look" at the tumor. ViT-based models, in particular, placed 33–42% of their attention on the outer 10% border of the image. Heatmaps that look plausible to the human eye often fail to overlap with expert masks.
  • Robustness > Pretraining: The standard ViT-L/16 and the CNNs (EfficientNet/ConvNeXt) were the most robust, losing the least accuracy when moving to external data.

Key Results

  • Data Audit: 35.6% of the test images had a verified near-duplicate in the training folder.
  • Internal Performance: Accuracies were statistically indistinguishable (94.5%–95.6%).
  • External Validation: Performance diverged sharply. ViT-L/16 led with 95.0%, while the medical foundation model (MedSigLIP) crashed to 77.6%.
  • XAI Failure: The "pointing-game" hit rate (how often the model's most important pixel is inside the tumor) was abysmal, peaking at only 25.5% for the best model-method pair.

Limitations and Future Work

The author acknowledges several constraints:

  • 2D vs. 3D: The study uses 2D slices rather than 3D volumetric scans, which is how radiologists actually diagnose tumors.
  • Metadata Gap: There was no access to patient demographics or scanner metadata, which could have helped further investigate shortcut learning.
  • Single External Source: While external validation was performed, it relied on only one additional dataset.

Future work should prioritize prospective, multicenter validation on volumetric MRI with realistic tumor prevalence and develop training strategies specifically designed to suppress source-specific shortcuts.

Summary updated

Abstract

Deep learning models for brain tumor MRI classification are usually compared by accuracy alone, on public datasets whose independence is rarely checked. We benchmarked six architectures: two convolutional networks (EfficientNetV2-M and ConvNeXt-V2-Tiny), two vision transformers (Swin-V2-S and ViT-L/16), and two foundation models (DINOv2-L and MedSigLIP-448), for four-class classification of 7,216 public MRI images. We assessed accuracy, calibration, explainability, ensembles, and decision curves, with every fitted quantity estimated on validation data only. Pixel-verified duplicate screening showed that 36% of the test images had a near-duplicate in the training data. Test accuracy ranged from 94.5% to 95.6%, with no pairwise difference after Holm correction, and fell to 90.3% to 91.7% on the 773 test images free of duplicates and shared patients. Ensembles did not outperform the best single model. Attribution maps passed a model-randomization test but localized expert-annotated tumors poorly. On 1,675 external BDNeuro-MRI images that did not duplicate the training data, accuracy ranged from 77.6% (MedSigLIP-448) to 95.0% (ViT-L/16), and 12 of 15 pairwise differences were significant. Internal accuracy on this public collection therefore did not distinguish the architectures, whereas the duplicate-filtered external test did.
BibTeX
@misc{asare2026benchmarking,
  author = {Akwasi Asare},
  title = {{Benchmarking CNNs, Vision Transformers, and Foundation Models for Explainable Brain Tumor MRI Classification: Calibration, Decision Analysis, and External Validation}},
  year = {2026},
  eprint = {2611.11793},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11793},
}
Figure