TL;DR
This paper challenges the "accuracy-first" mindset in medical AI by demonstrating that ResNet50 and Vision Transformers (ViT) achieve statistically equivalent accuracy for gastrointestinal disease classification, yet differ vastly in computational efficiency. The authors reveal that high-performing models often rely on "shortcuts"—non-clinical artifacts like endoscope overlays—and warn that significant performance drops on external datasets make no current model truly deployment-ready without local validation.
What Problem Does This Paper Address?
In the race to achieve state-of-the-art (SOTA) results, researchers often select deep learning architectures based solely on marginal gains in accuracy. However, in clinical settings—especially resource-constrained ones—computational efficiency, statistical reliability, and interpretability are just as critical.
The authors address three main gaps:
- The Accuracy Trap: Small differences in accuracy between CNNs and Transformers might not be statistically significant or clinically meaningful.
- The Efficiency Gap: The practical cost of running these models on hospital hardware (CPUs and older GPUs) is rarely profiled.
- Shortcut Learning: Models might "cheat" by learning to recognize acquisition artifacts (like green screen insets) rather than actual pathology, leading to a false sense of security.
Methods
The study provides a controlled, reproducible benchmark comparing three architectures: ResNet50 (a standard CNN), ViT-B/16 (a standard Transformer), and MobileViT-S (a hybrid "edge-optimized" model).
Key methodological highlights include:
- Statistical Equivalence Testing: Instead of just looking for a "significant difference," the authors used a Two One-Sided Tests (TOST) procedure to prove that ResNet50 and ViT are equivalent within a $\pm3$ percentage-point margin.
- Standardized Profiling: They measured MACs (multiply-accumulate operations), parameters, and inference latency on both a Tesla T4 GPU and an Intel Xeon CPU.
- Robustness Checks: An occlusion experiment was conducted where the "ScopeGuide" inset (a common artifact in endoscopic images) was blacked out to see if model accuracy collapsed.
- Multi-method Interpretability: They used Grad-CAM, LIME, SHAP, and attention rollout to visualize where models were looking, validating these against expert-drawn polyp masks.
Datasets
- Kvasir v1: The primary dataset containing 4,000 endoscopic images across 8 classes (e.g., polyps, esophagitis, ulcerative colitis). The authors performed rigorous near-duplicate screening to prevent data leakage.
- GastroVision: Used as an external validation set (1,998 images). This dataset is "partially independent" as it shares one hospital source with Kvasir but includes data from a different country (Sweden).
- Kvasir-SEG: Used specifically for lesion-level localization analysis, providing 500 unseen polyp images with expert-verified pixel masks.
Key Insights and Ideas
- Accuracy is a Wash: In cross-validation, ResNet50 (92.12%) and ViT-B/16 (91.70%) were statistically equivalent. The choice between them shouldn't be based on accuracy.
- Efficiency is the Real Differentiator: ResNet50 is the clear winner for deployment. It has 3.6x fewer parameters, 4.3x fewer operations, and runs 2.5x faster on a CPU. It also reaches its best performance four times faster during training.
- The Danger of "Edge" Models: MobileViT-S was the most accurate on the internal test set (95.5%) but proved to be the most fragile. It relied heavily on the "ScopeGuide" artifact; when that artifact was removed, its accuracy dropped by nearly 10 points.
- The External Reality Check: All models suffered a massive 13–17 point accuracy drop when tested on the external GastroVision dataset. This highlights that "SOTA" numbers on curated benchmarks rarely translate directly to the real world.
Key Results
- Equivalence: The pooled difference between ResNet50 and ViT was only +0.43 points (90% CI -0.33 to +1.18), proving equivalence within the pre-defined 3-point margin ($p < 0.001$).
- Latency: ResNet50 achieved a single-image CPU latency of 69 ms compared to 177 ms for ViT-B/16.
- Throughput: On a GPU, ResNet50 processed 337 images/sec, while ViT-B/16 managed only 88 images/sec.
- Interpretability: Both models focused on lesions at roughly 3.6 times the rate of chance, but Grad-CAM revealed that ResNet50 also frequently "looked" at the ScopeGuide inset in the corner of images.
Limitations and Future Work
The authors acknowledge several limitations:
- Partial Independence: The external dataset (GastroVision) shares a hospital with the training set, meaning the domain shift might be even more severe in a truly independent multi-center trial.
- Patient-Level Splitting: The Kvasir dataset lacks patient identifiers, making it impossible to ensure that images from the same patient aren't split across training and testing sets (though they attempted to mitigate this with perceptual hashing).
- Qualitative Interpretability: While they used expert masks for polyps, other disease classes lacked ground-truth masks for quantitative explainability.
Future Directions: The authors call for prospective multi-center validation, training strategies that proactively mask shortcuts (like "Cutout" on artifacts), and the evaluation of quantized models on specific target medical hardware.
Summary updated