Accuracy Is Not Enough: An Efficiency- and Statistics-Aware Benchmark of ResNet50 and Vision Transformer for Explainable Multi-Class Gastrointestinal Diseases Classification

Akwasi AsarearXiv 2026 · arXiv

TL;DR

This paper challenges the "accuracy-first" mindset in medical AI by demonstrating that ResNet50 and Vision Transformers (ViT) achieve statistically equivalent accuracy for gastrointestinal disease classification, yet differ vastly in computational efficiency. The authors reveal that high-performing models often rely on "shortcuts"—non-clinical artifacts like endoscope overlays—and warn that significant performance drops on external datasets make no current model truly deployment-ready without local validation.

What Problem Does This Paper Address?

In the race to achieve state-of-the-art (SOTA) results, researchers often select deep learning architectures based solely on marginal gains in accuracy. However, in clinical settings—especially resource-constrained ones—computational efficiency, statistical reliability, and interpretability are just as critical.

The authors address three main gaps:

  1. The Accuracy Trap: Small differences in accuracy between CNNs and Transformers might not be statistically significant or clinically meaningful.
  2. The Efficiency Gap: The practical cost of running these models on hospital hardware (CPUs and older GPUs) is rarely profiled.
  3. Shortcut Learning: Models might "cheat" by learning to recognize acquisition artifacts (like green screen insets) rather than actual pathology, leading to a false sense of security.

Methods

The study provides a controlled, reproducible benchmark comparing three architectures: ResNet50 (a standard CNN), ViT-B/16 (a standard Transformer), and MobileViT-S (a hybrid "edge-optimized" model).

Key methodological highlights include:

  • Statistical Equivalence Testing: Instead of just looking for a "significant difference," the authors used a Two One-Sided Tests (TOST) procedure to prove that ResNet50 and ViT are equivalent within a $\pm3$ percentage-point margin.
  • Standardized Profiling: They measured MACs (multiply-accumulate operations), parameters, and inference latency on both a Tesla T4 GPU and an Intel Xeon CPU.
  • Robustness Checks: An occlusion experiment was conducted where the "ScopeGuide" inset (a common artifact in endoscopic images) was blacked out to see if model accuracy collapsed.
  • Multi-method Interpretability: They used Grad-CAM, LIME, SHAP, and attention rollout to visualize where models were looking, validating these against expert-drawn polyp masks.
Overview of the benchmarking pipeline

Datasets

  • Kvasir v1: The primary dataset containing 4,000 endoscopic images across 8 classes (e.g., polyps, esophagitis, ulcerative colitis). The authors performed rigorous near-duplicate screening to prevent data leakage.
  • GastroVision: Used as an external validation set (1,998 images). This dataset is "partially independent" as it shares one hospital source with Kvasir but includes data from a different country (Sweden).
  • Kvasir-SEG: Used specifically for lesion-level localization analysis, providing 500 unseen polyp images with expert-verified pixel masks.
Table 1: Class distribution and accessible provenance metadata of the Kvasir dataset used in this study (no EXIF metadata were present in any image).

Key Insights and Ideas

  • Accuracy is a Wash: In cross-validation, ResNet50 (92.12%) and ViT-B/16 (91.70%) were statistically equivalent. The choice between them shouldn't be based on accuracy.
  • Efficiency is the Real Differentiator: ResNet50 is the clear winner for deployment. It has 3.6x fewer parameters, 4.3x fewer operations, and runs 2.5x faster on a CPU. It also reaches its best performance four times faster during training.
  • The Danger of "Edge" Models: MobileViT-S was the most accurate on the internal test set (95.5%) but proved to be the most fragile. It relied heavily on the "ScopeGuide" artifact; when that artifact was removed, its accuracy dropped by nearly 10 points.
  • The External Reality Check: All models suffered a massive 13–17 point accuracy drop when tested on the external GastroVision dataset. This highlights that "SOTA" numbers on curated benchmarks rarely translate directly to the real world.

Key Results

  • Equivalence: The pooled difference between ResNet50 and ViT was only +0.43 points (90% CI -0.33 to +1.18), proving equivalence within the pre-defined 3-point margin ($p < 0.001$).
  • Latency: ResNet50 achieved a single-image CPU latency of 69 ms compared to 177 ms for ViT-B/16.
  • Throughput: On a GPU, ResNet50 processed 337 images/sec, while ViT-B/16 managed only 88 images/sec.
  • Interpretability: Both models focused on lesions at roughly 3.6 times the rate of chance, but Grad-CAM revealed that ResNet50 also frequently "looked" at the ScopeGuide inset in the corner of images.

Limitations and Future Work

The authors acknowledge several limitations:

  • Partial Independence: The external dataset (GastroVision) shares a hospital with the training set, meaning the domain shift might be even more severe in a truly independent multi-center trial.
  • Patient-Level Splitting: The Kvasir dataset lacks patient identifiers, making it impossible to ensure that images from the same patient aren't split across training and testing sets (though they attempted to mitigate this with perceptual hashing).
  • Qualitative Interpretability: While they used expert masks for polyps, other disease classes lacked ground-truth masks for quantitative explainability.

Future Directions: The authors call for prospective multi-center validation, training strategies that proactively mask shortcuts (like "Cutout" on artifacts), and the evaluation of quantized models on specific target medical hardware.

Summary updated

Abstract

Automated classification of gastrointestinal (GI) findings in endoscopic images could support early diagnosis, but architectures are often chosen on accuracy alone. We present a controlled, reproducible benchmark of ResNet50 and ViT-B/16 for eight-class GI classification on the public Kvasir dataset (4,000 images), using a stratified split (400 test images) and stratified group 5-fold cross-validation over all images. In cross-validation, ResNet50 and ViT-B/16 reached 92.12% and 91.70% accuracy, and a two one-sided tests (TOST) procedure showed equivalence within a +/-3 percentage-point margin (difference +0.43 points, 90% CI -0.33 to +1.18; p < 0.001). The architectures differed mainly in cost: ResNet50 had 3.6 times fewer parameters and 4.3 times fewer operations, 2.5-fold lower single-image CPU latency (69 vs. 177 ms) and 3.8-fold higher GPU throughput. Cross-validated calibration was similar (expected calibration error 0.021 vs. 0.023). An edge-oriented transformer, MobileViT-S, was most accurate on the test split (95.5%) but lost 9.4 points when an acquisition artefact present in some classes (the ScopeGuide inset) was occluded, whereas ResNet50 was not significantly affected. On 500 unseen polyp images with expert masks, both models' Grad-CAM maps focused on the lesion (3.6 times chance). On a de-duplicated external subset of GastroVision (1,998 images), accuracy fell to 78-80% for all three models and differences between them remained small. Accuracy differences between architectures were thus small relative to the effects of shortcut learning and domain shift, whereas efficiency differences were large. An efficient CNN is a defensible default for resource-constrained screening, but no model is deployment-ready without local validation.
BibTeX
@misc{asare2026accuracy,
  author = {Akwasi Asare},
  title = {{Accuracy Is Not Enough: An Efficiency- and Statistics-Aware Benchmark of ResNet50 and Vision Transformer for Explainable Multi-Class Gastrointestinal Diseases Classification}},
  year = {2026},
  eprint = {2610.33965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33965},
}
Figure