Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention

Omid Nejati Manzari, Hojat Asgariandehkordi, Taha Koleilat, Yiming Xiao, Hassan RivazarXiv 2025 · DOI · arXiv

TL;DR

MedViTV2 is a novel hybrid vision architecture that integrates Kolmogorov-Arnold Networks (KANs) and Dilated Neighborhood Attention (DiNA) into Transformers for robust medical image classification. It achieves state-of-the-art performance across 27 out of 29 benchmarks while being 44% more computationally efficient than its predecessor, effectively solving the "feature collapse" problem that plagues scaled-up hybrid models.

What Problem Does This Paper Address?

Computer-aided diagnosis (CAD) systems powered by deep learning have made massive strides in medical image analysis. However, standard architectures (like CNNs, Vision Transformers, and Mamba variants) are typically designed and validated on clean, curated datasets.

Real-world clinical settings are rarely that pristine. Multi-center clinical studies frequently suffer from image corruptions, noise, and artifacts arising from variations in imaging equipment across different manufacturers. Furthermore, earlier hybrid models like MedViTV1 suffer from poor scalability: as you increase the model size and parameter count, accuracy often drops due to feature collapse—a phenomenon where feature maps in deeper layers become highly redundant or entirely inactive.

The authors set out to design a generalized, robust, and highly efficient architecture capable of maintaining top-tier performance even when faced with severe, real-world image corruptions.

Methods

The authors introduce MedViTV2, a hierarchical hybrid network that balances local and global feature learning across four distinct stages. Its architecture features three primary innovations:

Figure 2: Overall architecture of the proposed Medical Vision Transformer (MedViTV2).
  1. KAN-Integrated Transformers: For the first time in medical vision, Kolmogorov-Arnold Network (KAN) layers replace traditional Multi-Layer Perceptrons (MLPs) in the feed-forward pathway of the Global Feature Perception (GFP) blocks. KANs learn activation functions directly on the edges (weights) rather than applying fixed activations to linear combinations. MedViTV2 utilizes an optimized variant called Reflectional Switch Activation Functions (RSWAFs) to capture complex non-linear structures efficiently.
  2. Dilated Neighborhood Attention (DiNA): Implemented within the Local Feature Perception (LFP) blocks, DiNA localizes self-attention to a pixel's nearest neighbors using a sliding window pattern. This grants linear time and space complexity while allowing the receptive field to expand exponentially, ensuring feature diversity and scaling stability.
  3. Hierarchical Hybrid Strategy (HHS): Instead of simply stacking more blocks linearly, the authors designed a specific alternating pattern ((LFP × N + GFP × 1) × L) across the network stages to find the mathematical sweet spot between local feature extraction and global context awareness.

Datasets

The architecture was rigorously evaluated across a massive suite of 29 experimental setups:

  • MedMNIST2D: 12 standardized datasets (e.g., PathMNIST, ChestMNIST, OCTMNIST) spanning 9 different imaging modalities and thousands of samples.
  • NonMNIST Datasets: 5 multi-center, real-world clinical datasets including PAD-UFES-20 (skin lesions via smartphones), CPN X-ray (COVID-19/pneumonia), Fetal-Planes-DB (ultrasound), Kvasir (endoscopy), and ISIC2018.
  • MedMNIST-C: A specialized benchmark extending MedMNIST to evaluate model robustness against 12 distinct types of simulated real-world image corruptions (e.g., blur, noise, brightness adjustments) across 5 severity levels.

Key Insights and Ideas

  • The Power of KANs over MLPs: Traditional MLPs struggle to separate true anatomical signals from background noise in small medical datasets. Because KANs decompose functions into learnable non-linear curves, they act as superior feature extractors that require fewer parameters to capture complex medical structures.
  • Mitigating Feature Collapse: Through feature cosine distance analysis, the authors proved that standard convolutional attention mechanisms tend to homogenize features in deep layers. Replacing them with sparse global attention (DiNA) combined with Local Feed-Forward Networks (LFFN) successfully preserves feature diversity during scaling.
  • Structured Stacking Beats Raw Scaling: Simply increasing the number of blocks in a single stage yields diminishing returns and rapid saturation. True performance gains come from an iterative, principled combination of low-frequency (LFP) and high-frequency (GFP) signal extractors.

Key Results

  • SOTA Dominance: MedViTV2 achieved state-of-the-art results in 27 out of 29 evaluation benchmarks.
  • Massive Efficiency Gains: The architecture is 44% more computationally efficient than MedViTV1. As shown in the accuracy-to-FLOPs trade-off graph, MedViTV2 variants deliver significantly higher accuracy at a fraction of the computational cost compared to baselines like ResNet and MedMamba.
Figure 1: Comparison between MedViTs (V1 and V2), MedMamba, and the baseline ResNets, in terms of Average Accuracy vs. FLOPs trade-off over all MedMNIST datasets. MedViTV2-T/S/L significantly improves average accu- racy by 2.6%, 2.5%, and 4.6%, respectively, compared to MedViTV1-T/S/L.
  • Robustness Under Corruption: On the corrupted MedMNIST-C benchmark, MedViTV2 achieved a striking 13.4% accuracy improvement over previous methods, demonstrating its readiness for messy, real-world clinical deployment. It also yielded improvements of 4.6% on MedMNIST and 5.8% on NonMNIST datasets.

Limitations and Future Work

Despite its strong performance against digital and noise corruptions, the authors noted that color corruptions still cause a noticeable performance drop for MedViTV2 (a vulnerability shared with many Vision Transformer architectures).

As a direction for future work, the team aims to design specialized architectural blocks to counteract color-based distribution shifts. They also plan to adapt the MedViTV2 backbone for more complex downstream tasks, specifically 3D medical imaging and medical image segmentation.

Summary updated

Abstract

Convolutional networks, transformers, hybrid models, and Mamba-based architectures have demonstrated strong performance across various medical image classification tasks. However, these methods were primarily designed to classify clean images using labeled data. In contrast, real-world clinical data often involve image corruptions that are unique to multi-center studies and stem from variations in imaging equipment across manufacturers. In this paper, we introduce the Medical Vision Transformer (MedViTV2), a novel architecture incorporating Kolmogorov-Arnold Network (KAN) layers into the transformer architecture for the first time, aiming for generalized medical image classification. We have developed an efficient KAN block to reduce computational load while enhancing the accuracy of the original MedViT. Additionally, to counteract the fragility of our MedViT when scaled up, we propose an enhanced Dilated Neighborhood Attention (DiNA), an adaptation of the efficient fused dot-product attention kernel capable of capturing global context and expanding receptive fields to scale the model effectively and addressing feature collapse issues. Moreover, a hierarchical hybrid strategy is introduced to stack our Local Feature Perception and Global Feature Perception blocks in an efficient manner, which balances local and global feature perceptions to boost performance. Extensive experiments on 17 medical image classification datasets and 12 corrupted medical image datasets demonstrate that MedViTV2 achieved state-of-the-art results in 27 out of 29 experiments with reduced computational complexity. MedViTV2 is 44\% more computationally efficient than the previous version and significantly enhances accuracy, achieving improvements of 4.6\% on MedMNIST, 5.8\% on NonMNIST, and 13.4\% on the MedMNIST-C benchmark.
BibTeX
@misc{manzari2025medical,
  author = {Omid Nejati Manzari and Hojat Asgariandehkordi and Taha Koleilat and Yiming Xiao and Hassan Rivaz},
  title = {{Medical Image Classification with KAN-Integrated Transformers and Dilated Neighborhood Attention}},
  year = {2025},
  eprint = {2502.13693},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.13693},
}
Figure