Abstract
Despite its clinical utility, medical image segmentation (MIS) remains a daunting task due to images' inherent complexity and variability. Vision transformers (ViTs) have recently emerged as a promising solution to improve MIS; however, they require larger training datasets than convolutional neural networks. To overcome this obstacle, data-efficient ViTs were proposed, but they are typically trained using a single source of data, which overlooks the valuable knowledge that could be leveraged from other available datasets. Naively combining datasets from different domains can result in negative knowledge transfer (NKT), i.e., a decrease in model performance on some domains with non-negligible inter-domain heterogeneity. In this paper, we propose MDViT, the first multi-domain ViT that includes domain adapters to mitigate data-hunger and combat NKT by adaptively exploiting knowledge in multiple small data resources (domains). Further, to enhance representation learning across domains, we integrate a mutual knowledge distillation paradigm that transfers knowledge between a universal network (spanning all the domains) and auxiliary domain-specific branches. Experiments on 4 skin lesion segmentation datasets show that MDViT outperforms state-of-the-art algorithms, with superior segmentation performance and a fixed model size, at inference time, even as more domains are added.
Introduction
Vision Transformers (ViTs) have revolutionized computer vision by modeling long-range dependencies more effectively than traditional Convolutional Neural Networks (CNNs). However, ViTs are notoriously "data-hungry" because they lack the structural assumptions—such as locality and translation equivariance—that CNNs possess. In the medical field, acquiring large, labeled datasets is exceptionally difficult due to privacy concerns and the need for expert annotation.
While multiple small datasets for the same task (e.g., skin lesion segmentation) may exist, they often exhibit significant inter-domain heterogeneity because they originate from different clinical settings using various scanners and protocols. Simply merging these datasets often leads to Negative Knowledge Transfer (NKT), where domain differences confuse the model, leading to worse performance than single-domain training. Existing multi-domain solutions often require separate models for each domain, which does not scale well. MDViT addresses these challenges by enabling a single transformer model to learn from multiple diverse datasets simultaneously.
Methodology
The MDViT architecture is designed to overcome data-hunger through two primary innovations: Domain Adapters (DA) and Mutual Knowledge Distillation (MKD).
1. Domain Adapter (DA)
Rather than adding heavy, domain-specific layers, MDViT exploits the existing structure of Multi-Head Self-Attention (MHSA). A lightweight DA module is inserted to "calibrate" attention heads based on a domain label. This allows the model to prioritize different features depending on the dataset's origin, effectively managing domain-specific characteristics without a massive increase in parameters.
2. Mutual Knowledge Distillation (MKD)
During training, MDViT employs a "Universal Network" that processes all data and several "Auxiliary Peers"—small, domain-specific branches. These components engage in mutual distillation: the peers provide expert knowledge on specific domains, while the universal network shares broader patterns learned across the entire data pool. Crucially, these auxiliary peers are discarded after training, ensuring the inference model size remains fixed regardless of the number of training domains.
Experimental Results
The researchers evaluated MDViT on four distinct skin lesion segmentation datasets: ISIC 2018 (2,594 images), Dermofit Image Library (1,212 images), Skin Cancer Detection (206 images), and PH2 (200 images). This mix of relatively large and very small datasets tested the model's ability to transfer knowledge to data-scarce domains.
MDViT consistently outperformed state-of-the-art data-efficient ViTs and other multi-domain learning methods. Key findings include:
- Significant Gains on Small Data: On the tiny SCD dataset, MDViT improved the Intersection over Union (IoU) by 10.16% compared to a baseline trained on that dataset alone.
- Superior Accuracy: The model achieved an average Dice score of 92.45% across all four domains, surpassing the joint-training baseline of 91.63%.
- NKT Mitigation: Unlike competing models that often saw performance dips on larger datasets (like ISIC) when multi-domain training was introduced, MDViT maintained or improved performance across the board.
Conclusion
MDViT represents a significant step forward in making Vision Transformers viable for medical imaging tasks where data is scarce and fragmented. By hacking the attention mechanism for domain adaptation and using mutual distillation, it solves the NKT paradox, allowing models to improve across all domains simultaneously. Future work will explore removing the requirement for explicit domain labels and extending the architecture to 3D medical data, such as MRI and CT volumes.
Summary updated