Introduction
Medical Image Segmentation (MIS) is a cornerstone of modern computer-aided diagnosis. However, it faces a persistent technical challenge: the trade-off between local detail and global context. Traditional Convolutional Neural Networks (CNNs) are proficient at extracting local textures and edges but lack the receptive field to understand long-range spatial relationships. Conversely, Vision Transformers (ViTs) utilize self-attention to capture global context but are notoriously "data-hungry" and often lose fine-grained spatial information necessary for precise boundary delineation.
The DDViT (Double-Level Fusion Domain Adapter Vision Transformer) architecture addresses these limitations by integrating CNNs for local detail with ViTs for global context, further enhanced by a specialized Domain Adapter to mitigate data scarcity issues.
Methodology
The DDViT framework is built upon three core components designed to optimize feature extraction and fusion:
- Hybrid Encoder (CNN + TransFuse+DA): The model employs a dual-pathway encoder. A CNN branch extracts high-resolution local features, while a TransFuse-based ViT branch captures global dependencies. Skip connections facilitate the transfer of spatial information from the CNN to the Transformer branch, preventing detail loss.
- Domain Adapter (DA): To combat the data-hunger of standard Transformers, a plug-in Domain Adapter module is introduced. By utilizing a domain label vector, the DA generates domain-aware attention, allowing the model to adapt more efficiently to specific medical imaging domains with limited training data.
- Double-Level Fusion (DLF): In the decoding stage, the DLF module employs a cross-attention mechanism to fuse the smallest (global) and largest (local) feature maps. This ensures the final segmentation mask benefits from both high-level semantic understanding and low-level spatial precision.
Experimental Setup
The model was evaluated using the EBHI-Seg (Enteroscope Biopsy Histopathological Hematoxylin and Eosin Image) dataset. This dataset contains 4,452 images categorized into six pathological stages of colorectal cancer: Normal, Polyp, High-grade Intraepithelial Neoplasia (IN), Low-grade IN, Adenocarcinoma, and Serrated adenoma. The experimental setup involved 890 images for training, 890 for validation, and the remainder for testing.
Results and Discussion
The performance of DDViT was benchmarked against established models, including Seg-Net, U-Net, and TransFuse, using the Dice Ratio as the primary evaluation metric.
As shown in the table above, DDViT achieved state-of-the-art performance in four out of the six pathological categories. Notably, it reached a Dice Ratio of 0.951 for Serrated adenoma and 0.937 for Low-grade IN. While Seg-Net and U-Net showed strengths in "Normal" and "Polyp" categories respectively, DDViT demonstrated superior consistency across the more complex pathological stages of tumor differentiation. The inclusion of the Domain Adapter and DLF modules successfully mitigated the performance degradation typically observed in Transformers when trained on smaller datasets.
Conclusion
DDViT represents a significant step forward in hybrid architectures for medical image segmentation. By effectively combining the strengths of CNNs and ViTs through domain-aware adaptation and multi-scale fusion, the model achieves robust performance even with limited data. Future work will explore the generalizability of this framework across other medical imaging modalities, such as MRI and CT scans, to further validate its clinical utility.
Summary updated