DB-SAM: Delving into High Quality Universal Medical Image Segmentation

Chao Qin, Jiale Cao, Huazhu Fu, Fahad Shahbaz Khan, Rao Muhammad Anwer

Introduction

The Segment Anything Model (SAM) has set a new standard for zero-shot segmentation in natural image domains. However, its direct application to medical imaging—characterized by distinct textures, lower signal-to-noise ratios, and the necessity for high-precision local boundaries—has been limited by a significant "domain gap." In the paper "DB-SAM: Delving into High Quality Universal Medical Image Segmentation," Qin et al. propose a dual-branch architecture designed to bridge this gap by integrating global context with local medical-specific features.

The DB-SAM Architecture

While previous adaptations like MedSAM focused primarily on fine-tuning the decoder, DB-SAM argues that medical-specific knowledge must be injected directly into the encoding process. The core innovation is the Dual-Branch Encoder, which processes images through two parallel paths:

  • The ViT Branch: This utilizes the original, frozen SAM Vision Transformer (ViT) backbone. To make it "medical-aware," the authors insert learnable Channel Attention Blocks after each frozen attention layer. These blocks capture high-level, domain-specific features while preserving the pre-trained weights.
  • The Convolution Branch: A lightweight CNN branch extracts shallow, local features. This provides the "locality inductive bias" that ViTs often lack, ensuring the model captures fine anatomical details.

To synchronize these branches, the authors introduced two fusion mechanisms: Bilateral Cross-Attention (BCA), which allows the branches to exchange information during encoding, and a ViT-Conv Fusion Block that dynamically weights global versus local information for the final output.

The overall architecture of DB-SAM (a) alongside detailed views of the Channel Attention (b), Bilateral Cross-Attention (c), and ViT-Conv Fusion (d) blocks.

Experimental Evaluation

The authors evaluated DB-SAM on a massive, heterogeneous collection of 30 public medical datasets, covering:

  • 21 3D Tasks: Including brain, heart, and liver segmentation across MR, CT, and PET modalities.
  • 9 2D Tasks: Including endoscopy (polyps), ultrasound (breast tumors), X-ray (lungs), and pathology (glands).
  • Scale: The training set contained over 161,000 images, with a test set of over 52,000 images.

Results and Discussion

DB-SAM sets a new benchmark for universal medical image segmentation, significantly outperforming both vanilla SAM and MedSAM.

  • Quantitative Gains: In 3D tasks, DB-SAM achieved an 8.8% absolute gain in Normalized Surface Distance (NSD) and a 6.03% gain in Dice Similarity Coefficient (DSC) compared to MedSAM.
  • Qualitative Superiority: Visualizations demonstrate that DB-SAM is significantly more effective at capturing the intricate outlines of small organs and complex tumor shapes where previous models produced "blurry" or incomplete masks.
Qualitative comparison showing DB-SAM's (Ours) superior ability to capture fine anatomical details compared to SAM and MedSAM.

Conclusion

DB-SAM represents a major step forward in adapting foundation models for clinical use. By combining the global understanding of ViTs with the local precision of CNNs through an efficient adapter-based approach, it achieves state-of-the-art performance without the prohibitive cost of full-model retraining. Future work will focus on reducing dependency on manual prompts and exploring native 3D volumetric adapters.

Reference: Qin, C., et al. (2024). DB-SAM: Delving into High Quality Universal Medical Image Segmentation. arXiv:2410.04172

Summary updated

Abstract

Recently, the Segment Anything Model (SAM) has demonstrated promising segmentation capabilities in a variety of downstream segmentation tasks. However in the context of universal medical image segmentation there exists a notable performance discrepancy when directly applying SAM due to the domain gap between natural and 2D/3D medical data. In this work, we propose a dual-branch adapted SAM framework, named DB-SAM, that strives to effectively bridge this domain gap. Our dual-branch adapted SAM contains two branches in parallel: a ViT branch and a convolution branch. The ViT branch incorporates a learnable channel attention block after each frozen attention block, which captures domain-specific local features. On the other hand, the convolution branch employs a light-weight convolutional block to extract domain-specific shallow features from the input medical image. To perform cross-branch feature fusion, we design a bilateral cross-attention block and a ViT convolution fusion block, which dynamically combine diverse information of two branches for mask decoder. Extensive experiments on large-scale medical image dataset with various 3D and 2D medical segmentation tasks reveal the merits of our proposed contributions. On 21 3D medical image segmentation tasks, our proposed DB-SAM achieves an absolute gain of 8.8%, compared to a recent medical SAM adapter in the literature. The code and model are available at https://github.com/AlfredQin/DB-SAM.
BibTeX
@inproceedings{qin2024db,
  author = {Chao Qin and Jiale Cao and Huazhu Fu and Fahad Shahbaz Khan and Rao Muhammad Anwer},
  title = {{DB-SAM: Delving into High Quality Universal Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  url = {https://openreview.net/forum?id=3umqKox4Id},
}
Figure