SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation

Javier Gamazo Tejero, Moritz J Schmid, Pablo Márquez Neila, Martin Zinkernagel, Sebastian Wolf, Raphael Sznitman

TL;DR

This paper introduces SAM-DA, a novel adapter for the Segment Anything Model (SAM) that significantly reduces computational costs for medical image segmentation. By strategically placing a lightweight adapter within SAM's decoder, SAM-DA achieves comparable or superior performance to full fine-tuning on various medical datasets, while training less than 1% of SAM's parameters. This approach offers a computationally efficient solution for domain adaptation in medical imaging, a critical step for deploying AI models in real-world clinical settings.

What Problem Does This Paper Address?

The Segment Anything Model (SAM) has shown remarkable zero-shot segmentation capabilities on natural images. However, its performance drastically drops when applied to medical imaging datasets. This is due to inherent differences in medical images, such as lower contrast and irregular object morphologies, compared to natural images. While fine-tuning SAM on medical data can improve performance, full end-to-end fine-tuning is computationally prohibitive due to the massive size of models like SAM. Furthermore, medical imaging data often comes from diverse sources (different scanners, protocols), leading to domain shifts. Adapting models to these unseen domains is crucial for their practical deployment, but traditional fine-tuning for each new domain is also resource-intensive and requires re-certification. Existing Parameter-Efficient Fine-Tuning (PEFT) methods, while promising, haven't been extensively explored for SAM in the medical domain, especially considering both fully supervised and test-time adaptation scenarios.

Methods

The core contribution of this paper is the SAM Decoder Adapter (SAM-DA). Inspired by adapter techniques in Natural Language Processing (NLP), SAM-DA introduces a small, trainable module into SAM's mask decoder. Specifically, at each layer of the decoder's transformer block, SAM-DA adds a learnable adaptation prompt ($A_l$). This prompt, along with the existing dense embeddings ($T_l$) from the cross-attention operation, is fed into an additional attention block. Here, the dense embeddings act as queries, and the adaptation prompts act as keys and values. This process generates correction factors that modify the original embeddings without retraining the entire decoder. A learnable gating factor ($g_l$) is used to control the influence of the adapter, initialized to zero to ensure no disruption during early training.

The authors deliberately chose to place the adapter in the decoder for several reasons:

  1. The image encoder, being a pre-trained Vision Transformer (ViT), already generates high-quality representations.
  2. Modifying the encoder might necessitate retraining the mask decoder, defeating the purpose of parameter efficiency.
  3. Focusing on the decoder allows for minimal parameter updates, making it highly efficient and easy to train.

SAM-DA is designed to be agnostic to the specific test-time domain adaptation (TTDA) algorithm used, allowing it to be combined with various TTDA strategies.

Datasets

The paper evaluates SAM-DA on four datasets:

  • Retouch: A medical dataset of retinal OCT volumes from three devices (Cirrus, Spectralis, Topcon). The study uses "Spectralis" as the source and "Cirrus" as the target domain for domain adaptation experiments.
  • MRI: Prostate MRI scans from different sites and devices. "Boston Medical Center (BMC)" is used as the source and "University College London (UCL)" as the target.
  • WMH: Brain MRI scans from three sites (NUHS Singapore, UMC Utrecht, VU Amsterdam) focusing on White Matter Hyperintensities. "UMC Utrecht" and "NUHS Singapore" are used as source and target domains, respectively.
  • HQSeg-44K: A large dataset of natural images for fine-grained segmentation, comprising data from multiple sources for training and testing. This dataset is used for evaluating generalization without a clear domain shift.

Key Insights and Ideas

  • Decoder-centric adaptation is key for SAM in medical imaging: The paper strongly advocates for adapting SAM's decoder rather than its encoder for medical domain adaptation. This strategy leverages SAM's strong pre-trained encoder while efficiently fine-tuning the part responsible for generating segmentation masks.
  • Parameter efficiency unlocks new possibilities: By training less than 1% of SAM's parameters, SAM-DA makes domain adaptation computationally feasible, even for resource-constrained environments. This is crucial for the practical deployment of AI in healthcare.
  • Robustness in low-data regimes: SAM-DA shows significant performance improvements in low-data scenarios (like the BMC domain in the MRI dataset). This is attributed to its reduced susceptibility to overfitting, allowing it to retain more knowledge from SAM's pre-trained weights.
  • Test-Time Domain Adaptation (TTDA) benefits from lightweight adapters: The paper demonstrates that SAM-DA performs well in TTDA, where models adapt on a sample-by-sample basis. This is because the adapter's low parameter count prevents it from learning noisy or unrepresentative data characteristics, a common pitfall in unsupervised TTDA.

Key Results

  • Full Supervision: SAM-DA achieves comparable or superior IoU scores to state-of-the-art methods and full fine-tuning across medical datasets, despite training significantly fewer parameters. For instance, on the Retouch dataset, SAM-DA achieves 75.4±0.6 IoU, outperforming other adapters.
  • Domain Generalization: SAM-DA demonstrates statistically significant superiority in zero-shot generalization compared to encoder-focused adaptation methods like Med-SA and LoRA, showing performance on par with fine-tuning in low-data regimes.
  • Test-Time Domain Adaptation: SAM-DA outperforms other baselines in most TTDA scenarios, showing strong results even when compared to methods that might overfit to noisy test data.
  • Parameter Efficiency: SAM-DA trains with only 0.66M learnable parameters on the HQSeg-44K dataset, a fraction of the parameters trained by methods like Fine-Tuning (90.60M) or Med-SA (11.03M).

Limitations and Future Work

  • HQSeg-44K performance: While SAM-DA performs well, its IoU score on the large HQSeg-44K dataset (79.6) falls behind methods like LoRA and Med-SA (average 83.5). The authors suggest this might be due to the dataset's size, where encoder-adaptation methods with more parameters might have an advantage.
  • Comparison with HQ-SAM in TTDA: Due to the challenge of zero-initializing HQ-SAM, it was omitted from the TTDA experiments.
  • Future Work: The paper suggests exploring the impact of adapter placement within the encoder, though they deliberately chose the decoder for efficiency. Further investigation into the optimal number of adapter layers and their specific placement within the decoder could be beneficial. The authors also hint at the potential for combining SAM-DA with other TTDA algorithms for further improvements.

Summary updated

Abstract

This paper addresses the domain adaptation challenge for semantic segmentation in medical imaging. Despite the impressive performance of recent foundational segmentation models like SAM on natural images, they struggle with medical domain images. Beyond this, recent approaches that perform end-to-end fine-tuning of models are simply not computationally tractable. To address this, we propose a novel SAM adapter approach that minimizes the number of trainable parameters while achieving comparable performances to full fine-tuning. The proposed SAM adapter is strategically placed in the mask decoder, offering excellent and broad generalization capabilities and improved segmentation across both fully supervised and test-time domain adaptation tasks. Extensive validation on four datasets showcases the adapter's efficacy, outperforming existing methods while training less than 1% of SAM's total parameters.
BibTeX
@inproceedings{tejero2025sam,
  author = {Javier Gamazo Tejero and Moritz J Schmid and Pablo Márquez Neila and Martin Zinkernagel and Sebastian Wolf and Raphael Sznitman},
  title = {{SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure