SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation

Javier Gamazo Tejero, Moritz Schmid, Pablo Márquez Neila, Martin S. Zinkernagel, Sebastian Wolf, Raphael Sznitman

TL;DR

SAM-DA introduces a highly efficient, decoder-focused adapter for the Segment Anything Model (SAM) tailored for medical image segmentation. By training less than 1% of SAM’s total parameters (just 0.66M), this approach achieves state-of-the-art domain generalization and test-time adaptation, bypassing the prohibitive computational costs of full fine-tuning.

What Problem Does This Paper Address?

Foundation models like SAM have revolutionized natural image segmentation, but they frequently stumble when applied to medical imagery. Medical images often suffer from low contrast, blurred boundaries, and complex 3D structures that look nothing like typical natural image datasets.

Compounding this issue is domain shift: a model trained on MRI or OCT scans from one hospital (using a specific scanner or protocol) often performs poorly when deployed at another hospital using different equipment. While full end-to-end fine-tuning can fix this, it is computationally expensive and requires massive resources. Existing Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA or Med-SA typically focus on modifying the heavy vision encoder, which can inadvertently destroy the model's pre-trained generalization capabilities or require retraining the decoder from scratch.

Methods

The authors propose SAM-DA (SAM Decoder Adapter), a lightweight prompt-based adapter inspired by language model fine-tuning techniques (specifically LLaMA-Adapter).

Instead of modifying the massive Vision Transformer (ViT) encoder, SAM-DA leaves the encoder completely frozen. It strategically injects a small set of learnable adaptation prompts ($A_\ell$) into each layer of SAM’s mask decoder.

Illustration of the proposed SAM-DA architecture within the mask decoder layer.

As shown in Figure 2, the dense embeddings ($T_\ell$) from the decoder's cross-attention layer act as queries ($Q$), while the learnable adapter prompts ($A_\ell$) act as keys ($K$) and values ($V$) in an additional attention block. The resulting correction factor ($S_\ell$) is scaled by a learnable gating factor ($g_\ell$) and added back to the original embeddings. Crucially, $g_\ell$ is initialized to zero, ensuring that the adapter does not disrupt SAM's pre-trained capabilities at the start of training.

Datasets

The model was validated across three medical cross-device/site datasets and one natural image dataset:

  • Retouch: Retinal OCT volumes from three different devices. The authors used Spectralis as the source domain and Cirrus as the target domain.
  • MRI: Prostate MRI scans from multiple sites. BMC served as the source domain and UCL as the target.
  • WMH (White Matter Hyperintensities): Brain MRI scans, using UMC Utrecht as the source and NUHS Singapore as the target.
  • HQSeg-44k: A large-scale natural image dataset used to baseline the model against standard non-medical benchmarks.

Key Insights and Ideas

  • Decoder-Only Adaptation is a Superpower for Generalization: By keeping the encoder frozen, the model retains its high-quality, robust feature representations. The decoder adapter simply acts as a lightweight "translator" that maps these general features to specific medical contexts.
  • Less Parameters, Less Overfitting: With only 0.66 million trainable parameters (compared to 11.03 million for Med-SA and over 90 million for full fine-tuning), SAM-DA is highly regularized. This prevents the model from overfitting to small, site-specific medical datasets.
  • Zero-Gated Initialization is Essential: Initializing the gating factor to zero allows the model to smoothly transition from zero-shot inference to targeted adaptation without experiencing early training instability or catastrophic forgetting.

Key Results

SAM-DA consistently outperformed or matched competitive baselines while using a fraction of the parameters.

Domain Generalization

When evaluated on completely unseen target domains (testing the model's ability to generalize across different hospital scanners without seeing target data during training), SAM-DA showed substantial improvements:

  • On the Retouch (Cirrus) target domain, SAM-DA achieved an IoU of 70.2%, significantly outperforming LoRA (56.0%) and Med-SA (61.7%).
  • On the MRI (UCL) target domain, it reached 80.6% IoU, nearly matching full fine-tuning (80.9%) despite training 137x fewer parameters.

Test-Time Domain Adaptation (TTDA)

In the challenging TTDA setting—where the model must adapt to an unlabeled target image on the fly during inference—SAM-DA proved highly resilient against the noisy signals typical of unsupervised test-time adaptation.

Figure 4. Qualitative and quantitative results on Retouch and MRI datasets for the proposed model and LoRA. For reference, each image includes its IoU score after five TTDA iterations.

As illustrated in Figure 4, SAM-DA successfully refines the segmentation boundaries over 5 iterations, achieving higher IoU scores (e.g., 39.7% and 44.0% on Retouch) compared to LoRA, which struggled to capture the correct fluid regions.

Limitations and Future Work

The authors note that on massive natural image datasets like HQSeg-44k, encoder-focused PEFT methods still hold an advantage. Because HQSeg-44k contains over 44,000 training images, models with larger trainable parameter capacities (like Med-SA or LoRA) can absorb more information, whereas decoder-only approaches hit a performance ceiling.

Additionally, for the test-time adaptation experiments, the authors utilized a relatively simple entropy minimization algorithm. Future work could explore pairing the SAM-DA architecture with more sophisticated, state-of-the-art TTDA frameworks to push medical segmentation boundaries even further.

Summary updated

Abstract

This paper addresses the domain adaptation challenge for semantic segmentation in medical imaging. Despite the impressive performance of recent foundational segmentation models like SAM on natural images, they struggle with medical domain images. Beyond this, recent approaches that perform end-to-end fine-tuning of models are simply not computationally tractable. To address this, we propose a novel SAM adapter approach that minimizes the number of trainable parameters while achieving comparable performances to full fine-tuning. The proposed SAM adapter is strategically placed in the mask decoder, offering excellent and broad generalization capabilities and improved segmentation across both fully supervised and test-time domain adaptation tasks. Extensive validation on four datasets showcases the adapter's efficacy, outperforming existing methods while training less than 1% of SAM's total parameters.
BibTeX
@inproceedings{tejero2025sam,
  author = {Javier Gamazo Tejero and Moritz Schmid and Pablo Márquez Neila and Martin S. Zinkernagel and Sebastian Wolf and Raphael Sznitman},
  title = {{SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}

Loading related content...

Figure