TL;DR
SAM-DA introduces a highly efficient, decoder-focused adapter for the Segment Anything Model (SAM) tailored for medical image segmentation. By training less than 1% of SAM’s total parameters (just 0.66M), this approach achieves state-of-the-art domain generalization and test-time adaptation, bypassing the prohibitive computational costs of full fine-tuning.
What Problem Does This Paper Address?
Foundation models like SAM have revolutionized natural image segmentation, but they frequently stumble when applied to medical imagery. Medical images often suffer from low contrast, blurred boundaries, and complex 3D structures that look nothing like typical natural image datasets.
Compounding this issue is domain shift: a model trained on MRI or OCT scans from one hospital (using a specific scanner or protocol) often performs poorly when deployed at another hospital using different equipment. While full end-to-end fine-tuning can fix this, it is computationally expensive and requires massive resources. Existing Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA or Med-SA typically focus on modifying the heavy vision encoder, which can inadvertently destroy the model's pre-trained generalization capabilities or require retraining the decoder from scratch.
Methods
The authors propose SAM-DA (SAM Decoder Adapter), a lightweight prompt-based adapter inspired by language model fine-tuning techniques (specifically LLaMA-Adapter).
Instead of modifying the massive Vision Transformer (ViT) encoder, SAM-DA leaves the encoder completely frozen. It strategically injects a small set of learnable adaptation prompts ($A_\ell$) into each layer of SAM’s mask decoder.
As shown in Figure 2, the dense embeddings ($T_\ell$) from the decoder's cross-attention layer act as queries ($Q$), while the learnable adapter prompts ($A_\ell$) act as keys ($K$) and values ($V$) in an additional attention block. The resulting correction factor ($S_\ell$) is scaled by a learnable gating factor ($g_\ell$) and added back to the original embeddings. Crucially, $g_\ell$ is initialized to zero, ensuring that the adapter does not disrupt SAM's pre-trained capabilities at the start of training.
Datasets
The model was validated across three medical cross-device/site datasets and one natural image dataset:
- Retouch: Retinal OCT volumes from three different devices. The authors used Spectralis as the source domain and Cirrus as the target domain.
- MRI: Prostate MRI scans from multiple sites. BMC served as the source domain and UCL as the target.
- WMH (White Matter Hyperintensities): Brain MRI scans, using UMC Utrecht as the source and NUHS Singapore as the target.
- HQSeg-44k: A large-scale natural image dataset used to baseline the model against standard non-medical benchmarks.
Key Insights and Ideas
- Decoder-Only Adaptation is a Superpower for Generalization: By keeping the encoder frozen, the model retains its high-quality, robust feature representations. The decoder adapter simply acts as a lightweight "translator" that maps these general features to specific medical contexts.
- Less Parameters, Less Overfitting: With only 0.66 million trainable parameters (compared to 11.03 million for Med-SA and over 90 million for full fine-tuning), SAM-DA is highly regularized. This prevents the model from overfitting to small, site-specific medical datasets.
- Zero-Gated Initialization is Essential: Initializing the gating factor to zero allows the model to smoothly transition from zero-shot inference to targeted adaptation without experiencing early training instability or catastrophic forgetting.
Key Results
SAM-DA consistently outperformed or matched competitive baselines while using a fraction of the parameters.
Domain Generalization
When evaluated on completely unseen target domains (testing the model's ability to generalize across different hospital scanners without seeing target data during training), SAM-DA showed substantial improvements:
- On the Retouch (Cirrus) target domain, SAM-DA achieved an IoU of 70.2%, significantly outperforming LoRA (56.0%) and Med-SA (61.7%).
- On the MRI (UCL) target domain, it reached 80.6% IoU, nearly matching full fine-tuning (80.9%) despite training 137x fewer parameters.
Test-Time Domain Adaptation (TTDA)
In the challenging TTDA setting—where the model must adapt to an unlabeled target image on the fly during inference—SAM-DA proved highly resilient against the noisy signals typical of unsupervised test-time adaptation.
As illustrated in Figure 4, SAM-DA successfully refines the segmentation boundaries over 5 iterations, achieving higher IoU scores (e.g., 39.7% and 44.0% on Retouch) compared to LoRA, which struggled to capture the correct fluid regions.
Limitations and Future Work
The authors note that on massive natural image datasets like HQSeg-44k, encoder-focused PEFT methods still hold an advantage. Because HQSeg-44k contains over 44,000 training images, models with larger trainable parameter capacities (like Med-SA or LoRA) can absorb more information, whereas decoder-only approaches hit a performance ceiling.
Additionally, for the test-time adaptation experiments, the authors utilized a relatively simple entropy minimization algorithm. Future work could explore pairing the SAM-DA architecture with more sophisticated, state-of-the-art TTDA frameworks to push medical segmentation boundaries even further.
Summary updated