SAM-aware Test-time Adaptation for Universal Medical Image Segmentation

Jianghao Wu, Yicheng Wu, Yutong Xie, Wenjia Bai, You Zhang, Feilong Tang, Yulong Li, Imran Razzak, Daniel F Schmidt, Yasmeen George

TL;DR

SAM-TTA is a lightweight, label-free framework that adapts the Segment Anything Model (SAM) to medical images on-the-fly during inference. By combining a novel Bezier curve-based input transformation with self-supervised guidance from SAM’s own internal quality scores, it significantly outperforms state-of-the-art fine-tuned models on unseen medical datasets without requiring any new manual annotations.

What Problem Does This Paper Address?

The Segment Anything Model (SAM) has revolutionized general image segmentation, but it hits a wall in the clinical world. Medical images present two major hurdles:

  1. Input Mismatch: SAM expects three-channel RGB images, while most medical scans (like CT or MRI) are single-channel grayscale. Simply duplicating the grayscale channel ignores the color-opponent cues SAM was trained to use.
  2. Semantic Mismatch: Natural objects usually have sharp, well-defined edges. Medical targets (organs, tumors) often have fuzzy, low-contrast boundaries that lead to over- or under-segmentation.

While researchers have created "medicalized" versions of SAM (like MedSAM) through supervised fine-tuning, these models often overfit. They perform well on the specific types of scans they were trained on but struggle when faced with new imaging protocols or different patient populations—a classic "out-of-distribution" (OOD) generalization problem.

Methods

The authors propose SAM-aware Test-time Adaptation (SAM-TTA), which adjusts the model for each specific test image without needing ground-truth labels. It consists of two main pillars:

1. Self-adaptive Bezier Curve-based Transformation (SBCT)

Instead of a simple grayscale-to-RGB copy, SBCT uses three learnable cubic Bezier curves (one per output channel) to remap pixel intensities. This process involves only 12 learnable parameters, making it incredibly lightweight. It effectively "translates" grayscale scans into a pseudo-color space that aligns better with SAM’s pretrained features, enhancing structural contrast without introducing "hallucinations" or artifacts.

2. IoU-guided Multi-scale Adaptation (IMA)

SAM has an internal head that predicts the Intersection-over-Union (IoU) score—essentially its own "confidence" in a mask. SAM-TTA leverages this internal signal for self-supervision. The framework uses a teacher-student setup with LoRA (Low-Rank Adaptation) modules to:

  • Maximize Confidence: Update parameters to increase the predicted IoU score.
  • Enforce Consistency: Ensure that high-resolution and low-resolution mask predictions agree with each other.
  • Align Features: Match intermediate feature representations between the student and a stable teacher model (updated via Exponential Moving Average).
Overview of the SAM-TTA framework showing the SBCT and IMA components.

Datasets

The authors evaluated SAM-TTA on eight public medical image segmentation tasks to prove its "universal" capability:

  • Brain Tumors: Four subsets from BraTS2023 (Adult Glioma and Pediatric) using T2W and T2-FLAIR MRI.
  • Abdominal Organs: Pancreas segmentation from the AMOS 2022 dataset.
  • Lesions: Pancreatic cancer segmentation from the MSWAL dataset (a recent 2025 benchmark).
  • Endoscopy: Two color-image datasets for polyp segmentation (CVC-ColonDB and Kvasir-SEG).

Key Insights and Ideas

  • Restoring the "Quality Compass": The authors found that SAM’s internal IoU predictor is highly reliable for natural images but fails for grayscale medical scans. Interestingly, applying SBCT "restores" this reliability, allowing the model to accurately judge its own performance again.
  • Input vs. Semantic Alignment: Most adaptation methods focus only on the model's internal logic. This paper argues that fixing the input representation (via SBCT) is just as critical for bridging the domain gap.
  • TTA vs. Full Supervision: In out-of-distribution scenarios, adapting a general model at test-time can actually yield better results than using a model that was fully fine-tuned on a different medical dataset.

Key Results

SAM-TTA established a new state-of-the-art for medical TTA:

  • Grayscale Superiority: On six grayscale datasets, SAM-TTA achieved an average Dice score improvement of 4.8% over MedSAM and 7.4% over SAM-Med2D.
  • Boundary Accuracy: It significantly reduced the 95th percentile Hausdorff Distance (HD95), meaning the predicted boundaries were much closer to the actual anatomy.
  • Efficiency: The adaptation process is fast, adding only about 0.26 seconds of overhead per image compared to standard SAM inference.
Fig. 3. Visual comparison of segmentation results across multiple medical image datasets.

Limitations and Future Work

  • Oracle Prompts: Like many SAM studies, this work uses "oracle" bounding boxes derived from ground truth to prompt the model. In a real clinical workflow, these would need to be generated by a human or another automated system.
  • 2D vs. 3D: The current framework operates on 2D slices. While medical scans are often 3D volumes, SAM-TTA does not yet exploit volumetric context.
  • Backbone Limits: The method is still bounded by the representational power of the original SAM. If the base model fundamentally cannot "see" a very low-contrast structure, TTA can only help so much.

Future work aims to extend this framework to native 3D segmentation and integrate more diverse self-supervised signals to further reduce the need for manual prompts.

Summary updated

Abstract

Leveraging the Segment Anything Model (SAM) for medical image segmentation remains challenging due to its limited adaptability across diverse medical domains. Although fine-tuned variants, such as MedSAM, improve performance in scenarios similar to the training modalities or organs, they may lack generalizability to unseen data. To overcome this limitation, we propose SAM-aware Test-time Adaptation (SAM-TTA), a lightweight and flexible framework that preserves SAM's inherent generalization ability while enhancing segmentation accuracy for medical images. SAM-TTA tackles two major challenges: (1) input-level discrepancy caused by channel mismatches between natural and medical images, and (2) semantic-level discrepancy due to different object characteristics in natural versus medical images (e.g., with clear boundaries vs. ambiguous structures). To this end, we introduce two complementary components: a self-adaptive Bezier Curve-based Transformation (SBCT), which maps single-channel medical images into SAM-compatible three-channel images via a few learnable parameters to be optimized at test time; and IoU-guided Multi-scale Adaptation (IMA), which leverages SAM's intrinsic IoU scores to enforce high output confidence, dual-scale prediction consistency, and intermediate feature consistency, to improve semantic-level alignments. Extensive experiments on eight public medical image segmentation tasks, covering six grayscale and two color (endoscopic) tasks, demonstrate that SAM-TTA consistently outperforms state-of-the-art test-time adaptation methods. Notably, on six grayscale datasets, SAM-TTA even surpasses fully fine-tuned models, achieving significant Dice improvements (i.e., average 4.8% and 7.4% gains over MedSAM and SAM-Med2D) and establishing a new paradigm for universal medical image segmentation. Code is available at https://github.com/JianghaoWu/SAM-TTA.
BibTeX
@misc{wu2025sam,
  author = {Jianghao Wu and Yicheng Wu and Yutong Xie and Wenjia Bai and You Zhang and Feilong Tang and Yulong Li and Imran Razzak and Daniel F Schmidt and Yasmeen George},
  title = {{SAM-aware Test-time Adaptation for Universal Medical Image Segmentation}},
  year = {2025},
  eprint = {2506.05221},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05221},
}
Figure