Cheap Lunch for Medical Image Segmentation by Fine-Tuning SAM on Few Exemplars

Weijia Feng, Lingting Zhu, Lequan Yu

Introduction

The Segment Anything Model (SAM) represents a milestone in computer vision, having been trained on over a billion masks to segment objects in natural images with remarkable zero-shot capability. However, its application in medical imaging—such as CT and MRI scans—is hindered by distinct textures, low contrast, and complex anatomical structures absent from its training set. While fine-tuning is the standard remedy, the scarcity and high cost of expert-labeled medical data present a significant barrier. This research proposes a "cheap lunch" solution: a framework to fine-tune SAM using only a few expert-labeled "exemplars," achieving high-quality segmentation with minimal data and computational overhead.

Methodology: Efficient Adaptation

The proposed approach addresses the data and compute challenges through two primary innovations:

1. Exemplar-Guided Synthesis

To overcome the lack of labeled data, the authors utilize an automated synthesis module. Starting with just a few expert-labeled slices (exemplars), the module crops target organs or tumors and applies various geometric and intensity transformations (e.g., rotation, scaling, blurring). These augmented structures are then pasted onto unlabeled background images, effectively "hallucinating" a diverse training set that teaches the model the specific characteristics of the medical domain without requiring thousands of manual annotations.

Exemplars and synthesized images for brain tumors (top) and abdominal organs (bottom).

2. Parameter-Efficient Fine-Tuning with LoRA

Fine-tuning a massive model like SAM is computationally expensive. To make this accessible, the authors employ Low-Rank Adaptation (LoRA). By freezing the original weights and adding small, trainable "rank decomposition" matrices to the Transformer layers, they update only 6.32 million parameters—approximately 1.77% of the total model. This allows for training on consumer-grade hardware, such as an NVIDIA RTX 3090, significantly lowering the barrier for clinical research labs.

Experimental Results

The method was evaluated on two challenging benchmarks: the BraTS 2018 MRI dataset (brain tumors) and the Synapse CT dataset (multi-organ segmentation).

Performance on BraTS 2018

Zero-shot SAM performed poorly on brain tumor segmentation, with a Dice Similarity Score (DSC) of 45.29. By fine-tuning on just 150 slices (1% of the dataset), the proposed method boosted the DSC to 83.4, nearly reaching the "Pseudo Upper Bound" of 85.28 achieved by training on the entire dataset.

Performance on Synapse

The Synapse dataset is more complex, requiring the segmentation of eight abdominal organs. As shown in the table below, the proposed method using only 18 exemplars (one per volume) achieved a DSC of 83.04. This not only vastly outperformed zero-shot SAM (74.54) but also surpassed specialized medical models like Att-UNet (77.77) that were trained on the full dataset.

Quantitative comparison on the Synapse multi-organ CT dataset.

Discussion and Conclusion

The results demonstrate that SAM is a "quick learner" when properly aligned. The combination of data synthesis and LoRA allows SAM to adapt to the medical domain with as little as 1% of the data typically required. Notably, the synthesis strategy proved crucial; training on synthesized data yielded better generalization than training on raw exemplars alone.

While the current reliance on point prompts during inference mimics a radiologist's workflow, future work could explore fully automatic segmentation and more advanced synthesis techniques like Diffusion Models. Ultimately, this "cheap lunch" approach democratizes state-of-the-art medical AI, making it feasible for smaller institutions to deploy high-performance segmentation tools with minimal expert labeling effort.

Summary updated

Abstract

The Segment Anything Model (SAM) has demonstrated remarkable capabilities of scaled-up segmentation models, enabling zero-shot generalization across a variety of domains. By leveraging large-scale foundational models as pre-trained models, it is a natural progression to fine-tune SAM for specific domains to further enhance performances. However, the adoption of foundational models in the medical domain presents a challenge due to the difficulty and expense of labeling sufficient data for adaptation within hospital systems. In this paper, we introduce an efficient and practical approach for fine-tuning SAM using a limited number of exemplars, making it suitable for such scenarios. Our approach combines two established techniques from the literature: an exemplar-guided synthesis module and the widely recognized Low-Rank Adaptation (LoRA) fine-tuning strategy, serving as data-level and model-level attempts respectively. Interestingly, our empirical findings suggest that SAM can be effectively aligned within the medical domain even with few labeled data. We validate our approach through experiments on brain tumor segmentation (BraTS) and multi-organ CT segmentation (Synapse). The comprehensive results underscore the feasibility and effectiveness of such an approach, paving the way for the practical application of SAM in the medical domain.
BibTeX
@inproceedings{feng2023cheap,
  author = {Weijia Feng and Lingting Zhu and Lequan Yu},
  title = {{Cheap Lunch for Medical Image Segmentation by Fine-Tuning SAM on Few Exemplars}},
  booktitle = {BrainLes/SWITCH @ MICCAI},
  year = {2023},
  url = {https://openreview.net/forum?id=vpqtsOYl1l},
}
Figure