SAM Fewshot Finetuning for Anatomical Segmentation in Medical Images

Weiyi Xie, Nathalie Willems, Shubham Patil, Yang Li, Mayank KumarWACV 2024

TL;DR

This paper introduces a novel few-shot fine-tuning strategy for adapting the powerful Segment Anything Model (SAM) to anatomical segmentation in medical images. By reformulating SAM's mask decoder to accept "few-shot embeddings" derived from a small set of labeled examples, the method significantly reduces the need for manual annotation and achieves performance comparable to fully supervised methods with an order of magnitude less data.

What Problem Does This Paper Address?

Automated segmentation of anatomical structures in medical images (like CT and MRI) is crucial for accurate diagnosis and treatment planning. While deep learning models have made strides, they often require extensive task-specific training data, which is expensive and time-consuming to acquire, especially for volumetric medical scans. The recent Segment Anything Model (SAM) offers impressive zero-shot segmentation capabilities but struggles with medical images due to its 2D nature and the challenges of providing effective prompts (like points or bounding boxes) slice-by-slice for 3D volumes. Manually annotating these prompts is tedious and can lead to suboptimal segmentation, particularly for complex or ambiguous anatomical regions. This paper aims to bridge this gap by enabling SAM to perform accurate anatomical segmentation with minimal labeled data, overcoming the limitations of prompt-based approaches.

Methods

The core innovation lies in adapting SAM's existing architecture without introducing new components. The authors propose a "few-shot fine-tuning" strategy that focuses on retraining only the mask decoder of SAM. Instead of relying on user-provided prompts (points, boxes), the method leverages "few-shot embeddings" derived from a small set of labeled medical images (the "few-shot set").

Here's a breakdown:

  1. Image Encoding: SAM's pre-trained image encoder (a Vision Transformer) is used to extract image embeddings from the input medical images.
  2. Target Embedding Generation: For each anatomical label of interest, target embeddings are computed by averaging the image embeddings within the corresponding down-sampled segmentation mask from the few-shot set. This effectively captures the "essence" of the target anatomy.
  3. Mask Decoder Reformulation: The original SAM mask decoder, which typically uses prompt embeddings, is modified to accept these few-shot target embeddings as queries. The image embeddings serve as keys and values.
  4. Two-Way Transformer: The mask decoder's two-way transformer layers allow the few-shot target embeddings and image embeddings to attend to each other. This cross-attention mechanism helps the model learn correlations between the target anatomy's representation and its appearance in the image. Self-attention within the few-shot embeddings further refines their representation.
  5. Frozen Image Encoder: Crucially, the image encoder remains frozen throughout the fine-tuning process. This preserves SAM's powerful general image understanding capabilities and significantly reduces computational cost and training time.

This approach eliminates the need for manual prompt engineering during inference, making it more practical for medical applications.

Datasets

The study evaluates the proposed method on a combination of publicly available and internally collected datasets, covering six anatomical structures across CT and MRI modalities:

  • AMOS22: Includes Tibia and Femur (CT).
  • MSD (Medical Segmentation Decathlon): Includes Left Atrium (MRI).
  • Verse20: Includes Vertebrae (CT).
  • Internal Dataset: Includes Heart, Aorta, and Postcava (CT).

For evaluation, 3D scans were sampled into 2D slices along predefined axes, with specific spacing and intensity clipping parameters detailed in Table 1.

Key Insights and Ideas

  • Few-Shot Embeddings as Prompts: The core idea of using learned embeddings from a small labeled set as prompts for the mask decoder is a powerful way to adapt SAM without extensive retraining or manual annotation.
  • Efficiency is Key: By freezing the image encoder and only fine-tuning the mask decoder, the method achieves significant computational efficiency, making it practical for real-world deployment.
  • Beyond Manual Prompts: The proposed method bypasses the limitations of manual prompting (points, boxes) in medical imaging, which can be tedious, error-prone, and suboptimal for complex 3D structures.
  • Generalizability: The approach is presented as generic and applicable to any 2D/3D segmentation task, not just medical imaging.
  • SAM's Limitations in Medical Context: The paper highlights that even with bounding box prompts, SAM can struggle with pathological changes or anatomies that deviate from common appearances, as its image encoder might not capture these nuances without fine-tuning.

Key Results

The proposed few-shot fine-tuning method demonstrates strong performance:

  • Superior to Point Prompts: It significantly outperforms SAM using only point prompts, showing an improvement of approximately 50% in IoU.
  • Comparable to Fully Supervised Methods: The method achieves performance on par with the fully supervised nnU-Net, a state-of-the-art medical segmentation model, when using a small number of labeled images (e.g., 50).
  • Order of Magnitude Data Reduction: The most significant finding is that it requires an order of magnitude less labeled data than fully supervised methods to achieve comparable results. For instance, with just 50 labeled images, it matches or exceeds SAM with idealized bounding box prompts for several anatomies.
  • Qualitative Improvements: Qualitative results show that the fine-tuned model can better distinguish adjacent anatomies and handle variations compared to prompt-based SAM.

Limitations and Future Work

  • Image Encoder Limitations: The paper acknowledges that if the pre-trained SAM image encoder fails to capture sufficient anatomical semantics (especially for pathological changes), the fine-tuned model may still exhibit errors, such as over-segmentation of adjacent structures.
  • Vertebrae Segmentation Challenge: Fine-tuning with only 5 labeled images on the Verse20 dataset showed sub-optimal results for vertebrae segmentation. This is attributed to the challenge of capturing the full anatomical variation of vertebrae across different 2D coronal views with such a small sample.
  • Future Directions: The authors suggest that their few-shot SAM approach provides a generic framework that can be extended beyond medical image segmentation to other tasks like object detection and classification. Further research could explore more sophisticated ways to leverage the few-shot embeddings or investigate strategies to improve the image encoder's adaptability for specific medical domains.

Summary updated

BibTeX
@inproceedings{xie2024sam,
  author = {Weiyi Xie and Nathalie Willems and Shubham Patil and Yang Li and Mayank Kumar},
  title = {{SAM Fewshot Finetuning for Anatomical Segmentation in Medical Images}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure