Empirical Evaluation of the Segment Anything Model (SAM) for Brain Tumor Segmentation

Mohammad Peivandi, Jason Zhang, Michael T. Lu, Dongxiao Zhu, Zhifeng KouarXiv 2023 · DOI · arXiv

Abstract

Brain tumor segmentation presents a formidable challenge in the field of Medical Image Segmentation. While deep-learning models have been useful, human expert segmentation remains the most accurate method. The recently released Segment Anything Model (SAM) has opened up the opportunity to apply foundation models to this difficult task. However, SAM was primarily trained on diverse natural images. This makes applying SAM to biomedical segmentation, such as brain tumors with less defined boundaries, challenging. In this paper, we enhanced SAM's mask decoder using transfer learning with the Decathlon brain tumor dataset. We developed three methods to encapsulate the four-dimensional data into three dimensions for SAM. An on-the-fly data augmentation approach has been used with a combination of rotations and elastic deformations to increase the size of the training dataset. Two key metrics: the Dice Similarity Coefficient (DSC) and the Hausdorff Distance 95th Percentile (HD95), have been applied to assess the performance of our segmentation models. These metrics provided valuable insights into the quality of the segmentation results. In our evaluation, we compared this improved model to two benchmarks: the pretrained SAM and the widely used model, nnUNetv2. We find that the improved SAM shows considerable improvement over the pretrained SAM, while nnUNetv2 outperformed the improved SAM in terms of overall segmentation accuracy. Nevertheless, the improved SAM demonstrated slightly more consistent results than nnUNetv2, especially on challenging cases that can lead to larger Hausdorff distances. In the future, more advanced techniques can be applied in order to further improve the performance of SAM on brain tumor segmentation.

Introduction

Identifying the precise boundaries of brain tumors is a high-stakes clinical task. Traditionally, this requires expert neuroradiologists due to the irregular shapes and "fuzzy" boundaries characteristic of neuro-oncological pathologies. While specialized deep learning architectures like U-Net have become standard, the emergence of "foundation models" like Meta’s Segment Anything Model (SAM) presents a new paradigm: a generalist model capable of zero-shot segmentation across diverse domains.

However, SAM’s training on 11 million natural images (e.g., animals, everyday objects) does not naturally translate to the complexities of medical MRI. Furthermore, MRI data is multi-modal, often consisting of four distinct sequences (FLAIR, T1w, T1-Gd, T2w), whereas SAM is designed for 3-channel RGB input. This study evaluates strategies to adapt SAM for brain tumor segmentation, investigating whether a general-purpose vision model can be fine-tuned to match or exceed specialized medical benchmarks.

Methods

The research focused on an efficient adaptation of the SAM architecture. To minimize computational overhead, the image and prompt encoders were frozen, and only the mask decoder was fine-tuned using transfer learning.

Data Encapsulation and Augmentation

To bridge the gap between 4D MRI data and SAM’s 3-channel requirement, three "encapsulation" methods were tested:

  1. Combined: Mapping FLAIR, T1-Gd, and T2w modalities directly to the Red, Green, and Blue channels.
  2. Repeated: Repeating the T1-Gd modality across all three channels.
  3. 2D-PCA: Compressing the four modalities into three informative dimensions using Principal Component Analysis.
Architecture of the adapted SAM model showing the mapping of MRI modalities to RGB channels and the fine-tuning of the mask decoder.

The model was trained on the Medical Segmentation Decathlon (MSD) brain tumor dataset, comprising 484 3D multimodal images. 3D volumes were sliced into 2D axial planes and filtered to exclude slices with minimal tumor presence. To improve generalization, the team employed on-the-fly data augmentation, including rotations and elastic deformations, alongside Stochastic Weight Averaging (SWA) for weight stability.

Results

The "Improved SAM" was compared against the original Pretrained SAM and the specialized nnUNetv2 model using the Dice Similarity Coefficient (DSC) and the Hausdorff Distance 95th Percentile (HD95).

Segmentation Performance

The Improved SAM demonstrated significant gains over the base pretrained model. For Whole Tumor segmentation, the Improved SAM achieved a DSC of 0.88, compared to 0.75 for the Pretrained SAM. While nnUNetv2 maintained the highest overall accuracy (0.93 DSC), the Improved SAM showed a notable advantage in consistency.

Comparative performance (DSC) of Pretrained SAM, Improved SAM, and nnUNetv2 across different tumor regions.

Consistency and Robustness

Analysis of the box plots reveals that while nnUNetv2 has a higher median DSC, it also exhibits a wider spread of errors and more extreme outliers on challenging cases. In contrast, the Improved SAM produced more tightly clustered results, suggesting greater robustness to anatomical anomalies that can cause specialized models to fail significantly. Regarding input modalities, the "Combined" approach (FLAIR + T1-Gd + T2w) outperformed PCA-based compression, indicating that preserving the raw physical properties of distinct MRI sequences is more beneficial than mathematical dimensionality reduction.

Discussion and Conclusion

While SAM does not yet surpass specialized models like nnUNetv2 in absolute accuracy, this evaluation highlights its potential as a robust foundation for medical tasks. Key challenges remain, such as the model's struggle with disconnected tumor regions and an inherent bias toward natural image textures. Future work should explore more sophisticated modality fusion techniques and selective unfreezing of the image encoder to better capture medical-specific features. The results suggest that fine-tuning foundation models is a viable path toward creating reliable, consistent tools for clinical neuro-oncology.

Summary updated

Abstract

Brain tumor segmentation presents a formidable challenge in the field of Medical Image Segmentation. While deep-learning models have been useful, human expert segmentation remains the most accurate method. The recently released Segment Anything Model (SAM) has opened up the opportunity to apply foundation models to this difficult task. However, SAM was primarily trained on diverse natural images. This makes applying SAM to biomedical segmentation, such as brain tumors with less defined boundaries, challenging. In this paper, we enhanced SAM's mask decoder using transfer learning with the Decathlon brain tumor dataset. We developed three methods to encapsulate the four-dimensional data into three dimensions for SAM. An on-the-fly data augmentation approach has been used with a combination of rotations and elastic deformations to increase the size of the training dataset. Two key metrics: the Dice Similarity Coefficient (DSC) and the Hausdorff Distance 95th Percentile (HD95), have been applied to assess the performance of our segmentation models. These metrics provided valuable insights into the quality of the segmentation results. In our evaluation, we compared this improved model to two benchmarks: the pretrained SAM and the widely used model, nnUNetv2. We find that the improved SAM shows considerable improvement over the pretrained SAM, while nnUNetv2 outperformed the improved SAM in terms of overall segmentation accuracy. Nevertheless, the improved SAM demonstrated slightly more consistent results than nnUNetv2, especially on challenging cases that can lead to larger Hausdorff distances. In the future, more advanced techniques can be applied in order to further improve the performance of SAM on brain tumor segmentation.
BibTeX
@misc{peivandi2023empirical,
  author = {Mohammad Peivandi and Jason Zhang and Michael T. Lu and Dongxiao Zhu and Zhifeng Kou},
  title = {{Empirical Evaluation of the Segment Anything Model (SAM) for Brain Tumor Segmentation}},
  year = {2023},
  eprint = {2310.06162},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.06162},
}
Figure