Abstract
This paper evaluates the performance of Meta’s Segment Anything Model (SAM) on the complex task of multi-phase liver tumor segmentation. While SAM demonstrates impressive zero-shot capabilities, its performance currently lags behind specialized models like U-Net. However, the study identifies SAM as an exceptionally powerful tool for accelerating the manual medical data annotation process, potentially alleviating the data bottleneck in medical AI.
Introduction
In medical AI, data acquisition and annotation represent significant bottlenecks. Segmenting tumors in multi-phase Contrast-Enhanced Computed Tomography (CECT) is a labor-intensive task requiring expert radiologists. The Segment Anything Model (SAM) has promised a revolution in computer vision similar to Large Language Models in text, offering "zero-shot" segmentation. This study investigates whether a general-purpose model trained primarily on natural images can handle the nuances of multi-phase liver tumors, which change appearance across different circulatory phases.
Methodology
The researchers conducted a systematic investigation of SAM across several variables:
- Model Variants: Three versions based on Vision Transformer backbones: ViT-B (Base), ViT-L (Large), and ViT-H (Huge).
- Prompting Strategy: Accuracy was tested against the number of "prompt points" (clicks) provided: 1, 5, 10, and 20 points.
- Data Resolution: Images were resized to 224x224, 512x512, and 1024x1024 to evaluate the impact of resolution on performance.
- Multi-phase Aggregation: Since SAM is designed for single-phase inputs, results from different CT phases were combined using either average (avg) or maximum (max) response aggregation.
Performance was benchmarked against the standard medical imaging architecture, the U-Net.
Dataset
The study utilized an in-house Multi-phase Liver Tumor Segmentation (MPLiTS) dataset comprising 388 patients and 1,552 CECT volumes. Each scan included four phases: non-contrast (NC), arterial (ART), portal venous (PV), and delayed (DE). Ground truth annotations were provided by two experienced radiologists with 10 and 20 years of experience.
Key Results and Insights
Quantitative analysis revealed several critical findings regarding SAM's current limitations and strengths in the medical domain:
- The Performance Gap: A standard U-Net achieved a Dice Global Score (DGS) of approximately 0.80. In contrast, the best-performing SAM configuration (ViT-H, 20 prompts, max aggregation) reached a DGS of 0.76. With only one prompt point, SAM’s performance dropped significantly to between 0.25 and 0.41.
- The Resolution Paradox: Surprisingly, higher image resolutions did not consistently improve performance; the 224x224 resolution often outperformed the 1024x1024 setting, suggesting SAM's internal features may be optimized for specific scales.
- Prompt Dependency: The number of prompt points was the primary driver of performance. The advantage of the larger ViT-H encoder over ViT-B only became apparent with a high number of prompts (e.g., 20 points).
Conclusion and Future Work
SAM is currently a "natural image" generalist that struggles to match specialized medical models like U-Net, largely because it lacks an inherent understanding of 3D relationships between CT slices or the physiological relationships between contrast phases. However, its ability to capture the general morphology of tumors makes it a powerful interactive assistant for radiologists.
Future research should focus on domain adaptation through fine-tuning on medical datasets, developing native multi-phase integration methods, and refining SAM's role as an interactive tool in clinical workflows.
Summary updated