Abstract
The Segment Anything Model (SAM) has set new benchmarks in image segmentation but remains limited by its reliance on manual prompting, particularly in specialized fields like industrial anomaly detection and medical imaging. We introduce ODS-SAM, a framework that automates SAM through a self-generating prompt encoder. By integrating Omni-directional State Space Models (SSMs) for global context and dynamic convolutions for multi-scale local details, ODS-SAM achieves state-of-the-art performance across diverse benchmarks while keeping the core SAM weights frozen.
Introduction
While SAM excels at general-purpose segmentation, its application to anomaly detection faces three primary hurdles: the manual bottleneck of point/box prompts, the domain gap between natural and specialized images (e.g., X-rays), and feature blindness regarding multi-scale anomalous targets. Existing automated methods often fail to balance global scene understanding with fine-grained local detail. ODS-SAM addresses these by introducing a trainable prompt encoder that leverages the strengths of State Space Models and dynamic convolutions.
Methodology
The ODS-SAM architecture consists of a frozen SAM image encoder and mask decoder, bridged by a novel trainable Prompt Encoder.
SSMODC Module
The core innovation is the SSMODC (State-Space Model-based residual Omni-Dimensional module), which contains:
- Omni-directional State-Space Model (OD-SSM): Unlike traditional SSMs that process data unidirectionally, OD-SSM scans images in multiple directions simultaneously. This prevents the "forgetting" of information in the middle of sequences and ensures comprehensive global context.
- Omni-Dimensional Dynamic Convolution (OD-Conv): This component utilizes dynamic kernels with varying dilation rates to adaptively capture features at multiple scales, essential for detecting anomalies of different sizes.
By freezing the heavy SAM backbone and only training this lightweight prompt encoder, the system remains efficient and preserves SAM's inherent generalization capabilities.
Experimental Results
ODS-SAM was evaluated on industrial (MVTec AD) and medical (MoNuSeg, GlaS, Polyp) datasets.
Quantitative Performance
On the MVTec AD dataset, ODS-SAM achieved a mean Absolute Error (MAE) of 0.009, representing an 18.2% improvement over previous state-of-the-art methods. In medical imaging, the model showed a 3.68% increase in Dice score on the MoNuSeg dataset. Notably, ODS-SAM is computationally efficient, requiring only 53.9 GFLOPs compared to AutoSAM's 80.3 GFLOPs.
Qualitative Analysis
Qualitative results demonstrate the model's ability to precisely segment subtle defects that other models miss or over-segment.
Conclusion
ODS-SAM effectively bridges the gap between foundation models and specialized anomaly detection tasks. By combining the global reasoning of multi-directional SSMs with the local precision of dynamic convolutions, it provides a robust, automated solution for high-stakes imaging domains. Future work will explore alternative prompt modalities to further enhance precision in complex medical scenarios.
Summary updated