Omni-Dimensional State Space Model-driven SAM for Pixel-level Anomaly Detection

Chao Huang, Qianyi Li, Jie Wen, Bob ZhangIJCAI 2025 · DOI

Abstract

The Segment Anything Model (SAM) has set new benchmarks in image segmentation but remains limited by its reliance on manual prompting, particularly in specialized fields like industrial anomaly detection and medical imaging. We introduce ODS-SAM, a framework that automates SAM through a self-generating prompt encoder. By integrating Omni-directional State Space Models (SSMs) for global context and dynamic convolutions for multi-scale local details, ODS-SAM achieves state-of-the-art performance across diverse benchmarks while keeping the core SAM weights frozen.

Introduction

While SAM excels at general-purpose segmentation, its application to anomaly detection faces three primary hurdles: the manual bottleneck of point/box prompts, the domain gap between natural and specialized images (e.g., X-rays), and feature blindness regarding multi-scale anomalous targets. Existing automated methods often fail to balance global scene understanding with fine-grained local detail. ODS-SAM addresses these by introducing a trainable prompt encoder that leverages the strengths of State Space Models and dynamic convolutions.

Methodology

The ODS-SAM architecture consists of a frozen SAM image encoder and mask decoder, bridged by a novel trainable Prompt Encoder.

The ODS-SAM architecture featuring the SSMODC module within the Prompt Encoder.

SSMODC Module

The core innovation is the SSMODC (State-Space Model-based residual Omni-Dimensional module), which contains:

  • Omni-directional State-Space Model (OD-SSM): Unlike traditional SSMs that process data unidirectionally, OD-SSM scans images in multiple directions simultaneously. This prevents the "forgetting" of information in the middle of sequences and ensures comprehensive global context.
  • Omni-Dimensional Dynamic Convolution (OD-Conv): This component utilizes dynamic kernels with varying dilation rates to adaptively capture features at multiple scales, essential for detecting anomalies of different sizes.

By freezing the heavy SAM backbone and only training this lightweight prompt encoder, the system remains efficient and preserves SAM's inherent generalization capabilities.

Experimental Results

ODS-SAM was evaluated on industrial (MVTec AD) and medical (MoNuSeg, GlaS, Polyp) datasets.

Quantitative Performance

On the MVTec AD dataset, ODS-SAM achieved a mean Absolute Error (MAE) of 0.009, representing an 18.2% improvement over previous state-of-the-art methods. In medical imaging, the model showed a 3.68% increase in Dice score on the MoNuSeg dataset. Notably, ODS-SAM is computationally efficient, requiring only 53.9 GFLOPs compared to AutoSAM's 80.3 GFLOPs.

Qualitative Analysis

Qualitative results demonstrate the model's ability to precisely segment subtle defects that other models miss or over-segment.

Qualitative comparison of ODS-SAM (Ours) against SOTA methods on industrial and texture datasets.

Conclusion

ODS-SAM effectively bridges the gap between foundation models and specialized anomaly detection tasks. By combining the global reasoning of multi-directional SSMs with the local precision of dynamic convolutions, it provides a robust, automated solution for high-stakes imaging domains. Future work will explore alternative prompt modalities to further enhance precision in complex medical scenarios.

Summary updated

BibTeX
@inproceedings{huang2025omni,
  author = {Chao Huang and Qianyi Li and Jie Wen and Bob Zhang},
  title = {{Omni-Dimensional State Space Model-driven SAM for Pixel-level Anomaly Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/129},
}
Figure