Siamese Transition Masked Autoencoders as Uniform Unsupervised Visual Anomaly Detector

Haiming Yao, Xue Wang, Wenyong YuarXiv 2022 · DOI · arXiv

Abstract

Visual Anomaly Detection (VAD) has traditionally been fragmented across specialized domains such as industrial inspection, medical imaging, and video surveillance. We present ST-MAE (Siamese Transition Masked Autoencoder), a novel hybrid framework that provides a uniform solution for unsupervised VAD. By shifting the learning objective from simple pixel reconstruction to a "deep feature transition" task between Siamese encoders, the model learns robust prototypical patterns while effectively avoiding the common "identity mapping" pitfall. ST-MAE achieves state-of-the-art performance across multiple benchmarks, demonstrating high efficiency and broad applicability.

Introduction

Visual Anomaly Detection (VAD) involves identifying images or regions that deviate from a learned "normal" pattern. Historically, the field has been divided: industrial defect detection, video surveillance, and medical imaging often required entirely different architectures and methodologies.

A persistent challenge in reconstruction-based models, such as standard Autoencoders, is the "trivial shortcut" or identity mapping. Because these models are highly effective at reconstructing data, they often reconstruct anomalies just as well as normal data. This results in a negligible residual error, making it difficult to distinguish flaws from normal variations. Furthermore, pixel-level comparisons often lack the high-level semantic context required to differentiate meaningful anomalies from harmless noise or lighting changes.

Uniform application of ST-MAE across diverse domains including industrial defect detection, medical imaging, and video surveillance.

Methodology

ST-MAE addresses these challenges by combining Deep Convolutional Neural Networks (DCNNs) with Vision Transformers (ViTs) in a hybrid architecture. The framework consists of three primary modules:

1. Local Perceptual Semantics Representation (LPSR)

Rather than operating on raw pixels, the model extracts hierarchical features from a pre-trained DCNN (e.g., VGG19 or ResNet). These features are fused into a Pyramid-Perceptual Fusion Dense Feature (PFDF), providing a rich semantic foundation that captures both local textures and broader structural information.

2. Siamese Transition Masked Autoencoder (ST-MAE)

This core innovation moves beyond simple reconstruction. The feature map is divided into two disjoint subsets of patches, each processed by a separate "Siamese" encoder. Crucially, the model performs a latent representation transition: the encoded features of one subset are used to predict the positions and characteristics of the other subset during reassembly. This non-trivial self-supervisory task forces the model to learn the underlying global structure of "normal" data, preventing it from simply copying inputs to outputs.

3. Semantic Deep Feature Residual (SDFR)

A lightweight decoder reconstructs the original features from this "transitioned" latent space. Anomalies are detected by measuring the residual between the input features and the reconstructed features using both intensity loss (Euclidean distance) and orientational loss (cosine similarity).

Detailed architectural overview of the Siamese Transition Masked Autoencoder (ST-MAE) framework.

Experimental Results

The authors evaluated ST-MAE on a diverse array of benchmarks to prove its uniform applicability:

  • Industrial (MVTec AD): Achieved a state-of-the-art 97.3% Image-level AUC and 97.2% Pixel-level AUC, outperforming both reconstruction-based and embedding-based competitors.
  • Medical (Retinal-OCT): Reached a 94.6% AUC, significantly higher than traditional methods like MemAE (87.6%) or GANomaly (91.6%).
  • Video Surveillance (UCSD Ped2): Achieved 91.7% AUC, demonstrating the ability to handle complex crowd scenes without explicit temporal modeling.
  • Efficiency: Despite the use of Transformers, the base version of the model achieves high inference speeds of approximately 30 FPS (33.2 ms per image), making it viable for real-world deployment.

Key Insights and Conclusion

The success of ST-MAE stems from the paradigm shift of transition over reconstruction. By forcing the model to predict one half of the image's features using the other half, it must learn the global semantics of the data rather than memorizing local pixel values. Calculating residuals in high-dimensional feature space further enhances robustness against textural noise.

While the model performs exceptionally well, future work may involve incorporating explicit temporal information to further improve surveillance performance and exploring end-to-end training to reduce dependency on pre-trained DCNN backbones. Overall, ST-MAE represents a significant step toward a unified, efficient, and robust solution for visual anomaly detection across industries.

Summary updated

Abstract

Unsupervised visual anomaly detection conveys practical significance in many scenarios and is a challenging task due to the unbounded definition of anomalies. Moreover, most previous methods are application-specific, and establishing a unified model for anomalies across application scenarios remains unsolved. This paper proposes a novel hybrid framework termed Siamese Transition Masked Autoencoders(ST-MAE) to handle various visual anomaly detection tasks uniformly via deep feature transition. Concretely, the proposed method first extracts hierarchical semantics features from a pre-trained deep convolutional neural network and then develops a feature decoupling strategy to split the deep features into two disjoint feature patch subsets. Leveraging the decoupled features, the ST-MAE is developed with the Siamese encoders that operate on each subset of feature patches and perform the latent representations transition of two subsets, along with a lightweight decoder that reconstructs the original feature from the transitioned latent representation. Finally, the anomalous attributes can be detected using the semantic deep feature residual. Our deep feature transition scheme yields a nontrivial and semantic self-supervisory task to extract prototypical normal patterns, which allows for learning uniform models that generalize well for different visual anomaly detection tasks. The extensive experiments conducted demonstrate that the proposed ST-MAE method can advance state-of-the-art performance on multiple benchmarks across application scenarios with a superior inference efficiency, which exhibits great potential to be the uniform model for unsupervised visual anomaly detection.
BibTeX
@misc{yao2022siamese,
  author = {Haiming Yao and Xue Wang and Wenyong Yu},
  title = {{Siamese Transition Masked Autoencoders as Uniform Unsupervised Visual Anomaly Detector}},
  year = {2022},
  eprint = {2211.00349},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.00349},
}
Figure