One-for-All: Proposal Masked Cross-Class Anomaly Detection

Xincheng Yao, Chongyang Zhang, Ruoqi Li, Jun Sun, Zhenyu LiuAAAI 2023 · OpenReview

TL;DR

This paper introduces PMAD, a novel anomaly detection method that can learn a single, unified model capable of detecting anomalies in both known (seen) and entirely new (unseen) classes without retraining. It achieves this by leveraging a Masked Autoencoder (MAE)-inspired patch-level reconstruction approach and a clever "prototype-guided proposal masking" strategy during inference.

What Problem Does This Paper Address?

Traditional anomaly detection (AD) methods often struggle with class adaptability. This means that a model trained to detect anomalies in one class (e.g., defects on a specific type of screw) might perform poorly or fail entirely when presented with a different class (e.g., defects on a different screw or a completely different object). This necessitates retraining or fine-tuning models for each new class, which is impractical and resource-intensive for real-world applications where new product types or scenarios frequently emerge.

The paper specifically tackles two challenging settings:

  • Multi-Class Setting (one-for-many): Training a single model on normal samples from multiple known classes and then detecting anomalies within those same classes.
  • Cross-Class Setting (one-for-all): Training a single model on normal samples from seen classes and then expecting it to detect anomalies in unseen classes without any further training or fine-tuning. This is the more challenging scenario.

Existing methods often fall short in these settings due to two main issues:

  1. Reconstruction of Unseen Classes: Models trained only on seen classes might fail to reconstruct normal samples from unseen classes, leading to false positives.
  2. "Shortcut Reconstruction" Issue: Models can sometimes overfit and reconstruct even anomalous regions well, making it difficult to distinguish them from normal regions.

Methods

The proposed Proposal Masked Anomaly Detection (PMAD) approach has two core components:

  1. MAE-based Patch-Level Reconstruction:
    • Inspiration from MAE: Instead of reconstructing entire images (common in many AD methods), PMAD adopts a patch-level reconstruction strategy, inspired by Masked Autoencoders (MAE). The idea is that by masking out portions of an image and training the model to reconstruct them using the surrounding visible patches, the model learns a more generalizable understanding of contextual relationships within an image.
  2. ViT Architecture: They utilize a standard Vision Transformer (ViT) encoder-decoder architecture, which they found to be more effective than the asymmetric MAE architecture for AD, particularly for the decoder, to ensure good reconstruction of normal regions.
  3. Combinational Masking: During training, they employ a mix of random, blockwise, and frequency-based masking strategies to create more robust and generalizable masked patches. This helps the model learn to reconstruct diverse patterns, including those that might appear in unseen classes.
  4. Visual Tokens as Objective: Instead of raw pixels, they use discrete "visual tokens" (obtained from a dVAE codebook) as the reconstruction objective. This is less sensitive to fine-grained image details and reduces the risk of overfitting to local statistics, which can be problematic when using raw pixel reconstruction errors.
  5. Prototype-Guided Proposal Masking (Inference Stage):
    • Addressing "Shortcut Reconstruction": This is a two-stage process during inference designed to prevent the model from reconstructing anomalies too well.
  6. Offline Prototype Generation: They first generate "nominal feature prototypes" from normal samples of both seen and unseen classes using coreset subsampling. These prototypes represent the typical features of normal data.
  7. Proposal Masking: For a test image, they extract features from its patches and compare them to the nominal feature prototypes. Patches whose features deviate significantly from the prototypes are identified as "suspicious anomaly proposals." These proposals are then masked.
  8. Masking Anomalies: By masking these suspicious regions, the model is forced to reconstruct them. Since the model was trained on normal patterns, it will struggle to reconstruct these masked anomalous patches accurately, leading to high reconstruction errors and thus anomalies being detected. This strategy aims to mask out the most likely anomalous areas, preventing the "shortcut reconstruction" issue.

Datasets

The paper evaluates PMAD on two widely used industrial anomaly detection datasets:

  • MVTecAD: A comprehensive dataset with 5354 high-resolution images across 15 categories (textures and objects), featuring 73 different defect types.
  • BTAD: A dataset with 2830 real-world images of 3 industrial products.

The authors use these datasets to evaluate their model in single-class, multi-class, and cross-class settings.

Key Insights and Ideas

  • Patch-level reconstruction is key for generalization: The paper strongly suggests that reconstructing image patches, rather than whole images, is crucial for learning generalizable representations that can adapt to unseen classes. This is a departure from many traditional AD methods.
  • MAE's potential beyond pre-training: While MAE is known for self-supervised pre-training, this work demonstrates its applicability and adaptability for anomaly detection, particularly in challenging multi- and cross-class scenarios.
  • Targeted masking is vital for inference: Simply masking random patches during inference can leak too much information and lead to anomalies being reconstructed. The "prototype-guided proposal masking" is a novel and effective way to specifically target and mask suspicious regions, forcing the model to reveal anomalies through its reconstruction failures.
  • Visual tokens are a better reconstruction target than raw pixels: Using discrete visual tokens as the reconstruction objective proves more robust and less prone to overfitting than using raw pixel reconstruction errors, especially for anomaly detection.
  • Unified models are achievable for cross-class AD: The paper provides strong evidence that a single, unified model can indeed detect anomalies in entirely new classes without any fine-tuning, a significant step towards more practical and scalable AD systems.

Key Results

  • Single-Class Setting: PMAD achieves comparable results to state-of-the-art methods in the single-class setting, demonstrating its strong baseline performance.
  • Multi-Class Setting: PMAD shows significantly better performance than baseline methods, with much smaller performance drops compared to other SOTA models when moving from single-class to multi-class settings. For example, it outperforms the best competitor (DRAEM) by a large margin (2.7% at image-level) in the multi-class setting.
  • Cross-Class Setting: This is where PMAD truly shines. It significantly outperforms other SOTA methods, which often fail completely in unseen classes without fine-tuning. PMAD achieves substantial improvements, particularly for complex object classes (e.g., 17.4%/4.7% and 14.7%/8.4% improvements on MVTecAD). It also outperforms a registration-based class-adaptive method (RegAD).
  • Ablation Studies: The results confirm the effectiveness of the ViT architecture, the combinational masking strategy, the visual tokens as reconstruction objective, and crucially, the proposal masking strategy for inference. The proposal masking strategy alone leads to a significant performance improvement (about -20%/-25% drop if not used).

Limitations and Future Work

  • Limited Patch Granularity: The authors acknowledge that the ViT model's current architecture can only reconstruct 16x16 image patches. This limits the granularity of anomaly localization.
  • Future Directions: They plan to investigate hierarchical transformers and design multi-scale masking strategies to improve the fine-grained localization ability of their model.

Summary updated

Abstract

One of the most challenges for anomaly detection (AD) is how to learn one unified and generalizable model to adapt to multi-class especially cross-class settings: the model is trained with normal samples from seen classes with the objective to detect anomalies from both seen and unseen classes. In this work, we propose a novel Proposal Masked Anomaly Detection (PMAD) approach for such challenging multi- and cross-class anomaly detection. The proposed PMAD can be adapted to seen and unseen classes by two key designs: MAE-based patch-level reconstruction and prototype-guided proposal masking. First, motivated by MAE (Masked AutoEncoder), we develop a patch-level reconstruction model rather than the image-level reconstruction adopted in most AD methods for this reason: the masked patches in unseen classes can be reconstructed well by using the visible patches and the adaptive reconstruction capability of MAE. Moreover, we improve MAE by ViT encoder-decoder architecture, combinational masking, and visual tokens as reconstruction objectives to make it more suitable for anomaly detection. Second, we develop a two-stage anomaly detection manner during inference. In the proposal masking stage, the prototype-guided proposal masking module is utilized to generate proposals for suspicious anomalies as much as possible, then masked patches can be generated from the proposal regions. By masking most likely anomalous patches, the “shortcut reconstruction” issue (i.e., anomalous regions can be well reconstructed) can be mostly avoided. In the reconstruction stage, these masked patches are then reconstructed by the trained patch-level reconstruction model to determine if they are anomalies. Extensive experiments show that the proposed PMAD can outperform current state-of-the-art models significantly under the multi- and especially cross-class settings. Code will be publicly available at https://github.com/xcyao00/PMAD.
BibTeX
@inproceedings{yao2023one,
  author = {Xincheng Yao and Chongyang Zhang and Ruoqi Li and Jun Sun and Zhenyu Liu},
  title = {{One-for-All: Proposal Masked Cross-Class Anomaly Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=mEX8EUK3GxC},
}
Figure