Unsupervised Anomaly Detection with Local-Sensitive VQVAE and Global-Sensitive Transformers

Mingqing Wang, Jiawei Li, Zhenyang Li, Chengxiao Luo, Bin Chen, Shu-Tao Xia, Zhi Wang

TL;DR

This paper introduces LSGS, a novel unsupervised anomaly detection method that combines a local-sensitive VQVAE with global-sensitive transformers. By leveraging an aggregated codebook for better latent space representation and transformers for global context, LSGS effectively reconstructs normal images, leading to improved anomaly localization in both medical and industrial applications.

What Problem Does This Paper Address?

Unsupervised Anomaly Detection (UAD) is crucial for applications like industrial quality control and medical diagnosis, where obtaining labeled anomaly data is difficult and expensive. Deep autoencoders (AEs) are commonly used, learning to reconstruct normal data and flagging deviations as anomalies. However, a key challenge is that AEs can sometimes "generalize too well," reconstructing anomalies so accurately that they are missed. Existing methods struggle with anomalies that are global in nature rather than just local texture variations, or they rely on prior knowledge of anomalies, undermining the unsupervised aspect. This paper aims to address these limitations by improving the representation of normal data in the latent space and incorporating global image context for more robust anomaly detection.

Methods

The proposed method, LSGS, consists of two main components:

  1. Local-Sensitive VQVAE with Aggregated Codebook:
    • A Vector Quantized-Variational Autoencoder (VQVAE) is trained on normal images to learn a reversible mapping between image patches and discrete latent codes.
  2. A novel aggregated codebook is introduced. Instead of using the standard VQVAE codebook, the authors aggregate image encodings from the training set and use k-means clustering to create a new codebook. This aims to better represent the distribution of discrete latent space and capture local image patch similarities.
  3. The VQVAE focuses on feature extraction and reconstruction, aiming to capture local image details.
  4. Global-Sensitive Transformers with Self-Supervised Training:
    • Full-attention transformers are employed to model the global information of images, complementing the VQVAE's local focus.
  5. These transformers are trained on the discrete encoding sequences generated by the VQVAE.
  6. A self-supervised training strategy is used: a portion of the encoding sequence is randomly "tampered" with, and the transformer's task is to reconstruct the original sequence. This is optimized using an "abnormally focal loss." This approach helps the transformer learn to rectify abnormal encodings by understanding the global context.

The final anomaly score is computed by reconstructing the image using the transformer-corrected encodings and comparing it to the original VQVAE reconstruction, with a weighted average of multiple reconstructions from normal data.

Datasets

The paper evaluates LSGS on two distinct datasets:

  • BraTS2018: A 3D MRI dataset (specifically, flair attenuated inversion recovery data) used for medical anomaly detection. The anomaly-free slices are used for training, and slices with anomalies for evaluation.
  • MVTec-AD: A widely used benchmark for industrial anomaly detection, comprising 15 different object categories. Each category has an anomaly-free training set and a test set with both normal and abnormal samples.

Key Insights and Ideas

  • Bridging Local and Global Information: The core idea is that combining local feature extraction (VQVAE) with global context modeling (Transformers) is crucial for robust anomaly detection, especially for anomalies that affect larger image regions.
  • Aggregated Codebook for Better Representation: Using an aggregated codebook derived from training data encodings, rather than a standard learned codebook, leads to a more representative discrete latent space, improving reconstruction quality.
  • Self-Supervised Transformer Training: The "tampering" and reconstruction strategy for transformers is an effective way to train them to understand and correct deviations in encoding sequences, leveraging global image structure.
  • Addressing Over-Generalization: By using transformers to enforce a normal distribution on latent encodings and then reconstructing, the method aims to mitigate the issue of autoencoders reconstructing anomalies too well.

Key Results

  • BraTS2018: LSGS achieved a significant improvement in anomaly detection performance, with an Average Precision (AP) of 0.757 and Dice score of 0.687, outperforming previous methods like f-AnoGAN, IS-cycle, PraNet, and CaraNet.
  • MVTec-AD: On average across 15 categories, LSGS achieved an AUROC of 89.9%, surpassing methods like US [9] and DRAEM [8].
  • Ablation Studies:
    • The aggregated codebook showed improved reconstruction loss and Dice scores on BraTS2018 compared to a standard VQVAE.
  • The global-sensitive transformers significantly outperformed vanilla transformers and models without transformers on BraTS2018.

Limitations and Future Work

  • Limitations:
    • The paper mentions that auto-regression models (like Pixel-CNN or GPT2) used in prior work [10, 11] fail to integrate global information effectively, which LSGS aims to solve. This implies that while LSGS improves global integration, there might still be room for further enhancement.
  • The method relies on training solely on normal data, which is a standard assumption but can be a limitation if subtle anomalies are present in the training set.
  • Future Work:
    • The paper's success on diverse datasets suggests potential for broader applications.
  • Further exploration of transformer architectures and self-supervised training strategies could lead to even better performance.
  • Investigating the trade-offs between local and global attention mechanisms could be a direction for future research.

Summary updated

Abstract

Unsupervised anomaly detection (UAD) has been widely implemented in industrial and medical applications, which reduces the cost of manual annotation and improves efficiency in disease diagnosis. Recently, deep auto-encoder with its variants has demonstrated its advantages in many UAD scenarios. Training on the normal data, these models are expected to locate anomalies by producing higher reconstruction error for the abnormal areas than the normal ones. However, this assumption does not always hold because of the uncontrollable generalization capability. To solve this problem, we present LSGS, a method that builds on Vector Quantised-Variational Autoencoder (VQVAE) with a novel aggregated codebook and transformers with global attention. In this work, the VQVAE focus on feature extraction and reconstruction of images, and the transformers fit the manifold and locate anomalies in the latent space. Then, leveraging the generated encoding sequences that conform to a normal distribution, we can reconstruct a more accurate image for locating the anomalies. Experiments on various datasets demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{wang2023unsupervised,
  author = {Mingqing Wang and Jiawei Li and Zhenyang Li and Chengxiao Luo and Bin Chen and Shu-Tao Xia and Zhi Wang},
  title = {{Unsupervised Anomaly Detection with Local-Sensitive VQVAE and Global-Sensitive Transformers}},
  booktitle = {ICIP},
  year = {2023},
  doi = {10.1109/icip49359.2023.10222596},
}
Figure