Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types

Kihyuk Sohn, Jinsung Yoon, Chun-Liang Li, Chen-Yu Lee, Tomas Pfister

Abstract

We study anomaly clustering, grouping data into coherent clusters of anomaly types. This is different from anomaly detection that aims to divide anomalies from normal data. Unlike object-centered image clustering, anomaly clustering is particularly challenging as anomalous patterns are subtle and local. We present a simple yet effective clustering framework using a patch-based pretrained deep embeddings and off-the-shelf clustering methods. We define a distance function between images, each of which is represented as a bag of embeddings, by the Euclidean distance between weighted averaged embeddings. The weight defines the importance of instances (i.e., patch embeddings) in the bag, which may highlight defective regions. We compute weights in an unsupervised way or in a semi-supervised way when labeled normal data is available. Extensive experimental studies show the effectiveness of the proposed clustering framework along with a novel distance function upon existing multiple instance or deep clustering frameworks. Overall, our framework achieves 0.451 and 0.674 normalized mutual information scores on MVTec object and texture categories and further improve with a few labeled normal data (0.577, 0.669), far exceeding the baselines (0.244, 0.273) or state-of-the-art deep clustering methods (0.176, 0.277).

Introduction: Beyond Binary Detection

In industrial quality control, anomaly detection is typically treated as a binary classification problem: is the product "Normal" or "Anomalous"? While useful, a simple binary label is often insufficient for process improvement. To fix a manufacturing pipeline, engineers need to know the specific type of defect—whether it is a scratch, a dent, or a structural crack.

Anomaly clustering aims to group unlabeled data into a normal set and multiple coherent clusters of anomaly types.

This paper introduces a framework for Anomaly Clustering. The challenge lies in the fact that anomalous patterns are often subtle and localized. In a high-resolution image, a tiny crack might occupy only 1% of the pixels. Standard image clustering algorithms focus on global object semantics and tend to ignore these fine-grained, local details. Furthermore, industrial datasets are often small, making it difficult to train complex deep clustering models from scratch.

The Proposed Framework

The authors propose a framework that reframes the problem as Multiple Instance Clustering (MIC), treating images as "bags of patches."

  1. Image Representation: Instead of a single vector representing the whole image, each image is represented as a collection of patch embeddings extracted from a pretrained deep network (e.g., WideResNet-50).
  2. Weighted Average (WA) Distance: This is the core innovation. Simply averaging all patches in an image causes the "normal" background noise to wash out the signal of a tiny defect. The researchers introduce a weighting mechanism ($\alpha$) to highlight "important" (likely defective) patches.
  3. Unsupervised Weighting: Weights are assigned based on how unique a patch is compared to patches in other images.
  4. Semi-supervised Weighting: If a few labeled "normal" images are available, the system assigns higher weights to patches that look significantly different from the normal set.
  5. Clustering: Once the weighted distances between images are calculated, off-the-shelf algorithms like Hierarchical (Ward) or Spectral clustering are used to group the images.
The proposed framework extracts patch embeddings and applies a weighting mechanism to highlight anomalous regions before computing a weighted average for clustering.

Experimental Evaluation

The framework was evaluated on the MVTec AD dataset (the gold standard for anomaly detection) and the Magnetic Tile Defect (MTD) dataset.

Key Results

The proposed framework achieved massive gains over existing baselines:

  • MVTec Objects: Achieved a 0.451 NMI (Normalized Mutual Information) score, compared to just 0.176 for state-of-the-art deep clustering methods like SCAN.
  • MVTec Textures: Achieved a 0.674 NMI, nearly tripling the performance of standard baselines.
  • Efficiency: Because the method uses pretrained features and off-the-shelf clustering, it is computationally efficient and doesn't require the massive GPU resources needed to train deep clustering models.

Key Insights

  • Local vs. Holistic: The paper demonstrates that "holistic" representations (averaging the entire image) perform poorly for anomalies. Treating the image as a collection of patches allows the model to "zoom in" on defects.
  • The Power of "Normal": Using just a few unlabeled normal images to "filter out" the background (semi-supervised weighting) drastically improved cluster purity.
  • Deep Clustering Limitations: Surprisingly, state-of-the-art deep clustering methods failed in this domain. These models require large amounts of data to learn global structures, whereas anomaly clustering requires capturing subtle, local differences in small datasets.

Conclusion and Future Work

While highly effective, the authors acknowledge that current systems assume an image belongs to one primary defect cluster. Future work could explore multi-label anomaly clustering for products with multiple defect types. Additionally, while ImageNet-pretrained features work well, they may not be perfectly optimized for specialized industrial textures. Finally, the authors suggest these cluster assignments could be used in active learning systems to intelligently prioritize which images a human should label first.

Summary updated

Abstract

We study anomaly clustering, grouping data into coherent clusters of anomaly types. This is different from anomaly detection that aims to divide anomalies from normal data. Unlike object-centered image clustering, anomaly clustering is particularly challenging as anomalous patterns are subtle and local. We present a simple yet effective clustering framework using a patch-based pretrained deep embeddings and off-the-shelf clustering methods. We define a distance function between images, each of which is represented as a bag of embeddings, by the Euclidean distance between weighted averaged embeddings. The weight defines the importance of instances (i.e., patch embeddings) in the bag, which may highlight defective regions. We compute weights in an unsupervised way or in a semi-supervised way when labeled normal data is available. Extensive experimental studies show the effectiveness of the proposed clustering framework along with a novel distance function upon exist-ing multiple instance or deep clustering frameworks. Over-all, our framework achieves 0.451 and 0.674 normalized mutual information scores on MVTec object and texture categories and further improve with a few labeled normal data (0.577, 0.669), far exceeding the baselines (0.244, 0.273) or state-of-the-art deep clustering methods (0.176, 0.277).
BibTeX
@inproceedings{sohn2023anomaly,
  author = {Kihyuk Sohn and Jinsung Yoon and Chun-Liang Li and Chen-Yu Lee and Tomas Pfister},
  title = {{Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=3KTAN5NssEA},
}

Loading related content...

Figure