Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types

Kihyuk Sohn, Jinsung Yoon, Chun-Liang Li, Chen-Yu Lee, Tomas Pfister

Abstract

We study anomaly clustering, grouping data into coherent clusters of anomaly types. This is different from anomaly detection that aims to divide anomalies from normal data. Unlike object-centered image clustering, anomaly clustering is particularly challenging as anomalous patterns are subtle and local. We present a simple yet effective clustering framework using a patch-based pretrained deep embeddings and off-the-shelf clustering methods. We define a distance function between images, each of which is represented as a bag of embeddings, by the Euclidean distance between weighted averaged embeddings. The weight defines the importance of instances (i.e., patch embeddings) in the bag, which may highlight defective regions. We compute weights in an unsupervised way or in a semi-supervised way when labeled normal data is available. Extensive experimental studies show the effectiveness of the proposed clustering framework along with a novel distance function upon existing multiple instance or deep clustering frameworks. Overall, our framework achieves 0.451 and 0.674 normalized mutual information scores on MVTec object and texture categories and further improve with a few labeled normal data (0.577, 0.669), far exceeding the baselines (0.244, 0.273) or state-of-the-art deep clustering methods (0.176, 0.277).

The Challenge of Localized Defects

In industrial quality control or medical imaging, detecting an anomaly is only the first step. To fix a manufacturing line, an engineer needs to know if the defect is a crack, a smudge, or a missing component.

Standard anomaly detection treats this as a binary problem: Normal vs. Abnormal. Standard image clustering, on the other hand, is usually "object-centered"—it’s great at telling a dog from a cat because the whole image changes. However, an anomalous image is often 99% identical to a normal one, with the "anomaly" being a tiny, subtle local pattern. Existing deep clustering methods fail here because they try to represent the whole image at once, washing out the small, critical details that define the anomaly type.

Comparison between standard anomaly detection (left) and the proposed anomaly clustering task (right).

Methodology: Bags of Patches

The authors propose a simple but powerful three-step framework that treats images as "bags of patches" to ensure local defects are not lost in global averages:

  1. Patch-based Representation: Instead of one global vector, the image is passed through a pretrained network (like WideResNet-50) to extract embeddings for small patches.
  2. Weighted Average Distance (WA): This is the core innovation. To compare two images, the system calculates a weight for each patch representing its "importance" (likelihood of being a defect).
  3. Unsupervised: Weights are found by looking for patches that are "unique" compared to other images in the dataset.
  4. Semi-supervised: If a few "normal" images are available, the system learns to ignore patches that look like the normal background, effectively "highlighting" the anomalies.
  5. Off-the-shelf Clustering: Once these weighted representations are generated, standard algorithms like Ward’s hierarchical clustering or Spectral clustering are used to group the images.
The proposed technical pipeline: from patch embedding to weighted average and final clustering.

Key Insights and Results

The researchers evaluated their method on two primary industrial benchmarks: MVTec AD and Magnetic Tile Defect (MTD). Several critical insights emerged:

  • Local > Global: Holistic image representations are the "enemy" of anomaly clustering. By treating the image as a bag of instances, the model can ignore the 95% of the image that is normal and focus on the 5% that matters.
  • The "Normal" Anchor: One of the most striking findings is how much normal data helps cluster anomalies. By showing the model a few examples of what "perfect" looks like, it becomes much better at identifying and grouping the specific ways things can go wrong.
  • Performance Gains: The framework significantly outperformed existing baselines. On MVTec object categories, it achieved a Normalized Mutual Information (NMI) score of 0.451, compared to just 0.176 for state-of-the-art deep clustering methods. When provided with a few labeled normal images, the NMI jumped to 0.577.

Conclusion

This framework demonstrates that effective anomaly clustering does not require complex new deep learning architectures. Instead, by using a smart distance metric on top of pretrained features, we can achieve high-purity clusters that could reduce human labeling effort by over 50%. Future work will look toward handling images with multiple simultaneous anomaly types and scaling the distance calculations for massive datasets.

Summary updated

Abstract

We study anomaly clustering, grouping data into coherent clusters of anomaly types. This is different from anomaly detection that aims to divide anomalies from normal data. Unlike object-centered image clustering, anomaly clustering is particularly challenging as anomalous patterns are subtle and local. We present a simple yet effective clustering framework using a patch-based pretrained deep embeddings and off-the-shelf clustering methods. We define a distance function between images, each of which is represented as a bag of embeddings, by the Euclidean distance between weighted averaged embeddings. The weight defines the importance of instances (i.e., patch embeddings) in the bag, which may highlight defective regions. We compute weights in an unsupervised way or in a semi-supervised way when labeled normal data is available. Extensive experimental studies show the effectiveness of the proposed clustering framework along with a novel distance function upon existing multiple instance or deep clustering frameworks. Overall, our framework achieves 0.451 and 0.674 normalized mutual information scores on MVTec object and texture categories and further improve with a few labeled normal data (0.577, 0.669), far exceeding the baselines (0.244, 0.273) or state-of-the-art deep clustering methods (0.176, 0.277).
BibTeX
@inproceedings{sohn2023anomaly,
  author = {Kihyuk Sohn and Jinsung Yoon and Chun-Liang Li and Chen-Yu Lee and Tomas Pfister},
  title = {{Anomaly Clustering: Grouping Images into Coherent Clusters of Anomaly Types}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
  url = {https://openreview.net/forum?id=3KTAN5NssEA},
}
Figure