Unsupervised Outlier Detection via Transformation Invariant Autoencoder

Zhen Cheng, En Zhu, Siqi Wang, Pei Zhang, Wang LiIEEE Access 2021 · OpenReview

Abstract

Autoencoder-based methods are the cornerstone of deep unsupervised outlier detection. However, these methods often underperform on complex image datasets and are sensitive to noise introduced by outliers during training. We present the Transformation Invariant AutoEncoder (TIAE), a framework that improves representation learning by requiring the model to reconstruct original images from transformed inputs. To further mitigate the impact of outliers in the training set, we incorporate an adaptive self-paced learning (SPL) strategy. Our evaluations show that TIAE advances unsupervised outlier detection performance by up to 10% AUROC compared to state-of-the-art autoencoder-based methods across five benchmark image datasets.

Introduction

Outlier detection (OD) involves identifying data points that deviate significantly from the majority of the data. In many real-world scenarios, such as fraud detection or medical diagnosis, labeled data is scarce, necessitating unsupervised approaches. The standard paradigm for unsupervised OD is the Autoencoder (AE). As shown below, a standard AE learns to compress an input $x$ into a latent representation and then reconstruct it. The underlying assumption is that the model, trained primarily on "normal" data, will fail to reconstruct outliers accurately, resulting in a high reconstruction error that serves as an anomaly score.

Standard Autoencoder (AE) architecture.

However, standard AEs face two major challenges. First, they often focus on low-level pixel correlations rather than high-level semantics, which is insufficient for complex datasets like CIFAR-10. Second, if the training data contains outliers, the AE may learn to reconstruct them, thereby reducing the detection sensitivity.

The TIAE Framework

To address these challenges, we propose the Transformation Invariant AutoEncoder (TIAE).

Transformation Invariant Representation Learning

The core innovation of TIAE is the shift from simple reconstruction to transformation-invariant reconstruction. As illustrated in the diagram below, the input $x$ undergoes a transformation $t(\cdot)$, such as rotation or color distortion. The encoder $E$ maps the transformed image $t(x)$ to a latent space, and the decoder $D$ must then reconstruct the original image $x$.

The proposed Transformation Invariant Autoencoder (TIAE) architecture.

This approach forces the network to capture the semantic "essence" of the object (e.g., "this is a cat") to successfully undo the transformation. By focusing on high-level structural information rather than pixel-wise identity, the model becomes much more effective at distinguishing between complex inlier classes and outliers.

Adaptive Self-Paced Learning

To handle outliers present in the training data, we integrate an adaptive self-paced learning (SPL) module. SPL is a curriculum learning strategy where the model initially trains on "easy" samples (those with low reconstruction error) and gradually includes "harder" ones.

In TIAE, we introduce an adaptive threshold ($\lambda$) that is automatically calculated in each epoch based on the mean and standard deviation of the reconstruction scores. This ensures that likely outliers—which naturally yield higher errors—are excluded from the gradient updates, preventing them from "polluting" the learned representation.

Experimental Results

We evaluated TIAE on five benchmark datasets: MNIST, Fashion-MNIST, SVHN, CIFAR-10, and CIFAR-100. We employed a "one-vs-rest" protocol, where one class is designated as the inlier and the remaining classes as outliers.

The results demonstrate the superiority of the TIAE framework:

  • Performance Gains: On CIFAR-10, TIAE achieved an average AUROC of 71.18%, a significant improvement over the standard Convolutional AE (55.86%) and the previous state-of-the-art, RSRAE (54.77%).
  • Robustness: TIAE maintained high performance even as the outlier ratio in the training set increased from 5% to 25%, whereas baseline models showed significant degradation.
  • Stability: The adaptive SPL module was found to significantly stabilize the AUROC across training epochs compared to versions of the model without the SPL component.

Conclusion

The Transformation Invariant AutoEncoder (TIAE) provides a robust and effective framework for unsupervised outlier detection. By leveraging transformation-based pretext tasks and adaptive self-paced learning, TIAE moves beyond simple pixel reconstruction to learn meaningful semantic features. This approach not only improves detection accuracy on complex datasets but also ensures stability in the presence of noisy training data. Future work may explore the application of these principles to video-based anomaly detection.

Summary updated

Abstract

Autoencoder based methods are the majority of deep unsupervised outlier detection methods. However, these methods perform not well on complex image datasets and suffer from the noise introduced by outliers, especially when the outlier ratio is high. In this paper, we propose a framework named Transformation Invariant AutoEncoder (TIAE), which can achieve stable and high performance on unsupervised outlier detection. First, instead of using a conventional autoencoder, we propose a transformation invariant autoencoder to do better representation learning for complex image datasets. Next, to mitigate the negative effect of noise introduced by outliers and stabilize the network training, we select the most confident inliers likely examples in each epoch as the training set by incorporating adaptive self-paced learning in our TIAE framework. Extensive evaluations show that TIAE significantly advances unsupervised outlier detection performance by up to 10% AUROC against other autoencoder based methods on five image datasets.
BibTeX
@inproceedings{cheng2021unsupervised,
  author = {Zhen Cheng and En Zhu and Siqi Wang and Pei Zhang and Wang Li},
  title = {{Unsupervised Outlier Detection via Transformation Invariant Autoencoder}},
  booktitle = {IEEE Access},
  year = {2021},
  url = {https://openreview.net/forum?id=NDshYG7i_sE},
}
Figure