TL;DR
This paper challenges the long-held belief that autoencoders (AEs) require a bottleneck to prevent learning the identity function, which would render them useless for anomaly detection. Through extensive experiments with various non-bottlenecked AE architectures, including infinitely-wide ones, the authors demonstrate that these models can effectively detect anomalies and even outperform their bottlenecked counterparts, suggesting that learning the identity function is not as trivial as previously thought.
What Problem Does This Paper Address?
Anomaly detection is a crucial task in many fields, and autoencoders (AEs) are a popular choice for unsupervised anomaly detection. The core idea is that AEs learn to reconstruct normal data well, so anomalous data, which differs significantly from the training data, will be reconstructed with high error. However, a common concern is that AEs might learn to simply copy the input to the output (the "identity function") if not constrained. This would mean they reconstruct everything perfectly, making it impossible to distinguish between normal and anomalous data. To prevent this, researchers typically impose a "bottleneck" in the AE's architecture, forcing it to compress the input into a lower-dimensional latent space. This paper questions whether this bottleneck is always necessary and explores the potential of AEs without a bottleneck for anomaly detection.
Methods
The authors investigate two primary ways to remove the bottleneck in AEs:
- Overparameterization: This involves making the latent layer larger than the input dimension (an "overcomplete" architecture).
- Skip Connections: These are connections that allow information to bypass the bottleneck layer, directly passing features from earlier layers to later ones, similar to U-Net architectures.
They also introduce infinitely-wide Bayesian Autoencoders (BAE-∞) as an extreme form of non-bottlenecked architecture. The core of their methodology involves using Bayesian Autoencoders (BAEs), which treat the AE's parameters as probability distributions, allowing for a more robust estimation of uncertainty. They compare various configurations of bottlenecked and non-bottlenecked AEs, including deterministic AEs, Variational Autoencoders (VAEs), and different types of BAEs (e.g., BAE-MCD, BAE-BBB, BAE-Ensemble, and the proposed BAE-∞).
Datasets
The paper evaluates their methods on a variety of datasets:
- Image Data:
- FashionMNIST (inliers) vs. MNIST (anomalies)
- CIFAR (inliers) vs. SVHN (anomalies)
- Tabular Data: Eight datasets from the Outlier Detection Datasets (ODDS) collection: Cardio, Lympho, Optdigits, Ionosphere, Pendigits, Thyroid, Vowels, and Pima.
- Sensor Data:
- ZeMA dataset (from industrial environments)
- STRATH dataset (from industrial environments)
Key Insights and Ideas
- Bottleneck is Not Always Necessary: The most significant finding is that non-bottlenecked AEs can perform very well, and often better, for anomaly detection. This directly challenges the conventional wisdom.
- Identity Function is Harder to Learn Than Assumed: The success of non-bottlenecked AEs implies that learning the identity function is not as trivial as previously believed, even with overparameterized or skip-connected architectures.
- Non-Bottlenecked Architectures Offer Potential: The paper highlights that by removing the bottleneck, researchers might unlock better performance and more robust anomaly detection capabilities.
- Bayesian Approach Enhances Performance: Using Bayesian Autoencoders (BAEs) generally leads to improved anomaly detection performance compared to deterministic AEs and VAEs, likely due to their ability to model parameter uncertainty.
- Infinitely-Wide AEs as a Theoretical Tool: The BAE-∞ serves as an extreme case to explore the behavior of non-bottlenecked models and offers a foundation for theoretical analysis.
Key Results
- Outperforming Bottlenecked Models: On the CIFAR vs. SVHN benchmark, the best non-bottlenecked model achieved an AUROC of 0.849, significantly outperforming the best bottlenecked model's AUROC of 0.696. Similar trends were observed across other datasets.
- High AUROC Scores: Many non-bottlenecked models achieved mean AUROC scores of 0.8 or higher across various datasets. For instance, on ODDS, the BAE-Ensemble (non-bottlenecked) achieved an AUROC of 0.857, while the best bottlenecked model achieved 0.892.
- FashionMNIST vs. MNIST Anomaly: Interestingly, on the FashionMNIST vs. MNIST task, the bottlenecked model (BAE-BBB, type A) marginally outperformed the second-best non-bottlenecked model (BAE-BBB, type C) by 0.1 AUROC, suggesting that for some specific tasks, a bottleneck might still be beneficial or that certain non-bottlenecked configurations are more sensitive.
- BAEs Show Consistent Improvement: Switching from deterministic AEs to BAEs consistently improved performance across most configurations.
Limitations and Future Work
- Theoretical Proof Lacking: The authors acknowledge that while their experiments suggest non-bottlenecked AEs don't always learn the identity function, they lack a formal theoretical proof for this.
- Dataset Specificity: They note that there might exist datasets where learning the identity function is trivial for an AE, and their findings might not generalize to all possible scenarios.
- Other Use Cases: The study focuses solely on unsupervised anomaly detection. For other tasks like clustering or dimensionality reduction, a bottleneck might still be necessary.
- Future Work: The authors suggest developing more variants of non-bottlenecked AEs and further exploring the theoretical underpinnings of why they don't learn the identity function. They also propose investigating the connection between BAEs as predictive density models and kernel density estimation.
Summary updated