Abstract
We propose a new method for novelty detection that can tolerate high corruption of the training points, whereas previous works assumed either no or very low corruption. Our method, Mixture Autoencoding with Wasserstein penalty (MAW), trains a robust Variational Autoencoder (VAE) by modeling the latent space as a mixture of low-rank "normal" data and full-rank "corrupted" data, replacing the standard Kullback-Leibler (KL) divergence with a Wasserstein-1 penalty.
Introduction
Novelty detection—the task of identifying test data points that deviate from a "normal" baseline—is a cornerstone of modern machine learning applications, from medical diagnostics to fraud detection. Traditional one-class classification models typically operate under the assumption of a "clean" training dataset. However, real-world data is often contaminated with mislabeled or corrupted samples, known as outliers.
When standard models like Variational Autoencoders (VAEs) are trained on such contaminated data, they tend to overfit to the noise, losing their ability to distinguish between normal inliers and novel outliers. This paper introduces MAW, a framework designed to maintain high detection accuracy even when the training set is corrupted by up to 50% outliers.
The MAW Framework
The proposed MAW framework introduces several structural innovations to the standard VAE architecture to enhance robustness.
Key Methodological Innovations
- Latent Gaussian Mixture Model (GMM): Instead of a unimodal prior, MAW models the latent space as a mixture of two distributions. The inlier mode is constrained to a low-rank structure, reflecting the hypothesis that normal data resides on a lower-dimensional manifold. Conversely, the outlier mode is given a full-rank covariance matrix to capture high-variance noise.
- Wasserstein-1 Penalty: The authors replace the standard KL divergence with the Wasserstein-1 ($W_1$) metric. Theoretically, KL divergence is ill-defined for low-rank distributions (leading to infinite values), whereas the Wasserstein distance remains stable and helps align the estimated inlier mean with the true prior, effectively ignoring outliers.
- Robust Reconstruction Loss: By employing an $\ell_1$-type norm (least absolute deviations) instead of the standard Mean Squared Error (MSE), the model becomes significantly less sensitive to extreme reconstruction errors caused by outliers.
- Dimension Reduction Component: A specialized layer within the latent space allows the model to handle full covariance matrices efficiently, overcoming the limitations of the diagonal matrices used in standard VAEs.
Experimental Evaluation
The authors evaluated MAW against seven baseline methods, including DAGMM, OCGAN, and Isolation Forests. The benchmarks spanned diverse domains, including image data (COVID-19 Radiography, CIFAR-10, Caltech101, Fashion MNIST) and tabular/text data (KDDCUP-99 and Reuters-21578).
Performance Under Corruption
The primary metric for evaluation was the model's ability to maintain high Area Under the Curve (AUC) and Average Precision (AP) scores as the training set corruption level ($c$) increased from 0.1 to 0.5.
As shown in Figure 2, MAW consistently outperforms existing state-of-the-art methods. Notably, on the KDDCUP-99 dataset with 50% corruption, MAW maintained an impressive AUC, while competing deep learning methods struggled to exceed 0.52. Ablation studies further confirmed that each component—the Wasserstein penalty, the GMM latent structure, and the robust loss function—is vital for this performance.
Discussion and Conclusion
MAW represents a significant step forward in robust unsupervised learning. By leveraging the geometric properties of the Wasserstein distance and structural assumptions about data manifolds (low-rank vs. full-rank), it successfully isolates "normal" patterns from heavy noise. While the model introduces some computational overhead due to the auxiliary WGAN structure, its reliability in high-noise environments makes it a powerful tool for real-world novelty detection.
Summary updated