Abstract
This paper introduces an unsupervised energy-based framework for Out-of-Distribution (OOD) detection using the Stiefel-Restricted Kernel Machine (St-RKM). By integrating deep autoencoders with manifold-constrained latent space analysis, the proposed method identifies anomalies without requiring labeled data or prior exposure to OOD samples. The approach significantly outperforms existing generative models and supervised energy-based baselines across image and time-series benchmarks.
Introduction
In real-world machine learning deployment, the ability of a model to flag test samples that differ significantly from its training data—Out-of-Distribution (OOD) detection—is a critical safety requirement. Most state-of-the-art OOD detectors rely on supervised classifiers and labeled data, which introduces two major flaws: a dependency on expensive labels and a tendency to be fine-tuned for specific "known" OOD examples, making them less effective at spotting truly novel anomalies.
Researchers at KU Leuven propose a powerful unsupervised alternative: the Stiefel-Restricted Kernel Machine (St-RKM). This model learns the "energy" of a distribution entirely from raw, unlabeled in-distribution data, providing a robust solution for "open-world" settings like autonomous driving or medical diagnostics.
The St-RKM Framework
The St-RKM is a hybrid architecture combining a deep autoencoder with a Kernel Principal Component Analysis (KPCA) objective. A key technical innovation is the use of Manifold Optimization, where the model's interconnection matrix $U$ is constrained to the Stiefel manifold. This ensures that internal feature-matching weights remain orthonormal, preventing model collapse and ensuring a robust representation of the data structure.
The framework defines energy through different components to capture various types of anomalies:
- $E_{FullEnergy}$: The total objective function, combining reconstruction error and latent space alignment.
- $E_{kPCAError}$: Focuses specifically on how well a sample fits into the learned latent subspace.
- $E_{AELoss}$: The standard autoencoder reconstruction error.
Experimental Evaluation
The model was rigorously tested on computer vision benchmarks (Fashion-MNIST as in-distribution vs. MNIST, dSprites, SVHN, and CIFAR-10 as OOD) and time-series data (ECG5000).
One of the most significant findings is the "Closeness Paradox": different energy components excel at different tasks. For OOD data "close" to training data (e.g., MNIST vs. Fashion-MNIST), $E_{kPCAError}$ is the superior detector. For highly distinct OOD data (e.g., CIFAR-10 vs. Fashion-MNIST), $E_{FullEnergy}$ performs better due to the inclusion of reconstruction error.
Key Results
- Superiority over Supervised Baselines: The unsupervised St-RKM outperformed supervised energy-based methods (e.g., Liu et al., 2020) when those methods were not specifically fine-tuned with OOD examples.
- Low False Positive Rates: On MNIST (OOD), $E_{kPCAError}$ achieved a False Positive Rate at 95% True Positive Rate (FPR95) of 0.38%, compared to 75.97% for the supervised baseline. On CIFAR-10, $E_{FullEnergy}$ achieved an FPR95 of 0.34%.
- Time Series Performance: The model achieved an AUROC of 94.02% on the ECG5000 dataset, outperforming Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN).
Conclusion
The Stiefel-Restricted Kernel Machine provides a robust, label-agnostic approach to OOD detection. By learning the underlying structure of data rather than just classification boundaries, it offers a more general-purpose solution for anomaly detection. Future work will focus on scaling this approach to high-resolution datasets like ImageNet and investigating its robustness against adversarial attacks.
Summary updated