Abstract
Out-of-distribution (OOD) detection is a desired ability to ensure the reliability and safety of intelligent systems. A scoring function is often designed to measure the degree of any new data being an OOD sample. While most designed scoring functions are based on a single source of information (e.g., the classifier’s output, logits, or feature vector), recent studies demonstrate that fusion of multiple sources may help better detect OOD data. In this study, after detailed analysis of the issue in OOD detection by the conventional principal component analysis (PCA), we propose fusing a simple regularized PCA-based reconstruction error with other source of scoring function to further improve OOD detection performance. In particular, when combined with a strong energy score-based OOD method, the regularized reconstruction error helps achieve new state-of the-art OOD detection results on multiple standard benchmarks.
Introduction
Deep learning models often exhibit overconfidence when presented with Out-of-Distribution (OOD) samples—data that falls outside the distribution of their training set. This behavior poses significant risks in safety-critical domains such as autonomous driving and medical diagnostics. While many detection methods rely on the final outputs of a classifier, such as logits or softmax probabilities, others investigate the internal feature representations.
Principal Component Analysis (PCA) is a classical technique often considered for this task. The intuition is straightforward: features from In-Distribution (ID) data should lie within a principal subspace learned during training, resulting in low reconstruction error, whereas OOD features should lie outside this subspace, resulting in high error. However, standard PCA-based reconstruction error often fails to distinguish between ID and OOD samples effectively. This research investigates the geometric reasons behind this failure and proposes a robust solution.
The Geometry of Feature Spaces: Why PCA Fails
The researchers identified a "Variance Paradox" in deep feature spaces. Contrary to common intuition, ID data typically exhibits significantly higher feature variance than OOD data. By decomposing the standard reconstruction error into its magnitude and angular components, the authors found that while ID data has a smaller angle relative to the principal subspace, it also has a much larger feature norm (magnitude).
In the standard reconstruction error formula, the large magnitude of ID features "masks" the benefit of their small angular deviation. Consequently, the error distributions of ID and OOD data overlap significantly, rendering the raw error metric ineffective for detection.
Proposed Solution: Regularized PCA and Multiplicative Fusion
To address the masking effect of feature magnitudes, the authors propose a Regularized Reconstruction Error ($r(x)$). By normalizing the reconstruction error by the norm of the feature vector, the metric isolates the angular component that truly distinguishes ID from OOD data:
$$r(x) = \frac{\|h(x) - \hat{h}(x)\|}{\|h(x)\|}$$
Furthermore, the study introduces a Multiplicative Fusion strategy to combine this regularized error with the Energy Score, a popular logit-based metric. Unlike additive fusion, which requires sensitive hyperparameter tuning, multiplicative fusion naturally balances the signals:
$$D(x) = (1 - r(x)) \times \text{Energy}(x)$$
This approach is entirely post-hoc, requiring no model retraining or access to OOD data during the development phase.
Experimental Evaluation
The proposed method was rigorously tested across major benchmarks, including ImageNet-1K, ImageNet-100, CIFAR-10, and CIFAR-100, using various backbones like ResNet50 and WideResNet.
The results demonstrate a significant leap in performance. On the ImageNet-1K benchmark with a ResNet50 backbone, the method achieved an AUROC of 95.78%, substantially outperforming the previous state-of-the-art baseline, ReAct (92.95%). More importantly, the False Positive Rate at a 95% True Positive Rate (FPR95) was reduced from 31.43% to 18.66%. The regularized PCA error proved to be a versatile "plug-and-play" improvement, enhancing various existing OOD scoring functions.
Conclusion
This work revisits PCA for OOD detection by uncovering the geometric properties of deep features. By identifying the variance paradox and introducing a regularized reconstruction error, the authors provide a simple yet powerful tool for enhancing model reliability. The multiplicative fusion with energy scores sets a new state-of-the-art for post-hoc OOD detection, offering a robust solution for deploying safer AI systems.
Summary updated