Introduction
Out-of-distribution (OOD) detection serves as a critical safety mechanism for machine learning models, enabling them to identify when an input falls outside their training distribution. While effective on small-scale datasets like CIFAR-10, traditional methods often struggle as the number of classes increases. For instance, the False Positive Rate (at 95% TPR) for standard baselines can escalate from 17.3% with 50 classes to 76.9% on ImageNet-1000. This paper introduces Predicted Embedding Power Regression (PEPR), a method designed to maintain robustness at scale with minimal computational overhead.
Methodology
The core of PEPR lies in modeling the relationship between a model's final class predictions and its internal intermediate embeddings.
Architecture and Scoring
The system consists of three primary components:
- The Embedder: Utilizing Exponential Linear Unit (ELU) activations and Batch Normalization, the embedder produces sparse, high-magnitude "tail-heavy" distributions for in-distribution (ID) data.
- The Regressor ($R(\hat{y})$): A small non-linear model trained to predict the expected embedding values ($z$) based on predicted class probabilities ($\hat{y}$).
- Scoring Function ($S_{PEPR}$): The OOD score is derived from the "power" (mean squared value) of the predicted embeddings. For ID data, the regressor accurately predicts high-magnitude sparse values; for OOD data, predictions tend toward zero.
To further enhance stability, the authors employ anchored ensembling, regularizing weights toward random initial values to emulate Bayesian uncertainty without the typical computational cost.
Experimental Setup
The method was evaluated on ImageNet-1k as the in-distribution dataset. Out-of-distribution performance was tested against standard benchmarks (Textures, SUN, Places, iNaturalist) and a novel Anomalous Species Dataset containing 700,000 species distinct from ImageNet. Training efficiency was a priority; by freezing the backbone and pre-computing outputs, training time was reduced by approximately 80%.
Results
PEPR and its variant C-PEPR (which integrates actual embedding power) demonstrate superior performance over existing state-of-the-art methods like Grouped Softmax (MOS).
As shown in the results, C-PEPR achieves an AUROC of 84.2 and an AUPR of 96.2, consistently outperforming MOS across multiple experimental runs.
Discussion and Limitations
A primary insight of this work is the shift from modeling $P(y|x)$ to $P(x|\hat{y})$, effectively verifying if internal features align with the predicted class. However, the method faces challenges with the Textures dataset (AUROC < 60), likely because textures lack the distinct object-oriented features that trigger high-magnitude embedding patterns. Future work will focus on improving robustness for non-object-oriented OOD data and reducing dependency on specific softmax configurations.
Summary updated