Predicted Embedding Power Regression for Large-Scale Out-of-Distribution Detection

Hong Yang, William Gebhardt, Alexander G. Ororbia Ii, Travis DesellarXiv 2023 · DOI · arXiv

Introduction

Out-of-distribution (OOD) detection serves as a critical safety mechanism for machine learning models, enabling them to identify when an input falls outside their training distribution. While effective on small-scale datasets like CIFAR-10, traditional methods often struggle as the number of classes increases. For instance, the False Positive Rate (at 95% TPR) for standard baselines can escalate from 17.3% with 50 classes to 76.9% on ImageNet-1000. This paper introduces Predicted Embedding Power Regression (PEPR), a method designed to maintain robustness at scale with minimal computational overhead.

Methodology

The core of PEPR lies in modeling the relationship between a model's final class predictions and its internal intermediate embeddings.

Architecture and Scoring

The system consists of three primary components:

  1. The Embedder: Utilizing Exponential Linear Unit (ELU) activations and Batch Normalization, the embedder produces sparse, high-magnitude "tail-heavy" distributions for in-distribution (ID) data.
  2. The Regressor ($R(\hat{y})$): A small non-linear model trained to predict the expected embedding values ($z$) based on predicted class probabilities ($\hat{y}$).
  3. Scoring Function ($S_{PEPR}$): The OOD score is derived from the "power" (mean squared value) of the predicted embeddings. For ID data, the regressor accurately predicts high-magnitude sparse values; for OOD data, predictions tend toward zero.
Overview of the PEPR architecture, illustrating the flow from input image to OOD score calculation via the regressor and embedding power.

To further enhance stability, the authors employ anchored ensembling, regularizing weights toward random initial values to emulate Bayesian uncertainty without the typical computational cost.

Experimental Setup

The method was evaluated on ImageNet-1k as the in-distribution dataset. Out-of-distribution performance was tested against standard benchmarks (Textures, SUN, Places, iNaturalist) and a novel Anomalous Species Dataset containing 700,000 species distinct from ImageNet. Training efficiency was a priority; by freezing the backbone and pre-computing outputs, training time was reduced by approximately 80%.

Results

PEPR and its variant C-PEPR (which integrates actual embedding power) demonstrate superior performance over existing state-of-the-art methods like Grouped Softmax (MOS).

Comparison of PEPR variants against baseline OOD detection methods across AUROC, AUPR, and FPR95 metrics.

As shown in the results, C-PEPR achieves an AUROC of 84.2 and an AUPR of 96.2, consistently outperforming MOS across multiple experimental runs.

Discussion and Limitations

A primary insight of this work is the shift from modeling $P(y|x)$ to $P(x|\hat{y})$, effectively verifying if internal features align with the predicted class. However, the method faces challenges with the Textures dataset (AUROC < 60), likely because textures lack the distinct object-oriented features that trigger high-magnitude embedding patterns. Future work will focus on improving robustness for non-object-oriented OOD data and reducing dependency on specific softmax configurations.

Summary updated

Abstract

Out-of-distribution (OOD) inputs can compromise the performance and safety of real world machine learning systems. While many methods exist for OOD detection and work well on small scale datasets with lower resolution and few classes, few methods have been developed for large-scale OOD detection. Existing large-scale methods generally depend on maximum classification probability, such as the state-of-the-art grouped softmax method. In this work, we develop a novel approach that calculates the probability of the predicted class label based on label distributions learned during the training process. Our method performs better than current state-of-the-art methods with only a negligible increase in compute cost. We evaluate our method against contemporary methods across $14$ datasets and achieve a statistically significant improvement with respect to AUROC (84.2 vs 82.4) and AUPR (96.2 vs 93.7).
BibTeX
@misc{yang2023predicted,
  author = {Hong Yang and William Gebhardt and Alexander G. Ororbia II and Travis Desell},
  title = {{Predicted Embedding Power Regression for Large-Scale Out-of-Distribution Detection}},
  year = {2023},
  eprint = {2303.04115},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.04115},
}
Figure