GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection

Xixi Liu, Yaroslava Lochman, Christopher ZachCVPR 2023 · DOI

Introduction

Deep learning models are notoriously overconfident. When presented with Out-of-Distribution (OOD) data—samples from classes they have never seen—they often assign high-probability predictions to known classes rather than signaling uncertainty. This overconfidence poses significant safety risks in critical applications like autonomous driving and medical diagnostics.

While many high-performing OOD detection methods require access to internal model features or training data statistics ("white-box" methods), real-world scenarios often limit access to a model's final softmax outputs ("black-box" or post-hoc methods). Historically, post-hoc methods have lagged in accuracy. Researchers from Chalmers University of Technology have introduced Generalized ENtropy (GEN), a simple yet powerful post-hoc score that closes this performance gap without requiring model retraining or access to training data.

Methodology

The core of the GEN approach lies in magnifying the subtle deviations in predictive distributions when a model encounters unfamiliar data. It consists of two primary components:

1. Generalized Entropy Function

Standard Shannon entropy treats all probabilities equally and can be relatively "flat" near the extremes. GEN utilizes a specific family of concave functions:

$$G_\gamma(\mathbf{p}) = -\sum_{j=1}^{M} p_j^\gamma (1-p_j)^\gamma$$

By setting the parameter $\gamma$ to a low value (typically 0.1), the score becomes hyper-sensitive to small changes in the predictive distribution. This sensitivity allows the system to flag OOD samples that might otherwise appear as high-confidence ID samples under standard metrics.

The Generalized Entropy function $G_\gamma(\mathbf{p})$ is much steeper near the boundaries compared to standard Shannon entropy $H(\mathbf{p})$, providing higher sensitivity to OOD signals.

2. Truncation (Trimming)

In large-scale datasets like ImageNet, the "tail" of the probability distribution (the hundreds of low-probability classes) often contains random noise that can degrade the reliability of an OOD score. GEN addresses this by only considering the top $M$ classes (e.g., the top 100). This truncation focuses the signal on the most relevant predictions, providing a cleaner indicator of uncertainty.

Experimental Setup

The authors evaluated GEN using ImageNet-1k as the In-Distribution (ID) dataset. Performance was tested against four diverse OOD benchmarks: OpenImage-O, Textures, iNaturalist, and the challenging ImageNet-O (natural adversarial examples). Testing spanned six architectures, including ResNet, RepVGG, and modern Vision Transformers (ViT, Swin, DeiT).

Key Results

GEN consistently outperformed existing post-hoc methods, improving the average AUROC (Area Under the Receiver Operating Characteristic curve) by at least 3.5% across all tested architectures.

  • State-of-the-Art Performance: On the difficult ImageNet-O dataset, GEN achieved significantly higher AUROC than previous standards like MaxLogit or Energy scores.
  • Synergy with Feature Clipping: When combined with methods like ReAct (which clips internal features), GEN's performance improved further, rivaling complex white-box methods despite its simpler post-hoc nature.
  • Architectural Robustness: The method proved effective for both traditional CNNs and newer Vision Transformers, suggesting it captures a fundamental property of how neural networks represent uncertainty in their final layers.
Comparison of OOD detection performance (AUROC) on OpenImage-O and ImageNet-O. GEN and its combination with ReAct* significantly outperform baseline methods like MSP, ODIN, and MaxLogit.

Conclusion

GEN demonstrates that the performance of softmax-based OOD detection has been significantly underestimated. By applying a more sensitive entropy function and trimming distributional noise, GEN provides a robust, easy-to-implement solution for safer AI deployment. Future work may explore the theoretical underpinnings of why the top 10% of logits consistently provide the cleanest OOD signal across such a wide variety of models.

Summary updated

BibTeX
@inproceedings{liu2023gen,
  author = {Xixi Liu and Yaroslava Lochman and Christopher Zach},
  title = {{GEN: Pushing the Limits of Softmax-Based Out-of-Distribution Detection}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02293},
}

Loading related content...

Figure