Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution

Kai Liu, Zhihang Fu, Sheng Jin, Chao Chen, Ze Chen, Rongxin Jiang, Fan Zhou, Yaowu Chen, Jieping Ye

Abstract

Detecting and rejecting unknown out-of-distribution (OOD) samples is critical for deployed neural networks to avoid unreliable predictions. In real-world scenarios, however, the efficacy of existing OOD detection methods is often impeded by the inherent imbalance of in-distribution (ID) data, which causes significant performance decline. Through statistical observations, we have identified two common challenges faced by different OOD detectors: misidentifying tail class ID samples as OOD, while erroneously predicting OOD samples as head class from ID. To explain this phenomenon, we introduce a generalized statistical framework, termed ImOOD, to formulate the OOD detection problem on imbalanced data distribution. Consequently, the theoretical analysis reveals that there exists a class-aware bias item between balanced and imbalanced OOD detection, which contributes to the performance gap. Building upon this finding, we present a unified training-time regularization technique to mitigate the bias and boost imbalanced OOD detectors across architecture designs. Our theoretically grounded method translates into consistent improvements on the representative CIFAR10-LT, CIFAR100-LT, and ImageNet-LT benchmarks against several state-of-the-art OOD detection approaches.

Introduction

In the controlled environments of machine learning research, datasets are often curated to be perfectly balanced, with an equal number of samples for every category. However, real-world data is rarely so cooperative. Most practical applications encounter "long-tailed" distributions, where a few "head" classes contain the vast majority of data, while many "tail" classes are represented by only a handful of examples.

While significant progress has been made in improving classification accuracy on such imbalanced data, the critical task of Out-of-Distribution (OOD) detection—the ability of a model to recognize and reject inputs it was never trained on—remains a major hurdle. Standard OOD detection techniques frequently fail when the in-distribution (ID) data is imbalanced, leading to dangerous overconfidence or unnecessary caution. This paper introduces ImOOD, a statistical framework designed to identify and correct the fundamental biases that emerge in imbalanced OOD detection.

The Failure Modes of Imbalanced OOD Detection

Existing OOD detectors suffer from two primary failure modes when trained on long-tailed data:

  1. Tail-Class Confusion: Because the model has seen very few examples of tail classes, it lacks the confidence to categorize them correctly. Consequently, these rare but valid ID samples are frequently misidentified as OOD.
  2. Head-Class Absorption: Conversely, head classes—which have an abundance of training data—develop over-generalized and "bloated" decision boundaries. This causes the model to erroneously absorb truly OOD samples into these common ID categories.

Previous research has often focused on improving tail-class accuracy, but the authors argue that this only addresses half the problem. The core issue lies in the geometric and probabilistic bias introduced by the class priors themselves.

The ImOOD Framework

To address these challenges, the researchers developed the ImOOD framework, which provides a mathematical foundation for understanding the gap between balanced and imbalanced OOD detection.

The Discovery of Class-Aware Bias

Through rigorous mathematical derivation, the authors proved the existence of a class-aware bias item ($\beta$). This term represents the inherent discrepancy between an ideal detector (trained on balanced data) and the practical detector learned from imbalanced data. This bias naturally skews the decision boundaries, favoring head classes and marginalizing tail classes in the OOD scoring process.

Unified Regularization

Building on this discovery, ImOOD introduces a training-time regularization technique. Instead of relying on fixed heuristics, the method utilizes a learnable parameter ($\gamma$) to estimate the bias for every sample dynamically. During training, the loss function is adjusted to:

  • Protect Tail Classes: The model is penalized more heavily for misidentifying rare ID samples as OOD.
  • Constrain Head Classes: The model is forced to shrink the over-extended decision spaces of head classes, preventing them from "absorbing" outliers.

One of the primary strengths of ImOOD is its versatility; it is not a standalone model but a regularization plug-in that can be integrated into various existing OOD detection architectures, such as Outlier Exposure (OE), Energy-based models, and PASCL.

Experimental Evaluation

The researchers evaluated ImOOD across three major Long-Tailed (LT) benchmarks: CIFAR10-LT, CIFAR100-LT, and the large-scale ImageNet-LT. To simulate "known unknowns," auxiliary datasets like TinyImages80M and ImageNet-Extra were used during training to help the model define the boundaries of the known world.

Key Results

ImOOD demonstrated significant performance gains across all metrics:

  • ImageNet-LT: On this challenging large-scale benchmark, ImOOD improved the Area Under the ROC Curve (AUROC) by 6.7% over the previous state-of-the-art (SOTA) method, PASCL.
  • FPR95 Reduction: The method significantly lowered the False Positive Rate (FPR95)—the frequency at which OOD samples are wrongly accepted as ID. On CIFAR10-LT, this error rate was reduced by 4.6%.
  • Error Distribution: Statistical analysis confirmed that ImOOD successfully balanced the error rates across head, middle, and tail classes, preventing the performance from being dominated by head-class biases.

Conclusion and Future Work

ImOOD represents a significant step forward in making OOD detection robust for real-world, imbalanced applications. By identifying the geometric "geometry of failure"—where head classes occupy too much decision space and tail classes too little—the framework provides a theoretically grounded way to correct for class-aware bias.

While the method currently benefits from access to auxiliary outlier data during training, future research will focus on "open-world evolution." This involves developing online learning techniques that allow models to continue adapting their decision boundaries after deployment, ensuring they remain robust even as they encounter unforeseen OOD samples in the wild.

Summary updated

Abstract

Detecting and rejecting unknown out-of-distribution (OOD) samples is critical for deployed neural networks to void unreliable predictions. In real-world scenarios, however, the efficacy of existing OOD detection methods is often impeded by the inherent imbalance of in-distribution (ID) data, which causes significant performance decline. Through statistical observations, we have identified two common challenges faced by different OOD detectors: misidentifying tail class ID samples as OOD, while erroneously predicting OOD samples as head class from ID. To explain this phenomenon, we introduce a generalized statistical framework, termed ImOOD, to formulate the OOD detection problem on imbalanced data distribution. Consequently, the theoretical analysis reveals that there exists a class-aware bias item between balanced and imbalanced OOD detection, which contributes to the performance gap. Building upon this finding, we present a unified training-time regularization technique to mitigate the bias and boost imbalanced OOD detectors across architecture designs. Our theoretically grounded method translates into consistent improvements on the representative CIFAR10-LT, CIFAR100-LT, and ImageNet-LT benchmarks against several state-of-the-art OOD detection approaches. Code will be made public soon.
BibTeX
@inproceedings{liu2024rethinking,
  author = {Kai Liu and Zhihang Fu and Sheng Jin and Chao Chen and Ze Chen and Rongxin Jiang and Fan Zhou and Yaowu Chen and Jieping Ye},
  title = {{Rethinking Out-of-Distribution Detection on Imbalanced Data Distribution}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=EWxNEnFjKR},
}
Figure