Learning by Erasing: Conditional Entropy Based Transferable Out-of-Distribution Detection

Meng Xing, Zhiyong Feng, Yong Su, Changjae Oh

Introduction

Deep learning models often exhibit high confidence when presented with Out-of-Distribution (OOD) data—inputs that differ significantly from their training set. This overconfidence poses a critical safety risk in high-stakes applications like autonomous driving and medical diagnostics. Traditional Deep Generative Models (DGMs) used for OOD detection face two primary challenges: the "Likelihood Paradox," where models assign higher likelihoods to simple OOD data (e.g., MNIST) than to complex in-distribution data (e.g., CIFAR-10), and a lack of transferability, requiring costly retraining for every new environment.

The CETOOD Framework: Learning by Erasing

To address these issues, researchers have introduced CETOOD (Conditional Entropy based Transferable OOD detection). The core innovation is a "Learning by Erasing" strategy. Instead of evaluating an entire image, the model is tasked with reconstructing a missing patch (e.g., a 16x16 center square) based on the surrounding context.

Reconstruction of natural images using the erasing strategy.

The system utilizes an Uncertainty Estimation Network (UEN), a specialized auto-encoder pre-trained on the massive and complex ImageNet32 dataset. By training on such a diverse dataset, the UEN becomes a "universal" feature extractor capable of capturing the underlying statistics of various image domains without needing further weight updates.

Conditional Entropy as a Discriminative Signal

The theoretical foundation of CETOOD lies in the relationship between a DGM's log-likelihood and the conditional entropy between the input surroundings and the erased target patch. By forcing the model to "fill in the blanks," the difficulty of the reconstruction task becomes a unique signature for the dataset's specific statistics.

To detect OOD samples, the system employs Kernel Density Estimation (KDE) to model the likelihood distribution of a small group of in-distribution (ID) samples. During testing, the distribution of a new group of samples is compared against the ID distribution using KL-divergence. If the deviation is significant, the group is flagged as OOD.

Comparison of empirical KDE distributions for different datasets across various metrics. Plot (c) shows the superior separation achieved using conditional entropy (-H).

Key Results and Efficiency

CETOOD demonstrates remarkable transferability. It achieves performance comparable to state-of-the-art methods that require dataset-specific retraining. For instance, in detecting SVHN from a CIFAR-10 baseline, the model achieved an AUROC of 99.7% using a group size of 5.

Beyond accuracy, the method is highly efficient. By bypassing the retraining phase, it significantly reduces deployment costs, training time, and computational overhead (FLOPs) compared to complex flow-based models. The "center-erasing" strategy was found to be particularly effective, as the center of an image typically contains the most semantically relevant information.

Limitations and Future Directions

While powerful, CETOOD faces challenges when ID and OOD datasets are semantically very similar, such as CIFAR-10 versus CIFAR-100, where conditional entropy distributions may overlap. Future work may explore "adaptive erasing," where the model learns to mask the most informative regions dynamically. Additionally, the requirement for group-based detection introduces a small latency buffer, which may need optimization for real-time applications.

Summary updated

Abstract

Out-of-distribution (OOD) detection is essential to handle the distribution shifts between training and test scenarios. For a new in-distribution (ID) dataset, existing methods require retraining to capture the dataset-specific feature representation or data distribution. In this paper, we propose a deep generative models (DGM) based transferable OOD detection method, which is unnecessary to retrain on a new ID dataset. We design an image erasing strategy to equip exclusive conditional entropy distribution for each ID dataset, which determines the discrepancy of DGM's posteriori ucertainty distribution on different ID datasets. Owing to the powerful representation capacity of convolutional neural networks, the proposed model trained on complex dataset can capture the above discrepancy between ID datasets without retraining and thus achieve transferable OOD detection. We validate the proposed method on five datasets and verity that ours achieves comparable performance to the state-of-the-art group based OOD detection methods that need to be retrained to deploy on new ID datasets. Our code is available at https://github.com/oOHCIOo/CETOOD.
BibTeX
@inproceedings{xing2024learning,
  author = {Meng Xing and Zhiyong Feng and Yong Su and Changjae Oh},
  title = {{Learning by Erasing: Conditional Entropy Based Transferable Out-of-Distribution Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28444},
}
Figure