OOD Detection with immature Models

Behrooz Montazeran, Ullrich KöthearXiv 2025 · DOI · arXiv

Abstract

Likelihood-based Deep Generative Models (DGMs) frequently fail at out-of-distribution (OOD) detection, often assigning higher probabilities to unseen, simpler data than to their own training sets—a phenomenon known as the "likelihood paradox." This paper reveals a surprising solution: "immature" models, stopped early in the training process, can significantly outperform fully converged models in OOD tasks. By leveraging layer-wise gradient-based scoring, these early-stage models capture low-level structural features that are more discriminative for OOD detection than the high-level semantic features learned later. Our findings suggest that the computational cost of full model convergence is not only unnecessary but potentially detrimental for robust OOD detection.

1. Introduction: The Likelihood Paradox

Deep Generative Models like Normalizing Flows (e.g., GLOW) and Variational Autoencoders (VAEs) are designed to model the underlying distribution of training data. Theoretically, they should assign high likelihood to in-distribution (ID) samples and low likelihood to out-of-distribution (OOD) samples. However, empirical evidence shows a persistent "likelihood paradox": models trained on complex datasets like CIFAR-10 often assign higher likelihoods to simpler datasets like SVHN.

The Likelihood Paradox: A GLOW model trained on CIFAR-10 (ID) assigns higher likelihood (lower negative bits per dimension) to SVHN (OOD) samples.

This paradox makes raw likelihood an unreliable metric for safety-critical applications. While recent research has turned to gradient norms as a more robust OOD signal, these methods typically rely on fully trained, "mature" models. This study challenges that convention, investigating whether full convergence is required for effective OOD detection.

2. Methodology

The proposed approach utilizes the GLOW architecture and a novel layer-wise gradient-based scoring mechanism:

  1. Gradient Scoring: Instead of output probability, the method calculates the gradient of the log-likelihood with respect to model parameters for each input.
  2. Fisher Information Approximation: By approximating the Fisher Information Matrix (FIM) as an identity matrix, the OOD score is simplified to the log-transformed squared Euclidean norm of layer-wise gradients.
  3. Statistical Integration: A Gaussian distribution is fitted to these gradient norms using a separate "fit" dataset. The final OOD score is the Gaussian negative log-likelihood (NLL) aggregated across all layers.
  4. Early Stopping: The core of this research is the comparison of OOD detection performance at various training stages, specifically contrasting "immature" models (e.g., Epoch 1 or 10) with fully converged "mature" models (e.g., Epoch 250).

3. Results and Analysis

The study evaluates performance across various 3-channel (CIFAR-10, CelebA, SVHN, ImageNet32, GTSRB) and 1-channel (MNIST, FashionMNIST, KMNIST, Omniglot) datasets.

3.1 Performance of Immature Models

A key finding is that OOD detection performance often peaks very early in training. In many benchmarks, immature models achieved near-perfect AUROC scores (up to 1.0000). For instance, a GLOW model trained on SVHN for only 10 epochs reached an AUROC of 0.9852 against CelebA, whereas the fully trained version's performance degraded to 0.9377.

Distribution of OOD scores for a model trained on ImageNet32, showing clear separation between in-distribution fit samples and various OOD test sets.

3.2 Complexity and Batching

When trained on highly complex datasets like ImageNet32, immature models reach near-optimal performance almost immediately. Furthermore, using a small batch size (e.g., 5) for the goodness-of-fit test significantly enhances detection, often yielding perfect scores as early as the first epoch.

4. Discussion: Low-Level Features and Support Overlap

The superior performance of immature models is attributed to the nature of feature learning in DGMs. Low-level features—such as textures, edges, and dominant colors—are learned rapidly at the start of training. These features are highly effective for distinguishing between disparate datasets.

As a model matures, it begins to incorporate complex semantic features. This increased "maturity" can lead to "support overlap," where the statistical distributions of ID and OOD gradient norms begin to converge, making them harder to separate. Thus, the focus on high-level semantics in later training stages can paradoxically degrade OOD detection capabilities.

5. Conclusion

This research highlights a "double win" for OOD detection: using immature models provides superior or equivalent performance compared to mature models while requiring significantly less training time and energy. Future work should investigate whether these benefits extend beyond the GLOW architecture to other generative models like Diffusion Models or to different data modalities such as text.

Summary updated

BibTeX
@misc{montazeran2025ood,
  author = {Behrooz Montazeran and Ullrich Köthe},
  title = {{OOD Detection with immature Models}},
  year = {2025},
  eprint = {2502.00820},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.00820},
}
Figure