Out of Distribution Detection, Generalization, and Robustness Triangle with Maximum Probability Theorem

Amir Emad Marvasti, Ehsan Emad Marvasti, Ulas BagciarXiv 2022 · DOI · arXiv

Introduction

Deep learning models are often criticized for their "brittleness." They frequently overfit to training data, fail under slight input noise, and struggle to identify out-of-distribution (OOD) samples. Traditionally, researchers have addressed generalization, robustness, and OOD detection as separate challenges. This paper introduces a unified framework based on the Maximum Probability Theorem (MPT) to bridge these gaps, forming what the authors call the "Robustness Triangle."

Methodology: The Maximum Probability Theorem

The core of this research is the MPT, which treats a neural network model as a probabilistic "event" rather than just a set of parameters. Unlike Bayesian approaches that require complex and often arbitrary priors on hidden weights, MPT focuses on the probability of the observables (inputs and outputs). By maximizing a mathematical upper bound on the model's probability, the authors create a "black-box" regularization scheme.

Key technical components include:

  • $\alpha$-Parametrization: A "softmin" function is used to make the MPT bound differentiable, where the parameter $\alpha$ controls the strength of the regularization.
  • Energy-Based Models (EBMs): By treating standard CNNs as EBMs, the network learns the joint distribution of inputs and labels, which is critical for effective OOD detection.
  • Joint-Intersection (JI) Loss: This novel loss function re-weights gradients based on "energy," preventing the model from becoming overconfident and suppressing the probability of the most probable states.

Generalization and Stability

To validate the framework, the authors trained 1,080 models across CIFAR10, CIFAR100, and MNIST. A significant finding was that increasing the $\alpha$ parameter led to remarkably more stable training. As $\alpha$ increases, the variance in test accuracy across different runs decreases, and the model converges to higher performance levels than standard Conditional Cross-Entropy.

Figure 1: Test accuracy across CIFAR10, CIFAR100, and MNIST as a function of the regularization parameter alpha. Note the increased stability and performance of the Joint-Intersection (JI) loss as alpha increases.

On the CIFAR10 dataset, the JI loss derived from MPT achieved a maximum test accuracy of 0.812, significantly higher than the 0.757 achieved by standard Conditional Cross Entropy.

Robustness and OOD Detection

The MPT framework successfully links generalization with robustness. Models regularized via MPT showed higher Area Under the Curve (AUC) scores when subjected to increasing levels of input noise, particularly on the CIFAR100 dataset. This suggests that the regularization forced the model to learn more robust features that are less susceptible to perturbations.

Figure 5: AUC test accuracy (robustness) for varying loss functions across different datasets. MPT-based methods show superior resistance to performance degradation under noise.

For OOD detection, the authors introduced the Max Energy Score ($S_{ME}$). Despite using a simple 12-layer vanilla CNN, the EBM+MPT approach achieved an AUROC of 0.866 on OOD tasks (using datasets like SVHN and Fashion MNIST as unseen data). This performance outperformed several established baselines, demonstrating the power of the unified approach even without high-capacity modern architectures.

Limitations and Future Work

While the results are promising, the study focused on relatively simple CNNs to avoid interference from modern techniques like Batch Normalization or Dropout. Future research will explore how MPT scales to massive architectures like Transformers or ResNets. Additionally, while MPT reduces the need for weight priors, it introduces the $\alpha$ hyperparameter, which requires further study for optimal tuning. The authors also aim to explore combining MPT with explicit parameter priors to potentially stack the benefits of both Bayesian and MPT approaches.

Summary updated

Abstract

Maximum Probability Framework, powered by Maximum Probability Theorem, is a recent theoretical development in artificial intelligence, aiming to formally define probabilistic models, guiding development of objective functions, and regularization of probabilistic models. MPT uses the probability distribution that the models assume on random variables to provide an upper bound on the probability of the model. We apply MPT to challenging out-of-distribution (OOD) detection problems in computer vision by incorporating MPT as a regularization scheme in the training of CNNs and their energy-based variants. We demonstrate the effectiveness of the proposed method on 1080 trained models, with varying hyperparameters, and conclude that the MPT-based regularization strategy stabilizes and improves the generalization and robustness of base models in addition to enhanced OOD performance on CIFAR10, CIFAR100, and MNIST datasets.
BibTeX
@misc{marvasti2022out,
  author = {Amir Emad Marvasti and Ehsan Emad Marvasti and Ulas Bagci},
  title = {{Out of Distribution Detection, Generalization, and Robustness Triangle with Maximum Probability Theorem}},
  year = {2022},
  eprint = {2203.12145},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.12145},
}
Figure