Epistemic Uncertainty and Excess Risk in Variational Inference

Futoshi FutamiAISTATS 2025 · OpenReview

Abstract

Bayesian inference is widely used for its ability to assess epistemic uncertainty (EU) in predictions. However, computational complexity often necessitates the use of variational inference (VI). While metrics like posterior predictive variance and conditional mutual information are commonly used to estimate EU, they have lacked a comprehensive theoretical foundation. This paper bridges the gap by establishing a novel relationship between EU metrics and "excess risk," enabling a convergence analysis based on PAC-Bayesian theory. We demonstrate that standard VI objectives often over-regularize EU, leading to underestimation. Consequently, we propose a novel objective function, regularized Bayesian Excess Risk (rBER), which directly optimizes both prediction and EU. Experimental results show significant improvements in EU estimation and decision-making tasks.

Introduction

In Bayesian deep learning, Epistemic Uncertainty (EU) represents the uncertainty stemming from limited training data. A robust model should signal high EU for data points far from its training distribution. While practitioners frequently use predictive variance or mutual information to quantify this, the theoretical justification for these metrics—and their convergence behavior as data increases—has remained elusive. Furthermore, standard VI methods are known to produce overconfident models that underestimate uncertainty. This research provides the theoretical "missing link" and introduces a refined objective function to address these shortcomings.

The core contribution of this work is the introduction of Bayesian Excess Risk (BER). Unlike standard excess risk, which compares a model to an unattainable optimal, BER evaluates the model's predictive distribution against itself as a ground truth.

Theorem 1 proves that standard EU metrics (predictive variance for squared loss and mutual information for log loss) are mathematically equivalent to this BER. This equivalence allows the application of PAC-Bayesian theory to derive generalization bounds and analyze EU behavior.

Convergence of Bayesian Excess Risk (BER) and other risk metrics with increasing sample size N across different Bayesian approximations.

As shown in Figure 1, the theoretical BER tracks closely with other risk metrics and decreases at a rate of $O(\sqrt{\log N / N})$ as the number of samples $N$ increases. This provides the first formal proof of convergence rates for common EU metrics.

The rBER Objective Function

Standard VI objective functions typically bundle prediction error and uncertainty regularization. The author demonstrates that this leads to the uncertainty term being "squeezed," causing the model to become overconfident. To counter this, the paper proposes the regularized Bayesian Excess Risk (rBER) objective:

$$ \mathcal{L}_{rBER} = \text{Prediction Error} + \lambda \cdot \text{Uncertainty Regularization} $$

By introducing the hyperparameter $\lambda < 1$, rBER allows for explicit control over uncertainty weighting, preventing the over-regularization inherent in standard VI.

Experimental Validation

The framework was tested across synthetic cubic regression, six UCI regression benchmarks (e.g., Concrete, Yacht, Protein), and four contextual bandit datasets (e.g., Mushroom, Statlog).

Key Results

  • Calibration and Coverage: In regression tasks, rBER significantly improved the Prediction Interval Coverage Probability (PICP). While standard VI often fell short of the 95% target, rBER consistently achieved it without excessively increasing the Mean Prediction Interval Width (MPIW).
  • Exploration in Bandits: In contextual bandit experiments, rBER achieved the lowest cumulative regret. The improved EU estimation allowed the model to balance exploration and exploitation more effectively.
  • Theoretical Validation: Numerical results confirmed a Spearman Rank Correlation of at least 0.97 between theoretical excess risks and practical EU metrics.
Impact of the regularization parameter λ on predictive performance (RMSE, NLL) and uncertainty calibration (PICP, MPIW).

Figure 3 illustrates the trade-offs managed by $\lambda$. As $\lambda$ increases, we observe a marked improvement in calibration (PICP) and a reduction in interval width (MPIW), while maintaining stable predictive accuracy (RMSE/NLL).

Discussion and Limitations

This work provides a rigorous foundation for EU estimation, showing that EU metrics act as a lower bound for true excess risk. While a significant advancement, the current theory is limited to squared and log loss functions. Future work will explore extending these bounds to robust statistics and investigating the behavior of EU within mean-field approximations of VI. Additionally, the interplay between Aleatoric (inherent noise) and Epistemic uncertainty remains a critical area for achieving perfect model calibration.

Summary updated

Abstract

Bayesian inference is widely used in practice due to its ability to assess epistemic uncertainty (EU) in predictions. However, its computational complexity necessitates the use of approximation methods, such as variational inference (VI). When estimating EU within the VI framework, metrics such as the variance of the posterior predictive distribution and conditional mutual information are commonly employed. Despite their practical importance, these metrics lack comprehensive theoretical analysis. In this paper, we investigate these EU metrics by providing their novel relationship to excess risk, which allows for a convergence analysis based on PAC-Bayesian theory. Based on these analyses, we then demonstrate that some existing objective functions of VI regularize EU metrics in different ways leading to different performance in EU evaluation. Finally, we propose a novel objective function for VI that directly optimizes both prediction and EU under the PAC-Bayesian framework. Experimental results indicate that our algorithm significantly improves EU estimation compared to existing VI methods.
BibTeX
@inproceedings{futami2025epistemic,
  author = {Futoshi Futami},
  title = {{Epistemic Uncertainty and Excess Risk in Variational Inference}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
  url = {https://openreview.net/forum?id=axbN98w6pe},
}
Figure