Abstract
While Epistemic Uncertainty (EU) is a cornerstone of reliable Bayesian Deep Learning, its theoretical foundations—particularly within the context of approximate inference—have remained underdeveloped. This paper establishes the first non-asymptotic theoretical analysis of EU for approximate Bayesian inference by connecting it to PAC-Bayesian generalization theory. We demonstrate that common EU measures are mathematically equivalent to Bayesian Excess Risk (BER) and prove that EU acts as a lower bound for test error. Building on these insights, we introduce the regularized Bayesian Excess Risk (rBER) objective, which generalizes standard Variational Inference (VI) to provide better-calibrated uncertainty estimates. Our experiments across regression tasks and contextual bandits confirm that rBER significantly mitigates the overconfidence typical of standard VI.
Introduction
In safety-critical domains such as autonomous driving and medical diagnostics, a model's ability to quantify its own ignorance—known as Epistemic Uncertainty (EU)—is as vital as its predictive accuracy. Unlike aleatoric uncertainty, which arises from inherent noise in data, EU stems from a lack of training data and should ideally vanish as more information is acquired.
In modern deep learning, calculating the exact posterior distribution required for Bayesian uncertainty is computationally intractable. Consequently, practitioners rely on Variational Inference (VI) to approximate the posterior. However, while the generalization properties of VI's predictive performance are well-studied, the theoretical guarantees for its uncertainty estimates have been largely absent. This gap often leads to models that are well-calibrated in theory but dangerously overconfident in practice. This paper bridges this gap by introducing a framework based on Bayesian Excess Risk (BER).
Theoretical Framework: Bayesian Excess Risk
The authors propose that the quality of uncertainty estimation can be rigorously analyzed through the lens of "excess risk"—the difference between a model's actual error and the theoretical minimum error. They identify a fundamental link between standard EU measures and the Bayesian Excess Risk (BER).
Linking EU to Generalization
For common loss functions, such as log loss and squared loss, the researchers prove that standard measures of EU—specifically Mutual Information and Predictive Variance—are equivalent to the BER. This equivalence allows the application of PAC-Bayesian theory to uncertainty estimation.
One of the most significant theoretical findings is that EU measurements serve as a lower bound for the test error. This provides a mathematical explanation for the common phenomenon of overconfidence in deep learning: because the uncertainty estimate is inherently smaller than the actual risk, the model naturally underestimates its potential for error unless specifically regularized.
Convergence Rates
The study derives non-asymptotic bounds showing that EU converges to zero at a rate of $O(\sqrt{\ln N / N^\alpha})$. Notably, this convergence is faster than the convergence of the total excess risk. This suggests that as data volume increases, our "confidence" in the model's parameters improves more rapidly than the model's overall predictive accuracy.
The rBER Algorithm
Standard Variational Inference (VI) objectives typically force a rigid relationship between prediction and uncertainty. The authors demonstrate that the standard Evidence Lower Bound (ELBO) used in VI is actually a special case of their broader BER framework where a regularization hyperparameter, $\lambda$, is set to 1.
To address the overconfidence inherent in standard VI, the authors propose regularized Bayesian Excess Risk (rBER). This modified objective function introduces $\lambda$ as a tunable parameter:
- When $\lambda = 1$, the model behaves like standard VI.
- When $\lambda < 1$, the regularization on uncertainty is "loosened," preventing the posterior from collapsing prematurely and allowing for more conservative, better-calibrated uncertainty estimates.
Experimental Results
The theoretical framework was validated through extensive testing on toy cubic regression, the NYU Depth v2 computer vision dataset, and the UCI regression suite.
Calibration and Accuracy
In the UCI regression benchmarks, rBER achieved superior Prediction Interval Coverage Probability (PICP) compared to standard VI and Deep Ensembles. This indicates that the "error bars" generated by rBER are more likely to contain the true data points. Furthermore, numerical experiments showed a Spearman Rank Correlation of at least 0.97 between the theoretical BER and actual test error, confirming BER as a highly reliable proxy for model performance.
Decision Making in Contextual Bandits
The practical utility of improved EU was tested in contextual bandit tasks (using Mushroom, Financial, and CoverType datasets). In these scenarios, an agent must balance exploring new options with exploiting known rewards. Because rBER provides a more accurate grasp of the model's own ignorance, it outperformed existing methods like f-SVGD in terms of cumulative regret, demonstrating that better uncertainty leads to more efficient exploration.
Discussion and Limitations
This work provides a rigorous foundation for uncertainty in approximate inference, yet some limitations remain. The current theorems assume a "well-specified" model, where the true data-generating process exists within the model's hypothesis space. In practice, deep learning models are often misspecified. Future work will focus on extending these bounds to misspecified regimes, automating the selection of the $\lambda$ hyperparameter, and exploring the relationship between BER and other frameworks like Evidential Learning.
By framing epistemic uncertainty as a form of excess risk, this research provides both a diagnostic tool for understanding overconfidence and a practical algorithm for building more reliable AI systems.
Summary updated