TL;DR
While Bayesian deep learning relies heavily on variational inference (VI) to quantify epistemic uncertainty (EU), standard VI frequently underestimates it. This paper grounds practical EU metrics (predictive variance and mutual information) in PAC-Bayesian excess risk theory, proving they act as lower bounds on generalization error. Leveraging this insight, the authors propose rBER—a modified variational objective that decouples prediction risk from uncertainty penalization, significantly improving uncertainty estimation in regression and bandit tasks.
What Problem Does This Paper Address?
In safety-critical machine learning, distinguishing between aleatoric uncertainty (irreducible data noise) and epistemic uncertainty (EU, uncertainty stemming from a lack of training data) is essential. Bayesian deep learning models estimate EU through posterior approximations like variational inference (VI).
However, theoretical analysis of EU has traditionally assumed exact Bayesian inference under correctly specified models in asymptotic settings (infinite data). In practice, deep networks rely on approximate posteriors, operate in finite-sample regimes, and often yield predictors whose variance severely underestimates EU. Until now, there was no non-asymptotic, algorithm-dependent theoretical guarantee linking practical EU measures (such as predictive variance) to generalization error in approximate inference.
Methods
The authors bridge PAC-Bayesian generalization theory and information-theoretic excess risk:
- Bayesian Excess Risk (BER): They define BER by assuming the test data generating process is governed by the approximate posterior $q(\theta|z^N)$. They prove that for squared error loss, BER exactly equals the predictive variance $\text{Var}_{\theta|z^N}(f_\theta(x))$, and for log loss, it equals conditional mutual information $I(\theta; Y | x, z^N)$.
-
Risk Decomposition: For squared loss, test error decomposes cleanly into Prediction Risk (PR) and BER: $$\mathcal{R}^{(2)} = \text{PR}^{(2)} + \text{BER}^{(2)}$$ The authors prove that $\text{PER}^{(2)} + \text{BER}^{(2)} \le \mathcal{O}(N^{-\alpha} \ln N)$ via PAC-Bayesian bounds, establishing formal non-asymptotic convergence rates for common EU metrics.
-
Deconstructing the Standard VI Loss: The standard VI objective for Gaussian likelihood decomposes into fitting the mean (PR) while penalizing variance (BER) with equal weight. This rigid coupling forces the model to aggressively shrink variance, driving EU underestimation.
- rBER (Regularized Bayesian Excess Risk VI): To fix this, they introduce a tunable hyperparameter $\lambda \in (0, 1]$ to scale the BER penalty relative to prediction risk: $$\mathcal{L}_{\text{rBER}}(\lambda) = \frac{1}{N}\sum_{i=1}^N \left[ \frac{|y_i - \mathbb{E}_q[f_\theta(x_i)]|^2}{2v^2} + \lambda \frac{\text{Var}_q(f_\theta(x_i))}{2v^2} \right] + \frac{\text{KL}(q(\theta)||p(\theta))}{N}$$ Setting $\lambda < 1$ prevents over-penalizing predictive variance. They implement this objective within Particle Variational Inference (PVI).
Datasets
The authors evaluate their theoretical bounds and proposed rBER method on:
- Synthetic 1D Regression: $y = 0.5x^3 + \epsilon$ with $x, \epsilon \sim \mathcal{N}(0, 1)$.
- Real-World Depth Estimation: The NYU Depth v2 dataset (RGB-to-depth indoor scenes) using a U-Net architecture.
- UCI Regression Benchmarks: 6 tabular datasets (Concrete, Boston, Wine, Power, Yacht, Protein).
- Contextual Bandits: 4 classification-to-bandit benchmark datasets from the bandit showdown suite (Mushroom, Financial, Statlog, CoverType).
Note: No new datasets were introduced.
Key Insights and Ideas
- Underestimation is Structurally Baked In: Proving that $\text{BER}^{(2)} \le \text{Excess Risk} \le \text{Test Error}$ reveals mathematically why variance-based EU measures act as lower bounds and systematically underestimate uncertainty in practice.
- Fast Non-Asymptotic Convergence: Under mild sub-Gaussian assumptions, mutual information and predictive variance converge toward zero as sample size grows at rate $\mathcal{O}(\sqrt{\ln N / N^\alpha})$, resolving open questions about finite-sample EU convergence.
- Why Alternative Losses Worked: Prior empirical successes like $\alpha$-divergence dropout and second-order PAC methods succeeded precisely because their entropic risk formulations implicitly weaken the penalty on BER compared to standard VI.
Key Results
- Bound Validation: On synthetic data and NYU Depth v2, empirical BER and prediction excess risk remained strictly bounded by the test error, converging at the theoretical $\mathcal{O}(N^{-1/2})$ rate with Spearman rank correlations exceeding $0.97$.
- Calibrated Prediction Intervals: On UCI regression benchmarks (evaluating 95% Prediction Interval Coverage Probability, PICP), rBER($\lambda = 0.05$) achieved near-nominal coverage while maintaining narrow interval widths (MPIW) and matching RMSE:
- Concrete: PICP of 0.95 (MPIW 0.25) vs. 0.82 (MPIW 0.13) for f-SVGD and 0.87 for standard VAR.
- Protein: PICP of 0.96 (MPIW 0.86) vs. 0.53 (MPIW 0.24) for f-SVGD.
- Contextual Bandits: In Thompson sampling tasks, balancing exploration via rBER($0.05$) cut normalized cumulative regret dramatically:
- Mushroom: 0.021 vs. 0.043 (f-SVGD) and 0.037 ($\text{PAC}_E^2$).
- Financial: 0.075 vs. 0.154 (f-SVGD) and 0.189 ($\text{PAC}_E^2$).
- CoverType: 0.290 vs. 0.372 (f-SVGD) and 0.396 ($\text{PAC}_E^2$).
Limitations and Future Work
- Well-Specified Model Assumptions: The core theoretical guarantees require the conditional mean or model class to be well-specified (e.g., $\mathbb{E}[Y|x] = f_{\theta^*}(x)$).
- Exclusion of Aleatoric Uncertainty: The paper focuses solely on epistemic uncertainty; full predictive calibration requires joint modeling of both aleatoric and epistemic uncertainties.
- Future Directions: The authors highlight extending this excess risk framework to analyze and improve evidential deep learning as a promising next step.
Summary updated