Abstract
We propose and investigate new complementary methodologies for estimating predictive variance networks in regression neural networks. We derive a locally aware mini-batching scheme that result in sparse robust gradients, and show how to make unbiased weight updates to a variance network. Further, we formulate a heuristic for robustly fitting both the mean and variance networks post hoc. Finally, we take inspiration from posterior Gaussian processes and propose a network architecture with similar extrapolation properties to Gaussian processes. The proposed methodologies are complementary, and improve upon baseline methods individually. Experimentally, we investigate the impact on predictive uncertainty on multiple datasets and tasks ranging from regression, active learning and generative modeling. Experiments consistently show significant improvements in predictive uncertainty estimation over state-of-the-art methods across tasks and datasets.
Introduction
Deep learning models excel at point estimation but often struggle with uncertainty quantification. In regression, a model should ideally output both a mean prediction $\mu(x)$ and a predictive variance $\sigma^2(x)$. However, standard neural networks tend to be overconfident, underestimating noise in sparse data regions and failing to signal uncertainty when extrapolating beyond the training distribution. This paper addresses these limitations by introducing a suite of techniques designed to make variance estimation more reliable.
Methodology
The authors propose a "Combined" approach integrating four key innovations:
- Locality-Aware Mini-Batching: Variance is inherently a local property. Standard random mini-batching treats data points as independent, which can lead to unstable variance gradients. By sampling points along with their $k$-nearest neighbors and using the Horvitz-Thompson algorithm for unbiased re-weighting, the model gains the necessary local context for stable training.
- Mean-Variance (MV) Split Training: Optimizing mean and variance simultaneously can be problematic. The authors suggest an alternating optimization scheme: training the mean network while holding variance fixed, and then training the variance network conditioned on the current mean.
- Robust Likelihood with Inverse-Gamma Priors: Standard Maximum Likelihood Estimation (MLE) often leads to variance underestimation. By placing an Inverse-Gamma prior on the variance, the resulting predictive distribution follows a Student-t distribution, which is more robust to outliers and small sample sizes.
- Extrapolation Architecture: To address poor out-of-distribution behavior, the authors introduce an "extrapolation layer" inspired by Gaussian Processes (GPs). This architecture uses inducing points to ensure that as inputs move away from the training data, the model reverts to a high-variance prior state.
Experimental Evaluation
The proposed methodologies were tested on a variety of tasks, including toy regression, weather data calibration, 13 UCI benchmarks, and generative modeling (VAEs).
On toy datasets, the "Combined" model demonstrates a superior ability to capture heteroscedastic noise and provide sensible uncertainty bands compared to baselines like Monte Carlo Dropout or Deep Ensembles, often achieving lower error rates.
Key Results and Discussion
The "Combined" model achieved state-of-the-art performance, outperforming established methods on 10 out of 13 UCI datasets in terms of test set log-likelihood.
As shown in the figure above, the "Combined" method tracks the true standard deviation more accurately than standard NNs or BNNs, which often either underestimate variance or fail to capture its spatial variation. In active learning tasks, this improved uncertainty estimation allowed models to reach high accuracy with fewer labeled samples by more effectively identifying regions of high ignorance.
Conclusion
By treating variance as a local property and incorporating GP-inspired architectural constraints, the proposed methods significantly enhance the reliability of uncertainty quantification in neural networks. These techniques are complementary and provide a robust framework for training variance networks across diverse applications, from regression to generative modeling.
Summary updated