Quantifying Uncertainty in the Presence of Distribution Shifts

Yuli Slavutsky, David Blei

Deep neural networks are notoriously overconfident. While they can make highly accurate predictions on data similar to their training set, they often fail catastrophically—and silently—when faced with distribution shifts. For example, a medical diagnostic model trained on data from one hospital might become dangerously unreliable when deployed at another hospital with different patient demographics or equipment.

Classical Bayesian Neural Networks (BNNs) attempt to solve this overconfidence by placing a prior distribution over the model's weights. However, these priors are typically fixed and remain agnostic to the input distribution. Consequently, the resulting predictive uncertainty does not naturally increase just because a new test point lies in an unseen, out-of-distribution region. While "distance-aware" methods exist to flag such points, they often rely on fixed distance metrics that do not necessarily reflect how a shift actually impacts predictive performance.

To address this, researchers have proposed Variational Inference under Distribution Shift (VIDS), a Bayesian framework that explicitly accounts for covariate shifts to provide more reliable uncertainty estimates.

The VIDS Approach

The core innovation of VIDS breaks from traditional Bayesian inference in three key ways:

  1. Adaptive Prior: Instead of a fixed prior $p(\theta)$, VIDS employs an adaptive prior $p(\theta | x_{1:n}, x^*)$ that depends on both the training covariates and the new test point. This energy-based prior is designed to spread its mass over weight values that are "plausible" given the observed covariates, naturally increasing uncertainty for inputs far from the training distribution.
  2. Amortized Variational Inference: To make this adaptive prior computationally feasible, the authors use an inference network that takes test covariates as input and outputs the parameters of an approximate posterior. This allows the model to quickly adapt its uncertainty estimates for any new test point $x^*$.
  3. Synthetic Environments: Because the exact nature of test-time shifts is unknown during training, VIDS creates "synthetic environments" by drawing small bootstrap samples from the training data. These small samples naturally deviate from the overall population, simulating plausible covariate shifts. The model is then trained to perform well across all these simulated environments using a variance-penalty objective.
The VIDS graphical model introduces new dependencies (red arrows) where model parameters $\theta$ depend on both training and test covariates.

Key Insights

A major conceptual leap in VIDS is treating prior shift as a signal. If many different models fit the training data perfectly but disagree wildly on a new test point, the prior should reflect that ambiguity, leading to higher posterior uncertainty.

Furthermore, unlike distance-based methods that use fixed kernels (such as RBF), VIDS effectively learns which covariate shifts matter. It understands that a shift in a feature irrelevant to the target outcome should not necessarily trigger an increase in uncertainty. The use of small bootstrap samples acts as a clever "inverse bootstrap"—while standard bootstrapping seeks to represent the population, small samples are expected to be unrepresentative, providing a computationally free source of simulated distribution shifts.

Visualizing VIDS on a 1D classification task. As test points from a shifted distribution are introduced, the model adapts its decision boundaries and uncertainty estimates.

Evaluation and Results

The authors evaluated VIDS against state-of-the-art methods like SNGP, DUE, and DUL across several benchmarks:

  • Synthetic Data: In tasks with intentional "gaps" in the training data, VIDS successfully captured the correct variance structure, showing high uncertainty in missing regions while remaining confident at the edges.
  • Image Classification: On Corrupted CIFAR-10 (testing robustness against blurs and contrast changes) and Celeb-A (testing subpopulation shifts), VIDS consistently achieved the highest classification accuracy and superior calibration compared to baselines.
  • Tabular Regression: On standard UCI datasets (Boston, Concrete, Wine) with simulated shifts, VIDS achieved the lowest average Root Mean Squared Error (RMSE) and maintained consistently low variance in its performance.

Limitations and Future Directions

While highly effective, VIDS does introduce computational overhead. For small models, the per-environment optimization can be slower than standard methods, though this gap narrows for larger architectures. Additionally, the current implementation relies on pre-trained embeddings. Developing a fully end-to-end version that learns representations and uncertainty simultaneously remains an exciting avenue for future work. Finally, exploring more sophisticated ways to generate synthetic environments could help the model cover even more extreme or structured distribution shifts.

Summary updated

Abstract

Neural networks make accurate predictions but often fail to provide reliable uncertainty estimates, especially when test-time covariates differ from those seen during training, as occurs with selection bias or shifts over time. To address this, we propose a Bayesian framework for uncertainty estimation that explicitly accounts for covariate shifts. Unlike conventional approaches that rely on fixed priors, a key idea of our method is an adaptive prior, conditioned on both training and new covariates. This prior naturally increases uncertainty for inputs that lie far from the training distribution in regions where predictive performance is likely to degrade. To efficiently approximate the resulting posterior predictive distribution, we employ amortized variational inference. Finally, we construct synthetic environments by drawing small bootstrap samples from the training data, simulating a range of plausible covariate shifts using only the original dataset. We evaluate our method on both synthetic and real-world data, demonstrating that it yields substantially improved uncertainty estimates under distribution shift compared to existing approaches.
BibTeX
@inproceedings{slavutsky2025quantifying,
  author = {Yuli Slavutsky and David Blei},
  title = {{Quantifying Uncertainty in the Presence of Distribution Shifts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=04p7u1gIsv},
}
Figure