Posterior Annealing: Fast Calibrated Uncertainty for Regression

Uddeshya Upadhyay, Jae-Myung Kim, Cordelia Schmid, Bernhard Schölkopf, Zeynep AkataarXiv 2023 · DOI · arXiv

Introduction

Deep regression models often face a trade-off between optimization efficiency and uncertainty calibration. Standard Maximum Likelihood Estimation (MLE) using Gaussian negative log-likelihood (NLL) frequently suffers from a "flat" gradient landscape when errors are large and a "steep" landscape when errors are small, leading to slow convergence and potential gradient instability. Furthermore, these models often produce poorly calibrated uncertainty estimates ($\sigma$), necessitating complex post-hoc calibration steps that may not scale to high-dimensional data.

Likelihood Annealing (LIKA) is proposed as a novel training scheme to address these dual challenges. By introducing a temperature-dependent "improper" likelihood that evolves during training, LIKA enables faster convergence and produces inherently well-calibrated models.

Methodology: Likelihood Annealing (LIKA)

LIKA modifies the standard Gaussian loss by incorporating two temperature-controlled regularizers, $T_2$ and $T_3$. The $T_2$ term encourages the model's predictions to stay close to the ground truth, while the $T_3$ term explicitly penalizes discrepancies between the predicted uncertainty ($\sigma$) and the actual residual.

The core of the method is the annealing process. Training begins at high temperatures (e.g., $T_2, T_3 = 100$), which reshapes the loss surface to provide steeper gradients early in training. As training progresses, these temperatures are exponentially decayed toward zero. By the end of the process, the loss function smoothly transitions back to the standard Gaussian NLL, but the model has already been guided toward a well-calibrated state.

Evolution of the loss landscape during the annealing process as temperatures $T_2$ and $T_3$ decrease.

The authors also derive "LIKA-Norm," a normalized version of this likelihood, ensuring theoretical consistency while maintaining the practical benefits of the simpler formulation.

Experimental Evaluation

The effectiveness of LIKA was evaluated across five diverse regression tasks:

  1. Chaotic System Denoising: 1D trajectory recovery using the Lorenz Attractor.
  2. Physical Properties of Molecules: Predicting dipole moments using Graph Neural Networks (Atom3D).
  3. House Price Prediction: Low-dimensional benchmark using MLPs (Boston Housing).
  4. Natural Image Super-resolution: Mapping low-resolution images to high-resolution (DIV2K).
  5. Medical Image Translation: Converting T1-weighted MRI scans to T2-weighted scans using U-Nets (IXI Dataset).

Results and Discussion

LIKA demonstrates significant improvements in both training speed and calibration quality. Empirically, the method achieves a speedup of 1.5x to 6x compared to standard NLL and other state-of-the-art baselines. For instance, in super-resolution tasks, LIKA converges in approximately 4,000 epochs, whereas baselines require over 8,000.

Training curves showing Loss and Expected Calibration Error (ECE) across three tasks. LIKA (orange) demonstrates significantly faster convergence and superior calibration compared to baselines.

In terms of calibration, LIKA consistently achieves the lowest Uncertainty Calibration Error (UCE). In medical imaging tasks, LIKA’s UCE was found to be nearly 10x lower than that of standard NLL. Furthermore, LIKA exhibits robust performance on Out-of-Distribution (OOD) data, maintaining reliable calibration even when faced with noise levels not seen during training—a scenario where traditional Bayesian methods like ensembles or dropout often become overconfident.

Conclusion

Likelihood Annealing proves that treating calibration as a dynamic training constraint rather than a post-hoc adjustment can yield "honesty" in model predictions without sacrificing accuracy. By fixing the gradient landscape of deep regression, LIKA offers a scalable and efficient solution for high-dimensional tasks where traditional uncertainty quantification methods struggle. Future work may involve further refining hyperparameter decay rates and exploring the integration of carefully crafted priors to further enhance performance.

Summary updated

Abstract

Recent advances in deep learning have shown that uncertainty estimation is becoming increasingly important in applications such as medical imaging, natural language processing, and autonomous systems. However, accurately quantifying uncertainty remains a challenging problem, especially in regression tasks where the output space is continuous. Deep learning approaches that allow uncertainty estimation for regression problems often converge slowly and yield poorly calibrated uncertainty estimates that can not be effectively used for quantification. Recently proposed post hoc calibration techniques are seldom applicable to regression problems and often add overhead to an already slow model training phase. This work presents a fast calibrated uncertainty estimation method for regression tasks called Likelihood Annealing, that consistently improves the convergence of deep regression models and yields calibrated uncertainty without any post hoc calibration phase. Unlike previous methods for calibrated uncertainty in regression that focus only on low-dimensional regression problems, our method works well on a broad spectrum of regression problems, including high-dimensional regression.Our empirical analysis shows that our approach is generalizable to various network architectures, including multilayer perceptrons, 1D/2D convolutional networks, and graph neural networks, on five vastly diverse tasks, i.e., chaotic particle trajectory denoising, physical property prediction of molecules using 3D atomistic representation, natural image super-resolution, and medical image translation using MRI.
BibTeX
@misc{upadhyay2023posterior,
  author = {Uddeshya Upadhyay and Jae-Myung Kim and Cordelia Schmid and Bernhard Schölkopf and Zeynep Akata},
  title = {{Posterior Annealing: Fast Calibrated Uncertainty for Regression}},
  year = {2023},
  eprint = {2302.11012},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.11012},
}
Figure