Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift

Alex J. Chan, Ahmed M. Alaa, Zhaozhi Qian, Mihaela van der Schaar

Introduction

Modern neural networks are increasingly deployed in high-stakes domains like healthcare, where "knowing what they don't know" is as critical as predictive accuracy. While Bayesian Neural Networks (BNNs) are designed to provide uncertainty estimates, they often become dangerously overconfident when faced with covariate shift—a scenario where the input distribution $p(x)$ changes between training and deployment, even if the underlying relationship $p(y|x)$ remains stable.

Standard techniques like Monte Carlo (MC) Dropout often fail to account for these shifts, leading to miscalibrated uncertainty that can mislead human experts. This paper introduces Transductive Dropout, a novel framework that leverages unlabelled data from the target domain to "teach" the model to be more humble in unfamiliar regions of the feature space.

Methodology: Transductive Dropout

The core innovation of Transductive Dropout is the use of unlabelled target data as a regularizer during the training phase. The authors propose a multi-faceted approach to refine the model's variational posterior:

  1. Posterior Regularization: A regularization term is added to the Variational Inference objective (the ELBO). This term explicitly penalizes low variance in regions populated by target data but sparse in source data, forcing the model to express higher uncertainty where it lacks direct experience.
  2. Input-Dependent Dropout Rates: Unlike standard dropout, which uses a fixed rate (e.g., 0.5), Transductive Dropout parameterizes the dropout rate as a function of the input $x$. An auxiliary neural network outputs a per-point dropout rate, allowing the model to adapt its regularization level based on the local data density.
  3. The Auxiliary Task: The model is trained to minimize standard prediction loss on labelled data while simultaneously performing an auxiliary task. It uses its own predictive variance to "classify" whether a data point belongs to the source or target distribution. This mechanism turns uncertainty quantification into a supervised signal: if a point is from the target set and looks different from the source, the network is forced to increase its variance.
The Transductive Dropout training framework, incorporating an auxiliary loss based on predictive distribution to regularize the model using unlabelled data.

Experimental Evaluation

The authors evaluated Transductive Dropout across synthetic and real-world datasets to demonstrate its calibration benefits.

Toy 1D Regression

In synthetic tests, standard MC Dropout (MCDP) typically produces narrow uncertainty bands even in regions far from the training data. In contrast, Transductive Dropout successfully "spreads" its uncertainty across the regions where unlabelled target data is present, providing a much more realistic estimate of the model's limitations.

Comparison of uncertainty estimation on a 1D regression task: (a) standard MC Dropout (MCDP) fails to capture uncertainty in shifted regions, while (b) Transductive Dropout leverages unlabelled data to produce well-calibrated uncertainty.

Clinical Utility: Prostate Cancer Mortality

The method was tested on a high-stakes task: predicting mortality for prostate cancer patients. The model was trained on US data (SEER program) and tested on UK data (Prostate Cancer UK), representing a significant covariate shift due to different healthcare systems and demographics.

  • Predictive Performance: Transductive Dropout achieved an AUROC of 0.861, significantly outperforming standard MC Dropout (0.729) and Concrete Dropout (0.791).
  • Calibration: The model's ability to "know when it was wrong" was measured by using variance to predict errors. It achieved an AUROC of 0.857 for error prediction, compared to just 0.730 for standard BNNs.
  • Impact: At a 15% risk threshold, this improved calibration allowed for the correct identification of 189 more patients for treatment who were otherwise misclassified as low-risk by baseline models.

Conclusion and Future Work

Transductive Dropout demonstrates that unlabelled data is a powerful tool for mapping the boundaries of a model's expertise. By treating uncertainty as a learnable, input-dependent feature, the framework significantly improves reliability under covariate shift. While the current method requires a batch of unlabelled data during training (the "transductive" requirement), future work may explore extending these principles to streaming data or frequentist ensembles, further bridging the gap between theoretical BNNs and industrial applications.

Summary updated

Abstract

Modern neural networks have proven to be powerful function approximators, providing state-of-the-art performance in a multitude of applications. They however fall short in their ability to quantify confidence in their predictions - this is crucial in high-stakes applications that involve critical decision-making. Bayesian neural networks (BNNs) aim at solving this problem by placing a prior distribution over the network's parameters, thereby inducing a posterior distribution that encapsulates predictive uncertainty. While existing variants of BNNs based on Monte Carlo dropout produce reliable (albeit approximate) uncertainty estimates over in-distribution data, they tend to exhibit over-confidence in predictions made on target data whose feature distribution differs from the training data, i.e., the covariate shift setup. In this paper, we develop an approximate Bayesian inference scheme based on posterior regularisation, wherein unlabelled target data are used as "pseudo-labels" of model confidence that are used to regularise the model's loss on labelled source data. We show that this approach significantly improves the accuracy of uncertainty quantification on covariate-shifted data sets, with minimal modification to the underlying model architecture. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
BibTeX
@inproceedings{chan2020unlabelled,
  author = {Alex J. Chan and Ahmed M. Alaa and Zhaozhi Qian and Mihaela van der Schaar},
  title = {{Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=fTGk219dBIW},
}
Figure