Bayesian Online Learning for Consensus Prediction

Samuel Showalter, Alex Boyd, Padhraic Smyth, Mark Steyvers

Abstract

Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.

Introduction

In high-stakes domains such as medical diagnosis or autonomous navigation, "ground truth" is often defined by the consensus of a panel of experts rather than a single objective fact. However, obtaining full consensus for every instance is economically and temporally expensive. While AI models provide "free" predictions, they are not always reliable. This paper introduces a Bayesian framework for online consensus prediction, aiming to predict the majority vote of a pool of human experts efficiently by strategically querying them only when necessary.

This approach addresses a more realistic setting than traditional "learning to defer" models by treating consensus as the target, acknowledging cost asymmetry, and operating in an online environment where model quality and expert disagreement are initially unknown.

Methodology

The proposed framework treats the AI model's prediction as a prior for the expert votes. The population of human opinions is modeled using a Dirichlet distribution, while the observed votes from a finite pool of $N$ experts are modeled via a Multivariate Hypergeometric distribution (sampling without replacement).

Probabilistic graphical model for online consensus prediction.

As illustrated in the graphical model above, the system integrates the AI's input ($f_t$) through three learnable parameters: $\theta$ (AI belief weight), $\phi$ (base uncertainty), and $\tau$ (per-class calibration). Two inference methods are proposed: FinExp (Finite Experts), which uses exact hypergeometric likelihood, and InfExp (Infinite Experts), a computationally efficient approximation. Decisions to query or predict are based on whether the "Expected Accuracy" of the current posterior exceeds a predefined threshold $\rho$. If the threshold is not met, the system queries another human expert.

Experimental Results

Evaluation on CIFAR-10H and ImageNet-16H demonstrates that the proposed methods, FinExp and InfExp, consistently outperform standard baselines like entropy-based querying or the Model Picker algorithm. A key strength of this approach is its robustness to distribution shifts.

Error rate and querying rate dynamics under distribution shift at timestep 1200.

As shown in the performance dynamics above, when the data distribution shifts (e.g., due to increased noise in ImageNet-16H at timestep 1200), the Bayesian posterior naturally reflects increased uncertainty. This triggers the system to automatically increase the querying rate, thereby maintaining significantly lower error rates compared to non-adaptive baselines. Furthermore, the learned $\tau$ parameters effectively identify classes where the AI model is less trustworthy, allowing for more targeted human intervention. Even when human experts become unavailable, the parameters learned during the active phase allow the AI model to perform better than its original baseline accuracy.

Conclusion

This work provides a principled Bayesian approach to human-AI collaboration where consensus is the target. By treating the AI as a "virtual expert" that informs a dynamic prior, the system achieves a superior trade-off between cost and accuracy. Future research will explore individual expert modeling (accounting for seniority or reliability), expansion to other modalities like text, and robustness against adversarial annotators in crowdsourcing pools.

Summary updated

Abstract

Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.
BibTeX
@inproceedings{showalter2024bayesian,
  author = {Samuel Showalter and Alex Boyd and Padhraic Smyth and Mark Steyvers},
  title = {{Bayesian Online Learning for Consensus Prediction}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=z4uOSsLZI2},
}
Figure