Bayesian Online Learning for Consensus Prediction

Samuel Showalter, Alex J. Boyd, Padhraic Smyth, Mark Steyvers

Abstract

In high-stakes domains like medicine or autonomous driving, ground truth is often defined by the consensus of human experts rather than an objective oracle. This paper introduces a Bayesian framework for online classification that treats AI predictions as priors and human votes as samples from a multivariate hypergeometric distribution. By dynamically learning model reliability and expert consensus, the proposed method minimizes expensive human queries while maximizing predictive accuracy.

Introduction

In many critical applications, "absolute truth" is frequently unavailable, leading practitioners to rely on the collective judgment of expert panels. While AI models provide rapid predictions, human expertise is costly and slow. Existing "learning to defer" frameworks typically assume a single, infallible ground truth. This research addresses the more realistic challenge of predicting the plurality vote of a finite expert pool under budget constraints, where the AI's accuracy and expert reliability must be learned on the fly.

Methods: A Generative Bayesian Model

The core of the approach is a generative Bayesian model that estimates expert consensus. The system treats observed human votes as a sub-sample of the total pool, modeled using a multivariate hypergeometric distribution to capture the nature of sampling without replacement from a finite group.

Probabilistic graphical model of the proposed framework.

The framework incorporates a pre-trained AI model $f(x)$ to inform the prior. It introduces learned parameters—$\theta, \phi, \tau$—to scale model confidence and perform per-class calibration. Two variants are proposed:

  • FinExp: An exact model for a finite number of experts.
  • InfExp: A limiting case approximation assuming an infinite expert pool, which simplifies the model into a Dirichlet-Multinomial framework that is easier to optimize.

Decisions to query humans are made dynamically: if the posterior confidence in the consensus is below a threshold $\rho$, an additional expert is queried; otherwise, the system stops and predicts.

Experimental Evaluation

The framework was evaluated on two benchmark datasets:

  • CIFAR-10H: 10,000 images with 50 human labels each.
  • ImageNet-16H: 4,800 images with 6 human annotations each, including variations with phase noise.

Key Results

The Bayesian methods consistently outperformed standard active learning baselines, such as Entropy-based querying and the "Model Picker" algorithm.

Error rate vs. query budget on CIFAR-10H.

On CIFAR-10H, the proposed methods achieved significantly lower error rates for the same query budget. Notably, the InfExp approximation performed remarkably well even for small expert pools, offering a computationally efficient alternative to the exact model. Furthermore, the system demonstrated an "accuracy boost" in query-free scenarios, improving the base model's accuracy by up to 3% through better calibration based on historical human feedback.

Robustness and Online Calibration

A critical advantage of this framework is its ability to handle distribution shifts. By learning the $\tau$ parameter (per-class trust), the system identifies which classes the AI struggles with and automatically increases human querying for those instances. When data becomes noisier—simulating sensor failure—the Bayesian framework quickly adjusts its priors, maintaining accuracy where standard methods degrade.

Conclusion

This work provides a robust framework for integrating AI predictions with human expert consensus in online settings. By treating the AI as a prior and human votes as statistical samples, the system achieves a superior balance between cost and accuracy. Future research will explore individual expert identities and the application of this framework to non-image modalities and open-set tasks.

Summary updated

Abstract

Given a pre-trained classifier and multiple human experts, we investigate the task of online classification where model predictions are provided for free but querying humans incurs a cost. In this practical but under-explored setting, oracle ground truth is not available. Instead, the prediction target is defined as the consensus vote of all experts. Given that querying full consensus can be costly, we propose a general framework for online Bayesian consensus estimation, leveraging properties of the multivariate hypergeometric distribution. Based on this framework, we propose a family of methods that dynamically estimate expert consensus from partial feedback by producing a posterior over expert and model beliefs. Analyzing this posterior induces an interpretable trade-off between querying cost and classification performance. We demonstrate the efficacy of our framework against a variety of baselines on CIFAR-10H and ImageNet-16H, two large-scale crowdsourced datasets.
BibTeX
@inproceedings{showalter2024bayesian,
  author = {Samuel Showalter and Alex J. Boyd and Padhraic Smyth and Mark Steyvers},
  title = {{Bayesian Online Learning for Consensus Prediction}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=z4uOSsLZI2},
}
Figure