Abstract
In high-stakes domains like medicine or autonomous driving, ground truth is often defined by the consensus of human experts rather than an objective oracle. This paper introduces a Bayesian framework for online classification that treats AI predictions as priors and human votes as samples from a multivariate hypergeometric distribution. By dynamically learning model reliability and expert consensus, the proposed method minimizes expensive human queries while maximizing predictive accuracy.
Introduction
In many critical applications, "absolute truth" is frequently unavailable, leading practitioners to rely on the collective judgment of expert panels. While AI models provide rapid predictions, human expertise is costly and slow. Existing "learning to defer" frameworks typically assume a single, infallible ground truth. This research addresses the more realistic challenge of predicting the plurality vote of a finite expert pool under budget constraints, where the AI's accuracy and expert reliability must be learned on the fly.
Methods: A Generative Bayesian Model
The core of the approach is a generative Bayesian model that estimates expert consensus. The system treats observed human votes as a sub-sample of the total pool, modeled using a multivariate hypergeometric distribution to capture the nature of sampling without replacement from a finite group.
The framework incorporates a pre-trained AI model $f(x)$ to inform the prior. It introduces learned parameters—$\theta, \phi, \tau$—to scale model confidence and perform per-class calibration. Two variants are proposed:
- FinExp: An exact model for a finite number of experts.
- InfExp: A limiting case approximation assuming an infinite expert pool, which simplifies the model into a Dirichlet-Multinomial framework that is easier to optimize.
Decisions to query humans are made dynamically: if the posterior confidence in the consensus is below a threshold $\rho$, an additional expert is queried; otherwise, the system stops and predicts.
Experimental Evaluation
The framework was evaluated on two benchmark datasets:
- CIFAR-10H: 10,000 images with 50 human labels each.
- ImageNet-16H: 4,800 images with 6 human annotations each, including variations with phase noise.
Key Results
The Bayesian methods consistently outperformed standard active learning baselines, such as Entropy-based querying and the "Model Picker" algorithm.
On CIFAR-10H, the proposed methods achieved significantly lower error rates for the same query budget. Notably, the InfExp approximation performed remarkably well even for small expert pools, offering a computationally efficient alternative to the exact model. Furthermore, the system demonstrated an "accuracy boost" in query-free scenarios, improving the base model's accuracy by up to 3% through better calibration based on historical human feedback.
Robustness and Online Calibration
A critical advantage of this framework is its ability to handle distribution shifts. By learning the $\tau$ parameter (per-class trust), the system identifies which classes the AI struggles with and automatically increases human querying for those instances. When data becomes noisier—simulating sensor failure—the Bayesian framework quickly adjusts its priors, maintaining accuracy where standard methods degrade.
Conclusion
This work provides a robust framework for integrating AI predictions with human expert consensus in online settings. By treating the AI as a prior and human votes as statistical samples, the system achieves a superior balance between cost and accuracy. Future research will explore individual expert identities and the application of this framework to non-image modalities and open-set tasks.
Summary updated