Dynamic Bayesian Combination of Multiple Imperfect Classifiers

Edwin Simpson, Stephen Roberts, Ioannis Psorakis, A.f.m. Smith

Abstract

Classifier combination methods need to make best use of the outputs of multiple, imperfect classifiers to enable higher accuracy classifications. In many situations, such as when human decisions need to be combined, the base decisions can vary enormously in reliability. A Bayesian approach to such uncertain combination allows us to infer the differences in performance between individuals and to incorporate any available prior knowledge about their abilities when training data is sparse. In this paper we explore Bayesian classifier combination, using the computationally efficient framework of variational Bayesian inference. We apply the approach to real data from a large citizen science project, Galaxy Zoo Supernovae, and show that our method far outperforms other established approaches to imperfect decision combination. We go on to analyse the putative community structure of the decision makers, based on their inferred decision making strategies, and show that natural groupings are formed. Finally we present a dynamic Bayesian classifier combination approach and investigate the changes in base classifier performance over time.

Introduction: The Challenge of Imperfect Information Aggregation

In the era of crowdsourcing and large-scale citizen science, researchers often rely on cohorts of "imperfect" agents to label data. Whether these are human volunteers or weak computational models, they come with varying levels of expertise, different biases, and often provide very little data per individual.

The core challenge is information aggregation: how to combine these noisy, "greedy" preferences (simple labels with no confidence scores) into a single, high-accuracy classification. Standard approaches like majority voting treat everyone as equally reliable, which is rarely true. More sophisticated Bayesian methods exist but are often computationally expensive or assume that an agent's performance is static, ignoring the fact that people learn, fatigue, or change their strategies over time.

Methodology: Variational and Dynamic Bayesian Frameworks

The authors build upon the Independent Bayesian Classifier Combination (IBCC) model, introducing two primary contributions:

  1. Variational Bayesian (VB) Inference: Instead of using slow, stochastic sampling (Gibbs sampling), the authors apply Variational Bayes. This provides a deterministic approximation that is significantly faster and offers a "lower bound" metric to easily check for algorithm convergence.
  2. Dynamic IBCC (DynIBCC): They extend the model to handle time-varying performance. Using a dynamic generalized linear model—essentially a Kalman-filter-like forward pass and a Bryson-Frazier smoother backward pass—the system tracks changes in an agent's confusion matrix over the course of their participation.
Plate notation for the Independent Bayesian Classifier Combination (IBCC) model.

The model treats the "true" label as a latent variable and the agents' outputs as observed data generated by a multinomial distribution (the agent's confusion matrix). By placing Dirichlet priors on these matrices, the model can incorporate prior knowledge and handle sparse data gracefully.

Experimental Evaluation: Galaxy Zoo Supernovae

The researchers evaluated their model using data from Galaxy Zoo Supernovae, a Zooniverse citizen science project involving 963 classification tasks (identifying potential supernovae in images) and 1,705 base classifiers (human volunteers). The dataset comprised 26,558 individual classifications. To verify accuracy, the authors used "true" labels obtained retrospectively via full spectroscopic analysis from the Palomar Transient Factory.

Results and Behavioral Insights

The VB-IBCC model achieved an AUC (Area Under Curve) of 0.9840, vastly outperforming simple majority voting (0.7809) and even Gibbs-sampled IBCC (0.9127). Beyond accuracy, the VB approach reached convergence in just 13 seconds, compared to 349 seconds for Gibbs sampling—a ~27x speedup critical for real-time applications.

Performance comparison of VB-IBCC against baseline methods using ROC curves and AUC values.

Key insights from the model include:

  • Confusion Matrices as Behavioral Fingerprints: The model infers a full matrix of how likely an agent is to mistake one class for another, identifying specific biases like "optimists" who rarely miss a true positive but have many false positives.
  • Discovery of Communities: By calculating the Hellinger distance between inferred confusion matrices, the authors identified distinct "communities" of decision-makers, such as The Extremists (who use only high/low scores) and The Non-committal (who prefer middle-of-the-road scores).
  • Dynamic Performance: The DynIBCC model revealed that volunteers often experience a "settling in" period where early classifications are volatile before performance drifts as they gain or lose focus.

Conclusion and Future Directions

While the model assumes conditional independence between classifiers—a simplification that makes the math tractable—it performs exceptionally well in practice. Future work includes intelligent task assignment, using inferred confusion matrices to route specific images to the best-suited agents, and apprenticeship learning to provide targeted feedback to improve human-agent collective performance.

Summary updated

Abstract

Classifier combination methods need to make best use of the outputs of multiple, imperfect classifiers to enable higher accuracy classifications. In many situations, such as when human decisions need to be combined, the base decisions can vary enormously in reliability. A Bayesian approach to such uncertain combination allows us to infer the differences in performance between individuals and to incorporate any available prior knowledge about their abilities when training data is sparse. In this paper we explore Bayesian classifier combination, using the computationally efficient framework of variational Bayesian inference. We apply the approach to real data from a large citizen science project, Galaxy Zoo Supernovae, and show that our method far outperforms other established approaches to imperfect decision combination. We go on to analyse the putative community structure of the decision makers, based on their inferred decision making strategies, and show that natural groupings are formed. Finally we present a dynamic Bayesian classifier combination approach and investigate the changes in base classifier performance over time.
BibTeX
@inproceedings{simpson2013dynamic,
  author = {Edwin Simpson and Stephen Roberts and Ioannis Psorakis and A.F.M. Smith},
  title = {{Dynamic Bayesian Combination of Multiple Imperfect Classifiers}},
  booktitle = {Decision Making and Imperfection},
  year = {2013},
  url = {https://openreview.net/forum?id=pX4QhrQQIvi},
}
Figure