Bayesian Classifier Fusion with an Explicit Model of Correlation

Susanne Trick, Constantin A. Rothkopf

Abstract

Combining the outputs of multiple classifiers or experts into a single probabilistic classification is a fundamental task in machine learning with broad applications from classifier fusion to expert opinion pooling. Here we present a hierarchical Bayesian model of probabilistic classifier fusion based on a new correlated Dirichlet distribution. This distribution explicitly models positive correlations between marginally Dirichlet-distributed random vectors thereby allowing explicit modeling of correlations between base classifiers or experts. The proposed model naturally accommodates the classic Independent Opinion Pool and other independent fusion algorithms as special cases. It is evaluated by uncertainty reduction and correctness of fusion on synthetic and real-world data sets. We show that a change in performance of the fused classifier due to uncertainty reduction can be Bayes optimal even for highly correlated base classifiers.

Introduction

In machine learning, ensembling—combining multiple models to achieve superior performance—is a standard technique for improving robustness and accuracy. However, most common fusion methods, such as the Independent Opinion Pool (IOP), rely on the assumption that the errors made by different classifiers are independent. In practice, this assumption is frequently violated; classifiers trained on the same data or human experts evaluating the same evidence often exhibit high degrees of correlation.

Ignoring these correlations leads to mathematically unjustified "uncertainty reduction," resulting in overconfident and often incorrect predictions. This paper introduces a hierarchical Bayesian framework for fusing probabilistic outputs while explicitly accounting for their correlations. The goal is to develop a normative model that recognizes when agreement is a byproduct of correlation rather than independent confirmation of the truth.

Methodology

The authors propose a progression of three models to address the fusion task. The baseline is the Independent Opinion Pool (IOP), a heuristic method that simply multiplies probabilities. To move toward a normative Bayesian approach, they first define the Independent Fusion Model (IFM). In IFM, each classifier's output is modeled by an individual Dirichlet distribution, capturing its unique bias and variance, but assuming independence between classifiers.

The primary contribution is the Correlated Fusion Model (CFM). Unlike IFM, CFM explicitly models the joint output of all classifiers using a novel correlated Dirichlet distribution. This allows the model to account for positive correlations, preventing the overconfidence that arises when multiple dependent experts agree.

Comparison of the Independent Fusion Model (IFM) and the Correlated Fusion Model (CFM) plate diagrams and generative equations.

The "secret sauce" of the CFM lies in the construction of this distribution. By utilizing shared latent Gamma variables to link the marginal Dirichlet distributions of different classifiers, the model can represent dependencies directly within the probability simplex. Parameter inference and fusion are performed using Gibbs Sampling (implemented via JAGS), allowing the model to learn from training data how much to "trust" the ensemble's consensus.

Detailed generative process for the correlated Dirichlet distribution, illustrating the use of latent variables $A$ and $D$ to induce correlation between classifiers.

Experimental Setup

The model was evaluated on both synthetic and real-world datasets:

  • Simulated Data (SIM 1-3): Controlled environments used to test the model's handling of varying correlation levels, variance, and classifier bias.
  • Bookies A, B, & C: Odds from professional bookmakers for football matches (English Premier League and German Bundesliga). These represent "experts" with extremely high correlations (often >0.95).
  • DNA A, B, & C: Based on the StatLog splice-junction dataset, where ensembles (kNN and Random Forests) were trained with different hyperparameters to create varying degrees of correlation.

Results and Discussion

The CFM demonstrated superior accuracy in terms of log-loss, which penalizes both incorrectness and overconfidence. It outperformed previous state-of-the-art Bayesian methods across all tested datasets.

Key insights include:

  • The Correlation Penalty: As correlation between classifiers increases, the benefit of fusion (uncertainty reduction) naturally decreases. The CFM correctly scales its confidence based on the learned correlation.
  • Bias Correction: Unlike simple averaging, the CFM can learn and correct for consistent biases in specific classifiers (e.g., a model that always over-predicts a certain class).
  • Handling High Correlation: On the "Bookies" datasets, where experts are nearly identical, the CFM avoided the overconfidence traps that led other models to perform worse than individual base classifiers.

Conclusion and Future Work

The proposed framework provides a theoretically grounded way to combine expert opinions while avoiding the pitfalls of independence assumptions. However, the model's reliance on Gibbs Sampling introduces significant computational complexity. Fusing 60 examples took approximately 940 seconds, compared to 3.5 seconds for non-hierarchical models. Future work will focus on scaling to larger ensembles using Variational Inference and addressing the high parameter count to prevent overfitting on small datasets.

Summary updated

Abstract

Combining the outputs of multiple classifiers or experts into a single probabilistic classification is a fundamental task in machine learning with broad applications from classifier fusion to expert opinion pooling. Here we present a hierarchical Bayesian model of probabilistic classifier fusion based on a new correlated Dirichlet distribution. This distribution explicitly models positive correlations between marginally Dirichlet-distributed random vectors thereby allowing explicit modeling of correlations between base classifiers or experts. The proposed model naturally accommodates the classic Independent Opinion Pool and other independent fusion algorithms as special cases. It is evaluated by uncertainty reduction and correctness of fusion on synthetic and real-world data sets. We show that a change in performance of the fused classifier due to uncertainty reduction can be Bayes optimal even for highly correlated base classifiers.
BibTeX
@inproceedings{trick2022bayesian,
  author = {Susanne Trick and Constantin A. Rothkopf},
  title = {{Bayesian Classifier Fusion with an Explicit Model of Correlation}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
  url = {https://openreview.net/forum?id=AVlHeLw6Fy},
}
Figure