Aggregating Learned Probabilistic Beliefs

Pedrito Maynard-Reid Ii, Urszula ChajewskaUAI 2001

TL;DR

This paper tackles the challenge of combining probabilistic beliefs from multiple experts, proposing a novel framework where experts learn from data. They introduce a method that leverages the Linear Opinion Pool (LinOP) operator for aggregating Bayesian Networks, demonstrating its effectiveness in approximating the ideal scenario where a single expert learns from all available data.

What Problem Does This Paper Address?

In many fields, from medicine to engineering, we rely on the expertise of multiple individuals to make informed decisions. This expertise is often captured in the form of probability distributions. However, simply averaging or combining these distributions can be problematic, as many existing aggregation methods lack a clear semantic grounding. They often rely on abstract properties that may not hold in real-world scenarios. The core problem this paper addresses is how to aggregate these expert beliefs in a principled way, ensuring that the aggregated belief is as close as possible to what a single, highly experienced expert would have learned from the combined knowledge of all individuals. This is particularly relevant when experts have learned their beliefs from different subsets of data, a common situation in practice.

Methods

The authors propose a framework where experts are assumed to have learned their probability distributions from data. The "optimal" aggregate distribution is then defined as the one a Decision Maker (DM) would learn if they had access to all the data from all experts. Since direct access to raw data is often impractical, the goal becomes approximating this ideal.

The paper focuses on two main scenarios:

  • Joint Distributions: If experts have learned joint probability distributions, and both the experts and the DM are using Maximum Likelihood Estimation (MLE), the paper shows that the DM can simply compute a weighted sum of the expert distributions. Crucially, the weights are semantically justified as the proportion of the total data each expert has observed. This elegantly addresses a common criticism of LinOP where weights are often ad-hoc.
  • Bayesian Networks (BNs): For more structured representations like BNs, the authors adapt the BN learning algorithm. Instead of using raw data samples, they use the learned distributions from the experts. They leverage the marginalization property of LinOP to efficiently compute the necessary distributions and scores. This allows them to aggregate BNs without needing the original data, by essentially using the LinOP of the expert BNs to represent the empirical distribution of the combined data.

Datasets

The experiments were conducted on two datasets:

  • Alarm Network: A well-known, real-life 37-node network used in intensive care patient monitoring.
  • Asia Network: A smaller, 8-node artificial network.

These networks were used to evaluate the performance of the proposed aggregation algorithm by comparing its accuracy against learning from the combined data (optimal) and the individual sources.

Key Insights and Ideas

  • Semantics Matter: The paper strongly argues that the evaluation of aggregation techniques should be grounded in the context of how expert beliefs are formed, particularly from data.
  • LinOP as the Right Tool: Under the assumption that experts and the DM are MLE learners, LinOP naturally emerges as the correct aggregation operator.
  • Semantically Justified Weights: The proportion of data observed by each expert provides a principled way to assign weights in LinOP, moving beyond arbitrary choices.
  • BN Aggregation without Data: The proposed method for aggregating BNs is a significant contribution, allowing for the combination of structured expert knowledge without requiring access to the original data.
  • Approximation is Key: The framework acknowledges that perfect reconstruction of the combined dataset is impossible and focuses on finding the best possible approximation.

Key Results

The experimental results, particularly on the Asia network, show that the proposed aggregation algorithm (AGGR) performs very well. It consistently improves upon the individual sources and closely approximates the optimal distribution learned from the combined data.

  • The KL divergence (a measure of difference between distributions) drops as the total number of samples (M) increases, indicating better performance with more data.
  • The algorithm shows robustness to variations in the DM's estimate of M, with reasonable approximations of M yielding good results.
  • The method also performs well when combining distributions learned from different subpopulations, demonstrating its flexibility.
  • A naive sampling-based aggregation algorithm (SAMP) performed poorly, often worse than the individual sources, highlighting the effectiveness of the proposed LinOP-based approach.

Limitations and Future Work

The authors acknowledge several limitations and suggest future research directions:

  • Continuous Variables: The current framework is restricted to discrete variables. Extending it to handle continuous variables is a natural next step.
  • Dependence Between Sources: The current work assumes independence of sample sets. Relaxing this assumption to allow for dependencies between sources' data would increase the applicability of the method.
  • Incorporating Source Priors: The current framework largely ignores the priors used by the sources. Finding valid ways to incorporate these priors could further improve aggregation quality. For instance, if sources use Dirichlet priors, the DM might use their estimated sample sizes to inform their own estimate of M.
  • Alternative Aggregation Operators: While LinOP is shown to be appropriate for MLE learners, exploring other operators for different learning paradigms could be beneficial.

Summary updated

BibTeX
@inproceedings{ii2001aggregating,
  author = {Pedrito Maynard-Reid II and Urszula Chajewska},
  title = {{Aggregating Learned Probabilistic Beliefs}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2001},
}
Figure