Corporate Evidential Decision Making in Performance Prediction Domains

Alex G. Büchner, Werner Dubitzky, Alfons Schuster, Philippe Lopes, Peter G. O'Donoghue, John G. Hughes, David A. Bell, Kenneth Adamson, John A. White, John M. C. C. Anderson, Maurice D. MulvennaUAI 1997

TL;DR

This paper introduces a novel approach to performance prediction in sports using the Mathematical Theory of Evidence. By combining subjective beliefs from multiple domain experts, the authors demonstrate a significant improvement in prediction accuracy compared to individual expert intuition and traditional statistical methods, particularly for complex scenarios like predicting soccer match outcomes.

What Problem Does This Paper Address?

Predicting sports outcomes is a challenging task that traditionally relies on large datasets and statistical models. However, these methods often fail to incorporate valuable human expertise, struggle with uncertainty and incomplete information, and are limited to specific disciplines. The authors identify a critical gap: the need for a framework that can systematically combine the subjective, often uncertain, knowledge of multiple domain experts to achieve more robust and accurate performance predictions. They argue that existing knowledge-based systems also fall short in handling the inherent uncertainty in such domains.

Methods

The core of this paper is the application of the Mathematical Theory of Evidence (also known as Evidence Theory or Dempster-Shafer Theory) to performance prediction. This theory offers a powerful way to model uncertainty and combine evidence from multiple sources. Key aspects of their methodology include:

  • Frame of Discernment: Defining the set of all possible outcomes (e.g., all plausible soccer scores).
  • Basic Probability Assignment (Mass Function): Experts express their beliefs not just in single outcomes but in sets of outcomes, allowing for explicit representation of ignorance or reservation of judgment.
  • Discount Operation: This mechanism quantifies the reliability of an information source (an expert), allowing for the discounting of beliefs from less reliable sources.
  • Orthogonal Sum Operation: This is the crucial mechanism for combining the mass functions (predictions) from multiple experts. It systematically pools their beliefs to arrive at a consensus prediction.

The authors also introduce a method to derive single-outcome predictions from the combined belief distributions, which is necessary for practical applications like betting systems.

Datasets

The study used data from the European Football Championship in 1996 (Euro'96). The experiment involved predicting the 90-minute scores for all 31 matches of the tournament. Information from previous matches within the tournament was incorporated as evidence for subsequent predictions.

Key Insights and Ideas

  • Expertise is Key, but Combination is Crucial: While individual experts possess valuable knowledge, their predictions are often biased and incomplete. The real power lies in systematically combining these diverse perspectives.
  • Evidence Theory Handles Uncertainty Gracefully: Unlike traditional statistics that force uncertain data into rigid numerical formats, Evidence Theory explicitly models uncertainty, ignorance, and the reliability of information sources.
  • Flexibility in Belief Representation: Experts can express beliefs over sets of outcomes, not just single outcomes, which better reflects the nuanced nature of subjective judgment.
  • Scalability to Different Domains: Although applied to soccer, the authors suggest the framework is adaptable to other sports and decision-making domains where expert knowledge and uncertainty are prevalent.

Key Results

The experiment, involving four soccer experts predicting all 31 Euro'96 matches, yielded encouraging results:

  • Combined Evidence Theory predictions achieved a higher percentage of "Correct Results" (19.4%) compared to the average individual intuition (9.7%) and individual Evidence Theory predictions (11.3%).
  • The combined Evidence Theory predictions also outperformed the best individual expert's forecast.
  • For "Correct Outcome" predictions, the combined Evidence Theory approach also showed superior performance.
  • The authors note that the "result mindset" (predicting exact scores) was more successful than the "outcome mindset" (predicting win/draw/loss) with their current setup, suggesting that explicitly defining outcomes in the frame of discernment could further improve results for outcome-based predictions.

Limitations and Future Work

The authors acknowledge several limitations and suggest avenues for future research:

  • Independence of Evidence: A common critique of Evidence Theory is its assumption of evidence independence. The authors mention work by Ling (1989a, 1989b) that addresses this by introducing parameters to quantify dependence, but note the challenge of estimating these parameters.
  • Subjectivity of Expert Selection: The choice of experts was subjective, and the study is described as "admittedly anecdotal." Future work should involve a larger number of experts and more rigorous evaluation.
  • Scalability and Generalizability: While the framework is promising, its scalability to a larger number of experts and its applicability to other sports disciplines need further exploration.
  • Refining Predictions: The authors suggest that explicitly defining more granular outcomes (e.g., "clear win," "normal win," "close win") or developing operations for refining outcome-based predictions into result-based predictions could enhance accuracy.
  • Automated Decision Making: While they developed a practical automated solution for deriving single-outcome predictions, they note that more sophisticated techniques exist for this purpose.

Summary updated

BibTeX
@inproceedings{buchner1997corporate,
  author = {Alex G. Büchner and Werner Dubitzky and Alfons Schuster and Philippe Lopes and Peter G. O'Donoghue and John G. Hughes and David A. Bell and Kenneth Adamson and John A. White and John M. C. C. Anderson and Maurice D. Mulvenna},
  title = {{Corporate Evidential Decision Making in Performance Prediction Domains}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1997},
}
Figure