Evidential Probability

Henry E. Kyburg Jr.IJCAI 1991

TL;DR

This paper introduces "evidential probability," a novel framework for quantifying uncertainty that bridges the gap between statistical frequencies and unique, unrepeatable events. It proposes a method for determining the "correct reference class" for a given statement, allowing for more robust probabilistic reasoning in situations where direct statistical data is scarce or conflicting.

What Problem Does This Paper Address?

The paper tackles a fundamental challenge in probability and reasoning: how to assign probabilities to unique events when we lack direct statistical data for that specific instance. Traditional approaches fall into two camps:

  1. Frequency-based probability: This view equates probability with the long-run frequency of an event. While intuitive for repeatable phenomena (like coin flips or casino games), it struggles with unique events (e.g., the outcome of a specific medical treatment for a particular patient). The problem is that for a unique event, the frequency is either 0 or 1, offering no nuanced uncertainty.
  2. Subjective probability: This view defines probability as an individual's degree of belief. While flexible, it allows for arbitrary initial beliefs and can lead to inconsistencies, as "one man's coherent opinion is, formally, as good as another's."

The gap lies in how to leverage existing statistical knowledge to make reasoned judgments about specific, non-repeatable situations, especially when multiple, potentially conflicting, statistical datasets are available.

Methods

The core of the paper is the development of evidential probability, an interval-valued measure of uncertainty. The key methodological contributions are:

  • Reference Class Selection: The paper argues that every statement is equivalent to some statement for which statistics are relevant. The central problem is identifying the "correct reference class" – the most appropriate group of similar instances to draw statistical inferences from.
  • Rules for Determining Reference Classes: The paper proposes a set of rules for selecting the most informative and relevant reference class. This involves considering:
  • Specificity: Preferring narrower classes for which more precise statistics are available.
  • Conflict Resolution: Developing strategies to handle situations where different reference classes yield conflicting probability intervals. This includes considering the intersection of classes and, in cases of disagreement, favoring intervals that are not included in others.
  • Bayesian Principle: A principle that suggests when dealing with multi-stage processes (like selecting a herd and then a cow), the probability should reflect the combined evidence from these stages.
  • Interval-Valued Probabilities: Instead of single point estimates, evidential probability uses intervals to represent uncertainty. This acknowledges that statistical data often provides bounds rather than exact figures, especially at given confidence levels.

Datasets

The paper does not present new datasets. Instead, it uses hypothetical examples and existing statistical data structures to illustrate its concepts:

  • Casino Rolls: Hypothetical examples of dice rolls at a specific time and location.
  • Measurement Errors: Examples related to the distribution of errors in scientific measurements.
  • Insurance Premiums: A hypothetical scenario involving life insurance for an individual with a complex family medical history and professional background.
  • Beef Cattle Founder Data: A table presenting survey data on the incidence of founder in beef cattle, categorized by various treatments (worming medicine, implant, louse powder, soybean supplement, corn supplement). This is used extensively to demonstrate reference class selection and conflict resolution.

Key Insights and Ideas

  • The "Surfeit of Information" Problem: A key insight is that the problem isn't a lack of statistical data, but often a surfeit of it. An individual or event can belong to numerous statistical classes, each offering potentially conflicting probability estimates.
  • Beyond Frequency and Subjectivity: Evidential probability offers a middle ground, grounding probabilities in statistical evidence while still being applicable to unique events. It formalizes the "evidential relation" between general knowledge and specific instances.
  • The Importance of "Correct" Reference Class: The paper strongly emphasizes that the choice of reference class is paramount. An inappropriate class can lead to misleading probabilities, even with abundant data.
  • Intervals as Natural Representation of Uncertainty: Using intervals for probability naturally captures the uncertainty inherent in statistical inference and the limitations of available data.
  • Conflict Resolution is Crucial: The paper highlights that simply picking the "narrowest" class isn't always sufficient. Strategies for reconciling conflicting intervals are essential for robust reasoning.

Key Results

The paper doesn't present quantitative results in the form of accuracy metrics or performance benchmarks. Instead, its key "results" are conceptual and methodological:

  • A formal framework for evidential probability: The paper lays out the principles and rules for this new approach.
  • Demonstration of reference class selection: Through detailed examples (like the cattle founder data), the paper illustrates how to apply its rules to select appropriate reference classes and derive interval-valued probabilities.
  • Illustrations of conflict resolution: The paper shows how to handle situations where different statistical sources disagree, leading to more defensible probability assignments than purely subjective or naive frequency approaches. For instance, in the cattle example, it shows how to derive a broader, more inclusive interval when specific sub-classes conflict.

Limitations and Future Work

  • Adjudicating Conflicts: While the paper proposes methods for conflict resolution, it acknowledges that in some difficult cases (disjoint or overlapping intervals without inclusion), the problem remains challenging.
  • "Judgment Call" Aspect: The paper notes that in the final analysis, some aspects of reference class selection might still involve a "judgment call," though it aims to formalize this process.
  • Relation to Other Frameworks: The paper briefly mentions the relation between evidential probability and belief functions, subjective probability, and frequency probability, suggesting this as a direction for further exploration.
  • Computational Complexity: The paper doesn't delve into the computational aspects of applying these rules, which could be a significant consideration for complex expert systems.
  • Formalization of "Adequate Statistics": The paper refers to "adequate statistics" but doesn't provide a strict formal definition, leaving room for interpretation.

Summary updated

BibTeX
@inproceedings{jr1991evidential,
  author = {Henry E. Kyburg Jr.},
  title = {{Evidential Probability}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1991},
}
Figure