Bayesian Inference without Point Estimates

Paul SnowAAAI 1986

TL;DR

This paper argues that restricting Bayesian inference to point estimates of probabilities is unnecessarily limiting. The authors propose using "set estimates" (ranges or sets of possible probability vectors) to represent uncertainty more naturally, showing how Bayesian updating can be applied to these sets with minimal computational overhead, offering a more expressive and rigorous approach to probabilistic reasoning.

What Problem Does This Paper Address?

Traditional Bayesian inference, especially in artificial intelligence, often insists on using point estimates for probabilities. This means an analyst must assign a single, precise number (e.g., "70% chance of rain") even when their knowledge is much vaguer (e.g., "more likely to rain than not"). This strict requirement can be problematic for several reasons:

  • Imprecise Knowledge: Analysts often don't know precise probabilities. Forcing them to provide point estimates can lead to either overconfidence or an inability to express genuine ignorance.
  • Expressing Qualitative Assertions: Statements like "event A is more likely than event B" are natural but don't translate directly to point estimates.
  • Representing Ignorance: Expressing complete ignorance (e.g., with three mutually exclusive states) by assigning equal probabilities (1/3 each) leads to paradoxes where a disjunction of two states (2/3) is more likely than a single state (1/3).

The paper aims to overcome these limitations by allowing probability estimates to be sets of possible probability vectors, thereby enabling more natural and rigorous representation of uncertainty.

Methods

The core innovation lies in extending Bayesian inference to handle "set estimates" of probabilities. Instead of a single probability vector, the analyst provides a set of vectors that are considered plausible.

  • Set Estimates: These sets can represent various forms of imprecise knowledge, such as linear inequalities (e.g., P1 > P2), orderings of probabilities, or bounds on sums of probabilities. The paper highlights that these sets often form polytopes (convex geometric shapes defined by linear inequalities), which have a convenient structure.
  • Bayesian Updating with Set Priors: The paper shows that if the prior probability information is represented as a set (specifically, a polytope), Bayesian updating can be performed by applying Bayes' formula point-by-point to the vertices of the polytope. The resulting posterior set is also a polytope. This significantly reduces computational burden compared to updating every possible vector within the set.
  • Handling Zero Probabilities: A key challenge is that zero probabilities in the prior set can lead to persistent distortions in posterior estimates. The paper suggests that asserting small, non-zero minima for probabilities (e.g., "at least 1%") can suppress these problematic zeros and ensure convergence.
  • Implementation: The authors note that existing AI systems using Bayesian networks (like PROSPECTOR) can be adapted to handle set estimates by treating nodes as containing multiple probability vectors instead of just one. The computational cost increases linearly with the number of vectors to be updated.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Naturalness of Set Estimates: Probabilistic information often comes in qualitative forms (e.g., "more likely," "usually") that are better captured by sets of probabilities than by single point estimates.
  • Polytopes as a Convenient Geometry: The geometric structure of polytopes, arising from linear constraints on probabilities, makes them particularly amenable to Bayesian updating.
  • Convergence with Set Priors: Similar to point estimates, set estimates with non-zero components (or asserted minima) can lead to convergence towards a single, most likely probability vector as evidence accumulates.
  • Avoiding Paradoxes of Ignorance: Set estimates, particularly the "vacuous set" (all possible probability vectors), provide a more consistent way to represent complete ignorance than assigning equal point probabilities.
  • Bridging Qualitative and Quantitative Reasoning: The approach allows for rigorous Bayesian inference to be applied to qualitative statements about probabilities, bridging a gap often exploited by fuzzy logic or other non-additive belief measures.

Key Results

The paper primarily presents a theoretical framework and a computational method.

  • Theoretical Result: If the prior probability set is a polytope, the posterior probability set after Bayesian updating is also a polytope. The vertices of the posterior polytope are the Bayesian posteriors of the prior polytope's vertices.
  • Computational Efficiency: Applying Bayesian updates to the vertices of a polytope is computationally feasible, especially when the number of vertices is small. This is a significant improvement over trying to update an infinite or very large set of possibilities.
  • Practical Implications: The framework allows AI systems to handle more nuanced and realistic expressions of uncertainty, potentially improving their diagnostic and decision-making capabilities.

Limitations and Future Work

  • Decision Making: While the paper discusses methods for using set estimates to inform decisions (e.g., selecting a point with maximum entropy, using maximin rules), it acknowledges that this area remains a "weak spot" with no universally accepted technique. Further research is needed to develop robust decision rules for set-informed decisionmaking.
  • Savage Axioms: The paper addresses the conflict with Savage's axioms, which strongly support point estimates. It argues that if an analyst does not subscribe to all of Savage's axioms (particularly complete ordering of values), then the inference about point estimates is unfounded, opening the door for set estimates.
  • Zero Components: The paper highlights that zero components in prior probability vectors can hinder convergence. While asserting minima is a solution, the paper notes that the "worst case" of complete ignorance (vacuous set) still results in a vacuous posterior, though it argues this is rare in practice.
  • General Set Geometries: While polytopes are emphasized for their convenience, the paper acknowledges that other geometries (like discrete sets of points) are also possible and can be updated efficiently. The restriction to polytopes is not obligatory.

Summary updated

BibTeX
@inproceedings{snow1986bayesian,
  author = {Paul Snow},
  title = {{Bayesian Inference without Point Estimates}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {1986},
}
Figure