An Experimental Comparison of Numerical and Qualitative Probabilistic Reasoning

Max Henrion, Gregory M. Provan, Brendan Del Favero, Gillian SandersUAI 1994

TL;DR

This paper explores whether "infinitesimal probabilities" (a qualitative approach to reasoning under uncertainty) can effectively approximate traditional numerical Bayesian networks for diagnostic tasks. The authors found that this qualitative method performs comparably to numerical methods for low prior probabilities, suggesting its potential for simplifying knowledge engineering and improving robustness in certain diagnostic applications.

What Problem Does This Paper Address?

Traditional probabilistic reasoning methods, like Bayesian networks, often require precise numerical probabilities and utilities. Eliciting these exact numbers from experts can be difficult, time-consuming, and prone to judgmental biases. This has fueled interest in qualitative approaches that rely on less precise, more intuitive judgments. The paper specifically investigates "infinitesimal probabilities," a formalization of qualitative reasoning that is consistent with probability axioms but avoids exact numerical quantification. The core question is whether this qualitative approach can achieve comparable diagnostic performance to its numerical counterpart in a real-world task.

Methods

The authors compare two approaches:

  1. Numerical Probabilistic Reasoning: Using a fully assessed numerical Bayesian belief network.
  2. Infinitesimal Probabilities (Kappa-Calculus): This method represents probabilities using a formalization based on powers of a small parameter $\epsilon$ (i.e., $\epsilon^k$). They developed mappings to convert numerical probabilities into these $\epsilon$-based values (kappa-values) and, crucially, to convert the resulting kappa-values back into probabilities for comparison. This conversion involves defining a "plausible set" of diagnoses with the minimal kappa-value and assigning probabilities based on the size of this set.

The experiment involved:

  • A car diagnostic Bayesian network.
  • Converting the numerical network to an infinitesimal representation using different $\epsilon$ values (0.0001, 0.001, 0.01, 0.1, 0.2).
  • Generating 116 test cases by observing different combinations of findings (evidence).
  • Varying the prior fault probabilities for the faults in the network.
  • Comparing the diagnostic performance of both schemes.

Datasets

The paper uses a single, pre-existing Bayesian network representing a car diagnosis domain. This network was provided by David Heckerman and Paul Dagum. The network includes:

  • Binary variables for most states (e.g., spark plugs bad: yes/no).
  • A ternary variable for battery charge (high, low, none).
  • Nine explicitly identified faults and three "leak" events representing unmodeled causes.
  • Fully quantified numerical conditional probability distributions and prior probabilities.

No new datasets were introduced; the focus was on evaluating existing methods on a representative diagnostic task.

Key Insights and Ideas

  • Qualitative Reasoning as a Practical Approximation: The paper demonstrates that a formal qualitative probabilistic reasoning scheme can indeed serve as a viable approximation to traditional numerical Bayesian networks for diagnostic tasks, especially when precise numerical inputs are hard to obtain.
  • Sensitivity to Prior Probabilities: A critical finding is the significant performance drop of the infinitesimal scheme when prior fault probabilities exceed a certain threshold (around 0.03). This suggests its strength lies in scenarios with rare faults.
  • Robustness to Evidence Quantity: The performance of the infinitesimal scheme tracks the numerical scheme closely as the amount of evidence (number of findings) increases, indicating it scales similarly with data.
  • The Role of $\epsilon$: The choice of $\epsilon$ impacts performance. While strictly correct as $\epsilon \to 0$, a larger $\epsilon$ (up to 0.2 in this study) provided a finer discretization and better results, suggesting a trade-off between theoretical purity and practical approximation.

Key Results

  • Comparable Performance for Low Priors: For prior fault probabilities up to approximately 0.03, the infinitesimal scheme's performance (measured by the probability assigned to the true fault) was comparable to the numerical scheme.
  • Performance Degradation at High Priors: As average prior fault probabilities increased beyond 0.03 (e.g., to 0.089 and 0.214), the infinitesimal scheme's performance dropped sharply compared to the numerical method.
  • Effect of Evidence: With increasing evidence (from 1 to 10 findings), the average probability assigned to the true fault increased for both schemes, from about 0.15 to 0.47. The infinitesimal scheme's performance closely followed the numerical scheme throughout this range.
  • Plausible Set Size: The average size of the "plausible set" (diagnoses with the minimal kappa-value) decreased significantly with more findings, from 7 faults with 1 finding to 1.21 faults with 10 findings, indicating increased specificity.
  • Optimal $\epsilon$: The best performance for the infinitesimal scheme was observed with $\epsilon$ values around 0.1 to 0.2.

Limitations and Future Work

Limitations: * Single Domain: The study was conducted on a single, relatively simple car diagnostic network. The authors acknowledge that results may not generalize to larger, more complex networks with loops. * Simple Mapping: The mapping from kappa-values back to probabilities is a simple one (assigning $1/n$ to each diagnosis in the plausible set of size $n$). More sophisticated mappings could potentially yield different results. * Scoring Method: The evaluation was based on the posterior probability of the true diagnosis, which is a stringent measure. Other scoring methods (e.g., rank ordering) might be less sensitive to representation differences.

Future Work: * Domain Independence of $\epsilon$: Investigate if the optimal $\epsilon$ value is domain-dependent. * Larger Networks: Test the approach on more complex network structures. * Extension to Decision Making: Explore extending the infinitesimal approximation to utilities and decision-making under uncertainty. * Analytic Characterization: Develop a clearer theoretical understanding of when the infinitesimal approximation is reliable. * Knowledge Engineering Methods: Develop practical methods for eliciting infinitesimal probabilities from experts. * Hybrid Approaches: Anticipate that future practical tools will likely combine qualitative and quantitative methods.

Summary updated

BibTeX
@inproceedings{henrion1994experimental,
  author = {Max Henrion and Gregory M. Provan and Brendan Del Favero and Gillian Sanders},
  title = {{An Experimental Comparison of Numerical and Qualitative Probabilistic Reasoning}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1994},
}
Figure