How to Elicit Many Probabilities

Linda C. van der Gaag, Silja Renooij, C. L. M. Witteman, Berthe M. P. Aleman, Babs G. TaalUAI 1999

TL;DR

Building complex Bayesian belief networks often gets bogged down by the tedious process of eliciting probabilities from domain experts. This paper introduces a novel method that significantly speeds up this process by using natural language descriptions and a combined numerical/verbal probability scale, making it much more efficient and comfortable for experts.

What Problem Does This Paper Address?

Developing sophisticated Bayesian belief networks (BBNs) for complex domains, like medical treatment planning, requires a vast number of probabilities to define the relationships between variables. A major bottleneck in this process is "probability elicitation" – getting these probabilities from human experts. Traditional methods, like using purely numerical scales or complex lottery systems, are often time-consuming, uncomfortable for experts, and can lead to biased or inaccurate assessments. This paper tackles the challenge of efficiently and effectively eliciting a large number of probabilities from domain experts, particularly when data is scarce and expert knowledge is the primary source.

Methods

The authors propose a new method for probability elicitation that combines several ideas:

  • Textual Representation: Instead of mathematical notation, conditional probabilities are presented as natural language fragments describing a specific scenario (e.g., "Consider a patient with a polypoid oesophageal carcinoma; the carcinoma has a length of less than 5 cm. How likely is it that this carcinoma invades into the muscularis propria (T2) of the patient's oesophageal wall, but not beyond?"). This makes the questions more intuitive and less intimidating for experts.
  • Combined Numerical and Verbal Scale: A probability scale is used that features both numerical anchors (0, 15, 25, 50, 75, 85, 100) and corresponding verbal anchors ("impossible," "improbable," "uncertain," "fifty-fifty," "expected," "probable," "certain"). This caters to the human tendency to express probabilities more naturally using words, while still providing a quantitative basis for assessment.
  • Grouping of Figures: Figures presenting probabilities for the same conditional distribution are grouped together on consecutive sheets. This reduces the cognitive load on experts by minimizing the need to switch between different conditioning contexts.
  • Trend Handling: The method incorporates a way to handle "trends," where experts express a relationship between probabilities for different scenarios (e.g., "patients with ulcerating tumors are 10% worse off regarding depth of invasion than patients with equivalent polypoid tumors"). This allows for more structured and consistent elicitation.

Datasets

Not discussed in the paper. The primary source of information for the influence diagram was expert knowledge.

Key Insights and Ideas

  • Natural Language is Key: Experts are more comfortable and effective when probabilities are framed in natural language scenarios rather than abstract mathematical notation.
  • Words and Numbers Together: A combined verbal and numerical scale is more effective than either alone, as it leverages the intuitive nature of verbal expressions while grounding them in quantitative anchors.
  • Expert Comfort Matters: The method's success hinges on making the elicitation process less daunting and more aligned with how experts naturally think and communicate.
  • Trends as a Natural Elicitation Tool: Experts naturally express relationships between probabilities as trends, and a method that can capture and process these trends is highly valuable.

Key Results

  • Efficiency: The new method allowed for the elicitation of 150-200 probabilities per hour, a significant improvement over previous methods.
  • Expert Satisfaction: Domain experts reported feeling "very comfortable" with the method, finding it "much easier to use" than prior techniques.
  • Time Savings: What previously took over a year of sporadic effort with other methods was accomplished for a major part of the influence diagram in just two months using the new method.
  • Initial Evaluation: An initial evaluation of the 40-node belief network built using these probabilities showed promising results, predicting the correct stage for 61% of patients, which is considered reasonable given the rough initial assessments and data limitations.

Limitations and Future Work

  • Preparation Time: While the elicitation itself is fast, the method requires significant preparation time from the elicitors (around 10 hours per interview).
  • Extreme Probabilities: The current scale might not perfectly accommodate assessments very close to 0 or 1, although experts indicated they rarely needed to express such extreme values.
  • Verbal Anchor Interpretation: While the study found good agreement, there was a slight nuance where "improbable" was interpreted as closer to 5% by experts, rather than the intended 15%, suggesting potential for further refinement of verbal anchors.
  • Future Work: The authors suggest that the method's effectiveness makes the preparation time "certainly well spent." They also mention the possibility of stepwise refinement of probability assessments, implying that the initial elicitation is a first step in an iterative process.

Summary updated

BibTeX
@inproceedings{gaag1999how,
  author = {Linda C. van der Gaag and Silja Renooij and C. L. M. Witteman and Berthe M. P. Aleman and Babs G. Taal},
  title = {{How to Elicit Many Probabilities}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {1999},
}
Figure