TL;DR
This paper introduces a method for evaluating and refining imprecise subjective probability assessments from medical experts within a belief network framework. By treating expert judgments as probability distributions and using scoring rules like the Brier score, the authors demonstrate how to quantify the quality of these assessments and how to update them with real-world data, leading to more robust and adaptive medical expert systems.
What Problem Does This Paper Address?
Developing effective medical expert systems often relies on encoding the knowledge of human experts. A key challenge in this process is representing the inherent uncertainty in medical diagnosis and prognosis. Belief networks offer a powerful way to model this, but they require quantitative conditional probability tables (CPTs) for each relationship in the network. Traditionally, eliciting these probabilities from experts has been difficult, often leading to demands for overly precise numbers that experts may not be comfortable providing. This paper tackles the problem of how to effectively elicit, assess, and improve imprecise subjective probability assessments from medical experts, recognizing that imprecision itself can be a valuable feature for learning and adaptation.
Methods
The core of the paper lies in its approach to handling imprecise probability assessments. Instead of forcing experts to give single point estimates, the authors allow them to provide ranges (e.g., 30-40%).
- Elicitation of Imprecise Probabilities: Experts provide consensus ranges for conditional probabilities, reflecting their uncertainty.
- Conversion to Point Probabilities: For evaluation purposes, the midpoints of these ranges are used, and if the midpoints don't sum to 1, they are rescaled. This allows for the application of standard scoring rules.
- Scoring Rules (Brier Score): The Brier score is used to penalize assessors for assigning low probabilities to events that occur. This score can be decomposed into measures of "discrimination" (how well probabilities distinguish between outcomes) and "reliability" (how well predicted probabilities match observed frequencies).
- Implicit Samples: Imprecise probability ranges are converted into "implicit samples." This is a crucial step that allows the expert's opinion to be treated as if it were derived from a certain number of observed cases.
- Learning from Experience (Bayesian Updating): The implicit samples are combined with actual observed data using Bayesian updating. This allows the system to learn from new cases and refine the conditional probabilities over time, effectively tempering expert judgment with real-world evidence.
Datasets
- Expert Judgments: Subjective conditional probabilities were elicited from three paediatric cardiologists for nearly 1000 assessments related to congenital heart disease. These assessments were provided as percentage ranges.
- Prospective Data: A dataset of 200 newborn infants with suspected congenital heart disease was collected prospectively. This data included recorded features observed over the telephone and definitive diagnoses made via echocardiogram or cardiac catheterization. This dataset was used to evaluate the accuracy of the expert assessments.
Key Insights and Ideas
- Imprecision as a Feature, Not a Flaw: The paper argues that imprecise probabilities are not a hindrance but a valuable mechanism for expert systems to learn and adapt.
- Decomposition of Assessment Quality: The Brier score's decomposition into discrimination and reliability provides a nuanced way to understand why an assessment might be good or bad, going beyond a single performance metric.
- Reliability is Key: The authors suggest that reliability (i.e., if an expert assigns a 70% probability, the event should occur about 70% of the time) is often more critical than discrimination for accurate disease presentation modeling.
- Expert Tendency Towards Extremes: The study found that experts tend to make overly extreme probability judgments (over-confidence), which can be identified through the reliability score.
- Learning Adapts Judgments: The conversion of imprecise assessments into implicit samples and subsequent Bayesian updating provides a practical and natural way for the system to revise its probabilities as more data becomes available.
Key Results
- Good Reliability: Overall, the expert assessments showed good reliability, with a tendency towards being too extreme (over-confidence).
- Varied Discrimination: Discrimination varied significantly across diseases and questions, highlighting areas where expert knowledge might be less clear or the presentation of the disease is inherently more varied.
- Hypoplastic Left Heart Assessments were Extraordinary: Judgments concerning hypoplastic left heart disease were found to be remarkably accurate when compared to observed data.
- Non-Urgent Heart Disease Assessments were Less Accurate: Conversely, expert opinions on the main problems associated with non-urgent heart disease were less aligned with the telephone-reported data.
- Adaptation is Slow with High Confidence: When experts are highly confident (providing narrow ranges), adaptation to new data is slower. Conversely, wider initial ranges lead to faster adaptation.
Limitations and Future Work
- Handling Zero Probabilities: The paper notes a problem with "zero" assessments, where 74 out of 3210 events assigned zero probability actually occurred. They suggest treating these as 0-4% based on an empirical error rate and are investigating alternative transformations.
- "Prequential" Approach: The authors highlight the need to investigate the "prequential" or sequential updating approach, where data from single cases is used to update probabilities sequentially, and these revised judgments are then used for subsequent cases. This promises to be the basis for automatic monitoring of expert system performance.
- Alternative Interval Transformations: Further research is needed to test the adequacy of the "one standard error interval" assumption used for transforming intervals into implicit samples.
- Contrast with Specialist Hospital Data: The current study used telephone-reported data, and the authors plan to contrast these results with data obtained at the specialist hospital to see if the over-confidence is related to the communication method.
Summary updated