TL;DR
Shapley values are not a "one-size-fits-all" tool for medical AI; their meaning changes drastically depending on how you handle "missing" data. This paper demonstrates that clinicians must carefully match their specific clinical question—whether it's about fairness, feature selection, or justifying a diagnosis—to the correct mathematical flavor of Shapley values to avoid dangerous misinterpretations.
What Problem Does This Paper Address?
As machine learning enters routine clinical practice, "black-box" models need to be explainable to be safe and trusted. Shapley values (popularized by tools like SHAP) have become the gold standard for this. They work by "removing" features to see how the prediction changes.
However, there is a fundamental technical hurdle: How do you "remove" a feature from a model that was trained to expect it? You have to fill that gap with a "reference distribution." The choice of this distribution leads to a heated debate between being "true to the model" (Marginal Shapley) or "true to the data" (Conditional Shapley). This paper addresses the confusion surrounding these choices and provides a roadmap for using them correctly in high-stakes healthcare settings.
Methods
The authors provide a theoretical and conceptual analysis of three main approaches to Shapley values:
- Marginal Shapley Values: These treat features as independent. When a feature is "removed," it is replaced by a random value from the population. This is often called "interventional."
- Conditional Shapley Values: These account for correlations. If you remove "smoking status," the reference value is sampled based on what we know about the patient's other features (like "age" or "lung capacity"). This is often called "observational."
- Causality-based Shapley Values: These incorporate a causal graph of the data, attempting to attribute importance to "root causes" rather than just correlated symptoms.
The authors use mathematical proofs and "toy" clinical examples (e.g., predicting cardiovascular disease based on cholesterol and blood sugar) to demonstrate how each method can lead to different—and sometimes misleading—conclusions.
Datasets
The paper does not use a large-scale real-world dataset. Instead, it relies on carefully constructed synthetic counter-examples and mathematical derivations. These examples are designed to isolate specific behaviors of the algorithms, such as how they handle shifted data distributions or different feature spreads, which might be obscured in complex real-world data.
Key Insights and Ideas
- The "True to Model" Myth: A common misconception is that Marginal Shapley values only depend on the model. The authors prove that even these are heavily influenced by the data's distribution (specifically the mean and spread). If two patients have the same clinical profile but belong to different cohorts, their "model-true" explanations could look totally different.
- Fairness is Tricky: If you want to check for counterfactual fairness (e.g., "Would the model treat this patient differently if they were a different gender?"), Marginal Shapley is necessary. Conditional Shapley can actually "hide" bias by smearing the influence of a sensitive attribute across other correlated features.
- The Feature Selection Trap: Many researchers use a Shapley value of zero to mean a feature is useless. The authors show this is false: a feature can have a zero Shapley value in a specific instance even if it is vital to the model's overall logic.
- Justification vs. Explanation: For a doctor needing to justify a treatment, Causality-based Shapley is the most intuitive because it follows biological "root causes." However, it is the hardest to implement because it requires an accurate causal map of human physiology.
Key Results
- Statistical Fairness: The paper notes that while Marginal Shapley is better for detecting direct bias, Conditional Shapley is better for evaluating how a model performs on the "data manifold" (real-world plausible patients), preventing the model from being evaluated on "impossible" synthetic patients.
- Clinical Use-Case Mapping: The authors provide a definitive guide for which distribution to use:
- Counterfactual Fairness: Use Marginal.
- Feature Selection: Use Conditional (with caution).
- Clinical Justification: Use Causality-based.
- Mathematical Proofs: The supplementary material provides rigorous proofs showing that even in a simple linear model ($x_1 + x_2$), the feature attributions will be unequal if the features have different variances, even though they are algebraically identical in the model.
Limitations and Future Work
The primary limitation is that Causality-based Shapley values—the most promising for clinical "why" questions—rely on having a correct causal graph. In medicine, these graphs are often unknown or incredibly complex.
Furthermore, the authors point out that if the underlying black-box model is performing poorly, no amount of clever "explanation" will make it safe. They suggest that future research should focus on developing "causal models" from the start, rather than trying to retroactively apply causal explanations to non-causal black boxes.
Summary updated