Abstract
In business and economics, researchers increasingly rely on post-hoc explainers like SHAP and LIME to interpret complex "black-box" models. However, a critical gap exists: these tools are often used to draw conclusions about real-world data relationships rather than the models themselves. This paper introduces the concept of "data-alignment" to measure how well explainers recover ground-truth effects. Through simulations and theoretical proofs, the authors demonstrate that SHAP and LIME frequently fail to align with data reality, especially in high-dimensional or correlated settings, and offer mitigation strategies to avoid common pitfalls.
The Problem: The Explanation Pipeline
In fields where understanding the "why" is as important as the prediction itself, researchers have turned to eXplainable AI (XAI) to peek inside models like XGBoost or Deep Neural Networks. The authors identify a dangerous trend where researchers treat the output of these explainers as direct evidence of real-world causal or marginal effects.
This process, termed the "Explanation Pipeline," involves two distinct stages where errors can accumulate. First, the ML model must accurately characterize the data. Second, the explainer must accurately characterize the model. If either stage fails, the resulting "insights" about the relationship between explanatory and target variables are compromised.
Methodology and Metrics
To quantify these risks, the researchers surveyed 150 papers using SHAP or LIME, finding that 42% used these tools to make inferences about the data itself. They then introduced three metrics for "Data-Alignment":
- Directionality: Does the explainer correctly identify the sign (+/-) of a feature's effect?
- Concordance: Does the explainer rank feature importance in the correct order?
- Relevance: Does the explainer correctly identify the "Top-K" most important features?
These metrics were tested using simulated datasets with known linear ground truths, large-scale synthetic evaluations (up to 100 features), and a real-world case study on home loan applications.
Key Insights: Why Explainers Fail
The "Situational Importance" Trap
A primary finding is that SHAP measures "situational importance"—how much a feature's value deviates from its mean to contribute to a specific prediction. This is mathematically distinct from a "marginal effect." If a feature is at its mean value, SHAP may report zero importance even if that feature is the primary driver of the outcome in the real world.
Dimensionality and Correlation
As the number of features increases, the ability of SHAP and LIME to maintain data-alignment drops significantly. In high-dimensional spaces, SHAP's directionality performance often degrades to the level of random guessing. Furthermore, when features are correlated, explainers struggle to assign correct importance, often "smearing" effects across variables or producing incorrect signs.
The Rashomon Effect
Multiple models can achieve identical accuracy while using different internal logic. Explaining one high-performing model does not guarantee that the explainer has uncovered the "true" relationship inherent in the data.
Results and Mitigation Strategies
Surprisingly, the study found that LIME generally provided explanations more aligned with ground-truth data than SHAP, particularly regarding directionality. However, the authors proposed and tested several fixes:
- Normalized SHAP: Dividing the Shapley value by the difference between the feature value and its mean ($e_{SHAP} / (x - \bar{x})$) significantly improved concordance from near 0 to roughly 0.6.
- Ensemble Averaging for LIME: Averaging the results of 100 different LIME runs with different random seeds boosted the signal-to-noise ratio, leading to more reliable feature rankings.
Conclusion
Post-hoc explainers are valuable for understanding model behavior but should be used to propose hypotheses, not to validate them. They are not a substitute for rigorous econometric analysis. Researchers must exercise extreme caution when interpreting SHAP or LIME plots as reflections of real-world data relationships, as the map is not the territory.
Summary updated