TL;DR
This paper reveals that common data-engineering choices, like binning a continuous variable, can significantly alter local feature explanations (SHAP and LIME) even when the underlying model remains frozen. The authors introduce a "seed-noise floor" to prove that while SHAP is highly sensitive to these representation changes, LIME’s response is often indistinguishable from its own random sampling noise.
What Problem Does This Paper Address?
Machine learning pipelines often involve "representation" choices—deciding whether to pass a person's age as a raw integer, bucket it into decades, or standardize it. While these choices are often made for engineering convenience, we read the resulting feature attributions (like "age was the most important factor") as statements about the underlying quantity itself.
The authors investigate a critical gap: Does the way we represent a feature change the explanation we get, even if the model's parameters never change? If explanations are hypersensitive to these "innocuous" preprocessing steps, their reliability for high-stakes decision-making (like loan approvals or medical diagnoses) comes into question.
Methods
The researchers designed a controlled experiment using a frozen Random Forest classifier. They focused on a single continuous feature—age—and applied two types of discretization:
- Equi-width binning: Dividing the age range into equal intervals.
- Quantile binning: Dividing age so each bin has roughly the same number of data points.
They tested these across different bin counts ($k \in \{3, 5, 10\}$). To ensure the results weren't just random artifacts, they established a seed-noise floor by re-running the explainers (KernelSHAP and LIME) on the same unchanged data with different random seeds. This allowed them to distinguish between a "representation effect" and simple algorithmic jitter.
Datasets
The study uses the UCI Adult income dataset, a standard benchmark in tabular machine learning. They used the official train/test split (32,561/16,281 rows) and focused their explanations on a stratified sample of 120 held-out instances.
Key Insights and Ideas
- The "Noise Floor" is the Missing Control: Both SHAP and LIME are stochastic (they involve sampling). The authors argue that any study claiming a "representation effect" must first prove the effect is larger than the variation you get just by clicking "run" twice on the same data.
- KernelSHAP's Mathematical Equivalence: The authors prove that for KernelSHAP, binning a feature before it hits a frozen model is mathematically identical to explaining the original model at a "quantized" point against a "quantized" background. This means the change isn't just an explainer quirk; it's a fundamental change in the "effective input" the model sees.
- Magnitude vs. Ranking: A major "aha moment" is the dissociation between attribution values and feature rankings. While the specific numbers (magnitudes) shifted wildly, the list of "top 3 most important features" remained almost perfectly stable.
- Prediction Agreement $\neq$ Explanation Stability: Even when the model's prediction remained exactly the same (no class flip), the SHAP attribution for age often changed drastically. Verifying that a pipeline change didn't hurt accuracy tells you almost nothing about whether the explanations survived.
Key Results
The quantitative results highlight a stark difference between the two explainers:
- SHAP is sensitive but consistent: KernelSHAP’s age attribution exceeded its noise floor in all 6 test conditions, with median ratios ranging from 2.4 to 59 times the noise. However, it was much more consistent across repeats than LIME.
- LIME is noisy and "robust" by accident: On a raw scale, LIME’s attribution change exceeded its noise floor in only 2 of 6 conditions. In the others, the change caused by binning was actually smaller than the variation caused by LIME's own internal sampling.
- Ranking Stability: Despite the shifts in magnitude, the median Spearman rank correlation between original and discretized explanations never fell below 0.979, and the top-3 feature overlap was a perfect 1.00 in every condition.
Limitations and Future Work
- Structural Confounding: In a frozen-model design, you cannot separate the "explainer effect" from the fact that the model is genuinely receiving a different input.
- Scope: The study is limited to one dataset (UCI Adult), one model type (Random Forest), and one feature (age). Whether these patterns hold for neural networks or different data types remains to be seen.
- Future Directions: The authors suggest sweeping LIME’s kernel width to see if its apparent robustness is just an artifact of "smoothing" over multiple bins. They also call for human-subject studies to determine if the magnitude shifts they measured actually change how a human would interpret the explanation.
Summary updated