When Feature Representation Changes Local Explanations: A Controlled Comparison of SHAP and LIME under Numerical Discretization

Parv PatelarXiv 2026 · arXiv

TL;DR

This paper reveals that common data-engineering choices, like binning a continuous variable, can significantly alter local feature explanations (SHAP and LIME) even when the underlying model remains frozen. The authors introduce a "seed-noise floor" to prove that while SHAP is highly sensitive to these representation changes, LIME’s response is often indistinguishable from its own random sampling noise.

What Problem Does This Paper Address?

Machine learning pipelines often involve "representation" choices—deciding whether to pass a person's age as a raw integer, bucket it into decades, or standardize it. While these choices are often made for engineering convenience, we read the resulting feature attributions (like "age was the most important factor") as statements about the underlying quantity itself.

The authors investigate a critical gap: Does the way we represent a feature change the explanation we get, even if the model's parameters never change? If explanations are hypersensitive to these "innocuous" preprocessing steps, their reliability for high-stakes decision-making (like loan approvals or medical diagnoses) comes into question.

Methods

The researchers designed a controlled experiment using a frozen Random Forest classifier. They focused on a single continuous feature—age—and applied two types of discretization:

  1. Equi-width binning: Dividing the age range into equal intervals.
  2. Quantile binning: Dividing age so each bin has roughly the same number of data points.

They tested these across different bin counts ($k \in \{3, 5, 10\}$). To ensure the results weren't just random artifacts, they established a seed-noise floor by re-running the explainers (KernelSHAP and LIME) on the same unchanged data with different random seeds. This allowed them to distinguish between a "representation effect" and simple algorithmic jitter.

Figure 1: Experimental design. The classifier’s parameters are frozen and the explained instance is never modified; what changes between conditions is the effective input the classifier receives. By Equation (3) this is exactly equivalent, for KernelSHAP, to explaining the unchanged model at q(x) against a quantized background—so the comparison is between two effective inputs, not a pure explainer effect. For LIME the two framings differ, because its perturbations are drawn in the raw coordinates.

Datasets

The study uses the UCI Adult income dataset, a standard benchmark in tabular machine learning. They used the official train/test split (32,561/16,281 rows) and focused their explanations on a stratified sample of 120 held-out instances.

Key Insights and Ideas

  • The "Noise Floor" is the Missing Control: Both SHAP and LIME are stochastic (they involve sampling). The authors argue that any study claiming a "representation effect" must first prove the effect is larger than the variation you get just by clicking "run" twice on the same data.
  • KernelSHAP's Mathematical Equivalence: The authors prove that for KernelSHAP, binning a feature before it hits a frozen model is mathematically identical to explaining the original model at a "quantized" point against a "quantized" background. This means the change isn't just an explainer quirk; it's a fundamental change in the "effective input" the model sees.
  • Magnitude vs. Ranking: A major "aha moment" is the dissociation between attribution values and feature rankings. While the specific numbers (magnitudes) shifted wildly, the list of "top 3 most important features" remained almost perfectly stable.
  • Prediction Agreement $\neq$ Explanation Stability: Even when the model's prediction remained exactly the same (no class flip), the SHAP attribution for age often changed drastically. Verifying that a pipeline change didn't hurt accuracy tells you almost nothing about whether the explanations survived.

Key Results

The quantitative results highlight a stark difference between the two explainers:

  • SHAP is sensitive but consistent: KernelSHAP’s age attribution exceeded its noise floor in all 6 test conditions, with median ratios ranging from 2.4 to 59 times the noise. However, it was much more consistent across repeats than LIME.
  • LIME is noisy and "robust" by accident: On a raw scale, LIME’s attribution change exceeded its noise floor in only 2 of 6 conditions. In the others, the change caused by binning was actually smaller than the variation caused by LIME's own internal sampling.
  • Ranking Stability: Despite the shifts in magnitude, the median Spearman rank correlation between original and discretized explanations never fell below 0.979, and the top-3 feature overlap was a perfect 1.00 in every condition.
Table 2: Comparison of attribution change ($\Delta$) against the seed-noise floor. Bold ratios indicate the effect exceeds the explainer's own noise.

Limitations and Future Work

  • Structural Confounding: In a frozen-model design, you cannot separate the "explainer effect" from the fact that the model is genuinely receiving a different input.
  • Scope: The study is limited to one dataset (UCI Adult), one model type (Random Forest), and one feature (age). Whether these patterns hold for neural networks or different data types remains to be seen.
  • Future Directions: The authors suggest sweeping LIME’s kernel width to see if its apparent robustness is just an artifact of "smoothing" over multiple bins. They also call for human-subject studies to determine if the magnitude shifts they measured actually change how a human would interpret the explanation.

Summary updated

Abstract

Local feature attributions are computed over a feature representation chosen by an upstream data-engineering step, yet they are read as statements about the underlying quantity. We ask how discretizing one continuous feature changes local KernelSHAP and LIME explanations when the classifier is held fixed, and whether the two explainers respond differently. A random forest is trained on UCI Adult and frozen; age is collapsed onto equi-width or quantile bin representatives (k in {3,5,10}) before the model receives it, and the same 120 held-out instances are explained in every condition across multiple seeds. We show analytically and numerically that for KernelSHAP this is exactly equivalent to explaining the unchanged model at the quantized instance against a quantized background, so the comparison is between two effective inputs rather than a pure explainer effect; for LIME no such equivalence holds. Each attribution change is compared with a seed-noise floor from re-running the explainer on the unchanged condition. KernelSHAP's age attribution exceeds its noise floor in all 6 pre-specified conditions (median ratios 2.4-59, all significant after Holm correction); on the raw scale LIME's exceeds its floor in only 2 of 6 and in the rest is no larger than its own run-to-run variation. SHAP's normalized change exceeds LIME's in every condition (rank-biserial 0.81-1.00), while LIME is 2.5-4.6x less consistent across repeats. Predicted classes change for only 0.8-5.8% of instances and held-out ROC-AUC never falls below 0.9097, yet 45-94% of instances show a large SHAP attribution change with no class flip. Feature-importance rankings barely move (median Spearman >= 0.979, top-3 overlap 1.00). The representation change and the change in effective input cannot be separated within a frozen-model design.
BibTeX
@misc{patel2026when,
  author = {Parv Patel},
  title = {{When Feature Representation Changes Local Explanations: A Controlled Comparison of SHAP and LIME under Numerical Discretization}},
  year = {2026},
  eprint = {2611.21159},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.21159},
}
Figure