To understand why a complex "black box" model makes a specific prediction, we often look at its immediate neighborhood. If we can approximate a complex model with a simple linear one in a small enough region, we can explain the prediction through feature importance. This is the core intuition behind methods like LIME and its derivatives.
However, the definition of "local" is not a neutral choice. Research across feature selection and model interpretability shows that the neighborhood bandwidth—the parameter governing how far we look from the point of interest—is the most critical factor in determining whether an explanation is insightful, stable, or even meaningful.
The Local Surrogate Intuition
The goal of local interpretability is to create a "surrogate" model that is easy to understand. As Mateusz Staniak and Przemyslaw Biecek demonstrate in their work on the live and localModel packages, this involves sampling points around a specific observation and weighting them based on their proximity.
In these frameworks, a kernel function (usually Gaussian) assigns weights to these samples. If a sample is close to the original point, it gets a high weight; if it is far, it gets a low weight. The "bandwidth" of this kernel acts like a zoom lens. A narrow bandwidth zooms in on the fine-grained behavior of the model, while a wide bandwidth pulls back to see the broader landscape.
The Stability-Fidelity Trade-off
The choice of bandwidth introduces a fundamental trade-off between local fidelity (how well the explanation matches the model at that specific point) and numerical stability.
Damien Garreau and Ulrike von Luxburg provide a rigorous theoretical analysis of this in Looking deeper into LIME. They find that when the bandwidth is too small, the resulting feature weights become highly volatile. In extreme cases, a tiny change in the sampling can cause the sign of a feature's importance to flip—telling you a feature is "positive" when it was "negative" a moment ago. This instability makes it difficult to trust explanations for high-stakes decisions.
Conversely, if the neighborhood is too large, the simple linear surrogate fails to capture the non-linear complexities of the original model. The explanation becomes a "global average" rather than a local insight.
From Local Sensitivity to Global Averages
The impact of bandwidth isn't just about noise; it changes the very nature of what the explanation represents. In What does LIME really see in images?, Damien Garreau and Dina Mardaoui prove that the bandwidth dictates a shift in perspective:
- Small Bandwidth: The explainer measures local sensitivity. It tells you how the model's output changes if you slightly nudge a feature.
- Large Bandwidth: The explainer measures mean difference. It tells you the difference between the model's average output and its output when a specific feature is present.
This mathematical connection reveals that for smooth models, LIME is essentially calculating a variation of Integrated Gradients over the input segments.
Beyond Interpretability: Local Geometric Structure
This sensitivity to locality is not unique to explainability; it is a fundamental property of how we model data geometry. Xiaofei He and Deng Cai explored this in the context of the Laplacian Score for Feature Selection.
By modeling data as a nearest-neighbor graph, they identified features that preserve the "local geometric structure." Just like in LIME, the choice of neighborhood size in the graph determines whether the algorithm prioritizes fine-grained local patterns or broader global structures. This suggests that the "bandwidth problem" is a universal challenge in any method that attempts to bridge the gap between local data properties and global model behavior.