Introduction
As machine learning models are increasingly deployed in high-stakes environments—ranging from legal recidivism predictions to financial loan approvals—the demand for "explainability" has surged. Feature importance, which identifies the specific input tokens (e.g., words in a text) that most heavily influence a model's prediction, is a primary tool for building trust and debugging these systems. However, a critical question remains: do different explanation methods agree on what is "important"? This paper investigates the consistency of feature importance across various model architectures and explanation techniques, revealing significant discrepancies that challenge our reliance on these interpretations.
Methodology and Experimental Framework
The researchers conducted a systematic comparison using five distinct models: traditional "shallow" models (Linear SVM with $\ell_1$ and $\ell_2$ regularization, and XGBoost) and deep learning architectures (LSTM with attention and BERT). They evaluated two categories of explanation methods:
- Built-in Methods: Model-specific metrics such as SVM coefficients, XGBoost training loss gain, and attention weights for LSTMs and BERT.
- Post-hoc Methods: Model-agnostic tools like LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations), which estimate importance by perturbing inputs and observing model reactions.
To quantify agreement, the study employed Jaccard similarity on the top-$k$ most important features. This metric measures the overlap between the sets of words identified as most influential by different methods. The evaluation spanned three benchmarks: Yelp sentiment analysis, the Stanford Sentiment Treebank (SST), and a complex Deception Detection task involving hotel reviews.
The Deep Learning Divide
A primary finding of the study is the fundamental divergence between traditional and deep learning models. Regardless of the explanation method used, SVMs and XGBoost models tended to agree more with each other than they did with LSTMs or BERT. This suggests that deep architectures process text in a fundamentally different manner than linear or tree-based models.
As illustrated in Figure 1, the Jaccard similarity between SVM ($\ell_2$) and XGBoost using built-in methods is approximately 0.38. In contrast, the similarity between these traditional models and BERT is significantly lower, hovering around 0.18. This "Deep Learning Divide" indicates that the internal logic of neural networks is not easily reconciled with the feature-weighting logic of traditional classifiers.
The Smoothing Effect of Post-hoc Explanations
Perhaps the most striking discovery is the "smoothing" effect of post-hoc explainers. The researchers found that methods like LIME and SHAP often generate more similar explanations for two different models than the models' own built-in mechanisms do. This suggests that post-hoc tools may inadvertently mask the unique internal logic of a model, imposing their own structural biases (such as LIME's local linearity) onto the explanation.
Figure 3 demonstrates this phenomenon across the Yelp, SST, and Deception datasets. When comparing an SVM and an LSTM, the Jaccard similarity of their explanations is consistently higher when using LIME or SHAP than when using their respective built-in importance metrics. This implies that LIME can make a BERT model and an SVM appear to be utilizing the same features, even when their internal decision-making processes are worlds apart.
Linguistic Complexity and Agreement Paradoxes
The study also highlighted how input complexity affects explanation consistency. Longer text instances and those with a higher type-token ratio (more unique words) resulted in lower agreement between methods. Furthermore, the researchers identified an "Agreement Paradox": even when two models agree on a predicted label (e.g., both correctly identifying a review as "Positive"), they do not necessarily rely on the same features to reach that conclusion.
Linguistic analysis revealed that while adjectives are universally important for sentiment tasks, models diverge on more complex tasks. In deception detection, LSTMs placed a much heavier emphasis on nouns than traditional models did. Additionally, deep learning models exhibited higher entropy in their feature importance, relying on a more diverse set of keywords across instances compared to the more concentrated reliance seen in XGBoost.
Conclusion
The findings of this paper serve as a critical warning for practitioners: feature importance is not an absolute truth. It is as much a product of the explanation method as it is a reflection of the model itself. Because post-hoc methods can smooth over fundamental model differences, they may provide a false sense of consistency. Future work must move beyond measuring consistency to determining which of these conflicting explanations is most useful for human decision-making and system auditing.
Summary updated