TL;DR
As machine learning models move into high-stakes domains like loan approval and recidivism prediction, "explaining" their decisions via feature importance has become standard practice. This paper reveals a sobering reality: the "important" features identified depend heavily on whether you use a model's built-in mechanisms or post-hoc tools like LIME. Crucially, post-hoc methods tend to make different models look more similar than they actually are, potentially masking fundamental differences in how these models process information.
What Problem Does This Paper Address?
We often rely on feature importance—highlighting the specific words in a text that "caused" a prediction—to build trust in AI. However, there are many ways to generate these highlights. You can use built-in mechanisms (like the weights in a linear model or attention scores in a Transformer) or post-hoc methods (like LIME or SHAP) that treat the model as a black box and try to approximate its behavior.
The authors tackle a fundamental question: Are these explanations consistent? If two models make the same prediction, do they do so for the same reasons? And if we use two different explanation methods on the same model, do they point to the same features? Understanding this consistency is vital for developers debugging models and for decision-makers who rely on these explanations to justify their actions.
Methods
The researchers conducted a systematic comparison using a variety of architectures and explanation techniques:
- Models: They compared traditional models (Linear SVM with $\ell_1$ and $\ell_2$ regularization, XGBoost) against deep learning architectures (LSTM with attention, BERT).
- Explanation Methods: They evaluated built-in importance (coefficients for SVM, gain for XGBoost, attention weights for LSTM/BERT) against post-hoc methods (LIME and SHAP).
- Evaluation Metric: To measure similarity, they used Jaccard similarity on the top-$k$ most important features (words). Essentially, they looked at the overlap between the sets of "top words" identified by different combinations of models and methods.
Datasets
The study focused on three text classification tasks:
- Yelp: Binary sentiment analysis (positive vs. negative reviews).
- SST (Stanford Sentiment Treebank): Sentence-level sentiment classification.
- Deception Detection: Identifying genuine vs. deceptive hotel reviews from TripAdvisor.
Key Insights and Ideas
- The Deep Learning Divergence: Traditional models like SVM and XGBoost tend to agree on which features are important. Deep learning models (LSTM, BERT), however, often focus on entirely different sets of words compared to their traditional counterparts.
- Post-hoc "Smoothing": Perhaps the most striking finding is that post-hoc methods like LIME generate more similar explanations across different models than built-in methods do. This suggests that post-hoc tools might be "smoothing over" the unique internal logic of different architectures, providing a consistent but potentially less "faithful" view of the model.
- Agreement $\neq$ Consistency: You might expect that when two models agree on a label (e.g., both say a review is "Positive"), their explanations would be more similar. The authors found this isn't always true. Models can arrive at the same conclusion for very different reasons, and conversely, they can disagree on a label while still finding similar features "important."
- The Entropy of Deep Models: Deep learning models exhibit higher entropy in their feature importance. While an SVM might fixate on a few highly predictive keywords, deep models spread their "attention" across a much wider and more diverse set of features across different instances.
Key Results
The researchers quantified these trends across thousands of instances:
- Model Similarity: As shown in Figure 1, the similarity between SVM ($\ell_1$) and SVM ($\ell_2$) is high (0.76), but the similarity between BERT and SVM is remarkably low (0.18).
- Method Comparison: Figure 2 demonstrates that post-hoc methods (dashed lines) consistently show higher similarity between models than built-in methods (solid lines). For example, in the Yelp dataset, LIME makes the SVM and LSTM look significantly more alike than their internal weights/attention scores suggest.
- Length Matters: There is a negative correlation between document length and explanation similarity (Figure 5). The longer the text, the less likely different methods are to agree on the "top" features, likely because the search space for "importance" becomes much larger.
Limitations and Future Work
The authors acknowledge several limitations:
- Feature Scope: The study is limited to unigrams (single words). Future work could explore how these findings hold for n-grams or more complex linguistic structures.
- Human-in-the-loop: While the paper characterizes the mathematical similarity of explanations, it doesn't measure how these differences affect human decision-making or trust.
- Task Specificity: The results are specific to text classification. It remains to be seen if these patterns of (in)consistency persist in image recognition or tabular data tasks.
Ultimately, this work serves as a cautionary tale: an "explanation" is not a direct window into a model's soul, but rather a product of the specific method used to generate it.
Summary updated