Many faces of feature importance: Comparing built-in and post-hoc feature importance in text classification

Vivian Lai, Jon Cai, Chenhao Tan

Introduction

As machine learning models are increasingly deployed in high-stakes environments—ranging from legal recidivism predictions to financial loan approvals—the demand for "explainability" has surged. Feature importance, which identifies the specific input tokens (e.g., words in a text) that most heavily influence a model's prediction, is a primary tool for building trust and debugging these systems. However, a critical question remains: do different explanation methods agree on what is "important"? This paper investigates the consistency of feature importance across various model architectures and explanation techniques, revealing significant discrepancies that challenge our reliance on these interpretations.

Methodology and Experimental Framework

The researchers conducted a systematic comparison using five distinct models: traditional "shallow" models (Linear SVM with $\ell_1$ and $\ell_2$ regularization, and XGBoost) and deep learning architectures (LSTM with attention and BERT). They evaluated two categories of explanation methods:

  1. Built-in Methods: Model-specific metrics such as SVM coefficients, XGBoost training loss gain, and attention weights for LSTMs and BERT.
  2. Post-hoc Methods: Model-agnostic tools like LIME (Local Interpretable Model-agnostic Explanations) and SHAP (SHapley Additive exPlanations), which estimate importance by perturbing inputs and observing model reactions.

To quantify agreement, the study employed Jaccard similarity on the top-$k$ most important features. This metric measures the overlap between the sets of words identified as most influential by different methods. The evaluation spanned three benchmarks: Yelp sentiment analysis, the Stanford Sentiment Treebank (SST), and a complex Deception Detection task involving hotel reviews.

The Deep Learning Divide

A primary finding of the study is the fundamental divergence between traditional and deep learning models. Regardless of the explanation method used, SVMs and XGBoost models tended to agree more with each other than they did with LSTMs or BERT. This suggests that deep architectures process text in a fundamentally different manner than linear or tree-based models.

Inter-model agreement (Jaccard similarity) using built-in feature importance methods.

As illustrated in Figure 1, the Jaccard similarity between SVM ($\ell_2$) and XGBoost using built-in methods is approximately 0.38. In contrast, the similarity between these traditional models and BERT is significantly lower, hovering around 0.18. This "Deep Learning Divide" indicates that the internal logic of neural networks is not easily reconciled with the feature-weighting logic of traditional classifiers.

The Smoothing Effect of Post-hoc Explanations

Perhaps the most striking discovery is the "smoothing" effect of post-hoc explainers. The researchers found that methods like LIME and SHAP often generate more similar explanations for two different models than the models' own built-in mechanisms do. This suggests that post-hoc tools may inadvertently mask the unique internal logic of a model, imposing their own structural biases (such as LIME's local linearity) onto the explanation.

Jaccard similarity between SVM and LSTM using built-in vs. post-hoc methods (LIME, SHAP) across three datasets.

Figure 3 demonstrates this phenomenon across the Yelp, SST, and Deception datasets. When comparing an SVM and an LSTM, the Jaccard similarity of their explanations is consistently higher when using LIME or SHAP than when using their respective built-in importance metrics. This implies that LIME can make a BERT model and an SVM appear to be utilizing the same features, even when their internal decision-making processes are worlds apart.

Linguistic Complexity and Agreement Paradoxes

The study also highlighted how input complexity affects explanation consistency. Longer text instances and those with a higher type-token ratio (more unique words) resulted in lower agreement between methods. Furthermore, the researchers identified an "Agreement Paradox": even when two models agree on a predicted label (e.g., both correctly identifying a review as "Positive"), they do not necessarily rely on the same features to reach that conclusion.

Linguistic analysis revealed that while adjectives are universally important for sentiment tasks, models diverge on more complex tasks. In deception detection, LSTMs placed a much heavier emphasis on nouns than traditional models did. Additionally, deep learning models exhibited higher entropy in their feature importance, relying on a more diverse set of keywords across instances compared to the more concentrated reliance seen in XGBoost.

Conclusion

The findings of this paper serve as a critical warning for practitioners: feature importance is not an absolute truth. It is as much a product of the explanation method as it is a reflection of the model itself. Because post-hoc methods can smooth over fundamental model differences, they may provide a false sense of consistency. Future work must move beyond measuring consistency to determining which of these conflicting explanations is most useful for human decision-making and system auditing.

Summary updated

Abstract

Feature importance is commonly used to explain machine predictions. While feature importance can be derived from a machine learning model with a variety of methods, the consistency of feature importance via different methods remains understudied. In this work, we systematically compare feature importance from built-in mechanisms in a model such as attention values and post-hoc methods that approximate model behavior such as LIME. Using text classification as a testbed, we find that 1) no matter which method we use, important features from traditional models such as SVM and XGBoost are more similar with each other, than with deep learning models; 2) post-hoc methods tend to generate more similar important features for two models than built-in methods. We further demonstrate how such similarity varies across instances. Notably, important features do not always resemble each other better when two models agree on the predicted label than when they disagree.
BibTeX
@inproceedings{lai2019many,
  author = {Vivian Lai and Jon Cai and Chenhao Tan},
  title = {{Many faces of feature importance: Comparing built-in and post-hoc feature importance in text classification}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=KrMJd2U1gF},
}
Figure