Integrated Directional Gradients: Feature Interaction Attribution for Neural NLP Models

Sandipan Sikdar, Parantapa Bhattacharya, Kieran HeeseACL 2021

TL;DR

This paper introduces Integrated Directional Gradients (IDG), a novel method for attributing importance scores to groups of features in neural networks, specifically for Natural Language Processing (NLP) tasks. IDG aims to better capture how complex feature interactions contribute to model predictions, addressing a key limitation of existing methods that often focus on individual features or lack a solid theoretical foundation for group attributions.

What Problem Does This Paper Address?

Deep Neural Networks (DNNs) excel at learning complex, higher-order feature interactions, which is a primary reason for their success in NLP. However, understanding how these interactions contribute to a model's decision is crucial for interpretability and trust. Existing methods for explaining DNNs often focus on individual feature importance or struggle to robustly attribute importance to groups of features (i.e., feature interactions). This gap makes it difficult to understand the nuanced ways words, phrases, and their relationships influence NLP model outputs. The authors aim to formally define and solve the "feature group attribution problem," ensuring that explanations are intuitive and theoretically sound.

Methods

The core of the paper is the Integrated Directional Gradients (IDG) method. IDG is inspired by cooperative game theory and builds upon the Integrated Gradients (IG) approach.

  • Feature Group Attribution Problem: The authors formally define this problem as assigning an importance score to any meaningful subset of features (a "coalition"). This is framed as a cooperative game where features are "players" contributing to the model's output.
  • Axiomatic Approach: They propose a set of axioms (properties) that a good feature group attribution method should satisfy, including Non-Negativity, Normality, Monotonicity, Superadditivity, Sensitivity, Symmetry Preservation, and Implementation Invariance.
  • IDG Formulation: IDG calculates an "Integrated Directional Gradient" for a feature group by integrating the directional derivative of the DNN's output along a straight path from a baseline input to the actual input. The absolute value of this integral represents the "dividend" of the feature group. The final "value" (importance score) of a group is the sum of dividends of all its constituent groups.
  • Efficient Computation: An efficient dynamic programming algorithm is proposed to compute IDG for all meaningful feature groups within a hierarchical structure (like a parse tree).

Datasets

The authors trained and evaluated their models on three standard NLP datasets for sentiment analysis:

  • Stanford Sentiment Treebank (SST)
  • Yelp Reviews
  • IMDB Reviews

Key Insights and Ideas

  • Feature Groups are Crucial: The paper strongly advocates for attributing importance to groups of features (interactions) rather than just individual features, as DNNs inherently learn these complex relationships.
  • Cooperative Game Theory Foundation: Framing feature attribution as a cooperative game provides a robust theoretical framework, allowing for the definition of intuitive axioms that guide the development of attribution methods.
  • Characteristic Functions are Key: The authors argue that characteristic functions (which define the value of any coalition) are more suitable for modeling group importance than just solution concepts (like Shapley values, which are typically for distributing value among players).
  • IDG Captures Nuance: IDG is shown to effectively capture semantic interactions, such as negations and conjunctions, in linguistic models.
  • Axiomatic Guarantees: IDG is designed to satisfy most of the proposed axioms, providing a theoretically sound explanation method.

Key Results

  • Qualitative Evaluation: The paper presents qualitative comparisons of IDG against other feature interaction attribution methods, highlighting IDG's adherence to desirable axiomatic properties.
  • Model Performance: The trained models (XLnet-base, XLnet-large, BERT-itpt) achieved high accuracies on the sentiment analysis tasks, with results reported in Table 2 (e.g., SST: 0.915-0.916, Yelp: 0.947-0.983, IMDB: 0.957-0.967).
  • Demonstrated Interactions: Figures 2, 3, 4, and 5 showcase IDG's ability to attribute importance to phrases and their interactions, correctly identifying positive and negative contributions within sentences, even when the overall sentiment is mixed or misclassified. For instance, IDG can highlight a positive phrase within a negative review, demonstrating its ability to capture nuanced contributions.

Limitations and Future Work

  • Linearity Axiom: IDG sacrifices the "Linearity" axiom for the sake of normalization and intuitive results. The authors acknowledge this but argue that the importance of uniqueness claims is diminished in modern attribution research.
  • Adversarial Attacks: While IDG doesn't evaluate out-of-distribution inputs (a common vulnerability for other methods), the authors acknowledge that adversarial training could potentially deceive IDG. They suggest exploring defenses against such attacks as a future direction.
  • Reproducibility: The implementation of IDG is made publicly available to facilitate reproducibility.

Summary updated

BibTeX
@inproceedings{sikdar2021integrated,
  author = {Sandipan Sikdar and Parantapa Bhattacharya and Kieran Heese},
  title = {{Integrated Directional Gradients: Feature Interaction Attribution for Neural NLP Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure