TL;DR
This paper investigates why different post-hoc explanation methods for NLP models often disagree on which parts of the input are most important. They find that methods systematically prefer certain word classes and that agreement improves when looking at syntactic spans instead of individual tokens, especially when using a dynamic approach to select the number of important tokens.
What Problem Does This Paper Address?
Post-hoc explanation methods are crucial for understanding how complex NLP models make decisions. However, these methods often produce conflicting explanations, raising questions about their reliability and faithfulness to the model's internal workings. This disagreement is particularly evident when comparing the top-k most important tokens identified by different methods. The paper aims to pinpoint the sources of this disagreement, moving beyond simple token-level comparisons to explore linguistic preferences and the impact of how "important" tokens are selected.
Methods
The authors employ a multi-faceted approach:
- Linguistic Analysis of Token Preferences: They analyze the part-of-speech (POS) tags, stop words, and punctuation of tokens identified as important by various explanation methods. This helps reveal systematic biases in what each method "looks at."
- Syntactic Span Analysis: Instead of just individual tokens, they group tokens into syntactically defined spans (like noun phrases or verb phrases) and analyze agreement at this higher level. This is done using a shallow parser (Flair chunker).
- Dynamic Top-k Estimation: They investigate the impact of how the number of "top-k" important tokens is determined. They compare a fixed
kwith a dynamic approach that automatically adjustskbased on the attribution profile of each instance, aiming to capture "signal peaks." - Agreement Metrics: They use standard agreement metrics (e.g., @k agreement) to quantify how much different methods and human annotations overlap in their selections, both at the token and span levels.
- Baseline Comparison: To gauge the difficulty of the task and the reliability of methods, they compare method agreement against a pseudo-random baseline.
Datasets
- e-SNLI Dataset: The default training split (549,361 instances) was used to fine-tune a DistilBERT model. The analysis was performed on the default test split (9,842 instances). This dataset is a multi-class problem for natural language inference (contradiction, entailment, neutral).
- Human Rationales: For each instance in the dataset, human annotators (3 per instance) identified important words, providing a "human preference" baseline for comparison.
Key Insights and Ideas
- Word Class Preferences Matter: Different explanation methods systematically favor certain word classes (e.g., nouns, verbs, punctuation). Methods that agree more with each other and with humans tend to exhibit similar word class preferences.
- Spans Smooth Out Disagreement: Agreement between explanation methods significantly improves when looking at syntactic spans rather than individual tokens. This suggests that while methods might disagree on the exact token, they often agree on the broader linguistic constituent.
- Dynamic
kis Key for Spans: A dynamic approach to selecting the number of important tokens (k) is more effective for span-level agreement than a fixedk. It helps capture important signals more robustly. - Punctuation as a Locality Signal: Some methods' consistent preference for punctuation might indicate they are picking up on the location of important signals rather than the punctuation itself being lexically important.
- Two Groups of Methods Emerge: The analysis reveals two distinct groups of methods based on their word class preferences, with one group (including Integrated Gradient and its variants) showing different patterns than the other and humans.
Key Results
- Span Agreement Improvement: Span-level agreement was generally higher than token-level agreement. For example, with dynamic
k, span agreement reached 0.69 on average, compared to 0.61 for tokens. - Dynamic
kBoosts Span Agreement: Dynamickprovided marginal boosts to token-level agreement but a more significant positive effect on span-level agreement, especially for methods like Gradient x Input and Integrated Gradient. - Human vs. Method Agreement: The study found that human-method agreement might reach a ceiling at the token level, suggesting that even faithful methods might not perfectly align with human intuition on individual token importance.
- Baseline Outperformance: Span-wise agreement with a fixed
k=4was outperformed by a baseline by 0.12, while token-wise agreement was outperformed by 0.07, indicating the relative difficulty of achieving agreement. - Chi-Square Tests: Statistical tests confirmed significant differences in word class preferences between the two identified groups of methods.
Limitations and Future Work
- Generalizability of Spans: The authors acknowledge that the utility of span-based agreement might depend on the task. For tasks where modifiers vs. heads are critical, span-level agreement might be less informative.
- Human-Method Agreement: While human rationales were used, a more direct assessment of human-method agreement was left out of scope.
- Faithfulness Measurement: Measuring true faithfulness remains challenging. Agreement is proposed as a proxy, but it's not a direct measure.
- Future Work:
- Investigate different local importance criteria for dynamic
k.
- Investigate different local importance criteria for dynamic
- Explore how syntactic span information can improve token-level interpretability.
- Develop better explanation aggregation techniques.
- Use datasets with explicit instance-annotator mappings to better assess plausibility and human subjectivity.
- Further research into the role of punctuation and locality information.
Summary updated