Abstract: Xi Ye, Rohan Nair, Greg Durrett. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
2021.
Introduction: The Faithfulness Gap in QA Explanations
Post-hoc explanation methods, such as feature attributions that highlight specific words (token attribution) as the "cause" of a prediction, have become popular tools for interpreting complex AI models. However, these explanations are often misleading. A highlight might appear "plausible" to a human observer, but it does not necessarily mean the model actually relied on that word in the way we assume. This discrepancy is known as the faithfulness problem.
In Reading Comprehension (RC), this issue is particularly acute. Answering a question like "Are both movies directed by the same person?" requires comparing two distinct entities. A simple highlight on the word "director" doesn't reveal if the model is performing a logical comparison or merely picking up on a keyword. This paper addresses the gap between low-level highlights and high-level model reasoning by asking: Can an explanation help us predict how the model will behave if we change the facts in the text?
A Framework for Evaluating Explanations
The authors propose a rigorous three-step methodology to evaluate whether AI explanations actually reflect a model's internal logic:
- Hypothesis Formulation: Define a theory of how the model works (e.g., "The model is comparing the 'national origin' property of these two people").
- Counterfactual Collection: Create a "local neighborhood" of examples where those specific properties are changed (e.g., changing "French" to "German").
- Simulation: Use the attribution method to predict if the model's answer will change on these new examples without actually running the model on them first. If the attribution "mass" is high on the changed tokens, we expect a prediction flip.
LATATTR: Layer-wise Attention Attribution
To improve faithfulness, the authors introduce LATATTR (Layer-wise Attention Attribution). While previous methods (like ATATTR) look at word-to-word interactions across all transformer layers simultaneously, LATATTR intervenes on one layer at a time. This prevents gradients from becoming distorted by the complex dependencies between layers, leading to more accurate "pairwise" explanations that capture how tokens interact rather than just identifying individual important words.
Key Insights: Plausibility vs. Faithfulness
A central finding of the study is that plausibility does not equal faithfulness. Some methods produce highlights that look excellent to humans but fail miserably at predicting model behavior. For instance, a model might highlight a "documentary" tag in a comparison task, yet continue to predict "Yes" even when that tag is changed to "romance." This indicates the model was not actually using that token for its decision, despite what the attribution suggested.
Furthermore, the authors found that pairwise attribution is essential for RC. Token-level attributions (like Integrated Gradients or LIME) struggle because RC tasks are fundamentally about the relationship between the question and the context. Pairwise methods are significantly better at capturing these interactions.
Results and Performance
The study evaluated these methods across three challenging datasets: HotpotQA (Yes/No), HotpotQA (Bridge), and SQuAD-ADV (Adversarial). Using Simulation Accuracy (S-ACC) to measure how well attributions predicted model flips, the results were conclusive:
- HotpotQA Yes-No: LATATTR achieved 84.0% S-ACC, significantly outperforming the best token-level method (72.0%).
- HotpotQA Bridge: Pairwise methods again led the pack, with LATATTR hitting 78.0% S-ACC.
- SQuAD Adversarial: LATATTR reached 70.0% S-ACC, proving it could identify when a model was relying on the right keywords versus being distracted by adversarial "noise."
Conclusion and Future Directions
By treating explanation as a "simulation" task, this research provides a concrete, quantitative way to measure how much an explanation actually helps us understand a model's "black box" logic. While the current approach relies on some manual heuristics and is labor-intensive due to the creation of realistic counterfactuals, it sets a new standard for faithfulness in NLP interpretability. Future work may involve using generative models to automate the creation of these counterfactual stress tests, further scaling the evaluation of model explanations.
Summary updated