Introduction
Post-hoc explanation methods, such as Integrated Gradients or LIME, are frequently used to interpret the decisions of complex neural models by highlighting "important" input tokens. However, in Reading Comprehension (RC), these highlights often fail to capture the multi-hop reasoning or entity comparisons required for a correct prediction. A significant challenge remains: do these attributions represent the model's internal reasoning (faithfulness), or are they merely "plausible" to human observers?
This paper introduces a framework to evaluate the faithfulness of RC model explanations by testing their simulatability—the degree to which an explanation allows a user to predict model behavior changes when the input is modified. The authors argue that standard token-level attributions are insufficient for RC and propose LaAttrAttn (Layer-wise Attention Attribution), a method that captures essential pairwise token interactions.
Methodology: Connecting Attributions to Behavior
The core of the evaluation framework relies on realistic counterfactuals. The process involves three steps:
- Hypothesis Formulation: Proposing a theory about which tokens the model relies on for a specific decision.
- Counterfactual Collection: Generating realistic variations of the input by perturbing the tokens identified in the hypothesis.
- Simulation: Using the attribution mass on the perturbed tokens to predict whether the model's output will change on the counterfactual example.
To improve simulation accuracy, the authors introduce LaAttrAttn. Unlike standard attention attribution that intervenes on all layers simultaneously—potentially introducing out-of-distribution artifacts—LaAttrAttn applies Integrated Gradients layer-by-layer. This isolates the contribution of specific attention links while maintaining the natural flow of computation through the rest of the transformer architecture.
Key Insights: Plausibility vs. Faithfulness
A central finding of this research is that "plausible" explanations are often unfaithful. For instance, a model might highlight the word "documentary" as a key feature for a "YES" prediction. However, if the model still predicts "YES" when "documentary" is changed to "romance," the original highlight was a poor representation of the model's actual logic.
Furthermore, the authors demonstrate that pairwise interactions are essential. Reading comprehension fundamentally involves relationships (e.g., matching a question entity to a context entity). Token-level attributions collapse these relationships into a flat list, losing the context necessary for accurate simulation.
Results and Discussion
The proposed LaAttrAttn method significantly outperformed baseline attribution methods in simulation accuracy (S-ACC) across several RC tasks:
- HotpotQA Yes-No: Achieved 84.0% S-ACC, compared to 72.0% for Integrated Gradients and LIME.
- HotpotQA Bridge: Reached 78.0% S-ACC, effectively revealing when models rely on "primary question" shortcuts rather than full reasoning.
- Adversarial SQuAD: Achieved 70.0% S-ACC, proving more effective at exposing model vulnerabilities than raw confidence scores.
While successful, the method currently relies on heuristic mapping to connect low-level scores to behavioral predictions. Future work could involve training meta-models for simulation and automating the generation of high-quality counterfactuals. Additionally, accounting for architectural biases, such as positional embeddings, remains an open challenge for attribution-based explanations.
Summary updated