Abstract: Researchers have developed a unified framework to compare different flavors of AI explanations—ranging from simple word highlights to complex interactions between text spans—on a level playing field. Their findings reveal that while highlighting individual words is common, explaining the interactions between semantic spans is far more effective for helping humans understand and simulate model behavior. This research introduces metrics for faithfulness, human agreement, simulatability, and complexity to provide an "apples-to-apples" comparison of feature attribution methods.
1. Introduction
In the field of Explainable AI (XAI), feature attribution explanations typically fall into three categories:
- Token Explanations (TokenEx): Highlighting specific words (e.g., "The word 'banjo' caused this prediction").
- Token Interactive Explanations (TokenIntEx): Highlighting the relationship between two specific words.
- Span Interactive Explanations (SpanIntEx): Highlighting the relationship between entire phrases or "spans."
Historically, these methods have been studied in isolation because they use different "units" of explanation. This paper introduces a unified diagnostic framework to compare them across four key dimensions: Faithfulness (does it reflect the model?), Human Agreement (does it make sense to us?), Simulatability (can we predict the model's output using the explanation?), and Complexity (is it easy to digest?).
2. Methodology
The authors’ core contribution is a set of "unified" metrics that allow for a fair comparison between these disparate explanation types.
- Dynamic Thresholding: To ensure fairness, the framework ensures that every explanation type uses the same number of unique tokens when being tested. If a span-based explanation involves 10 words, the token-based explanation is allowed its top 10 words for that specific comparison.
- Base Attribution Techniques: The framework was tested using three standard "engines": Shapley Values, Integrated Gradients (IG), and Attention mechanisms.
- Span Extraction: For the more complex SpanIntEx, the researchers used the Louvain community detection algorithm to automatically group related tokens into semantic chunks before calculating their interactive importance.
- Agent-based Simulatability: Instead of relying solely on human intuition, the study trained "agent models" to see if an AI could predict the original model's behavior when given only the explanations.
3. Experimental Setup
The study focused on two "complex reasoning" tasks where relationships between different parts of the text are critical:
- SNLI (Stanford Natural Language Inference): Determining if a hypothesis contradicts or entails a premise.
- FEVER (Fact Extraction and VERification): Verifying claims against evidence.
The researchers utilized BERT and BART models and compared the AI-generated explanations against human-annotated datasets (e-SNLI and SpanEx) to measure alignment with human logic.
4. Results and Analysis
The comparative analysis yielded several key insights into the effectiveness of different attribution methods:
- The "Context" Advantage: SpanIntEx (span interactions) emerged as the overall winner. By providing semantic context rather than isolated words, these explanations were significantly more "simulatable" and aligned better with human intuition. SpanIntEx outperformed other types in 9 out of 12 cases, improving an agent's ability to predict model outputs by up to 7.9 F1 points.
- The Faithfulness Trade-off: There is a distinct tension between Comprehensiveness (how much of the model's reasoning is captured) and Plausibility (how much it makes sense to a human). Token-level highlights are often more "faithful" to the raw math of the model, but they are harder for humans to use for reasoning.
- Complexity isn't Linear: While one might assume interaction-level explanations are more complex, the study found that TokenIntEx (word-to-word pairs) was the most complex and "noisy." SpanIntEx achieved a level of simplicity (low entropy) comparable to basic word highlights because the spans are semantically coherent.
5. Conclusion and Future Directions
This research demonstrates that span-based interactions provide a superior balance of faithfulness and human utility compared to traditional token highlights. While the study was limited to BERT and BART models, it provides a robust framework for future evaluation. Future work will focus on scaling this analysis to massive decoder-only Large Language Models (LLMs) and exploring hybrid approaches that combine the raw faithfulness of token-level attribution with the clarity of span-level interactions.
Summary updated