Abstract
To interpret Vision Transformers (ViTs), post-hoc explanations assign salience scores to input pixels, providing human-understandable heatmaps. However, whether these interpretations reflect true rationales behind the model's output is still underexplored. To address this gap, we study the faithfulness criterion of explanations: the assigned salience scores should represent the influence of the corresponding input pixels on the model's predictions. To evaluate faithfulness, we introduce the Salience-guided Faithfulness Coefficient (SaCo), a novel evaluation metric leveraging essential information of salience distribution. Specifically, we conduct pair-wise comparisons among distinct pixel groups and then aggregate the differences in their salience scores, resulting in a coefficient that indicates the explanation's degree of faithfulness. Our explorations reveal that current metrics struggle to differentiate between advanced explanation methods and Random Attribution, thereby failing to capture the faithfulness property. In contrast, our proposed SaCo offers a reliable faithfulness measurement, establishing a robust metric for interpretations. Furthermore, our SaCo demonstrates that the use of gradient and multi-layer aggregation can markedly enhance the faithfulness of attention-based explanation, shedding light on potential paths for advancing Vision Transformer explainability.
The Challenge of Faithfulness in ViTs
Vision Transformers (ViTs) have become a cornerstone of modern computer vision, yet they remain largely "black-box" models. To demystify their decision-making, researchers employ post-hoc explanation methods that generate salience maps—heatmaps highlighting pixels deemed "important" for a specific prediction. However, a visually intuitive heatmap does not necessarily guarantee faithfulness; it may not accurately reflect the model's internal logic.
A significant hurdle in this field is the inadequacy of current evaluation metrics. Standard metrics, such as Area Over the Perturbation Curve (AOPC) or Area Under the Curve (AUC), typically rely on cumulative perturbation. This process involves removing the "top 10%" most important pixels, then the "top 20%," and so on, while measuring the subsequent drop in model confidence. This approach suffers from a critical flaw: once the most vital features (e.g., the head of an animal) are removed, the model's confidence often collapses immediately. This makes it impossible to measure the true influence of the remaining 90% of pixels, effectively conflating their impacts and making it difficult to distinguish sophisticated explanation methods from random noise.
Introducing SaCo: A New Standard for Faithfulness
To address the limitations of cumulative metrics, we propose the Salience-guided Faithfulness Coefficient (SaCo). Rather than removing pixels cumulatively, SaCo evaluates pixel groups individually and compares their relative impacts. The methodology follows four key steps:
- Partitioning: The image is divided into $K$ subsets ($G_1, G_2, ... G_K$) based on their salience scores (e.g., the top 10%, 10-20%, etc.).
- Individual Perturbation: Each subset is replaced by the mean pixel value of the image one at a time. The resulting drop in model confidence ($\nabla pred$) is recorded for each specific subset.
- Pairwise Comparison: SaCo performs a statistical analysis of all possible pairs of subsets. It operates on a core assumption: if Subset A has a higher salience score than Subset B, removing A should cause a larger drop in model confidence than removing B.
- The Coefficient: If the assumption holds, the difference in salience scores is added as a "reward"; if it fails, it is subtracted as a "penalty." The final score is normalized between -1 and 1.
Experimental Insights
We validated SaCo across three standard benchmarks (CIFAR-10, CIFAR-100, and ImageNet) using popular ViT architectures, including ViT-B, ViT-L, and DeiT-B. We tested ten different explanation methods, ranging from Integrated Gradients and Grad-CAM to attention-specific methods like Rollout and Transformer-MM.
The Failure of Traditional Metrics
Our most striking finding was that existing metrics like AOPC often assign high scores to "Random Attribution"—essentially heatmaps of static noise. In contrast, SaCo correctly assigns Random Attribution a score near zero. Furthermore, the correlation between SaCo and existing metrics is remarkably low (0.18 to 0.22), while existing metrics correlate highly with each other (~0.47). This suggests that SaCo captures a fundamentally different and more rigorous dimension of faithfulness.
What Makes an Explanation Faithful?
Our analysis revealed several key insights into ViT interpretability:
- Attention is Not Enough: Raw attention maps are surprisingly unfaithful, suggesting that attention weights alone do not capture the full reasoning process of the Transformer.
- The Power of Gradients: The most faithful explanations are those that incorporate gradient information and multi-layer aggregation. Gradients provide class-specific direction, while multi-layer views capture the Transformer's hierarchical processing. In our experiments, adding gradient information to an attention-based method boosted its SaCo faithfulness score by 106%.
- Top Performers: Under the SaCo metric, Transformer-MM and ATTCAT emerged as the most faithful methods, significantly outperforming traditional approaches like LRP.
Conclusion and Future Work
While SaCo provides a more honest look at model faithfulness, it also highlights a significant gap in the field. Even the highest-performing explanation methods achieved SaCo scores between 0.4 and 0.6, indicating that current interpretations are only partially faithful to the model's actual operations.
Future research must address the computational cost associated with the pairwise comparisons in SaCo and explore the optimal granularity ($K$) for different tasks. Ultimately, SaCo serves as both a diagnostic tool and a roadmap, suggesting that the path to truly understanding Vision Transformers lies in the sophisticated integration of gradients and multi-layer attention analysis.
Summary updated