Abstract
Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.
Introduction: The Limits of Attention
Transformers have revolutionized AI, moving from natural language processing (NLP) to becoming a dominant force in computer vision (CV). However, as these models grow more complex, understanding their decision-making process remains a challenge. While researchers often use "attention maps" to explain Transformer behavior, attention is not synonymous with relevance.
Existing visualization methods face three major hurdles:
- Attention ≠ Relevance: High attention might be given to background patches for context, but those patches may not drive the final classification.
- Class Agnosticism: Standard attention maps often look identical regardless of the target class. In an image with both a cat and a dog, raw attention fails to distinguish which pixels led to the "dog" prediction versus the "cat" prediction.
- Mathematical Instability: Transformers rely on skip connections and matrix multiplications that break standard explainability rules like Layer-wise Relevance Propagation (LRP), often leading to numerical instability.
Methodology: Propagating Relevancy
The proposed method uses Deep Taylor Decomposition (DTD) to propagate relevancy scores backward from the output to the input. Unlike previous heuristics, this approach ensures a "conservation rule," where the total relevance remains constant across layers.
Key technical innovations include:
- Gradient Integration: To make the method class-specific, attention maps are multiplied by their gradients with respect to a target class. This filters out noise and highlights only the information contributing to a specific label.
- The Identity Matrix Trick: To account for skip connections (which allow information to bypass attention layers), the authors add an identity matrix to the weighted attention relevance. This prevents the model from "inhibiting" itself during visualization.
- Normalization for Non-Parametric Layers: Specific normalization terms for skip connections and matrix multiplications ensure that relevancy scores do not explode or vanish.
Experimental Results
The authors benchmarked the method against established baselines like Rollout, GradCAM, and LRP across both Computer Vision (ImageNet) and NLP (Movie Reviews) datasets.
Qualitative Superiority
Visually, the method produces significantly cleaner heatmaps that tightly hug the contours of classified objects. A major advantage is its ability to distinguish between multiple objects in a single scene.
Quantitative Performance
- Computer Vision: On the ImageNet-Segmentation dataset, the method achieved a 61.95% mIoU (mean Intersection over Union), outperforming the next best method, Partial LRP (57.94%). In perturbation tests, removing the pixels identified as most relevant by this method caused the sharpest decline in model accuracy, proving the method correctly identifies critical features.
- NLP: On the Movie Reviews task, the method achieved higher token-F1 scores, meaning the words it highlighted as relevant were much closer to the human-annotated "rationales" for sentiment classification.
Conclusion
By moving beyond raw attention and incorporating class-specific gradients with a mathematically grounded propagation rule, this work provides a more reliable lens into the "black box" of Transformers. While challenges remain—such as handling sub-token artifacts in NLP and the computational cost of gradient propagation—this method sets a new standard for Transformer interpretability.
Summary updated