Transformer Interpretability Beyond Attention Visualization

Hila Chefer, Shir Gur, Lior Wolf

Abstract

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.

Introduction: The Limits of Attention

Transformers have revolutionized AI, moving from natural language processing (NLP) to becoming a dominant force in computer vision (CV). However, as these models grow more complex, understanding their decision-making process remains a challenge. While researchers often use "attention maps" to explain Transformer behavior, attention is not synonymous with relevance.

Existing visualization methods face three major hurdles:

  1. Attention ≠ Relevance: High attention might be given to background patches for context, but those patches may not drive the final classification.
  2. Class Agnosticism: Standard attention maps often look identical regardless of the target class. In an image with both a cat and a dog, raw attention fails to distinguish which pixels led to the "dog" prediction versus the "cat" prediction.
  3. Mathematical Instability: Transformers rely on skip connections and matrix multiplications that break standard explainability rules like Layer-wise Relevance Propagation (LRP), often leading to numerical instability.

Methodology: Propagating Relevancy

The proposed method uses Deep Taylor Decomposition (DTD) to propagate relevancy scores backward from the output to the input. Unlike previous heuristics, this approach ensures a "conservation rule," where the total relevance remains constant across layers.

Overview of the relevancy propagation method, showing how gradients and attention are combined with an identity matrix to account for skip connections.

Key technical innovations include:

  • Gradient Integration: To make the method class-specific, attention maps are multiplied by their gradients with respect to a target class. This filters out noise and highlights only the information contributing to a specific label.
  • The Identity Matrix Trick: To account for skip connections (which allow information to bypass attention layers), the authors add an identity matrix to the weighted attention relevance. This prevents the model from "inhibiting" itself during visualization.
  • Normalization for Non-Parametric Layers: Specific normalization terms for skip connections and matrix multiplications ensure that relevancy scores do not explode or vanish.

Experimental Results

The authors benchmarked the method against established baselines like Rollout, GradCAM, and LRP across both Computer Vision (ImageNet) and NLP (Movie Reviews) datasets.

Qualitative Superiority

Visually, the method produces significantly cleaner heatmaps that tightly hug the contours of classified objects. A major advantage is its ability to distinguish between multiple objects in a single scene.

Comparison of class-specific visualizations. Our method clearly distinguishes between the 'Dog' and 'Cat' or 'Elephant' and 'Zebra' in the same image, whereas baselines are often class-agnostic.

Quantitative Performance

  • Computer Vision: On the ImageNet-Segmentation dataset, the method achieved a 61.95% mIoU (mean Intersection over Union), outperforming the next best method, Partial LRP (57.94%). In perturbation tests, removing the pixels identified as most relevant by this method caused the sharpest decline in model accuracy, proving the method correctly identifies critical features.
  • NLP: On the Movie Reviews task, the method achieved higher token-F1 scores, meaning the words it highlighted as relevant were much closer to the human-annotated "rationales" for sentiment classification.

Conclusion

By moving beyond raw attention and incorporating class-specific gradients with a mathematically grounded propagation rule, this work provides a more reliable lens into the "black box" of Transformers. While challenges remain—such as handling sub-token artifacts in NLP and the computational cost of gradient propagation—this method sets a new standard for Transformer interpretability.

Summary updated

Abstract

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.
BibTeX
@inproceedings{chefer2021transformer,
  author = {Hila Chefer and Shir Gur and Lior Wolf},
  title = {{Transformer Interpretability Beyond Attention Visualization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=KyW34is-zsXn},
}
Figure