Transformer Interpretability Beyond Attention Visualization

Hila Chefer, Shir Gur, Lior Wolf

Abstract

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.

Introduction

While Transformers have revolutionized Natural Language Processing (NLP) and are rapidly becoming the standard for Computer Vision (ViT), they remain largely "black boxes." Understanding why a model makes a specific prediction is crucial for trust and debugging. Traditionally, researchers have relied on raw attention maps to visualize model logic. However, these maps suffer from several critical flaws. First, they are often class-agnostic; the attention map for a "dog" and a "cat" in the same image often looks identical. Second, as information flows through deep networks, simple attention averages become blurry, leading to noise accumulation. Finally, the complex architecture of Transformers—featuring skip connections and non-linearities like GELU—presents significant hurdles for traditional explainability tools like Layer-wise Relevance Propagation (LRP).

Methodology: Relevancy Propagation

To address these issues, we introduce a novel framework based on Deep Taylor Decomposition (DTD) principles. Our method moves beyond simple attention visualization by propagating relevancy scores backward through the network in a mathematically grounded way.

Overview of the relevancy propagation method

The approach is built on three pillars:

  • Relevancy Propagation Rules: We define specific rules to pass relevance scores through the network, accounting for non-ReLU activations and maintaining a "total sum of relevancy" across layers.
  • Normalization for Non-Parametric Layers: Transformers utilize "add" (skip connections) and "matmul" (attention) operations that lack learned weights. We introduce a normalization term to ensure numerical stability, preventing scores from exploding or vanishing.
  • Class-Specific Integration: By weighting attention maps with their gradients relative to a target class, we ensure the resulting heatmaps are class-specific. The final relevancy map $\mathbf{C}$ is computed by iteratively multiplying weighted transition matrices across all layers: $\mathbf{C} = \bar{\mathbf{A}}^{(1)} \cdot \bar{\mathbf{A}}^{(2)} \dots \bar{\mathbf{A}}^{(B)}$. Each block's contribution is defined as: $$\bar{A}^{(b)} = I + \mathbb{E}_h(\nabla A^{(b)} \odot R^{(n_b)})^+$$ This formulation ensures that only features positively contributing to the specific class are highlighted, while the identity matrix ($I$) accounts for the crucial role of skip connections.

Experimental Results

We evaluated our method across both Computer Vision and NLP domains. For vision, we utilized the ImageNet validation set for perturbation tests and the ImageNet-Segmentation dataset to measure alignment with object boundaries. For NLP, we used the Movie Reviews dataset from the ERASER benchmark.

Comparison of visualization methods across different classes

As shown in the visual comparisons, our method produces significantly clearer and more accurate heatmaps than baselines like Grad-CAM or Rollout. Notably, it successfully distinguishes between different objects in the same scene (e.g., Dog vs. Cat), whereas previous methods often produce identical, blurry maps.

Quantitatively, our method outperformed state-of-the-art baselines across nearly all metrics:

  • Segmentation Accuracy: On ImageNet-Segmentation, we achieved a 61.95% mIoU, a substantial improvement over Partial LRP (57.94%) and Grad-CAM (40.82%).
  • Perturbation Tests: In positive perturbation tests, model accuracy dropped most rapidly when pixels identified by our method were removed, confirming we correctly identify the most influential features.
  • NLP Performance: Our method achieved the highest token-F1 scores in the Movie Reviews task, demonstrating that the relevancy propagation logic generalizes effectively to 1D text sequences.

Discussion and Conclusion

Our findings highlight that "attention is not relevancy." High attention scores do not always correlate with classification importance; tokens may simply act as hubs for information flow. By incorporating gradients and handling architectural nuances, our method provides a more faithful representation of model logic.

While our approach is more computationally intensive than simple attention visualization due to gradient calculations, it remains far more efficient than black-box perturbation methods. Future work will explore extending this framework to models without a global [CLS] token and investigating its utility in model pruning by identifying redundant layers or heads that contribute little to the final relevancy score.

Summary updated

Abstract

Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.
BibTeX
@inproceedings{chefer2021transformer,
  author = {Hila Chefer and Shir Gur and Lior Wolf},
  title = {{Transformer Interpretability Beyond Attention Visualization}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=KyW34is-zsXn},
}
Figure