Introduction
Vision Transformers (ViTs) have become a cornerstone of modern computer vision. However, their internal decision-making processes remain largely opaque. To deploy these models in high-stakes environments like medical diagnostics or autonomous driving, we must develop reliable "attribution methods"—techniques that generate heatmaps indicating which pixels most influenced a specific prediction.
Current attribution methods for ViTs face two significant hurdles. First, vanishing attributions occur in deep models when information is multiplied across layers, causing values to shrink toward zero. Second, noisy early layers often focus on low-level textures and edges that are irrelevant to the final classification, leading to cluttered heatmaps that are difficult for humans to interpret.
The SkipPLUS Framework
This paper introduces SkipPLUS, a strategy designed to enhance the faithfulness and clarity of ViT explanations. The framework consists of two core components:
- PLUS (Progressive Layer Unification through Summation): Unlike traditional methods like "Rollout" that aggregate information multiplicatively, PLUS utilizes summation. By summing attribution maps across layers, the method prevents the vanishing attribution problem and ensures that contributions from every layer are preserved.
- SkipPLUS: The authors observe that the first half of a ViT typically performs general feature extraction that adds noise to class-specific explanations. SkipPLUS addresses this by initiating the PLUS summation from the middle layer of the network to the end. By skipping the initial "pre-processing" noise, the resulting heatmaps become significantly more focused on the target object.
Additionally, the authors present FullGrad+, an optimized implementation of the FullGrad algorithm tailored for efficient integration with these summation-based techniques.
Experimental Evaluation
The researchers evaluated SkipPLUS across several architectures (including EVA Large and ViT Base) and diverse datasets:
- ImageNet-1k: For standard classification performance.
- ImageNet-S: Utilizing ground-truth segmentation masks to quantitatively measure how well heatmaps align with actual object boundaries.
- MURA: A medical imaging dataset of bone X-rays to test the method's utility in specialized domains.
Results and Discussion
SkipPLUS demonstrates a clear advantage over existing techniques. Quantitatively, the combination of DecompX + SkipPLUS established a new state-of-the-art (SOTA) for white-box token attribution. In "deletion tests"—where the most important pixels are removed to observe the drop in model accuracy—SkipPLUS methods showed the steepest decline, indicating they more accurately identify the features critical to the model's decision.
Qualitatively, the visual improvements are striking. As shown in the comparison above, standard methods and even previous SOTA techniques like GenAtt often highlight large portions of the background. In contrast, SkipPLUS produces tightly focused heatmaps that clearly isolate the object of interest, such as the newt in the example image. This suggests a fundamental functional split in ViTs: the first half handles general feature extraction, while the second half performs class-specific reasoning.
Conclusion and Future Work
SkipPLUS offers a simple yet powerful heuristic for explaining Vision Transformers: less is more. By skipping the noisy early layers and using additive aggregation, the method achieves superior faithfulness and interpretability. While the current 50% skip-rate is a robust heuristic, future work may explore dynamic layer selection based on specific architectures. Furthermore, the principles of SkipPLUS could potentially be extended to Large Language Models (LLMs) to provide better explanations for transformer-based natural language processing.
Summary updated