Introduction
To deploy deep learning models in high-stakes domains like healthcare or security, interpretability is paramount. Perturbation-based explainability methods typically generate saliency maps by masking input regions and observing changes in model output. However, traditional approaches (e.g., RISE or Occlusion) replace masked areas with "neutral" values like black or gray pixels. These artificial "outlier" features fall outside the original data distribution, potentially triggering unrelated neurons and misleading the explanation. Furthermore, for Vision Transformers (ViTs), these replacement values still require processing a full token grid, yielding no computational benefit.
Methodology: Transformer Input Sampling (TIS)
Transformer Input Sampling (TIS) addresses these issues by treating the Transformer as a set-processing engine rather than a fixed-grid processor. The method follows a four-step pipeline:
- Mask Generation via Clustering: Instead of random masks, TIS applies K-Means clustering to internal embeddings across all transformer layers. This identifies groups of patches that the model perceives as semantically related.
- Token Sampling: The core innovation of TIS is the complete removal of masked tokens from the sequence before they enter the first transformer layer.
- Scoring: The model is probed with these reduced sequences. Because self-attention naturally handles variable-length inputs, the Transformer processes these "shorter" images without artificial padding.
- Saliency Map Construction: A final heatmap is generated via a weighted sum of the masks, where weights correspond to the model's confidence scores for the target class, normalized by token sampling frequency.
Key Advantages
By removing tokens rather than replacing them, TIS ensures the model only processes "real" features from the original image, eliminating the outlier bias inherent in pixel-replacement methods. This approach also offers significant efficiency gains; reducing the number of tokens passed through the network lowers inference time during the perturbation phase. Additionally, TIS demonstrates high class specificity, accurately highlighting features relevant to a specific target label rather than just the most salient object in the scene.
Results and Discussion
TIS was evaluated on 5,000 images from the ImageNet validation set using ViT-Base and DeiT-Base architectures. It was benchmarked against state-of-the-art (SOTA) methods including ViT-CX, Chefer’s methods, and Integrated Gradients.
In quantitative evaluations, TIS achieved superior performance in the "Pointing Game" localization metric, being the only method to score over 0.8 on both ViT and DeiT models. It also demonstrated high faithfulness, leading in "Insertion" metrics (accuracy gain as important pixels are added) and ranking second in "Deletion" metrics. Furthermore, the method showed low "Max-Sensitivity," indicating that its explanations are robust to minor input perturbations.
Limitations and Future Work
While effective for standard ViTs, TIS is not directly applicable to hierarchical models like Swin Transformers, which rely on fixed spatial grids for window-based attention. Future research will focus on adapting token sampling to these architectures and exploring its potential in multimodal transformers (e.g., vision-language models). The authors also noted that bypassing the final Softmax layer during generation can enhance class specificity, a direction for further investigation.
Summary updated