Visualizing Paired Image Similarity in Transformer Networks

Samuel Black, Abby Stylianou, Robert Pless, Richard SouvenirWACV 2022 · OpenReview

Abstract

Transformer architectures have shown promise for a wide range of computer vision tasks, including image embedding. As was the case with convolutional neural networks and other models, explainability of the predictions is a key concern, but visualization approaches tend to be architecture-specific. In this paper, we introduce a new method for producing interpretable visualizations that, given a pair of images encoded with a Transformer, show which regions contributed to their similarity. Additionally, for the task of image retrieval, we compare the performance of Transformer and ResNet models of similar capacity and show that while they have similar performance in aggregate, the retrieved results and the visual explanations for those results are quite different. Code is available at https://github.com/vidarlab/xformer-paired-viz.

Introduction

Deep learning models are often criticized as "black boxes," particularly in complex tasks like image retrieval. While tools like Grad-CAM provide insights into classification decisions by highlighting salient pixels, they are typically optimized for Convolutional Neural Networks (CNNs) and single-image analysis. As the field transitions toward Vision Transformers (ViTs), there is a growing need for visualization techniques that account for global attention mechanisms and explain the relationship between pairs of images.

Methodology: Similarity Decomposition and Attention Rollout

The proposed method generates interpretable heatmaps for pairs of images by bridging the gap between high-level embeddings and input pixels.

  1. Similarity Decomposition: The process begins with the cosine similarity of two image embeddings. This scalar score is mathematically decomposed into the contributions of individual output tokens (patches) from the final Transformer layer.
  2. Attention Flow Approximation (Rollout): To map these contributions back to the original input pixels, the authors employ the "Rollout" technique. This approximates the flow of information across all Transformer layers by iteratively multiplying attention weights, accounting for the global "token mixing" characteristic of the architecture.
Query image from the Hotels-50k dataset, featuring a room with a balcony and specific furniture layout.

Results: Global Structure vs. Local Texture

The study evaluated models on benchmarks including Hotels-50k, Google Landmarks V2, and Stanford Online Products (SOP). While ViT-B/16 and ResNet-101 showed nearly identical performance metrics (e.g., Recall@1 of ~0.82 on SOP), their qualitative behavior diverged significantly.

Global Structure vs. Local Texture

A key finding is that CNNs tend to be "texture-driven," often matching images based on localized patterns like carpet weaves. In contrast, Transformers act as "architects," focusing on large, contiguous structures and the overall layout of a scene.

Similarity heatmap (C|A) showing regions of the query image that contribute to its similarity with another image (A), focusing on global structure.

The Pose Factor

Transformers exhibit a higher sensitivity to camera pose and object layout. Quantitative analysis using semantic segmentation confirmed that top matches for Transformers have significantly higher correlation in object-layout and camera-pose to the query image compared to ResNet matches.

Conclusion

This visualization method provides a window into the decision-making process of Vision Transformers in retrieval tasks. By revealing a preference for global structure and pose over local texture, the research highlights fundamental differences in how these architectures perceive image similarity. Future work may extend this approach to other pairwise tasks, such as change detection or medical image comparison.

Summary updated

Abstract

Transformer architectures have shown promise for a wide range of computer vision tasks, including image embedding. As was the case with convolutional neural networks and other models, explainability of the predictions is a key concern, but visualization approaches tend to be architecture-specific. In this paper, we introduce a new method for producing interpretable visualizations that, given a pair of images encoded with a Transformer, show which regions contributed to their similarity. Additionally, for the task of image retrieval, we compare the performance of Transformer and ResNet models of similar capacity and show that while they have similar performance in aggregate, the retrieved results and the visual explanations for those results are quite different. Code is available at https://github.com/vidarlab/xformer-paired-viz.
BibTeX
@inproceedings{black2022visualizing,
  author = {Samuel Black and Abby Stylianou and Robert Pless and Richard Souvenir},
  title = {{Visualizing Paired Image Similarity in Transformer Networks}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=ZRjxj0Odymw},
}
Figure