Overview
Standard methods for measuring how neural networks perceive image similarity are often "distracted" by where objects are located within the frame. This paper introduces Semantic Representational Similarity Matrices (RSMs), a new framework that uses set-matching to decouple what a network sees from where it sees it. This decoupling leads to a more accurate understanding of model behavior and significantly better performance in downstream tasks like image retrieval.
The Problem with Spatial Coupling
To understand the "black box" of deep learning, researchers frequently rely on Representational Similarity Matrices (RSMs). An RSM functions as a heat map, illustrating how similar a network's internal responses are for every pair of images in a given dataset.
However, traditional RSMs are "spatio-semantic": they inextricably link the content of an image with its spatial location. For example, if you have two images of the exact same cat, but the cat is positioned on the left in one image and on the right in the other, a standard RSM will often report these representations as highly dissimilar.
This spatial sensitivity is problematic because most neural networks—and human observers—are trained to be invariant to these kinds of translational shifts. If a network’s final classification remains unchanged when an object moves, the analytical tools used to probe the network's internal state shouldn't be fooled by that movement either. Spatial alignment acts as "noise" that obscures the true semantic similarity a model has learned.
A Set-Matching Approach to Similarity
To resolve this, the authors propose shifting from rigid spatial alignment to a flexible set-matching approach.
Instead of comparing features at strict $(x, y)$ coordinates, the internal representation of an image (typically a 3D tensor of channels, width, and height) is treated as an unordered set of "semantic concept vectors"—one for each spatial location. The goal is to find the optimal permutation (rearrangement) of these vectors that maximizes the similarity between two images, effectively making the comparison invariant to spatial shifts.
Formulated as a bipartite matching problem, finding the perfect match using the Hungarian Algorithm is computationally expensive, scaling at $O(S^3)$ where $S$ is the number of spatial locations. To make this practical for large, high-resolution models, the authors introduce a Batch-Optimal approximation. By sorting vectors by their L2-norm and matching them in smaller batches, this method drastically reduces computation time while maintaining over 90% of the optimal similarity score.
Key Insights and Results
The researchers evaluated Semantic RSMs across diverse benchmarks, including Tiny-ImageNet, EgoObjects, Cityscapes, and ImageNet-1k, testing standard similarity measures like Linear, RBF, and Cosine similarity.
The results demonstrate that decoupling spatial alignment provides a much clearer view of a model's conceptual space:
- Massive Gains in Image Retrieval: On the EgoObjects dataset, Semantic RSMs significantly outperformed standard versions. For the CLIPSeg model using Cosine Similarity, the F1 score for image retrieval jumped from a mere 0.280 to an impressive 0.835.
- Better Alignment with Model Logic: Semantic RSMs correlate more strongly with the model's final output behavior. They showed a consistently higher Pearson correlation with Jensen-Shannon Divergence, meaning they are better at predicting which images a classifier will ultimately group into the same category.
- High-Resolution Benefits: The improvement was most dramatic for models with high spatial resolution or those designed for dense tasks, such as the Segment Anything Model (SAM) and CLIPSeg. These models retain granular spatial details that easily confuse standard RSMs.
- Computational Efficiency: The proposed "Batch-Optimal" approximation achieved a 36x speedup over the optimal matching algorithm for images with a $64 \times 64$ spatial extent, keeping the error rate under 8%.
Limitations and Future Directions
While Semantic RSMs offer a massive leap forward in interpretability, they are still more computationally demanding than standard RSMs, particularly for early CNN layers where spatial resolution is highest. Furthermore, while the correlation between internal representations and final outputs improved, it remains relatively low overall. This suggests that while spatial decoupling is a crucial step, the choice of the underlying "kernel" (how we measure the similarity of individual vectors) remains an open challenge for future research.
Summary updated