Abstract: A multitude of (dis)similarity measures between neural network representations have been proposed, resulting in a fragmented research landscape. Most of these measures fall into one of two categories. First, measures such as linear regression, canonical correlations analysis (CCA), and shape distances, all learn explicit mappings between neural units to quantify similarity while accounting for expected invariances. Second, measures such as representational similarity analysis (RSA), centered kernel alignment (CKA), and normalized Bures similarity (NBS) all quantify similarity in summary statistics, such as stimulus-by-stimulus kernel matrices, which are already invariant to expected symmetries. Here, we take steps towards unifying these two broad categories of methods by observing that the cosine of the Riemannian shape distance (from category 1) is equal to NBS (from category 2). We explore how this connection leads to new interpretations of shape distances and NBS, and draw contrasts of these measures with CKA, a popular similarity measure in the deep learning literature.
Introduction: Two Paradigms for Comparing Representations
As deep learning and computational neuroscience have matured, researchers have developed dozens of ways to answer the question: "How similar is the internal 'logic' of Network A to Network B?" Currently, these methods are divided into two distinct paradigms:
- Alignment-based methods: These approaches attempt to find an explicit transformation (such as a rotation or scaling) that maps the neurons of one network onto the neurons of another. By finding the best possible alignment, researchers can quantify the residual distance between the two representations. Examples include Procrustes distance and Canonical Correlation Analysis (CCA).
- Summary-statistic methods: These methods bypass individual neurons and focus on how a network treats a set of stimuli. They generate a "kernel matrix" or Representational Similarity Matrix (RSM) that captures the similarity between every pair of inputs. Comparing these matrices directly provides a measure of representational similarity. Examples include Representational Similarity Analysis (RSA), Centered Kernel Alignment (CKA), and Bures similarity.
Historically, these two camps have been treated as separate worlds. It has not been clear if they measure the same underlying properties or how they behave as models scale toward infinite width or massive datasets.
The Duality Principle: Theorem 1
The core contribution of this work is a mathematical proof establishing a "duality principle" between these two paradigms. Through Theorem 1, the authors demonstrate that the Procrustes size-and-shape distance (an alignment method) is mathematically identical to the Bures distance (a summary-statistic method) when applied to linear kernel matrices.
Furthermore, the paper proves that the cosine of the Riemannian shape distance is identical to the Normalized Bures Similarity (NBS). This unification allows researchers to translate insights between geometric alignment and information-theoretic summary statistics. For instance, the authors use this duality to generalize shape distances to networks with different numbers of neurons ($N_x \neq N_y$) by reformulating the distance using the nuclear norm of cross-covariance matrices.
Geometric Insights and CKA
The authors contrast Bures and Shape distances with Centered Kernel Alignment (CKA), a dominant metric in modern deep learning. A key insight is that CKA assumes a "flat" Euclidean geometry for kernel matrices. In contrast, Bures and Shape distances respect the "curved" geometry inherent to the space of positive semidefinite (PSD) matrices.
By deriving analytical upper and lower bounds relating CKA to NBS, the authors show that while the two are correlated, they are not interchangeable. The choice of metric depends heavily on the geometry one assumes for the representation space. In practice, CKA can vary significantly from NBS depending on the rank of the matrices involved, suggesting that researchers must be deliberate in their choice of similarity measure.
Theoretical Implications: Quantum Mechanics and Scaling
This duality opens new avenues for theoretical analysis:
- Uhlmann’s Connection: The findings link neural representation comparison to Uhlmann’s theorem in quantum mechanics, which describes the maximum overlap between two "purified" quantum states. This suggests that Bures similarity represents the "best-case scenario" alignment between two representations.
- Scaling Laws: The duality provides a theoretical foundation for comparing "infinitely wide" models. By applying the Law of Large Numbers, the authors show how these distances converge as the number of stimuli ($M$) or neurons ($N$) approaches infinity.
- Optimal Transport: The paper hints at a deeper connection to the 2-Wasserstein distance (the "Earth Mover's Distance"), suggesting that comparing networks can be viewed as a problem of "transporting" information from one architecture to another.
Conclusion and Future Directions
While this work provides a rigorous bridge between alignment and summary-statistic methods, it focuses primarily on linear kernel matrices. Extending this duality to non-linear kernels (such as RBF) remains an open challenge. Additionally, while the paper establishes mathematical equivalence, further empirical work is needed to determine which metric is most predictive of downstream utility, such as model generalization or matching artificial networks to biological neural recordings. By unifying the fragmented landscape of similarity measures, this research provides a clearer path forward for understanding the internal representations of complex neural systems.
Summary updated