Deep neural networks are notoriously difficult to inspect. If you train two identical architectures on the exact same dataset, their final weight matrices will look completely different due to random initialization and stochastic optimization. If you feed the same image into both networks, the raw neuron activations will reside in entirely different coordinate systems.
This creates a fundamental challenge: how do we determine if two neural networks have learned the same underlying concepts? How do we compare a convolutional network to a vision transformer, or track how a language model's internal logic changes during fine-tuning?
Directly comparing raw activations fails because neural representations are invariant to arbitrary rotations, reflections, and scaling. To solve this, researchers shift their perspective from absolute coordinates to relational geometry. Instead of asking where an input sits in a latent space, we ask how that input relates to every other input in the dataset. This insight forms the basis of Representational Similarity Analysis (RSA) and its most mathematically robust implementation: Centered Kernel Alignment (CKA).
The Relational Perspective
To understand why CKA works, imagine showing a neural network a dataset of $N$ images. If we extract the activations from a specific layer, we get a matrix $X$ of size $N \times D_1$, where $D_1$ is the number of neurons in that layer. A second network might yield an activation matrix $Y$ of size $N \times D_2$. Because $D_1$ and $D_2$ can be completely different, we cannot compare $X$ and $Y$ directly.
Instead, we compute an $N \times N$ similarity matrix (or Gram matrix) for each network:
$$K = XX^T \quad \text{and} \quad L = YY^T$$
Each entry $(i, j)$ in matrix $K$ represents the inner product between the representations of example $i$ and example $j$ in the first network. If example $i$ and example $j$ are close together in the first network's mind, $K_{i,j}$ will be large.
By transforming our data from an $N \times D$ coordinate space to an $N \times N$ relational space, we have completely eliminated the dimension $D$. We can now directly compare the relational geometries of the two networks by measuring how similar $K$ and $L$ are.
The Mechanics of Centered Kernel Alignment
Simply taking the correlation between $K$ and $L$ is not enough. If one network merely shifts or scales its activations, the raw inner products will change even though the underlying topology remains identical. To make the metric robust, CKA introduces two critical steps: centering and normalization.
First, the similarity matrices are centered so that the column and row means are zero. This process ensures that global offsets do not artificially inflate the similarity score. The similarity between these centered matrices is quantified using the Hilbert-Schmidt Independence Criterion (HSIC):
$$\text{HSIC}(K, L) = \frac{1}{(N-1)^2} \text{tr}(K H L H)$$
where $H$ is the centering matrix. Intuitively, HSIC acts as a dot product between the flattened, centered similarity matrices.
Second, to make the metric invariant to isotropic scaling (multiplying all activations by a constant), CKA normalizes the HSIC score:
$$\text{CKA}(K, L) = \frac{\text{HSIC}(K, L)}{\sqrt{\text{HSIC}(K, K) \cdot \text{HSIC}(L, L)}}$$
This yields a score between 0 and 1. A CKA score of 1 means the two layers judge the relationships between all data points in the exact same way, even if their internal dimensions and orientations differ completely. As shown by Francesco Insulla et al. in Towards a learning theory of representation alignment, these empirical alignment metrics can be unified through spectral decomposition, proving that the performance drop when "stitching" two different models together is theoretically bounded by their kernel alignment.
Why CKA Succeeds Where Other Metrics Fail
Prior to CKA, researchers often relied on Canonical Correlation Analysis (CCA) to compare representations. While CCA is invariant to any linear transformation, it suffers from a major flaw in deep learning contexts: it can be overly sensitive to low-variance dimensions. If two networks share a tiny, noisy dimension but differ completely on their primary components of variance, CCA might still report a high similarity.
CKA strikes a balance. It is invariant to orthogonal transformations (rotations and reflections) and scaling, but it is not invariant to arbitrary linear transformations. This constraint ensures that dimensions explaining the most variance dominate the metric, making it highly robust to the noise inherent in deep networks. For instance, Aniruddh Raghu et al. utilized both CCA and CKA in Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML to demonstrate that meta-learning algorithms succeed primarily through feature reuse rather than radical representational shifts during adaptation.
Mapping the Internal Dynamics of Models
Because CKA can compare layers of different widths and architectures, it has become a standard tool for peering inside "black box" models.
Transfer Learning and Feature Reuse
A classic question in deep learning is whether fine-tuning a model actually builds new features or simply rearranges existing ones. In What is being transferred in transfer learning?, Behnam Neyshabur et al. used CKA to show that pre-training anchors models in specific optimization basins where their internal logic remains remarkably similar. Similarly, Christos Matsoukas et al. demonstrated in What Makes Transfer Learning Work For Medical Images: Feature Reuse & Other Factors that Vision Transformers are highly dependent on reusing pre-trained early layers when adapting to data-scarce domains like medical imaging.
Layer-by-Layer Evolution
Researchers use CKA to plot a model's internal pipeline by comparing its layers against themselves or other models. In Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models, Zheng Zhao et al. mapped out a clear three-stage functional pipeline in instruction-tuned LLMs, showing where task-specific logic diverges from general features.
For time-series foundation models, Atharva Pandey employed CKA in On the Internal Semantics of Time-Series Foundation Models to identify where temporal information processing plateaus within transformer blocks. Aylin Caliskan used linear CKA in Low Frequency Names Exhibit Bias and Overfitting in Contextualizing Language Models to track the exact depth at which token representations become contextualized, revealing that low-frequency names fail to integrate context properly and retain higher social bias.
Cross-Lingual and Cross-Modal Alignment
CKA can reveal how models bridge different languages or modalities. In First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERT, Benjamin Müller et al. used CKA on parallel sentences to pinpoint the exact depth at which multilingual BERT maps disparate languages into a single, unified conceptual space. In the acoustic domain, Badr Abdullah et al. showed in How Familiar Does That Sound? Cross-Lingual Representational Similarity Analysis of Acoustic Word Embeddings that the CKA alignment between speech models directly mirrors human linguistic and typological distances.
Moving across modalities, Mayug Maniparambil demonstrated in Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment that CKA can serve as a zero-shot predictive indicator for how easily frozen vision and text encoders can be aligned using minimal projection layers.
Engineering Applications
Beyond pure interpretability, representational alignment serves as a powerful diagnostic tool for optimizing, compressing, and protecting models.
Efficiency and Compression
By identifying layers that produce nearly identical relational geometries, developers can safely compress networks. In Exploring Representations and Interventions in Time Series Foundation Models, Michal Wilinski et al. used CKA to uncover blocks of redundant layers, providing a clear signal for structural pruning. Similarly, Haoyue Deng et al. applied CKA in Uncovering the Redundancy in Graph Self-supervised Learning Models to verify that more than half of a graph model's parameters could be removed without altering its core internal representations.
CKA can also drive dynamic training schedules. In SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezing, Sheng Li et al. used CKA to detect when intermediate layers reach a representational plateau, automatically freezing them to save up to 80% of training memory.
Model Merging and Sharing
For resource-constrained edge computing, Abhinav Sharma et al. demonstrated in Representation Similarity: A Better Guidance of DNN Layer Sharing for Edge Computing without Training that CKA can act as a training-free proxy to predict the accuracy of merged models, allowing different architectures to share layers seamlessly without requiring ground-truth validation data.
Intellectual Property Protection
Because CKA is invariant to dimension permutations and scaling, it can act as a robust "fingerprint" for model ownership. In REEF: Representation Encoding Fingerprints for Large Language Models, Jie Zhang et al. leveraged CKA to identify derivative models. Even if a suspect model has undergone aggressive pruning, weight shuffling, or fine-tuning, its underlying relational geometry remains detectable, exposing intellectual property theft.
Summary
Centered Kernel Alignment fundamentally changes how we evaluate deep learning systems. By discarding absolute coordinates and focusing entirely on the relative geometry of data points, CKA provides a universal language for comparing neural networks. Whether diagnosing training instability, verifying model fairness across demographic subgroups, or tracking how a network corrects errors layer-by-layer, CKA offers a mathematically rigorous window into the otherwise opaque world of high-dimensional representations.