The Core Intuition: Testing Compatibility by Intersecting Networks
Deep neural networks are notoriously chaotic in their parameter space. Two networks trained on the exact same dataset from different random initializations will arrive at completely different weight configurations. Yet, empirically, they achieve nearly identical performance. Do they learn completely different ways to solve the problem, or do they converge to a shared internal language?
To answer this, researchers use a technique called model stitching. Instead of comparing weights directly or relying on static similarity metrics like Centered Kernel Alignment (CKA), model stitching evaluates functional compatibility by creating a hybrid network. We take the early layers of a source network, connect them to the later layers of a target network, and insert a simple, low-capacity linear transformation layer (a "stitch") at the junction.
The intuition is elegant: if the hybrid network maintains the performance of the original models without retraining any of the original layers, the internal representations at that junction must be functionally equivalent. Because the stitching layer is strictly limited in capacity (typically a linear mapping or a $1 \times 1$ convolution), it cannot compute new features; it can only translate, rotate, or scale the existing latent space. High performance proves that the source features are directly usable by the target model's downstream decision logic.
The Mechanics of a Stitch
As formalized early on by Vedaldi and Lenc and later expanded by Csiszárik et al. and Bansal et al., the stitching process freezes the weights of both the sender and receiver sub-networks.
Let $f_A$ be the front half of Model A and $g_B$ be the back half of Model B. The stitched model output for an input $x$ is given by:
$$\hat{y} = g_B(W \cdot f_A(x) + b)$$
where $W$ and $b$ represent the parameters of a linear stitching layer. This layer is trained using the standard task loss while $f_A$ and $g_B$ remain completely frozen.
Balogh and Jelasity distinguish between task-optimized stitching (minimizing final output error) and direct representation matching, noting that optimizing for the final task loss provides a truer measure of functional interchangeability. If the resulting hybrid model avoids a significant performance drop, it establishes what Bansal et al. term "stitching connectivity"—a structural property showing that stochastic gradient descent (SGD) consistently guides networks toward a shared representational manifold. In fact, Gu et al. demonstrate through layer grafting that independently trained networks follow nearly identical trajectories in feature space, explaining why their internal representations are so interchangeable.
Convergence Across Architectures and Objectives
The implications of stitching extend far beyond networks with identical architectures. Modern research demonstrates that models converge toward a universal representation space even when their underlying structures or training paradigms differ significantly.
- Across Architectures: Sun et al. show that networks of varying architectures converge toward a similar linear representation space, allowing features to act as transportable training targets. Similarly, Jones et al. demonstrate that adversarial training acts as a "universalizer": robustly trained ResNets, VGGs, and MobileNets develop highly translatable internal features that can be stitched seamlessly into one another's classification heads. Even when comparing robust and non-robust models, Balogh and Jelasity find that while their intermediate features diverge, their final layers remain surprisingly interchangeable.
- Across Generative Paradigms: Bhalla explored whether completely different generative families (such as GANs, VAEs, and Diffusion Models) learn the same internal maps of reality. By connecting the encoder of one generative model to the decoder of another via a linear transformation, Bhalla demonstrated a striking degree of cross-model latent compatibility.
- In Language Models: Baek et al. used model stitching to show that diverse large language models (LLMs) like LLaMA and Gemma converge on nearly identical geometric structures to represent relational knowledge graphs, implying that certain concepts have an optimal geometric form that all capable models eventually discover.
Applications: Dynamic Scaling and Feature Editing
Stitching is not merely a diagnostic tool; it enables practical engineering shortcuts and novel intervention methods.
- Stitchable Neural Networks: Instead of training multiple models for different deployment environments (e.g., mobile vs. server), Pan et al. introduced SN-Net. By inserting simple $1 \times 1$ convolutions between different scales of pretrained anchor models (like DeiT-Tiny and DeiT-Base), they create a runtime spectrum of scalable models at a fraction of the traditional training cost.
- Cross-Model Activation Patching: To understand how fine-tuning alters a model, Prakash et al. substituted internal activations from a fine-tuned model directly into a base model. The base model's immediate performance boost proved that fine-tuning optimizes existing internal mechanisms rather than creating entirely new circuits. Similarly, Zhou used model stitching to validate the functional utility of interpolated features across different fine-tuned tasks, proving that cross-task features behave linearly with respect to weights.
- Knowledge and Truth Spaces: Zhang et al. leveraged representation alignment to isolate a "truthful space" across language models, allowing a single editing vector to generalize across an entire architectural family. Similarly, Shrivastava and Holtzman showed that linear probes trained on unaligned base models can successfully decode "refused knowledge" in instruction-tuned variants, proving that the underlying representation of knowledge remains structurally intact after safety alignment. For vision models, Kalibhat et al. showed that a linear transformation can map the semantic representation space of one model onto another, allowing for the transfer of feature explanations between different architectures.
Theoretical Bounds and Methodological Caveats
Why does model stitching work so reliably, and when should we be skeptical?
Providing a formal mathematical foundation, Insulla et al. proved that the performance drop experienced by a stitched hybrid model is theoretically bounded by the kernel alignment of their respective representations. This bridges empirical stitching success directly with statistical learning theory.
However, researchers must watch out for structural illusions. Hernandez et al. uncovered a "triangle pattern" where later layers of a sender network can almost always be successfully stitched into earlier layers of a receiver network. They caution that a successful stitch might occasionally "hack" the downstream task by passing raw information rather than reflecting true representational alignment. Furthermore, Brown and Godfrey note that while stitching can locate where a model decides to learn a task versus relying on cheap heuristics, structural asymmetries between activation functions can distort these measurements. This highlights the importance of using low-capacity adapters to ensure that the stitch is merely aligning spaces rather than performing heavy feature engineering.
Key Foundational Work
How to Characterize The Landscape of Overparameterized Convolutional Neural Networks (2020) — contributed 2 concept elements to this direction.
Understanding image representations by measuring their equivariance and equivalence (2015) — contributed 1 concept elements to this direction.