Imagine two cartographers mapping the same city. One uses GPS coordinates; the other uses a custom grid based on local landmarks. Their final maps look completely different on paper, but they describe the exact same physical reality. If you want to combine their insights, you do not need to redraw the maps from scratch. Instead, you just need a mathematical formula to translate coordinates from one system to the other.
This is the core intuition behind cross-architecture feature alignment. When different neural networks—whether they vary in size, layer depth, or fundamental design (like CNNs versus Transformers)—are trained on similar data, they develop distinct internal coordinate systems, known as latent spaces. Feature alignment provides the mathematical bridge to translate between these spaces, allowing models to communicate, share knowledge, or even swap components without exhaustive retraining.
The Geometry of Representation Universality
Why is it possible to align entirely different models with lightweight transformations? The answer lies in representation universality. Deep networks trained on the same underlying distribution tend to learn isomorphic semantic abstractions. They discover the same features (e.g., edges, textures, or semantic concepts), but they arrange them differently in their respective hidden layers.
Because the underlying geometry of these latent spaces is highly correlated, simple mathematical operations can reconcile their coordinate discrepancies. For example, Yung-Sung Chuang et al. demonstrate in Lookback Lens that standard linear regression is sufficient to map internal attention head weights from a smaller 7B language model to a larger 13B model. This alignment allows diagnostic tools trained on smaller, cheaper models to be transferred directly to larger architectures without retraining.
When the relationship between latent spaces is more complex, non-linear mappings can bridge the gap. In Activation Space Interventions Can Be Transferred Between Large Language Models, Narmeen Fatimah Oozeer et al. use lightweight, non-linear autoencoders to translate activation spaces between entirely different large language models (LLMs). This translation enables safety interventions—such as forcing a model to refuse harmful prompts—to be discovered on a small model and seamlessly deployed inside a massive foundation model.
Bridging the Dimensionality Mismatch
The most common hurdle in cross-architecture alignment is a mismatch in dimensionality. If a student model represents a concept using a 256-dimensional vector, and a teacher model uses a 1024-dimensional vector, they cannot be compared directly.
To resolve this, researchers introduce lightweight projection layers that act as dimensional translators:
- Linear Projection Matrices: A simple matrix multiplication can upscale or downscale feature dimensions. In Dual-Space Knowledge Distillation for Large Language Models, Songming Zhang 0001 et al. employ bidirectional linear layers to project hidden states between teachers and students with entirely different vocabularies and architectures, allowing them to share prediction heads.
- 1x1 Convolutions: For computer vision models, 1x1 convolutions serve as channel-wise alignment tools. Zizheng Pan introduces this in Stitchable Neural Networks, using 1x1 convolutions to connect layers from a small "anchor" model to a larger one at runtime. Similarly, Yang Shu et al. use 1x1 convolutions in Zoo-Tuning to synchronize and aggregate internal representations from a diverse bank of pre-trained models before fusing their parameters.
[Source Feature: Shape (B, C_in, H, W)]
│
▼
[1x1 Convolution Layer] <─── Learns coordinate translation
│
▼
[Target Feature: Shape (B, C_out, H, W)]
By keeping these projection layers lightweight, the system learns to rotate, scale, and shift the source coordinate system until it aligns with the target coordinate system, without altering the core features learned by either model.
Reconciling Structural Prior Gaps
Alignment becomes more challenging when models possess completely different structural priors. Consider a Convolutional Neural Network (CNN) and a Vision Transformer (ViT). A CNN processes images using two-dimensional spatial grids, focusing heavily on local textures. A ViT flattens images into one-dimensional sequences of tokens, prioritizing global context via self-attention.
To align these disparate structures, alignment modules must perform both geometric reshaping and dimensional mapping:
- Spatial Reshaping: The 1D token sequences of a transformer are reshaped back into a 2D spatial grid to match the coordinate layout of the CNN.
- Dimension Matching: Depthwise and pointwise convolutions are applied to match channel widths and smooth out structural discrepancies.
This strategy is highly effective for cross-architecture knowledge distillation. In DearKD, Xianing Chen et al. use this exact combination of reshaping and depthwise convolutions to force the early layers of a ViT to mimic the intermediate representations of a CNN. This teaches the transformer the "spatial common sense" (inductive bias) of the convolutional network, drastically reducing the amount of data required to train the transformer.
A similar approach is taken by Kehan Li et al. in Locality Guidance for Improving Vision Transformers on Tiny Datasets, where 1x1 convolutions and spatial reformatting allow a compact CNN to guide a transformer's attention toward local features on small datasets.
Applications: From Franken-Models to Edge AI
Once we accept that latent spaces can be aligned, neural networks stop being rigid, monolithic structures and instead become modular components.
Deep Model Reassembly
Instead of training a customized model from scratch for a specific hardware constraint, we can treat existing models like a box of Lego bricks. In Deep Model Reassembly, Xingyi Yang et al. demonstrate that you can take a stage from a ResNet, a block from a ViT, and a piece of an MLP-Mixer, and stitch them together. By inserting lightweight adapter layers between these reassembled blocks, information flows smoothly across boundaries that were originally trained in completely different architectural contexts.
Model-Agnostic Edge Collaboration
In distributed environments like Edge AI, devices have vastly different hardware capabilities. A tiny edge sensor cannot run the same massive model as a cloud server. Songjie Xie et al. address this in Toward Real-Time Edge AI: Model-Agnostic Task-Oriented Communication with Visual Feature Alignment. By using shared "anchor data" to estimate a linear transformation matrix, they align the incoherent feature spaces of heterogeneous edge and server models. This allows the edge device to transmit compact, aligned feature maps instead of raw data, enabling real-time collaborative inference without requiring identical network architectures.