Functional Alignment of Neural Networks via Relative Depth

16 concepts12 papers1 proposal

Imagine trying to swap components between a compact sports car and a massive cargo truck. If you try to match parts by their absolute distance from the front bumper, the alignment will fail completely. Instead, you must look at their proportional position: the transmission sits in the middle of the drivetrain for both vehicles, regardless of their total length.

In deep learning, we face a similar challenge when comparing, combining, or transferring knowledge between different neural networks. A 12-layer model and an 80-layer model process information at different granularities, making absolute layer indices (like "Layer 8") meaningless for cross-model mapping.

However, a powerful insight has emerged across recent research: deep networks distribute feature extraction along a highly predictable, proportional timeline. By mapping layers to a normalized scale of relative depth, we can uncover striking functional similarities across models of completely different sizes, architectures, and even modalities.

The Proportional Timeline of Abstraction

Why do models align so well when their depths are normalized? The answer lies in how neural networks naturally structure their computation. To transform raw inputs into meaningful outputs, every network must transition from low-level features to highly abstract representations. The total number of layers determines the computational resolution of this journey, but the relative position dictates the level of abstraction.

David D. Baek et al. demonstrated this convergence in Generalization from Starvation: Hints of Universality in LLM Knowledge Graph Learning, showing that diverse language models develop a universal geometric language to encode semantic concepts. Crucially, these representations are most aligned at equivalent relative depths across different model families. This universal progression—from basic token processing to abstract reasoning—means that the structural evolution of representations scales proportionally with the total depth of the network.

This phenomenon spans multiple domains:

Mechanics of Relative Depth Normalization

To put this intuition into practice, researchers use normalized depth indexing. Instead of identifying a layer by its absolute index $l$, its position is mapped to a continuous value $d \in [0, 1]$ representing the percentage of the total network architecture traversed:

$$d = \frac{l}{L}$$

where $L$ is the total number of layers in the network.

Absolute Index (Model A, L=12):  0 --- 3 --- 6 --- 9 --- 12
Normalized Depth Scale:          0.0 - 0.25 - 0.5 - 0.75 - 1.0
Absolute Index (Model B, L=32):  0 --- 8 --- 16 -- 24 -- 32

This simple transformation allows researchers to track internal milestones across highly heterogeneous architectures. For instance, Fan Yang et al. applied this in Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers? to measure knowledge acquisition. They proved that larger architectures often process specific abstractions at the exact same relative depth as smaller ones, despite having vastly more parameters.

Similarly, Ian Su et al. leveraged relative layer depth in Evaluation Awareness Scales Predictably in Open-Weights Large Language Models to isolate where models process evaluation-specific context, observing consistent internal behaviors across models with wildly different layer counts.

However, relative depth also exposes where architectures diverge. In Categorize Early, Integrate Late: Divergent Processing Strategies in Automatic Speech Recognition, Nathan Roll et al. used normalized depth indexing to show that while Transformers and Conformers achieve similar speech recognition accuracy, they utilize opposite processing timelines. Conformers extract phonetic information early in their relative depth, whereas Transformers defer it to later proportional stages.

Applications in Reassembly, Stitching, and Acceleration

Understanding that relative depth preserves functional similarity unlocks powerful engineering capabilities, allowing us to treat neural networks as modular components that can be stitched together, cross-pollinated, or used to accelerate one another.

Model Stitching and Reassembly

If layers at the same relative depth perform analogous transformations, we can swap blocks between entirely different models. Adriano Hernandez et al. explored this in Model Stitching: Looking For Functional Similarity Between Representations, confirming that functional similarity is tightly bound to proportional position rather than absolute index.

Taking this further, Xingyi Yang et al. introduced Deep Model Reassembly, demonstrating that blocks from completely different architectural families (like CNNs and Vision Transformers) can be mixed and matched into a single customized model because they converge on similar feature transformations at comparable relative depths.

Collaborative Learning

In federated learning environments with heterogeneous clients, relative depth provides a common language. Minhyuk Seo et al. developed Co-LoRA: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients, which aggregates adapter parameters across models of different depths. By matching layers based on their relative position rather than absolute index, the framework ensures compatible knowledge transfer across diverse client models.

Inference Acceleration

Relative depth alignment can also eliminate latency. In KV Prediction for Improved Time to First Token, Chenfan Sun et al. used a small, fast auxiliary model to predict the Key-Value (KV) cache for a larger base model. This works because neighboring layers in transformer architectures exhibit similar internal states, and mapping layers based on relative depth ensures that the predicted cache aligns perfectly with the semantic expectations of the larger target model.

Cross-Modal Synergies

The principle even spans across entirely different data types. In Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities, Yiyuan Zhang et al. connected target and auxiliary transformer blocks based on their depth hierarchy. Because transformers trained on audio, text, or images develop a similar hierarchical structure of abstraction, sequential connections at corresponding relative depths maximize the transfer of general sequence-processing abilities.

The Boundaries of Functional Similarity

While relative depth alignment provides an elegant framework for cross-model mapping, it is bounded by fundamental architectural constraints. Jaehui Hwang et al. analyzed this in Similarity of Neural Architectures Using Adversarial Attack Transferability, revealing that neural networks naturally form distinct functional clusters based on their architectural families (such as non-hierarchical transformers versus residual networks).

While relative depth allows us to find the closest functional equivalents within or across these families, the underlying architecture still dictates the unique geometric properties of the representations. Relative depth alignment does not erase these structural differences; rather, it gives us the precise coordinate system needed to navigate them.

Key Foundational Work

How Deep is the Feature Analysis underlying Rapid Visual Categorization? (2016) — contributed 1 concept elements to this direction.
Deep Model Reassembly (2022) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relative_depth_layer_alignment_66447677,
  title = {relative depth layer alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-depth-layer-alignment/rdr_66447677f8d24aad8faef5266b5ede34},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_relative_depth_layer_alignment_66447677_v1,
  title = {relative depth layer alignment},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-depth-layer-alignment/rdr_66447677f8d24aad8faef5266b5ede34/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake