Standard deep learning architectures operate on a simple premise: to increase a model's representational capacity, you stack more layers, and each layer introduces a brand-new set of trainable parameters. While this approach has driven the success of large Transformer models, it introduces a severe bottleneck. The model's memory footprint and parameter count scale linearly ($O(L)$) with its depth.
Cross-layer weight sharing breaks this linear dependency. Instead of assigning unique weights to every single layer in a deep network, this paradigm reuses the same parameters across multiple steps of the architecture. By decoupling a network's computational depth from its parameter count, researchers can build models that possess the representational capacity of deep networks but retain the compact memory footprint of much shallower ones.
The Intuition: Iterative Refinement and Dynamical Systems
To understand why a model can reuse the exact same weights sequentially without its performance collapsing, it helps to shift how we view a neural network's layers. Instead of treating each layer as a distinct feature extractor operating at a different level of abstraction, we can view the sequence of layers as an iterative refinement process.
When a Transformer layer shares its weights with the next, passing a hidden state through the stack is conceptually identical to unrolling a recurrent neural network (RNN) over time. Rather than learning entirely new transformations, the shared layer repeatedly applies the same mathematical function to progressively refine the input representation.
This perspective bridges the gap between discrete deep networks and continuous dynamical systems. For example, Yu et al. leverage this connection in Learning to Encode Position for Transformer with Continuous Dynamical Model by using a shared neural network to define position dynamics across all layers. Because the underlying dynamics are invariant to depth, sharing the parameters ensures that the complexity of the position encoding does not scale with the model's depth. Similarly, Huang et al. note in Augmented Normalizing Flows: Bridging the Gap Between Generative Flows and Latent Variable Models that deep generative models utilizing discrete transformation steps can use weight-sharing or continuous-time approximations to mitigate the high parameter cost of unique layers.
Topologies of Weight Sharing
Weight sharing is not an all-or-nothing architectural choice. Researchers have mapped out a spectrum of sharing topologies, balancing the trade-off between parameter compression and specialized representational capacity.
Universal Sharing: [Layer A] -> [Layer A] -> [Layer A] -> [Layer A]
Block-wise Sharing: [Layer A] -> [Layer A] -> [Layer B] -> [Layer B]
Sandwich Sharing: [Layer A] -> [Layer B] -> [Layer B] -> [Layer C]
Universal Weight Sharing
The most aggressive strategy collapses the entire Transformer stack into a single set of parameters that are executed recursively. This approach is utilized in W2V2-Light: A Lightweight Version of Wav2vec 2.0 for Automatic Speech Recognition, where a single set of parameters is used for every layer within the Transformer module. This collapses the parameter count of the Transformer stack by a factor equal to the total number of layers, stripping away over 90% of the parameters while keeping performance remarkably competitive.
Block-wise and Group-wise Sharing
Instead of sharing a single set of weights globally, layers can be partitioned into distinct blocks or groups. Layers within the same group share weights, but weights differ between groups. Tan et al. apply this in ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioning to preserve deep reasoning capacity while minimizing memory overhead. To optimize these structures automatically, Zhao et al. introduce an entropy-driven framework in MeRino: Entropy-driven Design for Mobile-friendly Generative Language Models that dynamically tunes block depths and sharing configurations to fit strict hardware constraints.
Sandwich-Style Sharing
A key insight in sequence processing is that the boundary layers of a network—the initial and final layers—perform highly specialized operations. The initial layers process raw token or feature embeddings, while the final layers project hidden states back to output distributions. The intermediate layers, by contrast, perform highly repetitive contextual routing.
"Sandwich-style" sharing keeps the boundary layers unique while tying the weights of the intermediate blocks. Reid et al. formalize this in Subformer: Exploring Weight Sharing for Parameter Efficiency in Generative Transformers, demonstrating that focusing parameter reduction on the middle blocks preserves the specialized representational capacity needed for input processing and output generation. Similarly, Li et al. show in Improving Transformer-Based Speech Recognition Systems with Compressed Structure and Speech Attributes Augmentation that reusing parameters from the initial encoder and decoder blocks across subsequent blocks maintains depth while drastically reducing unique weights.
Extending Sharing Beyond Sequential Layers
The principle of parameter reuse can be extended beyond adjacent layers to encompass entire architectural components or parameter-efficient fine-tuning (PEFT) modules.
Cross-Component Sharing
In sequence-to-sequence models, the encoder and decoder are traditionally treated as separate entities. However, Dabre et al. demonstrate in Recurrent Stacking of Layers for Compact Neural Machine Translation Models that sharing weights between the encoder and decoder components enables extreme model compression with only marginal drops in accuracy. This suggests that the underlying feature extraction logic required to understand a source language is fundamentally symmetric to the logic required to generate a target language.
Global PEFT and Adapter Pools
When fine-tuning large models, maintaining unique adapters or prompts for every single layer introduces redundant parameters. Recent work addresses this by sharing a single parameter-efficient module across the entire network.
- Masked Prototypes: Zeng et al. propose One Network, Many Masks: Towards More Parameter-Efficient Transfer Learning, which shares a single global "prototype" adapter across all layers and tasks, using layer-specific binary masks to activate distinct sub-networks.
- Shared Prompt Generators: In IAPT: Instruction-Aware Prompt Tuning for Large Language Models, Zhu et al. share the parameters of prompt generators across multiple Transformer layers to maintain efficiency while allowing layer-specific adaptation through pooling modules.
- Cross-Layer Mixture-of-Experts: Li et al. introduce Sparser Mixture-of-Adapters with Cross-Layer Generalization, which replaces layer-specific experts with a unified pool of expert modules shared across all layers. A global router allows tokens to reuse specialized knowledge at different stages of the processing pipeline, eliminating cross-layer redundancy.
System-Level Advantages
The benefits of cross-layer weight sharing extend beyond reducing disk storage; they fundamentally alter the computational and communication dynamics of training and deployment.
Mitigating the Distributed Training Bottleneck
In large-scale decentralized training, the primary bottleneck is not computation, but communication—specifically, the overhead of synchronizing massive gradient vectors across independent nodes over the internet. As Borzunov et al. demonstrate in Training Transformers Together, cross-layer weight tying significantly lowers this communication overhead. Because layers share parameters, the total number of unique gradients that must be aggregated and synchronized across the network is drastically reduced, making collaborative, internet-scale training viable.
Enabling Edge and Low-Resource Deployment
By shrinking the memory footprint without sacrificing architectural depth, weight sharing makes deep Transformers viable for resource-constrained environments. This footprint reduction is critical across a wide array of localized applications:
- Localized NLP Benchmarks: Developing compact models via parameter sharing allows high-performance deployment for low-resource languages, as shown by Wilie et al. in the development of IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding.
- Specialized Domains: In fields with highly specialized vocabularies, such as historical document translation (Kang et al. in Restoring and Mining the Records of the Joseon Dynasty via Neural Language Modeling and Machine Translation), cross-layer sharing allows models to process complex linguistic structures efficiently without requiring parameter-heavy architectures that fail to fit on edge hardware.
Key Foundational Work
Improving Transformer-Based Speech Recognition Systems with Compressed Structure and Speech Attributes Augmentation (2019) — contributed 2 concept elements to this direction.
IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding (2020) — contributed 1 concept elements to this direction.
Tied Transformers: Neural Machine Translation with Shared Encoder and Decoder (2019) — contributed 1 concept elements to this direction.