Recurrent Weight Sharing for Parameter Efficient Depth

955 concepts524 papers9 proposals

Standard deep learning operates on a simple premise: to solve more complex problems, you add more layers. However, stacking unique layers creates a strict coupling between a model's computational depth (the number of sequential operations it performs) and its parameter count (the number of unique weights stored in memory). This coupling leads to parameter explosion, high memory footprints, and a tendency to overfit.

Recurrent weight sharing breaks this coupling. Instead of passing data through a linear chain of distinct layers ($x_{l+1} = f_{\theta_l}(x_l)$), a weight-shared architecture routes the data through the same physical layer or block multiple times ($x_{t+1} = f_\theta(x_t)$). This approach allows a model to achieve the computational depth of a massive network while maintaining the compact parameter footprint of a single layer.

Mimicking Iterative Optimization

In computer vision and signal processing, many tasks are traditionally framed as iterative optimization problems—such as minimizing an energy function or progressively refining a rough estimate. When we share weights across sequential processing steps, the network naturally learns to act as an iterative solver, applying the same underlying "update rule" to refine its state.

For example, in optical flow estimation, Junhwa Hur et al. demonstrated in Iterative Residual Refinement for Joint Optical Flow and Occlusion Estimation that applying a single backbone architecture repeatedly to its own outputs prevents the parameter explosion and overfitting common in deep-stacked networks. Instead of learning different features at every layer, the model learns a consistent logic for iterative spatial updates, effectively mimicking a classical optimization procedure.

This perspective is also shared by Filippos Kokkinos et al. in Deep Image Demosaicking using a Cascade of Convolutional Residual Denoising Networks, where a classical Majorization-Minimization algorithm is unrolled into a shared residual denoising network. By reusing the same weights across all stages, the model treats the network as a recursive solver rather than a deep feed-forward chain. Similarly, Zhiyang Yu et al. in Deep Bayesian Video Frame Interpolation reuse a shared backbone to execute a learned gradient descent process, focusing computational effort on high-error regions over time to achieve high-fidelity synthesis with a fraction of the weights.

Learning Scale-Invariant Algorithms

For algorithmic and compositional reasoning tasks, standard feed-forward networks struggle because they tend to memorize position-dependent patterns. Forcing layers to share weights acts as a powerful structural constraint: it compels the network to learn a single, reusable primitive operation.

In Making Transformers Solve Compositional Tasks, Santiago Ontanon et al. showed that forcing all layers within a transformer module to share weights significantly improves the model's ability to solve algorithmic tasks requiring repeated logical steps. This mechanism is explored from a compiler perspective by Peter Shaw 0004 et al. in ALTA: Compiler-Based Analysis of Transformers, where symbolic algorithms are compiled directly into weight-tied layers to execute length-invariant loops internally.

Crucially, because the underlying operation is consistent across steps, these models can generalize from easy to hard problems by simply "thinking for longer" at test time. As shown by Avi Schwarzschild et al. in Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks, a weight-shared recurrent module trained on simple problems can solve significantly harder instances at test time by increasing the number of evaluation iterations. Because the network executes a scalable procedure rather than a fixed-depth mapping, it breaks the structural boundaries that limit standard feed-forward networks.

Breaking the Symmetry of Pure Loops

If a network applies the exact same function $f_\theta$ at every step, it faces a fundamental limitation: it can struggle to track its own progress or break the symmetry of the loop, which restricts its expressive power. To overcome this, modern architectures introduce subtle, parameter-efficient mechanisms to modulate the shared layer based on the current iteration.

In On Expressive Power of Looped Transformers: Theoretical Analysis and Enhancement via Timestep Encoding, Kevin Xu et al. analyzed the approximation rates of looped transformers and proposed Timestep-Modulated Looped Transformers (TMLT). By injecting a unique timestep encoding at each iteration, they "break the symmetry" of the loops, vastly increasing the model's expressive power without adding unique layer weights.

Another approach is demonstrated by Mohammadmahdi Nouriborji et al. in MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers, where tiny, trainable bottleneck adapters are integrated into a recursive structure. These adapters provide iteration-specific memory, allowing a single shared layer to adapt its behavior dynamically across different processing steps and approximate the performance of much larger, non-recursive counterparts.

The Inference-Time Trade-off

One of the most compelling advantages of recurrent weight sharing is the flexibility it grants during inference. In a standard network, the computational cost is fixed at training time by the number of layers. In a weight-shared architecture, computation becomes a dynamic dial.

A model can be run for fewer iterations when resources are constrained or a quick answer is needed, or it can be allowed to iterate longer to achieve higher precision on difficult inputs. This decoupling of depth from parameters allows edge devices to trade execution time for accuracy dynamically, all while maintaining a constant, lightweight memory footprint on disk.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_recurrent_weight_sharing_11c687c3,
  title = {recurrent weight sharing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recurrent-weight-sharing/rdr_11c687c3211648bc8ae8204ceb559d47},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_recurrent_weight_sharing_11c687c3_v1,
  title = {recurrent weight sharing},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recurrent-weight-sharing/rdr_11c687c3211648bc8ae8204ceb559d47/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake