The Latency Cost of Weight Sharing in Recursive Networks

26 concepts23 papers1 proposal

In deep learning, we often equate a model's parameter count with its computational weight. We assume that a "lightweight" model with fewer parameters will naturally run faster, consume less power, and deploy more easily on edge devices.

Recursive architectures expose the flaw in this assumption. By reusing the same set of weights across multiple sequential iterations or time steps, a network can drastically reduce its disk and memory footprint. However, this compactness introduces a hidden tax: inference latency. While weight sharing allows a model to remain small in storage, executing a layer $N$ times sequentially requires the same—or greater—number of floating-point operations (FLOPs) as a deep, non-recursive network.

Understanding this parameter-latency trade-off is essential for designing efficient models across computer vision, natural language processing, and signal reconstruction.


The Illusion of Compactness

To build intuition, consider two ways to construct a deep network capable of complex contextual refinement:

  1. A Feed-Forward Approach: Stack 10 distinct convolutional or transformer layers. Each layer has unique weights ($\theta_1, \theta_2, \dots, \theta_{10}$). Data passes through them in a single forward pass.
  2. A Recursive Approach: Define a single layer with weights $\theta$. Pass the input through this same layer 10 times, updating an internal state at each step.
Feed-Forward:  Input ──> [ Layer 1 ] ──> [ Layer 2 ] ──> ... ──> [ Layer 10 ] ──> Output
                         (Unique θ1)     (Unique θ2)             (Unique θ10)

Recursive:     Input ──> [         Shared Layer θ         ] ───> Output
                            ▲                          │
                            └──── Iterates 10 times ───┘

From a storage perspective, the recursive model is roughly 10 times smaller. As shown by Ming Liang et al. in their work on Recurrent convolutional neural network for object recognition, integrating recurrent connections allows neurons to integrate contextual information dynamically, matching the performance of deep feed-forward networks with a fraction of the parameters.

However, the computational cost tells a different story. Because the data must still cycle through the shared weights 10 times, the total number of operations remains high. As Shaohui Lin et al. point out in Towards Compact Single Image Super-Resolution via Contrastive Self-distillation, recursive architectures repeatedly execute the same layers, meaning they fail to improve actual wall-clock inference speed despite their small memory footprint.


Why Parallelization Fails: The Sequential Bottleneck

Modern hardware accelerators (GPUs and TPUs) derive their speed from massive parallelization. In a wide feed-forward layer, thousands of activations can be computed simultaneously.

Recursive architectures introduce a strict mathematical dependency that breaks this parallelization:

$$h_t = f(h_{t-1}; \theta)$$

The state at iteration $t$ cannot be computed until the state at iteration $t-1$ is fully resolved. This sequential unrolling creates a bottleneck.

This issue is particularly pronounced in recursive Transformers. For instance, Mohammadmahdi Nouriborji et al. demonstrated in MiniALBERT: Model Distillation via Parameter-Efficient Recursive Transformers that while recursive parameter sharing creates ultra-compact models suitable for storage-limited devices, the data must still pass through the shared layers a fixed number of times. The computational cost scales with the number of iterations, not the number of unique parameters.

Similarly, when scaling up language and multimodal systems, Ibrahim Alabdulmohsin et al. introduced Recursive Inference Scaling (Recursive Inference Scaling: A Winning Path to Scalable Inference in Language and Multimodal Systems). They explicitly frame this as a direct trade-off: reusing weights saves memory but scales up inference-time FLOPs and latency, acting as a mechanism to trade time for performance without growing the model's physical size.


Domain Manifestations

The tension between parameter efficiency and sequential latency appears across a wide variety of application domains.

Low-Level Vision and Image Restoration

In tasks like single-image super-resolution, deblurring, and deraining, models use recursive loops to progressively clean or upscale an image.

Dense Prediction and Multi-Scale Fusion

When models process multi-scale representations recursively—such as image pyramids or multi-granularity maps—the sequential overhead compounds.


Mitigating the Bottleneck

To make recursive models practical, recent research focuses on architectural modifications that balance or decouple the parameter-latency equation.

Memory Splitting

Instead of forcing a single recurrent state to handle all operations, some architectures separate state tracking from feature processing. Jun-Ho Choi et al. proposed this in Volatile-Nonvolatile Memory Network for Progressive Image Super-Resolution, splitting the network's memory into "volatile" components for temporary state tracking and "nonvolatile" components for structural image features. This helps manage the sequential dependencies that slow down traditional recursive pipelines.

Hybrid Bimodal Designs

Other approaches combine the parameter savings of recursion with the speed of feed-forward layers. For example, Guangwei Gao et al. developed Lightweight Bimodal Network for Single-Image Super-Resolution via Symmetric CNN and Recursive Transformer, which pairs a fast, parallelizable symmetric CNN for local features with a recursive Transformer for global context, isolating the high-latency recursive steps to where they provide the most value.

Ultimately, recursive weight sharing is not a free lunch for model compression. It shifts the burden from the storage drive to the processor, offering a highly tunable knob: engineers can minimize a model's physical size, but they must be prepared to pay for it in wall-clock time.

Key Foundational Work

Multi-Path Feedback Recurrent Neural Networks for Scene Parsing (2017) — contributed 1 concept elements to this direction.
Recurrent convolutional neural network for object recognition (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_recursive_inference_latency_e54af170,
  title = {recursive inference latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recursive-inference-latency/rdr_e54af170f98a488c8e5d636a1afee0cb},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_recursive_inference_latency_e54af170_v1,
  title = {recursive inference latency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/recursive-inference-latency/rdr_e54af170f98a488c8e5d636a1afee0cb/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake