Temporal Rank in Recurrent Neural Networks

Leo Dobson, Anton RagniarXiv 2026 · arXiv

TL;DR

This paper identifies "temporal rank collapse"—a hidden state's inability to distinguish between independent input directions over time—as a fundamental bottleneck in RNNs that exists independently of the well-known vanishing gradient problem. By introducing a structured "Shift RNN" that explicitly preserves these input directions, the authors demonstrate that maintaining high-dimensional temporal capacity is essential for solving long-range dependency tasks.

What Problem Does This Paper Address?

For decades, the failure of Recurrent Neural Networks (RNNs) on long sequences has been blamed almost exclusively on vanishing and exploding gradients. While architectures like LSTMs and techniques like gradient clipping help stabilize training, RNNs have still been largely supplanted by Transformers, which process sequences in parallel.

The authors of this paper argue that gradient stability is only half the story. They identify a structural limitation they call a temporal rank bottleneck. In a standard RNN, the hidden state dynamics often cause information from different time steps to "align" or collapse into a low-dimensional subspace. When this happens, the network loses the ability to distinguish between inputs that occurred at different times, effectively "forgetting" details even if the gradient signal remains strong. This collapse is reflected in the stable rank of the input-to-state Jacobian—a measure of how many independent input directions the hidden state can actually track.

Methods

The researchers propose a novel, yet strictly standard, RNN formulation called the Shift RNN.

  1. Stable Rank Analysis: They use "stable rank" (a continuous proxy for matrix rank) to quantify the effective dimensionality of the mapping from input sequences to the hidden state.
  2. The Shift Construction: To prevent rank collapse, they structure the recurrent weight matrix $U$ as a non-parametric shift operator.
  3. The hidden state is expanded into a higher-dimensional space.
  4. At each time step, the existing state is "shifted" down, and the new input is injected only into the "top" block.
  5. This prevents different time steps from mixing prematurely and ensures that information from time $t$ stays separable from information at time $t-1$.
  6. Standard Formulation: Crucially, despite this internal structure, the model remains a standard Elman RNN. It doesn't rely on complex gating (like LSTMs) or attention mechanisms, allowing the authors to isolate the effect of temporal rank preservation.

Datasets

The model was evaluated on two classic "stress tests" for long-range memory in recurrent models:

  • The Copy Task: The network must memorize a short sequence of tokens, wait for a long "lag" period ($L$), and then reproduce the original sequence exactly.
  • The Adding Problem: The network receives a long sequence of random numbers, with two specific positions marked by a secondary binary input. It must output the sum of the numbers at those two marked positions.

Key Insights and Ideas

  • Gradients $\neq$ Rank: Even when gradient norms are perfectly preserved (e.g., using orthogonal initialization), the stable rank can still plateau far below the hidden dimension. This means a "stable" network can still be functionally "blind" to past inputs.
  • Linear Growth is Key: In the Shift RNN, the stable rank increases linearly with time, matching the theoretical maximum. This allows the model to "accumulate" information rather than just "overwriting" it.
Figure 1: Stable rank and input contribution norms to the final hidden state hT across random, autoregressive, and adding sequence tasks for different recurrent parametrisations.
  • Structural Prior over Mixing: Standard RNNs rely on "mixing" dynamics that often lead to premature saturation. The Shift RNN acts as a proof-of-concept that explicitly managing how representational rank is allocated across time is a powerful design principle.

Key Results

The Shift RNN significantly outperformed both vanilla and orthogonally-constrained RNNs on the benchmark tasks:

  • Copy Task: For lags of $L=100$ and $L=250$, the Shift RNN solved the task completely, while the vanilla RNN diverged and the orthogonal RNN failed to outperform a simple baseline.
Figure 2: Performance on the copy task for lags L = 100 and L = 250: the vanilla RNN diverges and falls outside the plotting range (hence not visible in the figure); the orthogonal RNN briefly approaches near-perfect performance at L = 100 but ultimately returns to the baseline and does not outperform it at L = 250, while the shift RNN solves both settings.
  • Adding Problem: The Shift RNN consistently solved the problem for sequence lengths up to $T=250$ (and even up to $T=2000$ in extended tests), while baseline models were unable to beat a naive mean-output strategy.
Figure 3: Performance on the adding problem for sequence lengths T = 100 and T = 250: the vanilla RNN diverges and falls outside the plotting range (hence not visible in the figure); the orthogonal RNN does not outperform the baseline, while the shift RNN solves the task.

Limitations and Future Work

  • Lack of Explicit Mixing: While the Shift RNN is excellent at preserving information, it doesn't explicitly handle the mixing of sequence elements (e.g., comparing token A at $t=1$ with token B at $t=50$). The authors suggest that future work should investigate "controlled mixing" mechanisms that preserve rank while allowing for complex interactions.
  • State Dimensionality: The construction requires a hidden state dimension that scales with the sequence horizon ($d = T \times d'$). While the authors provide a memory-efficient sparse implementation, this scaling might be challenging for extremely long sequences compared to models that compress information more aggressively.
  • Gating Mechanisms: The study focuses on simple RNNs. A natural next step is exploring how gating (as in LSTMs or GRUs) interacts with temporal rank—does gating help preserve rank, or does it introduce its own bottlenecks?

Summary updated

Abstract

Recurrent Neural Networks (RNNs) are a natural framework for temporal modelling, yet are often dismissed due to their sequential, non-parallelisable computation and difficulty in capturing long-range dependencies, typically attributed to vanishing and exploding gradients. We identify a distinct and under-explored limitation: an information bottleneck induced by rank-constrained hidden state dynamics, which limits the number of independent input directions that can be preserved over time, as reflected in the rank of the input-to-state mapping. We show that this constraint restricts temporal capacity independently of gradient stability. To address this, we introduce structured high-dimensional state dynamics that preserve temporal separability and prevent saturation of the input-to-state Jacobian over time. On controlled sequence tasks, the resulting model exhibits rapid learning despite remaining strictly within a standard RNN formulation. Our results demonstrate that RNN performance is governed by the evolution and preservation of temporal rank, and suggest that mitigating gradient pathologies alone is insufficient without explicitly maintaining temporal capacity.
BibTeX
@misc{dobson2026temporal,
  author = {Leo Dobson and Anton Ragni},
  title = {{Temporal Rank in Recurrent Neural Networks}},
  year = {2026},
  eprint = {2611.24394},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24394},
}
Figure