Do Wide Recurrent Networks Learn Memory, or Inherit It?

Theory says that a very wide recurrent neural network trained by gradient descent reaches zero training loss at a linear rate. A recurrent network is a model that reads a sequence one step at a time while carrying a hidden state forward. The same theory says that such a network barely moves its weights while it trains. Practitioners, meanwhile, find that recurrent networks struggle most on tasks that require remembering something from hundreds or thousands of steps back. This note proposes that these two facts are one fact seen from two sides. In the nearly frozen regime that makes the convergence proofs work, a recurrent network can only use temporal information that was already reachable at initialization. Its memory span is inherited, not learned. When long-range memory does appear during training, the network must have left that regime.

The proposal turns this into something measurable. We define a kernel memory horizon, computed from the network at initialization before any training, which states how far back in time the frozen, linearized version of the network can see. We then test whether this horizon predicts where training succeeds or fails on long-memory benchmarks, and whether the networks that succeed beyond it are exactly those whose internal features change substantially. If the prediction holds, width-based convergence guarantees for recurrent networks certify memorization of training sequences, not the learning of long memory. It would also give a cheap check, run before training, that says whether an architecture and initialization can solve a given memory length.

Core Claim

Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent shows that as width grows, training becomes equivalent to fitting a linear model in the gradient features of the network at initialization. That linear model is governed by the neural tangent kernel (NTK), the inner product of those parameter gradients between two inputs. The kernel stays fixed during training. On the Convergence Rate of Training Recurrent Neural Networks proves global linear convergence for a ReLU Elman network in this same polynomial-width setting, through a local Polyak–Łojasiewicz condition, a guarantee that gradient size grows with distance from the optimum.

For a recurrent network, the gradient features that reference an input $k$ steps back pass through a product of $k$ hidden-to-hidden Jacobians evaluated at initialization. On the difficulty of training recurrent neural networks showed that such products shrink or blow up exponentially with $k$. Its authors also argued that solving long-dependency tasks requires the network to reshape its dynamics, for example by forming attractors or slow directions. A frozen kernel cannot do that. The two papers therefore imply a claim neither of them tests. The lazy regime has a memory horizon fixed at initialization, and long-memory success beyond that horizon requires feature learning. Feature Learning in $L_2$-regularized DNNs and the mean-field view of Mean-field theory of two-layers neural networks describe the non-lazy alternative for feedforward networks. We carry that distinction into recurrence, where it has a concrete, time-indexed signature.

To make the horizon precise, let $f_0$ be the network at initialization and $\theta$ its parameters. For an input sequence $x$, let $x^{(k)}$ be the same sequence with the input at $k$ steps before the final step resampled from its distribution. Define the lag sensitivity

$$S(k)=\frac{\mathbb{E}_x\,\|\nabla_\theta f_0(x)-\nabla_\theta f_0(x^{(k)})\|^2}{\mathbb{E}_x\,\|\nabla_\theta f_0(x)\|^2},$$

which is the fraction of the network's initial gradient features that change when the input at lag $k$ changes. The kernel memory horizon $H_\varepsilon$ is the largest lag with $S(k)\ge\varepsilon$. We report $H_\varepsilon$ over a range of $\varepsilon$ rather than a single tuned value. $H_\varepsilon$ can be estimated at initialization with a few hundred sequences and random projections of the gradient.

Research Program

The first strand measures horizons. We compute $H_\varepsilon$ for ReLU Elman networks across widths, recurrent weight scales and depths. We also compute it for the element-wise recurrence of IndRNN, which bounds each scalar recurrent weight, and for the diagonal linear recurrence used in BLUR, whose eigenvalues are constrained to the unit circle. Our expectation is that $H_\varepsilon$ scales roughly like $\log\varepsilon/\log\rho$, where $\rho$ is the typical per-step Jacobian gain at initialization. That form follows directly from the exponential Jacobian products in Pascanu et al. Architectures that initialize eigenvalues near one should therefore have very long horizons without any learning. That is a nontrivial prediction about why IndRNN and linear recurrences succeed.

The second strand is a controlled comparison of lazy and rich training. On the same architecture and data, we train three variants. The first is the linearized model, which trains exactly the first-order expansion around $\theta_0$ with the kernel frozen by construction. The second is standard full training. The third is full training with the output multiplied by a small constant $\alpha$ and the loss rescaled to match, which pushes the network away from the lazy regime at fixed width; increasing width pushes it toward lazy behaviour. During training we log three things: the relative kernel change $\|K_t-K_0\|/\|K_0\|$, the relative weight displacement, and the spectrum of the hidden-state Jacobian along trajectories. The Jacobian spectrum lets us detect eigenvalues migrating toward the unit circle, the slow directions that Pascanu et al. associate with memory.

The third strand decomposes error by lag. On the adding problem, the error of every model is grouped by the lag of the earlier flagged number. The prediction is sharp. The linearized model should achieve near-zero error for lags below $H_\varepsilon$ and error near the random-guessing level of 0.167 beyond it. Full training should push the boundary further only when kernel change is large.

Baselines

The naive reference point is predicting the mean. On the adding problem this gives mean squared error 0.167, the random-guessing level stated in the benchmark description. Any model whose lag-bucketed error stays at 0.167 has learned nothing about that lag.

The first trained baseline is the clipped Elman network of On the difficulty of training recurrent neural networks, run from its released code, pascanur/trainingRNNs. That paper reports solving the adding and temporal-order problems for sequences up to 200 steps using gradient norm clipping together with its Jacobian-preserving regularizer. It is the strongest dense-recurrence competitor. We run it at identical width and data, and give its clipping threshold the same tuning budget as all other models.

The strongest existing long-memory method in the file is IndRNN. It is reported to converge on the adding problem at T=1000, where LSTMs failed to drop below the 0.167 baseline, to converge on sequences of over 5,000 steps, and to reach 1.0% test error on sequential MNIST against 1.8% for an LSTM (IndRNN). We do not try to beat IndRNN's accuracy; the proposal is diagnostic. "Beating" it here means something different. Its measured $H_\varepsilon$ at initialization, together with the lazy/rich split, should correctly predict the lengths and recurrent-weight bounds at which it succeeds or fails. That includes deliberately bad bounds under which we expect it to fail.

The key ablation is the linearized model against full training at identical width, initialization, optimizer, step budget and learning-rate sweep. This isolates feature learning as the only difference between them. A second ablation replaces $H_\varepsilon$ with a simpler predictor, the spectral radius of the recurrent Jacobian at initialization. This tests whether the kernel adds anything over classical dynamics analysis.

Why not just…?

Why not just clip gradients and train with AdamW? Clipping addresses the steep cliffs that launch parameters away, as Pascanu et al. showed. It does not change which temporal features exist at initialization. Our lag-bucketed error will show whether a clipped Elman network fails beyond a lag set by its initialization, regardless of clipping threshold. If clipping alone moves the success boundary well past $H_\varepsilon$ while the kernel stays nearly frozen, our claim is wrong. The experiment is designed to reveal exactly that.

Why not just use IndRNN or a linear recurrent unit, which already solve thousands of steps? These architectures are part of the test, not an answer to it. Our hypothesis is that they succeed largely because their initialization places the horizon beyond the task length, so even lazy training suffices. The linearized IndRNN should then nearly match the fully trained one on the adding problem. If instead linearized IndRNN fails where full IndRNN succeeds, then those architectures do rely on feature learning, and the practical advice changes.

Why not just predict success from the Jacobian spectral radius, as in classical dynamics analysis? This is a real threat. For the Elman network, $H_\varepsilon$ may be largely a function of the spectral radius. The spectral-radius ablation settles it. If both predictors are equally accurate across the Elman, IndRNN and diagonal families, we drop the kernel horizon as a tool. We would then report only the lazy-versus-rich finding, which does not depend on which predictor is used. We expect the kernel horizon to win where the readout and input weights shape which lags reach the gradient features. The diagonal family, with heterogeneous eigenvalues, is where that should show.

Why not just make the network wider, as the convergence theory suggests? Wider networks are lazier (Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent). Our hypothesis predicts that width improves training loss but does not extend test-time memory beyond $H_\varepsilon$. The width sweep tests this directly. Allen-Zhu et al. assume sequences are separable within their initial steps, which may be exactly what lets a short-horizon kernel memorize them. We will test whether their guarantee is consistent with this reading by building training sets that are separable only at long lags.

Decisive Experiments

The main benchmarks are the adding problem and the temporal order problem, at lengths from 50 to 5,000. Sequential MNIST serves as a realistic long-integration check, with pixels grouped by distance from the final step so that error can be attributed to lag. Character-level Penn Treebank, where IndRNN reports 1.21 bits per character, serves only as a scale sanity check on whether lazy fractions measured on synthetic tasks carry over.

Three results would confirm the proposal. First, linearized-model error on the adding problem rises from near zero to 0.167 at the lag predicted by $H_\varepsilon$, across initialization scales. Second, every fully trained model that succeeds beyond its $H_\varepsilon$ shows large kernel change and Jacobian eigenvalues migrating toward one, while models that stay lazy do not succeed. Third, increasing width at a fixed small-horizon initialization lowers training loss but not long-lag test error.

Two results would refute it. One is lazy models succeeding far beyond $H_\varepsilon$. The other is rich models showing no advantage at long lags. Either outcome is informative, because it would mean frozen-kernel theory already explains recurrent memory. Fluctuation-dissipation relations for stochastic gradient descent found strongly non-quadratic loss basins. That finding warns that the linearization may fit poorly even when training looks lazy. We therefore also report how far the full model's outputs deviate from the linearized model's along each run, so that we do not mislabel a run as lazy.