A theory of noisy linear recurrent neural network learning dynamics

John J. Vastola, Kanaka RajanarXiv 2026 · arXiv

TL;DR

This paper provides a new theoretical framework for understanding how intrinsic noise—a hallmark of biological brains—fundamentally shapes how Recurrent Neural Networks (RNNs) learn. The authors demonstrate that noise is not just a nuisance but a "representational pressure" that breaks solution degeneracies, forces specific weight alignments, and can even trigger phase transitions where a network "gives up" on a task to minimize noise.

What Problem Does This Paper Address?

Recurrent Neural Networks are powerful tools for modeling brain function, but biological neurons are notoriously noisy. Most existing theories of RNN learning assume a "noiseless" environment, treating noise as a secondary factor that might slightly blur a solution.

However, in high-dimensional parameter spaces, many different weight configurations can solve the same task (a property called degeneracy). The authors ask: Does noise act as a tie-breaker? Does it force the network toward specific types of solutions that a noiseless theory would miss? By exploring the "rich regime"—where networks start with small weights and develop complex, low-rank structures—the paper seeks to determine if noise is a fundamental determinant of the representations these networks eventually form.

Methods

The authors utilize an analytically tractable model: a noisy linear RNN trained to perform a leaky integration task (matching a target filter).

Key technical aspects include:

  • Gradient-Flow Limit: They analyze learning dynamics using ordinary differential equations (ODEs) derived from the gradient of a supervised objective function.
  • Objective Function Decomposition: The loss function $J$ is split into three parts: a filter-matching term (task performance), a noise reduction term (minimizing output variance), and a regularization term.
  • Dimensionality Reduction: They show that during the "alignment phase," the high-dimensional learning problem ($N^2 + 2N$ parameters) collapses into a much simpler 3D system involving the scales of read-in, readout, and recurrent weights.
  • Rich Regime Analysis: They specifically focus on learning from small initial weights, which is thought to be more representative of how performant, task-trained RNNs function compared to the "lazy" (kernel) regime.
The theoretical setting: a noisy linear RNN trained to match a filter while minimizing internal noise.

Datasets

Not discussed in the paper. As a theoretical work, it relies on synthetic white noise input signals $x(t)$ and target filters $f_*(t)$ rather than external empirical datasets.

Key Insights and Ideas

  • Representational Pressure: Noise reduces the number of "good" solutions. In a noiseless system, you can rotate or scale weights arbitrarily without changing the output. Noise breaks this symmetry because some directions in the network's state space are "noisier" than others.
  • Preferred Alignment: During the initial stages of learning, weights don't just grow in random directions; they preferentially align with the least noisy eigenvectors of the system's noise covariance matrix. This makes the final solution less dependent on random initialization and more dependent on the noise structure.
  • Symmetry Breaking between $u$ and $v$: In noiseless linear networks, the read-in ($u$) and readout ($v$) weights often evolve symmetrically. Noise breaks this: to minimize output variance, the network tends to drive the readout scale toward zero and the read-in scale toward infinity (unless stopped by regularization).
  • The "Escape Route" Phase Transition: If the intrinsic noise is too high, the network undergoes a phase transition. Instead of trying to solve the task poorly, the learning dynamics drive the network to a "trivial solution" (zero output) because the cost of the noise outweighs the benefit of matching the target filter.
Noise determines weight alignment: higher-noise modes grow more slowly, forcing the network to utilize "quieter" state-space directions.

Key Results

  • Alignment Dynamics: The authors derived exact growth rates for different state-space modes, showing that noise acts as a filter that suppresses learning in "loud" directions.
  • Amplitude Collapse: They identified a critical noise threshold $\sigma^2_{crit}$. When noise exceeds this value, the optimal filter amplitude $N = uv$ collapses to zero. They provided an exact algebraic characterization of this boundary as a function of regularization strength $\kappa$.
  • Frequency Runaway: In scenarios where read-in/readout scales are fixed, excessive noise causes a different failure mode: the recurrent weights drive the system to an "infinitely fast" filter ($c \to \infty$) to effectively "shunt" the noise.
  • Analytic Solutions: The paper provides closed-form solutions for the optimal readout weights in "reservoir computing" setups, showing they naturally weigh input directions by their signal-to-noise ratio.

Limitations and Future Work

  • Linearity: The theory is strictly for linear RNNs. While linear models offer deep insights into learning dynamics, biological and artificial RNNs are non-linear. The authors suggest future work could involve linearizing non-linear networks around fixed points.
  • Simple Tasks: The task (leaky integration) involves only one time scale. Real-world tasks are far more complex.
  • Whitened Inputs: The assumption of white noise input simplifies the math but ignores the temporal correlations found in natural stimuli.
  • Rich Regime Only: The findings are specific to small-weight initializations. It remains to be seen how these noise effects manifest in the "lazy" regime or in networks with large, chaotic initial weights.

Summary updated

Abstract

Biological neural networks are noisy: their dynamics involve difficult-to-predict fluctuations which are likely due not just to chaos, but also to a variety of low-level biophysical processes. Despite the omnipresence of intrinsic noise, its theoretical consequences are incompletely understood, especially in the context of learning. For example, does noise meaningfully shape representation learning, or is it better viewed as a nuisance factor whose presence or absence changes little about what solution is found and when? Here, we argue that intrinsic noise can significantly modulate representation learning in recurrent neural networks (RNNs), since it can strongly break solution-space degeneracies and in some cases control whether a non-degenerate solution exists at all. We show that this is true for an analytically tractable model of noisy RNN learning with a linear activation function and white noise inputs. We find that, in the so-called rich regime (where weights are initialized with small values), noise strongly modulates the directions in which the read-in, readout, and recurrent weights align. Moreover, we find that noise produces a kind of phase transition: for sufficiently high noise, a non-degenerate solution is not even optimal if parameter scales are limited (e.g., via regularization). These results extend recent analytic work on RNN representation learning and show that noise can act as a fundamental determinant of representations.
BibTeX
@misc{vastola2026theory,
  author = {John J. Vastola and Kanaka Rajan},
  title = {{A theory of noisy linear recurrent neural network learning dynamics}},
  year = {2026},
  eprint = {2610.35785},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35785},
}
Figure