The interplay between randomness and structure during learning in RNNs

Friedrich Schüßler, Francesca Mastrogiuseppe, Alexis M. Dubreuil, Srdjan Ostojic, Omri Barak

When Recurrent Neural Networks (RNNs) learn low-dimensional tasks, they do not overhaul their entire connectivity structure. Instead, they make precise, "low-rank" adjustments that leverage the existing random structure of the network. Recent research reveals that random initial weights are not just a starting point to be overwritten—they act as a critical scaffold that significantly accelerates the learning process.

The Paradox of High-Dimensional Learning

In both machine learning and neuroscience, RNNs are typically initialized with random weights and then trained to perform specific tasks. While it is well-established that these networks can learn, the exact relationship between the final connectivity and the initial random state has remained elusive.

Specifically, researchers face a paradox: if a network starts with a massive, high-dimensional random matrix ($W_0$) and is trained using unconstrained gradient descent, why does the resulting network often behave as if it has a very simple, low-dimensional structure? Understanding this interplay between randomness and structure is crucial for making RNNs more interpretable, efficient, and biologically plausible.

Methods and Tasks

To investigate this, the authors employed a two-pronged approach combining numerical simulation and analytical theory:

  1. Numerical Simulation: They trained nonlinear RNNs (using the tanh activation function and the Adam optimizer) on classic neuroscience-inspired tasks. They then decomposed the final weight matrix $W$ into the initial weights $W_0$ and the changes induced by training, $\Delta W$.
  2. Analytical Theory: To understand the underlying mechanics, they analyzed a simplified, mathematically tractable linear RNN using a continuous-time approximation of gradient descent ("gradient flow").
  3. Validation: Finally, they tested their findings on a more complex, "real-world" architecture by training a two-layer Long Short-Term Memory (LSTM) network on a sentiment analysis task.

The study utilized three primary neuroscience tasks alongside a natural language processing dataset:

  • Flip-flop task: A working memory task requiring the network to store and update bits of information.
  • Mante task: A context-dependent decision-making task.
  • Romo task: A delayed-discrimination task involving comparing two pulses separated by time.
  • Sentiment Analysis: A natural language processing task classifying movie reviews as positive or negative.
Task performance and low-rank weight updates

Key Insights: The "Scaffold" Effect

Even though gradient descent is free to change every single weight in the network, the observed changes ($\Delta W$) are effectively low-rank. This means the "learning" happens along just a few key directions in the high-dimensional space.

The initial random connectivity ($W_0$) is not merely noise to be overcome; it is a vital resource. The gradient "picks out" directions in the random matrix that align with the task. This correlation between $W_0$ and $\Delta W$ is what actually drives the network's performance.

Furthermore, increasing the strength of the initial random weights (up to a certain point) makes learning faster and smoother. The initial weights amplify the gradient along the necessary task-relevant directions. Learning also exhibits a temporal hierarchy: different "ranks" of the connectivity matrix grow at different speeds, meaning simple structures emerge first, followed by more complex ones.

Key Results and Applications

  • Rank Truncation: For the neuroscience tasks, the authors found that $\Delta W$ could be truncated to a rank as low as 2 to 6 without losing performance, even though the full matrix contains hundreds of dimensions.
  • Learning Speedup: In the linear model, the authors analytically proved that initial connectivity accelerates learning by a factor of $1/(1-g^2)^2$, where $g$ is the strength of the initial weights.
  • LSTM Compression: In the sentiment analysis task, the authors demonstrated a massive opportunity for neural network compression. By keeping the initial random weights and only a rank-10 approximation of the changes, they recovered full model accuracy.
LSTM accuracy after rank truncation

As shown above, truncating only the weight updates ($\Delta W$) preserves accuracy at a much lower rank than the standard technique of truncating the entire final weight matrix ($W$). This suggests we could store only a random seed and a few low-rank vectors instead of millions of weight parameters.

Limitations and Future Work

While the authors show these results generalize to nonlinear RNNs and LSTMs, much of the rigorous mathematical proof relies on linear networks. The exact analytical mapping for nonlinear systems remains an open challenge.

Additionally, there is a limit to the "randomness is good" rule. If the initial weights are too strong ($g > 1$), the network enters a chaotic regime, and learning performance drops sharply. Future work will also need to explore how these "low-rank" changes scale when the task itself is highly dimensional, such as in complex video processing.

Summary updated

Abstract

Training recurrent neural networks (RNNs) on low-dimensional tasks has been widely used to model functional biological networks. However, the solutions found by learning and the effect of initial connectivity are not well understood. Here, we examine RNNs trained using gradient descent on different tasks inspired by the neuroscience literature. We find that the changes in recurrent connectivity can be described by low-rank matrices. This observation holds even in the presence of random initial connectivity, although this initial connectivity has full rank and significantly accelerates training. To understand the origin of these observations, we turn to an analytically tractable setting: training a linear RNN on a simpler task. We show how the low-dimensional task structure leads to low-rank changes to connectivity, and how random initial connectivity facilitates learning. Altogether, our study opens a new perspective to understand learning in RNNs in light of low-rank connectivity changes and the synergistic role of random initialization.
BibTeX
@inproceedings{schuler2020interplay,
  author = {Friedrich Schüßler and Francesca Mastrogiuseppe and Alexis M. Dubreuil and Srdjan Ostojic and Omri Barak},
  title = {{The interplay between randomness and structure during learning in RNNs}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=Ru_iFY7FiF},
}
Figure