Abstract
Recurrent neural networks (RNNs) trained on low-dimensional tasks have been widely used to model functional biological networks. However, the solutions found by learning and the effect of initial connectivity are not well understood. Here, we examine RNNs trained using gradient descent on different tasks inspired by the neuroscience literature. We find that the changes in recurrent connectivity can be described by low-rank matrices, despite the unconstrained nature of the learning algorithm. To identify the origin of the low-rank structure, we turn to an analytically tractable setting: training a linear RNN on a simplified task. We show how the low-dimensional task structure leads to low-rank changes to connectivity. This low-rank structure allows us to explain and quantify the phenomenon of accelerated learning in the presence of random initial connectivity. Altogether, our study opens a new perspective to understanding trained RNNs in terms of both the learning process and the resulting network structure.
Introduction
Recurrent Neural Networks (RNNs) are the workhorses of both AI and computational neuroscience, often used to model biological neural dynamics. However, a significant "black box" problem remains: we do not fully understand how unconstrained gradient descent modifies weights to solve tasks. Specifically, the relationship between the initial random connectivity ($W_0$) and the final learned solution is unclear. Does the network ignore initial randomness, or does it build upon it? This paper investigates the interplay between initial noise and learned structure.
Methods
The study employs a two-pronged approach combining numerical simulation and analytical theory.
- Numerical Simulation: Non-linear RNNs (using
tanhactivation) were trained on neuroscience-inspired tasks: the Flip-flop task (working memory), the Mante task (context-dependent decision making), and the Romo task (delayed discrimination). Additionally, a 2-layer LSTM was trained on the SST-2 sentiment analysis dataset to test generalizability. - Analytical Theory: To derive the mathematical principles of weight evolution, the authors analyzed linear RNNs on simplified tasks using gradient flow equations.
- Rank Analysis: The researchers decomposed the final weight matrix $W$ into $W_0 + \Delta W$ and used Singular Value Decomposition (SVD) to measure the "functional rank" of the changes.
Key Results
The Low-Rank Nature of Learning
Despite the high dimensionality of the weight space, the changes in connectivity ($\Delta W$) are remarkably low-rank. For low-dimensional tasks, the network only modifies its behavior along a few specific directions. For example, the Flip-flop task requires only a rank-2 change, while more complex tasks like Mante or Romo require rank-4 to rank-6 changes.
Randomness as a Learning Accelerator
Initial random connectivity is not merely a starting point; it acts as a scaffold that accelerates learning. The authors demonstrate that the stronger the initial random weights (quantified by $g$), the faster the network converges. Their analytical model predicted that learning time scales with $(1-g^2)^2$, a curve closely matched by numerical results.
Correlation and Structure
The learned structure is intrinsically tied to the specific initial random state. If the initial weights are shuffled after training, the network's performance collapses. This indicates that $\Delta W$ is not just added "next" to $W_0$, but is highly correlated with the specific realization of the initial randomness.
Discussion and Limitations
This research suggests that trained RNNs can be viewed as low-rank "patches" applied to random matrices. This perspective allows for significant model compression; in LSTM tasks, truncating $\Delta W$ to rank 10 maintained full accuracy, whereas truncating the full matrix $W$ required a rank of 128.
However, limitations remain. The study primarily focuses on low-dimensional tasks; it is yet to be determined if $\Delta W$ remains low-rank for high-dimensional inputs like video. Furthermore, while the linear theory is robust, the full impact of non-linearities in the chaotic regime ($g > 1$) requires further exploration. Future work may extend these findings to modern architectures like Transformers.
Summary updated