Abstract
Recurrent neural networks (RNNs) are computational models regarded as dynamical systems. Modularity is a key ingredient of complex systems. Thus, the composition of RNN modules provides a simple paradigm for building complex computational models, with the potential to approach the human brain capability. We devise strategies for training RNNs assembled into a larger RNN of RNNs, provided with theoretical guarantees of stability that hold during training for the composed global network. Experiments on pixel-by-pixel image classification benchmarks prove the effectiveness of this approach.
Introduction
Recurrent Neural Networks (RNNs) are powerful tools for processing sequential data, but they are notoriously difficult to train because they behave like dynamical systems. If not carefully constrained, their internal states can "explode" or "vanish," leading to instability.
One promising approach to scaling RNNs is modularity—building a large network out of smaller, specialized sub-units, much like the human brain. However, previous attempts at these "assemblies" faced a catch-22: either the training process was computationally exhausting to ensure stability, or the internal modules had to be kept "frozen" (untrained) to keep the math simple. This research addresses the gap by creating modular RNNs where internal components can learn and adapt without sacrificing global stability guarantees.
Methodology: The AdaDiag Sparse Net
The authors introduce AdaDiag Sparse Net, a framework for coupling multiple RNN modules. The architecture relies on two main pillars to ensure stability while allowing for adaptivity.
Skew-Symmetric Coupling
To connect different RNN modules, the authors use a specific mathematical constraint where the coupling matrices $L_{ij}$ satisfy $L_{ij} = -L_{ji}^T$. This skew-symmetric coupling ensures that the energy exchange between modules does not lead to system-wide instability, regardless of the number of modules.
Adaptive Diagonal Internal Weights
Instead of full weight matrices inside each RNN module, which are difficult to stabilize during training, the proposed architecture uses diagonal matrices ($W_i$). In this configuration, neurons within a single module only possess "self-loops."
To maintain stability during training, two strategies were proposed to keep the spectral norm of these internal matrices below unity:
- AdaDiag with tanh: Passing the diagonal weights through a
tanhfunction to force them into the $(-1, 1)$ range. - AdaDiag with clip: Manually clipping the weights to the $[-0.99, 0.99]$ range.
By keeping these internal matrices contractive and using skew-symmetric coupling, the entire assembly remains stable throughout the training process.
Experimental Evaluation
The models were evaluated on two standard benchmarks for sequential processing:
- Sequential MNIST (sMNIST): Pixels of MNIST digits are fed into the network one by one (784 steps).
- Permuted Sequential MNIST (psMNIST): The same as sMNIST, but the pixel order is randomly shuffled, significantly increasing the difficulty of capturing long-range dependencies.
Key Results
The AdaDiag Sparse Net consistently outperformed the baseline Sparse Combo Net (SCN) and Vanilla RNN:
- psMNIST Performance: With a coupling factor of $C=20$, the
clipstrategy achieved 89.63% accuracy, significantly higher than the SCN baseline of ~85.91%. - sMNIST Performance: The
clipstrategy reached 87.74% with low coupling ($C=5$), whereas the baseline reached only ~77.99%. - Efficiency: Training time increased only slightly (from ~185s per epoch for the baseline to ~206s for the adaptive strategies), demonstrating that stability and adaptivity do not require prohibitive computational overhead.
Discussion and Key Insights
A primary insight of this work is that local simplicity can lead to global complexity. While diagonal internal weights restrict intra-module communication to self-loops, the coupling matrices between modules restore representational power. Communication is shifted from occurring within modules to occurring across them.
Furthermore, stability was found to breed consistency. The adaptive models exhibited much lower standard deviation in performance across different runs compared to non-adaptive baselines, suggesting that the ability to tune internal dynamics allows the network to recover from poor initializations.
Limitations and Future Work
While diagonal matrices provide a clean path to stability, they limit the "local" expressiveness of each module. Future research aims to generalize this approach to non-diagonal internal architectures, investigate deeper hierarchical structures, and test the framework on a broader variety of sequential tasks beyond image classification.
Summary updated