Towards Adaptive and Stable Compositional Assemblies of Recurrent Neural Network Modules

Valerio De, Andrea Ceni, Davide Bacciu, Claudio Gallicchio2025 · DOI

Abstract

Recurrent neural networks (RNNs) are computational models regarded as dynamical systems. Modularity is a key ingredient of complex systems. Thus, the composition of RNN modules provides a simple paradigm for building complex computational models, with the potential to approach the human brain capability. We devise strategies for training RNNs assembled into a larger RNN of RNNs, provided with theoretical guarantees of stability that hold during training for the composed global network. Experiments on pixel-by-pixel image classification benchmarks prove the effectiveness of this approach.

Introduction

Recurrent Neural Networks (RNNs) are powerful tools for processing sequential data, but they are notoriously difficult to train because they behave like dynamical systems. If not carefully constrained, their internal states can "explode" or "vanish," leading to instability.

One promising approach to scaling RNNs is modularity—building a large network out of smaller, specialized sub-units, much like the human brain. However, previous attempts at these "assemblies" faced a catch-22: either the training process was computationally exhausting to ensure stability, or the internal modules had to be kept "frozen" (untrained) to keep the math simple. This research addresses the gap by creating modular RNNs where internal components can learn and adapt without sacrificing global stability guarantees.

Methodology: The AdaDiag Sparse Net

The authors introduce AdaDiag Sparse Net, a framework for coupling multiple RNN modules. The architecture relies on two main pillars to ensure stability while allowing for adaptivity.

Skew-Symmetric Coupling

To connect different RNN modules, the authors use a specific mathematical constraint where the coupling matrices $L_{ij}$ satisfy $L_{ij} = -L_{ji}^T$. This skew-symmetric coupling ensures that the energy exchange between modules does not lead to system-wide instability, regardless of the number of modules.

Adaptive Diagonal Internal Weights

Instead of full weight matrices inside each RNN module, which are difficult to stabilize during training, the proposed architecture uses diagonal matrices ($W_i$). In this configuration, neurons within a single module only possess "self-loops."

Comparison between the baseline Sparse Combo Net (left) and the proposed AdaDiag Sparse Net (right).

To maintain stability during training, two strategies were proposed to keep the spectral norm of these internal matrices below unity:

  1. AdaDiag with tanh: Passing the diagonal weights through a tanh function to force them into the $(-1, 1)$ range.
  2. AdaDiag with clip: Manually clipping the weights to the $[-0.99, 0.99]$ range.
Heatmaps of the global weight matrices for different sparsity and diagonal adaptation strategies.

By keeping these internal matrices contractive and using skew-symmetric coupling, the entire assembly remains stable throughout the training process.

Experimental Evaluation

The models were evaluated on two standard benchmarks for sequential processing:

  • Sequential MNIST (sMNIST): Pixels of MNIST digits are fed into the network one by one (784 steps).
  • Permuted Sequential MNIST (psMNIST): The same as sMNIST, but the pixel order is randomly shuffled, significantly increasing the difficulty of capturing long-range dependencies.

Key Results

The AdaDiag Sparse Net consistently outperformed the baseline Sparse Combo Net (SCN) and Vanilla RNN:

  • psMNIST Performance: With a coupling factor of $C=20$, the clip strategy achieved 89.63% accuracy, significantly higher than the SCN baseline of ~85.91%.
  • sMNIST Performance: The clip strategy reached 87.74% with low coupling ($C=5$), whereas the baseline reached only ~77.99%.
  • Efficiency: Training time increased only slightly (from ~185s per epoch for the baseline to ~206s for the adaptive strategies), demonstrating that stability and adaptivity do not require prohibitive computational overhead.

Discussion and Key Insights

A primary insight of this work is that local simplicity can lead to global complexity. While diagonal internal weights restrict intra-module communication to self-loops, the coupling matrices between modules restore representational power. Communication is shifted from occurring within modules to occurring across them.

Furthermore, stability was found to breed consistency. The adaptive models exhibited much lower standard deviation in performance across different runs compared to non-adaptive baselines, suggesting that the ability to tune internal dynamics allows the network to recover from poor initializations.

Limitations and Future Work

While diagonal matrices provide a clean path to stability, they limit the "local" expressiveness of each module. Future research aims to generalize this approach to non-diagonal internal architectures, investigate deeper hierarchical structures, and test the framework on a broader variety of sequential tasks beyond image classification.

Summary updated

Abstract

Recurrent neural networks (RNNs) are computational models regarded as dynamical systems.Modularity is a key ingredient of complex systems.Thus, the composition of RNN modules provides a simple paradigm for building complex computational models, with the potential to approach the human brain capability.We devise strategies for training RNNs assembled into a larger RNN of RNNs, provided with theoretical guarantees of stability that hold during training for the composed global network.Experiments on pixel-by-pixel image classification benchmarks prove the effectiveness of this approach.
BibTeX
@misc{de2025towards,
  author = {Valerio De and Andrea Ceni and Davide Bacciu and Claudio Gallicchio},
  title = {{Towards Adaptive and Stable Compositional Assemblies of Recurrent Neural Network Modules}},
  year = {2025},
  doi = {10.14428/esann/2025.es2025-48},
}
Figure