Reservoir Topology in Deep Echo State Networks

Claudio Gallicchio, Alessio Micheli

Introduction

Reservoir Computing (RC) offers an efficient framework for Recurrent Neural Networks (RNNs) by utilizing a fixed, randomly initialized "reservoir" and training only a linear readout layer. Deep Echo State Networks (DeepESNs) extend this concept by stacking multiple reservoir layers, enabling the capture of information across diverse temporal scales. While traditional RC relies on sparse random connectivity, this research investigates whether specific internal reservoir topologies can further enhance performance in deep architectures.

DeepESN Architecture

The DeepESN architecture consists of a hierarchy of reservoir layers. The external input $\mathbf{u}(t)$ is fed only into the first layer. Each subsequent layer $L$ receives the state of the previous layer $\mathbf{x}^{(L-1)}(t)$ as its input. A key feature of this design is that the final output is derived from a linear combination of states across all layers, allowing the model to leverage multiple levels of temporal abstraction.

The hierarchical structure of a Deep Echo State Network, showing the flow of information through successive reservoir layers.

Structured Reservoir Topologies

The study compares four distinct topologies for the recurrent weight matrices within each reservoir layer:

  1. Sparse (Baseline): Random connectivity where each unit connects to a small subset of others.
  2. Permutation: Each unit has exactly one incoming and one outgoing connection, forming disjoint cyclic structures.
  3. Ring: A single continuous loop of units.
  4. Chain: A delay-line structure where units feed forward without a final loop-back.
The Permutation topology, characterized by disjoint cyclic structures, which emerged as the most effective configuration.

The Permutation topology proved particularly effective. As a subset of orthogonal matrices, permutation matrices preserve signal energy and enhance memory capacity, which are critical for processing complex time-series.

Experimental Results

The models were evaluated on chaotic time-series benchmarks, including NARMA10, Santa Fe Laser, and Mackey-Glass (MG17 & MG30). The results demonstrated a clear synergy between network depth and structured topology.

The Permutation-based DeepESN consistently outperformed other configurations. For instance, on the MG17 task, it achieved a Mean Squared Error (MSE) of $4.576 \times 10^{-10}$, nearly ten times better than its shallow counterpart. In the NARMA10 task, it reached an MSE of $1.243 \times 10^{-4}$, significantly surpassing the sparse random baseline. Optimal performance was generally observed at depths of 2 to 5 layers.

Conclusion

Moving beyond random connectivity in DeepESNs yields significant performance gains. Structured topologies, especially permutation matrices, provide a parsimonious yet powerful way to improve the predictive accuracy of reservoir-based models. These findings suggest that the architectural design of untrained layers is a crucial factor in the success of deep recurrent systems, offering a path toward more efficient and hardware-friendly AI.

Summary updated

Abstract

Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. The major outcome of our investigation is to show the remarkable effect, in terms of predictive performance gain, achieved by the synergy between a deep reservoir construction and a structured organization of the recurrent units in each layer. Our results also indicate that a particularly advantageous architectural setting is obtained in correspondence of DeepESNs where reservoir units are structured according to a permutation recurrent matrix.
BibTeX
@inproceedings{gallicchio2019reservoir,
  author = {Claudio Gallicchio and Alessio Micheli},
  title = {{Reservoir Topology in Deep Echo State Networks}},
  booktitle = {ICANN (Workshop)},
  year = {2019},
  doi = {10.1007/978-3-030-30493-5\_6},
}
Figure