Hierarchical Temporal Representation in Linear Reservoir Computing

Claudio Gallicchio, Alessio Micheli, Luca PedrelliNeural Advances in Processing Nonlinear Dynamic Signals 2017 · OpenReview

TL;DR

This paper investigates the intrinsic hierarchical temporal representations learned by deep recurrent neural networks (RNNs) by using a simplified linear version of deep Echo State Networks (L-deepESNs). By analyzing the frequency spectrum of the network's internal states, they demonstrate that layering in RNNs naturally creates a hierarchy of temporal feature processing, with deeper layers focusing on slower dynamics, even without non-linearities or explicit training of the recurrent weights. This finding is validated by achieving state-of-the-art performance on challenging time-series prediction tasks.

What Problem Does This Paper Address?

Deep recurrent neural networks (RNNs) have shown great promise in modeling sequential data, with recent work highlighting the importance of "layering" or creating deep architectures. However, understanding how these layers contribute to learning complex temporal patterns, especially those with multiple time scales, remains an active research area. Specifically, the paper aims to disentangle the role of network architecture (layering) from the role of learning and non-linear activation functions in developing hierarchical temporal representations. They want to show that the hierarchical structure itself, even in a simplified linear setting, is crucial for capturing multi-scale dynamics.

Methods

The core of the paper's methodology lies in using a Linear Deep Echo State Network (L-deepESN). This is a deep RNN architecture where each layer is a reservoir of interconnected recurrent units, and crucially, these units employ linear activation functions. This simplification is key because it allows for a direct analysis of the temporal dynamics without the confounding effects of non-linear distortions.

The authors then employ frequency analysis, specifically the Fast Fourier Transform (FFT), on the internal states of each reservoir layer. By examining the frequency components of these states, they can understand what temporal scales each layer is sensitive to. The L-deepESN architecture allows for an algebraic simplification, showing that the entire deep network can be represented by an equivalent single-layer system with a structured (lower triangular block) weight matrix, reflecting the hierarchical flow of information.

Datasets

The paper uses the Multiple Superimposed Oscillator (MSO) task for experimental assessment. This task involves predicting a time-series generated by summing multiple sinusoidal functions with different frequencies. The authors specifically focus on versions of the MSO task with n ranging from 5 to 12 sine waves, which creates a rich and challenging scenario with many distinct time scales. They use a standard split for training (400 steps, with 100 washout), validation (300 steps), and testing (300 steps). No new datasets are introduced.

Key Insights and Ideas

  • Layering as Intrinsic Hierarchy: The most significant insight is that the hierarchical structure of deep RNNs, even with linear units, inherently leads to a hierarchical processing of temporal information. Deeper layers naturally focus on slower time scales, while earlier layers capture faster dynamics.
  • Linearity Reveals Structure: By removing non-linearities, the authors can isolate the effect of the network's architecture. This allows them to demonstrate that the hierarchical organization itself is a powerful mechanism for temporal representation, independent of complex learned non-linear mappings.
  • Incremental Filtering: The frequency analysis reveals an "incremental filtering effect." As signals propagate through deeper layers, higher frequencies are progressively filtered out, and lower frequencies become more dominant. This is mathematically supported by the structure of the L-deepESN's state transition matrix.
  • Compositionality in Temporal Features: The work suggests that deep RNNs exhibit a form of compositionality in temporal feature representation, where different layers specialize in different temporal scales, and these representations are ordered by depth.

Key Results

  • State-of-the-Art Performance: The L-deepESN model achieved state-of-the-art performance on the MSO5-12 tasks, outperforming existing methods by 3-4 orders of magnitude in terms of Normalized Root Mean Square Error (NRMSE).
  • Hierarchical Advantage: The L-deepESN consistently outperformed its shallow counterpart, the Linear Echo State Network (L-ESN), demonstrating the benefit of the layered architecture for capturing multi-scale dynamics.
  • Frequency Spectrum Analysis: Figure 3 clearly shows that the state activations in deeper layers (levels 4, 7, 10) exhibit diversified magnitudes in their frequency components, with a stronger focus on lower frequencies, compared to the relatively uniform magnitudes in the first layer. This visually confirms the hierarchical processing of time scales.

Limitations and Future Work

  • Linearity Assumption: While the use of linear units simplifies analysis, real-world deep RNNs typically employ non-linear activation functions. The authors acknowledge that their findings provide a foundational understanding, and future work could explore how non-linearities interact with this inherent hierarchical structure.
  • Readout Layer Training: While the recurrent weights are not trained, the readout layer is trained using standard methods (ridge regression). The paper suggests that the structured temporal representations learned by the L-deepESN layers can be effectively exploited by this linear readout.
  • Future Research Directions: The authors propose that future research should consider shifting focus from solely learning and non-linearities to the concepts of hierarchical organization and distribution of representation as key defining aspects of deep learning for recurrent architectures. They hope their work opens avenues for exploring these architectural merits.

Summary updated

Abstract

Recently, studies on deep Reservoir Computing (RC) highlighted the role of layering in deep recurrent neural networks (RNNs). In this paper, the use of linear recurrent units allows us to bring more evidence on the intrinsic hierarchical temporal representation in deep RNNs through frequency analysis applied to the state signals. The potentiality of our approach is assessed on the class of Multiple Superimposed Oscillator tasks. Furthermore, our investigation provides useful insights to open a discussion on the main aspects that characterize the deep learning framework in the temporal domain.
BibTeX
@inproceedings{gallicchio2017hierarchical,
  author = {Claudio Gallicchio and Alessio Micheli and Luca Pedrelli},
  title = {{Hierarchical Temporal Representation in Linear Reservoir Computing}},
  booktitle = {Neural Advances in Processing Nonlinear Dynamic Signals},
  year = {2017},
  url = {https://openreview.net/forum?id=4t4FUYq6W3B},
}
Figure