Reservoir computing for spatiotemporal signal classification without trained output weights

Ashley PraterarXiv 2016 · OpenReview

Abstract

Reservoir computing (RC) is a machine learning paradigm well-suited for processing spatiotemporal data. Traditionally, RC networks utilize fixed internal connections and weights, with only the output layer weights trained via linear regression. This paper proposes a novel approach that bypasses the weight training step entirely, instead employing a supervised clustering method based on the principal components of the norms of reservoir states. Mathematical analysis and numerical experiments on real-world data demonstrate that this method can outperform traditional trained output weight approaches in both classification accuracy and robustness to reservoir hyperparameters.

Introduction

Reservoir computing (RC) is a bio-inspired recurrent neural network paradigm designed for efficient spatiotemporal data processing. In a standard RC setup, the hidden layer—the "reservoir"—consists of nodes with fixed, randomly assigned weights. The traditional "readout" method involves training a final output layer using simple linear or ridge regression to map high-dimensional reservoir dynamics to class labels.

While computationally efficient, this linear readout method is often sensitive to hyperparameters such as reservoir size and input gain, and it can struggle with datasets exhibiting high intra-class variation or non-linear separability. This research investigates whether signal classification can be achieved more effectively by directly clustering the reservoir's dynamic responses rather than training linear weights.

Methodology

The study compares the traditional linear readout approach against a novel clustering method across two primary reservoir architectures: Echo State Networks (ESNs) and Time Delay Reservoirs (TDRs).

Architecture of an Echo State Network (ESN) reservoir, showing fixed internal connections and the traditional trained output weights $W_{out}$.

PCA-Based Clustering of Reservoir Norms

The proposed method assumes that similar inputs generate similar reservoir dynamics. The process involves two phases:

  1. Training: Training signals are fed into the reservoir. At each time step, the $L_2$ norm of the reservoir's state vector is calculated. These time-series norm vectors are grouped by class, and Principal Component Analysis (PCA) is used to define a characteristic subspace for each class.
  2. Testing: A test signal's norm vector is projected onto the PCA subspace of each class. The signal is assigned to the class that yields the smallest projection residual (the distance between the vector and the subspace).

Theoretical support for this approach is provided through mathematical proofs (Theorems 1 and 2) that establish upper bounds on the distance between reservoir states for different inputs, ensuring that small input perturbations lead to bounded differences in the reservoir response.

Experimental Evaluation

The methods were evaluated using the USPS handwritten digit dataset, consisting of 16x16 pixel grayscale images. To adapt these for spatiotemporal processing, images were flattened into 256-length 1D vectors. The dataset includes 10 classes (digits 0-9), with 400 images per class used for training and 7,000 images for testing.

Results and Discussion

The clustering approach demonstrated significant advantages over the traditional linear readout:

  • Superior Accuracy and Robustness: The clustering method consistently outperformed trained linear weights across various configurations. Notably, while the traditional method's performance fluctuated significantly with changes in input gain ($\alpha$) and reservoir size ($N$), the PCA-clustering method remained remarkably stable.
  • Separation Metrics: Analysis of "Max Separation"—a proxy for class distinguishability—shows that the proposed method maintains higher separation across a wider range of hyperparameters compared to the traditional approach.
Max separation vs. input gain ($\alpha$) for ESN and TDR reservoirs. The proposed clustering method (red) shows higher stability and separation compared to the traditional linear readout (blue).
  • Reservoir Utility: Applying PCA clustering directly to raw input data achieved 95.27% accuracy. However, passing data through the reservoir first improved this result, confirming that the reservoir's non-linear spatiotemporal processing adds valuable discriminative information.
  • Computational Trade-off: The primary drawback of the clustering method is a slight increase in inference time. Classifying the 7,000 test images took approximately 35-40 seconds longer than the simple matrix multiplication required by the traditional linear readout.

Conclusion

By replacing trained output weights with a supervised clustering method based on reservoir state norms, this research offers a more robust and accurate alternative for spatiotemporal signal classification. While slightly more computationally intensive during inference, the method significantly reduces the need for tedious hyperparameter tuning. Future work will focus on optimizing inference speed and exploring the use of subclasses to better handle extreme intra-class variations.

Summary updated

Abstract

Reservoir computing is a recently introduced machine learning paradigm that has been shown to be well-suited for the processing of spatiotemporal data. Rather than training the network node connections and weights via backpropagation in traditional recurrent neural networks, reservoirs instead have fixed connections and weights among the `hidden layer' nodes, and traditionally only the weights to the output layer of neurons are trained using linear regression. We claim that for signal classification tasks one may forgo the weight training step entirely and instead use a simple supervised clustering method based upon principal components of norms of reservoir states. The proposed method is mathematically analyzed and explored through numerical experiments on real-world data. The examples demonstrate that the proposed may outperform the traditional trained output weight approach in terms of classification accuracy and sensitivity to reservoir parameters.
BibTeX
@misc{prater2016reservoir,
  author = {Ashley Prater},
  title = {{Reservoir computing for spatiotemporal signal classification without trained output weights}},
  year = {2016},
  eprint = {1604.03073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1604.03073},
}
Figure