Random Oscillators Network for Time Series Processing

Andrea Ceni, Andrea Cossu, Maximilian W. Stölzle, Jingyue Liu, Cosimo Della Santina, Davide Bacciu, Claudio GallicchioAISTATS 2024 · OpenReview

Abstract

We introduce the Random Oscillators Network (RON), a physically-inspired recurrent model derived from a network of heterogeneous oscillators. Unlike traditional recurrent neural networks, RON keeps the connections between oscillators untrained by leveraging on smart random initialisations, leading to exceptional computational efficiency. A rigorous theoretical analysis finds the necessary and sufficient conditions for the stability of RON, highlighting the natural tendency of RON to lie at the edge of stability, a regime of configurations offering particularly powerful and expressive models. Through an extensive empirical evaluation on several benchmarks, we show four main advantages of RON. 1) RON shows excellent long-term memory and sequence classification ability, outperforming other randomised approaches. 2) RON outperforms fully-trained recurrent models and state-of-the-art randomised models in chaotic time series forecasting. 3) RON provides expressive internal representations even in a small parametrisation regime making it amenable to be deployed on low-powered devices and at the edge. 4) RON is up to two orders of magnitude faster than fully-trained models.

Introduction

Recurrent Neural Networks (RNNs) are essential for sequence processing, yet they frequently struggle with two major issues: the "vanishing gradient" problem, which hinders the learning of long-term dependencies, and the high computational cost associated with training. While physically-inspired models based on ordinary differential equations (ODEs) have emerged as stable alternatives, they often still rely on expensive backpropagation through time (BPTT). The Random Oscillators Network (RON) addresses these challenges by combining the natural dynamics of physical oscillators with the efficiency of the Reservoir Computing (RC) paradigm.

The Random Oscillators Network (RON)

RON is designed as a pool of heterogeneous, forced mechanical oscillators. Its architecture follows the Reservoir Computing framework, where the internal recurrent weights, input weights, and biases are randomly initialized and kept fixed (untrained). Only a simple linear "readout" layer is trained, typically using efficient ridge regression.

Architecture of the Random Oscillators Network (RON) showing the reservoir of heterogeneous oscillators and the linear readout layer.

The model is mathematically derived from a second-order ODE, discretized using an Euler scheme. As shown in the architecture above, an input signal $\mathbf{u}(t)$ is fed into an excitation layer of neurons, which in turn drives the $N$ oscillators. The state of the reservoir $\mathbf{y}(t)$ is then mapped to the output $\mathbf{r}(t)$ via a trained linear readout.

Physical Foundations and Stability

The core of RON lies in the physics of a damped harmonic oscillator, characterized by stiffness ($\gamma$) and damping ($\epsilon$) coefficients. By employing a heterogeneous pool where these parameters vary across units, the network can respond to a wide range of frequencies in the input data, creating a rich, high-dimensional representation.

Physical intuition of RON: (a) schematic of a damped oscillator, (b) frequency response (Bode plot), and (c) time-domain responses for varying stiffness and damping parameters.

A key theoretical contribution of this work is the stability analysis. The authors prove that RON naturally gravitates toward the "edge of stability." This regime is critical for computational power, as it allows the network to maintain memory across thousands of time steps without the signal vanishing or exploding. Furthermore, RON is shown to be a mathematical generalization of the Leaky-ESN (Echo State Network) model.

Experimental Evaluation

RON was evaluated across several challenging benchmarks, demonstrating significant advantages in speed and performance:

  • Computational Efficiency: RON is remarkably fast. On the Sequential MNIST (sMNIST) task, RON achieved comparable accuracy to fully-trained models like hcoRNN but trained in approximately 11 minutes compared to hcoRNN's 230 minutes—a 20x speedup.
  • Chaotic Forecasting: In predicting the Lorenz96 system, RON achieved a Normalized Root Mean Squared Error (NRMSE) of 0.0016, vastly outperforming LSTMs (0.24) and even state-of-the-art randomized models like Leaky-ESN (0.0020).
  • Long-Term Memory: Using the npAdiac task (shape classification with added noise), RON demonstrated superior robustness. While LSTM performance plummeted as noise padding increased, RON’s accuracy remained stable, proving its ability to ignore interference and maintain long-term memory.
  • Edge Readiness: Due to its expressive representations even with few parameters, RON is highly suitable for deployment on low-power edge devices.

Conclusion

The Random Oscillators Network offers a powerful, physics-inspired alternative to traditional RNNs. By leveraging the inherent stability of damped oscillators and the training efficiency of Reservoir Computing, RON achieves state-of-the-art results on chaotic and long-memory tasks with a fraction of the computational overhead. Future work includes exploring multi-layer "Deep RON" architectures and potential implementations on neuromorphic or analog hardware.

Summary updated

Abstract

We introduce the Random Oscillators Network (RON), a physically-inspired recurrent model derived from a network of heterogeneous oscillators. Unlike traditional recurrent neural networks, RON keeps the connections between oscillators untrained by leveraging on smart random initialisations, leading to exceptional computational efficiency. A rigorous theoretical analysis finds the necessary and sufficient conditions for the stability of RON, highlighting the natural tendency of RON to lie at the edge of stability, a regime of configurations offering particularly powerful and expressive models. Through an extensive empirical evaluation on several benchmarks, we show four main advantages of RON. 1) RON shows excellent long-term memory and sequence classification ability, outperforming other randomised approaches. 2) RON outperforms fully-trained recurrent models and state-of-the-art randomised models in chaotic time series forecasting. 3) RON provides expressive internal representations even in a small parametrisation regime making it amenable to be deployed on low-powered devices and at the edge. 4) RON is up to two orders of magnitude faster than fully-trained models.
BibTeX
@inproceedings{ceni2024random,
  author = {Andrea Ceni and Andrea Cossu and Maximilian W. Stölzle and Jingyue Liu and Cosimo Della Santina and Davide Bacciu and Claudio Gallicchio},
  title = {{Random Oscillators Network for Time Series Processing}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=idgMr9iPkz},
}
Figure