Random projections: A remedy for overfitting issues in time series prediction with echo state networks

Lucie Daubigney, Matthieu Geist, Olivier Pietquin

TL;DR

This paper introduces a novel method to improve time series prediction using Echo State Networks (ESNs) by combining them with random projections. This approach effectively mitigates overfitting, allowing for richer reservoir dynamics without sacrificing prediction accuracy, even with limited training data.

What Problem Does This Paper Address?

Accurately modeling time series, which represent data evolving over time (like temperature or stock prices), is a challenging task. Reservoir computing, particularly Echo State Networks (ESNs), has emerged as a powerful tool for this. ESNs use recurrent neural networks with a large number of "reservoir" neurons to capture complex temporal dependencies. However, a critical challenge arises: the trade-off between the reservoir's complexity (number of neurons) and the risk of overfitting. A larger reservoir can model more intricate dynamics but requires more training data to avoid learning noise instead of the underlying patterns. Conversely, a smaller reservoir reduces overfitting but may miss important temporal features. This paper aims to break this trade-off, enabling the use of large, complex reservoirs without the penalty of overfitting.

Methods

The core innovation lies in combining Echo State Networks (ESNs) with random projections (RPs).

  • Echo State Networks (ESNs): These are a type of Recurrent Neural Network (RNN) where the vast majority of connections (input-to-reservoir and reservoir-to-reservoir) are fixed randomly. Only the connections from the reservoir to the output layer are trained. This "echo state property" allows the reservoir to implicitly store information about past inputs, making it adept at capturing time dependencies.
  • Random Projections (RPs): The paper leverages a mathematical theorem (based on Johnson-Lindenstrauss lemma) that guarantees it's possible to project high-dimensional data into a lower-dimensional space while preserving the distances between data points with high probability. The key is that the dimension of the projected space depends on the amount of data, not the original dimension.
  • The Combination: Instead of training the output weights directly from the high-dimensional reservoir state x(k), the paper proposes projecting x(k) into a lower-dimensional space using a random projection matrix Π. The output weights Wout are then trained using this projected state Πx(k). This effectively reduces the number of parameters to be learned, thereby combating overfitting.

Datasets

The experiments were conducted using a 10th order non-linear NARMA time series. This is a standard benchmark dataset for time series prediction, previously used in related research. The dataset was generated using an independent and identically distributed uniform input signal.

Key Insights and Ideas

  • Decoupling Reservoir Size from Learnable Parameters: The most significant insight is that random projections allow researchers to decouple the number of neurons in the reservoir (which determines the richness of dynamics) from the number of parameters that need to be learned. This is a fundamental shift in how ESNs can be designed.
  • Overfitting Mitigation: By projecting the reservoir state to a lower dimension, the number of effective features for the output layer is reduced. This directly addresses the overfitting problem, especially when the number of training samples is limited relative to the reservoir size.
  • Improved Prediction with Large Reservoirs: The method enables the use of very large reservoirs (thousands of neurons) without the typical degradation in prediction quality due to overfitting. This allows ESNs to capture more complex and subtle temporal patterns.
  • Online Learning Capability: The proposed approach retains the online learning capability of ESNs, making it suitable for real-time applications where data arrives sequentially.

Key Results

  • Lower MSE with Random Projections: When using random projections, the Mean Squared Error (MSE) for time series prediction was consistently lower compared to using an ESN alone, especially when the number of parameters to learn was kept constant.
  • Performance Gains with Increased Reservoir Size: With random projections, increasing the number of hidden neurons in the reservoir did not degrade performance and often led to better results, indicating that overfitting was effectively avoided.
  • Competitive with Feature Selection Methods: The performance of the proposed method was compared to Least Angle Regression (LAR), an algorithm known for feature selection. The ESN with random projections, trained with Recursive Least Squares (RLS), showed comparable or better results, suggesting the projection acts as a form of implicit feature selection.
  • Robustness to Parameter Choice: The paper demonstrates that the projection dimension can be chosen to be significantly smaller than the reservoir size, leading to a substantial reduction in the number of learnable parameters without a significant loss in prediction quality.

Limitations and Future Work

  • Choice of Projection Dimension: While the paper shows the benefits of random projections, the optimal choice of the projection dimension (p) is not explicitly detailed beyond its relation to the number of parameters. Future work could explore more systematic methods for selecting this dimension.
  • Randomness in Projection: The random nature of the projection matrix means that results can vary slightly between runs. While the paper averages over multiple trials to mitigate this, further investigation into deterministic or more structured projection methods could be beneficial.
  • Generalizability to Other Time Series: The experiments were conducted on a specific NARMA time series. While promising, testing the method on a wider range of real-world time series (e.g., financial, environmental) would be valuable to assess its broader applicability.
  • Theoretical Guarantees: While the paper relies on the theoretical guarantees of random projections, exploring the specific impact of the ESN's internal dynamics on the effectiveness of these projections could be an area for future research.

Summary updated

Abstract

Modelling time series is quite a difficult task. The last recent years, reservoir computing approaches have been proven very efficient for such problems. Indeed, thanks to recurrence in the connections between neurons, this approach is a powerful tool to catch and model time dependencies between samples. Yet, the prediction quality often depends on the trade-off between the number of neurons in the reservoir and the amount of training data. Supposedly, the larger the number of neurons, the richer the reservoir of dynamics. However, the risk of overfitting problem appears. Conversely, the lower the number of neurons is, the lower the risk of overfitting problem is but also the poorer the reservoir of dynamics is. We consider here the combination of an echo state network with a projection method to benefit from the advantages of the reservoir computing approach without needing to pay attention to overfitting problems due to a lack of training data.
BibTeX
@inproceedings{daubigney2013random,
  author = {Lucie Daubigney and Matthieu Geist and Olivier Pietquin},
  title = {{Random projections: A remedy for overfitting issues in time series prediction with echo state networks}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2013},
  url = {https://openreview.net/forum?id=tQ-tHeJtta},
}
Figure