Time Series Forecasting Using Manifold Learning

Panagiotis Papaioannou, Ronen Talmon, Daniela di Serafino, Constantinos I. Siettos

TL;DR

This paper introduces a novel three-tier framework for forecasting high-dimensional time series by leveraging manifold learning. The approach embeds time series into a lower-dimensional manifold, forecasts within this reduced space, and then lifts the predictions back to the original dimension, outperforming traditional methods on synthetic and real-world financial data.

What Problem Does This Paper Address?

Forecasting time series, especially those with many features (high-dimensional), is notoriously difficult due to the "curse of dimensionality." This phenomenon means that as the number of features increases, the amount of data needed to accurately model the system grows exponentially, making it impractical with limited real-world datasets. Traditional methods struggle to capture the underlying dynamics when faced with this high dimensionality. This paper aims to tackle this by finding a lower-dimensional representation (a manifold) that captures the essential dynamics of the time series, allowing for more efficient and accurate forecasting.

Methods

The proposed framework is a three-tier process:

  1. Embedding: The high-dimensional time series is first reduced to a lower-dimensional manifold. The paper explores two nonlinear manifold learning techniques for this:
  2. Locally Linear Embedding (LLE): This method assumes that each data point can be reconstructed as a linear combination of its neighbors and preserves these reconstruction weights in the lower-dimensional space.
  3. Diffusion Maps (DMs): This technique constructs a graph based on a kernel function and then uses a random walk process to reveal the underlying geometry of the data, mapping it to a lower-dimensional space defined by the eigenvectors of a transition matrix.
  4. Forecasting on the Manifold: Once embedded, reduced-order regression models are trained on the low-dimensional representation to forecast the dynamics. The paper investigates:
  5. Multivariate Autoregressive (MVAR) models: These models predict future values based on past values of multiple time series.
  6. Gaussian Process Regression (GPR): This probabilistic model provides predictions with associated uncertainty estimates.
  7. Lifting: The forecasted low-dimensional time series are then mapped back to the original high-dimensional space. Two methods are used for this reconstruction:
  8. Radial Basis Functions (RBFs) interpolation: This method uses a set of basis functions to interpolate between known points.
  9. Geometric Harmonics (GHs): This technique extends the embedding of new points on the manifold, effectively providing an inverse mapping.

The paper also compares these approaches against standard methods like Principal Component Analysis (PCA), a naive random walk model, and MVAR/GPR models trained directly in the high-dimensional space.

Datasets

The paper uses the following datasets for training and evaluation:

  • Synthetic Time Series: Three sets of synthetic data were generated to mimic EEG signals. These were produced by:
  • Linear five-dimensional discrete stochastic models with white noise.
  • A nonlinear five-dimensional discrete stochastic model with white noise.
  • A linear stochastic model with a model order greater than one.
  • Real-World Data: Daily spot closing prices for 10 USD-based FOREX pairs (EUR/USD, GBP/USD, AUD/USD, NZD/USD, JPY/USD, CAD/USD, CHF/USD, SEK/USD, NOK/USD, DKK/USD) were downloaded from investing.com. This dataset spans from September 3, 2001, to October 29, 2020, and includes periods of significant market crashes. The data was pre-processed into compounded daily returns and "carry-adjusted" returns to account for interest rate differentials.

Key Insights and Ideas

  • Manifold Learning for Forecasting: The core idea is that complex, high-dimensional time series often evolve on a lower-dimensional "manifold." By finding and utilizing this manifold, we can bypass the curse of dimensionality and achieve better forecasting.
  • "Embed-Forecast-Lift" Paradigm: This three-step process (embedding, forecasting on the manifold, and lifting back) offers a structured way to handle high-dimensional time series forecasting.
  • GHs as Superior Lifting: Geometric Harmonics consistently outperformed RBF interpolation for reconstructing the high-dimensional predictions, suggesting it's a more robust lifting operator in this context.
  • DM and LLE Effectiveness: Both Diffusion Maps and Locally Linear Embedding proved effective for dimensionality reduction, with Diffusion Maps often showing a slight edge in certain scenarios.
  • Forecasting vs. Interpolation: The paper emphasizes that this work focuses on forecasting (out-of-sample extrapolation), which is a more challenging task than interpolation (predicting within the observed data range).

Key Results

  • Synthetic Data: For synthetic time series, the proposed "embed-forecast-lift" scheme, particularly with Diffusion Maps for embedding and Geometric Harmonics for lifting, reconstructed predictions with very high accuracy, nearly matching those from models trained directly in the high-dimensional space.
  • FOREX Trading: In the FOREX trading application, the proposed scheme (using DMs/LLE for embedding, MVAR/GPR for prediction, and GHs for lifting) significantly outperformed naive random walk, PCA-based methods, and even full MVAR/GPR models trained directly in the original space, as measured by the Sharpe Ratio. The best performance was achieved with LLE and MVAR within a 100-day training window.
  • RBF Limitations: Lifting with RBF interpolation, especially when using LU decomposition for solving the linear system, often resulted in poor reconstructions and wide forecasting intervals.

Limitations and Future Work

  • Manifold Assumption: The framework relies on the assumption that the data lies on a "smooth enough" low-dimensional manifold. This assumption might not always hold for real-world data, especially financial data which can be non-stationary.
  • Non-stationarity: The paper acknowledges that non-stationary dynamics pose a significant challenge for forecasting and that some methods (like GPR with Gaussian kernels) require stationarity.
  • Data Density: Kernel-based manifold learning methods can be sensitive to data density and the presence of strong stochasticity.
  • RBF Solver Issues: The paper notes potential numerical instabilities with RBF interpolation, particularly with LU decomposition for ill-conditioned matrices.
  • Future Work: The authors suggest exploring alternative lifting methods (e.g., Moore-Penrose pseudoinverse for RBFs, GMRES with singular preconditioners), investigating other embedding techniques like autoencoders, reservoir computing, and LSTMs, and developing methods to handle non-stationarity and data sparsity more effectively. They also plan to explore techniques for graph metric adjustment and outlier removal.

Summary updated

Abstract

We address a three-tier numerical framework based on manifold learning for the forecasting of high-dimensional time series. At the first step, we embed the time series into a reduced low-dimensional space using a nonlinear manifold learning algorithm such as Locally Linear Embedding and Diffusion Maps. At the second step, we construct reduced-order regression models on the manifold, in particular Multivariate Autoregressive (MVAR) and Gaussian Process Regression (GPR) models, to forecast the embedded dynamics. At the final step, we lift the embedded time series back to the original high-dimensional space using Radial Basis Functions interpolation and Geometric Harmonics. For our illustrations, we test the forecasting performance of the proposed numerical scheme with four sets of time series: three synthetic stochastic ones resembling EEG signals produced from linear and nonlinear stochastic models with different model orders, and one real-world data set containing daily time series of 10 key foreign exchange rates (FOREX) spanning the time period 03/09/2001-29/10/2020. The forecasting performance of the proposed numerical scheme is assessed using the combinations of manifold learning, modelling and lifting approaches. We also provide a comparison with the Principal Component Analysis algorithm as well as with the naive random walk model and the MVAR and GPR models trained and implemented directly in the high-dimensional space.
BibTeX
@inproceedings{papaioannou2022time,
  author = {Panagiotis Papaioannou and Ronen Talmon and Daniela di Serafino and Constantinos I. Siettos},
  title = {{Time Series Forecasting Using Manifold Learning}},
  booktitle = {Chaos},
  year = {2022},
  doi = {10.1063/5.0094887},
}
Figure