Predicting future dynamics from short-term time series using an Anticipated Learning Machine

Chuan Chen, Rui Li, Lin Shu, Zhiyu He, Jining Wang, Chengming Zhang, Huanfei Ma, Kazuyuki Aihara, Luonan ChenNational 2020 · DOI

TL;DR

This paper introduces the Anticipated Learning Machine (ALM), a novel approach for predicting future time series dynamics using short-term, high-dimensional data. ALM leverages non-linear dynamical systems theory to transform spatial correlations within high-dimensional variables into temporal predictions for a target variable, effectively overcoming the "small-sample problem" and outperforming 12 existing methods on various real-world datasets.

What Problem Does This Paper Address?

Accurate time series prediction is crucial across many scientific and engineering fields. However, a significant challenge arises when dealing with short-term time series data, which often lacks sufficient historical information for traditional methods to make reliable predictions. This problem is exacerbated when the data is also high-dimensional, meaning it comprises a large number of variables measured simultaneously. While high-dimensional data theoretically contains rich information, effectively extracting and utilizing this information for prediction, especially with limited historical data, remains a difficult task. Existing methods often struggle with either short time series or high dimensionality, but rarely both simultaneously.

Methods

The core of the proposed solution is the Anticipated Learning Machine (ALM), which is built upon principles from non-linear dynamical systems theory and delay embedding theory. ALM's key innovation lies in its ability to transform spatial information (correlations among high-dimensional variables at a given time point) into temporal information (future dynamics of a target variable).

Here's a breakdown of the methodology:

  • Delay Embedding Theory: This theory suggests that a high-dimensional dynamical system can be reconstructed from a single time series by using time-delayed versions of that series. The paper extends this by considering multiple variables.
  • Randomly Distributed Embedding (RDE): ALM utilizes RDE to create numerous "sampled nondelay attractors" from the high-dimensional input data. Each sampled attractor represents a different facet of the system's dynamics.
  • Anticipated Learning (AL): The training samples are generated such that they implicitly contain information about the unknown future values of the target variable. This is termed "anticipated learning" because the model learns from data that "looks ahead."
  • AL Neural Network: A multi-layer neural network is employed to learn the complex mapping (referred to as the Spatial-Temporal Information Transformation or STI equation) between these sampled nondelay attractors and the delay attractor of the target variable.
  • Consistent-Training Scheme: To ensure robust and accurate multistep-ahead predictions, ALM uses a "consistent-training scheme." This involves co-training multiple sub-maps simultaneously, forcing them to adhere to the underlying dynamical system's constraints and improving the capture of global system properties.
  • Dropout: The Dropout technique is used during neural network training to prevent overfitting and improve generalization.

Datasets

The paper evaluates ALM on a diverse range of datasets, demonstrating its versatility:

  • Synthetic Data Set: A 90D time-variant coupled Lorentz system was used to test ALM's ability to predict non-periodic behavior and handle noise.
  • Gene Expression Data: Affymetrix microarray data for studying circadian rhythm in rat suprachiasmatic nucleus (SCN) was used, featuring 31,099 genes and 22 time points.
  • Plankton Data Set: Data from an optical plankton counter and CTD mounted on a ScanFish platform, containing 58 attributes, representing a small ecosystem.
  • Climate Data Sets:
    • A 72-dimension ground-ozone-level data set from 1998-2004.
  • Wind speed data collected by the Japan Meteorological Agency, with 155 wind stations.
  • Stock Index Data Set: A 1130-dimension stock index data set (Shanghai Stock Exchange A Share Index) to assess performance on highly unstable systems.
  • Traffic Data Set: Traffic speed data from 207 loop detectors on a highway in Los Angeles.
  • Satellite Cloud-Image Data Set: Data recording the route of typhoon Marcus, composed of 241 cloud images with 2402 variables.

Key Insights and Ideas

  • Spatial-to-Temporal Transformation: The fundamental insight is that correlations within high-dimensional data (spatial) can be a powerful proxy for future dynamics (temporal), especially when direct temporal data is scarce. ALM formalizes this transformation.
  • Overcoming the "Small-Sample Problem": By leveraging the rich information embedded in high-dimensional data, ALM can achieve accurate predictions even with very short time series, a significant improvement over methods that rely on extensive historical data.
  • "Anticipated Learning": The training data generation process, which implicitly includes future values, is a novel way to imbue the model with predictive foresight, distinguishing it from standard supervised learning.
  • Robustness to Noise: The method of integrating information from multiple sampled nondelay attractors helps to suppress the effects of noise in the data, leading to more robust predictions.
  • Dynamics-Based Machine Learning: ALM represents a shift towards "dynamics-based machine learning," moving beyond purely statistical pattern recognition to incorporate underlying physical or biological system dynamics.

Key Results

  • Superior Performance: ALM consistently outperformed 12 existing methods (including ARIMA, VAR, LSTM, RDE, etc.) across all tested real-world datasets, as measured by composite indicators that combine numerical errors and trend accuracy.
  • Accurate Multistep-Ahead Prediction: The consistent-training scheme enabled ALM to make accurate predictions for multiple time steps into the future.
  • Effective Noise Handling: On the synthetic Lorentz system, ALM showed remarkable resilience to Gaussian noise, maintaining accurate predictions where other methods struggled.
  • Prediction of Unexperienced Dynamics: ALM demonstrated the ability to predict system behaviors that were not explicitly present in the training data, highlighting its capacity to generalize based on underlying dynamics.
  • Efficiency with Short Data: For the 90D Lorentz system, ALM achieved accurate predictions with only 30 training time points.

Limitations and Future Work

  • Sensitivity to Hyperparameters: The authors acknowledge that hyperparameter settings can still be sensitive, particularly for highly nonlinear or stochastic systems with noisy data.
  • Sudden Changes: ALM may not accurately predict sudden changes in real-world data, as these are often caused by external factors not included in the trained model.
  • Future Work: The paper suggests that further theoretical analysis and the development of more appropriate frameworks are needed to address the sensitivity to hyperparameters and the challenges posed by strong nonlinearity, stochasticity, and noisy data. The authors also point to the need for more in-depth theoretical analysis to better handle these complexities.

Summary updated

Abstract

Predicting time series has significant practical applications over different disciplines. Here, we propose an Anticipated Learning Machine (ALM) to achieve precise future-state predictions based on short-term but high-dimensional data. From non-linear dynamical systems theory, we show that ALM can transform recent correlation/spatial information of high-dimensional variables into future dynamical/temporal information of any target variable, thereby overcoming the small-sample problem and achieving multistep-ahead predictions. Since the training samples generated from high-dimensional data also include information of the unknown future values of the target variable, it is called anticipated learning. Extensive experiments on real-world data demonstrate significantly superior performances of ALM over all of the existing 12 methods. In contrast to traditional statistics-based machine learning, ALM is based on non-linear dynamics, thus opening a new way for dynamics-based machine learning.
BibTeX
@inproceedings{chen2020predicting,
  author = {Chuan Chen and Rui Li and Lin Shu and Zhiyu He and Jining Wang and Chengming Zhang and Huanfei Ma and Kazuyuki Aihara and Luonan Chen},
  title = {{Predicting future dynamics from short-term time series using an Anticipated Learning Machine}},
  booktitle = {National},
  year = {2020},
  doi = {10.1093/nsr/nwaa025},
}
Figure