Manifold-based time series forecasting

Nikita Puchkin, Aleksandr Timofeev, Vladimir SpokoinyarXiv 2020 · DOI · arXiv

Abstract

Prediction for high dimensional time series is a challenging task due to the curse of dimensionality problem. Classical parametric models like ARIMA or VAR require strong modeling assumptions and time stationarity and are often overparametrized. This paper offers a new flexible approach using recent ideas of manifold learning. The considered model includes linear models such as the central subspace model and ARIMA as particular cases. The proposed procedure combines manifold denoising techniques with a simple nonparametric prediction by local averaging. The resulting procedure demonstrates a very reasonable performance for real-life econometric time series. We also provide a theoretical justification of the manifold estimation procedure.

Introduction

In modern econometrics and data science, predicting the future behavior of high-dimensional systems is a critical yet daunting task. When dealing with dozens of interacting variables, researchers frequently encounter the curse of dimensionality: as the number of variables increases, the volume of the space grows so rapidly that the available data becomes sparse, making it difficult to build reliable models.

Traditional workhorses like ARIMA (AutoRegressive Integrated Moving Average) and VAR (Vector AutoRegression) often fall short in these settings. They typically rely on rigid assumptions of stationarity and linearity. Furthermore, in high-dimensional contexts, these models become overparameterized, leading them to fit the noise in the data rather than the underlying signal. A recent study by Nikita Puchkin, Aleksandr Timofeev, and Vladimir Spokoiny proposes a solution by leveraging the geometric structure of data through manifold learning.

The Manifold Hypothesis in Time Series

The core premise of this research is the manifold hypothesis: the idea that high-dimensional data actually lives on or near a much simpler, low-dimensional "surface" or manifold. By identifying this underlying structure, one can effectively denoise the data and reduce the complexity of the forecasting problem without losing essential information.

Methodology

The authors propose a three-stage framework for forecasting:

  1. Patching (Temporal Embedding): The time series is first transformed into a collection of high-dimensional vectors, or "patches," using a sliding window technique. This step ensures that the local temporal structure of the data is preserved within the geometric representation.
  2. Manifold Denoising: This is the central innovation of the paper. Rather than assuming a global linear structure (as in Principal Component Analysis), the method assumes the data lies near a smooth, curved manifold. The authors utilize two sophisticated algorithms for this:
  3. SAME (Structure-Adaptive Manifold Estimation): An iterative approach that estimates local tangent spaces to project noisy points back onto the manifold.
  4. LDMM (Low Dimensional Manifold Model): A method that utilizes split Bregman iterations to find a local minimum for the manifold structure.
  5. Weighted k-NN Prediction: Once the data is denoised and projected onto the manifold, a non-parametric prediction is made using a weighted k-Nearest Neighbors rule. The weights are determined by the distance in the manifold space and include a "time discount" factor to prioritize more recent observations.

Theoretical Foundations

Beyond empirical application, the paper provides a rigorous theoretical dive. The authors establish non-asymptotic upper bounds for the accuracy of the manifold reconstruction. Crucially, these proofs extend to "non-mixing" time series—data that does not quickly lose its dependence on initial conditions, such as periodic or unit-root models. This makes the mathematical framework applicable to a wider range of real-world, unstable systems than traditional asymptotic theories allow.

Empirical Evaluation

The researchers tested their approach using a real-world econometric dataset from the Russian Presidential Academy of National Economy and Public Administration. The data tracked various indicators of living standards in Russia (e.g., real disposable income, accrued wages) from 1999 to 2018. This dataset is particularly challenging due to strong internal correlations, seasonality, and sudden "leaps" or economic shocks.

Key Findings: * The Necessity of Denoising: In all experiments, the manifold-based methods (SAME and LDMM) significantly outperformed raw k-NN. This confirms that projecting data onto the manifold is essential for filtering out high-dimensional noise. * Responsiveness to Shocks: While ARIMA models performed well on stable data, they often struggled with sudden shifts, tending to "revert to the mean." The manifold-based methods were notably more responsive to sudden "leaps" in the economic indicators, which is a vital trait for real-world decision-making. * Competitive Accuracy: The proposed methods achieved Root Mean Square Error (RMSE) levels that were comparable to, and often better than, optimized ARIMA models, despite requiring fewer rigid assumptions about the data's distribution.

Conclusion and Future Directions

This research bridges the gap between high-level differential geometry and practical econometrics. By treating high-dimensional time series as points on a low-dimensional manifold, the authors provide a robust tool for forecasting in noisy, complex environments.

While the method currently requires careful tuning of hyperparameters (such as the manifold dimension $d$ and the number of neighbors $k$), it represents a significant step toward more flexible, data-driven forecasting. Future work may focus on automating the estimation of the manifold's intrinsic dimension, further reducing the manual oversight required for these powerful models.

Summary updated

Abstract

Prediction for high dimensional time series is a challenging task due to the curse of dimensionality problem. Classical parametric models like ARIMA or VAR require strong modeling assumptions and time stationarity and are often overparametrized. This paper offers a new flexible approach using recent ideas of manifold learning. The considered model includes linear models such as the central subspace model and ARIMA as particular cases. The proposed procedure combines manifold denoising techniques with a simple nonparametric prediction by local averaging. The resulting procedure demonstrates a very reasonable performance for real-life econometric time series. We also provide a theoretical justification of the manifold estimation procedure.
BibTeX
@misc{puchkin2020manifold,
  author = {Nikita Puchkin and Aleksandr Timofeev and Vladimir Spokoiny},
  title = {{Manifold-based time series forecasting}},
  year = {2020},
  eprint = {2012.08244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.08244},
}
Figure