Second Order Techniques for Learning Time-series with Structural Breaks

Takayuki OsogamiAAAI 2021 · OpenReview

Abstract

We study fundamental problems in learning nonstationary time-series: how to effectively regularize time-series models and how to adaptively tune forgetting rates. The effectiveness of L2 regularization depends on the choice of coordinates, and the variables need to be appropriately normalized. In nonstationary environment, however, what is appropriate can vary over time. Proposed regularization is invariant to the invertible linear transformation of coordinates, eliminating the necessity of normalization. We also propose an ensemble learning approach to adaptively tuning the forgetting rate and regularization-coefficient. We train multiple models with varying hyperparameters and evaluate their performance by the use of multiple hyper forgetting rates. At each step, we choose the best performing model on the basis of the best performing hyper forgetting rate. The effectiveness of the proposed approaches is demonstrated with real time-series.

Introduction

Learning from nonstationary time-series data presents significant challenges, particularly when the underlying data-generating process undergoes sudden changes known as structural breaks. In such environments, traditional models often fail because their learned parameters become obsolete. Common strategies involve using "forgetting rates" to prioritize recent data, but this introduces two critical problems: the difficulty of adaptively tuning these rates in real-time and the sensitivity of standard $L_2$ regularization to coordinate scaling, which can fluctuate in nonstationary settings.

Methodology

This research introduces two primary innovations within a Recursive Least Squares (RLS) framework to address these challenges.

Transformation-Invariant Regularization

The authors propose a novel regularization technique based on the Hessian of the loss function. Unlike standard $L_2$ regularization, which is sensitive to how data is scaled, this approach is invariant to any invertible linear transformation of the coordinates. This eliminates the need for manual normalization as the data distribution shifts, allowing the optimization to handle coordinate changes automatically.

Adaptive Hyperparameter Tuning via Ensemble Learning

To dynamically tune the forgetting rate ($\gamma$) and regularization coefficient ($\lambda$), the paper employs an ensemble-based "Follow the Leader" strategy.

Hierarchical structure of the proposed ensemble learning approach, showing the relationship between hyper forgetting rates, loss functions, and specific model weights.

As illustrated in Figure 1, the system maintains an ensemble of models trained with various hyperparameter combinations. Their performance is evaluated using a set of "hyper forgetting rates" ($\eta$), which track model accuracy over different time horizons. By selecting the best-performing hyper forgetting rate, the system adaptively determines whether to trust a model that has been stable for a long time or one that has quickly adapted to a recent structural break.

Experimental Results

The proposed framework was evaluated on both synthetic datasets with engineered change points and real-world financial indices from 2008 to 2018, including the S&P 500 (SPX), Nikkei 225, and DAX.

Performance on Financial Data

The method demonstrated significant improvements in predictive accuracy over popular first-order baselines such as Adam, RMSProp, and vSGD.

Relative Mean Squared Error (MSE) on the SPX dataset. The proposed method consistently achieves lower error across various model orders compared to established baselines.

On the S&P 500 (SPX) dataset, the proposed approach achieved a 22% lower Mean Squared Error (MSE) than the best-performing baseline. As shown in Figure 6, the performance advantage remains consistent across different model orders, highlighting the robustness of the second-order techniques and the adaptive tuning mechanism.

Discussion and Limitations

The key insight of this work is that tying regularization to the curvature (Hessian) of the loss surface makes models naturally resilient to coordinate shifts. Furthermore, moving the tuning problem to a "hyper" level allows for principled adaptation to structural breaks.

However, the $O(n^2)$ computational complexity per step limits the current application to medium-sized models with approximately 1,000 features. While highly effective for industrial and financial time-series, further research is needed to scale these second-order techniques to massive deep learning architectures.

Summary updated

Abstract

We study fundamental problems in learning nonstationary time-series: how to effectively regularize time-series models and how to adaptively tune forgetting rates. The effectiveness of L2 regularization depends on the choice of coordinates, and the variables need to be appropriately normalized. In nonstationary environment, however, what is appropriate can vary over time. Proposed regularization is invariant to the invertible linear transformation of coordinates, eliminating the necessity of normalization. We also propose an ensemble learning approach to adaptively tuning the forgetting rate and regularization-coefficient. We train multiple models with varying hyperparameters and evaluate their performance by the use of multiple hyper forgetting rates. At each step, we choose the best performing model on the basis of the best performing hyper forgetting rate. The effectiveness of the proposed approaches is demonstrated with real time-series.
BibTeX
@inproceedings{osogami2021second,
  author = {Takayuki Osogami},
  title = {{Second Order Techniques for Learning Time-series with Structural Breaks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=ckUXa4bCcI},
}
Figure