Abstract
In this work, we study the problem of aggregating a finite number of predictors for nonstationary sub-linear processes. We provide oracle inequalities relying essentially on three ingredients: (1) a uniform bound of the $\ell^{1}$ norm of the time varying sub-linear coefficients, (2) a Lipschitz assumption on the predictors and (3) moment conditions on the noise appearing in the linear representation. Two kinds of aggregations are considered giving rise to different moment conditions on the noise and more or less sharp oracle inequalities. We apply this approach for deriving an adaptive predictor for locally stationary time varying autoregressive (TVAR) processes. It is obtained by aggregating a finite number of well chosen predictors, each of them enjoying an optimal minimax convergence rate under specific smoothness conditions on the TVAR coefficients. We show that the obtained aggregated predictor achieves a minimax rate while adapting to the unknown smoothness. To prove this result, a lower bound is established for the minimax rate of the prediction risk for the TVAR process. Numerical experiments complete this study. An important feature of this approach is that the aggregated predictor can be computed recursively and is thus applicable in an online prediction context.
1. Introduction
In time series analysis, the assumption of stationarity—where statistical properties remain constant over time—is often treated as the gold standard. However, real-world data from high-frequency trading to sensor networks frequently violate this assumption. Predicting such nonstationary processes requires a delicate balancing act: if the environment changes rapidly, the model must "forget" the past quickly; if it changes slowly, it should utilize as much historical data as possible to minimize noise.
In their 2015 study, Giraud, Roueff, and Sanchez-Perez address this challenge by providing a robust framework for aggregating multiple predictors. Their method focuses on Time-Varying Autoregressive (TVAR) models and sub-linear processes, offering an online, recursive algorithm that adapts to unknown changes in the underlying system dynamics.
2. The Challenge of Unknown Smoothness
A central problem in nonstationary forecasting is that the "smoothness" (the rate at which parameters change) is rarely known beforehand. Standard methods like Least Mean Squares (LMS) or Kalman filters require tuning parameters, such as learning rates or forgetting factors, that depend on this unknown smoothness. If these parameters are misaligned with the actual rate of change, prediction accuracy suffers significantly. This paper proposes a solution that automatically adapts to the unknown smoothness of a process in a computationally efficient, online manner.
3. Methodology: Exponentially Weighted Aggregation
The authors utilize Exponentially Weighted Aggregation (EWA) to manage a "committee" of $N$ different predictors. Instead of attempting to identify a single perfect model, the framework maintains a library of predictors (e.g., several TVAR estimators tuned with different learning rates) and assigns weights to them based on their performance.
3.1 Aggregation Strategies
Two primary strategies for weight assignment ($\alpha$) are explored:
- Strategy 1 (Gradient-based): Updates weights based on the gradient of the quadratic loss. This strategy is designed to compete with the best convex combination of all available predictors.
- Strategy 2 (Loss-based): Updates weights based directly on the cumulative quadratic loss, aiming to compete with the single best individual predictor in the library.
3.2 Computational Efficiency
Crucially, the weights are updated at each time step $t$ using only the previous observation and the previous weights. This recursive structure results in a memory requirement of $O(dN)$ and a computational complexity of $O(dNT)$, making the approach highly feasible for high-frequency data streams.
4. Theoretical Guarantees and Minimax Optimality
The authors establish "oracle inequalities," which are mathematical guarantees that the risk of the aggregated predictor is bounded by the risk of the best expert in the pool, plus a small penalty term that vanishes as more data is processed.
Key theoretical insights include:
- Adaptivity: By aggregating predictors tuned for different smoothness indices ($\beta$), the aggregate automatically achieves the optimal rate without prior knowledge of $\beta$.
- Minimax Optimality: The authors establish a lower bound for the prediction risk of TVAR processes and demonstrate that their aggregated predictor achieves this minimax rate.
- Regret Bounds: Under Strategy 2, if the noise has an exponential moment, the "regret" (the additional error compared to the best expert) is bounded by $O(\frac{\log N}{T \eta} + \frac{\log N}{T})$.
5. Numerical Results
The framework was evaluated using synthetic TVAR processes (order $d=3$) where coefficients evolved smoothly over time. When compared against a suite of Normalized Least Mean Squares (NLMS) predictors with varying step sizes, the aggregated predictor using Strategy 1 consistently outperformed even the best single NLMS predictor. This confirms the power of the convex aggregation approach, as it effectively mimics the best possible weighted average of experts.
6. Conclusion and Future Work
This research demonstrates that adaptive online forecasting for nonstationary processes is not only possible but can achieve theoretical optimality. While the performance depends on the presence of at least one high-quality predictor in the initial pool, the recursive nature of the EWA framework provides a scalable solution for complex time series. Future extensions may involve adapting to models where the order of the autoregression ($d$) is also unknown and time-varying.
Summary updated