Abstract
Concept drift—the evolution of statistical properties over time—poses a significant challenge to time series forecasting. Traditional models often struggle to adapt to these shifts, either lagging behind long-term trends or failing to capture short-term anomalies. This paper introduces LEAF (Learning to Extrapolate and Adjust for Forecasting), a model-agnostic meta-learning framework. LEAF decouples drift into macro-drift (long-term shifts) and micro-drift (sample-specific fluctuations). By operating in a low-dimensional latent space, LEAF employs an extrapolation module to predict future model parameters and a meta-learned surrogate loss for real-time sample adjustment. Empirical evaluations across diverse benchmarks, including new real-world electric load datasets, demonstrate that LEAF significantly reduces forecasting error and enhances the adaptability of various neural architectures.
Introduction
In dynamic environments like financial markets or energy grids, time series data rarely remains stationary. This phenomenon, known as concept drift, renders static forecasting models obsolete as the underlying data-generating processes change. While existing online learning methods attempt to mitigate drift through frequent retraining or reactive updates, they often suffer from "lag" during steady transitions or fail to account for idiosyncratic, sample-level variations.
The LEAF Framework: Decoupling Drift
The core innovation of the LEAF framework is the conceptual separation of concept drift into two distinct scales:
- Macro-Drift: Stable, inter-period changes that reflect long-term trends or seasonal shifts.
- Micro-Drift: Erratic, intra-period fluctuations that are specific to individual data samples.
To handle these efficiently, LEAF maps model parameters into a low-dimensional latent space. This avoids the computational burden of updating entire high-dimensional parameter sets (like those in Transformers) while maintaining the flexibility to adapt.
Methodology: Extrapolation and Adjustment
LEAF operates through a two-stage meta-learning process:
1. Extrapolation Stage (Macro-Drift)
Instead of simply adopting the most recent model, LEAF "forecasts the forecaster." An extrapolation network (typically an LSTM) analyzes the trajectory of optimal latent embeddings ($H^*$) from previous periods to predict the embedding ($H_p$) for the upcoming period. This proactive approach allows the model to stay ahead of steady macro-drifts.
2. Adjustment Stage (Micro-Drift)
To address sample-specific micro-drifts, LEAF utilizes a meta-learned surrogate loss ($L^{surrogate}$). During inference, the model performs a quick gradient descent step on the predicted latent embedding for each specific input sample. This "nudges" the model parameters to better fit the immediate context before the final prediction is made.
The entire system is optimized in an "outer loop" to minimize forecasting error on a meta-test set, ensuring that both the extrapolation and adjustment modules learn to facilitate effective adaptation.
Experimental Evaluation
The authors evaluated LEAF against state-of-the-art (SOTA) methods using standard benchmarks (ETT, ECL) and three new Electric Load datasets characterized by aggressive drift.
Key Results
- Performance Gains: LEAF achieved an average 14.4% reduction in error (MSE/MAE) compared to SOTA online learning methods like FSNet and SOLID.
- Model Agnosticism: The framework provided consistent improvements across various architectures, including MLPs, RNNs, and Transformers. Notably, standard LSTMs saw a 35.4% decrease in MSE when wrapped in LEAF.
- Ablation Studies: Removing either the extrapolation or adjustment module resulted in significant performance degradation, confirming the necessity of addressing both macro and micro scales of drift.
Conclusion
LEAF represents a shift from reactive model updating to proactive model forecasting. By decoupling drift and leveraging latent space meta-learning, it provides a robust and efficient solution for online time series forecasting in non-stationary environments. Future work may explore combining LEAF with continual learning to prevent catastrophic forgetting over extended horizons.
Summary updated