Abstract
We approach the time-series forecasting problem in the presence of concept drift by automatic learning rate tuning of stochastic gradient descent (SGD). The SGD-based approach is preferable to other concept drift algorithms in that it can be applied to any model and it can keep learning efficiently whilst predicting online. Among a number of SGD algorithms, the variance-based SGD (vSGD) can successfully handle concept drift by automatic learning rate tuning, which is reduced to an adaptive mean estimation problem. However, its performance is still limited because of its heuristic mean estimator. In this paper, we present a concept-drift-aware stochastic gradient descent (Cogra), equipped with more theoretically-sound mean estimator called sequential mean tracker (SMT). Our key contribution is that we define a goodness criterion for the mean estimators; SMT is designed to be optimal according to this criterion. As a result of comprehensive experiments, we find that (i) our SMT can estimate the mean better than vSGD’s estimator in the presence of concept drift, and (ii) in terms of predictive performance, Cogra reduces the predictive loss by 16–67% for real-world datasets, indicating that SMT improves the prediction accuracy significantly.
Introduction
In real-world time-series forecasting—ranging from sensor data to financial markets—the underlying data-generating processes often shift over time, a phenomenon known as concept drift. To maintain accuracy, forecasting models must be capable of "forgetting" obsolete patterns and rapidly adapting to new regimes. While Stochastic Gradient Descent (SGD) is a standard for online learning, most variants struggle with drift. Standard SGD often uses decaying learning rates that become too small to react to sudden changes. Variance-based SGD (vSGD) attempted to automate this tuning, but its heuristic estimators often suffer from "bias persistence," where the model remains stuck in an old, stable regime long after a drift has occurred.
Methodology: The Sequential Mean Tracker
Cogra (Concept-drift-aware SGD) addresses these limitations by refining how gradient moments are estimated. The core innovation is the Sequential Mean Tracker (SMT). Unlike heuristic moving averages, SMT utilizes an "oracle" forgetting rate ($a_t$) derived to greedily minimize the Mean Squared Error (estimation risk). This allows the optimizer to adaptively decide how much historical data to ignore based on the current shift in gradient distribution.
To balance flexibility and stability, the authors propose three grouping strategies for parameter updates:
- Cogra-local: Individual learning rates for every parameter.
- Cogra-global: A single learning rate for the entire model.
- Cogra-block: Parameters are grouped into logical blocks (e.g., weights vs. biases), which provides the best empirical performance.
Experimental Results
The researchers evaluated Cogra against state-of-the-art optimizers including ADAM, AdaGrad, RMSProp, and vSGD across synthetic and real-world datasets (Activity Recognition, EEG Eye State, and Gas Sensor Arrays).
Adaptation Speed and Tracking
A critical advantage of Cogra is its response time to abrupt changes. As shown in the tracking analysis, Cogra's estimation error spikes briefly at a change point but returns to near-zero almost immediately. In contrast, vSGD remains biased for a significantly longer duration because its internal time constant becomes too large during stable periods to react quickly to new evidence.
Robustness and Performance
Cogra demonstrates remarkable robustness across various drift scenarios. Unlike traditional optimizers that require extensive hyperparameter tuning (like initial learning rates) to handle specific drift magnitudes, Cogra is largely hyperparameter-free. Synthetic tests varying bias, transition periods, and dimensions show that Cogra consistently maintains lower relative cumulative loss than its competitors.
On real-world datasets, Cogra-block reduced predictive loss by 16–67% compared to the best-performing baselines. This significant improvement stems from SMT's ability to accurately track the mean and variance of gradients during drift, allowing for near-optimal automatic learning rate tuning.
Conclusion
Cogra provides a theoretically grounded framework for online learning under concept drift. By framing learning rate tuning as an adaptive moment estimation problem and solving it with the Sequential Mean Tracker, Cogra achieves superior adaptation speeds and predictive accuracy. Future work may explore Cogra's performance in high-capacity non-linear models like Transformers or its application in Reinforcement Learning environments where targets are inherently non-stationary.
Summary updated