Abstract
Non-stationary time series present a significant challenge for machine learning due to evolving data distributions, often referred to as concept drift. We present the Mixture of Online and Offline Experts (MOOE) framework, which adaptively integrates historical offline knowledge with real-time online learning. By leveraging a library of experts trained on past distributions, MOOE mitigates the "cold-start" problem in non-stationary environments. Theoretical analysis demonstrates that MOOE achieves a regret bound of $O(\sqrt{T \ln K})$, ensuring scalability and robust performance. Empirical evaluations across diverse real-world datasets show that MOOE consistently outperforms state-of-the-art methods, particularly in scenarios with recurring concept drift.
Introduction
Traditional machine learning models often rely on the assumption of stationarity, where the underlying data distribution remains constant. However, real-world applications such as financial forecasting, sensor monitoring, and network security frequently encounter non-stationary data. This phenomenon, known as concept drift, renders static offline models obsolete and poses a "scarcity of labels" challenge for pure online learning models during the initial phases of a distribution shift. MOOE addresses this by identifying and reusing relevant historical knowledge, bridging the gap between static offline learning and adaptive online learning.
The MOOE Framework
The MOOE architecture consists of three core components: Offline Experts, an Online Expert, and a Meta-Expert. Offline experts are static models derived from previously completed time intervals. The Online Expert is a dynamic model updated via Online Gradient Descent (OGD) to capture the most recent trends. The Meta-Expert acts as an orchestrator, employing an exponentially weighted average forecaster to assign weights to each expert based on their predictive accuracy.
A key innovation is the use of regularized knowledge transfer when transitioning online intervals into new offline experts. This ensures the model retains broad historical relevance while adapting to new data, effectively transferring knowledge across time intervals.
Experimental Setup
The framework was evaluated on a comprehensive suite of benchmarks, including synthetic 2D Gaussian distributions with shifting means and real-world datasets like Usenet, GasSensor, Electricity, and Kitsune. These datasets cover various types of drift, including abrupt, gradual, and recurring concepts. For instance, the Kitsune dataset focuses on network intrusion detection, while WESAD addresses wearable stress detection.
Theoretical Analysis
MOOE provides strong theoretical guarantees. The regret—the cumulative loss difference between the model and the best-performing expert—is bounded by $O(\sqrt{T \ln K})$. This logarithmic dependence on the number of experts ($K$) allows the system to scale effectively without significant performance degradation. Furthermore, the framework achieves a convergence rate of $O(1/\sqrt{T})$, matching the performance of stationary learning models even in non-stationary environments. The meta-expert's weighting mechanism specifically addresses "expert dilution" by exponentially penalizing experts with high cumulative loss.
Results and Discussion
MOOE demonstrates superior accuracy across all tested datasets. Notably, on the Kitsune network intrusion dataset, MOOE achieved 93.4% accuracy, a substantial improvement over competing methods like NSE and DTEL. In scenarios with recurring drift, such as email spam filtering, MOOE showed an 11.8% accuracy gain by effectively re-activating relevant historical experts.
As shown in the comparative analysis, MOOE significantly reduces both loss and regret compared to standard Online Gradient Descent (OGD). This is particularly evident in the early stages of new data windows, where MOOE leverages relevant offline experts to maintain high performance while OGD is still adapting.
Conclusion and Future Work
While MOOE offers a robust solution for non-stationary time series, it introduces computational overhead due to the management of multiple experts. Performance is also somewhat sensitive to hyperparameters like the maximum number of experts ($K_{max}$) and interval size ($B$). Future research will focus on automated hyperparameter tuning and advanced expert pruning techniques to optimize performance on resource-constrained edge devices.
Summary updated