Introduction
In modern industrial environments, servers generate a continuous stream of monitoring data, including metrics and logs, to track service availability. Traditional machine learning models often fail in these dynamic settings due to two primary challenges: the Stability-Plasticity Dilemma and Concept Drift. A stable model retains historical knowledge but fails to adapt to new patterns, while a plastic model adapts quickly but risks "catastrophic forgetting." Furthermore, real-world data is non-static; service patterns change due to maintenance or seasonal usage, rendering static models obsolete. This research proposes a lightweight, incremental learning framework capable of processing infinite data streams with finite resources.
Methodology: The Learn++ Framework
The proposed solution utilizes the Learn++ algorithm, an ensemble-based meta-learning approach designed for incremental learning. Unlike batch models that require full retraining, Learn++ incorporates new data "chunks" as they arrive.
The framework employs an ensemble of K-Nearest Neighbor (KNN) weak learners. KNN was selected for its simplicity, minimal hyperparameter tuning, and efficiency in handling noisy, high-volume data. The process involves:
- Partial-Fitting: The model predicts outcomes for new data chunks and updates its internal weights based on ground truth.
- Weighting Strategy: Misclassified instances receive higher weights in subsequent training rounds, forcing the ensemble to focus on difficult cases.
- Dimensionality Reduction: Using the Total Explained Variance Ratio (TEVR), the authors reduced the feature set to the top 75 features, preserving over 99% of the information while minimizing computational overhead.
Dataset Characteristics
The study evaluated the framework using a real-world dataset from Ericsson Canada, consisting of six labeled time-series "chunks" collected over 39 days. Each chunk represents one week of monitoring data, including both "Counter" (monotonically increasing) and "Gauge" (fluctuating) metrics.
As shown in the table above, the dataset exhibits varying characteristics across chunks. While most are relatively balanced, Chunk #2 presents a significant 1:3 imbalance between service "UP" and "DOWN" states, testing the model's robustness to class distribution shifts.
Key Results and Performance Analysis
The model's performance was measured using F1-score, Area Under the Curve (AUC), and False Negative Rate (FNR).
- Stable Performance: For the first four adaptive chunks, the model achieved high accuracy, with F1-scores between 0.9665 and 0.9986 and AUCs consistently above 0.90.
- Impact of Sudden Drift: A significant performance drop occurred at Chunk #5, where the F1-score fell to 0.6464 and the FNR rose to 0.6801. This was identified as a case of sudden concept drift, where abrupt changes in system behavior or extreme outliers temporarily overwhelmed the adaptive mechanism.
Conclusion and Future Directions
The research demonstrates that incremental learning via Learn++ and KNN is a viable path for long-term industrial monitoring. However, the struggle with sudden drift in the final data chunk highlights a key limitation: the current model does not explicitly distinguish between permanent concept drift and temporary noise.
Future work will focus on:
- Outlier Detection: Integrating clustering-based methods to filter anomalous data points.
- Drift Categorization: Developing mechanisms to identify and respond differently to gradual versus sudden drift.
- Topology Optimization: Refining the interaction between Learn++ and KNN to accelerate recovery after drift events.
Summary updated