Adaptive Learning for Service Monitoring Data

Farzana Anowar, Samira Sadaoui, Hardik DalalarXiv 2022 · DOI · arXiv

Introduction

In modern industrial environments, servers generate a continuous stream of monitoring data, including metrics and logs, to track service availability. Traditional machine learning models often fail in these dynamic settings due to two primary challenges: the Stability-Plasticity Dilemma and Concept Drift. A stable model retains historical knowledge but fails to adapt to new patterns, while a plastic model adapts quickly but risks "catastrophic forgetting." Furthermore, real-world data is non-static; service patterns change due to maintenance or seasonal usage, rendering static models obsolete. This research proposes a lightweight, incremental learning framework capable of processing infinite data streams with finite resources.

Methodology: The Learn++ Framework

The proposed solution utilizes the Learn++ algorithm, an ensemble-based meta-learning approach designed for incremental learning. Unlike batch models that require full retraining, Learn++ incorporates new data "chunks" as they arrive.

Flowchart of the Learn++ incremental classifier system.

The framework employs an ensemble of K-Nearest Neighbor (KNN) weak learners. KNN was selected for its simplicity, minimal hyperparameter tuning, and efficiency in handling noisy, high-volume data. The process involves:

  1. Partial-Fitting: The model predicts outcomes for new data chunks and updates its internal weights based on ground truth.
  2. Weighting Strategy: Misclassified instances receive higher weights in subsequent training rounds, forcing the ensemble to focus on difficult cases.
  3. Dimensionality Reduction: Using the Total Explained Variance Ratio (TEVR), the authors reduced the feature set to the top 75 features, preserving over 99% of the information while minimizing computational overhead.

Dataset Characteristics

The study evaluated the framework using a real-world dataset from Ericsson Canada, consisting of six labeled time-series "chunks" collected over 39 days. Each chunk represents one week of monitoring data, including both "Counter" (monotonically increasing) and "Gauge" (fluctuating) metrics.

Summary of dataset chunks, including size, dimensionality, and class distribution.

As shown in the table above, the dataset exhibits varying characteristics across chunks. While most are relatively balanced, Chunk #2 presents a significant 1:3 imbalance between service "UP" and "DOWN" states, testing the model's robustness to class distribution shifts.

Key Results and Performance Analysis

The model's performance was measured using F1-score, Area Under the Curve (AUC), and False Negative Rate (FNR).

  • Stable Performance: For the first four adaptive chunks, the model achieved high accuracy, with F1-scores between 0.9665 and 0.9986 and AUCs consistently above 0.90.
  • Impact of Sudden Drift: A significant performance drop occurred at Chunk #5, where the F1-score fell to 0.6464 and the FNR rose to 0.6801. This was identified as a case of sudden concept drift, where abrupt changes in system behavior or extreme outliers temporarily overwhelmed the adaptive mechanism.

Conclusion and Future Directions

The research demonstrates that incremental learning via Learn++ and KNN is a viable path for long-term industrial monitoring. However, the struggle with sudden drift in the final data chunk highlights a key limitation: the current model does not explicitly distinguish between permanent concept drift and temporary noise.

Future work will focus on:

  • Outlier Detection: Integrating clustering-based methods to filter anomalous data points.
  • Drift Categorization: Developing mechanisms to identify and respond differently to gradual versus sudden drift.
  • Topology Optimization: Refining the interaction between Learn++ and KNN to accelerate recovery after drift events.

Summary updated

Abstract

Service monitoring applications continuously produce data to monitor their availability. Hence, it is critical to classify incoming data in real-time and accurately. For this purpose, our study develops an adaptive classification approach using Learn++ that can handle evolving data distributions. This approach sequentially predicts and updates the monitoring model with new data, gradually forgets past knowledge and identifies sudden concept drift. We employ consecutive data chunks obtained from an industrial application to evaluate the performance of the predictors incrementally.
BibTeX
@misc{anowar2022adaptive,
  author = {Farzana Anowar and Samira Sadaoui and Hardik Dalal},
  title = {{Adaptive Learning for Service Monitoring Data}},
  year = {2022},
  eprint = {2208.12281},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2208.12281},
}
Figure