Autoregressive based Drift Detection Method

Mansour Zoubeirou A. Mayaki, Michel Riveill

TL;DR

This paper introduces ADDM, a novel concept drift detection method that leverages autoregressive time series models to monitor a machine learning model's error rate. ADDM significantly outperforms existing methods on both synthetic and real-world datasets, offering improved accuracy and a lower false alarm rate. The authors also propose a complementary method for adapting models based on the severity of detected drifts.

What Problem Does This Paper Address?

Machine learning models are typically trained assuming that the data distribution remains constant over time (stationarity). However, in many real-world applications, this assumption is violated as the underlying data generation process changes. This phenomenon, known as "concept drift," leads to a degradation in model performance over time. Detecting these drifts accurately and efficiently is crucial for maintaining model effectiveness, especially in streaming data scenarios where data arrives continuously and in large volumes. Existing methods often struggle with high false alarm rates or require direct access to ground-truth labels, which is not always feasible.

Methods

The core of the proposed method, ADDM (Autoregressive based Drift Detection Method), lies in modeling the error rate of a primary machine learning model as a time series. Specifically, they employ a Self-Exciting Threshold Autoregressive (SETAR) model. This model is well-suited for capturing non-linear dynamics and regime shifts, which are characteristic of changing error rates due to concept drift.

Here's a breakdown of the ADDM approach:

  1. Primary Model Training: A standard machine learning model (e.g., deep neural network, logistic regression) is trained on historical data for the main prediction task.
  2. Error Rate Monitoring: As new data arrives, the primary model makes predictions, and the prediction errors are calculated. These errors form a time series, denoted as $Y_t$.
  3. SETAR Model for Drift Detection: A SETAR model is trained on this error rate time series ($Y_t$). The SETAR model is designed to detect changes in the behavior of the error rate, indicating a potential concept drift. The threshold values identified by the SETAR model correspond to the detected drift points.
  4. Concept Drift Adaptation: In addition to detection, ADDM proposes a method for adapting the model. This involves aggregating the old model with a newly trained model. The weighting for this aggregation is determined by the severity of the detected drift, calculated using quantiles of the error rate in old and new concepts. This ensures that more severe drifts lead to a greater reliance on the new model.

Datasets

The authors evaluated ADDM on a combination of synthetic and real-world datasets:

  • Synthetic Datasets: Six datasets were used, generated using the scikit-multiflow package. These datasets were designed to simulate artificial concept drifts. Examples include:
  • Friedman multi-variate regression data set
    • Brieman regression data set
    • Agrawal stream generator data sets (two versions)
    • Mixed data set
  • Real-world Datasets: Five publicly available datasets from the UCI Machine Learning Repository were used:
  • Panama electricity data set
    • Italian air quality data set
    • NSW electricity data set
    • Gas sensor array drift data set
    • Beijing Multi-Site air quality data set

Key Insights and Ideas

  • Shifting Focus to Error Rate: Instead of directly analyzing input data distributions or relying solely on model uncertainty, ADDM cleverly uses the model's own error rate as the signal for concept drift. This is a powerful insight because a degraded model performance is a direct consequence of drift.
  • SETAR's Suitability for Error Dynamics: The choice of SETAR is key. Its ability to model non-linear, regime-switching behavior makes it adept at capturing the often erratic and sudden changes in error rates that occur during concept drift.
  • Severity-Based Adaptation: The proposed adaptation mechanism, which weights new models based on drift severity, is a practical improvement. It allows for more nuanced model updates, avoiding drastic retraining when drifts are minor and emphasizing new models when drifts are substantial.
  • Theoretical Guarantees: The use of OLS for SETAR parameter estimation provides theoretical guarantees for convergence, adding robustness to the method.

Key Results

  • Superior Detection Accuracy: ADDM consistently outperformed seven state-of-the-art drift detection methods across both synthetic and real-world datasets.
  • Low False Alarm Rate: On synthetic datasets, ADDM achieved a significantly lower false alarm rate (FA) compared to other methods, which often generated many spurious detections. This is crucial for practical applications where retraining is costly.
  • Effective on Real-World Data: For real-world datasets, ADDM showed competitive or superior performance in terms of model loss (MSE for regression, cross-entropy for classification) and, importantly, required significantly fewer model retrainings (nb_train). For instance, on the Gas sensor drift dataset, ADDM retrained only 7 times compared to KSWIN's 42 times, while maintaining a comparable loss.
  • Theoretical Foundation: The method is theoretically grounded, and confidence intervals for detected drift points can be constructed.

Limitations and Future Work

  • Reliance on Error Rate: While effective, the method relies on the availability of ground-truth labels to compute the error rate for the SETAR model. The paper notes that in cases where true labels are unavailable, model uncertainty (as explored by others) could be an alternative signal.
  • Future Work: The authors express interest in extending their approach to directly use input data for drift detection, bypassing the need for error rates or labels altogether. This would further enhance its applicability in scenarios with limited or no ground truth.

Summary updated

Abstract

In the classic machine learning framework, models are trained on historical data and used to predict future values. It is assumed that the data distribution does not change over time (stationarity). However, in real-world scenarios, the data generation process changes over time and the model has to adapt to the new incoming data. This phenomenon is known as concept drift and leads to a decrease in the predictive model's performance. In this study, we propose a new concept drift detection method based on autoregressive models called ADDM. This method can be integrated into any machine learning algorithm from deep neural networks to simple linear regression model. Our results show that this new concept drift detection method outperforms the state-of-the-art drift detection methods, both on synthetic data sets and real-world data sets. Our approach is theoretically guaranteed as well as empirical and effective for the detection of various concept drifts. In addition to the drift detector, we proposed a new method of concept drift adaptation based on the severity of the drift.
BibTeX
@inproceedings{mayaki2022autoregressive,
  author = {Mansour Zoubeirou A. Mayaki and Michel Riveill},
  title = {{Autoregressive based Drift Detection Method}},
  booktitle = {IJCNN},
  year = {2022},
  doi = {10.1109/ijcnn55064.2022.9892066},
}

Loading related content...

Figure