Unsupervised Unlearning of Concept Drift with Autoencoders

André Artelt, Kleanthis Malialis, Christos G. Panayiotou, Marios M. Polycarpou, Barbara Hammer

Introduction

In streaming data applications, such as industrial monitoring or consumer behavior analysis, the underlying data distribution often shifts over time—a phenomenon known as "concept drift." When drift occurs, machine learning models trained on historical data typically experience a significant drop in performance. Conventional solutions involve retraining models or employing incremental learning techniques. However, these "local" adaptation strategies face major hurdles in complex systems where a single data stream feeds a multitude of heterogeneous models. Retraining dozens of models is computationally expensive, and the lack of immediate ground-truth labels often makes supervised adaptation impossible.

This paper proposes a "global" alternative: instead of adapting the models to the new data, we adapt the data back to the models. This unsupervised, model-agnostic approach "unlearns" the drift by transforming the shifted data to match the distribution the models were originally trained on.

Methodology

The proposed framework operates in two distinct phases: Distribution Learning and Unsupervised Transfer Learning.

Distribution Learning

Before any drift is detected, an Autoencoder (AE) is trained on the "clean" or original data distribution. The AE learns to compress the input into a latent representation and then reconstruct it. By minimizing reconstruction error, the AE effectively captures the manifold of the original data distribution.

Unsupervised Unlearning

When concept drift is detected—manifesting as a spike in the AE's reconstruction error—the system initiates the "unlearning" process. The goal is to find a mapping function $f(\cdot)$ that transforms drifted samples $x$ into $f(x)$ such that the pre-trained AE can reconstruct them with minimal error.

The mapping function $f(\cdot)$ is optimized to minimize the reconstruction loss of the original AE on the transformed data. To ensure the transformation is conservative and preserves as much information as possible, an $L_1$ regularization term is added to the objective function. This encourages the mapping to only modify features necessary to correct the drift, effectively "undoing" the shift without requiring labels or model retraining.

Experimental Evaluation

The method was evaluated on two datasets representing different types of drift and downstream tasks:

  1. MNIST Digits (0-4): Drift was simulated by zeroing out the top half of the images. A logistic regression classifier served as the downstream model.
  2. Hanoi Water Distribution Network: This realistic simulation of a water grid involves 31 regression models predicting pressure at various nodes. Drift was introduced through four types of sensor faults: constant offsets, Gaussian noise, power failures, and linear scaling.

Results and Analysis

The experimental results demonstrate that the "unlearning" approach significantly recovers performance across various scenarios. In the MNIST task, classification accuracy dropped from 98% to 68% post-drift; the proposed method restored it to 77% without any modification to the classifier.

In the Hanoi Water Network, the impact was even more pronounced. The method consistently improved $R^2$ scores across the 31 regression tasks.

Performance comparison across different stages: before drift, after drift, after naive AE reconstruction, and after the proposed unlearning method.

As shown in the figure above, the proposed unlearning method (red box) significantly outperforms both the drifted state (orange box) and a naive reconstruction using the AE without the learned mapping (green box). Notably, the unlearning method reduces the variance in performance, providing a more stable recovery across different fault types.

Consistency of the unlearning method across 31 different downstream regression tasks in the Hanoi Water Network.

Furthermore, the framework demonstrates excellent scalability. The figure above illustrates the $R^2$ scores for all 31 downstream tasks after applying the unlearning transformation. The consistency of these scores highlights the advantage of a data-centric approach: a single transformation successfully corrects the data for a diverse fleet of models simultaneously.

Conclusion

The "Unsupervised Unlearning" framework offers a scalable, label-free solution to concept drift. By shifting the focus from model adaptation to data transformation, it enables complex systems to maintain reliability without the overhead of individual model retraining. While the method's effectiveness depends on the drift being detectable via reconstruction error, its model-agnostic nature makes it a powerful tool for modern, multi-model data pipelines.

Citation: Artelt, A., et al. (2023). Unsupervised Unlearning of Concept Drift with Autoencoders. arXiv:2211.12989v2.

Summary updated

BibTeX
@inproceedings{artelt2023unsupervised,
  author = {André Artelt and Kleanthis Malialis and Christos G. Panayiotou and Marios M. Polycarpou and Barbara Hammer},
  title = {{Unsupervised Unlearning of Concept Drift with Autoencoders}},
  booktitle = {SSCI},
  year = {2023},
  doi = {10.1109/ssci52147.2023.10372001},
}
Figure