LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised Anomaly Detection

Feiyi Chen, Zhen Qing, Yingying Zhang, Shuiguang Deng, Yi Xiao, Guansong Pang, Qingsong Wen

Introduction

In web services and cloud computing, the definition of "normal" system behavior is constantly evolving due to software updates, configuration changes, and shifting user patterns. This phenomenon, known as distribution shift, renders static anomaly detection models obsolete. Standard solutions typically involve either full retraining, which is computationally expensive, or few-shot retraining, which often leads to overfitting because the model lacks sufficient new data to learn general patterns.

LARA (Light Anti-overfitting Retraining Approach) addresses this "catch-22" by providing a framework that is fast, data-efficient, and resistant to overfitting. It allows models to adapt to new distributions using as little as 1% of the new data while maintaining high accuracy.

Methodology

LARA is built upon Variational Auto-Encoders (VAEs) and introduces three key innovations to streamline the retraining process:

  1. The Ruminate Block: This component restores historical knowledge directly from the old model's weights, eliminating the need to store massive amounts of raw historical data. Using Online Bayesian Learning, it estimates new latent vectors by balancing prior knowledge with fresh observations.
  2. Linear Adjusting Functions ($M_z$ and $M_x$): Rather than retraining the entire deep neural network, LARA applies simple linear transformations to the latent vectors and reconstructed data. The authors mathematically prove that these linear forms achieve the least error when adapting to a new distribution.
  3. Convex Loss Design: Retraining is formulated as a convex optimization problem. This ensures a single global optimum and a fast convergence rate of $O(1/k)$, preventing the model from falling into the suboptimal traps that cause overfitting in deeper architectures.
The LARA architecture illustrates the iterative retraining process, highlighting the Ruminate block and linear adjustment modules (Mz and Mx) used to adapt to distribution shifts over time.

Experimental Evaluation

The authors evaluated LARA on four real-world multivariate time series datasets: the Server Machine Dataset (SMD), monitoring metrics from Alibaba (J-D1 & J-D2), and NASA spacecraft telemetry (SMAP).

Key Results

  • Few-Shot Performance: When provided with only 1% of the new distribution data, LARA (specifically the LARA-LO variant) achieved higher F1 scores than state-of-the-art models trained on the entire new dataset.
  • Efficiency and Speed: LARA is significantly faster than competing methods, capable of updating a model in less than 8 seconds. It also maintains a much lower memory overhead compared to standard neural network retraining approaches.
  • Stability: While other models experienced performance degradation when retraining on small samples due to overfitting, LARA remained stable. Experimental results confirmed the theoretical $O(1/k)$ convergence rate, indicating that the model reaches its optimal state very quickly.
A comparison of time and memory ratios shows that LARA (LA) is significantly more efficient than other state-of-the-art retraining methods.

Conclusion

LARA demonstrates that for rapid model adaptation, a "light" linear touch can be more effective than complex non-linear retraining. By leveraging a ruminate block for knowledge extraction and ensuring mathematical convexity, LARA provides a robust solution for maintaining anomaly detection accuracy in dynamic environments. Future work may involve extending this framework to non-VAE architectures like Transformers or GANs and further optimizing hyperparameter sensitivity.

Summary updated

BibTeX
@inproceedings{chen2024lara,
  author = {Feiyi Chen and Zhen Qing and Yingying Zhang and Shuiguang Deng and Yi Xiao and Guansong Pang and Qingsong Wen},
  title = {{LARA: A Light and Anti-overfitting Retraining Approach for Unsupervised Anomaly Detection}},
  booktitle = {The Web Conference},
  year = {2024},
  doi = {10.1145/3589334.3645472},
}
Figure