An Empirical Study on Log-based Anomaly Detection Using Machine Learning

Shan Ali, Chaima Boufaied, Domenico Bianculli, Paula Branco, Lionel C. Briand, Nathan Aschbacher

TL;DR

This paper presents a comprehensive empirical study comparing various traditional and deep machine learning techniques for log-based anomaly detection (LAD). It reveals that traditional methods, particularly Random Forest, often perform comparably to deep learning approaches in accuracy and are significantly less sensitive to hyperparameter tuning, making them a practical choice for many LAD tasks.

What Problem Does This Paper Address?

Modern systems generate vast amounts of execution logs, which are crucial for identifying anomalies and potential failures. Log-based anomaly detection (LAD) aims to automatically detect these anomalies from these logs. While deep learning (DL) techniques have dominated recent research in LAD, this paper questions whether this focus is always justified. The authors argue that traditional machine learning (ML) techniques might offer comparable or even superior performance in many scenarios, especially when considering practical aspects like hyperparameter tuning sensitivity and computational efficiency. Furthermore, existing evaluations often focus solely on detection accuracy, neglecting other critical factors like training/prediction times and robustness to hyperparameter variations. This study aims to provide a more holistic and evidence-based comparison of LAD techniques.

Methods

The study evaluates a range of supervised and semi-supervised ML techniques, encompassing both traditional and deep learning approaches.

Traditional ML Techniques: * Support Vector Machine (SVM): A supervised technique using kernels (specifically RBF) to handle non-linear data. * Random Forest (RF): A supervised ensemble method known for its robustness. * One-Class SVM (OC-SVM): A semi-supervised technique that learns from normal data to detect anomalies.

Deep Learning Techniques: * LSTM (Long Short-Term Memory): A type of Recurrent Neural Network (RNN) capable of learning long-term dependencies. * LogRobust: An attention-based Bi-LSTM model designed to capture contextual semantics in log sequences. * DeepLog: A semi-supervised forecasting-based technique that predicts the next log event. * NeuralLog: A transformer-based supervised technique that directly processes raw log messages without parsing. * Logs2Graphs: A GNN-based semi-supervised technique that models log data as graphs for anomaly detection and interpretability.

The evaluation is based on four key criteria: detection accuracy (F1-score), time performance (re-training and prediction time), and the sensitivity of both accuracy and time performance to hyperparameter tuning. The paper also details log representation techniques, including word vectorization (FastText), log event occurrence vectorization (TF-IDF), and sequence vectorization (average operator).

Datasets

The study utilizes seven public benchmark datasets for evaluation:

  • Session-based Datasets:
    • HDFS (Hadoop Distributed File System)
    • Hadoop
    • F-dataset (synthesized from OpenStack)
  • Log message-based Datasets:
    • Hades
    • BGL (BlueGene/L)
    • Thunderbird
    • Spirit

The paper notes that datasets like HDFS and BGL are commonly used, while Hades is newer. It also discusses the characteristics of these datasets, including their imbalance ratios and sequence lengths.

Key Insights and Ideas

  • Traditional ML holds its own: Contrary to the trend of focusing solely on deep learning, this study demonstrates that traditional ML techniques, particularly Random Forest, are highly competitive with deep learning methods for log anomaly detection, often achieving similar or better results.
  • Hyperparameter sensitivity matters: Deep learning models, while powerful, are often significantly more sensitive to hyperparameter tuning than traditional ML models. This means achieving optimal performance with deep learning can require more effort and computational resources.
  • Practicality over complexity: For many LAD tasks, the trade-off between accuracy and practical considerations like training time, prediction time, and ease of tuning favors traditional methods. Random Forest, in particular, emerges as a strong candidate due to its robustness and lower sensitivity to hyperparameter variations.
  • Window size impact: The choice of window size for processing log sequences significantly impacts detection accuracy and time performance, with smaller windows often leading to better accuracy but longer training times for deep models, and vice-versa for traditional models.
  • Semi-supervised limitations: Semi-supervised techniques, while useful when anomalies are scarce, generally underperform supervised methods in terms of detection accuracy.

Key Results

  • Detection Accuracy: Supervised traditional (SVM, RF) and deep (LSTM, LogRobust, NeuralLog1, NeuralLog2) ML techniques show comparable detection accuracy (F1-score) on most session-based datasets (HDFS, Hadoop). However, on the F-dataset, traditional methods significantly outperform deep learning models, with RF achieving 96.58% F1-score. On log message-based datasets, supervised techniques generally perform well, with SVM showing strong performance on the imbalanced Hades dataset.
  • Time Performance: Traditional ML techniques (especially RF) are significantly faster in terms of re-training time (about an order of magnitude) compared to deep learning techniques. Prediction times are generally comparable across most techniques, except for transformer-based models (NeuralLog1, NeuralLog2) which can be slower.
  • Hyperparameter Sensitivity: RF exhibits much lower sensitivity to hyperparameter tuning for detection accuracy compared to deep learning techniques. SVM also shows less sensitivity than deep models, though it can be sensitive on highly imbalanced datasets. Deep learning models, especially transformer-based ones, are generally more sensitive.
  • Semi-supervised Performance: DeepLog generally outperforms OC-SVM and Logs2Graphs in detection accuracy on HDFS. However, Logs2Graphs shows superior performance on log message-based datasets, suggesting its GNN-based approach is more effective for anomaly detection and interpretability. Semi-supervised techniques generally have lower detection accuracy than supervised ones.

Limitations and Future Work

Limitations: * Limited Technique Selection: The study evaluated a selection of nine ML techniques. The authors acknowledge that this might limit the generalization of their findings and suggest that other techniques could yield different results. * Reproducibility Challenges: The authors note that many existing LAD studies do not fully share hyperparameter settings or code, making direct reproduction difficult. * Data Preprocessing: The study did not explore advanced data preprocessing techniques, which could potentially enhance detection accuracy. * Computational Cost: The extensive hyperparameter tuning and cross-dataset evaluation required significant computational resources, limiting the scope of some analyses.

Future Work: * Investigate the impact of different data distributions and model complexity on LAD performance. * Develop solutions to facilitate anomaly diagnosis, moving beyond just detection to understanding the causes of anomalies. * Explore more advanced data preprocessing techniques to further improve detection accuracy.

Summary updated

BibTeX
@article{ali2025empirical,
  author = {Shan Ali and Chaima Boufaied and Domenico Bianculli and Paula Branco and Lionel C. Briand and Nathan Aschbacher},
  title = {{An Empirical Study on Log-based Anomaly Detection Using Machine Learning}},
  journal = {Empir. Softw. Eng.},
  year = {2025},
  doi = {10.1007/s10664-025-10669-3},
}
Figure