TL;DR
This paper presents a comprehensive empirical study comparing various machine learning techniques for log-based anomaly detection (LAD). It reveals that traditional ML methods, particularly Random Forest, often perform comparably to deep learning models in accuracy and speed, while being less sensitive to hyperparameter tuning, making them a practical choice.
What Problem Does This Paper Address?
As systems grow in complexity, automatically detecting anomalies from execution logs (Log-based Anomaly Detection or LAD) becomes crucial. While deep learning (DL) techniques have gained popularity, their dominance might be arbitrary. This paper addresses the need for a thorough comparison of traditional ML, deep learning, and semi-supervised techniques across multiple evaluation criteria beyond just detection accuracy. The authors argue that factors like training/prediction time and sensitivity to hyperparameter tuning are vital for practical adoption by engineers.
Methods
The study evaluates nine ML techniques: three traditional (SVM, RF, OC-SVM) and six deep learning (LSTM, LogRobust, DeepLog, NeuralLog1, NeuralLog2, Logs2Graphs). They employ a rigorous methodology involving:
- Log Parsing: Using the Drain tool to extract log templates.
- Feature Extraction: Employing FastText for word vectorization, TF-IDF for aggregation, and sequence vectorization.
- Hyperparameter Tuning: Conducting extensive grid searches for each technique across multiple datasets and window sizes.
- Evaluation Criteria: Assessing detection accuracy (F1-score), time performance (re-training and prediction time), and sensitivity of both to hyperparameter tuning.
- Statistical Analysis: Using Kruskal-Wallis and Dunn's tests to determine the statistical significance of observed differences.
Datasets
The study utilizes seven public benchmark datasets:
- Session-based: HDFS, Hadoop, F-dataset. These are labeled at the session level, representing a full system execution.
- Log message-based: Hades, BGL, Thunderbird, Spirit. These are labeled at the individual log message level.
The paper notes that OpenStack was excluded due to severe imbalance and high overlap between normal and anomalous sequences. The F-dataset is a synthesized version of OpenStack.
Key Insights and Ideas
- Traditional ML is competitive: Contrary to the trend favoring deep learning, traditional ML techniques, especially Random Forest, often match or exceed deep learning models in detection accuracy and prediction time, particularly when considering practical aspects like hyperparameter tuning.
- Hyperparameter sensitivity matters: Deep learning models, especially transformer-based ones, are significantly more sensitive to hyperparameter tuning than traditional ML models, requiring more effort and resources for optimal performance.
- Semi-supervised learning lags: Semi-supervised techniques generally show lower detection accuracy compared to their supervised counterparts, making them less suitable for current LAD tasks, especially when anomaly data is scarce.
- Window size impact: Smaller window sizes (10-20) generally lead to higher detection accuracy for supervised models due to more training sequences, but this can vary for semi-supervised methods (OC-SVM prefers larger windows).
- Data imbalance is a challenge: Highly imbalanced datasets negatively impact detection accuracy across all techniques, though some models (like SVM) show better resilience due to their handling of misclassification penalties.
Key Results
- Accuracy: Supervised traditional and deep ML techniques perform similarly on most datasets. RF and SVM achieve high F1-scores across various datasets. Semi-supervised techniques generally yield lower F1-scores.
- Time Performance: Traditional ML techniques (especially RF) have significantly shorter re-training times than deep learning models. Prediction times are generally comparable, except for transformer-based models which can be slower.
- Sensitivity: RF is the least sensitive supervised ML technique to hyperparameter tuning for detection accuracy. Deep learning models, particularly transformer-based ones, are the most sensitive.
- Window Size: Smaller window sizes generally yield better detection accuracy for supervised models, while OC-SVM performs better with larger windows.
- Specific Findings:
- On the F-dataset, deep ML models struggled, while traditional ML performed well.
- LogRobust and NeuralLog models showed high sensitivity to hyperparameter tuning.
- Logs2Graphs outperformed OC-SVM and DeepLog on log message-based datasets, suggesting GNNs' potential.
Limitations and Future Work
- Limited Technique Selection: The study evaluated nine techniques; a broader selection could yield different insights.
- Reproducibility Concerns: Many LAD studies lack shared code or detailed hyperparameter settings, hindering reproducibility.
- Data Preprocessing: The impact of advanced data preprocessing techniques on detection accuracy was not explored.
- Future Work: The authors plan to investigate the influence of data distributions and model complexity on LAD performance. They also highlight the need for solutions that go beyond anomaly detection to facilitate anomaly diagnosis (identifying the cause of anomalies).
Summary updated