TL;DR
This paper systematically evaluates various deep learning (DL) models and log sequence embedding strategies for predicting software failures from logs. It introduces a modular architecture and a novel synthetic data generation approach to overcome limitations of existing studies, finding that CNNs with Logkey2vec embeddings offer the best performance, outperforming traditional ML methods and other DL configurations.
What Problem Does This Paper Address?
As software systems become increasingly complex, their reliability is paramount. Analyzing system logs generated during runtime offers a promising avenue for predicting failures before they occur, enabling proactive maintenance and minimizing disruptions. However, existing research on using deep learning (DL) for this task is fragmented, often evaluating only a few DL architectures or embedding strategies on limited, non-diverse datasets. This lack of systematic evaluation hinders the development of practical guidelines for selecting the most effective DL models and strategies for log-based failure prediction, especially considering the varying characteristics of real-world log data.
Methods
The authors propose a modular architecture for failure prediction, consisting of two main steps:
- Embedding Step: Converts log sequences into vector representations using various embedding techniques.
- Classification Step: Processes these embeddings using DL encoders to predict failures.
They systematically investigate combinations of:
- DL Encoders: LSTM, BiLSTM, CNN, and Transformer-based models.
- Log Sequence Embedding (LSE) Strategies: Logkey2vec (template ID-based), BERT (semantic-based), and a hybrid of FastText + TF-IDF.
To address dataset limitations, they developed a novel synthetic data generation approach based on behavior models inferred from real-world logs. This approach allows for controlled generation of datasets with specific characteristics (size, failure percentage, log sequence length, failure pattern type), enabling a more comprehensive and unbiased evaluation.
Datasets
- Synthesized Datasets: 360 datasets were generated for each of three behavior models (M1, M2, M3), varying in dataset size (6 levels), maximum log sequence length (5 levels), failure percentage (6 levels), and failure pattern type (2 types).
- Real-world Dataset: OpenStack_PF, a processed version of the OpenStack dataset, was used for validation.
Key Insights and Ideas
- CNNs are King for Log Failure Prediction: The CNN-based encoder consistently outperformed other DL architectures (LSTM, BiLSTM, Transformer) across various datasets and embedding strategies. This is attributed to CNNs' ability to capture spatial information and their robustness with longer sequences.
- Logkey2vec Shines with CNNs: While BERT generally performed well across DL models, Logkey2vec, a template ID-based strategy, achieved the best overall performance when paired with the CNN encoder. This suggests that for CNNs, capturing the order of log templates is crucial, and Logkey2vec's ability to be trained jointly with the encoder is beneficial.
- Synthetic Data is a Game-Changer: The ability to generate controlled, diverse datasets was critical for systematically evaluating the impact of various factors, overcoming the limitations of scarce and non-diverse real-world datasets. The study found that results from synthesized data closely matched real-world data, validating the synthetic generation approach.
- Dataset Characteristics Matter (A Lot): Dataset size and failure percentage were found to be the most influential factors for failure prediction accuracy. Larger datasets and higher failure percentages generally led to better performance. Log sequence length had a notable impact on some configurations (like BiLSTM+B), while failure pattern type showed less consistent influence.
- DL Outperforms Traditional ML: The best DL configuration (CNN+Logkey2vec) significantly outperformed Random Forest, a leading traditional ML classifier, in both accuracy and robustness.
Key Results
- Best Configuration: CNN-based encoder with Logkey2vec embedding strategy achieved the highest F1 scores across most evaluations.
- Accuracy: The best configuration (CNN+L) achieved an average F1 score of 0.969 on synthesized data and 0.974 on the OpenStack_FP dataset.
- Dataset Size Impact: Accuracy significantly decreased with smaller datasets (below 350 samples).
- Failure Percentage Impact: Accuracy generally increased with higher failure percentages, with CNN+L performing well even with unbalanced data (10% failure).
- Comparison to RF: The best DL configuration showed significantly higher accuracy and robustness compared to Random Forest.
Limitations and Future Work
- Hyper-parameter Tuning: While efforts were made to optimize hyper-parameters, different settings might yield even better results.
- Synthetic Data Generation: While the approach is novel and validated, further research could explore more sophisticated methods for generating failure patterns and ensuring broader representativeness of real-world scenarios.
- Timeliness of Prediction: The study focused on prediction accuracy rather than how early failures can be predicted. Future work should investigate metrics like lead time and the number of log messages required for timely predictions.
- Data Noise: The impact of various data noises (mis-labeled logs, parsing errors, evolving logs) on model accuracy was not explored and is a direction for future research.
- Real-world Data Diversity: While OpenStack_PF was used, evaluating the proposed models on a wider range of real-world datasets with diverse characteristics would further enhance generalizability.
Summary updated