Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction

Fatemeh Hadadi, Joshua Heneage Dawes, Donghwan Shin, Domenico Bianculli, Lionel C. Briand

TL;DR

This paper systematically evaluates various deep learning models and log embedding strategies for predicting software failures from system logs. They introduce a novel synthetic data generation approach to overcome dataset limitations and find that a CNN-based model with the Logkey2vec embedding strategy offers the best performance, outperforming traditional ML methods and showing robustness across various dataset characteristics.

What Problem Does This Paper Address?

As software systems become increasingly complex, their reliability is paramount. Analyzing system logs generated during runtime is a promising way to predict failures before they occur, allowing for corrective actions and minimizing disruptions. However, existing research on using deep learning (DL) for this task is limited. Studies often focus on only a few DL architectures (like RNNs or CNNs) or a narrow range of log embedding strategies. Crucially, the scarcity of diverse, well-characterized datasets hinders the development of generalizable guidelines for selecting the best models. This paper aims to bridge this gap by systematically evaluating a comprehensive set of DL models and embedding strategies across a wide range of controlled synthetic datasets, and validating findings on a real-world dataset.

Methods

The authors propose a modular architecture for failure prediction, allowing for flexible combinations of log embedding strategies and DL encoders.

  • Log Embedding Strategies: They investigate three main categories:
  • Template ID-based: Logkey2vec (focuses on the order of log template IDs).
  • Semantic-based: BERT (leverages contextual information of words in log templates).
  • Hybrid: FastText + TF-IDF (combines word embeddings with template frequency).
  • Deep Learning Encoders: Four main DL architectures are evaluated:
  • LSTM (Long Short-Term Memory)
    • BiLSTM (Bidirectional LSTM)
    • CNN (Convolutional Neural Network)
    • Transformer
  • Synthetic Data Generation: To address dataset limitations, they developed a novel approach using behavioral models (inferred from real logs using finite-state automata) and regular expressions to define failure patterns. This allows for controlled generation of datasets with specific characteristics (size, failure percentage, log sequence length, failure pattern type).
  • Evaluation: The study uses the F1 score as the primary metric and compares DL models against a traditional ML baseline (Random Forest).

Datasets

  • Synthesized Datasets: 360 datasets were generated for each of three inferred behavioral models (M1, M2, M3). These datasets were systematically varied across four characteristics: dataset size (6 levels), log sequence length (5 levels), failure percentage (6 levels), and failure pattern type (2 levels).
  • Real-world Dataset: OpenStack_PF, a processed version of the OpenStack dataset, was used for validation.

Key Insights and Ideas

  • CNN+Logkey2vec is the winning combination: The study consistently finds that a CNN-based encoder paired with the Logkey2vec embedding strategy provides the best failure prediction accuracy and robustness. This is surprising because Logkey2vec doesn't inherently capture semantic meaning, unlike BERT, but its focus on template order and joint training with the CNN proves effective.
  • Dataset characteristics matter significantly: Dataset size and failure percentage are the most critical factors influencing prediction accuracy. Larger datasets and higher failure percentages generally lead to better performance.
  • Synthetic data is a valid proxy: The results on synthesized datasets closely match those on the real-world OpenStack_PF dataset, validating their novel data generation approach and its utility for research.
  • DL models offer a significant advantage over traditional ML: The best DL configuration (CNN+Logkey2vec) consistently outperforms Random Forest, highlighting the power of deep learning for capturing complex log patterns.

Key Results

  • Best Performing Model: The CNN-based encoder with Logkey2vec achieved the highest F1 scores across various configurations and datasets, with an average F1 score of 0.969 on the OpenStack_PF dataset.
  • DL vs. Traditional ML: The best DL configuration (CNN+Logkey2vec) achieved significantly higher accuracy and robustness compared to Random Forest.
  • Impact of Dataset Size: Accuracy decreases significantly for smaller datasets (below 350 samples).
  • Impact of Failure Percentage: Accuracy generally increases with a higher failure percentage.
  • Robustness: Failure predictors are very accurate (F1-score > 0.95) and robust (IQR < 0.01) when dataset size is above 350 or failure percentage is above 7.5%.

Limitations and Future Work

  • Hyperparameter Tuning: While efforts were made to optimize hyperparameters, further fine-tuning could potentially yield even better results.
  • Synthetic Data Generalizability: While the synthetic data generation approach is novel and validated, the generated failure patterns are necessarily dependent on the specific behavioral models and may not perfectly represent all real-world failure patterns. More case studies with finer-grained failure patterns are needed.
  • Timeliness of Prediction: This study focuses on prediction accuracy. Future work should investigate how early DL models can predict failures, which is crucial for practical AIOps applications.
  • Data Noise: The impact of various data noises (e.g., mislabeled logs, parsing errors) on model accuracy was not explored and is a direction for future research.

Summary updated

BibTeX
@article{hadadi2024systematic,
  author = {Fatemeh Hadadi and Joshua Heneage Dawes and Donghwan Shin and Domenico Bianculli and Lionel C. Briand},
  title = {{Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction}},
  journal = {Empir. Softw. Eng.},
  year = {2024},
  doi = {10.1007/s10664-024-10501-4},
}
Figure