Impact of Log Parsing on Log-based Anomaly Detection

Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, Lionel C. Briand

TL;DR

This paper investigates the impact of log parsing on deep learning-based anomaly detection, finding that traditional log parsing accuracy metrics have little correlation with anomaly detection performance. Instead, the "distinguishability" of log parsing results—how well they separate normal from abnormal system behavior—is crucial for effective anomaly detection.

What Problem Does This Paper Address?

Software systems generate massive amounts of log data, which are invaluable for monitoring and detecting anomalies. Log-based anomaly detection aims to identify abnormal system behavior by analyzing these logs. A common preprocessing step in many anomaly detection techniques, especially those using deep learning, is log parsing. This process transforms raw, unstructured log messages into structured formats (templates) that machine learning models can process. However, despite the critical role of log parsing, there's been surprisingly little empirical research on how its accuracy directly impacts anomaly detection performance. Existing studies have either focused on theoretical aspects or used limited datasets and metrics, leaving a gap in understanding the true relationship and guiding the development of better log parsing techniques for anomaly detection.

Methods

The authors conducted a comprehensive empirical study to address their research questions.

  • Log Parsing Techniques: They evaluated 13 different log parsing techniques, leveraging a replication package from a prior study.
  • Anomaly Detection Techniques: They employed seven anomaly detection techniques, including five deep learning-based models (DeepLog, LogAnomaly, LogRobust, CNN, PLELog) and two traditional machine learning models (SVM, Random Forest).
  • Accuracy Metrics: For log parsing, they considered three established metrics: Grouping Accuracy (GA), Parsing Accuracy (PA), and Template Accuracy (TA). For anomaly detection, they used standard metrics like Precision, Recall, and F1-score.
  • Distinguishability Analysis: To investigate the impact of "distinguishability" (a concept proposed theoretically by Shin et al. [44]), they developed methods to generate pairs of distinguishable and indistinguishable log parsing results from the same log parsing technique. This allowed for a direct comparison of anomaly detection performance under these two conditions. They also proposed an additional analysis to measure the degree of distinguishability.

Datasets

The study utilized three publicly available log datasets from the LogHub benchmark:

  • HDFS (reduced): A large dataset that was reduced in size to manage computational resources.
  • Hadoop: Another dataset from the benchmark.
  • OpenStack: A dataset that was ultimately excluded from anomaly detection analysis due to a very small number of abnormal logs.

The authors emphasize that they used the source code of the systems that generated these logs to extract correct "oracle templates," which are crucial for accurately evaluating log parsing accuracy.

Key Insights and Ideas

  • Accuracy is Not King: The most surprising finding is that traditional log parsing accuracy metrics (GA, PA, TA) show no strong correlation with anomaly detection accuracy. This challenges the common assumption that better log parsing accuracy directly leads to better anomaly detection.
  • Distinguishability is Key: The "distinguishability" of log parsing results—how well they separate normal from abnormal log sequences—is the critical factor for anomaly detection performance. If log parsing makes normal and abnormal logs indistinguishable, anomaly detection will fail, regardless of the parsing accuracy.
  • Artificial vs. Real-World: While the study used artificial methods to create distinguishable and indistinguishable log parsing pairs, the authors argue this mimics real-world scenarios where imperfect log parsing can lead to indistinguishable results.
  • Deep Learning vs. Traditional ML: The impact of distinguishability was more pronounced and consistent across deep learning models compared to traditional machine learning models, which seemed less sensitive to the number of templates.

Key Results

  • No Correlation: Across multiple datasets and anomaly detection techniques, the study found no significant correlation between log parsing accuracy (GA, PA, FTA) and anomaly detection accuracy (F1 score).
  • Distinguishability Matters: When comparing distinguishable and indistinguishable log parsing results, anomaly detection accuracy consistently decreased when using indistinguishable results. The differences were statistically significant across most tested techniques. For example, on the HDFS dataset, the F1 score decreased from an average of 0.676 (distinguishable) to 0.508 (indistinguishable) for DL models.
  • Degree of Distinguishability: An additional analysis showed a positive correlation between the degree of distinguishability and anomaly detection accuracy for most deep learning models, reinforcing the importance of this property.

Limitations and Future Work

  • Threats to Validity:
    • Oracle Templates: While efforts were made to use source code for accurate oracle templates, the potential for bias in manual extraction remains a concern.
  • Hyperparameters: The study used default or author-recommended hyperparameters, but individual techniques might perform differently with optimized settings.
  • Dataset Generalizability: The reliance on a few specific datasets (HDFS, Hadoop, OpenStack) might limit the generalizability of findings to other system types.
  • Future Work:
    • More Datasets and Techniques: Expand the study with more diverse datasets and newer log parsing/anomaly detection techniques.
  • Few-Shot Anomaly Detection: Investigate techniques that require less training data.
  • Improved Distinguishability Metrics: Develop more sophisticated metrics to better quantify the degree of distinguishability.
  • Log Parsing without Anomaly Detection: Explore techniques that do not require log parsing.

Summary updated

BibTeX
@article{khan2024impact,
  author = {Zanis Ali Khan and Donghwan Shin and Domenico Bianculli and Lionel C. Briand},
  title = {{Impact of Log Parsing on Log-based Anomaly Detection}},
  journal = {Empir. Softw. Eng.},
  year = {2024},
  doi = {10.1007/s10664-024-10533-w},
}
Figure