TL;DR
This paper investigates the crucial but understudied relationship between log parsing accuracy and anomaly detection performance in software systems. Surprisingly, they found that how well logs are parsed (accuracy) doesn't strongly correlate with how well anomalies are detected. Instead, the distinguishability of parsed logs (whether normal and abnormal events can be told apart) is the key factor for effective anomaly detection.
What Problem Does This Paper Address?
Software systems generate massive amounts of log data, which are vital for monitoring and troubleshooting. Log-based anomaly detection aims to automatically identify unusual system behavior by analyzing these logs. A common preprocessing step for many anomaly detection techniques, especially those using deep learning, is "log parsing." This process converts unstructured log messages into structured templates, making them easier for algorithms to process.
Despite the apparent importance of log parsing for anomaly detection, its impact on the accuracy of anomaly detection has received surprisingly little attention. Previous studies have either been theoretical, used limited datasets, or didn't rigorously assess log parsing accuracy itself. This paper aims to fill this gap by conducting a comprehensive empirical study to understand:
- To what extent does the accuracy of log parsing affect anomaly detection accuracy?
- How does the distinguishability of log parsing results (a theoretical concept suggesting that normal and abnormal logs should remain separable after parsing) impact anomaly detection accuracy?
Methods
The authors conducted a large-scale empirical study involving:
- 13 Log Parsing Techniques: They used a variety of established log parsing algorithms.
- 7 Anomaly Detection Techniques: This included five deep learning models (DeepLog, LogAnomaly, LogRobust, PLELog, CNN) and two traditional machine learning models (SVM, Random Forest).
- 3 Publicly Available Log Datasets: HDFS, Hadoop, and OpenStack datasets were chosen for their availability of normal/abnormal labels and source code for accurate oracle template generation.
To address their research questions, they employed the following methodologies:
- RQ1 (Log Parsing Accuracy vs. Anomaly Detection Accuracy): They ran each log parsing technique on each dataset, computed standard log parsing accuracy metrics (Grouping Accuracy - GA, Parsing Accuracy - PA, and Template Accuracy - TA), and then fed these parsed logs into each anomaly detection technique. The anomaly detection accuracy was measured using Precision, Recall, and F1-score. They then analyzed the correlation between parsing accuracy and detection accuracy.
- RQ2 (Distinguishability vs. Anomaly Detection Accuracy): To isolate the effect of distinguishability, they artificially generated pairs of "distinguishable" and "indistinguishable" log parsing results from existing ones. This was done by either merging templates (to make logs indistinguishable) or removing logs (to make them distinguishable). They then compared the anomaly detection performance on these paired results. They also performed an additional analysis using a "degree of distinguishability" metric.
Datasets
The study utilized three publicly available log datasets from the LogHub benchmark:
- HDFS (reduced): A large dataset of system logs, reduced to manage computational load.
- Hadoop: Another system log dataset.
- OpenStack: A system log dataset.
The authors specifically selected these datasets because they provided labels for normal and abnormal logs and, crucially, had publicly available source code. This allowed them to extract "oracle templates" (ground truth) directly from the source code, ensuring accurate evaluation of log parsing accuracy, which is a significant improvement over previous studies that relied on manually generated or sampled templates.
Key Insights and Ideas
- Accuracy is Not King (for Parsing): The most surprising finding is that higher log parsing accuracy (measured by GA, PA, or TA) does not necessarily lead to better anomaly detection accuracy. This challenges the intuitive assumption that better parsed logs should always improve downstream analysis.
- Distinguishability is Everything: The theoretical concept of "distinguishability" – the ability of parsed logs to clearly separate normal from abnormal behavior – emerged as the critical factor for effective anomaly detection.
- Why Accuracy Fails: The paper explains that log parsing accuracy metrics often treat all logs and templates as equally important. However, anomaly detection needs to focus on discriminating specific patterns. A log parsing result with lower overall accuracy might still be better if it clearly separates normal and abnormal sequences, even if some individual messages are misparsed.
- Artificial Generation Mimics Reality: While they artificially generated distinguishable/indistinguishable log pairs, the authors argue this process mimics real-world scenarios where imperfect log parsing can lead to indistinguishable outputs.
- Deep Learning vs. Traditional ML: Deep learning-based anomaly detection techniques showed a stronger correlation with distinguishability than traditional ML methods, which seemed more sensitive to the sheer number of templates (features) rather than their separability.
Key Results
- No Strong Correlation (Accuracy): Across all tested log parsing and anomaly detection techniques and datasets (HDFS, Hadoop), there was no significant positive correlation between log parsing accuracy metrics (GA, PA, FTA) and anomaly detection accuracy (F1 score).
- Significant Impact of Distinguishability: When comparing distinguishable vs. indistinguishable log parsing results, anomaly detection accuracy (F1 score) consistently decreased when using indistinguishable results. This decrease was statistically significant across all tested anomaly detection techniques. For example, on the HDFS dataset, the F1 score decreased by up to 90 percentage points when using indistinguishable logs.
- Degree of Distinguishability: An additional analysis showed that for most deep learning models, anomaly detection accuracy generally increased with a higher "degree of distinguishability" of the log parsing results.
Limitations and Future Work
- Dataset Specificity: The study used a specific set of log datasets (HDFS, Hadoop, OpenStack). While these are common in the field, they might not capture the full diversity of log data characteristics. Future work should involve more datasets to improve generalizability.
- Artificial Distinguishability Generation: The method for generating distinguishable/indistinguishable log pairs, while designed to minimize bias, is still artificial. Future work could explore more natural ways to achieve this.
- Metric for Distinguishability: The proposed metric for the "degree of distinguishability" is heuristic. Developing more sophisticated and precise metrics is suggested for future research.
- Anomaly Detection Techniques: The study focused on established DL and ML techniques. Future work could evaluate techniques that do not require log parsing or explore few-shot anomaly detection methods.
- Log Parsing Technique Selection: The paper did not aim to find the "best" log parsing technique but rather to understand the impact of their properties. Future work could focus on developing log parsing techniques that prioritize distinguishability.
Summary updated