Self-supervised Log Parsing

Sasho Nedelkoski, Jasmin Bogatinovski, Alexander Acker, Jorge Cardoso, Odej Kao

TL;DR

This paper introduces NuLog, a novel self-supervised log parsing method that leverages masked language modeling (MLM) to automatically extract structured log templates from raw, unstructured log messages. NuLog achieves state-of-the-art performance in parsing accuracy and edit distance across various real-world datasets and demonstrates its utility as a powerful feature extractor for downstream tasks like anomaly detection.

What Problem Does This Paper Address?

Modern IT systems, with their complex and diverse architectures, generate massive amounts of log data. This data is crucial for troubleshooting and maintenance, but its semi-structured and free-form text nature makes automated analysis challenging. Traditional log parsing methods often rely on manual rule extraction or heuristics, which are brittle, require domain expertise, and struggle to generalize across different log types and systems. This leads to significant overhead for operators managing large infrastructures, necessitating a more robust and automated approach to log parsing.

Methods

NuLog employs a self-supervised learning approach based on the Transformer architecture, specifically utilizing Masked Language Modeling (MLM), a technique popularized in Natural Language Processing (NLP). The core idea is to treat log parsing as a language modeling task: the model learns to predict masked tokens within a log message based on their context. The intuition is that tokens that are consistently part of a log template will be accurately predicted, while variable parts will be harder to predict.

The process involves:

  1. Preprocessing: Stripping meta-information and tokenizing log messages into sequences of words or sub-word units.
  2. Masking: Randomly masking tokens in the input sequence and using the original token as the prediction target.
  3. Transformer Model: A Transformer encoder processes the masked token sequences to generate contextualized embeddings.
  4. Log Embedding: A special (CLS) token embedding is used to represent the entire log message, which is then used for prediction.
  5. Template Extraction: During an "online" phase, the model's confidence in predicting each token is used to distinguish between constant template parts and variable parameters. Tokens with high prediction confidence are deemed part of the template, while low confidence suggests a variable.

A key advantage is that NuLog does not require any domain-specific knowledge or hand-crafted rules, making it more generalizable.

Datasets

NuLog was evaluated on ten real-world log datasets, covering a wide range of systems:

  • BGL: BlueGene Supercomputer logs
  • Android: Mobile Operating System logs
  • OpenStack: Cloud Operating System logs
  • HDFS: Hadoop Distributed File System logs
  • Apache: Apache HTTP Server logs
  • HPC: High Performance Cluster logs
  • Windows: Windows 7 Computer Operating System logs
  • HealthApp: Mobile Application for Android Devices logs
  • Mac: MacOS Operating System logs
  • Spark: Unified Analytics Engine for Big Data Processing logs

The paper follows the benchmark established by Zhu et al. [19], using a random sample of 2000 log messages from each dataset where ground truth templates are available.

Key Insights and Ideas

  • Self-Supervised Learning for Logs: The paper demonstrates that powerful NLP techniques like MLM can be effectively adapted for log parsing, eliminating the need for manual feature engineering or domain expertise.
  • Log Parsing as Language Modeling: The core insight is that predicting masked tokens in a log message can reveal its underlying template structure. High confidence predictions indicate template components, while low confidence suggests variable parameters.
  • Generalizable Representations: The Transformer model learns rich, contextualized embeddings for log messages. These embeddings capture semantic meaning and are robust across diverse log types, making them suitable for various downstream tasks.
  • Unified Approach: NuLog provides a single, unified framework for both log parsing and generating log embeddings, simplifying the overall log analysis pipeline.

Key Results

  • Parsing Accuracy: NuLog achieves an average parsing accuracy of 99% across the evaluated datasets, outperforming twelve existing log parsing methods. It significantly outperforms baselines on more complex log data from OpenStack, HPC, and HealthApp by 4-13%.
  • Edit Distance: NuLog achieves the lowest edit distance to ground truth templates on several datasets (HDFS, Windows, Android, HealthApp, Mac), indicating accurate identification of template structure and variables.
  • Robustness: NuLog demonstrates superior robustness in parsing accuracy and edit distance across a broad range of log types, with a median parsing accuracy of 0.99 and a median edit distance of 5.00.
  • Anomaly Detection: Case studies show that NuLog's learned log embeddings are highly effective for both unsupervised and supervised anomaly detection, achieving F1 scores of 0.999 and 0.99 respectively on the BGL dataset.

Limitations and Future Work

  • Computational Cost: While NuLog is effective, Transformer-based models can be computationally intensive, especially for very large-scale log streams.
  • Fine-tuning for Downstream Tasks: The paper highlights the benefit of fine-tuning the model for specific downstream tasks like anomaly detection, suggesting that further research could explore more sophisticated fine-tuning strategies.
  • Generalization Across Domains: The authors suggest that future research should focus on improving generalization across different domains and exploring transfer learning techniques for log analysis. They believe that deep language models hold significant promise for advancing log parsing and anomaly detection.

Summary updated

Abstract

Logs are extensively used during the development and maintenance of software systems. They collect runtime events and allow tracking of code execution, which enables a variety of critical tasks such as troubleshooting and fault detection. However, large-scale software systems generate massive volumes of semi-structured log records, posing a major challenge for automated analysis. Parsing semi-structured records with free-form text log messages into structured templates is the first and crucial step that enables further analysis. Existing approaches rely on log-specific heuristics or manual rule extraction. These are often specialized in parsing certain log types, and thus, limit performance scores and generalization. We propose a novel parsing technique called NuLog that utilizes a self-supervised learning model and formulates the parsing task as masked language modeling (MLM). In the process of parsing, the model extracts summarizations from the logs in the form of a vector embedding. This allows the coupling of the MLM as pre-training with a downstream anomaly detection task. We evaluate the parsing performance of NuLog on 10 real-world log datasets and compare the results with 12 parsing techniques. The results show that NuLog outperforms existing methods in parsing accuracy with an average of 99% and achieves the lowest edit distance to the ground truth templates. Additionally, two case studies are conducted to demonstrate the ability of the approach for log-based anomaly detection in both supervised and unsupervised scenario. The results show that NuLog can be successfully used to support troubleshooting tasks. The implementation is available at https://github.com/nulog/nulog .
BibTeX
@inproceedings{nedelkoski2020self,
  author = {Sasho Nedelkoski and Jasmin Bogatinovski and Alexander Acker and Jorge Cardoso and Odej Kao},
  title = {{Self-supervised Log Parsing}},
  booktitle = {ECML/PKDD},
  year = {2020},
  url = {https://openreview.net/forum?id=hfAiWTSd1z},
}
Figure