Log representation as an interface for log processing applications

Mohammad Amin Sadeghi, Shameem Parambath, Ji Lucas, Youssef Meguebli, Maguette Toure, Fawaz Al Qahtani, Ting Yu, Sanjay Chawla

TL;DR

This paper introduces a novel framework for representing system logs as vector embeddings across multiple levels of abstraction, from individual log fields to entire log sequences. By leveraging Transformer networks and incorporating temporal information, their approach significantly enhances the performance of various log processing applications like anomaly detection and predictive analysis, offering a more unified and powerful way to understand system behavior.

What Problem Does This Paper Address?

System logs are a treasure trove of information, recording every event, message, and transaction within a computer system. However, their raw, often unstructured, textual nature makes them incredibly difficult to analyze at scale. Traditional methods struggle with the sheer volume, diversity, and complexity of log data, especially when dealing with systems that generate logs from numerous devices and processes. This makes tasks like identifying anomalies, diagnosing failures, or understanding system behavior a significant challenge. The paper aims to bridge this gap by proposing a standardized, multi-level representation of logs that can be readily consumed by machine learning models, thereby simplifying and improving the effectiveness of log processing applications.

Methods

The core of this work lies in a multi-level abstraction for log representation, built upon Transformer networks.

  • Levels of Abstraction: They define five levels:
  • Log Sequence: The raw sequence of log entries.
  • Parsed Log: Log entries are parsed into key-value pairs, extracting structured information.
  • Field Embedding: Each key-value pair is converted into a vector representation.
  • Log Embedding: A vector representation for an entire log entry, computed from its field embeddings.
  • Sequence Embedding: A vector representation for a sequence of log entries (e.g., a sliding window or a whole file), computed from log embeddings.
  • Transformer Networks (TNs): They adapt TNs, a state-of-the-art architecture for sequence modeling, to process log data.
  • Time Encoding: Crucially, they incorporate temporal information (timestamps) into the Transformer's input, recognizing its importance for understanding event sequences. This is achieved through a time encoding mechanism.
  • Self-Supervised Learning (Masked Language Model - MLM): The Transformer is trained using a masked language modeling task, where it learns to predict masked log entries based on their context. This allows the model to learn rich embeddings without requiring explicit human-labeled data for training.

Datasets

The authors evaluated their approach on two datasets:

  • HDFS Dataset: An open-source dataset of Hadoop job logs, containing over 11 million log entries, with approximately 2.9% labeled as anomalous. This dataset has a vocabulary size of 29 after tokenization.
  • Radio Data: A proprietary dataset from a leading telecommunications vendor, comprising logs from a 4G cell tower product. This dataset includes 4783 distinct log files, 146,4010 log entries, about 860 log templates, and 180 software programs, collected over two months.

Key Insights and Ideas

  • Logs as a Universal Data Format: The paper highlights that logs can represent a wide variety of sequential data, including natural languages and temporal signals, suggesting their potential as a unifying data representation.
  • Temporal Information is Crucial: Incorporating timestamps significantly improves the performance of log processing models, outperforming models that ignore time. This is a key differentiator from standard NLP models.
  • Multi-Level Abstraction Simplifies Complexity: Breaking down log representation into distinct levels (field, log, sequence) provides a structured way to handle complex log data and makes it easier to develop downstream applications.
  • Transformer's Adaptability: The Transformer architecture, with its self-attention mechanism, proves highly effective for log sequence modeling, overcoming limitations of traditional RNNs and enabling parallel processing.
  • Self-Supervised Learning for Log Embeddings: Using MLM for training allows for the creation of powerful log embeddings without the need for extensive, costly manual labeling, making the approach more scalable.

Key Results

  • Anomaly Detection: Their unsupervised anomaly detection method, based on clustering log embeddings, shows promising results. When evaluated on the HDFS dataset, their approach, when compared to baselines like PCA, Invariant Mining, N-grams, and DeepLog, achieved superior performance in terms of Precision, Recall, and F-measure.
  • Predictive Analysis: The model demonstrated the ability to reliably predict future log entries within a short time frame (next few seconds), showcasing its potential for proactive system monitoring and failure prediction.
  • Log Search: Their embedding-based search method showed a useful approximation to edit distance, with top matches being more sensitive to displacement than edit distance matches, suggesting a more nuanced understanding of log sequence similarity.
  • Performance Gains: The paper consistently shows that their Transformer-based approach, especially when incorporating time, outperforms existing methods on various benchmarks.

Limitations and Future Work

  • Causal Analysis Evaluation: The authors acknowledge that they did not quantitatively evaluate their causal analysis method due to the lack of ground truth data, though they illustrated its potential.
  • Log Search Evaluation: Evaluating log search performance is challenging due to the absence of labeled datasets. While they used edit distance as a proxy, they note its limitations in handling time and continuous parameters.
  • Future Work: The paper suggests that the developed framework can be extended to other downstream tasks like predictive analysis, causal analysis, log search, and log generation/synthesis. They also mention the potential for using the learned language model for log synthesis and test generation. The authors plan to release their anonymized telecommunications dataset to the research community.

Summary updated

Abstract

Log files are files that record events, messages, or transactions. Logs are rich containers of data because they can store a sequence of structured textual and numerical data. Many sequential forms of data including natural languages and temporal signals can be represented as logs. We propose to represent logs at a few levels of abstraction including field level, log level, and log sequence level. The representation for each level can be computed from the previous level. These representations are in vector format and serve as interfaces to downstream applications. We use a version of transformer networks to encode numerical information as well as textual information that is suitable for log embedding. We show how a number of log processing applications can be readily solved with our representation.
BibTeX
@misc{sadeghi2020log,
  author = {Mohammad Amin Sadeghi and Shameem Parambath and Ji Lucas and Youssef Meguebli and Maguette Toure and Fawaz Al Qahtani and Ting Yu and Sanjay Chawla},
  title = {{Log representation as an interface for log processing applications}},
  year = {2020},
  url = {https://openreview.net/forum?id=-5VpoDCExrU},
  howpublished = {OpenReview},
}
Figure