Interactive Log Parsing via Light-weight User Feedbacks

Liming Wang, Hong Xie, Ye Li, Jian Tan, John C. S. Lui

This paper, presented at the 2023 ACM Web Conference, introduces a novel framework for Interactive Log Parsing that bridges the gap between fully automated template mining and the practical needs of system engineers.

TL;DR

The authors propose a human-in-the-loop framework that uses light-weight user feedback to refine automated log parsing results. By performing atomic operations like merging, splitting, and completing templates, the system achieves near-perfect accuracy across diverse datasets while keeping the manual effort independent of the total log volume.

What Problem Does This Paper Address?

Modern web applications generate massive volumes of logs, which are essential for troubleshooting. Template mining is the process of clustering these logs into groups and extracting a "template" (e.g., turning Failed password for user root into Failed password for user <*>).

While many automated algorithms exist (like Drain or Spell), they often suffer from three types of errors:

  1. Loss-pure error: A cluster is correct, but the extracted template is missing key information.
  2. Complete-partial error: Logs of the same type are split into multiple different clusters.
  3. Loss-mixed error: Logs of different types are incorrectly lumped into a single cluster.

Existing tools are often "black boxes" that don't allow engineers to interactively adjust the granularity of these results during a live investigation.

Methods

The authors developed a generic framework that can sit on top of any existing log parsing algorithm. It relies on three types of light-weight feedback:

  • Message Loss Check: A simple binary question asking if a template is missing information.
  • Dummy Token Selection: The user identifies specific tokens (like IP addresses or IDs) that should be treated as variables.
  • Template Selection: The user picks the correct match from a list of candidates, sorted by similarity to reduce cognitive load.

These feedbacks power three atomic operations:

  1. Message Completion: Uses Longest Common Subsequence (LCS) logic to fill in missing parts of a template.
  2. Merge: Combines clusters that represent the same underlying event.
  3. Separation: Splits clusters that have been incorrectly mixed.

To test this at scale, the authors also designed a Feedback Simulator (Algorithm 7) that mimics human responses based on ground-truth data.

Algorithm 7: Feedback Simulator

Datasets

The framework was evaluated using 16 widely used benchmark datasets from the LogPAI project, including logs from Android, HDFS, BGL (BlueGene/L supercomputer), OpenStack, and Spark. These represent a broad spectrum of log formats and complexities.

Key Insights and Ideas

  • Log-Volume Invariance: Perhaps the most significant finding is that the "query complexity" (the number of questions asked of the human) depends on the number of unique templates, not the millions of log lines. This makes the approach highly scalable for big data.
  • LCS with a Human Touch: While automated LCS often fails to distinguish between static text and variables, human-guided "trimming" of dummy tokens allows the LCS algorithm to find the perfect template quickly.
  • Efficiency Gains: Surprisingly, the interactive refinement can actually be faster than the base algorithms. As log volume increases, the computational efficiency of this method improves relative to traditional parsers.

Key Results

The authors applied their framework to five representative base algorithms (Drain, Spell, IPLoM, Logram, and PrefixGraph).

  • Near-Perfect Accuracy: In almost every case, the framework boosted Group Accuracy (GA) and Message Accuracy (MA) to 1.0 (100%) or very close to it.
  • Human vs. Simulator: The results showed that the simulator was a highly accurate proxy for real human feedback, as seen in Table 1.
Table 1: Accuracy (human feedback vs. simulator).
  • Dramatic Improvements: For the Proxifier dataset, the base Drain algorithm had a GA of only 0.5265. With the interactive framework, this jumped to 1.0. Similar massive gains were seen in the Linux and OpenStack datasets.
Table 2: Accuracy (Drain, fine tuned parameter).

Limitations and Future Work

  • User Expertise: The framework assumes the user provides "correct" feedback. If an engineer is unfamiliar with the specific log domain, they might provide incorrect labels, though the authors' user study suggests the tasks are simple enough for most to handle.
  • Initial Parsing Quality: While the framework can fix many errors, it still benefits from a decent starting point provided by the base algorithm.
  • Future Directions: The authors suggest that future work could involve further reducing user burden by incorporating more advanced semantic understanding or active learning to prioritize which templates need human review first.

Summary updated

Abstract

Template mining is one of the foundational tasks to support log analysis, which supports the diagnosis and troubleshooting of large scale Web applications. This paper develops a human-in-the-loop template mining framework to support interactive log analysis, which is highly desirable in real-world diagnosis or troubleshooting of Web applications but yet previous template mining algorithms fails to support it. We formulate three types of light-weight user feedbacks and based on them we design three atomic human-in-the-loop template mining algorithms. We derive mild conditions under which the outputs of our proposed algorithms are provably correct. We also derive upper bounds on the computational complexity and query complexity of each algorithm. We demonstrate the versatility of our proposed algorithms by combining them to improve the template mining accuracy of five representative algorithms over sixteen widely used benchmark datasets.
BibTeX
@inproceedings{wang2023interactive,
  author = {Liming Wang and Hong Xie and Ye Li and Jian Tan and John C. S. Lui},
  title = {{Interactive Log Parsing via Light-weight User Feedbacks}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583456},
}
Figure