Interactive Log Parsing via Light-weight User Feedback

Liming Wang, Hong Xie, Ye Li, Jian Tan, John C. S. Lui

Abstract

Log parsing is a critical first step in automated system analysis, yet existing automated tools often fail to meet the specific semantic needs of engineers or suffer from inherent clustering errors. This paper introduces a human-in-the-loop framework that utilizes light-weight user feedback to refine the output of any base log parser. By implementing three atomic operations—Message-Completion, Merge, and Separation—the framework achieves near-perfect parsing accuracy across 16 diverse datasets. The approach is computationally efficient, scalable for streaming data, and requires minimal human intervention, making it suitable for production environments.

Introduction

System logs are essential for troubleshooting large-scale applications, but their volume necessitates automated template mining. Traditional log parsers cluster similar logs and extract templates (e.g., converting Failed password for user root to Failed password for user <*>). However, these tools are often "one-size-fits-all" and struggle with three primary error types: loss-pure errors (missing template information), complete-partial errors (over-splitting clusters), and loss-mixed errors (incorrectly merging distinct categories).

A Light-weight Human-in-the-Loop Framework

The proposed framework acts as a "booster" for existing log parsers. It introduces three atomic operations to correct common errors:

  • Message-Completion: Ensures templates capture all necessary semantics.
  • Merge: Combines clusters representing the same underlying event.
  • Separation: Splits mixed clusters into distinct categories.

To minimize user burden, the system employs three light-weight feedback types: binary indicators, dummy token selection (identifying missed variables), and template selection from a sorted list of candidates. This framework is agnostic to the underlying parser, meaning it can enhance the output of algorithms like Drain, Spell, or IPLoM.

The framework achieves near-perfect accuracy across various base algorithms, with simulated feedback closely matching human performance.

Experimental Evaluation

The framework was evaluated using 16 benchmark datasets from the LogPAI repository, covering operating systems, distributed systems, and supercomputers.

Key Results

  • Accuracy Boost: The framework improved both Group Accuracy (GA) and Message Accuracy (MA) to nearly 1.0 (100%) across almost all datasets. For instance, on the HealthApp dataset, the base Drain algorithm's GA improved from 0.78 to 1.0.
  • Efficiency and Scalability: The algorithms are designed for single-pass processing, making them ideal for streaming data. In tests on the BGL dataset (4M+ logs), the interactive method outperformed the base Drain algorithm in speed as log volume increased.
  • Low User Burden: User studies showed that participants took an average of only 1.54 seconds per query, with most datasets requiring only a few dozen interactions to process millions of logs.
Comparison of base Drain accuracy versus the improved results using simulated feedback across 16 benchmark datasets.

Conclusion and Future Work

By involving the user in a light-weight manner, this framework bridges the gap between automated parsing and the subjective semantic needs of system engineers. While it requires an initial human effort, the theoretical rigor and empirical results demonstrate its effectiveness as a scalable, high-accuracy solution. Future work may explore automating feedback through Large Language Models (LLMs) to further reduce human intervention.

Summary updated

Abstract

Template mining is one of the foundational tasks to support log analysis, which supports the diagnosis and troubleshooting of large scale Web applications. This paper develops a human-in-the-loop template mining framework to support interactive log analysis, which is highly desirable in real-world diagnosis or troubleshooting of Web applications but yet previous template mining algorithms fail to support it. We formulate three types of light-weight user feedback and based on them we design three atomic human-in-the-loop template mining algorithms. We derive mild conditions under which the outputs of our proposed algorithms are provably correct. We also derive upper bounds on the computational complexity and query complexity of each algorithm. We demonstrate the versatility of our proposed algorithms by combining them to improve the template mining accuracy of five representative algorithms over sixteen widely used benchmark datasets.
BibTeX
@inproceedings{wang2023interactive,
  author = {Liming Wang and Hong Xie and Ye Li and Jian Tan and John C. S. Lui},
  title = {{Interactive Log Parsing via Light-weight User Feedback}},
  booktitle = {The Web Conference},
  year = {2023},
  doi = {10.1145/3543507.3583456},
}
Figure