Abstract
Log parsing is a critical first step in automated system analysis, yet existing automated tools often fail to meet the specific semantic needs of engineers or suffer from inherent clustering errors. This paper introduces a human-in-the-loop framework that utilizes light-weight user feedback to refine the output of any base log parser. By implementing three atomic operations—Message-Completion, Merge, and Separation—the framework achieves near-perfect parsing accuracy across 16 diverse datasets. The approach is computationally efficient, scalable for streaming data, and requires minimal human intervention, making it suitable for production environments.
Introduction
System logs are essential for troubleshooting large-scale applications, but their volume necessitates automated template mining. Traditional log parsers cluster similar logs and extract templates (e.g., converting Failed password for user root to Failed password for user <*>). However, these tools are often "one-size-fits-all" and struggle with three primary error types: loss-pure errors (missing template information), complete-partial errors (over-splitting clusters), and loss-mixed errors (incorrectly merging distinct categories).
A Light-weight Human-in-the-Loop Framework
The proposed framework acts as a "booster" for existing log parsers. It introduces three atomic operations to correct common errors:
- Message-Completion: Ensures templates capture all necessary semantics.
- Merge: Combines clusters representing the same underlying event.
- Separation: Splits mixed clusters into distinct categories.
To minimize user burden, the system employs three light-weight feedback types: binary indicators, dummy token selection (identifying missed variables), and template selection from a sorted list of candidates. This framework is agnostic to the underlying parser, meaning it can enhance the output of algorithms like Drain, Spell, or IPLoM.
Experimental Evaluation
The framework was evaluated using 16 benchmark datasets from the LogPAI repository, covering operating systems, distributed systems, and supercomputers.
Key Results
- Accuracy Boost: The framework improved both Group Accuracy (GA) and Message Accuracy (MA) to nearly 1.0 (100%) across almost all datasets. For instance, on the HealthApp dataset, the base Drain algorithm's GA improved from 0.78 to 1.0.
- Efficiency and Scalability: The algorithms are designed for single-pass processing, making them ideal for streaming data. In tests on the BGL dataset (4M+ logs), the interactive method outperformed the base Drain algorithm in speed as log volume increased.
- Low User Burden: User studies showed that participants took an average of only 1.54 seconds per query, with most datasets requiring only a few dozen interactions to process millions of logs.
Conclusion and Future Work
By involving the user in a light-weight manner, this framework bridges the gap between automated parsing and the subjective semantic needs of system engineers. While it requires an initial human effort, the theoretical rigor and empirical results demonstrate its effectiveness as a scalable, high-accuracy solution. Future work may explore automating feedback through Large Language Models (LLMs) to further reduce human intervention.
Summary updated