Abstract
System logs are critical for debugging and monitoring, yet their unstructured nature necessitates complex parsing into structured schemas for large-scale analysis. Traditional parsers rely on fragile, hand-crafted regular expressions or token-centric statistical models that fail on the "mixed-content" logs typical of industrial domains like Electronic Design Automation (EDA). This paper presents SchemaCoder, the first fully automated, end-to-end framework that leverages Large Language Models (LLMs) to synthesize parsing code. By introducing a novel Residual Q-Tree Boosting mechanism, SchemaCoder iteratively refines its extraction logic to handle complex nested structures and rare edge cases. Experimental results demonstrate a 21.3% accuracy improvement on standard benchmarks and a 57.9% boost in code generation success for complex EDA logs compared to state-of-the-art AI coding agents.
Introduction
As software systems grow in complexity, the volume and variety of logs they generate have exploded. Converting these logs into structured data—identifying templates and extracting variables—is a prerequisite for automated root-cause analysis and performance monitoring. However, the industry faces a "Regex Bottleneck," where engineers must manually maintain thousands of regular expressions that break as software evolves. Furthermore, modern logs, particularly in the EDA domain, are no longer simple sequences of text; they are a dense weave of command calls, progress updates, and nested performance tables that traditional parsers cannot navigate.
Methodology
SchemaCoder shifts the paradigm from static rule-matching to an optimization-based code generation approach. The framework operates through four primary stages:
1. Cluster-Based Chunk Selection
To manage the high latency and cost associated with processing multi-gigabyte log files via LLMs, SchemaCoder employs an embedding-based sampling strategy. By using k-means clustering on log line embeddings, the system identifies representative "chunks" that capture the diverse structural patterns within the file, reducing the input volume by orders of magnitude.
2. Hierarchical Q-Tree Pattern Recognition (Q-TREE-PR)
Rather than relying on a single monolithic prompt, SchemaCoder utilizes a three-layer "Question Tree" to guide the LLM's reasoning:
- Exploration Layer: The model identifies high-level structural patterns or anomalies.
- Segment Selection Layer: The model pinpoints specific lines corresponding to the identified patterns.
- Pattern Code Generation Layer: The model synthesizes Python code (utilizing regex) to extract data from those specific segments.
3. Textual-Residual-Guided Evolutionary Optimizer
The generated parser is treated as an evolving program. Using the OpenEvolve algorithm, SchemaCoder iteratively improves the code based on feedback from an automated evaluator, ensuring the synthesized logic is both syntactically correct and semantically accurate.
4. Residual Q-Tree Boosting
Inspired by gradient boosting techniques like XGBoost, SchemaCoder introduces a "Residual" mechanism. After an initial parsing pass, the system identifies log lines that the current code failed to process (the residuals). It then constructs a new Q-Tree specifically targeting these failed cases. The resulting specialized code is merged into the main parser, allowing the system to systematically conquer rare edge cases and complex nested tables.
Experimental Evaluation
Datasets
SchemaCoder was evaluated on two distinct targets:
- LogHub-2.0: A standard benchmark of 14 system log types (e.g., HDFS, Apache, Thunderbird).
- Industrial EDA Logs: Real-world logs from chip design tools characterized by tabular data and multi-stage execution flows.
Key Results
- Accuracy: On LogHub-2.0, SchemaCoder achieved an average 21.3% improvement in parsing accuracy over existing statistical and LLM-based baselines.
- EDA Dominance: In the challenging EDA domain, SchemaCoder outperformed Cursor, a leading AI coding agent, by 57.9% in the
pass@kmetric, demonstrating its superior ability to reason about complex, non-linear log structures. - Scalability: The chunk-based selection allowed the system to process massive files while maintaining high precision, effectively "slashing" LLM token usage by two to three orders of magnitude.
Conclusion and Future Work
SchemaCoder represents a significant leap forward in automated log analysis by treating schema extraction as a residual-guided code generation problem. While the optimization process is currently computationally intensive (averaging four hours per run), the resulting parsers are highly efficient and robust. Future research will focus on integrating domain-specific feedback loops to further enhance reliability in specialized industrial sectors and exploring the framework's performance on smaller, open-source LLMs to reduce operational costs.
Summary updated