Evidence-Grounded Debate as a Gate for LLM Log-Parser Caches

Software systems emit billions of text log lines, and almost every automated monitoring tool first has to split each line into a fixed developer-written skeleton (the template, e.g. Connection from <*> failed on port <*>) and the values that change at runtime (the parameters). Recent parsers do this with large language models (LLMs). To stay fast, they store every template the model produces in a cache, so the model only runs on lines the cache has never seen. This design has an unexamined failure mode. When the model hallucinates a template, either inventing a constant or turning a meaningful status word like ERROR_TIMEOUT into a wildcard, the bad template enters the cache. From then on it silently mislabels every later line that matches it.

We propose to guard the cache with a short, adversarial check that runs only when it is needed. When the model proposes a template that the log stream's own statistics cannot confirm, two LLM agents argue over the disputed tokens. One argues each token is fixed developer text. The other argues it is a runtime value. Each argument counts only if it cites real log lines that a deterministic checker can verify. If the evidence is too thin, the template is held provisionally instead of committed. The bet is that grounding arguments in checkable lines from the stream does two things. It stops the confirmation bias that limits single-model self-correction. It also stops the "distraction effect", where one confidently wrong agent drags a debate off course. Together these should make LLM parsers much more robust when their prompts and caches are polluted by near-miss logs.

Core Claim

Most LLM template errors that matter happen in two places. The first is rare templates, which have too few instances for frequency statistics to separate constants from variables. The second is low-cardinality tokens such as status words, which vary across lines yet belong in the template. Both errors get worse when the prompt contains distracting but plausible demonstrations or interleaved logs from unrelated processes.

We claim that a cache admission gate can recover most of these errors at a small fraction of the cost of running debate on every cache miss. The gate has three steps. Statistics settle the easy cases. A debate settles the ambiguous ones, and its arguments must be backed by verifiable line citations. A provisional state handles the cases where evidence is still too thin. The novelty is not "debate for log parsing" in general. It is debate whose only admissible evidence is the log stream itself, used as admission control on a template cache, and evaluated under deliberate contextual distraction.

Where This Sits

The parsers we build on all trade per-line LLM calls for caching. LILAC pairs nearest-neighbour demonstration selection with a tree-structured cache that refines neighbouring templates by string matching. AdaParser adds a single-model "Template Corrector". It checks that a template can reconstruct the original line and restores status words that were wrongly abstracted. MicLog meta-trains Qwen-2.5-3B across shot counts and uses a two-tier cache. LLMLog prunes demonstrations because extra examples add noise and trigger hallucinations, which is direct motivation for our distraction setting.

These systems supply our base parser and caching machinery. None of them asks whether what enters the cache is correct, beyond a reconstruction check that a hallucinated wildcard passes trivially. Adaptive Memory Admission Control For LLM Agents shows that agents accumulate hallucinated content into long-term memory without an admission filter. We carry that idea to template caches.

The debate machinery comes from two papers that also constrain how we can use it. The Markov-chain debate detector sends atomic claims through Trust, Skeptic and Leader agents and outperforms single-model checks on Factool and HaluEval. However, it grounds its agents in web search, and web search cannot verify proprietary internal logs. We swap that external retrieval for the stream itself. FORD supplies the stance–debate–summary protocol. It also supplies the key negative result: a stubborn weaker model can pull a stronger one below its solo accuracy. Our verifier-filtered arguments are designed as a structural defense against exactly this, rather than relying on a stronger judge.

On the noise side, Breaking Focus shows that coherent but irrelevant context causes large accuracy drops, and that telling a model to "ignore irrelevant information" does not fix them. Its benchmarks are multiple-choice question sets. We build the log analogue of its distractors.

The closest system-level work is mABC. It uses weighted multi-agent voting for root-cause analysis in microservices, assumes parsing has already happened, and does not study distractor logs.

Research Program

The base parser is a LILAC-style in-context LLM parser with a cache. On a cache miss, the model proposes a template. A support retriever then collects up to $N$ recent stream lines that the candidate template matches, plus near-miss lines under the semantic edit distance from LLMLog.

A deterministic evidence checker labels each token position in one of three ways. It is confirmed variable if at least two supporting lines differ at that position. It is confirmed constant if it never varies across at least $m$ supporters. Otherwise it is ambiguous. This covers sparse support, low-cardinality alternation, and conflicts with an existing cached template.

Templates with no ambiguous positions are admitted directly, with no extra LLM cost. For ambiguous positions, an Invariant agent and a Parameter agent each make claims about specific tokens. Every claim must cite line identifiers, such as "lines 17 and 42 differ only at position 5" or "token FAIL co-occurs with a distinct downstream message that SUCCESS never does". The checker verifies each citation mechanically and strikes any that fail. A judge sees only the surviving claims. After at most two rounds, the template is admitted, revised, or quarantined. A quarantined template is used provisionally for matching and re-adjudicated once support reaches $m$. Lines it matched in the meantime are re-labelled if the verdict changes. This explicit "not enough evidence yet" state targets the loss of abstention that heavy clutter causes.

The analysis has three parts. First, we measure how often hallucinated templates enter each baseline's cache, and how many downstream lines each error mislabels before it is noticed. Second, we sort errors into the two regimes named above to check whether they really are where the errors concentrate. If they are not, the gate's design is wrong, and we will report that. Third, we deliberately pair a 3B agent with a larger one to test whether verification removes FORD's distraction effect.

Baselines

The naive reference is Drain with standard regex pre-masking. It is the fastest parser, was best on 9 of 16 original Loghub datasets, and needs no LLM. We tune its depth and similarity threshold per dataset on the same small labelled split the LLM methods use. Our method must beat it clearly on template accuracy, not just on grouping.

The strongest accuracy competitor is MicLog, which reports 97.6% average parsing accuracy (PA) on Loghub-2.0. No MicLog template-level F1 (FTA) was found on Lacuna. The strongest self-correction competitor is AdaParser, which reports 93.8% PA and 87.9% FTA on Loghub-2.0. AdaParser lists LILAC at 87.1% PA and 78.9% FTA, while LILAC reports 81.0% FTA for itself. We will rerun all three under one protocol rather than reconcile those numbers. Each runs with the same backbone where its design allows (Qwen-2.5-3B for MicLog, the same instruction model elsewhere), the same labelled budget, and the same stream order. Because 97.6% PA leaves little headroom on clean data, we treat matching MicLog's clean PA while beating it on FTA and on distracted streams as the bar. A higher clean PA is not required.

Three ablations isolate the key ingredient. Evidence-rule-only admits or rejects by the checker alone and sends ambiguous cases to a single re-prompt. Ungrounded debate is the same two agents and judge with citation verification turned off. Single-agent with evidence lets AdaParser's corrector see the same supporting lines and checker output. All LLM-based variants get a matched token budget per million lines.

Beating the field means three things. On distracted streams, our method's drop in FTA must be significantly smaller than every baseline's, with paired bootstrap over datasets. Its cache poisoning rate must be lower. Its tokens per million lines must stay within a stated multiple of AdaParser's. For cache poisoning, no reported result was found on Lacuna for any baseline.

Why not just…?

Why not just use the token-variability statistics and skip the LLM agents? This is the most serious threat, and the design already concedes most ground to it: confirmed constants and variables are settled by statistics alone. The open question is the ambiguous remainder. Tools and Benchmarks for Automated Log Parsing reports that frequency-based parsers struggle to separate meaningful status values from noise identifiers, and statistics are by definition silent for rare templates. The evidence-rule-only ablation settles this. If it matches the full gate on the ambiguous slice, we will report that the contribution is evidence-grounded admission control and drop the debate.

Why not just sample the parser several times and take a majority vote? Self-consistency is cheaper and does reduce random variance. However, distraction errors are systematic. Breaking Focus induces consistent wrong answers, and repeated samples under the same distracting prompt should share the same error. We run self-consistency at a matched token budget. If its poisoning rate under distraction equals ours, the debate is unnecessary.

Why not just give AdaParser's self-corrector the same evidence? This would separate the benefit of evidence from the benefit of an adversary. The Markov debate and FORD results suggest that a single model tends to defend its first answer, but neither tested structured extraction. The single-agent-with-evidence ablation is the decisive comparison for the debate itself.

Why not just meta-train a better small model, as MicLog does? A better base parser lowers the error rate but provides no check on what enters the cache, and the two approaches stack. We therefore also run the gate on top of MicLog. A gain there would show the gate is complementary to stronger parsers rather than a replacement for them.

Proposed Experiments and Decisive Checks

All experiments use the 14 Loghub-2.0 datasets, replayed in timestamp order as streams. Metrics are PA, FTA, grouping F1, cache poisoning rate, LLM tokens and wall time per million lines.

We build two distracted variants. Demonstration distraction replaces a share of retrieved demonstrations with near-miss templates chosen by edit distance, from the same dataset and from other datasets. Stream distraction interleaves coherent logs from an unrelated Loghub-2.0 system into the stream. This mirrors the domain-specific semantic noise that Breaking Focus left untested. We report clean-to-distracted drops for every method.

The first decisive check is the ambiguous-slice comparison between the full gate, evidence-rule-only, ungrounded debate, and single-agent-with-evidence. If grounded debate does not beat single-agent-with-evidence there, the debate claim fails. The second checks the distraction effect. We pair a 3B agent with a larger model, with and without citation verification, and test whether verification keeps joint accuracy at or above the stronger model's solo accuracy, which FORD found ungrounded debate fails to do. The third measures cost. We report the fraction of cache misses that reach debate. The proposal is only practical if this fraction stays small enough that total time remains within the same order of magnitude as AdaParser's.