When a large language model is used as a judge to grade dialogue summaries, the hardest thing it has to check is also the thing summarizers most often get wrong: who committed to what by the end of the conversation. People propose, revise, retract, and hand off tasks across many turns. A summary that reports Tom's first offer instead of the time Anna finally accepted reads fluently and uses all the right names, and it is still false. We propose to build dialogues from a hidden, machine-readable record of each speaker's commitments and how those commitments change over the conversation. Because the correct final state is known exactly, we can plant specific, labeled errors in summaries and measure whether judges catch them. We can also make each error harder by increasing how many revisions stand between the first mention and the final value. Because every dialogue is freshly generated, a judge cannot succeed by recognizing a memorized benchmark.
The expected payoff is a clear answer to a question the field currently cannot settle. Asking judges to reason step by step before scoring (chain-of-thought, or CoT) improves agreement with humans on general benchmarks. The same step-by-step reasoning also makes judges accept confident but false explanations. We do not know which effect dominates when the judge has to track conversational state. We hypothesize that free-form CoT helps on shallow errors, such as a swapped speaker name in a single-turn fact, but does not help, or makes things worse, on deep errors where a commitment has been revised several times. We further hypothesize that a judge forced to first write out a turn-cited ledger of final commitments closes most of that gap without becoming overly harsh on correct summaries.
Core Claim
LLM judges of dialogue summaries fail mainly on state-tracking errors, and this failure is hidden by static benchmarks for two reasons: they do not control error depth, and their dialogues and reference summaries are likely present in the judges' pretraining data. Making the judge reconstruct the final commitment state, citing turn numbers for each entry, before comparing it with the summary is the ingredient we expect to matter. Unconstrained CoT is the control. The contribution has two parts: a way to measure this failure that resists contamination, and a test of whether structured intermediate reasoning, rather than more reasoning, is what fixes it.
Relation to Prior Work
Four lines of work motivate this proposal.
Dialogue summarization research has documented the error we target. DialogSum, a real-life spoken dialogue dataset, reports that models misattribute actions and agreements to the wrong speaker. DialSummEval, which meta-evaluated 18 metrics against 12,600 human ratings on SAMSum, shows that word-overlap metrics are blind to such consistency errors. Neither can say how deep an error is, because real dialogues do not come with a ground-truth commitment record. DialSummEval also predates prompted LLM judges and uses 100 static public dialogues.
The design follows DyVal, which generates reasoning problems from directed graphs with controllable depth. DyVal showed that models strong on static benchmarks collapse on generated ones, and it names discourse-heavy domains as an open extension. Our commitment record plays the role of DyVal's graph, and revision depth plays the role of graph depth. What is new is rendering that structure as natural multi-party talk and using it to test judges rather than solvers. The argument-graph linearization in ConvoSumm shows that structured representations of conversation are usable by models. We reuse this idea in reverse, going from structure to dialogue.
Judge-bias work constrains the design. Gaming the Judge shows that web-agent judges believe fabricated reasoning traces, with false positive rates rising 20–30 absolute points. It also shows that "be adversarial" prompting imposes a robustness tax on correct outputs. We carry both findings into dialogue: one condition attaches a fabricated rationale to the summary, and every judge is scored on false rejections as well as missed errors. The Silent Judge shows that judges' written rationales never mention the cues that actually drive their verdicts, a 0% acknowledgment rate. For that reason we do not trust a CoT judge's explanation as evidence of verification. Instead, we check the turn citations in the ledger mechanically. Preference Leakage shows that judges favor outputs stylistically related to their own model family, with a leakage score up to 37.1%. This is a direct threat to any benchmark whose dialogues are written by an LLM, so we render every dialogue with several model families and treat renderer family as a measured factor.
CoT work supplies both the method and a warning. TRACT shows that CoT combined with regression-aware fine-tuning raises judge–human Pearson correlation to 0.650, compared with 0.591 for Prometheus-2-7B. Those gains are on general single-turn benchmarks, not long conversations. Reflect, Not Reflex shows that forcing an explicit intermediate inference about the dialogue improves responses. Our ledger is the evaluation-side counterpart of that idea. Local Coherence or Global Validity? and Unveiling Confirmation Bias in Chain-of-Thought Reasoning show that fluent reasoning traces can be globally wrong and can entrench initial mistakes. They predict that free CoT will fail precisely at deep revisions, which is the hypothesis we test.
Research Program
The generator. The generator samples a commitment record: a small set of speakers, objects under negotiation (times, amounts, tasks, owners), and a sequence of events such as propose, counter, accept, retract, and delegate. Each event is attached to a speaker and a turn. The depth of a commitment is the number of revisions between its first mention and its final value. Depth is our main difficulty knob and is set per dialogue from 0 to 5. Filler turns (greetings, tangents, backchannels) control the distance between the final revision and the end of the conversation.
A rendering model turns the record into informal dialogue. Rendering is done separately with models from at least three families, and names, settings, and phrasing are randomized. A second, independent model extracts the record back from each rendered dialogue. Dialogues whose extraction disagrees with the source record are discarded. A human-annotated sample of several hundred dialogues checks both extraction fidelity and naturalness, with naturalness rated against real DialogSum dialogues.
Planted errors. For each dialogue we produce a correct summary written from the record and paraphrased by a model from a family not used as a judge. We also produce variants with exactly one planted error of a known type: speaker swap, stale value (a superseded proposal reported as final), fabricated consensus (a retracted or never-accepted item reported as agreed), dropped owner, and invented entity. Since each variant differs from the correct summary by one controlled edit, any change in the judge's verdict is attributable to that error.
Isomorphic twins. To measure contamination, we extract commitment records from a sample of real SAMSum and DialogSum dialogues, correcting them by hand, and re-render each as a twin. A twin has the same structure but new names and surface text. The gap between how a judge performs on originals and on their twins, holding error type and depth fixed, estimates how much judging on public benchmarks benefits from familiarity. The same comparison applied to summarizers tells us whether ROUGE-era scores, such as BART's 47.28 ROUGE-1 on DialogSum reported in DialogSum, survive re-rendering.
Judge conditions. Each judge model is run in four conditions:
- Direct: it outputs a score with no reasoning.
- Free CoT: it reasons in free form, then scores.
- Ledger: it first writes the final state of each commitment with the turn numbers that establish it, then compares that state claim by claim against the summary. Its citations are checked automatically against the source record.
- Ledger with shuffled citations: the ledger's turn numbers are replaced with wrong ones before the comparison step. This tests whether the judge actually uses its ledger.
Each condition is crossed with a rationale attack. In the attack, the summary arrives with a plausible summarizer explanation that asserts the erroneous fact, for example: "Both agreed to Friday in turn 14." This follows the manipulation design of Gaming the Judge, moved from agents to dialogue.
Judges include GPT-4o and Gemini-2.5-Flash, both examined in the cited bias papers, plus open Qwen-2.5 and Mistral-7B judges, including a TRACT-trained Mistral where available.
Baselines
The naive reference point is ROUGE-L of each summary against the correct generated summary, thresholded to flag errors. It shows how much of the planted-error signal is visible to word overlap alone. Following DialSummEval, we expect it to be near chance on speaker swaps and stale values, because those edits change few words.
The strongest non-LLM competitor is QuestEval. It reached system-level correlation r=0.85 with human consistency judgments on SAMSum in DialSummEval, which is the best consistency figure in our material. We run it unchanged on the same dialogue–summary pairs. Its question-answering design might catch stale values on its own, which makes it a serious rival to the ledger judge.
The strongest LLM-judge competitor is the Free CoT condition using the same underlying model as the Ledger condition, with the same prompt budget and the same few-shot examples. It also gets matched output length: we allow Free CoT as many reasoning tokens as the ledger uses. This rules out "more tokens" as the explanation for any ledger advantage. For a trained judge we also run a TRACT-style Mistral-7B, whose reported average Pearson correlation is 0.650 on general judge benchmarks in TRACT. We found no reported result on Lacuna for any prompted LLM judge on DialSummEval or on planted dialogue-summary errors, so this study provides the first such numbers.
The ablation that isolates our key ingredient is the shuffled-citation ledger. If the correct ledger beats Free CoT but the shuffled ledger does not fall back toward Free CoT, then the benefit comes from format rather than from actually tracking state, and we would report it that way.
The ledger counts as beating the competition if it meets two conditions:
- Detection: on depth-3-and-deeper errors, its balanced detection accuracy exceeds both Free CoT and QuestEval by a margin outside bootstrap confidence intervals.
- No robustness tax: its false-rejection rate on correct summaries rises by no more than two points over Direct.
Why not just…?
Why not just perturb real SAMSum summaries instead of generating dialogues? This is the main threat. Swapping names in real reference summaries gives controlled errors on natural text. It has two weaknesses. Real dialogues lack a ground-truth record of revisions, so depth cannot be controlled or even measured reliably. And public references may be memorized, so a judge could flag a perturbed summary simply because it differs from a summary it has seen. We run this alternative directly: planted errors on original SAMSum/DialogSum dialogues versus the same errors on their twins. If detection rates match across originals and twins, and depth can be approximated well enough on real data, the generator adds little. In that case we would narrow the contribution to the ledger judge evaluated on perturbed real data.
Why not just use QuestEval or another question-answering metric? Its 0.85 correlation in DialSummEval is at the system level, over 14 summarizers. That tells us little about catching a single stale value in a single summary. Question-generation metrics also ask about entities mentioned in the summary and may not ask "was this later retracted?" We include it as a full baseline. Its depth curve will show whether it degrades like free CoT or holds up like the ledger.
Why not just tell the judge to think step by step or be skeptical? Gaming the Judge found that manipulation-aware prompting makes judges fail correct outputs. Local Coherence or Global Validity? shows that fluent traces can be globally invalid. The Free CoT and skeptical-prompt conditions, scored on both misses and false rejections, test this directly. If free CoT matches the ledger at matched length, our central hypothesis is wrong, and we will say so.
Why not just fine-tune a judge on dialogue consistency labels? A trained judge is useful, and a TRACT-style model is in the baseline set. But fine-tuning on existing labels inherits their lack of depth control, and possibly their contamination. We additionally fine-tune a judge on generated training data from held-out depths 0–2 and test it at depths 3–5. This shows whether training generalizes to deeper revisions or only memorizes shallow patterns.
Decisive Checks
The first check is the depth curve: detection accuracy versus revision depth for each judge condition, broken down by error type. The hypothesis is supported if Free CoT's curve falls with depth at least as steeply as Direct's while the ledger's curve stays flatter.
The second check is the rationale attack. We compare the rise in false acceptances of wrong summaries against Gaming the Judge's 20–30 point increase for agents. We then test whether the ledger, which cites turns before reading the summary, reduces that rise.
The third check is the twin gap. A consistent advantage on originals over twins for the same judge would be evidence that static dialogue benchmarks overstate judge competence.
The fourth check is renderer-family effects, measured as in Preference Leakage. If a judge is more lenient on dialogues and summaries rendered by its own family, the testbed must balance families before it can be trusted.
Finally, the ledger judge must be run on DialSummEval's human ratings. It should not lower correlation with human consistency judgments on real data relative to Free CoT; otherwise, gains on generated dialogues do not transfer.
References
- DialogSum: A Real-Life Scenario Dialogue Summarization Dataset
- DialSummEval: Revisiting Summarization Evaluation for Dialogues
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- ConvoSumm: Conversation Summarization Benchmark and Improved Abstractive Summarization with Argument Mining
- Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- TRACT: Regression-Aware Fine-tuning Meets Chain-of-Thought Reasoning for LLM-as-a-Judge
- Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality
- Local Coherence or Global Validity? Investigating RLVR Traces in Math Domains
- Unveiling Confirmation Bias in Chain-of-Thought Reasoning