Does the Longer Argument Win? Verbosity as a Hidden Vote in Fact-Checking Debates

Multi-agent debate is a popular way to catch hallucinations, meaning fluent but false statements from a language model. Several model instances argue for and against an answer, and a judge model decides who is right. Two findings in the literature suggest this setup has a quiet failure mode. First, judges prefer longer text. Second, models write more when they are unsure and wrong, a behavior called verbosity compensation. Put these together and the debater who is least sure of the facts may produce the most words, and the judge may reward those words. Debate would then pass along the very hallucinations it is supposed to catch, and scoring the final answer would never show it.

This proposal measures that effect causally, then tests a fix. The measurement is simple. For each debate we build counterfactual transcripts in which one side's argument is padded or compressed without adding or removing any factual claim, and we count how often the judge's verdict flips. We run this separately at each level of a task hierarchy: retrieving a fact, relating facts, and composing a multi-step answer. We expect length to mean different things at different levels. At the retrieval level, extra length mostly signals uncertainty. At the composition level, some of it is genuine derivation. That is why a single length cap should help at one level and hurt at another. The fix follows from this. The judge sees each turn as a list of the atomic claims it makes, so style-length disappears. Each agent's length is compared with what is normal for that level of the task, and an excess is flagged as a possible sign of uncertainty.

Core Claim

In debates over factual questions, the length of an argument acts as a vote that is separate from its content. The size of that vote depends on where the question sits in a perception-to-composition hierarchy, and a length control that knows the question's level beats both no control and a uniform cap.

The claim is new along two axes. Judging with Many Minds shows that sequential debate amplifies verbosity bias in LLM-as-judge settings, and that parallel meta-judges resist it. It measures judge consistency on MT-Bench-style preference data, though, not whether the debate's factual verdict is correct. LegalHalluLens builds typed, calibrated debate for contract extraction and cuts fabricated clauses by 45%. It never measures or controls length, and it covers one domain with one backbone. Verbosity ≠ Veracity establishes verbosity compensation, reporting a 27.61% performance drop for verbose answers on Qasper. That work is single-agent, and it neither asks what happens when a verbose uncertain agent faces a judge nor splits the effect by task level. Our work sits in the gap those three papers leave, and its central measurement is a counterfactual intervention on length, not a correlation.

Research Program

Task hierarchy used as a control signal, not a report card. A known limitation of hierarchical benchmarks is that they diagnose failures after the fact and are costly to annotate. We avoid the annotation cost by using benchmarks whose structure already defines the levels.

Around the World in 24 Hours has four templates of increasing difficulty with exact answers:

  • verifying the time in a known place;
  • time arithmetic within one place;
  • time across places;
  • multi-shift arithmetic across places.

It also documents a "complexity cliff": accuracy falls to 25.4% for Llama 3-70B on the hardest template. ToolBeHonest has three levels: detecting whether a task is solvable, identifying the bottleneck sub-goal, and specifying the missing tool. Notably, it reports a "verbosity trap" in open-weight models alongside a positive length–accuracy relation in proprietary ones. That mixed sign is exactly what a level- and model-dependent account predicts. HaluEval question answering and TruthfulQA supply mostly retrieval-level factual items. For these, we assign levels with a lightweight classifier trained on the level labels of the two structured benchmarks. We report how the results change when predicted levels replace gold ones.

Debate protocol. The debate follows the proponent/skeptic/judge pattern of the Markov-chain debate detector and FORD. A candidate answer is produced by a base model. A proponent defends it, a skeptic attacks it, and a judge outputs a verdict of correct or hallucinated, plus a final answer.

We run open-weight models so that we can log token-level uncertainty. We include matched pairings (equal-strength agents) and mismatched pairings (unequal strength). The mismatched case matters because Xiong et al. show that a weaker, stubborn debater can derail a stronger one. We test whether verbosity is the channel through which that happens.

The length-counterfactual measurement. For each debate transcript, a rewriting model produces two versions of one side's turn:

  • a padded version, with restatements, hedges, and connective elaboration;
  • a compressed version.

In both cases the rewriter is instructed to preserve the claim set. We check preservation by extracting atomic claims before and after rewriting, in the style of the decomposition step in RT4CHART. We discard any pair whose claim sets differ under two independent extractors.

The judge then re-rules on each counterfactual transcript. Our main quantity is the verbosity flip rate: the fraction of verdicts that change when only length changes, reported per level, per model pairing, and for padding versus compression. We also check the verbosity-compensation half of the story directly. We ask whether debaters defending wrong answers write longer turns than debaters defending right ones at the same level, and whether the extra length appears alongside the early-token perplexity spikes that Verbosity ≠ Veracity reports.

The intervention: claim-ledger judging with a level-conditioned excess flag. The intervention has two parts.

  1. Claim ledger. Each turn is converted into a ledger of deduplicated atomic claims. Each claim is tagged as either grounded in a retrieved passage or asserted from memory, borrowing the evidence/memory split of DRAG. The judge rules on the ledgers, not the prose.
  2. Excess flag. For each level we estimate the typical turn length of confident, correct debaters on a held-out calibration split. The ledger also carries a flag when an agent's raw turn exceeds that level's norm by a set quantile.

The judge is told that the flag may indicate uncertainty. This turns verbosity from a hidden vote into an explicit, level-appropriate piece of evidence.

Baselines

Naive references. The "do nothing" reference is the base model's candidate answer with no verification. The second reference is standard sequential three-role debate with unconstrained turns. The flip-rate study measures the latter's susceptibility, and every intervention is compared against it.

Strongest existing hallucination detectors. The strongest existing debate method for detection is the Markov-chain Trust/Skeptic/Leader debate. It is evaluated on HaluEval but I found no reported result for it on Lacuna. We therefore rerun it with the same backbone, the same retrieval corpus, and the same call budget.

As published reference points on HaluEval, the file reports:

  • ChatGPT detecting question-answering hallucinations at 76.83% accuracy with retrieval (HaluEval);
  • G2LDetect reaching 72.98% F1 on HaluEval-Summary with GPT-4.

Our numbers use different backbones, so these anchor plausibility rather than serve as head-to-head targets.

Strongest verbosity-resistant judging. The strongest bias-resistant judging method is the parallel meta-judge from Judging with Many Minds, which that paper finds resists verbosity bias in synthesis mode. It is the most direct competitor on the length question. I found no reported factual-accuracy result for it on Lacuna, because it was evaluated only on judge consistency.

Fairness conditions. Every method receives the same number of model calls per question (capped near the roughly ten calls per question that DRAG uses), the same total token budget, and the same prompt-tuning budget on a shared development split.

Ablations. These isolate the key ingredients. Each removes or replaces one component of the full method:

  • claim ledger without the excess flag;
  • excess flag without the ledger;
  • a uniform per-turn token cap tuned on the development split;
  • an excess flag computed from a single global length norm rather than a per-level norm.

What counts as beating the baselines. The proposal succeeds if two things hold. First, the full method must reduce the verbosity flip rate at least by half relative to standard debate. Second, it must improve verdict accuracy over the parallel meta-judge and the Markov-chain debate. This improvement must hold at the retrieval level and must not lose accuracy at the composition level, where the uniform cap is expected to hurt.

Why not just…?

Why not just cap every debater's turn at a fixed number of tokens? This is the most serious threat, and it may well work at the retrieval level. The evidence against it as a general fix is indirect but consistent:

  • length normalization in SimPO costs some accuracy on GSM8K math;
  • ToolBeHonest finds length helping proprietary models;
  • both suggest that some length is real work.

The uniform-cap ablation settles it. If a tuned cap matches the full method on the hardest time-and-place template and at ToolBeHonest's third level, the level-conditioned machinery is unnecessary, and we would report the cap as the recommendation.

Why not just replace sequential debate with a parallel meta-judge? Judging with Many Minds shows that parallel meta-judges resist verbosity bias, but it shows this only for preference judgments. A parallel judge still reads each debater's prose, so padding inside a single turn can still sway it. We run the meta-judge through the same counterfactual test. If its flip rate is already near zero on factual verdicts, the ledger adds little and the proposal narrows to the finding itself.

Why not just subtract length from the judge's score after the fact? Post-hoc reward calibration removes the length trend from reward scores by local regression, and it is cheap. It treats length as pure noise, however. Verbosity compensation says length is partly signal, because excess length predicts that the debater is wrong. Regressing length out throws that signal away. We include regression-calibrated judge scores as an additional baseline. The flag-only ablation tells us whether keeping the signal matters.

Why not just skip debate and use retrieval-grounded single-model checking? This is a fair question, given how costly debate is. We include retrieval-augmented single-judge checking at matched token cost. If it beats every debate variant, that result would itself argue against debate for factual verification at these levels.

Proposed Experiments and Decisive Checks

The first decisive check is whether the phenomenon exists. The verbosity flip rate in unconstrained debate should be well above the rate from content-free rephrasing of equal length. We use that rephrasing rate as the noise floor. The flip rate should also be larger when the padded side defends the wrong answer. If flip rates sit at the noise floor across all levels, verbosity is not a meaningful vote in factual debate, and the proposal ends as a useful negative result.

The second check is level dependence. Flip susceptibility and the uniform-cap penalty should move in opposite directions across the four time-and-place templates and the three ToolBeHonest levels. If both are flat across levels, the hierarchy adds nothing and the method collapses to the claim ledger alone.

The third check is transfer to unstructured factual data. Using classifier-predicted levels on HaluEval question answering and TruthfulQA, we measure the accuracy loss relative to gold levels. This tells us whether the approach works without hand-built hierarchies, which addresses the annotation-cost problem that motivates it.

Across all three checks we report:

  • verdict accuracy and F1;
  • flip rate;
  • turn length per level;
  • calls and tokens per question.

We run three seeds each. We also repeat the full study with a mismatched debater pairing to test whether verbosity is how weaker agents derail stronger ones.