When a language model reads a long contract and reports what it contains, it often invents clauses that are common in contracts generally but absent from this one. LegalHalluLens traces this to the model's training: what it has seen in many other contracts overrides what the document in front of it says. A popular remedy is debate, where several model copies critique each other's answers. Debate is expensive, though. If every agent reads every other agent, cost grows with the square of the number of agents. It also fails when the original answer is confidently wrong, because a model that has absorbed a strong prior does not doubt itself.
This note proposes a debate design built around two ideas. Agents do not talk to each other directly. Each agent sends its critique to a small set of shared hubs, one per claim type (obligation, numeric, temporal, and so on), so cost grows with agents times hubs rather than agents squared. The second idea decides which claims get debated at all. We do not trust the answering model's own confidence. Instead, a model pre-trained on contracts compares two likelihoods: how plausible a claim is with no document in view, and how plausible it is given the actual contract text. A claim that is plausible from priors alone but is not supported, or is actively undercut, by the document is the signature of the invention failure. Only those claims are sent to the hubs, and the hubs for high-risk claim types get more critics.
Core Claim
In long-document technical extraction, hallucinations cluster by claim type and are driven by pre-training priors. Debate should therefore be organized around claim types rather than around agents, and it should be triggered by a measure of how strongly a claim rests on priors rather than by self-confidence. Concretely, we claim three things. First, routing critiques through claim-type hubs keeps cost roughly linear in the number of agents while matching or beating fully connected debate on fabricated-clause rate. Second, a prior-override score computed by a domain pre-trained model catches confidently wrong inventions that self-confidence gating misses. Third, the domain pre-training is what makes that score work. A general model scoring the same quantity should be worse, because its priors about contracts are weaker and noisier.
For a candidate claim $c$ answering clause-category question $q$ about document $d$, the prior-override score under a domain model $p_D$ is
$$ r(c) = \log p_D(c \mid q) - \log p_D(c \mid d, q), $$
Large positive $r$ means the claim is more expected from contract boilerplate than from this contract. Claims with $r$ above a threshold tuned on a development split go to debate.
Positioning Against Prior Work
The motivating evidence comes from two papers. LegalHalluLens measures hallucination on CUAD. It finds an aggregate rate near 52% that hides a 38–40 point gap between obligation/numeric claims (65–74%) and temporal claims (29–35%), and it shows that a fixed six-role debate pipeline with asymmetric gates cuts fabricated clause detections by 45%. Its pipeline treats every claim alike and is a fixed sequential chain. We keep its typed failure analysis but use the type structure to decide where debate happens and how large it is. The second is TruthfulQA, which shows that larger models more faithfully reproduce common falsehoods they absorbed in training. This is the general form of the prior-override problem.
Three papers supply the sparse-debate methods we compare against and build on. Sparse communication topology for debate replaces all-to-all debate with fixed rings of at most six agents, cutting input tokens by over 40% on math and improving MATH accuracy by 2% by delaying premature agreement. LASE gates debate on the leader's own token confidence and uses a star around a leader. It reaches 65.2% on GPQA-Diamond against 62.1% for standard debate and 62.7% for majority voting. By its own account, it breaks when the leader is confidently wrong. Memory masking for debate filters low-confidence or high-perplexity messages between rounds, but the rounds themselves stay all-to-all. None of these organizes communication around the content of the claims. All three rely on signals from the general model's own confidence, which is exactly the signal that fails under prior override.
The hub structure is borrowed from multi-agent reinforcement learning. f-MAT connects agents only to a small set of latent factor nodes, reducing attention cost from quadratic to factor count times factor size. It reports an 18x inference speedup over the Multi-Agent Transformer on 144-agent grids, but its factors are static and its setting is non-linguistic. VAIN similarly avoids pairwise networks with factorized attention, and it warns that such approximations degrade when interactions are dense and nonlinear. This warning constrains our design. If critiques of one claim type routinely depend on another, as when an obligation's meaning hinges on a date, separate hubs could lose information. We therefore test a variant with a cross-hub exchange round.
Domain pre-training supplies the scorer. BioBART shows that continued generative pre-training on in-domain text improves generation (MeQSum Rouge-L 53.15 vs. 51.05). Its finding that sentence permutation hurts biomedical pre-training tells us objective choice matters. CUAD is a constraining result: 8GB of legal pre-training improved extraction by only about 3%. We take this seriously. We do not use domain pre-training to make the extractor better. We use it to give us a sharper model of what contracts usually say, which is precisely what an invention detector needs.
Research Program
We build four components. The first is a contract-domain scorer. We continue pre-training a small open generative model on commercial contract text with text-infilling, holding out every CUAD contract and near-duplicates to avoid leakage. Following BioBART, we skip sentence permutation.
The second is a claim decomposer. It breaks each extractor output into atomic claims labeled with CUAD's 41 clause categories and with LegalHalluLens's claim types.
The third is the hub debate. For each document, a hub is created only for claim types that have at least one flagged claim, which makes the factor set dynamic per document, unlike f-MAT's fixed factors. Critics are assigned to hubs in proportion to the measured hallucination rate of that type on the development split. Each critic sees the flagged claim, the retrieved contract windows, and the hub's current summary. It returns a supporting or refuting critique with a quoted span. The hub, a single model call, merges critiques into a bounded-length summary that is broadcast back to the hub's critics. A final judge per hub accepts or rejects.
Per round, cost is roughly $N$ critic messages plus $m$ hub summaries, where $N$ is the number of critics and $m$ the number of active hubs. Fully connected debate instead requires $N(N-1)$ message reads. The fourth component is instrumentation that logs tokens, wall-clock time, and the LegalHalluLens Risk Direction Index, its measure of whether errors lean toward invention or omission. We need this because a method that suppresses inventions by deleting true clauses has not solved anything.
Baselines
The naive reference is the single extractor with no verification, run with the same backbone. Its hallucination rate by claim type anchors everything. LegalHalluLens reports aggregate rates near 52% for comparable models, and we expect a similar profile.
The strongest existing method on this exact question is the calibrated six-role debate of LegalHalluLens, which reduced fabricated clause detections by 45% on a matched subset. We reimplement it on the same backbone and same retrieved windows, with its role prompts and gates. We give it the same total token budget as our method by letting it run on every claim, since that is how it was designed. Beating it means a fabricated-clause reduction at least as large as 45% relative to the naive extractor, reached with fewer tokens, and without a shift of the Risk Direction Index toward omission beyond a tolerance fixed in advance.
We also run fully connected debate and the sparse ring topology at matched agent counts. These two give the cost-scaling comparison, and the ring is the strongest prior answer to quadratic cost in debate. We further run LASE, which gates on the extractor's minimum log-probability and uses a star topology. This is the direct competitor for the gating component, since it reaches 65.2% on GPQA-Diamond against 62.1% for standard debate. Finally, we run majority voting over independent extractions at equal token cost.
Three ablations isolate the key ingredients. The first replaces the domain scorer with the untuned general model computing the same $r(c)$. This isolates the contribution of domain pre-training. The second replaces $r(c)$ with the extractor's own confidence while keeping hubs. This isolates the gating signal. The third keeps the gating but sends flagged claims to one undifferentiated pool of critics instead of typed hubs. This isolates the claim-type structure.
All methods share the backbone, the retrieval windows, the development split for threshold tuning, and an equal number of tuning trials. Results are averaged over three seeds, which addresses LegalHalluLens's single-run limitation.
Why not just…?
Why not just check whether the extracted quote appears in the contract? A string-match verifier is cheap, and it will catch fully fabricated quotes. This is a real threat for the subset of hallucinations that are verbatim inventions, and we include it as a baseline. It cannot catch the obligation and numeric errors that dominate LegalHalluLens's 65–74% rates, where the quoted text exists but its obligor, amount, or condition is misattributed. We will report what fraction of hallucinations survive string match. If that fraction is small, the debate machinery is unjustified for CUAD and the claim shrinks to the residue.
Why not just use majority voting? Voting costs grow linearly and need no communication. On GPQA-Diamond, however, it trailed LASE (62.7% vs. 65.2%). On medical notes, debate between differently grounded agents reached 78.8% versus 72.6% for the best single agent. More fundamentally, prior-driven inventions are shared across samples from the same model, so voting will tend to agree on them. The equal-token voting baseline tests this directly on fabricated clauses.
Why not just gate debate on the model's own confidence, as LASE does? LASE's authors identify confident wrong answers as the case their gate misses, and prior override is exactly that case. The ablation that swaps $r(c)$ for self-confidence while holding hubs fixed settles this. If self-confidence catches as many fabrications, the domain scorer is unnecessary and we will say so.
Why not just use a general model to compute the prior-override score? This may well work. The CUAD result that legal pre-training adds only about 3% to extraction makes it a serious possibility. The general-scorer ablation decides it. Our prediction is that the domain scorer's advantage concentrates in obligation and numeric claims, where contract boilerplate is most specific.
Why not just run the existing six-role pipeline on every claim? It works, but it spends the same effort on temporal claims (29–35% hallucination) as on obligation claims (65–74%), and its cost grows with document length times roles. Our matched-budget comparison measures whether type-aware allocation buys the same reduction for less.
Proposed Experiments
The main testbed is CUAD v1.0: 510 contracts, 41 categories, and 13,101 expert annotations. We use contract-level train, development, and test splits, with LegalHalluLens's matched-subset protocol for fabricated clause detections. We report per-claim-type hallucination rate, the Risk Direction Index, and extraction AUPR. The extraction reference point is DeBERTa-xlarge's 47.8% AUPR reported in CUAD, which confirms that verification is not destroying recall. Cost is measured in tokens and wall-clock time.
The first decisive check is a scaling sweep that grows the number of critics from 3 to 24. It should show token cost growing roughly linearly for hub debate and quadratically for fully connected debate. Fabricated-clause reduction should stay flat or improve for hubs, while the ring topology, which was never tested beyond six agents, is measured on the same sweep.
The second decisive check is a stratified analysis of fabrications the extractor produced with high confidence. On this slice, $r(c)$ gating must recover substantially more than LASE-style gating. If it does not, the core mechanism fails.
The third decisive check is the cross-hub variant. It tests whether VAIN's warning about dense interactions applies, which would show up as obligation errors that depend on dates being missed by isolated hubs.
As a transfer test, we apply the same design to the 500-case MS collection derived from MedQA, using BioBART as the domain scorer. The comparison target is the 78.8% accuracy of the Mayo/WebMD debate, at a cost that paper puts at $30 per evaluation versus $5 for a single agent. Matching that accuracy at substantially lower cost would show that the idea is not specific to contracts. Failure would localize it to domains whose boilerplate priors are as strong as legal drafting's.
References
- LegalHalluLens: Typed Hallucination Failure Modes and Calibrated Multi-Agent Debate Mitigation
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Improving Multi-Agent Debate with Sparse Communication Topology
- Efficient Multi-Agent Reasoning via Confidence-Guided Adaptive Debate
- Multi-Agent Debate with Memory Masking
- Towards Efficient Collaboration via Graph Modeling in Reinforcement Learning
- VAIN: Attentional Multi-agent Predictive Modeling
- BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model
- CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review
- Error Detection in Medical Note through Multi Agent Debate