Knowing When to Stop Arguing About a Contract

Systems that answer questions about long legal documents increasingly use several language models in a loop. One model retrieves passages, another asks follow-up questions, and a third drafts and revises an answer. Every extra round costs tokens and latency, and the literature shows that extra rounds do not reliably help. In legal judgment prediction, debate accuracy peaks around the second round and then decays. Yet these systems run as if computation were free: each stops after a hand-picked number of rounds. This note asks whether a stopping rule that weighs the expected gain of another round against its cost can do better on contract reasoning. The rule would also be allowed to abstain, meaning it declines to answer, when further checking is not worth it.

The specific bet is that the "early peak" in legal debate is an average over two kinds of questions. Most contract questions are settled by the first retrieved clause, and more rounds only add drift. A minority hinge on an exception or qualification stated pages away from the main obligation. For those, later rounds are exactly what finds the decisive text. If so, no fixed round count is right. A controller that looks at the history of the loop, such as whether the last round surfaced a new section or whether the tentative answer has flipped, should give a better trade-off between accuracy and cost than either a fixed count or a rule that sees only the current state. This follows work on controlling anytime algorithms in operations research, where stopping rules that condition on the full execution path beat memoryless ones.

Core Claim

In a multi-agent contract question-answering loop, the value of one more round of interrogation depends on the path the loop has taken, not just on its current answer or confidence. The clearest example is whether the latest round retrieved evidence from a part of the document not seen before. A stopping controller estimated from logged trajectories, with three actions (continue, answer now, abstain), should reach a given accuracy with fewer tokens than fixed-round and memoryless stopping. It should spend most of its extra rounds on items whose evidence is spread across distant clauses.

The objective is the per-item net utility of Halpern and Pass's costly-computation view, applied to a stopping time:

$$U(\tau, a) = R(a) - \lambda \, C(\tau)$$

Here $\tau$ is the round at which the loop stops, $a$ is the final action (a label taken from some round, or abstain), $R(a)$ is $1$ for a correct label, $0$ for a wrong one, and a fixed value $r_{abs}\in(0,1)$ for abstaining. $C(\tau)$ is the token cost consumed up to $\tau$, and $\lambda$ is the price per token. Sweeping $\lambda$ and $r_{abs}$ traces out the accuracy–cost and risk–coverage curves we compare.

Grounding in Prior Work

Four papers motivate the gap directly. PAKTON builds a three-agent system for long agreements, with an Archivist that parses, a Researcher that retrieves, and an Interrogator that probes and revises. The authors acknowledge its latency and cost but give it no mechanism to decide when more interrogation stops paying. MASLegalBench shows that decomposition into agents that mirror the Issue–Rule–Application–Conclusion structure of legal analysis beats single-agent retrieval in 44 of 60 settings. Its pipeline, however, is feed-forward, and it does not track cost. Debate-Feedback supplies the key empirical puzzle: accuracy peaks near two rounds. Its authors patch this with a fixed cutoff and a smoothing average over rounds. We treat that finding as a hypothesis to decompose rather than a constant to hard-code. Dr. MAMR seems to point the opposite way, showing that in math and code more multi-agent deliberation keeps helping when agents can restart and are credited for their causal contribution. We reconcile the two findings as follows. Dr. MAMR has a binary verifier that makes extra rounds safe. Contract reasoning has none, so the benefit of extra rounds is item-dependent and must be predicted.

The stopping method comes from Using Performance Profile Trees to Improve Deliberation Control and its companion Deliberation in Equilibrium. A performance profile tree records how solution quality evolved along each observed execution path. It then chooses to stop or continue by dynamic programming over that tree. On scheduling and vehicle routing it came within 99.0% of hindsight-optimal stopping and beat memoryless curves and tables, most clearly at mid-range costs. Those results are for combinatorial solvers. We carry the construction over to language-model rounds, where the "quality" signal is not observable at test time and must be proxied by features of the conversation. The utility framing comes from Algorithmic Rationality: Game Theory with Costly Computation. Under that framing, abstaining when checking costs more than it is worth is a rational action rather than a failure, which is why abstention is a first-class action here.

The legal side sets the target. ContractNLI asks whether a hypothesis is entailed, contradicted, or not mentioned by a non-disclosure agreement. Roughly 30% of its evidence is discontinuous, and contradiction F1 stays around 0.36 even for its strongest fine-tuned model, which is the "exception stated pages later" failure. The work also constrains two design choices. Separable Representations of Task Complexity and Deliberation shows that suppressing a model's internal deliberation hurts procedural tasks, and that a "skip thinking" prompt costs about 20 points on hard math. We therefore control the number of rounds rather than cutting reasoning inside a round. LeSeR shows that dense retrieval should be paired with lexical reranking on regulatory text, so our Researcher uses that hybrid.

Research Program

We first build a fixed interrogation loop and hold it constant across all conditions. A hybrid retriever (dense recall followed by BM25 reranking) serves an Interrogator agent. Each round, the Interrogator issues one probing query aimed at qualifications, definitions, or exceptions to the evidence gathered so far. A Judge agent then outputs a label, cited evidence spans, and a verbal confidence. Every round is logged up to a cap of eight rounds on the ContractNLI training documents, using two open-weight backbones and one API model.

From the logs we compute, for every item, the hindsight-optimal stopping round and the label that would have been returned at each round. The first analysis is purely descriptive and tests the mixture hypothesis. It asks whether items whose gold evidence is discontinuous have later optimal stopping rounds than contiguous-evidence items, and whether the population-average peak near round two disappears once we stratify by this property.

We then fit the controller. The path state at round $t$ is a small discrete summary of the history: the sequence of labels so far, whether each round retrieved a section not previously seen, the confidence bucket, and the number of label flips. Following the tree construction, we estimate the expected final utility of continuing versus stopping at each node by backward induction over the logged trajectories. Stopping can return the current label, the label from the round the controller judges most reliable, or an abstention. Real trajectories make trees sparse, so we also fit a smoothed version, a small gradient-boosted model over the same path features, and report both. The controller never sees gold labels at test time. It uses only the conversation.

Baselines

The naive reference point is a single-round pass: retrieve, then judge once, with no interrogation. It fixes the floor on cost and shows how much any extra rounds buy.

The strongest existing method on this question is a fixed-round multi-agent loop in the style of PAKTON and Debate-Feedback, run at every fixed depth from one to eight rounds. We also run Debate-Feedback's smoothing over rounds. These use our identical agents, retriever, backbones, and prompts, so the only thing varying is the stopping policy. Its fixed depth is tuned on the same development split used to fit our controller. For context, the file reports PAKTON's retrieval Recall@1 on ContractNLI as 53.14%, and ContractNLI's fine-tuned DeBERTa v2 xlarge at 88.5% accuracy and roughly 0.36 contradiction F1. No reported result was found on Lacuna for an LLM multi-agent loop's ContractNLI label accuracy at a given token cost. The fixed-round curve we produce is therefore the comparison that matters. Beating it means that our controller's accuracy–token curve lies above the upper envelope of all fixed depths over a contiguous range of prices $\lambda$. This must hold with a paired bootstrap interval excluding zero on area under the curve.

Two stronger heuristic stoppers test whether path information is needed. The first stops when the label has been unchanged for two consecutive rounds. The second stops when verbal confidence crosses a threshold tuned on the development split. The ablation that isolates our key ingredient is the same fitted controller with path features removed. It sees only the round index and current confidence, which makes it the analogue of the memoryless performance-profile table. An oracle that stops at each item's hindsight-best round gives the ceiling. Every stopper is charged the same token accounting and draws on the same logged rollouts, which makes the comparison exact rather than stochastic.

Why not just…?

Why not just stop after two rounds, as Debate-Feedback found? If the two-round peak holds uniformly across items, a fixed cutoff captures nearly all the benefit and our controller is overhead. That is exactly what the stratified analysis tests. If discontinuous-evidence items peak at the same round as contiguous ones, we will report that the fixed cutoff suffices and the proposal's claim fails. We expect otherwise, because ContractNLI's weakest class, contradiction, is the one where the decisive exception tends to sit far from the obligation it qualifies.

Why not just stop when the answer stops changing? This is the most serious threat, because answer stability is cheap, needs no training, and partially encodes history. It can mislead in one specific way: a loop that has not yet retrieved the exception clause will stably repeat the wrong "entailed" label. Our controller's distinctive feature, whether the last round reached a new part of the document, is aimed at exactly this case. If the stability rule matches our controller within noise, we will conclude that the path signal adds nothing beyond label persistence and say so.

Why not just spend the same tokens on a single long-context pass with several samples and a majority vote? Voting helps when errors are independent. When the relevant clause is never attended to, however, every sample makes the same mistake, and the yes-man tendency documented in MALR pushes all samples toward plausible affirmation. We run matched-token self-consistency as an additional baseline and expect it to lag specifically on discontinuous-evidence contradiction items.

Why not just fine-tune a classifier, given that DeBERTa already reaches 88.5% on ContractNLI? For ContractNLI's seventeen fixed hypotheses, a fine-tuned classifier is a real competitor on label accuracy, and we do not claim to beat it there. The agent loop is justified only for hypotheses the classifier has not seen, and for outputs that need cited evidence and an option to abstain. We therefore hold out a subset of hypotheses during fine-tuning and compare on those. If the classifier still wins on unseen hypotheses, the practical case for any agent loop on this benchmark weakens, whatever its stopping rule.

Proposed Experiments

The primary benchmark is ContractNLI, using its document-level splits. We report label accuracy, per-class F1 with contradiction singled out, evidence-span recall, and total tokens and wall-clock time per item. Abstention is evaluated through risk–coverage curves as $r_{abs}$ varies. To check that the mixture explanation extends beyond contracts, we log trajectories on CaseLaw outcome prediction, where Debate-Feedback reports 0.67 accuracy against 0.64 for single-model GPT-4o. This tests whether the round-two peak there also splits by item type. MASLegalBench serves as a transfer test. There the controller, fit on contracts, is applied without refitting to GDPR case questions, where the file reports 82–86% peak accuracy for the full agent suite. LegalBench-RAG serves as a retrieval sanity check for the fixed agent loop; no reported numerical result was found on Lacuna for it.

The decisive checks are three. First, the per-item oracle stopping round must differ systematically between discontinuous and contiguous evidence; if it does not, the core premise is wrong. Second, the full path-conditioned controller must beat both the tuned fixed depth and the memoryless ablation on the accuracy–token curve. The gain should be concentrated at mid-range token prices, which is where the operations-research results located the advantage of path-dependent stopping. Third, at matched coverage, the abstaining controller must lower the error rate on contradiction items relative to the confidence-threshold stopper. That would show abstention is being triggered by a real signal that a crucial clause may be missing, not by generic uncertainty.