Several recent fact-checking systems use multi-agent debate. Multiple copies of a language model argue for and against a claim, and a judge model then picks a verdict. A separate body of work shows that language models change their answers when a prompt is reworded in ways that should not matter, such as a different persona description, a different speaking order, or an extra polite phrase. Put these together and there is an uncomfortable possibility: a debate verdict on a political claim may depend partly on how the debaters were instructed. We expect this dependence to be worst for claims from places the models know poorly, such as Indian state politics, Canadian parliamentary exchanges, or Chinese domestic topics. On such claims the model is less confident, and earlier work finds that low confidence is exactly where prompt wording flips decisions.
This note proposes to measure that effect directly and to test a specific fix. Debaters keep the job of proposing evidence. They lose the job of deciding. Each debate turn is broken into short factual sub-claims. A fixed natural language inference model checks each sub-claim against a short evidence passage. Such a model is a classifier that answers "does this passage support, contradict, or say nothing about this sentence?" The verdict is then computed from those checks by a fixed rule instead of by a judge model. Our core claim is twofold. First, verdicts from debate fact-checkers vary measurably across meaning-preserving prompt variants, and they vary more on claims from outside the United States and from outside major metropolitan areas. Second, routing the decision through a fixed entailment check removes most of that variation without lowering accuracy. Whether the debate itself adds anything beyond a single model doing the same decomposition is an open question, and the experiments are built to answer it.
What prior work establishes and what it leaves open
Three papers apply debate to fact verification or bias, and none of them measures whether the verdicts are stable under prompt changes. Can LLMs Produce Faithful Explanations For Fact-checking? runs a four-role debate (two debaters, a judge, a refiner) to improve fact-check explanations on the US-only PolitiHop data. Humans still judged only 30% of its explanations fully faithful, and the authors flag prompt sensitivity as an uncontrolled factor. MAD-Fact splits long answers into atomic propositions and has a jury of personas debate each one with retrieval, scoring Chinese (LongHalluQA) and English text separately. It never varies the jury instructions. Mitigating Cultural Bias in LLMs via Multi-Agent Cultural Debate assigns regional cultural personas to debaters. It does so for soft questions about food and etiquette rather than verifiable facts, and it uses no entailment check.
These papers motivate the proposal, but they share an assumption: that a persona prompt produces a stable perspective. Two other lines of work constrain that assumption. Revealing Political Bias in LLMs through Structured Multi-Agent Debate finds that persona agents drift during debate, revert to their earlier positions when the pressure is removed, and are swayed by speaking order. ProSA ties prompt sensitivity to low decoding confidence. Rethinking Hallucinations shows that answers on factual benchmarks often shift across prompt variants (ambiguity frequently above 50%) and that hallucination detectors track this inconsistency more than correctness.
None of these studies examines a debate system, and none stratifies by geography. The geographic-bias literature supplies the regional axis. (Im)balance in the Representation of News? finds that Delhi and Mumbai take 60–70% of city mentions in Indian national news. That concentration gives a natural split between well-covered and poorly covered regions within a single country.
The method borrows from natural language inference work. TRUE shows that entailment classifiers agree with human consistency judgments better than surface-overlap metrics. It also shows that they degrade once the evidence passage exceeds about 200 tokens, so we keep every evidence passage under that length. Scalable and Domain-General Abstractive Proposition Segmentation supplies the step that splits text into atomic sub-claims. Conditional Natural Language Inference handles a harder case: claims that are true only under a stated condition, such as "according to ministry figures" or "in this state." We reuse that idea for regionally framed claims. Position: Logical Soundness is not a Reliable Criterion warns that internally coherent reasoning can still be empirically false. For that reason the checker compares sub-claims against retrieved evidence, never against the debate's own logic.
Core Claim and Mechanism
In a standard debate fact-checker, wording in the prompt can reach the verdict through two channels. It can change which arguments get made, and it can change how the judge weighs them. The proposed system closes the second channel.
Debaters work as in MADR: one supports the claim and one attacks it. One rule is added. The attacking agent must also try to name a condition under which the claim holds or fails. Every turn is segmented into atomic sub-claims, and each sub-claim must cite an evidence passage of at most 200 tokens. For multi-hop claims, a sub-claim may cite a pair of passages, provided the two together stay under the same budget.
A fixed entailment model labels each sub-claim and its cited passage as supported, contradicted, or unsupported, and unsupported sub-claims are struck. A deterministic rule then produces the verdict from the surviving sub-claims. The verdict is false if any surviving sub-claim on the attacking side contradicts the claim. It is true if the supporting side's surviving sub-claims cover every component of the claim. It is conditionally true if an entailed condition was surfaced. Otherwise it is "not enough evidence."
Once this structure is in place, prompt wording can affect the verdict only through which sub-claims get proposed. That residual effect is directly measurable.
The main quantity is verdict multiplicity, adapted from the ambiguity measure in Rethinking Hallucinations. Let $V$ be a set of $K$ meaning-preserving prompt variants. Let $y_k(c)$ be the system's verdict on claim $c$ under variant $k$. Then
$$M = \frac{1}{|C|}\sum_{c \in C} \mathbb{1}\left[\exists\, j,k \le K : y_j(c) \neq y_k(c)\right],$$
which is the fraction of claims whose verdict changes under at least one variant pair.
The variants cover four factors: persona wording, speaking order, instruction paraphrase, and ProSA's four template dimensions (simple input, emotional support, role player, output requirement). The regional gap is $M$ on a non-US or peripheral claim set minus $M$ on the US or metropolitan claim set.
Research Program
The program has three parts. The first is a measurement study. Two existing debate fact-checkers, MADR-style and MAD-Fact-style, are run under the variant set on every claim pool. We report $M$, accuracy, and the regional gap. We also report how much of $M$ each variant factor contributes, by changing one factor at a time.
The second part builds the entailment-gated system described above. Its multiplicity is decomposed into two parts: verdict changes caused by different sub-claims being proposed, and verdict changes caused by the checker itself. The second part should be zero by construction, apart from retrieval nondeterminism, which is logged.
The third part tests whether multiplicity predicts errors in debate systems, as On the Relation between Sensitivity and Accuracy in In-Context Learning found for single-model in-context learning (correlation up to −0.40). If it does, multiplicity can serve as an abstention signal: the system declines to issue a verdict on claims where it is unstable.
Claim pools are drawn as follows. PolitiHop from Multi-Hop Fact Checking of Political Claims serves as the US multi-hop reference. The Canadian Oral Question Period labels from Automated Fact-Checking of Claims in Argumentative Parliamentary Debates provide a multi-party Western set. Atomic facts from MAD-Fact's LongHalluQA provide a Chinese set.
The research file contains no labeled multi-hop fact-verification set for Indian or European politics, so we will annotate a new one. It will hold about 600 claims from the Indian news corpus, split evenly between claims about Delhi or Mumbai and claims about other states, plus claims from EuroParlVote speeches (Demographics and Democracy). Each claim gets gold evidence passages and a conditional-truth label where it applies. Two annotators will label every claim, and we will report agreement. The TRUE collection is used only to choose and calibrate the entailment checker before any experiment runs.
Baselines
The naive reference is a single model giving a zero-shot verdict on the claim plus retrieved evidence, run under the same $K$ variants. This sets the floor for both multiplicity and accuracy.
The strongest existing debate methods are a MADR-style four-role pipeline (Can LLMs Produce Faithful Explanations…) and a MAD-Fact-style persona jury with retrieval (MAD-Fact). Each is reimplemented on the same backbone, retriever, and evidence pool as the proposed system. Each also gets the same prompt-tuning budget on a shared development split. This matters because the reported numbers come from different setups: MADR reached 30% fully faithful explanations against 20% for zero-shot, with 17 errors against 42 for chain-of-thought in a 20-sample study, and MAD-Fact's best F1 on LongHalluQA was 0.694 (QwQ-32B). Neither paper reports verdict multiplicity. No reported result was found on Lacuna for prompt-variant stability of any debate fact-checker.
The fine-tuned reference on PolitiHop is Transformer-XH, with a FEVER score of 24.5 and evidence F1 of 47.2 (Multi-Hop Fact Checking of Political Claims). It is included as a point of comparison rather than as a competitor on stability.
Two ablations isolate the proposal's ingredients. The first keeps the proposed debate and segmentation but lets a judge model overrule the entailment labels. This isolates the effect of removing the judge from the decision. The second has a single agent produce the decomposition, sub-claims, and evidence citations with no adversary, scored by the same checker and the same rule. This isolates what the debate itself contributes.
To keep comparisons fair, every LLM-based system is matched on total generated tokens per claim. Where a method uses fewer tokens, it receives the difference as extra samples for majority voting.
The proposal counts as beating a baseline if two conditions hold together. Multiplicity must fall by at least half relative to the best debate baseline on the non-US pools. Macro-F1 must stay within one point of that baseline or above it.
Why not just…?
Why not just run the entailment model on the claim and the retrieved evidence, with no debate at all? This is cheap, prompt-free, and strong: the ANLI-based metric reached an area under the ROC curve of 81.5 on TRUE, and an ensemble of metrics reached 86.0 (TRUE). Its weaknesses are the ones political claims expose. Evidence for a multi-hop claim easily exceeds the 200-token range where TRUE saw degradation, and a whole compound claim is harder to classify than its parts. We run this baseline, giving it the same retrieved passages as the proposed system, both truncated and in sliding windows. If it matches our accuracy on PolitiHop and the new multi-hop set, the debate is unnecessary and we will report that.
Why not just have one model split the claim into parts and check them? This is the most serious threat, and it is why the single-agent ablation exists. The argument for an adversary is that a single decomposer tends to propose only sub-claims consistent with its first reading of the claim. An attacker assigned to find contradictions and conditions should surface contradicting evidence and conditional framings that a single pass misses. That benefit should show up mainly on claims labeled false or conditionally true. If the single-agent version matches the debate on those claims at equal token budget, the conclusion becomes "entailment gating, not debate, is what stabilizes verdicts," and that would still be a useful result.
Why not just average a single model's verdicts over many prompt paraphrases? Majority voting over variants, which is the core of SENSEL's approach (an average gain of 4.1 points in area under the F1–coverage curve over MaxProb and Entropy, On the Relation between Sensitivity and Accuracy in In-Context Learning), hides instability by averaging over it. It also multiplies cost by $K$. We will apply it at matched token budget to both the naive baseline and the debate baselines, and measure multiplicity across held-out variant sets. If voting closes the regional gap, the gating mechanism is not needed for stability.
Why not just add a few demonstrations? ProSA finds that moving from zero to one example gives the largest single drop in sensitivity. Few-shot prompting, however, remains sensitive to the order of the examples (On the Relation between Sensitivity and Accuracy). It also says nothing about the judge's freedom to weigh arguments differently. We include one-shot versions of every baseline, and we treat demonstration order as an additional variant factor.
Decisive Checks
The first check is whether the problem exists. Debate baselines must show nonzero multiplicity, and that multiplicity must be higher on peripheral-Indian, European, and Chinese claims than on US and metropolitan claims, with confidence intervals bootstrapped over claims. If there is no regional gap, the geographic framing is dropped. What remains is a general finding about the stability of debate fact-checkers.
The second check is whether gating fixes it. The proposed system must meet the dual criterion above, and its residual multiplicity must come almost entirely from changes in which sub-claims were proposed. This is verified by replaying a fixed set of sub-claims through the checker under every prompt variant.
The third check compares the debate system against the single-agent decomposition at equal budget, separately on true, false, and conditionally true claims. It determines whether the debate earns its cost.
The fourth check tests whether multiplicity in the debate baselines correlates negatively with correctness on the regional pools. If so, multiplicity serves as an abstention signal. The final metric is the area under the F1–coverage curve, compared against verbalized confidence from the debate judge.
References
- Can LLMs Produce Faithful Explanations For Fact-checking? Towards Faithful Explainable Fact-Checking via Multi-Agent Debate
- MAD-Fact: A Multi-Agent Debate Framework for Long-Form Factuality Evaluation in LLMs
- Mitigating Cultural Bias in LLMs via Multi-Agent Cultural Debate
- Revealing Political Bias in LLMs through Structured Multi-Agent Debate
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- (Im)balance in the Representation of News? An Extensive Study on a Decade Long Dataset from India
- TRUE: Re-evaluating Factual Consistency Evaluation
- Scalable and Domain-General Abstractive Proposition Segmentation
- Conditional Natural Language Inference
- Position: Logical Soundness is not a Reliable Criterion for Neurosymbolic Fact-Checking with LLMs
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- Multi-Hop Fact Checking of Political Claims
- Automated Fact-Checking of Claims in Argumentative Parliamentary Debates
- Demographics and Democracy: Benchmarking LLMs' Gender Bias and Political Leaning in European Parliament