Systematic reviews often leave out studies not published in English. That skews medical evidence toward the populations English-language journals happen to cover. Automated review pipelines built on large language models could fix this, since they can read any language. Many of these pipelines split the work so that one model instance reads each study, and then the instances argue to a verdict, a setup called multi-agent debate. This note argues that debate may quietly rebuild the bias inside the machine. If one agent holds a Swahili or Chinese trial report and three agents hold English ones, the English agents tend to agree with each other. Agreement exerts social pressure, and the lone agent's evidence can be voted down even when it is decisive. The reverse failure is also likely. Models reading linguistically distant text are badly overconfident, so a misreading custodian can drag the group into error.
We propose to measure both failures on a clinical evidence-update task. In that task, the latest study in a time-ordered evidence stream is written in a language other than English, and the question is whether it changes the cumulative conclusion. We then test a narrow fix, which we call custody debate. Each agent is the sole reader of its document. It must ground every claim in quoted spans from that document. The final judge weighs agents by confidence calibrated separately for each language, not by head count. The central question is whether evidence survives the debate as a function of how far its language is from English. A strong single-model baseline may make debate unnecessary, and the experiments are designed to show that if it does.
Core Claim
In debate over distributed clinical evidence, two rates should rise with the linguistic distance between a document's language and English. The first is the suppression rate: the agent holding the decisive non-English document is initially correct, but the group ends wrong. The second is the derailment rate: the custodian is initially wrong, the others are right, and the group ends wrong.
Two prior findings support this prediction. Calibration error in multilingual models grows with typological distance. On the Calibration of Massively Multilingual Language Models reports XLM-R's Expected Calibration Error (ECE) on XNLI at 7.32 in English and 19.07 in Swahili. Calibrating Zero-shot Cross-lingual (Un-)structured Predictions shows ECE climbing steadily from German to Arabic to Chinese. Separately, debate dynamics favor conformity. Beyond Correctness: Distance-Based Social Dynamics of Multi-Agent Debate finds that identical wrong answers pressure an agent far more than diverse ones, and that correct agents are pulled off course by "near-correct" peers.
The new element is the combination. We place linguistic distance on the evidence side and an answer-space distance on the output side, in a task where the right answer depends on one specific document. We then ask whether custody and calibrated weighting close the gap better than simpler fixes.
The evidence-update framing also gives the ConceptARC distance analysis a natural non-geometric home. That paper left open how to define distance in semantic domains. Here the answer is a pooled effect estimate, so distance between agents' answers is simply a difference on the effect-size scale. This lets us test whether the plausibility trap and the contraction effect carry over to clinical numbers.
Research Program
The evidence-update benchmark. We build it from TrialReviewBench. That benchmark comes from the TrialMind paper, which automates search, screening, extraction and meta-analysis but evaluates only on English PubMed Central. It contains 100 systematic reviews with 1,049 manually annotated study results, which lets us recompute pooled effects without guessing.
For each review we order the included studies by publication date. We compute the cumulative pooled estimate before and after the latest study. We label whether adding that study flips the direction of effect, changes statistical significance, or leaves the conclusion stable. These are the improving, worsening and stable categories that temporal radiology models such as BioViL-T and CoCa-CXR predict for image pairs. We apply that "what changed since the baseline" logic to an evidence stream instead of a scan series.
The latest study's report is then rendered into one of the eight non-English languages used by HealMed: German, Spanish, Portuguese, Japanese, Chinese, Thai, Swahili and Zulu. The rendering is done with physician verification, following HealMed's protocol. HealMed showed that unverified machine translation creates artifacts that even inflate LLM judges' scores, so unverified translation is not acceptable here. We index each language by its URIEL syntactic distance from English, as the calibration papers do.
A second, shorter task. We use the comparison instances of NLI4CT. Each instance asks whether a statement comparing two trial reports is entailed, and we render one of the two reports in the target language. This task tests numerical comparison reasoning, which already caps English performance.
Limitation and natural-text check. Translated English reports are not the same as naturally regional literature. We therefore add a small natural set: non-English trial publications from the reviews' excluded-by-language lists, with outcomes extracted by bilingual clinicians. This set serves as an external-validity check, not a training resource. We found no existing benchmark of naturally non-English trial reports on Lacuna.
The custody debate protocol. The protocol builds on the formal debate structure of FORD, which has three steps: independent stances, then alternating rounds of refutation or explicit concession, then a judge that rules on argument quality. We add three constraints.
The first is custody. Each document is read by exactly one agent, and that agent reads it in its original language. Other agents see only the custodian's quoted source spans plus the custodian's own English gloss of those spans.
The second is span grounding. A concession or rebuttal is valid only if it cites a span. A custodian therefore cannot be argued out of its position by assertion alone, only by a span-level challenge to its reading.
The third is calibrated weighting. Before judging, the judge receives each agent's confidence, rescaled by a calibration map fit separately for each language. Each map is fit on a small labeled development set in that target language. This follows the finding of On the Calibration of Massively Multilingual Language Models that target-language few-shot data with label smoothing (ECE 2.55 to 5.69) beats English-fit temperature scaling (Swahili ECE still 11.54). The maps can be temperature scaling or Gaussian-process calibration, which Calibrating Zero-shot Cross-lingual (Un-)structured Predictions found stronger for structured outputs.
Concretely, the judge's verdict on the update label is a weighted vote. Agent $i$'s vote counts with weight $w_i = g_{\ell_i}(c_i)$, where $c_i$ is the agent's raw confidence, $\ell_i$ is the language of its document, and $g_{\ell}$ is the calibration map for language $\ell$. A verdict that disagrees with a high-weight custodian must cite the span-level rebuttal that justifies the override.
What prior debate methods do differently. Several debate methods address weak or divergent agents by pruning dissent. Masked Agent Collaboration drops the least semantically consistent agent at each layer. The critic in Separate the Wheat from the Chaff merges or discards divergent contributions. In our setting, these rules would systematically remove the one agent holding the foreign-language study, because it is by construction the odd one out. We therefore treat these methods both as the strongest existing competitors and as a prediction: consistency pruning should have the highest suppression rate of any method tested.
Models and analysis. All runs use one open model family (Qwen3-32B, which HealMed found to be a top open performer yet still 21.7 points behind in low-resource languages) and one frontier proprietary model. Debates have three to four agents. We log every round so we can measure, on the effect-size scale, whether revisions contract toward the true pooled estimate. We also test whether near-correct peer estimates derail correct agents more than distant ones, as observed on ConceptARC.
Baselines
Naive reference points. Two reference points bound the problem. The first is an English-only upper bound: the same instances with every document in English, under the same protocols. The gap between this and each language is the "distance penalty." The second is an exclusion baseline that mimics an English-restricted review: the non-English study is dropped and the model answers from the English studies alone. Any method that does no better than exclusion has not recovered the excluded evidence.
Single model with full context. The most serious competitor is a single frontier model, prompted with chain-of-thought, that sees every document in context. It receives the non-English report either in the original language or translated to English, and both variants are run. This removes the social channel entirely.
For fairness, the single model gets a token budget equal to the whole debate's. We spend that budget on self-consistency sampling with majority voting, and use the same prompts' clinical framing.
On NLI4CT, the strongest reported figure for this family is from IITK at SemEval-2024 Task 2. That system simulates an expert panel inside one model with tree-of-thought prompting and retrieval. It reports F1 0.691 with Gemini Pro (faithfulness 0.901, consistency 0.712), against the original NLI4CT ceiling of F1 0.644 reported in NLI4CT. These are English-only figures. We found no reported result on Lacuna for NLI4CT with a non-English report, or for any evidence-update task.
Standard debate. The second family is debate as usually practiced. This includes FORD-style debate without custody, where all agents see all documents, and MAC-style consistency pruning. On NEJMQA's obstetrics and gynecology subset, the MAC paper reports beating standard collaboration by 16.55% and GPT-4 by 9.35%. Both run with the same agent count, base models and number of rounds as custody debate.
Calibration without debate. The third baseline gives each agent independent per-language calibration, then takes a calibrated weighted vote with no debate at all. This isolates whether interaction adds anything beyond calibration.
Ablations. We run custody debate without calibrated weights, custody debate without span grounding, and calibrated debate without custody. Together these show which ingredient carries the effect.
What counts as winning. Custody debate beats the field if it does two things. First, it lowers the suppression rate in the most distant languages, Swahili and Zulu, without raising derailment. Second, it shrinks the slope of update-label accuracy against URIEL distance relative to the single-model full-context baseline, with confidence intervals from bootstrap over reviews that exclude zero. Matching the single model's accuracy in German while losing in Zulu would count as a failure.
Why not just…?
Why not just give one strong model everything? This is a real threat, and it may win. A single model has no conformity pressure, so suppression in our sense cannot occur. It still faces the distance problem: HealMed reports drops of 15.4 points in Swahili and 28.9 in Zulu, and observed a model reasoning in English before answering in Thai. Reading the decisive study as one item among many in a long context may also bury it, much as ClinSeekAgent found its single agent distracted by non-pertinent history. The compute-matched comparison settles this. If the single model matches custody debate on the distant-language slope, our conclusion becomes that distributed pipelines should not use debate for cross-lingual synthesis. The suppression measurement remains useful as a warning for the many multi-agent review pipelines being built.
Why not just translate everything into English first? Translation to English followed by a standard pipeline is what TrialMind-style systems would do by default. Its failure points are documented. Multilingual Clinical NER: Translation or Cross-lingual Transfer? found translate-test worse than direct transfer for large models, even between closely related European languages. HealMed found terminology errors in over 64% of Zulu open-ended responses. Both translate-test variants are in the baseline set. If translation wins in German and Spanish but loses in Zulu, that is the predicted pattern.
Why not just calibrate each agent and skip the debate? If miscalibration explains derailment, per-language calibration with a weighted vote might be enough. It cannot address suppression, though, because a correct but outnumbered custodian loses a vote even with honest confidence. The calibration-only baseline directly tests whether span-level challenges add anything beyond reweighting.
Why not just use an existing medical debate method? Consistency-based pruning like MAC removes the dissenting agent, and in this task the dissenter is the one holding the new study. We expect it to perform worst on suppression while looking competitive on English-only controls. That outcome would itself show that agreement-seeking ensembles reproduce publication bias.
Proposed Experiments and Decisive Checks
Experiment 1: does the phenomenon exist? We run standard FORD debate and MAC on the evidence-update and NLI4CT-comparison tasks across all nine languages. We report suppression and derailment rates against URIEL distance. If neither rate rises with distance, the core claim is false, and we report that debate is robust to linguistic evidence asymmetry.
Experiment 2: the full comparison. We run every baseline and ablation under matched token budgets. The primary metrics are update-label accuracy, with flips and significance changes reported separately because these are what change guidelines. We also report the distance slope, suppression and derailment rates, and ECE of the final verdict.
Experiment 3: solution-space dynamics. We use logged numeric estimates to test whether the distance-based dynamics transfer from grids to effect sizes. The specific questions are whether far-off estimates are revised more, whether revisions contract toward the true pooled value, and whether near-correct English-majority estimates derail a correct custodian more often than distant ones. We also ask whether this trap deepens with linguistic distance, which is the most direct link between the two kinds of distance.
Experiment 4: natural text. We repeat the headline comparison on the natural non-English set. This checks whether translated-report results carry over to studies written for regional readers.
The decisive check. Each benchmark has a subset of instances where the non-English study alone flips the conclusion. On those instances, does custody debate recover the flip more often than both the exclusion baseline and the compute-matched single model, in the two most distant languages? If yes, multi-agent synthesis can include regional evidence without silencing it. If no, the right recommendation is to drop debate for this purpose. Either result is directly useful to anyone automating systematic reviews.
References
- On the Calibration of Massively Multilingual Language Models
- Calibrating Zero-shot Cross-lingual (Un-)structured Predictions
- Beyond Correctness: Distance-Based Social Dynamics of Multi-Agent Debate
- Accelerating clinical evidence synthesis with large language models
- Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing
- CoCa-CXR: Contrastive Captioners Learn Strong Temporal Structures for Chest X-Ray Vision-Language Understanding
- HealMed: Multilingual Evaluation of Large Language Models in Medicine
- NLI4CT: Multi-Evidence Natural Language Inference for Clinical Trial Reports
- Examining Inter-Consistency of Large Language Models Collaboration: An In-depth Analysis via Debate
- Adaptive Cluster Collaborativeness Boosts LLMs Medical Decision Support Capacity
- Separate the Wheat from the Chaff: Winnowing Down Divergent Views in Retrieval Augmented Generation
- IITK at SemEval-2024 Task 2: Exploring the Capabilities of LLMs for Safe Biomedical Natural Language Inference for Clinical Trials
- ClinSeekAgent: Automating Multi-modal Evidence Seeking for Agentic Clinical Reasoning
- Multilingual Clinical NER: Translation or Cross-lingual Transfer?