Large language models asked to reproduce the conclusions of medical systematic reviews take weak studies at face value. The best of 25 models tested on the MedEvidence benchmark reach roughly 60–62% agreement with expert Cochrane conclusions, against more than 75% for time-limited human experts. When the source studies disagree with each other, model accuracy falls to about 41% (Can Large Language Models Match the Conclusions of Systematic Reviews?). Human reviewers do something models do not. They discount small, unblinded or off-target trials. They avoid counting the same result twice when an abstract restates it. They say "the evidence is insufficient" when the question's key outcomes are never actually measured.
We propose to make that skepticism explicit. Each study is broken into small "evidence units": individual finding statements, each tagged by the section it came from. These units carry an amount of "mass" (how much they are allowed to count). The mass is moved onto the parts of the review question using unbalanced optimal transport, a matching method in which mass may be thrown away when matching it costs too much. The idea in two sentences: weak, indirect or redundant evidence should be discarded by the matching itself rather than by the model's unguided attention. The pattern of what was kept, what was discarded, and which parts of the question were left uncovered should then be read off as the review's conclusion and its certainty.
Core Claim
Optimal transport finds the cheapest way to move mass from a set of sources to a set of targets. The classical version must move all of it. Unbalanced optimal transport (UOT) replaces those hard totals with soft penalties, so mass that is too expensive to place is simply destroyed. Unbalanced Optimal Transport through Non-negative Penalized Linear Regression and Unbalanced minibatch Optimal Transport; applications to Domain Adaptation show this filters outliers and unshared classes in vision and domain adaptation. Our claim is that the UOT objective has a natural correspondence to the five GRADE downgrade criteria, the international standard for rating how much to trust a body of clinical evidence (Grading the Certainty of Clinical Evidence). Each GRADE criterion becomes a separate, auditable term of the objective:
- Indirectness becomes the transport cost: how far a study's population, intervention or outcome is from the question.
- Risk of bias becomes a per-source destruction penalty: weak studies are cheap to discard.
- Imprecision becomes the starting mass, scaled by sample size, together with any question mass left uncovered.
- Inconsistency becomes signed disagreement among the mass that lands on the same part of the question.
The design choice that separates this from scoring each study independently is the penalty on the target side. Each part of the question can absorb only a bounded amount of evidence. Duplicate statements of one trial's result, and floods of small trials on a secondary outcome, therefore cannot outvote a large trial on the primary outcome.
Relation to Prior Work
The nearest method is UOTSumm (Joint learning of text alignment and abstractive summarization for long documents via unbalanced optimal transport). It aligns sections of one long paper to the sentences of its summary with UOT, using the summarizer's own loss as the cost, and raises arXiv ROUGE-L from 39.68 to 44.61. We reuse its insight that unhelpful sections should lose their mass. We differ in three ways:
- We operate across many conflicting studies rather than within one document.
- Our cost encodes methodological directness rather than how likely the summarizer is to generate the text.
- Our output is a graded conclusion, not prose.
Structured Optimal Transport adds group structure to transport costs. It is mass-conserving and slow, because each step requires a projection through a convex relaxation of the group cost. We take only its lesson that group structure matters, and encode study membership through the target-side penalty instead.
Structural relevance weighting supplies the source masses. The Next Phase of Scientific Fact-Checking shows that Introductions mostly recount prior or contested work while Results and Discussion carry primary evidence. It exploits this by re-ranking each claim–document pair independently. Our proposal places those section weights inside a joint allocation over all studies. Hand-set position curves of the kind in CaseSummarizer or the parabolic prior of Ferrara et al. fail on documents that break convention. We therefore learn one weight per section type from data rather than fixing a position curve.
Two papers constrain the design. Decide less, communicate more finds low expert agreement on unstructured medical claims. This is why we target Cochrane-adjudicated questions rather than free claims. Evaluating and Guarding Citation Faithfulness finds that automatic verifiers agree on which citations are unsupported only 0.27–0.30 of the time (negative-specific agreement). This warns us that any model scoring direction or risk of bias will be noisy. We therefore measure how sensitive conclusions are to swapping the extractor model.
Research Program
Extraction. For each MedEvidence question, an LLM breaks every provided study into evidence units. Each unit records:
- the finding,
- its effect direction $s_i \in \{-1, 0, +1\}$ (harm, no difference, benefit),
- the section it came from (results text or table, abstract, discussion, introduction),
- the study's sample size,
- a short risk-of-bias judgment along the Cochrane domains (randomization, blinding, attrition, selective reporting).
The review question is broken into facets $j$, each a population–outcome pair, with facet mass $b_j$ (uniform, or with primary outcomes weighted up).
Allocation. Unit $i$ starts with mass $a_i = w_{z(i)} \sqrt{n_i}$. Here $w_z$ is a learned weight for section type $z$, and $n_i$ is the sample size. The square root mirrors how a study's precision grows with sample size. We solve
$$\min_{P \ge 0} \; \langle C, P \rangle + \sum_i \tau_i \, \mathrm{KL}\big((P\mathbf{1})_i \,\|\, a_i\big) + \tau_b \, \mathrm{KL}\big(P^\top \mathbf{1} \,\|\, b\big) - \varepsilon H(P).$$
The terms are defined as follows:
- $P_{ij}$ is the mass unit $i$ sends to facet $j$.
- $C_{ij}$ is an indirectness cost: an LLM- or embedding-scored mismatch between the unit's population, intervention and outcome and facet $j$.
- $\tau_i = \tau_0 \, r_i$ is the per-unit penalty for discarding mass, where $r_i \in (0,1]$ falls with judged risk of bias, so biased evidence is cheap to discard.
- $\tau_b$ caps how much evidence each facet can absorb.
- $\varepsilon H(P)$ is a small entropy term, which permits the fast scaling iterations used by Sinkhorn-type solvers.
Readout. From the solved plan we compute three quantities for each facet:
- the transported direction $d_j = \sum_i P_{ij} s_i / \sum_i P_{ij}$,
- the coverage $\kappa_j = \sum_i P_{ij} / b_j$,
- the conflict, measured as the mass-weighted variance of $s_i$ on $j$.
A small classifier, or simply two thresholds fitted on training folds, maps these to the benchmark's conclusion labels. Low coverage maps to insufficient-evidence style answers, high conflict to uncertain ones, and otherwise the label follows the sign of $d_j$. The section weights $w_z$, the penalty scale $\tau_0$ and the readout are fitted by cross-validation grouped by review. This addresses the known sensitivity to the destruction penalty, and keeps only about seven free parameters on a 284-question benchmark.
Outputs beyond accuracy. Every conclusion comes with an inspectable plan. The plan shows which studies were discarded and why: was it high cost (indirect), a low penalty (biased), or a saturated facet (redundant)? This is the audit trail that GRADE asks for.
Baselines
Naive reference points. These are majority-label prediction, and an unweighted vote over extracted study directions. Beating majority shows the readout carries any signal at all. The unweighted vote shows how much the conflict cases depend on weighting.
Strongest reported existing system. This is direct prompting of a frontier model given all studies, as in Can Large Language Models Match the Conclusions of Systematic Reviews?. The best reported accuracy is roughly 60–62% (DeepSeek V3, GPT-4.1), and roughly 41% when sources conflict. We will re-run the strongest open model available to us under our cross-validation folds rather than rely on the published figures. The same model also serves as our extractor, so the comparison isolates the aggregation step rather than model strength.
GRADE-checklist prompting. The same model is prompted to rate each study's risk of bias, indirectness and precision step by step before answering. We found no reported result on Lacuna for GRADE-structured prompting on MedEvidence. It gets the same token budget and number of prompt iterations as our extraction prompts.
Extract-then-aggregate. The same extracted units are combined by a weighted vote, with weights $a_i \cdot r_i \cdot \exp(-\min_j C_{ij})$, and with thresholds fitted on the same folds. We found no reported result on Lacuna for this baseline. It is the decisive competitor, because it uses every signal our method uses except the joint coupling and the per-facet capacity.
Section-aware filtering. The language model receives only the Results and Discussion sections, in the spirit of The Next Phase of Scientific Fact-Checking. This tests whether structural weighting alone, without transport, closes the gap.
Ablations of our method. We run five:
- balanced transport, where all mass must move;
- a uniform $\tau_i$, which removes the risk-of-bias link;
- uniform $w_z$, which removes structural weighting;
- $\tau_b \to 0$, which removes facet capacity;
- swapping the extractor model, which tests sensitivity to verifier noise.
What counts as winning. Beating direct prompting by a margin larger than the across-fold standard deviation counts as a win on accuracy. To claim that the transport itself matters, we additionally need a statistically significant advantage over extract-then-aggregate on the conflicting-source subset, using a paired bootstrap over questions.
Why not just…?
Why not just prompt the model with a GRADE checklist? This is plausible and untested on Lacuna, so we run it. The diagnostic study reports two problems that suggest it falls short: accuracy degrades as more studies are added, and models trust weak trials even when the text describes their flaws (Can Large Language Models Match the Conclusions of Systematic Reviews?). A checklist does not remove that degradation with study count, since every study still competes for the same attention. If checklist prompting matches our method, the honest conclusion is that skepticism can be elicited from the model and does not need to be computed outside it. The study-count analysis below would show which is true.
Why not just extract per-study fields and do a weighted vote or a meta-analysis? This is the real threat. Cochrane reviews themselves pool studies by meta-analysis. Our method differs only where evidence overlaps or partially matches the question. Duplicated statements of one result across abstract, results and discussion are capped by facet capacity, rather than counted three times. Evidence on the wrong outcome cannot stand in for the primary outcome. The extract-then-aggregate baseline and the $\tau_b \to 0$ ablation test exactly this. If neither loses to the full method, we will report that the gain comes from structured extraction rather than transport, and reframe the contribution accordingly. A formal meta-analysis requires effect sizes and variances that abstracts often omit. We therefore also report it on the subset where the extractor recovers them.
Why not just give the model only the Results sections? Section filtering cuts context length and background noise, as The Next Phase of Scientific Fact-Checking found for retrieval (Recall@5 on SciFact-Open from 57.17% to 62.61% with verification reranking). It does not tell the model which trial is biased or indirect, so it should help mainly when sources agree. It should not help on the conflicting subset, where the 41% failure lives.
Proposed Experiments
Main evaluation. This is MedEvidence (100 reviews, 284 questions, 329 studies), using five-fold cross-validation grouped by review. We report accuracy overall, separately for agreeing and conflicting sources, and as a function of the number of studies per question.
Decisive check 1: conflicting questions. The full method must exceed extract-then-aggregate there, not just direct prompting.
Decisive check 2: number of studies. The full method's accuracy should stay roughly flat as studies increase, while direct prompting declines. Transport cost grows with the number of units, not with how long a context the model must reason over.
Decisive check 3: discarded mass versus expert downgrades. Where the Cochrane source review reports a summary-of-findings certainty, we would extract it ourselves from the reviews. We then test whether the fraction of mass destroyed correlates with downgrade count, and whether low coverage predicts insufficient-evidence conclusions. This is the test of whether the GRADE correspondence is real rather than decorative.
Decisive check 4: stability. We sweep $\tau_0$ across two orders of magnitude and swap the extractor model. Agreement on discarded studies is reported alongside the 0.27–0.30 verifier agreement from Evaluating and Guarding Citation Faithfulness, as a reference for how much extractor noise to expect.
Supporting checks. As a secondary check on the learned section weights, we test whether they transfer: used as a document re-ranker on SciFact-Open and Check-COVID, do they recover the gains the fact-checking paper reports (Check-COVID Recall@5 from 82.06% to 84.04%)? TrialPanorama (TrialPanorama) supplies registry fields such as sample size and design. We would use these to validate the extractor's sample-size and blinding fields on trials with known registry entries.
References
- Can Large Language Models Match the Conclusions of Systematic Reviews?
- Unbalanced Optimal Transport through Non-negative Penalized Linear Regression
- Unbalanced minibatch Optimal Transport; applications to Domain Adaptation
- Does Tranexamic Acid Reduce the Blood Loss in Various Surgeries? An Umbrella Review of State-of-the-Art Meta-Analysis
- Joint learning of text alignment and abstractive summarization for long documents via unbalanced optimal transport
- Structured Optimal Transport
- The Next Phase of Scientific Fact-Checking: Advanced Evidence Retrieval from Complex Structured Academic Papers
- CaseSummarizer: A System for Automated Summarization of Legal Texts
- Unsupervised Detection of Argumentative Units though Topic Modeling Techniques
- Decide less, communicate more: On the construct validity of end-to-end fact-checking in medicine
- Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
- Developing Large Language Models for Clinical Research Using One Million Clinical Trials