Systems that write literature reviews automatically are now graded mostly by other language models acting as judges. The judge reads the generated review and the papers it cites, then scores it for correctness and quality. This setup has a blind spot. When the search step quietly drops evidence, the judge has no way to notice. Search drops evidence in predictable ways: small trials, regional studies, and papers outside a single index are the usual casualties. A review built on such a skewed set of studies can reach the wrong conclusion while being perfectly faithful to every paper it cites, so it will score well.
We propose to measure how often this happens and to test a judging protocol designed to catch it. The protocol has two parts. The judge is given a broader reference pool of studies drawn from several sources. Before it reads the candidate review, it must reason through which studies the review left out and whether including them would change the conclusion. The central claim fits in two sentences. Retrieval bias changes the conclusions of machine-written syntheses, and current judges reward those wrong conclusions about as much as correct ones. Making the judge reason about omissions before it reads the review's prose is what closes the gap, and having more context or using free-form reasoning does not.
Core Claim
Three findings in the file motivate this work. Do AI chatbots find what experts would? tested chatbot study retrieval against 20 Cochrane reviews. It found that a single response recovers on average 39.2% of the included studies, and that sample size was the only independent predictor of whether a study was retrieved. The authors call this "landmark bias." That paper evaluated retrieval only and explicitly did not ask whether the omissions change clinical conclusions.
The MedEvidence benchmark (Can Large Language Models Match the Conclusions of Systematic Reviews?) studies the opposite end of the pipeline. It hands models pre-screened Cochrane study sets and finds that accuracy falls from about 92% when the studies agree to about 41% when they conflict. Because the study sets are pre-screened, the benchmark bypasses retrieval entirely.
Articulate Intuition or Genuine Analysis? shows that judges of peer reviews reward analytical form over epistemic function: about 74% of the apparent quality advantage of AI-written reviews was explained by length and venue. Those judges also worked without access to the underlying papers.
No one has connected these three facts. Our claim is that the missing link is omission. Landmark bias removes exactly the small and often discordant trials that create conflict. Removing them makes the evidence look more concordant than it is, and the result is confident, cleanly argued, wrong syntheses. A judge restricted to the cited set is structurally unable to penalize such a synthesis, and its preference for polished form makes it likely to reward it.
Research Program
The work has three stages. Each stage depends on the result of the one before.
Stage 1: Does biased retrieval change conclusions? We build a testbed with a known full evidence pool. For every MedEvidence question, we know the complete set of studies the Cochrane panel used and the panel's conclusion. We tag each study with:
- its sample size,
- its language and country,
- whether it appears in an open-access index comparable to the OpenScholar DataStore of 45 million open-access papers (Synthesizing scientific literature with retrieval-augmented language models).
From these tags we create degraded pools under three bias models.
- Size-weighted dropout. Retrieval probability rises with sample size. We calibrate it so overall recall matches the 39.2% reported by the chatbot-retrieval study.
- Single-source restriction. We keep only studies present in one index, either the open-access index or PubMed-only records.
- Random dropout. We drop studies at random at matched recall. This is the control that separates bias from mere loss of evidence.
For the Cochrane 2026 reviews, which postdate model training cutoffs, we also re-collect real chatbot retrievals using the published persona protocol. This gives a naturally biased pool in addition to the simulated ones.
The first quantity we measure is the conclusion flip rate: the fraction of questions where a fixed synthesizer reaches a different conclusion from the degraded pool than from the full pool. Random dropout is the baseline flip rate. If size-weighted dropout flips conclusions more often than random dropout at the same recall, that would be the first direct evidence that landmark bias changes machine-generated reviews, and not just their coverage.
Stage 2: Can current judges see the damage? We test judges on matched pairs, borrowing the logic of the function-probe pilots in the peer-review benchmark. Each pair consists of two syntheses of the same question by the same model, one written from the full pool and one from a biased pool. We keep only pairs whose conclusions differ and where only the full-pool version agrees with Cochrane. We length-match the pairs so the verbosity bias catalogued by CALM (Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge) cannot explain the results.
The judges use the rubric styles currently applied to scientific synthesis: the eight-dimension protocol of SciSurveyBench (Automating Cognitive Distillation for Expert-Level Scientific Literature Synthesis) and the ScholarQABench correctness rubrics. They see only the papers each synthesis cites. The prediction is that they will score the wrong, biased-pool synthesis about as high as the correct one, and sometimes higher, because the biased one reads as more decisive.
Stage 3: A judging protocol that reasons about omissions first. The judge receives a reference pool built as the union of several sources. In MedEvidence this is the full Cochrane set. In open-domain settings it is the union of open-access dense retrieval, PubMed, and chatbot search. The judge then proceeds in a fixed order, and it does not see the candidate's prose until the second step.
- It lists the reference-pool studies the candidate did not cite, together with their sizes and reported directions of effect.
- It commits to a judgment of whether those studies, taken with the cited ones, support the same conclusion, a weaker one, or an opposite one.
- Only then does it read the candidate's argument and score it.
The ordering is the key ingredient, and two findings motivate it. Unveiling Confirmation Bias in Chain-of-Thought Reasoning shows that step-by-step reasoning tends to become a justification for an early hunch, and that supplying explicit facts reduces this effect. A judge that reads a fluent synthesis first acquires exactly such a hunch. Forcing the reasoning about omissions to happen before the prose is visible is meant to prevent that.
This judge does not have to be a good synthesizer in absolute terms. Its job is relative: it only has to detect that the evidence the candidate dropped would move the conclusion. That task may be easier than full synthesis, but whether it actually is easier is an empirical question we test, not an assumption.
Baselines
Do-nothing reference. The do-nothing reference is the judge as currently deployed: a frontier model scoring with the SciSurveyBench or ScholarQABench rubric, seeing only cited papers, with no instructed reasoning. It defines how blind current practice is. On matched pairs, we expect its score difference between correct and omission-damaged syntheses to be close to zero.
Strongest existing method. The strongest existing judging method in the file is a rubric judge that reasons step by step before scoring. CALM reports that this "generally improved judgment accuracy" across judges, and it found Claude 3.5 Sonnet to be the most robust judge overall. We run that model and one open model with identical rubrics, identical candidate syntheses, and the same token budget for reasoning as our protocol. One difference is unavoidable: this baseline sees only the cited papers. The file contains no reported result for any judge's sensitivity to omission-induced conclusion errors. The nearest number is the 33-of-35 matched-pair discrimination ($d_z = +2.52$) in the peer-review benchmark, which concerns fault-finding in review text rather than omissions.
Context-only ablation. This ablation removes the ordering and keeps everything else. The judge receives the same multi-source reference pool and the same compute budget but uses free-form reasoning with the candidate visible from the start. Comparing it against our protocol isolates whether the gain comes from simply having more evidence or from reasoning about omissions before reading the candidate.
Feature heuristic. This baseline uses no language model. It is a logistic model that flags likely-wrong syntheses from features of the evidence pool alone:
- the fraction of reference studies omitted,
- the omitted share of total sample size,
- concordance of reported effect directions among the omitted studies.
It is fit on held-out MedEvidence questions.
Measures and the bar for success. We measure:
- the area under the receiver operating characteristic curve (AUROC) for separating correct syntheses from omission-damaged ones,
- the paired score difference on matched pairs.
Beating the baselines means two things. Our protocol must improve AUROC over both the context-only ablation and the feature heuristic, with confidence intervals that do not overlap under bootstrap over questions. It must also produce a paired difference in the correct direction on a clear majority of pairs.
Why not just…?
Why not just fix retrieval, for example by taking the union of several databases and chatbots? Better retrieval is the right fix for the systems that produce reviews, and nothing here argues against it. It is also a real threat to the urgency of Stage 1. If unions of sources push recall high enough, the flip rate may become negligible. The chatbot-retrieval study's best single system still recovered only 63.1% of included studies, and its results show that recall is driven by how many studies a response cites, which suggests a ceiling for any single source. We will report flip rate as a function of recall, including for multi-source unions.
Even with better production retrieval, the evaluation problem remains. A benchmark judge must be able to tell whether a given system's retrieval was adequate, and it cannot do that from the cited set alone.
Why not just compare the candidate's conclusion to the gold conclusion? On MedEvidence this is correct, and we use it as ground truth there. A reviewer would rightly note that no judge is needed when gold conclusions exist. That is why MedEvidence serves only for calibration. The protocol is aimed at open-ended synthesis, such as ScholarQABench long-form answers and SciSurveyBench surveys, where no categorical gold conclusion exists. The SurveyMaster authors explicitly could not verify human surveys against full texts at scale. The test of value is whether the protocol's MedEvidence accuracy carries over to an expert-checked subset of open-domain tasks.
Why not just prompt the synthesizer to hedge when evidence is thin, or use a reasoning model? The MedEvidence results argue against relying on reasoning models: models such as o1 and R1 did not consistently outperform standard models. The confirmation-bias results predict that unstructured reasoning will justify whatever conclusion the retrieved pool suggests.
Hedging instructions could still raise calibration cheaply. We include a hedge-prompted synthesizer in Stage 1. If it eliminates the gap in flip rate between size-weighted and random dropout, the synthesis problem is largely solved by prompting. The judging question would then shrink to whether judges reward appropriate hedging, which we would report.
Why not just use the feature heuristic? This is the most serious alternative. If simple features of the evidence pool predict omission damage as well as the language-model judge does, the right contribution is the measurement study plus that heuristic, and we will say so. The heuristic cannot read what the omitted studies actually conclude in prose. It also cannot weigh study quality, which is the core of the skepticism gap. Our protocol has to show it adds value on exactly those cases, so we will report AUROC separately for questions where omitted studies are concordant and where they are discordant with the cited ones.
Proposed Experiments and Decisive Checks
Experiment 1: propagation. This runs Stage 1 across all 284 MedEvidence questions with two synthesizers that the MedEvidence paper reports near its 60–62% accuracy ceiling (DeepSeek-V3 and GPT-4.1), plus OpenScholar-8B as a retrieval-based system. We report flip rate under each bias model at matched recall, broken down by whether the full pool contains conflicting studies.
- Decisive check: size-weighted dropout must produce a higher flip rate than random dropout at the same recall.
- Falsification: if it does not, landmark bias reduces coverage without changing conclusions, and the rest of the program loses its motivation.
Experiment 2: judge blindness. This builds the length-matched pair set from Experiment 1 and scores it with the do-nothing judge and the CALM-style reasoning judge.
- Decisive check: the paired score difference between correct and damaged syntheses should be indistinguishable from zero or negative.
- Also tested: whether that difference tracks the single "analytical-trace" factor the peer-review benchmark found judges collapse into.
Experiment 3: omission-first judging. This compares our protocol against the context-only ablation and the feature heuristic on the same pairs, using AUROC and paired differences.
- Decisive check: the protocol must beat the context-only ablation. Otherwise ordering is irrelevant, and the honest recommendation is simply to give judges broader reference pools.
- Second check: whether the protocol beats the heuristic on the discordant-omission subset.
Experiment 4: transfer. This moves to the biomedicine portion of ScholarQABench. We construct multi-source reference pools and have domain experts annotate, for a sample of OpenScholar and GPT-4o answers, whether omitted studies change the answer's main claim. Experts preferred OpenScholar-GPT-4o over human answers 70% of the time, so this experiment also asks whether some of that preference hides omission errors invisible to evaluators who see only what was cited.
Decisive outcome. The program succeeds if two things hold. Landmark-style retrieval must flip conclusions beyond chance. And only judges that reason about omitted studies before reading the candidate must be able to detect those flips.
References
- Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions
- Can Large Language Models Match the Conclusions of Systematic Reviews?
- Articulate Intuition or Genuine Analysis? Benchmarking Epistemic Reliability in LLM-as-a-Judge Peer Review
- Synthesizing scientific literature with retrieval-augmented language models
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- Automating Cognitive Distillation for Expert-Level Scientific Literature Synthesis
- Unveiling Confirmation Bias in Chain-of-Thought Reasoning