Auditing AI peer reviewers: a benchmark on real scientific papers

Íñigo Zubeldia, Boris Bolliet, Francisco Villaescusa-Navarro, Pablo Villanueva-Domingo, Miles CranmerarXiv 2026 · arXiv

TL;DR

Researchers have developed a rigorous new benchmark for AI peer reviewers by injecting objective, localizable errors—ranging from subtle typos to blatant mathematical absurdities—into real scientific papers. They also introduced SKEPTHICAL, a multi-agent system that outperforms standard LLMs by using external tools like computer algebra and automated citation retrieval to verify claims.

What Problem Does This Paper Address?

Large Language Models (LLMs) are already being used to review scientific papers at a "production scale," yet we lack objective ways to measure their reliability. Previous evaluations typically relied on comparing AI reviews to subjective human reviews or using purely synthetic datasets.

The authors argue that we need to move beyond "vibe-based" evaluation. They focus on objective, localizable defects: Can the AI catch a sign flip in an equation? Does it notice when a citation is misrepresented? Does it flag numerical inconsistencies between an abstract and a table? By injecting known errors into real LaTeX sources, the authors create a "ground truth" for what a reviewer should catch, while also measuring the "false positive" rate—how often the AI hallucinates errors in perfectly clean papers.

Methods

The paper introduces two major components:

  1. The Evaluation Pipeline: The authors took 30 real arXiv preprints and created four versions of each: one clean and three with injected errors. Errors were categorized into four types (General, Mathematics, Numerics, Citations) and three severity levels (Subtle, Clear, Blatant). To determine if a reviewer "caught" an error, they used a cross-family "LLM-as-judge" panel (GPT, Claude, and Gemini) to adjudicate the reviews.
  2. SKEPTHICAL: A novel multi-agent reviewer designed to overcome the limitations of pure LLMs. It consists of four specialist modules:
  3. General Issues: An ensemble of frontier models (Claude Opus and Gemini Pro) merged by a third model (GPT-5.1).
  4. Mathematics: Extracts equations and verifies them using WolframScript (for symbolic equality) and Pint (for dimensional analysis).
  5. Numerics: Uses Python code execution to check internal consistency of numbers.
  6. Citations: This is the "killer app"—it identifies cited claims, retrieves the actual PDFs of the cited papers from arXiv, and uses an LLM to verify the claim against the source text.

Datasets

The study uses two newly introduced corpora:

  • Astrophysics Corpus: 15 recent preprints from astro-ph, chosen for their heavy use of formal mathematics and numerical results.
  • Machine Learning (ML) Corpus: 15 recent preprints from cs.LG, used to see if models perform better on the data they were likely trained on.
  • Total Errors: 357 injected errors across the 30 papers.

The authors have released the evaluation pipeline, injected errors, and adjudication logs as a public benchmark.

Key Insights and Ideas

  • Model Choice is Paramount: Even with identical prompts, frontier models (like GPT-5.5 vs. Gemini 3.1 Pro) can differ by as much as 50 percentage points in detection rate.
  • Scaffolding vs. Raw Power: "Agentic scaffolding"—giving the model tools and a structured workflow—significantly boosts performance. For example, Claude Code (an agentic wrapper) performed much better than a single call to the underlying Claude model.
  • The "Knowledge Cutoff" Advantage: In the ML corpus, LLMs were surprisingly good at catching citation errors even without retrieval, likely because the "frontier" models are already familiar with the core ML literature. However, in the Astrophysics corpus, SKEPTHICAL’s mandatory citation retrieval gave it a massive lead, catching errors that models simply didn't have the "background knowledge" to spot.
  • Math is the False Positive King: Most AI reviewers have a low false-positive rate (under 10%), but when they do hallucinate errors, they are most likely to do so in mathematical sections.

Key Results

The quantitative results highlight a clear hierarchy in reviewer capability. As shown in the detection rates below, SKEPTHICAL and structured frontier models lead the pack.

Table 2: Error detection rate (as %) by error type for all the reviewers on the astrophysics corpus (177 injections, top block) and the ML corpus (180 injections, bottom block). Quoted uncertainties are paper-level bootstrap 95 % confidence intervals (CIs); bold corresponds to column-best within each block.
  • Detection vs. Severity: As expected, "blatant" errors are caught more often than "subtle" ones. However, even the best systems struggle with subtle citation and math errors.
  • SKEPTHICAL's Performance: It achieved the highest overall detection rate on the Astrophysics corpus (70.6%) and a strong 76.7% on ML.
  • Open-Weights Struggle: Smaller, fine-tuned models like OpenReviewer-8B (based on Llama) performed poorly, often failing to detect almost any injected errors (detecting ~1-2%).
Figure 1: Error detection rate across error severity, error type, and architectural family for all our reviewers, on the astrophysics corpus (top block) and the ML corpus (bottom block). In each panel, the x-axis is the injected error severity, the y-axis the detection rate, and the error bars are paper-level bootstrap CIs, with the solid whisker showing the 68 % interval and the thinner half-transparent overlay the 95 % interval.

Limitations and Future Work

  • Subjectivity Gap: The benchmark intentionally ignores "high-level" review qualities like assessing a paper's significance, novelty, or the adequacy of its experimental design, which remain difficult to quantify objectively.
  • Scale: The corpus is relatively small (30 papers). Expanding this to hundreds of papers across more diverse fields (Biology, Chemistry, etc.) is a logical next step.
  • LLM-as-Judge: While the authors validated their LLM adjudication panel against a human annotator (finding 96% agreement), the evaluation still relies on LLMs to grade other LLMs, which could introduce subtle biases.
  • Cost: Running a full SKEPTHICAL audit is significantly more expensive (~$2.41 per paper) than a single LLM call, though still far cheaper than human labor.

Summary updated

Abstract

LLM-based peer review is now used at production scale, but its evaluation remains limited. We introduce a benchmark that injects objective, localisable errors of four types (general issues, mathematics, numerics, citations) at three severity levels into real papers and measures each reviewer's detection rate by error type and severity, together with its false-positive rate on clean copies. We also present Skepthical, a multi-agent reviewer that verifies citation-backed claims by retrieving the cited papers, mathematics with computer algebra, and numerical claims with executed code. We apply the benchmark to two corpora of 15 recent arXiv preprints each, one in astrophysics and one in machine learning (ML), injecting a total of 357 errors, and evaluate twelve reviewer systems: six single-LLM reviewers, two Claude Code configurations, three publicly available reviewers, and Skepthical. The choice of model has the largest impact on detection rate: given the same prompt, three frontier LLMs differ by about 50 percentage points in total detection on both corpora. Tools and scaffolding help when they contribute what the model cannot supply on its own. In particular, on astrophysics, Claude Code improves on a single Claude call in mathematics and numerics detection, and enforced citation retrieval gives Skepthical the highest citation error detection rate of all systems by a large margin. This citation lead, however, is not observed on ML, where the models appear to know the literature better and Skepthical's citation detection rate is statistically indistinguishable from that of the best single LLM. False-positive rates are below 10% for most systems, but vary with prompt, model, and scaffolding, with mathematics flags the main source of false positives. The evaluation pipeline, the injected errors, and the adjudication logs are released as a benchmark.
BibTeX
@misc{zubeldia2026auditing,
  author = {Íñigo Zubeldia and Boris Bolliet and Francisco Villaescusa-Navarro and Pablo Villanueva-Domingo and Miles Cranmer},
  title = {{Auditing AI peer reviewers: a benchmark on real scientific papers}},
  year = {2026},
  eprint = {2611.17052},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17052},
}
Figure