TL;DR
Researchers have developed a rigorous new benchmark for AI peer reviewers by injecting objective, localizable errors—ranging from subtle typos to blatant mathematical absurdities—into real scientific papers. They also introduced SKEPTHICAL, a multi-agent system that outperforms standard LLMs by using external tools like computer algebra and automated citation retrieval to verify claims.
What Problem Does This Paper Address?
Large Language Models (LLMs) are already being used to review scientific papers at a "production scale," yet we lack objective ways to measure their reliability. Previous evaluations typically relied on comparing AI reviews to subjective human reviews or using purely synthetic datasets.
The authors argue that we need to move beyond "vibe-based" evaluation. They focus on objective, localizable defects: Can the AI catch a sign flip in an equation? Does it notice when a citation is misrepresented? Does it flag numerical inconsistencies between an abstract and a table? By injecting known errors into real LaTeX sources, the authors create a "ground truth" for what a reviewer should catch, while also measuring the "false positive" rate—how often the AI hallucinates errors in perfectly clean papers.
Methods
The paper introduces two major components:
- The Evaluation Pipeline: The authors took 30 real arXiv preprints and created four versions of each: one clean and three with injected errors. Errors were categorized into four types (General, Mathematics, Numerics, Citations) and three severity levels (Subtle, Clear, Blatant). To determine if a reviewer "caught" an error, they used a cross-family "LLM-as-judge" panel (GPT, Claude, and Gemini) to adjudicate the reviews.
- SKEPTHICAL: A novel multi-agent reviewer designed to overcome the limitations of pure LLMs. It consists of four specialist modules:
- General Issues: An ensemble of frontier models (Claude Opus and Gemini Pro) merged by a third model (GPT-5.1).
- Mathematics: Extracts equations and verifies them using WolframScript (for symbolic equality) and Pint (for dimensional analysis).
- Numerics: Uses Python code execution to check internal consistency of numbers.
- Citations: This is the "killer app"—it identifies cited claims, retrieves the actual PDFs of the cited papers from arXiv, and uses an LLM to verify the claim against the source text.
Datasets
The study uses two newly introduced corpora:
- Astrophysics Corpus: 15 recent preprints from
astro-ph, chosen for their heavy use of formal mathematics and numerical results. - Machine Learning (ML) Corpus: 15 recent preprints from
cs.LG, used to see if models perform better on the data they were likely trained on. - Total Errors: 357 injected errors across the 30 papers.
The authors have released the evaluation pipeline, injected errors, and adjudication logs as a public benchmark.
Key Insights and Ideas
- Model Choice is Paramount: Even with identical prompts, frontier models (like GPT-5.5 vs. Gemini 3.1 Pro) can differ by as much as 50 percentage points in detection rate.
- Scaffolding vs. Raw Power: "Agentic scaffolding"—giving the model tools and a structured workflow—significantly boosts performance. For example, Claude Code (an agentic wrapper) performed much better than a single call to the underlying Claude model.
- The "Knowledge Cutoff" Advantage: In the ML corpus, LLMs were surprisingly good at catching citation errors even without retrieval, likely because the "frontier" models are already familiar with the core ML literature. However, in the Astrophysics corpus, SKEPTHICAL’s mandatory citation retrieval gave it a massive lead, catching errors that models simply didn't have the "background knowledge" to spot.
- Math is the False Positive King: Most AI reviewers have a low false-positive rate (under 10%), but when they do hallucinate errors, they are most likely to do so in mathematical sections.
Key Results
The quantitative results highlight a clear hierarchy in reviewer capability. As shown in the detection rates below, SKEPTHICAL and structured frontier models lead the pack.
- Detection vs. Severity: As expected, "blatant" errors are caught more often than "subtle" ones. However, even the best systems struggle with subtle citation and math errors.
- SKEPTHICAL's Performance: It achieved the highest overall detection rate on the Astrophysics corpus (70.6%) and a strong 76.7% on ML.
- Open-Weights Struggle: Smaller, fine-tuned models like OpenReviewer-8B (based on Llama) performed poorly, often failing to detect almost any injected errors (detecting ~1-2%).
Limitations and Future Work
- Subjectivity Gap: The benchmark intentionally ignores "high-level" review qualities like assessing a paper's significance, novelty, or the adequacy of its experimental design, which remain difficult to quantify objectively.
- Scale: The corpus is relatively small (30 papers). Expanding this to hundreds of papers across more diverse fields (Biology, Chemistry, etc.) is a logical next step.
- LLM-as-Judge: While the authors validated their LLM adjudication panel against a human annotator (finding 96% agreement), the evaluation still relies on LLMs to grade other LLMs, which could introduce subtle biases.
- Cost: Running a full SKEPTHICAL audit is significantly more expensive (~$2.41 per paper) than a single LLM call, though still far cheaper than human labor.
Summary updated