FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification

Ling Yue, Chaoqian Ouyang, Hang Xu, Ruijun Huang, Yuchen Liu, Libin Zheng, Wei Liu, Shaowu Pan, Shimin Di, Min-Ling ZhangarXiv 2026 · arXiv

TL;DR

FactReview is an automated peer-review pipeline that extracts review-relevant claims from AI/ML manuscripts and audits them against external literature and actual code execution. By grounding evaluations in real-world evidence rather than just text, it significantly improves review quality and cuts human reviewer time by up to 58% while expanding claim coverage.

What Problem Does This Paper Address?

Large Language Models (LLMs) are increasingly being deployed to assist with peer review, but they typically evaluate manuscripts in isolation. If a paper claims to achieve state-of-the-art (SOTA) results or introduces a completely novel method, a standard LLM reviewer takes it at face value. It cannot verify if the code actually runs, if the reported metrics match the code's outputs, or if a closely related paper already solved the exact same problem. This results in superficial, uninformative reviews that fail to flag overclaims or reproducibility issues.

Methods

FactReview introduces an end-to-end pipeline consisting of four main stages:

  1. Document Parsing and Claim Extraction: Using MinerU, the system parses the PDF into structured text and uses an LLM to break down broad assertions into atomic, testable claims (e.g., specific datasets, metrics, and baselines).
  2. Literature Positioning and Reference Verification: It queries Semantic Scholar and OpenAlex to build a comparison matrix of related work, while using RefCopilot to check for citation hallucinations or integrity issues.
  3. Execution-Based Claim Verification: For papers with public code repositories, FactReview spins up a sandboxed workspace. It automatically infers tasks, sets up environments, and runs the code. Crucially, it features a Run-Review-Fix (RRF) loop that allows up to 3 rounds of minor automated repairs (like fixing dependency paths or launch arguments) without altering the core scientific logic.
  4. Claim Assessment & Review Synthesis: Finally, it cross-references the extracted claims with the gathered evidence, assigning each claim one of four labels: Supported, Partially supported, In conflict, or Inconclusive. It then generates a concise review linked directly to an evidence report.
Figure 2: FactReview extracts review-relevant claims, constructs manuscript, literature, reference-integrity, and optional execu- tion evidence, assigns conservative outcomes, and links the review to evidence.

Datasets

To evaluate FactReview, the authors constructed a custom benchmark of 35 machine-learning papers containing 495 human-verified reference claims.

  • Development Set: Consists of 9 papers (including CompGCN) used to tune prompts, thresholds, and rules.
  • Test Set: Consists of 26 paper-disjoint manuscripts (24 from OpenReview with known accept/reject outcomes, and 2 from arXiv). Out of these, 12 papers had usable code repositories for the execution-based evaluation.

Key Insights and Ideas

  • Code Execution is the Ultimate Truth-Teller: Ablation studies revealed that removing execution evidence changed 17.0% of claim statuses—more than removing any other single evidence source (like literature retrieval).
  • Atomic Splitting is Essential: Broad claims (e.g., "Our model outperforms all baselines across all tasks") must be broken down. A paper might have runnable code that reproduces one specific local result, but fails to support the broader generalized claim.
  • Automated Code Repair is Viable: The Run-Review-Fix (RRF) loop successfully resolved common environment and dependency blockers, boosting the paper execution success rate from 55% to 65% without violating scientific integrity.

Key Results

  • High Accuracy: FactReview achieved an 84.3% F1 score for claim recovery and a 94.7% status agreement rate on matched claims.
  • Human-Approved Quality: In a blind controlled study, human evaluators rated FactReview 4.72/5 overall, significantly outperforming a direct LLM reviewer (3.98/5).
  • Massive Time Savings: When human reviewers were assisted by FactReview's reports and visual summaries, their average review time dropped by 58% (from 50.6 minutes down to 21.3 minutes), while their coverage of major benchmark claims jumped from 87% to 99%.
Figure 4: Reviewer-assistance results for 20 participants and five papers. Left: mean review time. Right: benchmark-claim coverage. Time excludes instruction and setup; coverage is the proportion of benchmark major claims addressed.

Limitations and Future Work

The authors note several limitations: the evaluation test set is relatively small (26 papers) and strictly limited to the machine-learning domain. Additionally, the claim-recovery metrics depend on an author-constructed reference inventory, and the code execution is bound to a single type of sandboxed harness. Future work includes expanding the system to evaluate broader paper populations across diverse scientific domains and testing its robustness against alternative execution harnesses and LLM backends.

Summary updated

Abstract

Large language model (LLM)-based reviewing systems typically assess manuscripts in isolation, leaving literature- and code-dependent claims difficult to verify. We present FactReview, an audit pipeline that extracts review-relevant claims, grounds them in related work and reference checks, and, when code is available, executes released artifacts under a fixed repair budget. On 26 paper-disjoint test papers with 354 human-verified claims, FactReview achieves 84.3\% F1 for claim recovery. In a same-backend, evidence-matched comparison, FactReview scores 4.72/5 overall, outperforming a direct LLM reviewer by 0.74 points. Removing execution evidence changes 17.0\% of claim statuses, more than removing any other single evidence source. In a reviewer-assistance study, FactReview reduces mean review time by 58\% while increasing benchmark-claim coverage from 87\% to 99\%. FactReview supports evidence-based claim auditing, with acceptance decisions reserved for human reviewers. The code is public at https://github.com/DEFENSE-SEU/FactReview.
BibTeX
@misc{yue2026factreview,
  author = {Ling Yue and Chaoqian Ouyang and Hang Xu and Ruijun Huang and Yuchen Liu and Libin Zheng and Wei Liu and Shaowu Pan and Shimin Di and Min-Ling Zhang},
  title = {{FactReview: Evidence-Grounded Peer Review with Execution-Based Claim Verification}},
  year = {2026},
  eprint = {2604.04074},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.04074},
}
Figure