Abstract
Large language models can generate fluent peer reviews, yet their assessments often lack sufficient critical rigor when substantive issues are subtle and distributed across a paper. In this paper, we introduce PaperAudit-Bench, which consists of two components: (1) PaperAudit-Dataset, an error dataset covering both errors identifiable within individual sections and those requiring cross-section reasoning, designed for controlled evaluation under long-context settings; and (2) PaperAudit-Review, an automated review framework that integrates structured error detection with evidence-aware review generation to support critical assessment. Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths, highlighting the difficulty of identifying such errors under long-context settings. Relative to representative automated reviewing baselines, incorporating explicit error detection into the review workflow produces systematically stricter and more discriminative evaluations, demonstrating its suitability for peer review. Finally, we show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost.
Introduction: The Need for Critical Rigor in AI Peer Review
Large Language Models (LLMs) have demonstrated a remarkable ability to generate professional-sounding peer reviews. However, these reviews often suffer from a lack of critical rigor, frequently resulting in "fluffy" summaries that miss substantive, distributed errors. Current models may overlook contradictions between an abstract and an appendix or fail to notice when experimental data does not support the narrative. This "polite bias" leads to overly permissive evaluations of flawed research.
PaperAudit-Bench addresses this gap by providing a systematic way to evaluate how well AI can detect "paper-level" errors. By moving beyond simple summarization toward rigorous scientific auditing, this work aims to create more reliable automated assistants for the scientific community.
Constructing the PaperAudit-Dataset
The researchers developed the PaperAudit-Dataset to serve as a rigorous testing ground for error detection. They selected 220 high-quality "oral" papers from premier AI conferences (ICLR, ICML, and NeurIPS 2025) and utilized a model-driven synthetic editing process to inject 10–20 realistic "research bugs" into each.
As shown in the pipeline above, these errors are not mere typos but substantive issues categorized into eight risk dimensions, including:
- Method Logic Errors (ML): Flaws in the underlying technical reasoning.
- Evidence Manipulation (CE): Altering results to appear more favorable.
- Claim Distortion (CD): Misrepresenting the significance of findings.
- Context Misalignment (CM): Inconsistencies between different sections of the paper.
The PaperAudit-Review Framework
To tackle these complex errors, the authors proposed PaperAudit-Review, a multi-tiered framework designed to enhance an LLM's analytical depth. The framework operates in three distinct modes:
- Fast Mode: A single-pass scan for high-level issues.
- Standard Mode: A section-aware approach utilizing a "Global Memory" to maintain context across the entire document while auditing individual sections.
- Deep Mode: A sophisticated multi-agent workflow where a Planner generates a task list, a Retriever gathers evidence, and a Specialist performs the final audit.
By forcing the model to perform explicit error detection before generating a review, the framework produces evaluations that are systematically stricter and more discriminative, aligning more closely with high-quality human reviews.
Key Results and Insights
Experiments on PaperAudit-Bench reveal that paper-level auditing remains a significant challenge for even the most advanced models. Top-tier models like GPT-5 and Gemini 2.5 Pro achieved Macro-F1 scores of only 40-41% in Fast mode. However, the multi-agent Deep Mode significantly improved performance; for instance, GPT-5's error coverage increased from 67.7% to 80.6%.
Other key findings include:
- The Abstract Blind Spot: Errors located in the Abstract were consistently the hardest to detect, suggesting models (like humans) often accept high-level claims at face value.
- Global Memory is Essential: Structured memory is required to bridge distant sections and identify distributed contradictions.
- Lightweight Auditing: Smaller models (7B to 14B parameters) can be highly effective. A Qwen3-8B model trained with Supervised Fine-Tuning (SFT) and Reinforcement Learning (GRPO) achieved an F1 score of 59.7%, rivaling much larger general-purpose models.
Conclusion and Future Directions
PaperAudit-Bench marks a transition in AI peer review from fluent summarization to critical auditing. While the synthetic nature of the errors and the potential for over-criticality in Deep mode remain areas for future refinement, the framework provides a robust foundation for building reliable automated reviewers. The authors suggest these tools are currently best suited for pre-submission self-auditing, helping researchers identify subtle flaws before their work reaches human reviewers.
Summary updated