OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews

Maximilian Idahl, Zahra AhmadiNAACL 2025

TL;DR

This paper introduces OpenReviewer, a specialized large language model (LLM) fine-tuned on a massive dataset of scientific paper reviews. OpenReviewer generates more critical and realistic reviews than general-purpose LLMs, aligning better with human reviewer judgments and providing valuable pre-submission feedback for authors.

What Problem Does This Paper Address?

The academic peer review process, especially in fast-moving fields like machine learning and AI, is facing significant challenges due to surging submission volumes. This leads to increased workload for reviewers and a lack of timely, expert feedback for authors before submission. While LLMs show promise in academic content generation, existing models often struggle to produce high-quality, critical reviews that adhere to the depth, specificity, and structured format expected in academic peer review. They tend to be overly positive and may miss field-specific conventions or fail to properly evaluate technical contributions.

Methods

The core of OpenReviewer is Llama-OpenReviewer-8B, an 8-billion parameter LLM that has been fine-tuned on a curated dataset of 79,000 expert reviews from top ML conferences (ICLR and NeurIPS). The system takes a PDF paper submission and a review template as input. It first uses transformer-based PDF processing models to extract the full text, including technical content like equations and tables, and converts it to markdown. Users can then edit this text. OpenReviewer is trained to follow conference-specific review templates and guidelines, ensuring structured and critical feedback. The model was fine-tuned for three epochs using a batch size of 64 and a learning rate of 2x10⁻⁵, employing techniques like Flash Attention V2 and Deepspeed ZeRO-3 for efficient training.

Datasets

The training dataset for Llama-OpenReviewer-8B was collected from OpenReview, comprising 36,000 submitted papers and 141,000 reviews from the International Conference on Learning Representations (ICLR) and the Conference on Neural Information Processing Systems (NeurIPS), specifically focusing on editions from 2022 onwards. The earliest revisions of papers were used to avoid incorporating feedback from later revisions. The evaluation was conducted on a test set of 400 held-out papers and their reviews from NeurIPS 2024 and ICLR 2025.

Key Insights and Ideas

  • Specialized fine-tuning is crucial for critical reviews: Unlike general-purpose LLMs that tend to be overly positive, OpenReviewer's specialized training on expert reviews allows it to generate more critical and realistic feedback, mirroring human reviewer sentiment.
  • LLMs can be a valuable pre-submission tool: OpenReviewer demonstrates the potential of LLMs to provide authors with rapid, constructive feedback to improve their manuscripts before formal submission, without replacing human peer review.
  • Matching human reviewer judgments is a key metric: The evaluation focuses on how well generated reviews align with human reviewer ratings and recommendations, suggesting that this alignment is a strong indicator of review quality.

Key Results

  • Recommendation Alignment: OpenReviewer's recommendations closely matched human reviewer judgments, achieving an exact match rate of 55.5% and an average error of 0.96 on a 1-10 scale. This significantly outperformed general-purpose LLMs like GPT-40 (23.8% exact match, 2.34 error) and Claude-3.5-Sonnet (15.5% exact match, 2.77 error).
  • Criticality: OpenReviewer's average recommendation score was 5.4, aligning with the human reviewer distribution, while other LLMs produced significantly higher average recommendations (6.9 to 8.1), indicating a tendency towards overly positive feedback.
  • LLM-as-a-judge Evaluation: In an arena-style evaluation using GPT-40 as a judge, OpenReviewer achieved win rates ranging from 60% against GPT-40 to 76% against Llama-3.1-70B-Instruct, indicating that human-like expert reviewers preferred OpenReviewer's output.

Limitations and Future Work

Limitations: * The study's dataset is restricted to ML/AI conferences from 2022 onwards, potentially limiting its applicability to other domains. * The PDF-to-markdown conversion accuracy can impact the quality of extracted text. * The 8B parameter size of the model, while balancing performance and accessibility, might limit its capabilities compared to larger models. * The evaluation relied on automated metrics and a limited set of baseline models due to resource constraints. * There's a risk of authors gaining false confidence from automated feedback alone. * The system's bias towards specific conferences and review templates, and its limited handling of figures and images, are also noted.

Future Work: * Expand training data to cover more venues and domains. * Incorporate related literature search and citation graph information to improve novelty assessment. * Develop better automatic evaluation metrics for review quality. * Create interfaces for collaborative human-AI reviewing. * Encourage further research into bias detection and mitigation.

Summary updated

BibTeX
@inproceedings{idahl2025openreviewer,
  author = {Maximilian Idahl and Zahra Ahmadi},
  title = {{OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure