Generative Reviewer Agents: Scalable Simulacra of Peer Review

Nicolas Bougie, Narimawa WatanabeEMNLP 2025

TL;DR

This paper introduces Generative Agent Reviewers (GAR), an LLM-powered framework that simulates human peer reviewers to provide scalable and detailed scientific manuscript evaluations. GAR aims to democratize access to expert-level feedback, improve the efficiency of the peer review process, and assist human reviewers.

What Problem Does This Paper Address?

The scientific peer review process is crucial for ensuring the quality and rigor of published research. However, it faces significant challenges: the sheer volume of scholarly output is overwhelming, leading to reviewer fatigue; specialization creates knowledge silos, making it hard for reviewers to assess diverse topics; and access to high-quality, in-depth feedback is often limited to researchers with established networks. These issues strain traditional mechanisms and can slow down scientific progress. This paper tackles these problems by proposing an AI-driven solution that can simulate faithful peer reviewers.

Methods

GAR employs a novel architecture that extends Large Language Models (LLMs) with memory capabilities and reviewer personas. Key components include:

  • Graph Construction: Manuscripts are converted into a knowledge graph, mapping relationships between ideas, claims, and evidence. This structured representation helps LLMs cross-reference and analyze complex papers more effectively.
  • Reviewer Personas: Agents are equipped with distinct personas (e.g., strictness, expertise, focus areas) derived from historical data through contrastive comparison. This allows for more nuanced and human-like evaluations.
  • Memory Module: Agents leverage a memory module to retrieve past reviews and community descriptors, enabling retrieval-augmented reviews and providing context for their assessments.
  • Multi-Round Evaluation: The review process involves multiple rounds of refinement, where agents iteratively improve their feedback based on retrieved information and previous assessments.
  • Novelty Module: This module assesses a manuscript's originality by comparing it against prior research using external knowledge sources.
  • Meta-Reviewer: A final agent synthesizes individual reviews to make a paper's acceptance decision, mimicking the final stage of human peer review.

Datasets

The experiments were primarily conducted on the ICLR 2023 dataset, with additional experiments on ICLR 2022 and NeurIPS 2023 datasets. These datasets consist of thousands of papers, each reviewed by multiple human reviewers. The authors also mention using historical data from OpenReview for initializing reviewer personas and evaluating expertise levels.

Key Insights and Ideas

  • Structured Representations are Key: Representing papers as knowledge graphs and using community descriptors significantly improves the ability of LLMs to perform detailed and accurate reviews, outperforming raw text representations.
  • Persona Matters: Assigning specific reviewer personas (e.g., based on historical data) leads to more consistent and human-like evaluations compared to random persona assignment.
  • Memory Enhances Review Quality: The memory module, which stores and retrieves past reviews and community information, is crucial for generating comprehensive and faithful feedback.
  • GAR as an Assistant: The research suggests GAR can not only simulate peer review but also act as a valuable assistant to human reviewers, reducing review time and improving clarity and confidence.

Key Results

  • Human-like Performance: GAR achieved performance comparable to human reviewers in providing detailed feedback and predicting paper outcomes. In human preference rankings, GAR outperformed other LLM-based reviewers and even human reviewers in some scenarios.
  • Improved Prediction Accuracy: GAR achieved an F1 score of 0.66 in predicting paper acceptance, significantly outperforming baselines and human reviewers.
  • Reduced Review Time: GAR reduced review time by approximately 20% when assisting human reviewers.
  • Reduced Bias: GAR's acceptance rates for papers from top-tier institutions were closer to the overall submission pool baseline than real-world outcomes, suggesting a potential to reduce institutional bias.
  • Robustness Across LLMs: The GAR framework demonstrated robust performance across various foundation LLMs, with GPT-40 and GPT-40-mini showing particularly strong human-likeness.

Limitations and Future Work

  • Novelty Detection: GAR, like other LLMs, may struggle to identify genuinely groundbreaking or paradigm-shifting ideas, especially those that deviate significantly from established norms. Future work could focus on more nuanced novelty detection using graph structures and citation embeddings.
  • LLM Biases: The framework is susceptible to inheriting biases from the LLMs' training data, which could amplify existing biases in peer review. Mitigation strategies and transparency are crucial.
  • Generalizability: The current evaluation is limited to machine learning conferences. Future work aims to adapt and validate GAR in other scientific domains with different review norms.
  • Practical Deployment: Challenges remain in ensuring effective oversight, handling sensitive information, and maintaining confidentiality in real-world deployment.
  • Training Data Contamination: There's a risk that papers in evaluation datasets might have been part of the LLMs' training corpus, potentially introducing bias. Future work could explore using locally trained models on disjoint data.

Summary updated

BibTeX
@inproceedings{bougie2025generative,
  author = {Nicolas Bougie and Narimawa Watanabe},
  title = {{Generative Reviewer Agents: Scalable Simulacra of Peer Review}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure