Can We Automate Scientific Reviewing?

Weizhe Yuan, Pengfei Liu, Graham Neubig

TL;DR

Researchers at Carnegie Mellon University have developed ReviewAdvisor, an NLP system designed to generate "first-pass" peer reviews for scientific papers. While the system excels at summarizing core ideas and covering a broad range of evaluation aspects, it struggles with factual accuracy and high-level reasoning, suggesting that AI is currently best suited as a human-assistant rather than a replacement for expert reviewers.

What Problem Does This Paper Address?

The scientific community is facing a scaling crisis: the number of published papers is growing exponentially, but the pool of expert reviewers is not. This leads to reviewer burnout, delayed feedback, and potentially lower-quality reviews.

The authors ask a provocative question: "Can we automate scientific reviewing?" To answer this, they move beyond simple summarization. A good review isn't just a recap; it requires a decisive stance (accept/reject), comprehensive coverage of different aspects (clarity, novelty, etc.), and justified, factual evidence. The challenge lies in teaching a model to read a long, complex document and provide a "critique" that is both helpful and fair.

Methods

The authors treat review generation as a task of aspect-based scientific paper summarization. Because scientific papers are far too long for standard Transformer models (like BART) to process in one go, they employ a two-stage "extract-then-generate" paradigm:

  1. Extraction: The system first identifies salient sentences from the paper using heuristics and a cross-entropy method to ensure diverse content coverage (e.g., looking for keywords like "propose" or "experimental results").
  2. Generation: These snippets are fed into a fine-tuned BART model.
  3. Aspect-Awareness: To ensure the model doesn't just summarize but actually critiques, they designed an auxiliary task where the model predicts specific "aspects" (like Originality, Clarity, or Replicability) for each sentence it generates.

Datasets

The researchers introduced a new dataset called ASAP-Review (ASpect-enhAnced Peer Review):

  • Source: Over 20,000 reviews crawled from ICLR (2017–2020) and NeurIPS (2016–2019).
  • Annotations: They manually annotated 1,000 reviews with eight specific aspects (Summary, Motivation, Originality, Soundness, Substance, Replicability, Clarity, and Comparison) to train an "Aspect Tagger."
  • Metadata: The dataset includes final decisions (accept/reject) and author/institutional information to facilitate bias analysis.

Key Insights and Ideas

  • Broader but Shallower: Interestingly, the AI-generated reviews often touched upon more aspects of a paper than human reviews. While a human might forget to mention "Replicability," the model (trained to be comprehensive) usually includes it.
  • The Hallucination Hurdle: The biggest "red flag" is factuality. The system frequently generates non-factual statements—essentially "hallucinating" flaws or strengths that aren't actually in the paper. In high-stakes peer review, this is a deal-breaker.
  • The "Reviewer 2" Bias: The study found that both humans and AI exhibit biases. For example, human reviewers tend to give higher clarity scores to native English speakers. While the AI narrowed this "clarity gap," it was significantly harsher on the "originality" of papers written by non-native speakers.
  • Style over Substance: The model learned to imitate the style of a review perfectly (e.g., using common phrases like "The paper is well-written and easy to follow"), but it lacks the high-level understanding to consistently distinguish a "top-tier" paper from a "reject."

Key Results

The authors evaluated the system using a mix of automated and human metrics:

  • Summary Accuracy (SACC): The system performed quite well here, with some configurations achieving over 80% accuracy in summarizing the paper's core contributions.
  • Recommendation Accuracy (RACC): The system struggled to match the final "Accept/Reject" decisions of the actual conference, performing significantly worse than humans.
  • Constructiveness (ACON): Human evaluators found that system-generated reviews were much less constructive than human ones, particularly in providing specific evidence for negative critiques.
  • Informativeness (INFO): Surprisingly, the system achieved high scores in providing evidence for negative comments, though the "validity" of that evidence was often questionable.

Limitations and Future Work

The authors are transparent about the system's current state: it is not ready to replace humans.

Limitations: * Factuality: The model cannot yet be trusted to be factually correct about a paper's technical details. * Long-Document Modeling: Even with the two-stage approach, much of the paper's nuance is lost during the extraction phase. * High-Level Reasoning: The model mimics the language of a critique without truly "understanding" the scientific merit.

Future Directions: * External Knowledge: Integrating citation graphs or knowledge bases to help the model understand how a paper fits into the broader field. * Improved Parsers: Better tools to extract information from tables and figures, which are currently ignored by the system. * Human-in-the-loop: Developing the system as a "Reviewer Assistant" that provides a template or a first draft for a human expert to refine.

Summary updated

Abstract

The rapid development of science and technology has been accompanied by an exponential growth in peer-reviewed scientific publications. At the same time, the review of each paper is a laborious process that must be carried out by subject matter experts. Thus, providing high-quality reviews of this growing number of papers is a significant challenge. In this work, we ask the question "can we automate scientific reviewing?", discussing the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. Arguably the most difficult part of this is defining what a "good" review is in the first place, so we first discuss possible evaluation measures for such reviews. We then collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews, but the generated text can suffer from lower constructiveness for all aspects except the explanation of the core ideas of the papers, which are largely factually correct. We finally summarize eight challenges in the pursuit of a good review generation system together with potential solutions, which, hopefully, will inspire more future research on this subject. We make all code, and the dataset publicly available: https://github.com/neulab/ReviewAdvisor, as well as a ReviewAdvisor system: http://review.nlpedia.ai/.
BibTeX
@article{yuan2022can,
  author = {Weizhe Yuan and Pengfei Liu and Graham Neubig},
  title = {{Can We Automate Scientific Reviewing?}},
  journal = {J. Artif. Intell. Res.},
  year = {2022},
  doi = {10.1613/jair.1.12862},
}
Figure