Automated scholarly paper review: Concepts, technologies, and challenges

Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, Xiaodong ShiarXiv 2021 · DOI · arXiv

TL;DR

This paper introduces the concept of Automated Scholarly Paper Review (ASPR), a proposed end-to-end pipeline where AI independently handles everything from manuscript parsing to final decision-making. By moving beyond mere "AI assistance," ASPR aims to solve the chronic efficiency and reproducibility crises currently plaguing academic publishing.

What Problem Does This Paper Address?

The traditional peer review system is under immense strain. The authors highlight several critical "pain points":

  • Poor Efficiency: The average review process takes 17 weeks, a "profligate of academic time" that can render research outdated before it's even published.
  • Low Reproducibility: Human judgment is notoriously inconsistent. The paper cites a famous study where 8 out of 9 previously published papers were rejected when resubmitted to the same journals under different names.
  • Volume Overload: With millions of papers published annually and the rise of preprint servers like arXiv, the pool of qualified human reviewers is simply overwhelmed.

The authors argue that while AI tools currently assist editors (e.g., finding reviewers or checking for plagiarism), we need a transition to a fully automated system to ensure a fairer, faster, and more scientific era of publishing.

Methods

The paper enunciates a formal pipeline for ASPR, contrasting it with the traditional human-centric model.

The ASPR pipeline replaces human intervention with an integrated AI process.

The proposed ASPR pipeline (shown below) consists of four main stages:

  1. Parsing and Representation: Converting PDFs or LaTeX files into structured data. This uses tools like GROBID for text and CNNs/ResNets for image extraction.
  2. Screening: Initial checks for formatting, plagiarism, and "scope fit."
  3. Main Review: Evaluating the "four pillars": Originality (novelty), Soundness (methodological rigor), Clarity (language and structure), and Significance (impact).
  4. Review Report Generation: Using NLP models (like BART or Transformers) to generate a summary, detailed comments, scores, and a final decision.
The ASPR pipeline replaces human intervention with an integrated AI process.

Datasets

The authors survey a wide range of datasets essential for training ASPR modules, including:

  • PeerRead: 14.7k papers with textual reviews and scores.
  • S2ORC: A massive corpus of 8.1M papers with full-text and citations for citation analysis.
  • ASAP-Review: 9k papers with aspect-annotated review comments.
  • TDMSci: A specialized corpus for extracting tasks, datasets, and metrics to enable SOTA (State-of-the-Art) comparisons.

Key Insights and Ideas

  • From Assistant to Independent Actor: The core shift is defining computers as the "review subject" rather than just a tool. In ASPR, the AI fulfills the roles of both editor and reviewer simultaneously.
  • Originality as Recombination: The paper suggests that AI can quantify originality by analyzing "novel combinations" of references—identifying when a paper links previously unrelated ideas in an unprecedented way.
  • The "PDF Nightmare": A significant technical hurdle is that PDFs lack structural tags. For ASPR to work, parsers must be perfect; even a single punctuation error in a mathematical formula can invalidate a review.
  • The Two-Phase Evolution: The authors envision a Coexistence Phase (where we are now), where AI acts as a sophisticated assistant, eventually leading to an Independent Phase (estimated by the 2050s) where AI handles the workload autonomously.

Key Results

While this is primarily a conceptual and survey paper, it highlights the impressive performance of existing sub-modules:

  • Acceptance Prediction: Some models (like Random Forests) have achieved up to 81% accuracy in predicting whether a paper will be accepted based on its content.
  • Plagiarism Detection: This is the most mature area, with systems achieving over 89% accuracy in detecting machine-generated "nonsense" papers.
  • Assisting Tools: The paper lists several existing tools that already handle parts of the pipeline, such as Statcheck for validating p-values and ReviewAdvisor for generating multi-aspect comments.
A summary of existing AI tools currently assisting the peer review process.

Limitations and Future Work

The authors are candid about the "mountain" still left to climb:

  • Data Imbalance: Most available datasets consist of accepted papers. AI trained on this data will have an "in-built bias" and may struggle to identify what makes a paper "bad" or truly "novel" (since novel papers are outliers).
  • Flawed Logical Reasoning: Current AI is great at pattern matching but struggles with "deep logical reasoning"—checking if a conclusion actually follows from the results.
  • Ethics and Authority: If an AI rejects a paper, who is responsible? The developers? The journal? The paper calls for independent third-party evaluations of ASPR algorithms before they are given "authority."
  • Adversarial Attacks: There is a risk that authors might "game the system" by writing papers specifically designed to trigger high scores from an AI reviewer, potentially degrading the quality of scientific writing.

Summary updated

Abstract

Peer review is a widely accepted mechanism for research evaluation, playing a pivotal role in academic publishing. However, criticisms have long been leveled at this mechanism, mostly because of its poor efficiency and low reproducibility. Recent years have seen the application of artificial intelligence (AI) in assisting the peer review process. Nonetheless, with the involvement of humans, such limitations remain inevitable. In this paper, we propose the concept and pipeline of automated scholarly paper review (ASPR) and review the relevant literature and technologies of achieving a full-scale computerized review process. On the basis of the review and discussion, we conclude that there is already corresponding research and preliminary implementation at each stage of ASPR. We further look into the challenges in ASPR with the existing technologies. The major difficulties lie in inadequate data, imperfect document parsing and representation, defective human$\unicode{x2013}$computer interaction, and flawed deep logical reasoning. Moreover, we point out the future directions and discuss the possible moral and ethical issues of ASPR. In the foreseeable future, ASPR and peer review will coexist in a reinforcing manner before ASPR is able to fully undertake the reviewing workload from humans.
BibTeX
@misc{lin2021automated,
  author = {Jialiang Lin and Jiaxin Song and Zhangping Zhou and Yidong Chen and Xiaodong Shi},
  title = {{Automated scholarly paper review: Concepts, technologies, and challenges}},
  year = {2021},
  eprint = {2111.07533},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.07533},
}
Figure