MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation

Xian Gao, Jiacheng Ruan, Zongyun Zhang, Jingsheng Gao, Ting Liu, Yuzhuo FuarXiv 2025 · DOI · arXiv

Abstract

The rapid growth of academic publishing has strained the peer review system, prompting interest in Large Language Model (LLM) assistance. However, existing benchmarks for evaluating AI-generated reviews often lack domain diversity, ignore multimodal content such as figures and tables, and rely on superficial metrics. We introduce MMReview, a comprehensive benchmark spanning 17 research fields across four major disciplines. MMReview evaluates models on 13 distinct tasks, including reasoning, decision-making, and adversarial robustness. Our findings highlight the importance of multimodal integration and Chain-of-Thought reasoning in achieving human-aligned review quality.

Introduction

The academic community faces a significant bottleneck due to the increasing volume of submissions, leading to reviewer fatigue and delayed feedback. While LLMs offer potential for automation, current evaluation frameworks are insufficient. They typically focus on text-only AI/CS papers and simple "Accept/Reject" classification, failing to capture the nuanced reasoning and multimodal interpretation required for high-quality peer review. MMReview addresses these gaps by providing a standardized, multidisciplinary, and multimodal evaluation suite.

Methodology and Pipeline

The construction of MMReview follows a rigorous three-stage process: Data Collection, Data Processing, and Task Construction.

Overview of the MMReview pipeline, including data collection from major venues, multi-stage filtering, and task generation.

Data Collection and Processing

Starting from a seed dataset of 51,881 papers from platforms like OpenReview and Nature Communications, we employed a collaborative pipeline using GPT-4o and Qwen3. This process filtered the pool down to 240 high-quality papers that demonstrated significant information gain when comparing full-text content to abstracts. The final dataset covers four disciplines: Artificial Intelligence (AI), Natural Sciences (NS), Engineering Sciences (ES), and Social Sciences (SS).

Task Design

We designed 13 tasks categorized into four themes:

  1. Step-based: Summarization, strength/weakness evaluation, and scoring (soundness/presentation).
  2. Outcome-based: Final decision-making (Direct vs. Chain-of-Thought).
  3. Preference-based: Pairwise ranking of papers.
  4. Attack-based: Robustness against adversarial "fake" content and prompt injection.

Experimental Results

We benchmarked 21 models, including 18 open-source and 3 closed-source variants, across three input modalities: text-only, multimodal (text + extracted figures/tables), and PDF-as-image.

Comprehensive performance of 21 LLMs across Step, Outcome, Preference, and Attack tasks.

Key Findings

  • Multimodal Advantage: Integrating visual data (figures/tables) significantly enhances a model's ability to resist prompt injection attacks compared to text-only processing.
  • Reasoning over Direct Output: Models utilizing Chain-of-Thought (CoT) reasoning consistently outperformed those providing direct scores, showing higher alignment with human editorial decisions.
  • Model Performance: GPT-4o and Claude-3.5-Sonnet emerged as top performers. Interestingly, mid-sized models showed competitive performance in technical grading tasks like "Soundness" and "Presentation."
  • Biases: We observed a persistent "length-induced bias," where models tended to favor longer papers regardless of quality, highlighting a critical area for future refinement in AI-based review systems.

Conclusion

MMReview provides a robust framework for evaluating the next generation of AI peer review assistants. By emphasizing multidisciplinary breadth, multimodal depth, and adversarial robustness, it sets a new standard for measuring the utility and safety of LLMs in academic workflows. Future work will aim to further balance domain distributions and mitigate inherent model biases.

Summary updated

Abstract

With the rapid growth of academic publications, peer review has become an essential yet time-consuming responsibility within the research community. Large Language Models (LLMs) have increasingly been adopted to assist in the generation of review comments; however, current LLM-based review tasks lack a unified evaluation benchmark to rigorously assess the models' ability to produce comprehensive, accurate, and human-aligned assessments, particularly in scenarios involving multimodal content such as figures and tables. To address this gap, we propose \textbf{MMReview}, a comprehensive benchmark that spans multiple disciplines and modalities. MMReview includes multimodal content and expert-written review comments for 240 papers across 17 research domains within four major academic disciplines: Artificial Intelligence, Natural Sciences, Engineering Sciences, and Social Sciences. We design a total of 13 tasks grouped into four core categories, aimed at evaluating the performance of LLMs and Multimodal LLMs (MLLMs) in step-wise review generation, outcome formulation, alignment with human preferences, and robustness to adversarial input manipulation. Extensive experiments conducted on 16 open-source models and 5 advanced closed-source models demonstrate the thoroughness of the benchmark. We envision MMReview as a critical step toward establishing a standardized foundation for the development of automated peer review systems.
BibTeX
@misc{gao2025mmreview,
  author = {Xian Gao and Jiacheng Ruan and Zongyun Zhang and Jingsheng Gao and Ting Liu and Yuzhuo Fu},
  title = {{MMReview: A Multidisciplinary and Multimodal Benchmark for LLM-Based Peer Review Automation}},
  year = {2025},
  eprint = {2508.14146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.14146},
}
Figure