Abstract
The rapid growth of academic publishing has strained the peer review system, prompting interest in Large Language Model (LLM) assistance. However, existing benchmarks for evaluating AI-generated reviews often lack domain diversity, ignore multimodal content such as figures and tables, and rely on superficial metrics. We introduce MMReview, a comprehensive benchmark spanning 17 research fields across four major disciplines. MMReview evaluates models on 13 distinct tasks, including reasoning, decision-making, and adversarial robustness. Our findings highlight the importance of multimodal integration and Chain-of-Thought reasoning in achieving human-aligned review quality.
Introduction
The academic community faces a significant bottleneck due to the increasing volume of submissions, leading to reviewer fatigue and delayed feedback. While LLMs offer potential for automation, current evaluation frameworks are insufficient. They typically focus on text-only AI/CS papers and simple "Accept/Reject" classification, failing to capture the nuanced reasoning and multimodal interpretation required for high-quality peer review. MMReview addresses these gaps by providing a standardized, multidisciplinary, and multimodal evaluation suite.
Methodology and Pipeline
The construction of MMReview follows a rigorous three-stage process: Data Collection, Data Processing, and Task Construction.
Data Collection and Processing
Starting from a seed dataset of 51,881 papers from platforms like OpenReview and Nature Communications, we employed a collaborative pipeline using GPT-4o and Qwen3. This process filtered the pool down to 240 high-quality papers that demonstrated significant information gain when comparing full-text content to abstracts. The final dataset covers four disciplines: Artificial Intelligence (AI), Natural Sciences (NS), Engineering Sciences (ES), and Social Sciences (SS).
Task Design
We designed 13 tasks categorized into four themes:
- Step-based: Summarization, strength/weakness evaluation, and scoring (soundness/presentation).
- Outcome-based: Final decision-making (Direct vs. Chain-of-Thought).
- Preference-based: Pairwise ranking of papers.
- Attack-based: Robustness against adversarial "fake" content and prompt injection.
Experimental Results
We benchmarked 21 models, including 18 open-source and 3 closed-source variants, across three input modalities: text-only, multimodal (text + extracted figures/tables), and PDF-as-image.
Key Findings
- Multimodal Advantage: Integrating visual data (figures/tables) significantly enhances a model's ability to resist prompt injection attacks compared to text-only processing.
- Reasoning over Direct Output: Models utilizing Chain-of-Thought (CoT) reasoning consistently outperformed those providing direct scores, showing higher alignment with human editorial decisions.
- Model Performance: GPT-4o and Claude-3.5-Sonnet emerged as top performers. Interestingly, mid-sized models showed competitive performance in technical grading tasks like "Soundness" and "Presentation."
- Biases: We observed a persistent "length-induced bias," where models tended to favor longer papers regardless of quality, highlighting a critical area for future refinement in AI-based review systems.
Conclusion
MMReview provides a robust framework for evaluating the next generation of AI peer review assistants. By emphasizing multidisciplinary breadth, multimodal depth, and adversarial robustness, it sets a new standard for measuring the utility and safety of LLMs in academic workflows. Future work will aim to further balance domain distributions and mitigate inherent model biases.
Summary updated