TL;DR
ReviewMap is a new framework that distinguishes between two fundamental evaluative acts in peer review: Assessment (checking if a paper’s claims are technically sound) and Calibration (judging if those claims actually matter for the venue). By testing this on thousands of ICLR 2026 review comments, the authors define a "measured automation boundary," showing that while AI can reliably categorize these acts, the nuanced justifications behind them still require human oversight.
What Problem Does This Paper Address?
The peer-review process is currently under immense strain due to the sheer volume of submissions in fields like Machine Learning. While there is a push to use Large Language Models (LLMs) to assist or even automate reviews, doing so blindly is dangerous.
The authors argue that current NLP tools for peer review fail to distinguish between what a paper does and why it matters. For example, saying "the baseline is missing" is a technical assessment, whereas saying "the contribution is too incremental for ICLR" is a value judgment (calibration). Without separating these, it’s hard to build responsible AI tools that help humans inspect the grounds of a reviewer's judgment rather than just replacing the human entirely.
Methods
The authors introduce ReviewMap, a two-level classification framework:
- Level 1 (Evaluative Act): Categorizes a comment as either Assessment (ASM) (evaluating claims, methods, or standards) or Calibration (CAL) (judging significance, novelty, or venue fit).
- Level 2 (Act-conditional Attributes): Adds detail. For ASM, it checks if the comment is claim-relevant (specific to the paper) or standard-relevant (field-wide conventions). For CAL, it checks if the judgment is grounded in explicit evidence.
To ensure reliability, they used a multi-model consensus approach. They ran three open-weight LLMs (Qwen 3.5, Gemma 4, and Nemotron 3-Super) three times each, using a two-stage voting pipeline to reach a final coding. They validated this pipeline against a "gold standard" of five human experts and a panel of frontier models (like GPT-4 and Claude).
Datasets
The study utilized a newly compiled corpus of 500 ICLR 2026 submissions sampled from OpenReview. This resulted in 17,654 individual review comments across nearly 2,000 reviewer-paper instances. The authors also created a development set of 12 papers (470 comments) for human validation.
Key Insights and Ideas
- The Automation Boundary: LLMs are remarkably good at identifying the type of evaluative act (ASM vs. CAL), achieving substantial agreement with humans ($\kappa \geq 0.75$). However, they struggle significantly more with the attributes (Level 2), such as determining if a value judgment is properly "grounded." This defines the boundary: AI can structure the review for inspection, but humans must verify the reasoning.
- Calibration Scarcity: Only 17.4% of review comments are explicit "calibrations." Surprisingly, 17% of all reviews contain no explicit calibration at all, meaning the reviewer's value judgment remains entirely implicit.
- Section Asymmetry: Reviewers are much more likely to explain why a paper is good than why it is bad. The share of calibration comments is nearly 3x higher in "Strengths" sections (34.5%) than in "Weaknesses" (12.2%). Most "Weaknesses" are actually technical assessments (85.5% ASM) rather than explicit judgments on the paper's significance.
Key Results
The multi-model consensus proved highly stable for the primary task of act classification. As shown in the agreement plot below, while the "Act" classification remains high, agreement drops sharply for the more subjective "Grounding" attribute.
Quantitatively:
- Act Distribution: 80.5% Assessment, 17.4% Calibration, 2.2% Flagged/Uncertain.
- Reliability: The open-weight LLM panel achieved 90% raw agreement with the human expert mode on Level 1 acts.
- Zero-CAL Reviews: 81.4% of reviewers issue two or fewer calibration comments per paper.
Limitations and Future Work
The authors acknowledge several limitations:
- Venue Specificity: The study is limited to ICLR 2026 and English-language reviews. Different venues may have different norms for "calibration."
- Implicit Reasoning: The framework only maps explicit text. It cannot capture a reviewer's unstated reasoning or the ultimate validity of their recommendation.
- LLM Bias: There is a slight tendency for LLMs to over-predict Calibration (CAL) compared to human annotators.
Future work includes validating the "Author-Side Strategy Scaffold"—a tool designed to help authors choose rebuttal strategies (like DEFEND, CONCEDE, or CLARIFY) based on ReviewMap codings—against real-world rebuttal corpora.
Summary updated