ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments

Kihyun Na, Injung KimarXiv 2026 · arXiv

TL;DR

ReviewMap is a new framework that distinguishes between two fundamental evaluative acts in peer review: Assessment (checking if a paper’s claims are technically sound) and Calibration (judging if those claims actually matter for the venue). By testing this on thousands of ICLR 2026 review comments, the authors define a "measured automation boundary," showing that while AI can reliably categorize these acts, the nuanced justifications behind them still require human oversight.

What Problem Does This Paper Address?

The peer-review process is currently under immense strain due to the sheer volume of submissions in fields like Machine Learning. While there is a push to use Large Language Models (LLMs) to assist or even automate reviews, doing so blindly is dangerous.

The authors argue that current NLP tools for peer review fail to distinguish between what a paper does and why it matters. For example, saying "the baseline is missing" is a technical assessment, whereas saying "the contribution is too incremental for ICLR" is a value judgment (calibration). Without separating these, it’s hard to build responsible AI tools that help humans inspect the grounds of a reviewer's judgment rather than just replacing the human entirely.

Methods

The authors introduce ReviewMap, a two-level classification framework:

  1. Level 1 (Evaluative Act): Categorizes a comment as either Assessment (ASM) (evaluating claims, methods, or standards) or Calibration (CAL) (judging significance, novelty, or venue fit).
  2. Level 2 (Act-conditional Attributes): Adds detail. For ASM, it checks if the comment is claim-relevant (specific to the paper) or standard-relevant (field-wide conventions). For CAL, it checks if the judgment is grounded in explicit evidence.

To ensure reliability, they used a multi-model consensus approach. They ran three open-weight LLMs (Qwen 3.5, Gemma 4, and Nemotron 3-Super) three times each, using a two-stage voting pipeline to reach a final coding. They validated this pipeline against a "gold standard" of five human experts and a panel of frontier models (like GPT-4 and Claude).

Figure 1: The ReviewMap ecosystem: reviewers check assessments and ground value judgments; authors choose responses; area chairs inspect panel judgments and disagreements. These proposed affordances require users to inspect source comments and confirm attribute- based suggestions (§6).

Datasets

The study utilized a newly compiled corpus of 500 ICLR 2026 submissions sampled from OpenReview. This resulted in 17,654 individual review comments across nearly 2,000 reviewer-paper instances. The authors also created a development set of 12 papers (470 comments) for human validation.

Key Insights and Ideas

  • The Automation Boundary: LLMs are remarkably good at identifying the type of evaluative act (ASM vs. CAL), achieving substantial agreement with humans ($\kappa \geq 0.75$). However, they struggle significantly more with the attributes (Level 2), such as determining if a value judgment is properly "grounded." This defines the boundary: AI can structure the review for inspection, but humans must verify the reasoning.
  • Calibration Scarcity: Only 17.4% of review comments are explicit "calibrations." Surprisingly, 17% of all reviews contain no explicit calibration at all, meaning the reviewer's value judgment remains entirely implicit.
  • Section Asymmetry: Reviewers are much more likely to explain why a paper is good than why it is bad. The share of calibration comments is nearly 3x higher in "Strengths" sections (34.5%) than in "Weaknesses" (12.2%). Most "Weaknesses" are actually technical assessments (85.5% ASM) rather than explicit judgments on the paper's significance.

Key Results

The multi-model consensus proved highly stable for the primary task of act classification. As shown in the agreement plot below, while the "Act" classification remains high, agreement drops sharply for the more subjective "Grounding" attribute.

Figure 3: Agreement across acts and attributes. Intra- model points use Fleiss’ κ over three runs; inter-model points use Cohen’s κ over pairs of model labels. Mean inter-model agreement decreases across act, P1, P2, and grounding. Attribute samples are conditional on match- ing applicable acts and binary labels; denominators and observed agreement appear in Appendix D.

Quantitatively:

  • Act Distribution: 80.5% Assessment, 17.4% Calibration, 2.2% Flagged/Uncertain.
  • Reliability: The open-weight LLM panel achieved 90% raw agreement with the human expert mode on Level 1 acts.
  • Zero-CAL Reviews: 81.4% of reviewers issue two or fewer calibration comments per paper.

Limitations and Future Work

The authors acknowledge several limitations:

  • Venue Specificity: The study is limited to ICLR 2026 and English-language reviews. Different venues may have different norms for "calibration."
  • Implicit Reasoning: The framework only maps explicit text. It cannot capture a reviewer's unstated reasoning or the ultimate validity of their recommendation.
  • LLM Bias: There is a slight tendency for LLMs to over-predict Calibration (CAL) compared to human annotators.

Future work includes validating the "Author-Side Strategy Scaffold"—a tool designed to help authors choose rebuttal strategies (like DEFEND, CONCEDE, or CLARIFY) based on ReviewMap codings—against real-world rebuttal corpora.

Summary updated

Abstract

Peer-review comments check what a paper does and judge why it matters, but no existing scheme distinguishes these two evaluative acts. We argue that making these acts explicit and measuring how reliably LLMs identify them is a prerequisite for responsible LLM-assisted review analysis: humans need to inspect review judgments and their grounds. This defines a more honest mission for NLP in peer review than full automation. We present ReviewMap, a framework that classifies each comment as assessment of claims and scholarly standards or calibration of contribution value and venue fit, with finer-grained attributes applied conditionally by act type. Applied to 500 ICLR~2026 papers through multi-model LLM consensus, the act layer proves reliably automatable under the operational guideline (inter-model $κ\geq 0.75$), supported by five-human validation on 470 development comments, while the attribute layer requires human oversight, defining a measured automation boundary. At the corpus level, 17% of reviewer-paper instances contain no panel-coded calibration, and the panel-coded calibration share is nearly $3\times$ as high in Strengths as in Weaknesses. This boundary informs three affordances: a review guideline that operationalizes the assessment-then-calibration workflow for reviewers, a per-comment rebuttal strategy scaffold for authors, and a discussion-phase dashboard for area chairs.
BibTeX
@misc{na2026reviewmap,
  author = {Kihyun Na and Injung Kim},
  title = {{ReviewMap: Mapping Evaluative Acts and Automation Boundaries in Peer-Review Comments}},
  year = {2026},
  eprint = {2611.07489},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07489},
}
Figure