Teaching Novice Reviewers to Judge the Paper They Were Given

NLP conferences now ask reviewers to judge very different kinds of work: new model architectures, annotated datasets, linguistic analyses, deployed software, and studies that borrow from psychology or medicine. Reviewers are new to the job in growing numbers, and many are graduate students. A common failure is simple to describe. The reviewer applies the checklist for a method paper, which asks whether the method beats the best prior score and whether the architecture is new, to a paper whose contribution is a dataset, a negative result, or a clinical analysis. This proposal builds and tests a lightweight review coach that targets that failure directly. While a novice writes a review, the coach infers what kind of contribution the paper claims to make. It then flags critical claims in the draft review that apply a criterion foreign to that contribution type, and claims that have no supporting evidence. Finally, it asks the reviewer to either reframe the criticism or justify it.

The bet is that "criterion mismatch" is a specific, detectable error. It should be distinct from generic low review quality, it should be concentrated among novices and on non-method papers, and correcting it should matter more than the generic "be more specific" advice that current writing tools give. If that holds, the community can scale reviewer mentoring that is now done by hand, without the multi-minute latency of agentic feedback systems. If it does not hold, the experiments are built to show that a static, contribution-specific review form is enough, and that result is itself a useful policy finding.

Where this sits

The gap is diagnosed by What Can We Do to Improve Peer Review in NLP?. That paper argues that forcing incomparable paper types onto one merit scale makes scores noisy and pushes overloaded reviewers toward chasing leaderboard results. It proposes review forms tailored to each contribution type, but it builds no tool and does not measure how often the mismatch actually occurs. This proposal turns its "apples-to-oranges" argument into a measurable quantity and tests it as an intervention.

The case that novices can be trained comes from A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers in Large Conferences. In that ICML 2020 study, 52 screened junior students received human scaffolding: a senior contact, example reviews, and reminders. Meta-reviewers rated their reviews higher than the main pool's (2.26 vs. 2.08 on a 1–3 scale). The authors note that the screening and mentoring were manual and would need automation to scale. They also note two confounds: novices reviewed 3 papers against 5.4 in the main pool, and they knew they were being studied. Our study design takes both confounds from them as constraints. Mentoring programs report the same bottleneck. Machine Learning for Health (ML4H) 2020 paired 55 junior reviewers with 35 mentors under tight time windows, and the NAACL 2025 Student Research Workshop reports rising demand for mentoring.

The method borrows from three argument-mining lines of work. Automatic Analysis of Substantiation in Scientific Peer Reviews tags subjective claims in reviews and links each to supporting evidence within the review. It reports claim-tagging F1 of 55.61 with RoBERTa and evidence-linkage F1 of 82.07 with SpanBERT, and its SubstanScore correlates with human quality judgments at Spearman 0.7568. That work checks only whether evidence is present, and it runs after the fact for chairs. We reuse its claim tagger and add a test of whether each claim's criterion fits the paper. The review-sentence aspect labels in DISAPERE, where BERT reaches 60.42 macro-F1 on review actions, and the proposition types in Argument Mining for Understanding Peer Reviews supply the vocabulary for which criterion a claim invokes. Paper and review pairs across ARR-22, COLING-20 and F1000 Research come from NLPeer. Its biomedical F1000 portion is the main test of whether the approach holds outside NLP venues.

Two existing tools are the closest systems, and both differ in target. Reviewriter gives inline writing prompts to student reviewers, but only for German business-model reviews, and it is evaluated on user acceptance rather than review rigor. SWIF²T critiques manuscript paragraphs with four agents, but it serves authors and takes 7 minutes 11 seconds per paragraph. Neither system models what kind of contribution a paper makes, and neither targets the reviewer's choice of criterion.

Core Claim

Novice reviews of non-method papers contain a measurable excess of negative claims whose criterion does not fit the paper's stated contribution. An example is a demand for a leaderboard comparison in a resource paper. These mismatched claims predict disagreement with senior reviewers on borderline papers. A coach that pairs a contribution-type classifier with a per-claim criterion tagger can reduce such claims during drafting. It should outperform both a static contribution-specific form and a prompted general-purpose language model, and it should respond in seconds.

Research Program

The first stage is measurement. We will annotate a sample of NLPeer reviews, stratified by venue and by contribution type. Annotators record the paper's primary contribution type: method, resource, analysis, system or software, interdisciplinary or applied study, or position paper. For each claim found by the SubstanReview tagger, they record the criterion it invokes and whether that criterion is appropriate for this contribution type.

This stage tests whether mismatch is common, whether it is concentrated in negative claims on non-method papers, and whether it predicts lower scores and wider reviewer disagreement. All three hypotheses come from Rogers and Augenstein and have not been quantified before. Reviewer seniority is not uniformly available, so seniority analyses use the controlled study described below.

The second stage builds the coach from three parts. The first part classifies contribution type from the abstract and introduction. The second reuses the claim tagger and evidence linker. The third assigns each claim a criterion label and scores fit against a contribution-type rubric drafted with senior area chairs and with editors from outside NLP, including clinical informatics for the F1000 portion. The coach shows only flagged claims, each with a one-line question asking the reviewer to reframe or justify, so it never writes the review for them. Small encoder models keep latency in seconds.

The third stage is a controlled study of novice reviewers recruited through student research workshops, screened as in Stelmakh et al. Each participant reviews exactly three papers. The papers are balanced across contribution types and drawn from NLPeer, which has real reviews and outcomes. All arms receive identical instructions about being studied, so the Hawthorne effect is held constant rather than removed.

Baselines

The naive reference point is the unassisted novice. The participant receives the standard review form and the same example reviews Stelmakh et al. provided, and nothing else. This arm anchors the quality scale. Stelmakh et al. report a meta-reviewer rating of 2.26 for supported novices, but at a different venue, with different raters and with human mentoring. We therefore re-measure rather than borrow that number.

The strongest non-computational competitor is the static contribution-specific review form that Rogers and Augenstein recommend. In this arm, the paper's true contribution type is shown and a matching checklist of criteria is given. The arm uses the same papers and the same time limit. We found no reported result on Lacuna for contribution-specific forms. Beating this arm means a lower rate of mismatched negative claims and a higher expert rating, judged by blind raters, at a pre-registered effect size.

The strongest computational competitor is a prompted large model, GPT-4 class. It is given the paper and the draft review and asked to critique the review. It receives the same paper text, a prompt-tuning budget matched to the development effort spent on the coach's rubric, and no gold contribution label. The nearest reported evidence comes from SWIF²T: vanilla GPT-4 scored a human helpfulness dominance score of 92.50, against 171.75 for SWIF²T and 121.25 for human reviews, on manuscript feedback. That is a different task, so it motivates but does not settle this comparison. A reduced version of SWIF²T adapted to review critique would be the agentic upper reference if latency allows, and its latency will be reported alongside quality.

The key ablation is the coach with type-awareness removed, which flags only unsubstantiated claims as SubstanScore does. If this arm does as well as the full coach, criterion mismatch is not the active ingredient. A second ablation feeds the coach the gold contribution type, which separates classifier error from rubric value.

Why not just…?

Why not just use contribution-specific review forms? This is the most serious threat, because it costs nothing to deploy. A form tells reviewers which criteria apply. It does not catch a reviewer who ticks the right boxes and then writes "no comparison to the best prior system" anyway. Whether that residual error matters is exactly what the form arm against the coach arm measures. If the two tie on mismatch rate and expert rating, we will report that forms suffice. The coach's remaining value would then be limited to the measurement tool from stage one.

Why not just ask a general-purpose model to critique the review? Prompted models give generic advice and hallucinate in the closest settings. The SWIF²T authors report weak human preference for vanilla GPT-4, and the Reviewriter authors report hallucinated, irrelevant instructions. Neither, however, tested review critique with an explicit contribution-type rubric in the prompt. Our prompted baseline will include that rubric, which makes it a fair and strong opponent. Its latency and cost per review will be reported as well.

Why not just score substantiation, which already correlates with quality? SubstanScore's correlation of 0.7568 shows that evidence-backed claims track quality. A well-substantiated claim can still apply the wrong criterion, though. A reviewer can carefully document that a dataset paper "lacks a new model". The substantiation-only ablation tests whether the extra type-aware signal adds anything.

Proposed Experiments

The decisive offline check comes first. On held-out NLPeer reviews, we test whether mismatched negative claims are more frequent on non-method papers, and whether they predict score divergence from the meta-review beyond what review length and SubstanScore explain. If mismatch is rare or uninformative, the coach is not built.

Detector quality is then reported against our annotations. This covers contribution-type accuracy, criterion-label macro-F1 compared against DISAPERE's 60.42 as a rough difficulty reference, and claim-tagging F1 compared against SubstanReview's 55.61. Accuracy is reported separately on F1000 to measure transfer across disciplines.

The controlled study reports four outcomes per arm: mismatched-claim rate, blind expert rating on the Stelmakh 1–3 scale, SubstanScore, and agreement with real senior scores on borderline papers. Effects are broken down by contribution type. The prediction is that gains concentrate on non-method papers and are near zero on method papers. A gain that is uniform across types would point to generic attention effects rather than the proposed mechanism.