Calibration Anchors: Designing Novice Review Loads So Their Scores Can Be Trusted

Conferences can no longer find enough senior reviewers, and a controlled study at ICML 2020 showed that mentored junior graduate students can write better reviews than the regular pool. The trouble is that a novice's numeric scores are the weakest link. Novices review only a few papers. Their personal leniency and their sense of scale are unknown. A score of 6 from a novice may mean 4 or 8 from anyone else, and no algorithm can untangle this unless the novice's papers overlap in the right way with papers whose quality is already well estimated.

This is an identifiability problem in the sense used by the matrix-factorization literature: are the hidden quantities uniquely determined by the data, or can many settings explain it equally well? This note proposes treating it as a design problem rather than a problem to fix after the fact. We would give each novice a small set of anchor papers that senior reviewers also score. The anchors are chosen so that the novice's personal scoring map, covering leniency, scale, and cross-aspect "halo" effects, is uniquely recoverable and well-conditioned. After submitting, the novice sees the senior reviews of those same anchors, which gives them formative feedback.

The core claim has two parts. First, how many anchors a novice needs is set by a rank condition that generalizes the known "double connectivity" result to multi-aspect scores. Second, which anchors matter as much as how many. Anchors whose estimated quality profiles are spread toward the extremes give far more accurate calibration than randomly chosen or most-similar papers. This mirrors how spread-out data pins down a factorization in simplex-structured matrix factorization.

What exists and what is missing

Least Square Calibration for Peer Review supplies the starting point. It models each score as a reviewer-specific function of a scalar paper quality, $f_i(q_j)=a_iq_j+b_i$. It proves that qualities and reviewer parameters are recoverable exactly when the reviewer graph is "doubly connected": one shared paper aligns leniency, and two align scale. It analyzes a given graph and does not design one. It also handles only scalar quality and is silent on conditioning, meaning how badly estimates degrade when the shared papers are nearly equal in quality.

PeerReview4All designs assignments, but it optimizes worst-case expertise per paper and treats calibration as out of scope. The ICML 2020 novice-reviewer experiment motivates the regime. It found that novices produced meta-review ratings of 2.26 vs 2.08 on a 1–3 scale, that 88% of the selected novices would have failed publication filters, and that they reviewed 3 papers versus 5.4 for the main pool. It also documents that selection and mentoring were manual and hard to scale.

The geometric intuition comes from the identifiability literature. Nonnegative Matrix Factorization for Signal and Data Analytics and Robustness of Minimum-Volume NMF under an Expanded Sufficiently Scattered Condition show that hidden factors are pinned down when the data press against the boundary of the latent region. Error bounds tighten as the data approach the extreme points. ReviewArena documents score compression toward the 4–7 range. That is exactly the regime where data sit in the interior and identification degrades.

We borrow the intuition, not the theorems. The proposal is not a new NMF theorem. It is an assignment-design method justified by a rank condition and a conditioning bound.

Core Claim

Let paper $j$ have a latent quality vector $q_j\in\mathbb{R}^r$ over $r$ review aspects (for example soundness, presentation, contribution). Let novice $i$ report aspect scores

$$y_{ij}=A_iq_j+b_i+\varepsilon_{ij},$$

where $A_i$ is an $r\times r$ matrix (its diagonal is per-aspect scale, its off-diagonal is halo leakage between aspects), $b_i$ is a leniency offset, and $\varepsilon_{ij}$ is noise of size $\sigma$.

The model has a gauge freedom: any affine transform of all $q_j$ can be absorbed into every $A_i,b_i$. We remove it by fixing the senior pool's average map to the identity. Given anchor qualities estimated from senior reviews, $(A_i,b_i)$ is identifiable exactly when the $(r+1)\times k$ matrix $M_i=[q_{j_1},\dots,q_{j_k};\,1,\dots,1]$ over the novice's $k$ anchors has full row rank.

This requires $k\ge r+1$ affinely independent anchors. With $r=1$ it reduces to the double-connectivity requirement of two shared papers. The estimation error of ordinary least squares then scales as $\sigma/\sigma_{\min}(M_i)$, the standard regression bound. So score compression, which shrinks the spread of anchor qualities, directly inflates error.

The design rule follows: choose anchors to maximize $\sigma_{\min}(M_i)$. We would do this greedily with a successive-projection procedure that repeatedly picks the paper farthest from the span of those already chosen, the same idea as the Successive Projection Algorithm used for pure-pixel extraction. We would also add a check that the senior-side estimates of the anchors are themselves well determined.

Research Program

The first component is analysis. We would prove the rank condition and the conditioning bound. We would also extend both to the case where anchor qualities are estimated with error, an errors-in-variables term that adds to $\sigma$. We would then ask when diagonal $A_i$ (no halo), which needs only two anchors per aspect, is an adequate restriction.

The second component is the design algorithm. It solves a joint problem: give every novice $r+1$ or more well-spread anchors, keep each paper's anchor burden on seniors bounded, and preserve the expertise floor of the main assignment. The anchor step would be layered on top of a PeerReview4All assignment, so that expertise fairness is not traded away.

The third component is the mentorship loop. After a novice submits, the system shows them senior reviews for their anchors. It also reports their estimated $A_i,b_i$ with uncertainty, for example "you score presentation about 1.5 points harsher than seniors and let clarity bleed into soundness." The uncertainty estimate also tells organizers which novices are already calibrated. This replaces part of the manual auxiliary-review selection used at ICML 2020.

Baselines

The naive reference point is the plain average of raw scores per paper. Its counterpart is per-reviewer z-score normalization. Both would be run on exactly the same assignment graphs as our method. On the semi-synthetic ICLR 2019 data, Least Square Calibration reports the Average baseline at 39.2% precision for recovering the top 10% of papers at noise $\sigma=0.5$. No z-score number is reported in the file, so we would compute one ourselves.

The strongest existing method is Least Square Calibration (linear) applied post hoc to a standard assignment. On the same semi-synthetic setting it reports 80.1% precision at $\sigma=0.5$. On the University of Hamburg peer-grading data it reports 82.2% top-paper precision and an average gap of 0.38 to TA grades. We would rerun it with the authors' hypothesis class on the same graphs. Beating it means higher top-$k$ precision, or lower gap to ground truth, at an equal total review budget. We would also apply our multi-aspect calibration model on LSC's original graphs, which isolates design effects from model effects.

The Bayesian calibration baseline reached 71.5% at $\sigma=0.5$ in the same paper. We would rerun it with hierarchical priors that shrink novices toward the population, giving it the same hyperparameter-tuning budget as our method.

The key ablation holds the anchor count fixed at $k$ and the calibration model fixed. It varies only how anchors are selected: randomly, by highest topical similarity (TPMS-style), or by spread-maximizing selection. This isolates the claim that which anchors are used matters beyond how many.

Why not just…?

Why not just z-score each novice's scores? With three papers, a novice's sample mean mixes their leniency with the quality of what they happened to receive. LSC's analysis shows that a single shared paper is needed even to align offsets, and that scale needs two. Z-scoring also cannot correct cross-aspect halo. The ablation measures this directly on identical graphs.

Why not just give novices more papers? Load alone does not guarantee identifiability. Five papers that share no seniors, or five near-identical mediocre papers, leave $\sigma_{\min}(M_i)$ near zero. Load also costs novice effort, which the ICML study flags as a confound (3 vs 5.4 papers). We would compare at equal load: three spread-out anchors plus regular papers versus a larger random load. If random extra load matches our method, the design contribution is falsified, and we would say so.

Why not a Bayesian hierarchical model that shrinks novices toward the population? This is the real threat. Shrinkage makes estimates look well-defined even when the data do not identify them, and on well-connected graphs it may match our method. Our expectation is that it wins when novices are average and fails for atypical novices, such as a very harsh one, whom shrinkage pulls toward the mean. The file reports 71.5% vs 80.1% for Bayesian vs LSC, but that does not settle our question. We would therefore stratify error by how far each novice's true map lies from the population mean in semi-synthetic data. We would also test whether the Bayesian model combined with designed anchors beats either alone. If it does, we would adopt it as the estimator and keep anchor design as the contribution.

Why not just check whether real assignment graphs already satisfy the condition? That check is our first experiment. If standard assignments already give novices well-conditioned overlaps, the proposal reduces to a diagnostic tool, and we would report it that way.

Proposed Experiments

We would begin with a diagnostic on ReviewArena (51,529 papers, 196,099 reviews). For venues whose forms record aspect sub-scores, we would build the reviewer graph and treat the lowest-load reviewers as stand-ins for novices. We would then compute, per reviewer, rank and $\sigma_{\min}(M_i)$ against well-connected reviewers. This tells us how often real assignments leave low-load reviewers unidentifiable or ill-conditioned.

Next comes a semi-synthetic study. We would fit the multi-aspect model to ReviewArena and to PeerRead aspect-annotated reviews. From the fitted model we would generate data with known truth, then sweep three things: noise, the degree of score compression (the narrowness of the score spread), and the number of anchors. On the semi-synthetic ICLR 2019 setup from LSC we would report top-10% precision, so the scalar case is directly comparable to the 80.1% figure.

On real ground truth, we would use the Hamburg peer-grading data. We would subsample graders' assignments to novice-sized loads and compare anchor policies against TA grades, with gap and precision as the metrics.

The decisive checks are three. The first is whether error tracks $1/\sigma_{\min}(M_i)$ as predicted across policies. The second is whether spread-maximizing anchors beat random anchors at equal $k$ under compressed scores. The third is whether the advantage survives a Bayesian estimator.

A pilot in a student research workshop of the scale reported for EACL 2021 SRW (59 submissions) would then test the feedback loop. Novices would be randomized to see either their calibration report plus anchor reviews or only the anchor reviews. The outcome is the change in calibration error on a later round. We acknowledge the Hawthorne concern raised by the ICML study, and the randomized comparison is designed to address it.