Local Tournaments: Removing the Novelty Tax from LLM Pairwise Peer Review

Large language model (LLM) reviewing works better when the model compares two papers than when it scores one paper alone. In one large study, papers chosen by a pairwise LLM tournament went on to collect about as many citations as papers accepted by human committees. The same study found a cost: the tournament favored elite institutions and pushed topically novel papers down the ranking. Genomics has a name for this kind of failure, reference bias. A DNA read that differs from the single reference genome gets penalized for the difference, whether or not it is correct. The field's fix was not to add more references to one linear backbone. It was to compare alternative variants against each other at the specific site where they diverge, inside a graph of shared and branching sequence.

This note proposes the same repair for LLM peer review. The hypothesis is that the novelty tax is not a general dislike of new ideas. We suspect it comes from comparisons between papers that are topically far apart, where the judge can only fall back on familiarity, fluent phrasing, and prestige cues. If that is right, restricting comparisons to topically close papers should remove most of the tax at the same comparison budget. These local pools would mirror the area-chair tracks that editorial boards already use. A small number of bridging comparisons would then join the pools into one ranking, much as a senior area chair (a mid-level editor overseeing several tracks) reconciles tracks. The editorial hierarchy becomes a structure for the comparison graph, not just an administrative tree.

Core Claim

From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review ran up to 3 million GPT-4o mini matchups per conference and combined them with the Bradley-Terry model, a standard method that turns win/loss records into one score per item. Papers it selected averaged 20.0 future citations, against 19.36 for human-accepted papers and 11.41 for absolute LLM scoring. The same system also showed a higher institutional Gini coefficient than human committees and lower topic novelty among its top picks. That paper samples opponents from one global pool, so every paper is effectively compared against the field's center of mass.

We claim two things. First, the LLM judge's bias against novelty and toward prestige grows with the topical distance between the two papers being compared. Second, a comparison graph shaped like an editorial hierarchy removes most of that bias without losing the citation signal. In this graph, dense comparisons happen within tracks and sparse bridge comparisons happen across them. The thing that is new here is the diagnosis that the tax depends on distance, together with a sampling design derived from it. Neither appears in the cited work. Debiasing the judge prompt is a separate matter and not our contribution.

Research Program

The first step is diagnostic. We fit an extended Bradley-Terry model to the existing matchups:

$$\operatorname{logit} P(i \succ j) = s_i - s_j + \gamma\,(n_j - n_i)\,d_{ij} + \delta\,(p_i - p_j)\,d_{ij}$$

The symbols are defined as follows. $s_i$ is paper $i$'s latent quality. $n_i$ is its topic novelty, measured as the embedding distance to the previous year's accepted papers. $p_i$ is an indicator for an elite-institution author. $d_{ij}$ is the topic distance between the two papers. The coefficients $\gamma$ and $\delta$ measure how much novelty and prestige bias grow with distance. We will also fit a version with piecewise distance bins, which does not assume the effect is linear. If $\gamma$ and $\delta$ are near zero, meaning the bias is uniform across distances, the local-tournament idea is wrong. In that case we report the negative result and stop there.

The second step is the design. We cluster submissions into pools of roughly area-chair size, using the venue's own tracks where available and embedding clusters otherwise. Most of the comparison budget goes to within-pool pairs. The remaining fraction, which we sweep from 2% to 20%, goes to bridge comparisons. Bridge pairs are chosen among papers that sit topically between two pools, so even cross-pool comparisons stay short-range. This is the direct analogue of the "bubbles" in Mitigating Reference Bias with Graph-Based Pangenomes, where variants meet only at the sites where they diverge. Bradley-Terry is then fit on the whole connected graph.

We also test a variant with human anchors. It uses a small budget of existing human reviewer scores, taken from OpenReview ICLR records, to calibrate offsets between pools. This plays the role a senior area chair plays in the hierarchies documented in Findings of the Association for Computational Linguistics: EMNLP 2023.

The genomics literature also tells us where this design will struggle. KAGE reports that its accuracy drops when an individual's ancestry is underrepresented in the reference panel. The editorial analogue is a paper that fits no existing track. We handle these orphan papers explicitly. Any paper whose distance to every pool exceeds a threshold gets its own small pool, formed with its nearest neighbors. We report these papers' outcomes separately.

Baselines

The naive reference point is absolute LLM scoring with a rubric, meaning one score per paper. Using GPT-4o mini on the same papers, the cited paper reports 11.41 mean citations for papers it would accept. We rerun it to measure its novelty and Gini on our metrics.

The strongest existing method is the global pairwise tournament of From Replication to Redesign, which reports 20.0 mean citations for accepted papers. The file gives no numeric Gini or novelty values, so we recompute both from our own rerun. We run it with the same judge model, prompts, papers, and total number of comparisons as our method, which puts the cost near the reported ~$1,350 per conference for both systems. Neither system gets extra prompt tuning.

Beating it requires three things together. Our method must keep mean top-k citations within noise of the global tournament's, using a paired bootstrap over papers. It must significantly reduce the gap in topic novelty between LLM and human selections. And it must lower the institutional Gini.

Two ablations isolate the key ingredient. The first is local pools with no bridges, where each pool is ranked alone and acceptances are allocated in proportion to pool size. This tests whether bridging matters or whether partitioning alone does the work. The second keeps the budget matched but samples pairs at random within distance deciles. This shows whether the effect tracks topic distance specifically rather than the density of the comparison graph.

Why not just…?

Why not just impose a novelty or institution quota on the global ranking? This is the real threat. A quota can match any target output distribution cheaply. What it cannot fix is the ordering among novel papers, if that ordering was itself distorted by far-away comparisons. We therefore evaluate within-novelty-tercile Spearman correlation between rank and future citations. A quota on global scores and our local tournament select the same number of high-novelty papers. If the quota's picks within the top tercile predict citations as well as ours, the quota wins and we will say so.

Why not just anonymize affiliations and strip hype phrasing before comparison? ReviewerGPT? shows GPT-4 is swayed by bombastic language and by $p<0.05$ results, so input cleaning is a reasonable fix for $\delta$. It does not address familiarity bias, which acts on content rather than surface cues. We run cleaned inputs with both global and local sampling as a two-by-two comparison. The interaction term shows whether the two fixes are complementary.

Why not just use calibrated absolute rubrics, or add more reviewers? Absolute scoring collapses into scores of 6–7 and selects papers averaging 11.41 citations (From Replication to Redesign). Human rubric scores in PeerRead track substance and clarity much more than originality. Adding judges to one global pool increases cost and leaves the distance structure of the comparisons unchanged.

Proposed Experiments

We use the ICLR, NeurIPS, EMNLP, and CoRL sets from From Replication to Redesign. We add ICLR 2017 from PeerRead, whose human originality annotations give an independent check on the embedding-based novelty measure. For every system we select the venue's actual number of acceptances. We then report mean future citations, the novelty gap relative to human selections, the institutional Gini, within-tercile citation correlation, and outcomes for orphan papers.

The decisive checks are three. The fitted $\gamma$ must be significantly positive. Local sampling must shrink the novelty gap relative to the global tournament at a matched budget. And the quota baseline must fail to match our within-tercile correlation.

Citations are themselves biased toward canonical topics and prestigious institutions, which would work against detecting any gain for novel papers. To account for this we also report agreement with PeerRead's human originality scores. We treat citation parity as a floor to maintain, not the objective.