Peer review is a cascade. An editor-in-chief hands a manuscript to a handling editor. That editor picks reviewers, the reviewers raise arguments, and the arguments split into sub-points and evidence. Each manuscript's journey can therefore be written as a rooted tree with labelled nodes. Comparing such trees is how one would audit a journal: check whether two venues run the same process, or spot a handling editor whose subtrees look systematically different from their peers'. Theory says this comparison becomes expensive as trees get deeper. Practice says review trees are shallow. We propose to measure, rather than assume, the depth at which the decision-relevant structure of review trees stops growing. We call that depth the saturation height. We measure it on real multi-round review data and on a simulated editorial board with planted bias at known tiers, using comparison methods whose cost is explicitly controlled by height.
The payoff runs in two directions. If outcome and bias signal saturates at height two or three, auditing tools can rely on cheap, exactly computable tree signatures. They can ignore the worst-case hardness results, with evidence rather than hope. If the signal keeps growing with height, especially for bias planted at intermediate editorial tiers, then the hardness results bite in a practically relevant regime. Auditors would then need approximate comparison methods, and we would know where.
Core Claim
The complexity literature shows that comparing trees, as opposed to computing on one tree, is governed by height. Subtree Isomorphism Revisited shows that asking whether one tree embeds inside another (subtree isomorphism) cannot be solved in $O(n^{2-\epsilon})$ time under the Strong Exponential Time Hypothesis, the standard conjecture that brute-force satisfiability cannot be beaten by much. This holds even for trees of height $O(\log\log n)$, which for any realistic $n$ means height four or five. The same paper finds that the largest common subtree problem stays quadratic even for shallow trees. The Wasserstein Weisfeiler-Lehman subtree distance of Wasserstein Graph Distance Based on L1-Approximated Tree Edit Distance between Weisfeiler-Lehman Subtrees approximates tree edit distance by counting shared complete subtrees, at a cost exponential in height. Graph Similarity Based on Matrix Norms shows that norm-based similarity is hard to approximate even on bounded-degree trees.
These results motivate the question. They do not answer it, because they are worst-case statements about unlabelled trees. Our claim is empirical and falsifiable. On review trees, the gain in predicting editorial outcomes, and in detecting planted editorial bias, from comparing trees truncated at height $h$ flattens at a small saturation height $h^*$. The exception is bias planted at tier $t$, which should require $h \ge t + 1$ to detect. If confirmed, the useful regime for auditing sits exactly where exact rooted-tree isomorphism classes are cheap, linear time by canonical encoding. The hard problems (containment, common subtree, edit distance) become optional refinements whose value we can price.
Concretely, let $T_{\le h}$ denote a labelled review tree cut off below depth $h$. Let $\phi_h(T)$ be the vector counting how often each isomorphism class of depth-$h$ rooted subtree appears in $T_{\le h}$. On rooted trees, Weisfeiler-Lehman colour refinement, an iterative relabelling of each node by its children's labels, identifies these classes exactly. We define
$$ h^* = \min\{h : \mathrm{Perf}(\phi_{h+1}) - \mathrm{Perf}(\phi_h) < \delta\}, $$
where $\mathrm{Perf}$ is held-out task performance of a fixed classifier on the signatures and $\delta$ is a preregistered tolerance (one macro-F1 point). We report the full performance-versus-height curve alongside measured comparison cost.
Research Program
The first component is a representation of review processes as labelled rooted trees. For real data we use FIRSTPASS, a multi-round, multi-domain set of 3,668 records with real editorial outcomes. The manuscript is the root. Its children are first-round reviewers, then aspect nodes (clarity, novelty, soundness, following PeerArg's aspect scheme), then argument nodes, then supporting or attacking sub-arguments and evidence. Node labels combine role, aspect, stance, and a coarse severity bucket.
We use only first-round material. FIRSTPASS's outcome task is distinguishing standard from extended review rounds, so including later rounds would leak the label. The depth we study is therefore argumentative depth within the first round, plus the editor tier where the record exposes editor-level text. Argument trees are extracted by an LLM, with a manually annotated audit sample to measure extraction error at each depth. Deeper levels are noisier, and that noise could masquerade as saturation.
The second component is a deeper editorial board in simulation. Editorial behaviors in peer review built a 1,000-agent model with three tiers and found that 10% biased editors lower accepted-paper quality by 11.01%, versus 6.81% for 10% biased reviewers. It attributed the gap to editors controlling reviewer assignment. We extend this model to boards of depth three to five: editor-in-chief, senior editor, associate editor, reviewer, and in the deepest setting an area-chair layer. We plant biased agents at a chosen tier $t$ and record each manuscript's full delegation tree.
Because bias at tier $t$ acts by reshaping the subtree below the biased agent, the model makes a sharp prediction. Detecting which board unit contains a biased agent should require signatures of height at least $t+1$ measured from the root. This supplies the ground truth the real data cannot.
The third component is a ladder of comparison methods ordered by cost. The bottom rung is the exact truncated isomorphism-class signature $\phi_h$, which is linear in tree size per height. The next rung is the L1-approximated tree edit distance inside an optimal-transport distance, following Wasserstein Graph Distance Based on L1-Approximated Tree Edit Distance between Weisfeiler-Lehman Subtrees, which tolerates small structural differences that $\phi_h$ treats as all-or-nothing. The top rung is exact tree edit distance and largest common subtree, run only on samples small enough to finish. For each rung and height we log wall-clock cost. The deliverable is a cost-versus-signal frontier, not a single score.
Baselines
The naive reference is the height-zero and height-one signature: counts of reviewers, aspects, and arguments, plus the majority class. If this matches deeper signatures, structure beyond counts carries nothing. On the bias task, its analogue is per-unit acceptance-rate statistics with no tree structure.
The strongest existing method on real outcomes is the fine-tuned Qwen2.5-7B-Instruct of FIRSTPASS. It reports 80.5% accuracy and 78.2% macro-F1 on the standard-versus-extended task, 10.4 points above a zero-shot Gemini model. We rerun it on our first-round-only split with the same data and tuning budget, because the published number may use material we exclude. We then compare text-only against text plus $\phi_{h^*}$. Beating it means a statistically reliable macro-F1 gain from adding structure on identical inputs, not an absolute number above 78.2.
The strongest tree-structured competitor is PeerArg: Argumentative Peer Review with LLMs. It builds height-two argument trees and aggregates them with bipolar argumentation semantics. It reports 76.64% macro-F1 on PRA versus 72.5% for end-to-end Mistral-7B, and 68.09% versus 59.5% on MOPRD. PeerArg is effectively our method frozen at $h=2$ with a hand-designed aggregator. We run it on PRA and MOPRD with its published setup and on FIRSTPASS with our extracted trees. Beating it at $h>2$ is the direct test that depth adds signal.
A graph-neural baseline in the style of the message passing in ResearchTown represents a learned, non-height-controlled alternative. We give it the same trees, the same node labels, and a matched hyperparameter search budget.
The key ablation shuffles labels within each depth while preserving tree shape. This separates "deeper structure helps" from "deeper text helps."
Why not just…?
Why not just feed the review text to a fine-tuned LLM? This is the real threat. The FIRSTPASS model already reaches 78.2% macro-F1, and a model reading the text may implicitly recover argument depth. Our experiments settle it with the text-versus-text-plus-structure comparison. If structure adds nothing on real data, we report that. The study then still delivers the simulated-board result, where the target is not outcome prediction but locating a biased tier, which text accuracy does not address.
Why not just use counts and simple process statistics? Counts are our height-zero and height-one rungs, so this alternative is built into the curve rather than argued away. Wang et al.'s mechanism, editors acting through who they assign, predicts that unit-level acceptance rates can hide bias that changes subtree composition without changing totals. The simulation tests this directly.
Why not just run an off-the-shelf Weisfeiler-Lehman kernel and skip the complexity framing? On rooted trees, our cheapest rung is a Weisfeiler-Lehman kernel, so the method itself is not the contribution. The contribution is three things: the saturation height, its link to the tier where bias is planted, and pricing what the expensive comparisons (edit distance, common subtree) buy beyond it. The hardness results tell us those expensive comparisons cannot be made cheap in general, so knowing whether they are needed is the useful output.
Proposed Experiments and Decisive Checks
On simulated boards of depth three to five, with 10% biased agents at tier $t \in \{1,2,3\}$, we measure unit-level bias-localisation AUROC as a function of signature height. The decisive check is whether detection AUROC rises sharply at $h=t+1$ and stays near chance below it. A null here would mean subtree shape does not carry the bias, contradicting the assignment mechanism in Wang et al.
On FIRSTPASS, PRA, and MOPRD, we plot macro-F1 against $h$ from zero to four for each comparison rung, with cost on a second axis, and report $h^*$ with bootstrap intervals. We stratify FIRSTPASS by its five domains to check whether $h^*$ is domain-dependent. We repeat on the hand-annotated audit subset to confirm that apparent saturation is not caused by extraction noise at depth.
We consider the hypothesis refuted if real-data performance keeps improving at $h=4$ with clean annotations. In that case review trees live in the regime where the hardness results apply, and we report how much the approximate distance recovers relative to exact edit distance on the samples where both finish.
References
- Subtree Isomorphism Revisited
- Wasserstein Graph Distance Based on L1-Approximated Tree Edit Distance between Weisfeiler-Lehman Subtrees
- Graph Similarity Based on Matrix Norms
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes
- Editorial behaviors in peer review
- PeerArg: Argumentative Peer Review with LLMs
- ResearchTown: Simulator of Human Research Community