Scientific journals are run as a hierarchy. Reviewers write critiques, and an editor reads them and decides what happens next: accept, or send the paper back for another round. Simulations of peer review assume editors follow fixed vote-counting rules. Machine learning work on review data instead feeds the whole review dialogue to a language model and predicts the outcome. Neither line of work answers a basic empirical question. When an editor sends a manuscript into a long revision loop, how much of that decision is a mechanical function of what the reviewers did across rounds, and how much is editorial judgment beyond the reviewers?
We propose to answer it on real multi-round review histories. Each reviewer's behavior across rounds becomes a short sequence of discrete actions: raising a concern of a given type, requesting new experiments, conceding a point after the author's reply, holding firm, or changing the recommendation. These sequences are embedded the way recommender systems embed click histories. We then fit a ladder of editor models to the actual editorial outcomes, from fixed voting rules up to a fine-tuned language model reading the full text. The accuracy gap between the best model that sees only reviewer behavior and the model that sees everything is a direct, interpretable estimate of editorial discretion.
Core Claim
The central hypothesis is that, on FIRSTPASS, the decision to extend review beyond two rounds is largely explained by a simple aggregation over per-reviewer stance trajectories. FIRSTPASS is a corpus of 3,668 complete Nature Communications review histories across five sciences, labeled as STANDARD (2 rounds) or EXTENDED (3+ rounds). The hypothesis implies that a set-structured model over reviewer trajectories should come within a small margin of the fine-tuned Qwen2.5-7B text classifier that FIRSTPASS reports. It also implies that one of a few identifiable aggregation rules should account for most decisions, and that this rule may differ by field.
If the hypothesis is false, the residual becomes the object of study. A large gap would mean editors act on information that reviewer behavior does not carry. We would then test where that information sits: the author response, manuscript surface features, or the domain.
What is new is the framing and the measurement, not a better accuracy number. FIRSTPASS treats the trajectory as text for one classifier and does not separate the reviewer tier from the editor tier. Editorial behaviors in peer review posits three editor rules: P-all (accept only if every reviewer approves), P-any (accept if at least one approves) and P-majority. It then simulates their consequences, finding that 10% biased editors cut accepted-paper quality by 11.01% versus 6.81% for 10% biased reviewers. It never checks which rule real editors follow. We fit those rules, and richer ones, to real decisions.
Research Program
The first component is a reviewer action vocabulary and a tagger. We define roughly twenty to forty action types covering concern type (methodological, novelty, clarity, missing experiment), severity, request versus suggestion, concession, persistence, and explicit recommendation level. A language model tags each sentence of each review with these types. A few hundred reviews are hand-annotated so we can report per-type agreement before anything downstream is trusted.
This step is needed because of a warning in prior work. Student Behavioral Embeddings embedded raw clickstreams with skip-gram and found they predicted social activity, improving error by 8–21%, but not learning outcomes: 0.06% on final scores and up to 4.80% worse on grade gains. Raw activity traces like review length or response timing may likewise capture pacing rather than evaluative stance. Our tokens are therefore defined over evaluative content, and a pure-activity variant is kept as a deliberate contrast.
The second component embeds the reviewer trajectories. Action embeddings are learned with a skip-gram objective over action sequences, following Improving Music Recommendation Using Distributed Representation, which places items that co-occur in local windows near each other. A small transformer then encodes each reviewer's ordered sequence across rounds into a few vectors. The idea of compressing a long history into a handful of tokens comes from User-LLM, which reports a 78.1x reduction in compute and 16.33% better accuracy than text prompting on Google Review category prediction.
To capture how a reviewer moves between rounds, we borrow the translation idea from TransRec. There, a user's next item is modeled as the previous item plus a user-specific offset vector. Here, a reviewer's second-round stance is modeled as their first-round stance plus an update vector conditioned on the author response. That update vector is what an editor plausibly reads.
The third component is the editor ladder, which models the hierarchy explicitly. At the bottom are fixed rules over per-reviewer final recommendations: P-all, P-any and P-majority read as "extend unless all/any/most reviewers are satisfied." Next comes a weighted rule with learned per-action-type weights. Above that is a permutation-invariant set function over reviewer trajectory embeddings. Permutation invariance matters because reviewer order is arbitrary, and a flat text concatenation ignores this. At the top is the fine-tuned full-text model. Every rung uses the same splits and the same decision point.
Because FIRSTPASS contains only accepted papers, all conclusions concern the extension decision conditional on eventual acceptance, and we state this throughout. As a secondary check, PeerRead ICLR reviews and decisions let us test whether area-chair accept/reject decisions follow the same rule family. PeerRead offers single-round reviews with accept/reject outcomes. Its own acceptance classifiers rely on surface features of the paper (65.3% on ICLR), not on reviews.
Baselines
The naive reference is the majority class (always STANDARD) on our split. FIRSTPASS reports STANDARD/EXTENDED proportions of roughly 63–71% across disciplines, so accuracy alone is misleading and F1-macro is primary.
The simplest structured reference is the best single fixed rule (P-all, P-any or P-majority) applied to tagged final-round recommendations. This is the hypothesis that the editor is a vote counter. Any learned model must beat it to show that trajectory detail, not just the final vote, matters.
The strongest existing method is the fine-tuned Qwen2.5-7B-Instruct classifier from FIRSTPASS. It reports 80.5% accuracy and 78.2% F1-macro on outcome prediction, 10.4 points above zero-shot Gemini-3.1-flash-lite-preview. We re-run it on our splits with FIRSTPASS's input protocol. We also run a variant restricted to exactly the information our stance models see at each decision point, so the comparison does not mix "better model" with "more input." Tuning budgets are matched in number of configurations tried, and the compute gap is reported rather than hidden.
"Beating" this baseline is defined in two ways. The stance-only set model wins on interpretability if it comes within two F1-macro points of Qwen at matched inputs. It wins outright if it exceeds Qwen on cross-domain transfer: train on four fields, test on the fifth.
Two ablations isolate the key ingredients. The first replaces evaluative action tokens with activity-only features (lengths, counts, round timing), which tests the student2vec failure mode directly. The second removes cross-round ordering by pooling actions without sequence position, which tests whether the trajectory, as opposed to a bag of concerns, carries the signal.
Why not just…?
Why not just fine-tune the LLM on full text and stop? It already reaches 80.5% accuracy. It is a real threat on raw accuracy, and we do not claim to beat it there. However, it cannot say whether editors are aggregating reviewer stances or exercising independent judgment, because reviewer and editor information are fused in one token stream. The matched-input Qwen variant settles the question. If text-only Qwen restricted to reviewer content scores no better than our stance model, the extra text adds no editor-relevant signal. If it scores much better, we have located discretion and can probe which spans it attends to.
Why not just use handcrafted features with gradient-boosted trees? Candidate features would include counts of major concerns, number of reviewers who are still negative, and review lengths. This may well work as well as the embeddings. The comparison in Customer Lifetime Value Prediction Using Embeddings does not decide the matter for us. There, learned customer embeddings outperformed 132 handcrafted features, at 0.798 churn AUC, but in a very different domain. So we run boosted trees on exactly the same tagged action counts. If they match the set model, the conclusion is that editors follow a low-dimensional rule. That is the scientific finding we care about, and we would report the simpler model as the result.
Why not just assume majority vote, as simulations do? That assumption is precisely what Editorial behaviors in peer review makes without testing. The fixed-rule rung tests it, separately for each field. If one rule fits well, simulation studies gain an empirically grounded editor model. If no rule fits, their conclusions about editor-versus-reviewer vulnerability rest on an unvalidated premise.
Proposed Experiments and Decisive Checks
The main experiment trains every rung of the ladder on FIRSTPASS. We report accuracy, F1-macro and calibration, stratified by field, at two decision points: after round one with the author response, and after round two. The headline quantity is the discretion gap, the F1-macro difference between matched-input Qwen and the best stance-only model, with bootstrap intervals.
The first decisive check concerns field-specific editing. If the best-fitting fixed rule differs across Biology, Physics and the other three fields, and per-field rules beat a pooled rule on held-out data, editorial practice is field-specific. If not, a single rule describes the journal.
The second decisive check is the activity ablation. If activity-only features predict extension nearly as well as evaluative tokens, extension tracks reviewer effort, not judgment. That would replicate the student2vec failure in reverse and argue against reading the outcome as a quality signal.
The third check is tagger robustness. All conclusions are recomputed under tagger outputs perturbed to the measured disagreement rate on the hand-annotated sample. Any finding that flips is reported as fragile.
Finally, the fitted rule family is tested on PeerRead ICLR reviews against area-chair decisions, to see whether the editor-as-aggregator picture transfers from journal editors to conference chairs.
References
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes
- Editorial behaviors in peer review
- Student Behavioral Embeddings and Their Relationship to Outcomes in a Collaborative Online Course
- Improving Music Recommendation Using Distributed Representation
- User-LLM: Efficient LLM Contextualization with User Embeddings
- Translation-based Recommendation
- A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
- Customer Lifetime Value Prediction Using Embeddings