Much of what we now claim to know about peer review comes from datasets that contain only the papers that got through. Transparent-review corpora publish reviews only for accepted papers. Opt-in conference releases lose the rejected papers whose authors declined to share. Models trained on these corpora therefore learn how surviving papers and their reviewers negotiate, not how the gate itself works. This is the same trap career researchers call survivorship bias: studying tenured professors to learn what gets people tenure. We propose to measure how badly accepted-only data distorts conclusions about editorial decisions, and to build a correction for it. The correction models the editorial filter as a two-tier process. Reviewers score a paper, and then an editor or area chair decides, possibly using information the reviews do not contain.
The central observation is simple. When a decision depends on two signals, keeping only the accepted cases creates a spurious negative relationship between those signals. Among accepted papers, a paper with weak clarity scores must have had strong substance scores to get in. How strong that artifact is, and whether it can be undone, depends on the structure of the board. If the deciding editor mostly follows the reviewers, the filter is a known function of observed scores, and standard reweighting can repair the survivor sample. If the editor acts on private judgment, for example from reading the paper, then part of the filter is invisible in the reviews. No reweighting on reviews alone can fully repair it. We turn this "editor's private-information share" into a measurable quantity. We estimate it on venues where rejected papers are observable, and use it to put honest bounds on conclusions drawn from accepted-only corpora such as FIRSTPASS.
Core Claim
We make two claims. First, survivor-only peer review data systematically misstates the relationships that editorial decisions depend on, and the size and sign of these errors can be measured on venues that expose their rejections. Second, a selection model split by tier recovers the true relationships substantially better than a single-stage correction. The first stage models how reviewer scores feed a recommendation. The second models how much of the final decision is explained by paper content beyond the reviews. This advantage should grow with how much the deciding tier relies on information outside the reviews. If the flat correction does just as well, that is itself a finding: the editorial tier adds little independent filtering at these venues, and the hierarchy can be ignored for bias correction. The experiments are designed so that either outcome is informative.
Where This Sits in Prior Work
The survivorship side motivates the problem.
- Gender and Academic Promotion to Full Professor in Ontario reconstructs a promotion pipeline from a salary disclosure list and a population baseline. It shows that conclusions change once the missing denominator is supplied: a raw productivity gap largely vanishes after adjusting for co-authorship.
- Bayesian modelling of elite sporting performance with large databases shows that cross-sectional aging curves are distorted because only non-declining athletes remain.
- Mid-career pitfall of consecutive success in science studies only careers lasting 20 years or more, a survivor population by construction.
We take their logic, that conditioning on surviving a gate manufactures patterns, and apply it one level down. The gate we study is the editorial decision on individual manuscripts, not the career.
The peer review datasets define the problem concretely.
- FIRSTPASS builds 3,668 multi-round Nature Communications review records. It predicts "standard" versus "extended" review, but contains only accepted papers.
- PeerRead includes ICLR 2017 and arXiv subsets with both accepted and rejected papers. Its authors acknowledge opt-in selection in the ACL and CoNLL portions.
We use PeerRead's full-denominator portions as ground truth. We deliberately degrade them into survivor-only views, then test whether corrections recover what the full data shows.
The editorial-structure side supplies the mechanism.
- Editorial behaviors in peer review simulates a single editor tier. It finds that 10% biased editors lower accepted-paper quality by 11.01% versus 6.81% for 10% biased reviewers, which suggests the editor's own filtering matters more than reviewer noise.
- Reviewer bias in single- versus double-blind peer review measures reviewer-level prestige effects in a WSDM experiment, but stops before the chairs' decisions.
Both papers treat the decision tier as either synthetic or unobserved. We instead estimate from real data how much the deciding tier departs from its reviewers, and show why that quantity determines whether survivor data can be trusted.
Research Program
We would build three things.
Survivor views of full-denominator venues. We take PeerRead's ICLR 2017 and arXiv subsets, which carry accept/reject labels, and produce two degraded views. One keeps only accepted papers. A second approximates FIRSTPASS by also dropping papers below a review-score threshold, mimicking triage losses. Every analysis run on these views has a known answer from the full data.
Survivor-sensitive target quantities. We pick three estimands whose values could plausibly change under selection:
- the association between review aspects (substance, clarity, originality, soundness) and the overall recommendation, which PeerRead reports as dominated by substance and clarity on full data;
- the calibration and accuracy of an outcome predictor;
- the correlation between pairs of review aspects.
The aspect-pair correlation is the most direct test of the manufactured negative relationship, because under the null of no selection it should not flip sign.
A two-tier selection model.
- Tier one predicts the final decision from reviewer scores alone. This is the "editor follows reviewers" hypothesis.
- Tier two adds features of the manuscript text that an area chair would read but that are not summarized in scores.
The extra explanatory power of tier two over tier one, measured as held-out log-likelihood gain on the full data, is our operational definition of the editor's private-information share. The correction reweights survivor papers by the inverse of their estimated acceptance probability. It then widens intervals according to the private-information share, because the part of selection that depends on unobserved content cannot be reweighted away. Where that share is large, we report partial-identification bounds rather than point estimates.
Application. We apply the fitted selection structure, transferred from the full-denominator venues, to FIRSTPASS. We then ask which of its published patterns survive correction, such as the consistent 63–71% standard-review share across domains and the learnability of the outcome. Transfer across fields is an assumption, not a fact. We therefore state results for FIRSTPASS as sensitivity ranges indexed by the private-information share, not as corrected truths.
Baselines
Naive survivor-only analysis. This is the "do nothing" reference: fit each estimand directly on the accepted-only view, exactly as a FIRSTPASS-style study would. It uses the same papers, features and model class as every other arm. Any correction that fails to move estimates closer to the full-data value than this baseline has failed.
Full-data oracle. This is the same analysis on the complete PeerRead ICLR and arXiv data. It is the target, not a competitor. The gap between naive and oracle is the survivorship distortion we are measuring. We will report it before claiming any correction works, because a small gap would make the rest of the program moot.
Flat inverse-probability reweighting. This is the strongest real competitor. It fits one acceptance model from reviewer scores, reweights survivors, and ignores any editor-tier structure. We will give it the same features available to tier one of our model and the same held-out tuning budget, selected by cross-validated likelihood on the full data. Beating it means the two-tier model's corrected estimates land significantly closer to the oracle across bootstrap resamples. In addition, its intervals must cover the oracle value at the nominal rate where flat reweighting's intervals do not. If flat reweighting matches us, we report that the editor tier is not an independent filter at these venues.
Ablation of the editor tier. This arm uses our full procedure with tier two replaced by a model given shuffled manuscript text. It isolates whether any gain comes from modeling the editor's use of paper content, rather than from extra parameters or the interval widening itself.
Published predictor numbers as context. On the predictive side, the strongest reported numbers are:
- a fine-tuned Qwen2.5-7B-Instruct at 80.5% accuracy and 78.2% macro-F1 on FIRSTPASS outcome prediction, 10.4 points above zero-shot Gemini (FIRSTPASS);
- 65.3% acceptance accuracy on ICLR and 75.7% on arXiv cs.CL (PeerRead).
We do not aim to beat these accuracies. We aim to show how much of an accepted-only predictor's apparent skill fails to carry over once rejected papers are included. No reported result was found on Lacuna for selection-corrected estimates on any peer review benchmark.
Why not just…?
Why not just collect the rejected papers? Where rejections are public, as in PeerRead's ICLR subset, that is exactly what we do, and it is our ground truth. The problem is that the most detailed corpora, such as FIRSTPASS's multi-round journal reviews, come from venues that never release rejections. PeerRead itself shows that opt-in release reintroduces selection. The proposal exists for the corpora where collection is not an option. The full-denominator venues are the lab where the correction is validated.
Why not just use standard single-stage reweighting? This is the real threat. If editors at these venues largely rubber-stamp reviewer scores, flat reweighting is correct and our tier structure adds nothing. PeerRead's finding that decisions are only moderately predictable leaves room for either outcome, since its 65.3% on ICLR came from paper text rather than scores. The head-to-head against flat reweighting, with the shuffled-text ablation, settles it. We commit to reporting a null here as a substantive result about how little independent filtering the editor tier does.
Why not just simulate the editorial hierarchy with agents? Editorial behaviors in peer review shows what synthetic editor bias could do but calibrates nothing to real decisions. LLM-agent simulators such as ResearchTown reach only 0.49 semantic similarity to human reviews. A simulation's selection function is whatever the modeler writes down. Our question is what the real selection function is, which only decision data can answer.
Why not just report worst-case bounds and skip modeling? Assumption-free bounds are always valid, and we report them as the fallback. On their own they are typically too wide to say anything about aspect associations. The point of estimating the private-information share is to shrink bounds exactly as much as the data justifies. We will show the bound width under both approaches so readers can judge the trade.
Proposed Experiments and Decisive Checks
Is the distortion real? On ICLR 2017 and the arXiv subsets, we compare the naive survivor-only and oracle estimates for aspect-acceptance associations and aspect-pair correlations, with bootstrap intervals. The decisive signature of survivorship is aspect pairs that are positively correlated in the full data but negatively correlated among accepted papers. If no sign flips or material shifts appear, the problem is smaller than claimed and we say so.
Does the tier structure help recovery? We measure absolute error to the oracle and interval coverage across naive, flat reweighting, two-tier, and ablated arms. We then plot the two-tier advantage against the estimated private-information share, computed separately per venue and per subfield of arXiv. The core claim predicts a positive slope. A flat line near zero refutes it.
Does the correction transfer? We fit the selection structure on ICLR and apply it to the survivor view of the arXiv subsets, then check against their true labels. This is the transfer step that application to FIRSTPASS relies on. If cross-venue transfer fails here, the FIRSTPASS analysis will report only assumption-free bounds.
Which FIRSTPASS findings are robust? We retrain the outcome predictor under corrected weights. We then report, as a function of the assumed private-information share, the range of plausible standard-review shares and predictor accuracies. A finding that holds across the whole range, such as consistent cross-domain round distributions, can be trusted. One that flips within plausible ranges should be flagged as a possible artifact of seeing only survivors.
References
- Gender and Academic Promotion to Full Professor in Ontario
- Bayesian modelling of elite sporting performance with large databases
- Mid-career pitfall of consecutive success in science
- FIRSTPASS: A Multi-Domain, Multi-Round Peer Review Dataset Grounded in Real Editorial Outcomes
- A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
- Editorial behaviors in peer review
- Reviewer bias in single- versus double-blind peer review
- ResearchTown: Simulator of Human Research Community