When a top machine learning conference accepts a paper, that paper goes on to be cited more than papers the conference rejected. Two stories explain this, and they have very different consequences. Under the first, reviewers are good at spotting strong work, and acceptance simply marks quality that would have drawn citations anyway. Under the second, the acceptance label itself creates citations through visibility, prestige, and the community's habit of reading what top venues publish. The second story matters most for the large group of sound papers where, as The Impact Market to Save Conference Peer Review argues, the decision is close to a coin flip. If that is how acceptance works, noisy review does more than misclassify papers. It hands out lasting citation advantages close to at random.
The obstacle is a textbook case of unobserved confounding: a hidden variable, here the paper's true quality, drives both the treatment (acceptance) and the outcome (citations), so a raw comparison mixes the two stories. This note proposes to estimate the causal effect of acceptance on citations at ICLR, where rejected papers are public. It draws on two sources of leverage that prior work has not combined. The first is reviewer harshness: the part of a reviewer's numerical score that is not explained by what the reviewer actually wrote, which plausibly moves the decision without reflecting quality. The second is the citations a paper's arXiv preprint collected before the decision. Acceptance cannot cause those, so any acceptance-related gap in them measures leftover confounding and can be used to set the width of formal sensitivity bounds.
Core Claim
The effect of selective acceptance on later citations can be bounded informatively, not just point-estimated under an untestable assumption. The approach pairs an instrument built from reviewer scoring idiosyncrasy with a sensitivity analysis whose confounding strength is calibrated on citations collected before the decision. The expected finding is a real but modest label effect concentrated among borderline papers. That result would put a number on the "equivalence class" argument for the first time.
Why This Is Not Already Done
Several papers come close, and each one leaves out the acceptance gate itself. Estimating the Causal Effect of Early ArXiving on Paper Acceptance uses the same ICLR 2018–2022 data, but it studies the reverse arrow: whether preprinting raises acceptance. It removes quality confounding by treating later citations as a negative outcome control, meaning an outcome assumed to reflect quality but not the treatment. Our question makes citations the outcome, so that device is unavailable to us. Our work also stress-tests the assumption behind it, since a large acceptance-to-citation effect would weaken the case for citations as a pure quality signal.
Twin Papers pairs papers that cite each other in order to estimate venue effects. It finds, for example, +2.78 for IEEE Transactions on Signal Processing over ICASSP. Because its pairs exist only when both papers were published, it compares venues and never compares accepted with rejected papers. CausalCite builds counterfactuals by matching on text, and it explicitly treats venue as a mediator to be left unadjusted rather than as a treatment. CiteTracked links reviews to citations for accepted NeurIPS papers and finds that the oral and poster tiers converge over time. Its analysis is correlational and contains no rejected papers.
On the methods side, Auditing Fairness under Unobserved Confounding supplies the core trick we reuse: data from before treatment narrows bounds under a confounding-strength parameter $\gamma$. That paper works on clinical drug allocation. Sharp Bounds for Generalized Causal Sensitivity Analysis supplies sharp bounds of the marginal-sensitivity type but gives no recipe for choosing $\gamma$ in this domain.
Research Program
The unit of analysis is an ICLR submission from 2018 to 2022. Each is linked to Semantic Scholar citations, following the linkage of the early-arXiving study. The treatment is the accept or reject decision. The primary outcome is log citations three years after the decision. We count citations to every version of the work, so that a rejected paper later published elsewhere is still credited. This makes the estimand an intent-to-treat effect of a first-round acceptance, which includes any delay a rejection imposes.
The instrument is built in three steps. First, we train a model that predicts each review's numerical score from its text alone. Second, we take each review's residual, the gap between the score the reviewer gave and the score their text predicts, and treat it as that reviewer's harshness or generosity in turning an opinion into a number. Third, the paper-level instrument is the average of these residuals across its reviewers.
The instrument rests on a specific assumption: once the text, the observed covariates, and the topic are accounted for, a reviewer's calibration affects citations only through the decision. Its main threat is that numbers sometimes carry quality information the text omits. This is exactly where the second ingredient enters.
The second ingredient uses the roughly 1,486 early-arXived papers identified in the early-arXiving study, along with any others we recover. For these papers we compute citations accumulated before the decision date. Acceptance cannot cause these citations. Any association between them and the instrument, after conditioning on covariates, therefore directly measures how much quality leaks into the instrument. Likewise, the gap between eventually accepted and eventually rejected papers measures confounding in the naive comparison.
We use these placebo gaps to calibrate $\gamma$, the maximum factor by which unobserved quality can shift the odds of acceptance. This is analogous to how the auditing paper calibrates its bounds with pre-allocation data. We then report sharp bounds on the acceptance effect at the calibrated $\gamma$, alongside the instrument estimate. We also report the breakdown value: the $\gamma$ at which the effect's sign is no longer identified.
A concrete risk is that early-arXived papers are not representative, since by that study's own account they differ in observable ways. We therefore report the calibrated bounds both for that subgroup and for the full population, with the transfer of the calibration to the full population stated explicitly as an assumption.
Baselines
The do-nothing reference is the raw citation gap between accepted and rejected submissions. Its main purpose is to show how much of that gap the later methods attribute to quality.
The first serious competitor is covariate matching in the style of the early-arXiving study. It matches on the same 18 observed features (author citation counts, institutional rank, topic clusters) and adds dense text embeddings of the abstract, in the spirit of CausalCite's text matching. We will run it on the identical sample, the same outcome window, and the same covariates the instrument model uses.
The strongest competitor is a fuzzy regression discontinuity on the mean reviewer score. A regression discontinuity compares papers just above and just below an acceptance cutoff, where quality should be nearly equal. "Fuzzy" means the cutoff only raises the chance of acceptance rather than determining it, which fits ICLR because area chairs override scores. The bandwidth will be chosen by a standard data-driven rule and its sensitivity reported.
We found no reported result on Lacuna for the causal effect of conference acceptance on citations under any of these designs. The nearest reported figures concern different questions. Matching inflated the early-arXiving acceptance effect to about 10%, against under 4% after quality adjustment, per Estimating the Causal Effect of Early ArXiving. The venue effect of +2.78 comes from Twin Papers.
Two ablations isolate the key ingredients. The first swaps the text-residual instrument for the raw deviation of a reviewer's score from the paper's other reviewers, which tests whether the text model is doing any work. The second reports bounds of the type in Sharp Bounds over a generic range of $\gamma$ without pre-decision calibration, which tests whether calibration actually narrows the answer.
The proposal beats the baselines if three conditions hold together. The calibrated bounds must exclude the raw gap. The instrument must pass the pre-decision placebo test where matching fails it. The calibrated interval must be materially narrower than the uncalibrated one at the same coverage.
Why not just…?
Why not just match on text embeddings and author covariates? Matching conditions only on what authors wrote and who they are. CiteTracked reports that review text predicts citation tier better than abstracts (F1 0.85 vs. 0.83), so author text misses evaluative information. The early-arXiving study shows that matching alone overstated an effect by more than double. Our placebo test settles the question directly: if matched accepted and rejected papers still differ in citations collected before the decision, matching has left quality unadjusted.
Why not just run a regression discontinuity on mean score? This is the real threat, and it may work. If a sharp enough jump in acceptance probability exists near some score, the discontinuity gives a clean local effect with fewer assumptions than our instrument. We will run it as described above. If its estimate falls inside our calibrated bounds and the instrument adds no precision, we will report that the simpler design suffices for borderline papers. Our remaining contribution would then be the calibrated sensitivity analysis, which tells the reader how far either estimate can be trusted. The instrument earns its place only if ICLR's fuzzy, area-chair-driven decisions leave the discontinuity with too weak a first stage, or if it reaches compliers away from the cutoff.
Why not just reuse the negative outcome control from the early-arXiving study? Here citations are the outcome, so they cannot also serve as a negative control. Pre-decision citations play the analogous role legitimately, because time order rules out any causal path from acceptance.
Decisive Checks
Three results decide the project. The first is the placebo test: neither the instrument nor its residual-free ablation should predict pre-decision citations conditional on covariates. Failure means the instrument carries quality, and only the bounds survive. The second is the instrument's first stage: the reviewer-residual instrument must shift acceptance strongly, both in the full ICLR sample and within the borderline band. The third is the calibrated breakdown value: if the effect stays positive at the $\gamma$ implied by the pre-decision gap, acceptance has a demonstrable label effect. If it does not, the citation premium of selective venues cannot be separated from reviewer skill with public data, which is itself a result relevant to the case for decoupling dissemination from credentialing. As a secondary check, we replicate on the ICLR portion of PeerRead and test whether the estimated label effect shrinks over time, as CiteTracked's convergence of the oral and poster tiers would predict.
References
- The Impact Market to Save Conference Peer Review: Decoupling Dissemination and Credentialing
- Estimating the Causal Effect of Early ArXiving on Paper Acceptance
- Twin Papers: A Simple Framework of Causal Inference for Citations via Coupling
- CausalCite: A Causal Formulation of Paper Citations
- CiteTracked: A Longitudinal Dataset of Peer Reviews and Citations
- Auditing Fairness under Unobserved Confounding
- Sharp Bounds for Generalized Causal Sensitivity Analysis
- A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications