Change the Funnel, Not Who May Submit: Engineering Peer Review for the Flood

Mikhail L. Arbuzov, Lee Mosbacker, Sisong BeiarXiv 2026 · arXiv

The "flood" of AI research is no longer a metaphor—it’s a logistical nightmare. In 2026, NeurIPS received over 30,000 submissions, requiring roughly 92 researcher-years of expert time to review. Despite this Herculean effort, the process remains notoriously noisy: historical experiments show that if you gave the same set of papers to two different committees, they would disagree on half of the accepted list.

A new paper, "Change the Funnel, Not Who May Submit," argues that our current response—capping submissions or banning new authors—is both ineffective and unfair. Instead, the authors propose a "staged funnel" that uses decision-making engineering to make peer review faster, more accurate, and naturally scalable.

TL;DR

Instead of limiting who can submit to conferences, we should change how we review. By using a staged funnel—automated quality checks, peer-ranked short summaries (gists), and full reviews only for finalists—we can cut expert labor by nearly 50% while significantly increasing the accuracy and reproducibility of the selection process.

What Problem Does This Paper Address?

The current "funnel" for major ML conferences is broken. Every paper gets three full reviews, regardless of quality. This leads to two major issues:

  1. Inefficiency: Three-quarters of expert hours are spent on papers that are ultimately rejected.
  2. Arbitrariness: Near the "accept" line, the outcome is often a coin toss. Current conferences are responding by imposing submission caps or "new author" rules, but the authors' simulations show these limits save little time and unfairly penalize newcomers without improving the quality of the accepted pool.

Methods

The authors used decision-making engineering: they built a simulation model of the review process and calibrated it using data from the famous NeurIPS 2014 and 2021 "consistency experiments" (where duplicate committees reviewed the same papers).

They proposed and simulated a Staged Funnel:

  1. AI Gate: An automated check for basic soundness and clarity (not judging the idea, just the "readiness").
  2. Gist Stage: Authors provide a "gist"—a four-part summary (problem, idea, result, evidence). Every submission brings one "peer" (an author) who ranks sets of 10 gists.
  3. Full Review: Only the top 30% of papers (finalists) proceed to full reviews, where experts check claims and finalize the ranking.
Figure 1: Where the papers of each quality tier (Section 2) end up (computed). Accepted papers are posters, spotlights or orals. The staged funnel’s one weak spot is the 3–10% row, just below the spotlight cut, where it promotes slightly more papers than today’s funnel (Section 6). Its figures rest on an assumed noise for its new kinds of judgment (Sections 2 and 7). Numbers: Table 4, page 15.

Datasets

The paper doesn't introduce a new ML training dataset. Instead, it uses meta-data from historical peer review experiments to calibrate its simulation:

  • NeurIPS 2014/2021 Consistency Experiments: Used to model the "noise" in human judgment.
  • ALMA Observatory Data: Used to validate the effectiveness of "distributed peer review" (proposers ranking one another).
  • NeurIPS 2025/2026 Statistics: Used to set realistic baselines for submission volume and acceptance rates.

Key Insights and Ideas

  • Limits are a blunt instrument: Rules that block authors without a past acceptance (like those proposed for ICLR 2027) mostly just block new researchers. Since the "credential" of a past acceptance is itself the product of a noisy process, it’s a poor filter for future quality.
  • Sorting > Scoring: The model shows that having peers rank sets of summaries is more accurate and faster than having them give individual scores. Ranking cancels out "leniency bias" (where some reviewers are just naturally "nicer" than others).
  • Scaling is built-in: Because every paper brings its own reader (one of its authors), the reviewer pool grows automatically with the submission volume.
  • The "Expert Hour" ROI: Spending two hours on a full review for a paper that is clearly not ready is a waste. The funnel reallocates those hours to the "finalists," where expert scrutiny matters most.

Key Results

The staged funnel outperforms the current system across almost every metric:

  • Efficiency: It requires only 3.4 expert hours per submission, compared to the current 6 hours.
  • Accuracy: It accepts 95.5% of papers in the top 3–10% quality tier, whereas the current system only catches 74%.
  • Prestige: It identifies the "best" papers (top 0.5%) much more reliably, making nearly all of them spotlights or orals (97% vs. 37% today).
  • Reproducibility: A rerun of the staged funnel would accept 72.6% of the same papers, compared to only ~50% in today's system.
Figure 2: What placement costs (computed). Today’s funnel buys placement with more full reviews per paper, labelled by number, and at the top by choosing spotlights and orals by the reviewers’ mean scores, not by the chairs. Today’s 3–10% acceptance at 6 hours plots at about 75, not the text’s 74: a different run, within simulation error. The dip at 20 hours is simulation noise. Table 12 in Appendix D gives the points; Figure 4 (page 14) adds every design simulated.

Limitations and Future Work

  • The AI Gate is speculative: While the model assumes an AI gate as reliable as one human review for basic soundness, this hasn't been fully stress-tested in a live conference setting.
  • Feedback Gap: Papers rejected at the Gist stage would receive a rank but no written feedback. The authors acknowledge this is the "price" authors pay for a more efficient system.
  • Gaming the System: While the authors suggest mechanisms to prevent collusion (like randomizing sets), they admit that "gaming lies outside the model."
  • Conservative Bias: There is a risk that short summaries might favor "safe" or easy-to-explain work over radical, complex breakthroughs. The authors suggest a "running funnel" would need to monitor this drift.

By treating peer review as an engineering problem rather than a tradition to be guarded, this paper offers a rare optimistic path forward for a research community currently drowning in its own success.

Summary updated

Abstract

Every reviewer knows the feeling: more papers each cycle, three full reviews for each of them, and a suspicion that near the accept line the outcome is luck. The numbers bear it out. NeurIPS received 30,709 submissions in 2026, more than three times as many as in 2021, and reviewing them took some 184,000 expert hours, or 92 years of researchers' time. Three-quarters of those hours went to papers that were rejected. And when NeurIPS twice had the same papers reviewed by two committees, the second rejected half of what the first accepted. Conferences are responding by limiting who may submit. We simulated that answer, and an alternative, in a model calibrated to the NeurIPS experiments. The limits save little and select no better, and those based on a past acceptance fall mostly on new authors. The alternative changes how papers are judged, spending full reads only where a decision turns on them. In a staged funnel, an automated quality check first turns back papers that are not ready. The rest are ranked by their value, from short summaries, by readers that each submission brings. Only the finalists get a full read, which checks their claims and settles the final ranking. The staged funnel needs a little over half of today's expert hours, accepts 95.5% of papers in the top 3-10% (against 74% today), and makes nearly every top-0.5% paper a spotlight or oral. And because every paper brings its own reader, the funnel keeps pace with the flood.
BibTeX
@misc{arbuzov2026change,
  author = {Mikhail L. Arbuzov and Lee Mosbacker and Sisong Bei},
  title = {{Change the Funnel, Not Who May Submit: Engineering Peer Review for the Flood}},
  year = {2026},
  eprint = {2611.01076},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01076},
}
Figure