The "flood" of AI research is no longer a metaphor—it’s a logistical nightmare. In 2026, NeurIPS received over 30,000 submissions, requiring roughly 92 researcher-years of expert time to review. Despite this Herculean effort, the process remains notoriously noisy: historical experiments show that if you gave the same set of papers to two different committees, they would disagree on half of the accepted list.
A new paper, "Change the Funnel, Not Who May Submit," argues that our current response—capping submissions or banning new authors—is both ineffective and unfair. Instead, the authors propose a "staged funnel" that uses decision-making engineering to make peer review faster, more accurate, and naturally scalable.
TL;DR
Instead of limiting who can submit to conferences, we should change how we review. By using a staged funnel—automated quality checks, peer-ranked short summaries (gists), and full reviews only for finalists—we can cut expert labor by nearly 50% while significantly increasing the accuracy and reproducibility of the selection process.
What Problem Does This Paper Address?
The current "funnel" for major ML conferences is broken. Every paper gets three full reviews, regardless of quality. This leads to two major issues:
- Inefficiency: Three-quarters of expert hours are spent on papers that are ultimately rejected.
- Arbitrariness: Near the "accept" line, the outcome is often a coin toss. Current conferences are responding by imposing submission caps or "new author" rules, but the authors' simulations show these limits save little time and unfairly penalize newcomers without improving the quality of the accepted pool.
Methods
The authors used decision-making engineering: they built a simulation model of the review process and calibrated it using data from the famous NeurIPS 2014 and 2021 "consistency experiments" (where duplicate committees reviewed the same papers).
They proposed and simulated a Staged Funnel:
- AI Gate: An automated check for basic soundness and clarity (not judging the idea, just the "readiness").
- Gist Stage: Authors provide a "gist"—a four-part summary (problem, idea, result, evidence). Every submission brings one "peer" (an author) who ranks sets of 10 gists.
- Full Review: Only the top 30% of papers (finalists) proceed to full reviews, where experts check claims and finalize the ranking.
Datasets
The paper doesn't introduce a new ML training dataset. Instead, it uses meta-data from historical peer review experiments to calibrate its simulation:
- NeurIPS 2014/2021 Consistency Experiments: Used to model the "noise" in human judgment.
- ALMA Observatory Data: Used to validate the effectiveness of "distributed peer review" (proposers ranking one another).
- NeurIPS 2025/2026 Statistics: Used to set realistic baselines for submission volume and acceptance rates.
Key Insights and Ideas
- Limits are a blunt instrument: Rules that block authors without a past acceptance (like those proposed for ICLR 2027) mostly just block new researchers. Since the "credential" of a past acceptance is itself the product of a noisy process, it’s a poor filter for future quality.
- Sorting > Scoring: The model shows that having peers rank sets of summaries is more accurate and faster than having them give individual scores. Ranking cancels out "leniency bias" (where some reviewers are just naturally "nicer" than others).
- Scaling is built-in: Because every paper brings its own reader (one of its authors), the reviewer pool grows automatically with the submission volume.
- The "Expert Hour" ROI: Spending two hours on a full review for a paper that is clearly not ready is a waste. The funnel reallocates those hours to the "finalists," where expert scrutiny matters most.
Key Results
The staged funnel outperforms the current system across almost every metric:
- Efficiency: It requires only 3.4 expert hours per submission, compared to the current 6 hours.
- Accuracy: It accepts 95.5% of papers in the top 3–10% quality tier, whereas the current system only catches 74%.
- Prestige: It identifies the "best" papers (top 0.5%) much more reliably, making nearly all of them spotlights or orals (97% vs. 37% today).
- Reproducibility: A rerun of the staged funnel would accept 72.6% of the same papers, compared to only ~50% in today's system.
Limitations and Future Work
- The AI Gate is speculative: While the model assumes an AI gate as reliable as one human review for basic soundness, this hasn't been fully stress-tested in a live conference setting.
- Feedback Gap: Papers rejected at the Gist stage would receive a rank but no written feedback. The authors acknowledge this is the "price" authors pay for a more efficient system.
- Gaming the System: While the authors suggest mechanisms to prevent collusion (like randomizing sets), they admit that "gaming lies outside the model."
- Conservative Bias: There is a risk that short summaries might favor "safe" or easy-to-explain work over radical, complex breakthroughs. The authors suggest a "running funnel" would need to monitor this drift.
By treating peer review as an engineering problem rather than a tradition to be guarded, this paper offers a rare optimistic path forward for a research community currently drowning in its own success.
Summary updated