A Defense of Academic Gatekeeping

Martin WeissLacuna 2026

I was the tech chair for ICCV 2021, a conference with over 6,000 submissions. I watched it attempt to combat fraud, plagiarism, and cheating. It was not a fair fight. There were no effective mechanisms for detecting and punishing authors who attempted to gain an unfair advantage.

We rely on academic gatekeepers to help us find truth. Truthseekers, which all academics should strive to be, need experienced, trustworthy, and ethical gatekeepers. That job used to be easier - we could rely on formality of language, typesetting conventions, and a stable and slow-changing community to help separate reliable from unreliable. But now they also need to keep out the Slopbarian hordes.

And so the gatekeepers must innovate. To that end, we need mechanisms that:

  1. Check whether someone understands the work that they are submitting
  2. Hold people accountable for what they have submitted
  3. Reward gatekeepers (reviewers, ACs, PCs) as much as authors
Submissions are outrunning human attention: NeurIPS, ICML and ICLR submissions by year Open in a new tab

This September, Thomas Dietterich, who chairs arXiv's computer science editorial committee, wrote that arXiv is seeing "a new trend in submissions … Authors submitting papers whose contents they likely do not understand." arXiv requires a human author to take responsibility for each submission, "but obviously an author cannot take responsibility (or claim credit) for a result that they do not understand." He floated a few options: screen authors by publication record, oral exams, or create a separate venue for AI-written results with no human author.

A few days later, Nihar Shah, an editor-in-chief of TMLR, wrote up the results of asking the authors of 10 submissions slated for desk rejection to meet with him and answer questions about their own papers. Of the seven who showed up, three couldn't answer basic questions about their paper (all three were solo-authored), three could talk about the high-level ideas but struggled with the technical details, and one answered everything, though Shah found a major flaw in that paper anyway. TMLR's desk-rejection rate has gone from about 6% in 2023 to about 53% now.

Science compounds through understanding, so it is bad for science if AI produces results faster than humans can understand them.

The scientific community relies on peer review to decide which results are worth building on, and it runs on the scarcest resource in research: expert human attention. A paper that no one can explain is worse than useless because it carries a higher probability of fooling others into wasting resources building on incorrect beliefs. I've never felt more betrayed than when I lost a month building on a "reputable lab"'s bad slop (which they lied to an AC about). Slight digression.

Isn't this just gatekeeping?

Replies to Thomas and Nihar called this gatekeeping, saying that their responsibility is to evaluate claims, not to certify who is entitled to make them. I agree with them in principle, but modern times require modern solutions. These gatekeepers are trying to hold the line, to do the community a service, and they are being overwhelmed. Reviewers are the scarce resource.

I see it as a signal-to-noise problem. When plausible-looking noise grows faster than the attention available to process it, we need better filters. The easy filter is who is submitting: track record, affiliation, seniority. But well-credentialed authors also submit papers they don't understand.

Rules per year at NeurIPS, ICML and ICLR. Grey is about the paper: page limit, anonymity, dual submission, missing checklist. Red is about you: didn't review, reviewed badly, LLM misuse, collusion, fake conflicts. Chart: Martin Weiss.

Venues are actively experimenting with filters. ICML 2027 just announced a 20 paper cap on author submissions. That is a filter on who is submitting. ICML is also trying a two-stage review as a cheaper filter per paper. An area chair can propose a first-stage rejection for a paper that would clearly not be accepted, a senior area chair must confirm it, and when in doubt it goes to full review. Journal editors have always done this, and I applaud ICML trying this policy.

Three possible ways to improve the scientific process

We need to make it cheaper to filter out poor quality papers, to make it more expensive to submit poor quality papers, and to increase the reviewer attention we have available. Here are a few proposals.

1. Random understanding audits. A random 5% of all submissions get a 15-minute video call with a peer reviewer or chair. One author, selected by the authors, must explain the main claim and answer follow-up questions. The bar to pass is basic fluency with your own paper. Polished delivery does not matter - you just need to be able to explain the basics of your work. If you can't, it's desk rejected. Optionally audited authors could request a second call, and if they fail again, we could record this (see below). \ \ The auditor doesn't need to understand the paper, just well enough to tell whether the author does. Anyone who has walked up to a poster at a conference knows the difference. \ \ For 30,000 submissions, a 5% draw is 1,500 calls. Even with 15 minutes for the auditor to skim the paper first, that's about 750 hours. Reviewing those same papers, at roughly four reviews each and an hour or two per review is 100,000+ hours.

Martin Weiss on X: Proposal for ML conferences: random understanding audits Open in a new tab

2. Hold people accountable. We need more significant penalties for certain kinds of rule-breaking, and a way to flag author profiles that carries across venues. This could enable future conferences and journals to have policies that use these. I know this is a controversial opinion - but having seen the scope of plagiarism and attempts to unfairly cheat the system, it was shocking to me that we have no institutional memory at all to stop bad actors.

Tax authorities only audit a fraction of returns, because everyone just needs to know an audit is possible. Deterrence research shows the same pattern for crime, where the certainty of getting caught deters far more than the severity of the punishment (NIJ, summarizing Nagin 2013).

3. Reward the gatekeepers. Right now we draft reviewers through mandates. ICLR and ICML both require every submission to put forward one of its authors as a reviewer, and at both venues, reviewers who fail to deliver their reviews may get their own papers desk-rejected. Reviewing is now a toll you pay to get published, when it needs to be a celebrated and sought-after skill. A credential, one that counts in hiring and tenure the way papers do. Area chairs could rate each review, reviewers could choose to sign their best ones and publish them, and it adds up to a meaningful and public record of contributions.

None of this requires an AI slowdown. Human Science just needs to ramp up fast.

Gatekeeping gets a bad reputation when it keeps out the wrong people. But we need these gates to separate truth from fiction. And we need to focus hard on building better ones.

It's going to be really important that we solved this.

Figure

Lacuna is a research map of machine learning by Tiptree Systems: papers linked to the research directions, authors, venues and institutions behind them, with summaries that link back to their sources. Free to read, no account needed.

The Lacuna paper is by Martin Weiss, Miles Q. Li, Alejandro H. Artiles, Yacine Mkhinini, Chris Pal, Hugo Larochelle and Nasim Rahaman.

Updated dailyFollow Lacuna on X