TL;DR
The Isotonic Mechanism is a new, incentive-compatible way to fix the "noisy review" problem at massive AI conferences by asking authors to rank their own submissions. By using these rankings to calibrate reviewer scores through isotonic regression, conferences can significantly improve score accuracy without encouraging authors to lie.
What Problem Does This Paper Address?
The "reviewer crisis" in machine learning and AI is well-documented. As conferences like NeurIPS and ICML explode in size—some seeing nearly 50% growth in a single year—the pool of qualified reviewers simply cannot keep up. This leads to "noisy" reviews: scores that are often arbitrary or inconsistent. A famous 2014 experiment at NeurIPS showed that if the same papers were reviewed by a different committee, 57% of the accepted papers would have been rejected.
While reviewers are often overworked or inexperienced, authors are arguably the most qualified people to assess the quality of their own work. However, you can't just ask an author for a score; they have every incentive to report a "10/10" to get accepted. This paper tackles the challenge of truthful elicitation: how can we design a system where authors are mathematically incentivized to tell the truth about their papers' relative quality?
Methods
The authors propose the Isotonic Mechanism. Here is how it works:
- Ranking: Before seeing their review scores, authors with multiple submissions must rank their papers from best to worst.
- Calibration: The conference collects noisy raw scores from traditional peer reviewers.
- Optimization: The mechanism "calibrates" these raw scores by solving a convex optimization problem (specifically, isotonic regression). It finds the set of adjusted scores that are as close as possible to the raw scores while strictly obeying the author’s provided ranking.
The core of the paper is a proof that this mechanism is truthful. If an author’s goal is to maximize their "utility" (modeled as a convex function where higher scores are increasingly valuable, such as for winning a "Best Paper" award), their best strategy is to provide their honest internal ranking.
Datasets
The paper references real-world implementation and experimental data:
- ICML 2023: The mechanism was used to collect 1,342 rankings corresponding to 2,592 submissions. Analysis showed that the adjusted scores were significantly more accurate than raw scores.
- ICML 2024 & 2025 / NeurIPS 2025: The paper notes continued experimental implementation at these major venues.
- ICLR 2020: The authors used submission data from this conference to demonstrate the prevalence of authors with multiple submissions (e.g., one individual submitted 32 papers).
Key Insights and Ideas
- Authors as Experts: The paper shifts the perspective of authors from "biased agents to be ignored" to "private experts to be utilized."
- Convex Utility is Key: The proof of truthfulness relies on the idea that authors care more about their "top" papers. Because isotonic regression "shrinks" scores together when a ranking is incorrect, an author who lies actually lowers the expected utility of their best papers.
- The Optimality of Pairwise Comparisons: The authors prove a "necessary condition" for truthfulness: any mechanism that elicits information more detailed than a simple ranking (like asking for exact scores) will inevitably encourage lying. The Isotonic Mechanism is the "most fine-grained" truthful mechanism possible because it is based on pairwise comparisons.
- Statistical Gains: The mechanism is most effective exactly where the current system fails: when noise is high and authors have many submissions. Mathematically, it reduces the estimation error from a factor of $n$ (number of papers) to $n^{1/3}$.
Key Results
- Truthfulness (Theorem 1.1): Under the assumption of nondecreasing convex utility, reporting the true ranking is the optimal strategy for authors.
- Accuracy Boost (Proposition 1): The adjusted scores are guaranteed to be more accurate (in terms of squared error) than the raw reviewer scores.
- Efficiency: The mechanism uses the "Pool Adjacent Violators Algorithm" (PAVA), which is computationally very efficient, making it practical for conferences with tens of thousands of papers.
- Robustness: The mechanism remains truthful even if authors only have partial information or "noisy" internal knowledge of their own papers' quality.
Limitations and Future Work
- Strategic "Trash" Submissions: A potential loophole is authors deliberately submitting low-quality papers to "boost" the scores of their good papers through the ranking process. The authors suggest preliminary screening or "negative utility" for very poor papers as a fix.
- Collaborative Authorship: Most papers have multiple authors. The paper provides an algorithm (Algorithm 1) to partition papers into disjoint groups based on authorship, but managing conflicting rankings between co-authors remains a complex social and technical challenge.
- Non-Convex Utility: If an author only cares about getting any paper accepted (a "flat" utility), the isotonic partition is no longer strictly truthful. Future work is needed to design rewards that maintain convexity in author incentives.
- Human Error: Authors might inadvertently provide a wrong ranking due to their own uncertainty. The paper explores "coarse rankings" as a more robust alternative for these cases.
Summary updated