You Are the Best Reviewer of Your Own Papers: The Isotonic Mechanism

Weijie Su

TL;DR

The Isotonic Mechanism is a new, incentive-compatible way to fix the "noisy review" problem at massive AI conferences by asking authors to rank their own submissions. By using these rankings to calibrate reviewer scores through isotonic regression, conferences can significantly improve score accuracy without encouraging authors to lie.

What Problem Does This Paper Address?

The "reviewer crisis" in machine learning and AI is well-documented. As conferences like NeurIPS and ICML explode in size—some seeing nearly 50% growth in a single year—the pool of qualified reviewers simply cannot keep up. This leads to "noisy" reviews: scores that are often arbitrary or inconsistent. A famous 2014 experiment at NeurIPS showed that if the same papers were reviewed by a different committee, 57% of the accepted papers would have been rejected.

While reviewers are often overworked or inexperienced, authors are arguably the most qualified people to assess the quality of their own work. However, you can't just ask an author for a score; they have every incentive to report a "10/10" to get accepted. This paper tackles the challenge of truthful elicitation: how can we design a system where authors are mathematically incentivized to tell the truth about their papers' relative quality?

Methods

The authors propose the Isotonic Mechanism. Here is how it works:

  1. Ranking: Before seeing their review scores, authors with multiple submissions must rank their papers from best to worst.
  2. Calibration: The conference collects noisy raw scores from traditional peer reviewers.
  3. Optimization: The mechanism "calibrates" these raw scores by solving a convex optimization problem (specifically, isotonic regression). It finds the set of adjusted scores that are as close as possible to the raw scores while strictly obeying the author’s provided ranking.

The core of the paper is a proof that this mechanism is truthful. If an author’s goal is to maximize their "utility" (modeled as a convex function where higher scores are increasingly valuable, such as for winning a "Best Paper" award), their best strategy is to provide their honest internal ranking.

Datasets

The paper references real-world implementation and experimental data:

  • ICML 2023: The mechanism was used to collect 1,342 rankings corresponding to 2,592 submissions. Analysis showed that the adjusted scores were significantly more accurate than raw scores.
  • ICML 2024 & 2025 / NeurIPS 2025: The paper notes continued experimental implementation at these major venues.
  • ICLR 2020: The authors used submission data from this conference to demonstrate the prevalence of authors with multiple submissions (e.g., one individual submitted 32 papers).

Key Insights and Ideas

  • Authors as Experts: The paper shifts the perspective of authors from "biased agents to be ignored" to "private experts to be utilized."
  • Convex Utility is Key: The proof of truthfulness relies on the idea that authors care more about their "top" papers. Because isotonic regression "shrinks" scores together when a ranking is incorrect, an author who lies actually lowers the expected utility of their best papers.
  • The Optimality of Pairwise Comparisons: The authors prove a "necessary condition" for truthfulness: any mechanism that elicits information more detailed than a simple ranking (like asking for exact scores) will inevitably encourage lying. The Isotonic Mechanism is the "most fine-grained" truthful mechanism possible because it is based on pairwise comparisons.
  • Statistical Gains: The mechanism is most effective exactly where the current system fails: when noise is high and authors have many submissions. Mathematically, it reduces the estimation error from a factor of $n$ (number of papers) to $n^{1/3}$.

Key Results

  • Truthfulness (Theorem 1.1): Under the assumption of nondecreasing convex utility, reporting the true ranking is the optimal strategy for authors.
  • Accuracy Boost (Proposition 1): The adjusted scores are guaranteed to be more accurate (in terms of squared error) than the raw reviewer scores.
  • Efficiency: The mechanism uses the "Pool Adjacent Violators Algorithm" (PAVA), which is computationally very efficient, making it practical for conferences with tens of thousands of papers.
  • Robustness: The mechanism remains truthful even if authors only have partial information or "noisy" internal knowledge of their own papers' quality.

Limitations and Future Work

  • Strategic "Trash" Submissions: A potential loophole is authors deliberately submitting low-quality papers to "boost" the scores of their good papers through the ranking process. The authors suggest preliminary screening or "negative utility" for very poor papers as a fix.
  • Collaborative Authorship: Most papers have multiple authors. The paper provides an algorithm (Algorithm 1) to partition papers into disjoint groups based on authorship, but managing conflicting rankings between co-authors remains a complex social and technical challenge.
  • Non-Convex Utility: If an author only cares about getting any paper accepted (a "flat" utility), the isotonic partition is no longer strictly truthful. Future work is needed to design rewards that maintain convexity in author incentives.
  • Human Error: Authors might inadvertently provide a wrong ranking due to their own uncertainty. The paper explores "coarse rankings" as a more robust alternative for these cases.

Summary updated

Abstract

Machine learning (ML) and artificial intelligence (AI) conferences including NeurIPS is Neural Information Processing Systems and International Conference on Machine Learning have experienced a significant decline in peer review quality in recent years. To address this growing challenge, we introduce the isotonic mechanism, a computationally efficient approach to enhancing the accuracy of noisy review scores by incorporating authors’ private assessments of their submissions. Under this mechanism, authors with multiple submissions are required to rank their papers in descending order of perceived quality. Subsequently, the raw review scores are calibrated based on this ranking to produce adjusted scores. We prove that authors are incentivized to truthfully report their rankings because doing so maximizes their expected utility, modeled as an additive convex function over the adjusted scores. Moreover, the adjusted scores are shown to be more accurate than the raw scores, with improvements being particularly significant when the noise level is high and the author has many submissions—a scenario increasingly prevalent at large-scale ML/AI conferences. We further investigate whether submission quality information beyond a simple ranking can be truthfully elicited from authors. We establish that a necessary condition for truthful elicitation is that the mechanism be based on pairwise comparisons of the author’s submissions. This result underscores the optimality of the isotonic mechanism because it elicits the most fine-grained truthful information among all mechanisms we consider. We then present several extensions, including a demonstration that the mechanism maintains truthfulness even when authors have only partial rather than complete information about their submission quality. Finally, we discuss future research directions, focusing on the practical implementation of the mechanism and the further development of a theoretical framework inspired by our mechanism. Funding: This research was supported by the National Science Foundation [Grants CCF-1934876 and CAREER DMS-1847415] and an Alfred Sloan Research Fellowship. Supplemental Material: The online appendix is available at https://doi.org/10.1287/opre.2022.0622.
BibTeX
@inproceedings{su2022you,
  author = {Weijie Su},
  title = {{You Are the Best Reviewer of Your Own Papers: The Isotonic Mechanism}},
  booktitle = {Operations Research},
  year = {2022},
  doi = {10.48550/arxiv.2206.08149},
}
Figure