Introduction
Peer review is the cornerstone of academic publishing, yet the process of aggregating diverse reviewer opinions into a single decision remains fraught with challenges. One of the most significant hurdles is commensuration bias—the reality that different reviewers weight evaluation criteria (such as novelty, technical rigor, or clarity) differently. While one reviewer might prioritize a groundbreaking idea despite minor technical flaws, another might view those same flaws as disqualifying.
In "No Agreement Without Loss: Learning and Social Choice in Peer Review," the authors provide a rigorous theoretical critique of recent machine learning-based attempts to automate the resolution of this bias. Specifically, they investigate the $L(p, q)$ framework—a method recently adopted by major venues like AAAI—and demonstrate that it faces fundamental mathematical trade-offs that limit its effectiveness in real-world scenarios.
The Problem of Commensuration Bias
To combat the subjectivity of how reviewers aggregate their own scores, researchers previously proposed a framework to "learn" a fair, global mapping from criteria scores to overall recommendations. By minimizing a specific loss function, $L(p, q)$, the system attempts to find a monotonic function that best fits the collective wisdom of the reviewer pool.
Previous influential work suggested that the $L(1, 1)$ loss—which effectively results in taking the median—was the optimal choice because it satisfied the social choice axiom of strategy-proofness (meaning reviewers cannot manipulate the outcome by lying about their preferences). However, this paper investigates whether this recommendation holds up when we demand other essential properties, such as consistency and stability.
Theoretical Framework and Axioms
The authors evaluate aggregation methods through the lens of Social Choice Theory, introducing a new axiom called Consistency. Consistency requires that if Paper A is strictly better than Paper B across all reviewers' evaluations (dominance), the final aggregated score for Paper A must be strictly higher than that of Paper B.
Through mathematical proofs and constructed counter-examples, the authors test various configurations of the $L(p, q)$ framework against three desired properties:
- Consistency: Ranking clearly superior papers higher.
- Stability (Continuity): Ensuring that small changes in reviewer scores do not lead to massive jumps in the final aggregate score.
- Strategy-proofness: Preventing malicious or tactical voting from distorting the results.
Key Insights: The "No Free Lunch" of Peer Review
The Median is Insensitive
The authors argue that the $L(1, 1)$ (median) approach is "too crude." While it is strategy-proof, it frequently fails the Consistency axiom. Because the median only looks at the middle value, it can assign the same final score to two papers even if the vast majority of reviewers clearly prefer one over the other. In contrast, they show that $L(p, p)$ for $p > 1$ (similar to the mean) satisfies consistency but loses other benefits.
The Impossibility Theorem
A central contribution of the paper is Proposition 3.2, which establishes a fundamental trade-off: no aggregation method can simultaneously be continuous, consistent, and strategy-proof. Designers of peer-review systems must essentially "pick their poison," choosing which of these three vital properties to sacrifice.
The Fragility of the Objectivity Assumption
Previous defenses of the $L(p, q)$ framework relied on the "Objectivity Assumption"—the idea that all reviewers agree on the base criteria scores (e.g., everyone agrees a paper's novelty is a 7) and only differ on how to weight them. The authors demonstrate that when this unrealistic assumption is removed (the "Non-Hidden Scores" setting), the framework’s benefits vanish.
Most strikingly, Theorem 4.5 proves that without this assumption, the $L(p, q)$ method becomes discontinuous. This means the system is inherently unstable; an infinitesimal change in a single reviewer's score could cause a paper's final ranking to swing wildly, making the system vulnerable to accidental errors or intentional gaming.
Conclusion and Future Directions
The findings in "No Agreement Without Loss" serve as a cautionary tale for the application of automated aggregation in high-stakes academic environments. The authors show that the current "perfect" solutions are only perfect under narrow, unrealistic assumptions.
While the results are largely negative, they point toward new research directions, such as the Linear Functional Setting, which may recover stability and consistency at the cost of "consensus." Future work must also address miscalibration bias—the "assassins vs. zealots" problem—and the reality that reviewers only see a small subset of total submissions. Ultimately, the paper suggests that we cannot calculate our way out of the fundamental tensions of social choice; we must instead make conscious, transparent trade-offs in how we define fairness in peer review.
Summary updated