In machine learning, we often treat evaluation metrics like thermometers: objective instruments that measure a single, underlying "truth" about a model’s performance. We expect that if Model A is better than Model B, it should score higher across all reasonable metrics.
However, a growing body of research across saliency, clustering, and classification reveals a fundamental "Metric Paradox." It is mathematically impossible for a single metric to satisfy all the properties we find desirable, and equally impossible for a single model output to optimize all metrics simultaneously. This is not a failure of model design, but a consequence of inherent geometric and axiomatic conflicts.
The Geometry of Conflicting Requirements
The most intuitive example of this impossibility appears in visual saliency—predicting where humans look in an image. Researchers use various metrics like AUC (Area Under the Curve), CC (Correlation Coefficient), and NSS (Normalized Scanpath Saliency) to rank models. Historically, these rankings were inconsistent: a model might win on AUC but lose on CC.
Matthias Kuemmerer, Thomas S. A. Wallis, and Matthias Bethge demonstrated in Consistent Saliency Benchmarking: How One Model Can Win on All Metrics that this inconsistency is baked into the math. Different metrics impose contradictory geometric requirements on the output map. For instance, some metrics reward high-contrast, "peaky" distributions, while others reward smooth, blurred densities.
Because a single static image cannot be both sharp and blurry at the same time, a single "saliency map" is mathematically barred from being optimal for all metrics. The researchers resolved this by treating the model as a probabilistic density and using Bayesian decision theory to derive "optimal maps" tailored to each specific metric. This proved that the model could be superior, even if its static output could not be.
The Axiomatic Triangle of Trade-offs
The problem extends beyond visualization into the very way we validate unsupervised and supervised learning. When comparing two clusterings or two classifications, we want our metrics to satisfy three "common sense" properties:
- Monotonicity: If the prediction gets objectively closer to the ground truth, the score should increase.
- Distance Property: The metric should behave like a physical distance (satisfying the triangle inequality).
- Constant Baseline: A random guess should always result in the same baseline score (usually zero), regardless of class imbalance or cluster size.
In Systematic Analysis of Cluster Similarity Indices: Towards Bias-free Cluster Validation, authors Liudmila Prokhorenkova and Martijn Gösgens prove an impossibility theorem: no single index can satisfy all three. If you want a metric that is unbiased against class sizes, you must sacrifice its status as a proper distance metric.
This same tension is explored in Good Classification Measures and How to Find Them, where the authors show that popular measures like F1-score or Accuracy fail these tests in different ways. This forces a difficult choice: do you prioritize geometric consistency (Distance) or practical robustness (Constant Baseline)?
Why Metrics Disagree on "Support"
This impossibility isn't limited to data points; it even affects how we evaluate logical arguments. In Evaluation of Arguments from Support Relations: Axioms and Semantics, Leila Amgoud and Jonathan Ben-Naim found that when determining the strength of an argument based on its supporters, you cannot simultaneously prioritize the quantity of supporters and the quality of those supporters without violating fundamental logical axioms.
A system that rewards having many weak supporters will eventually conflict with a system that rewards having one very strong supporter. There is no "perfect" middle ground that doesn't break a core logical principle.
Trustworthy Evaluation in Practice
If universal metrics are impossible, how do we rank models? The research suggests two paths:
First, we must move toward Axiomatic Selection. Instead of picking a metric because it is popular (like mAP or NMI), researchers like Hamid Rezatofighi argue in How Trustworthy are the Existing Performance Evaluations for Basic Vision Tasks? that we must choose metrics based on the mathematical properties required by the specific domain. For example, they advocate for the Optimal Sub-Pattern Assignment (OSPA) because it satisfies the triangle inequality, which standard vision metrics often ignore.
Second, we can use Decision-Theoretic Frameworks. Rather than looking for a single "score," Yiding Feng and Wei Tang propose the Informativeness Gap in Measuring Informativeness Gap of (Mis)Calibrated Predictors. This approach evaluates models based on their utility to a decision-maker across various tasks, acknowledging that while a "perfect" metric is impossible, a "most useful" model is still identifiable.
Ultimately, understanding these impossibility theorems changes the goal of benchmarking. We are no longer looking for the "right" metric, but rather the "right set of trade-offs" for the problem at hand.