Axiomatic Validation of Evaluation Metrics

16 concepts13 papers1 proposal

In machine learning, metrics are the ultimate arbiters of progress. We use them to rank models on leaderboards, select hyperparameters, and guide optimization loss functions. However, many of our most trusted metrics are heuristics born out of empirical convenience rather than mathematical necessity. When an evaluation metric lacks formal rigor, it can behave in deeply counter-intuitive ways—silently penalizing good models, rewarding flawed ones, or failing to generalize across datasets.

To address this, a growing body of research advocates for axiomatic metric validation. Instead of evaluating a metric based purely on its popularity or empirical correlation with human judgment, this paradigm treats metrics as mathematical objects that must satisfy a strict set of logical constraints, or axioms. If a metric fails these foundational "unit tests," its rankings cannot be fully trusted.

How Metrics Secretly Break

To understand why axiomatic validation is necessary, we must look at how purely empirical or heuristic metrics fail basic logical sanity checks.

The Triangle Inequality Failure

For any valid geometric distance metric, the shortest path between two points $A$ and $B$ must be a direct line; detouring through a third point $C$ cannot shorten the distance. Mathematically, this is the triangle inequality:

$$d(A, B) \leq d(A, C) + d(C, B)$$

Yet, many proximity and similarity measures used in complex domains violate this rule. For example, Nils Morten Kriege et al. demonstrated that a prominent distance measure used to compare outerplanar graphs (representing molecular structures in drug discovery) failed the triangle inequality, rendering it mathematically invalid for downstream applications like fast similarity search or hierarchical clustering. Similarly, Enrique Fita Sanmartín et al. noted that defining new graph distances often lacks the mathematical rigor needed to guarantee this property, which severely hinders the development of valid novel graph metrics for machine learning.

The Diversity Paradox

Consider measuring the diversity of a generative model's output or a synthetic dataset. Intuitively, if you have a set of unique items and you add exact duplicates to it, the overall diversity of the set should either decrease or stay the same—it should never increase.

However, Mikhail Mironov et al. showed that popular diversity metrics frequently lack mathematical consistency, leading to absurd scenarios where sets containing duplicates are erroneously ranked as more diverse than sets of entirely unique items. They argue that diversity metrics must be bound by fundamental axioms like monotonicity (adding variety increases diversity) and uniqueness (adding duplicates does not).

The Axiomatic Framework as a Software Unit Test

To systematically expose and fix these flaws, researchers construct axiomatic frameworks. This methodology acts exactly like software unit testing: before running a metric on real-world data, it is subjected to synthetic edge cases to see if its mathematical invariants hold.

                  +---------------------------------------+
                  |       Axiomatic "Unit Tests"          |
                  +---------------------------------------+
                  |  [ ] Identity of Indiscernibles       |
Model Outputs --->|  [ ] Symmetry: f(A,B) == f(B,A)       |---> Trustworthy
                  |  [ ] Triangle Inequality              |     Model Rankings
                  |  [ ] Monotonicity                     |
                  +---------------------------------------+

For instance, the authors of Explaining Human Preferences via Metrics for Structured 3D Reconstruction explicitly designed a suite of unit tests checking for identity of indiscernibles, symmetry, the triangle inequality, and monotonicity to evaluate whether 3D reconstruction metrics truly align with human preferences.

In the space of core machine learning tasks, Martijn Gösgens et al. established rigorous axiomatic frameworks to test standard classification and clustering validation indices. By testing metrics against formal properties like the existence of a constant baseline and monotonicity, they proved an "impossibility theorem" showing that no single classification metric can satisfy all desirable properties simultaneously. This forces practitioners to make deliberate, mathematically informed trade-offs rather than blindly choosing a popular metric.

This principle-based approach spans across diverse subfields:

  • Process Mining: Niek Tax et al. proposed five fundamental axioms for "precision" measures (which quantify how well a process model fits an event log), proving through formal counter-examples that every existing industry-standard metric failed basic logical requirements.
  • Knowledge Bases: Badran Raddaoui et al. introduced a framework of formal postulates (such as independence and monotonicity) to evaluate how much an individual formula contributes to conflicts within a contradictory knowledge base.
  • Visual Anomaly Detection (VAD): Standard VAD metrics are often "spatially blind," rewarding models that produce fragmented or redundant detections. To fix this, Nasar Iqbal et al. introduced a formal framework of eight axiomatic properties, including redundancy penalization and distance-aware severity, to validate whether a metric aligns with actual operational requirements.

From Measurement Theory to Axiomatic Design

Validating a metric isn't just about exposing flaws; it provides the blueprint for designing robust alternatives. By anchoring metric design in formal measurement theory—as advocated by Dong Nguyen et al. for NLP data diversity and Yupei Du et al. for fine-tuning instability—researchers ensure that metrics possess construct validity (meaning they accurately capture the intended underlying concept) and reliability (robustness to parameter variations).

When a metric is built from the ground up to satisfy a strict set of axioms, it behaves predictably and reliably:

  • Symmetric Balanced Accuracy (SBA): Developed by Martijn Gösgens et al., this metric was designed specifically to overcome the axiomatic failures discovered in traditional multiclass classification measures.
  • DCScore: To evaluate the diversity of LLM-generated datasets without suffering from the mathematical inconsistencies exposed by previous heuristic measures, Yuchang Zhu et al. introduced a classification-based diversity metric. Because it is mathematically proven to satisfy key axioms like symmetry, monotonicity, and the effective number of samples, it remains robust against data duplication and mode collapse.
  • SAAM-ALARM: Proposed by Nasar Iqbal et al., this metric satisfies spatial coherence constraints, ensuring a trustworthy ranking of visual anomaly detection models.
  • Trustworthy Vision Metrics: Tran Thien Dat Nguyen et al. established a framework requiring object tracking and detection metrics to pass sanity tests with pre-determined rankings, ensuring mathematical consistency (identity, symmetry, and the triangle inequality) alongside reliability against parameter tuning.

By shifting the paradigm from "empirical popularity" to "formal property verification," the machine learning community is building a mathematical safety net. Axiomatic validation ensures that when a model wins on a leaderboard, it wins because it is genuinely superior, not because it successfully exploited a hidden mathematical loophole in an unvalidated metric.

Key Foundational Work

The imprecisions of precision measures in process mining (2017) — contributed 1 concept elements to this direction.
A note on block-and-bridge preserving maximum common subgraph algorithms for outerplanar graphs (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_axiomatic_metric_validation_f87e2948,
  title = {axiomatic metric validation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/axiomatic-metric-validation/rdr_f87e2948e1a64870b7333408af52546c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_axiomatic_metric_validation_f87e2948_v1,
  title = {axiomatic metric validation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/axiomatic-metric-validation/rdr_f87e2948e1a64870b7333408af52546c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake