When evaluating machine learning models, natural language processing systems, or information retrieval pipelines, we routinely reduce complex behaviors to a single number. We score a model's performance across hundreds of test cases, compute the arithmetic mean, and declare the model with the highest average the winner of the leaderboard.
However, this ubiquitous practice relies on a hidden, foundational assumption: that the numbers we are averaging live on an interval scale. An interval scale guarantees that the numerical distance between any two points reflects a consistent, uniform change in the underlying quality being measured (e.g., the difference between 0.1 and 0.2 is the exact same magnitude of improvement as the difference between 0.8 and 0.9).
When this assumption is violated, our evaluation pipelines break down. A growing body of research warns against the "parametric-ordinal mismatch"—the practice of applying parametric statistical operations, like calculating means and variances, to metrics that are fundamentally ordinal.
The Hierarchy of Measurement Scales
To understand why standard evaluations can mislead, we must look to formal measurement theory. As Peter A. Flach outlines in a foundational analysis of machine learning performance evaluation, metrics are not just arbitrary numbers; they belong to distinct scale types—nominal, ordinal, interval, or ratio. Each scale type strictly dictates which mathematical operations are coherent.
[Nominal] --> Identifies categories (No ordering)
[Ordinal] --> Preserves order, but distances are meaningless (e.g., 1st, 2nd, 3rd)
[Interval] --> Uniform distances; addition/subtraction are valid (e.g., Celsius)
[Ratio] --> True zero point; multiplication/division are valid (e.g., Kelvin, Length)
If an evaluation metric is merely ordinal, it can reliably tell us whether Model A is better than Model B on a specific task, but it cannot quantify by how much. Computing an arithmetic mean across multiple tasks or queries on an ordinal scale is mathematically invalid. It creates an illusion of quantitative precision, masking the fact that the resulting "average" may not correspond to any consistent physical or psychological reality.
Why Common Metrics Fail the Interval Test
This scale mismatch is pervasive across both human and automated evaluation frameworks.
Human Ratings and Likert Scales
In human evaluation, researchers frequently rely on discrete Likert-type scales (e.g., 1 to 5 stars) to assess text quality, translation fluency, or clinical acceptability. However, as Belz et al. and Baek et al. point out, these discrete categories are inherently ordinal. A judge's cognitive leap from 2 to 3 stars does not necessarily represent the same increment in quality as a leap from 4 to 5 stars. Because the intervals are non-uniform, treating these scores as continuous numbers in parametric tests can distort statistical significance and lead to inaccurate conclusions.
Automated Benchmarks and Leaderboards
The problem runs even deeper in automated metrics. In information retrieval and search evaluation, standard metrics like Average Precision (AP) and normalized Discounted Cumulative Gain (nDCG) are widely used to aggregate system performance across diverse search queries. Yet, formal measurement-theoretic analysis reveals that these metrics do not operate on an interval scale.
As summarized by Fang et al., because the numerical distance between scores does not represent a consistent change in search quality across different queries, calculating the arithmetic mean of these scores is mathematically questionable.
This vulnerability is vividly illustrated on competitive leaderboards. For instance, Lin et al. evaluated the MS MARCO document ranking leaderboard and demonstrated that the standard metric, Mean Reciprocal Rank (MRR@100), conflates two entirely different types of model improvements: finding more answers versus ranking known answers higher. Because the metric forces these distinct behaviors onto a non-interval scale, traditional significance frameworks struggle to confirm whether "State of the Art" (SOTA) jumps actually represent true, reliable improvements in system quality.
Solution 1: Shifting to Continuous Measurement Scales
If discrete scales trap us in ordinal limitations, how do we capture true interval data? One elegant solution is to change how we collect human judgments by replacing discrete boxes with continuous Visual Analogue Scales (VAS)—essentially, continuous sliders.
Discrete Scale: [ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ] (Ordinal Mismatch)
Continuous Scale: |-------------------|-------------------| (Interval-like)
Slider
Belz et al. demonstrated that continuous scales are not only more sensitive to subtle differences between AI systems, but they are also strongly preferred by human evaluators, who find rigid discrete boxes frustratingly restrictive.
The true power of continuous scales, however, lies in their mathematical tractability. When human judges use discrete scales, the resulting ordinal data prevents researchers from applying standard variance-based adjustments to account for individual judge biases (such as a judge who is consistently harsh or lenient).
By switching to continuous sliders, Graham et al. showed that the data behaves sufficiently like an interval scale to permit rigorous statistical quality control. This mathematical freedom allows researchers to normalize individual judge biases and filter out unreliable annotators. Ultimately, this shift yields massive boosts in annotator agreement without requiring an expensive expert "gold standard," transforming subjective human perception into a mathematically sound metric.
Solution 2: Nonparametric Analysis and Axiomatic Design
When changing the data collection method is impossible and we must work with ordinal metrics, two theoretical alternatives exist: changing the statistical tools, or redesigning the metrics from first principles.
Nonparametric Frameworks
If data is strictly ordinal or highly skewed, we must abandon parametric tests like ANOVA or t-tests, which incorrectly assume normal distributions and equal intervals. Instead, we can utilize robust, rank-based nonparametric methods. For example, Noguchi et al. introduced the nparLD framework, which provides a mathematically sound approach for analyzing longitudinal and ordinal data in complex factorial experiments, ensuring that conclusions remain valid even when means cannot be trusted.
Axiomatic Metric Design
Rather than fixing the statistics after the data is collected, we can design metrics that are guaranteed to satisfy formal measurement properties from the start. This is the core of axiomatic thinking in evaluation.
As Amigó et al. argue, many widely used evaluation metrics are developed ad-hoc to satisfy immediate empirical needs, frequently failing fundamental measurement constraints. By treating evaluation metrics as formal measurements that must satisfy strict mathematical axioms, researchers can explicitly design metrics that possess interval scale properties. This foundational rigor ensures that when we average scores across a benchmark, the math genuinely mirrors reality—moving the field away from fragile leaderboard victories and toward reproducible scientific knowledge.
Key Foundational Work
nparLD: AnRSoftware Package for the Nonparametric Analysis of Longitudinal Data in Factorial Experiments (2012) — contributed 1 concept elements to this direction.
Discrete vs. Continuous Rating Scales for Language Evaluation in NLP (2011) — contributed 1 concept elements to this direction.