Many ordinal prediction tasks, such as diabetic retinopathy grading, age groups and star ratings, are scored with Quadratic Weighted Kappa (QWK). QWK measures agreement between predicted and true grades, penalizes each disagreement by the squared distance between grades, and subtracts the agreement expected by chance. It is non-differentiable, and its chance term depends on the whole set of predictions. Two literatures respond to this in different ways. One builds smooth, differentiable stand-ins for such metrics and trains through them. The other adds ordinal-consistency constraints, meaning constraints that keep predicted probabilities ordered, for example ensuring that the probability of "grade above 2" never exceeds the probability of "grade above 1."
This proposal starts from a short derivation that we think changes what both literatures should be doing. At the population level, the QWK-maximizing classifier takes a simple form. Take the model's estimate of the expected grade for an input, move it away from the dataset's average grade by a factor of one over the best achievable kappa, and round to the nearest grade. If this is right, training through a soft kappa loss is unnecessary for consistency. Any loss that estimates the conditional mean well suffices, followed by a one-parameter decision rule. The real question becomes which ordinal model estimates that mean best in the tails, where the stretch factor magnifies its errors.
Core Claim
Let $K$ grades be indexed $0,\dots,K-1$. Let $\mu(x)=\mathbb{E}[Y\mid x]$ be the conditional mean grade, $m=\mathbb{E}[Y]$ the overall mean grade, and $\kappa^*>0$ the optimal QWK. QWK is one minus a ratio. The numerator is the expected squared disagreement $\mathbb{E}[(Y-h(X))^2]$. The denominator is $\sum_k r_k\, c(k)$, where $r_k$ is the fraction of predictions equal to $k$ and $c(k)=\mathbb{E}[(Y-k)^2]$ under the true label marginal.
The denominator is linear in the prediction frequencies. We can therefore apply Dinkelbach's method, which minimizes a ratio by minimizing a numerator-minus-$t$-times-denominator objective at the optimal ratio $t^*=1-\kappa^*$. This objective decomposes pointwise, and each input gets a convex quadratic in $k$. Its minimizer is
$$ h^*(x)=\mathrm{clip}_{[0,K-1]}\,\mathrm{round}\!\left(m+\frac{\mu(x)-m}{\kappa^*}\right). $$
Three consequences follow. First, QWK depends on the predictive distribution only through its mean. The spread and shape of a model's output distribution are irrelevant to the optimal decision once the mean is fixed. Second, the familiar recipe of fitting a regressor and then grid-searching $K-1$ cutoffs has a justification, but it over-parameterizes the decision. The optimal cutoffs are equally spaced in $\mu$ with spacing $\kappa^*$, centered on $m$, which leaves one free scalar rather than $K-1$. Third, under the label skew described in The Problem with Kappa, which shows that chance-corrected agreement degrades under prevalence and bias skew, $\kappa^*$ shrinks. The stretch $1/\kappa^*$ then grows and amplifies any error in $\mu(x)$. This is where ordinal consistency regularization should matter: not for shaping distributions, but for producing low-variance, well-ordered estimates of the mean at rare extreme grades.
The contrast case sharpens this. For linearly weighted kappa, the same derivation gives a pointwise objective $\mathbb{E}[|Y-k|\mid x]-t\,\mathbb{E}|Y-k|$. That objective depends on the full conditional distribution, not only its mean. The theory therefore predicts a clean crossover. Distribution-shaping ordinal losses should beat plain regression on linear kappa. They should beat it on QWK only insofar as they estimate the mean better.
Relation to Prior Work
Several papers supply the machinery that the derivation reuses. Consistency Conditions for Differentiable Surrogate Losses shows that a smooth surrogate is consistent exactly when all distributions sharing a surrogate minimizer share the same target decision, a condition it calls strong indirect elicitation. It then builds a one-dimensional consistent ordinal surrogate. Our result is an elicitation statement of the same kind: QWK's decision is a function of a one-dimensional property, the mean, plus one global scalar. That paper targets decomposable ordinal risk. We extend the style of argument to a metric whose normalizer couples all predictions. On the Consistency of Ordinal Regression Methods proves consistency of all-threshold surrogates for squared error but explicitly leaves agreement-adjusted metrics open. Adversarial Surrogate Losses for Ordinal Regression derives a consistent convex surrogate for absolute error only, and does so for kernel models. Neither paper covers the chance-corrected case we derive.
A second group motivates the question empirically. Unimodal Probability Distributions for Deep Ordinal Classification makes network outputs a single-parameter Poisson or Binomial distribution to better serve QWK. It reports that on the large retinopathy dataset the advantage over cross-entropy narrows. Our theory offers an explanation. With enough data, cross-entropy already estimates the mean well, and unimodality does not affect the QWK decision beyond the mean. The known flaw of that design, variance tied to the mean, is under our claim harmless for QWK but harmful for linear kappa. Unimodal-Concentrated Loss learns adaptive unimodal label distributions and reports mean absolute error (MAE) 1.86 on MORPH. A Simple Log-based Loss Function for Ordinal Text Classification penalizes distant errors via a distance power law and benchmarks against a Weighted Kappa Loss. Both are candidate mean estimators in our study rather than competitors to the decision rule.
A third group supplies the ordinal-consistency tools we use to estimate $\mu$. Ordinal Regression by Extended Binary Classification reduces an ordinal task to ordered binary questions, and Estimating Class Membership Probabilities using Classifier Learners enforces monotone cumulative probabilities. Together they give a head whose implied mean is a valid expectation. By contrast, crossing cumulatives imply negative probability mass and a corrupted mean. Order Regularization on Ordinal Loss keeps thresholds ordered in deep networks, though the file notes it degrades in high-dimensional feature spaces.
On the metric-relaxation side, sigmoidF1 relaxes confusion-matrix counts with sigmoids to train directly on F1. Its batch-level dependence and steepness trade-off are exactly the costs our plug-in route avoids.
Research Program
The work has three parts.
First, we state and prove the theorem, covering finite-sample deviations. We bound the QWK loss of the plug-in rule in terms of the mean-estimation error $\mathbb{E}|\hat\mu(X)-\mu(X)|$ and the error in the estimated $\kappa^*$. The bound should scale like $1/\kappa^*$, since the stretch multiplies mean errors by that factor. We derive the linear-kappa counterpart and show that it requires conditional quantile-like information rather than the mean.
Second, we build the estimator that the theory calls for. It is a deep network with a non-crossing cumulative ordinal head, trained with a proper scoring rule so that its implied mean is consistent. We compare this head against a plain squared-error head and against cross-entropy's expected value. The decision layer is the stretch rule. We estimate $m$ from training labels and find $\kappa^*$ on validation data by a fixed-point iteration: apply the rule with the current kappa, recompute QWK, and repeat.
Third, we test the theory's predictions on synthetic ordinal data, where $\mu(x)$ is known exactly. There we check that the stretch rule attains the brute-force QWK optimum, that predictive variance is irrelevant at fixed mean, and that the $1/\kappa^*$ error amplification appears as label skew increases.
Baselines
The naive reference is a cross-entropy classifier with argmax decisions. It is what most pipelines ship, and it ignores both order and the metric.
The strongest practical competitor is regression-plus-free-cutoffs. This means a squared-error regressor, or the expected value from cross-entropy, with $K-1$ cutoffs tuned on validation to maximize QWK. It is the common recipe for retinopathy grading, and our theorem says it is consistent. We will run it with the same backbone, training epochs and validation split as our method, using the same search budget for the cutoffs as for our single scalar. Beating it means higher test QWK on small or skewed datasets. The mechanism we expect is lower variance from one decision parameter instead of $K-1$, plus a better tail mean from the ordinal head.
We also run a direct soft-QWK training loss: the Weighted Kappa Loss that A Simple Log-based Loss Function for Ordinal Text Classification uses as a baseline, with batch sizes swept. We run the Binomial head of Unimodal Probability Distributions for Deep Ordinal Classification, the loss of Unimodal-Concentrated Loss, and the loss of A Simple Log-based Loss Function for Ordinal Text Classification itself. Each is evaluated both with its native decision and with our stretch rule, which separates the effect of the loss from the effect of the decision.
Two external reference points come from the file. On Kaggle-EyePACS, the strongest reported number is QWK 0.884 for Lesion-Aware Transformers for Diabetic Retinopathy Grading. We treat it as a ceiling reference obtained with a stronger, lesion-aware architecture. The fair test is whether adding the stretch rule to that class of backbone moves its QWK. For FGADR Grade-set, the file reports only a relative gain of up to 8.04% from lesion masks in A Benchmark for Studying Diabetic Retinopathy, and no absolute QWK. We found no reported QWK results on Lacuna for Adience or for the text rating sets.
Two ablations isolate the key ingredients. The first swaps the stretch rule for free cutoffs while holding the estimator fixed, which isolates the decision-rule claim. The second swaps the non-crossing head for a squared-error head while holding the rule fixed, which isolates the role of ordinal consistency in estimating the mean.
Why not just…?
Why not just fit a regressor and tune cutoffs, as practitioners already do? This is the most serious alternative, and our theory partly endorses it: it is consistent for QWK. The proposal differs in two testable ways. It replaces $K-1$ free cutoffs with a one-parameter rule, which should matter on FGADR's 1,000 graded images and on Adience, where cutoff tuning can overfit. It also asks whether an ordinal-consistent head estimates the tail mean better than squared error, which the $1/\kappa^*$ amplification makes consequential under skew. If free cutoffs match the stretch rule on small data and squared error matches the ordinal head, we will report that the practical recipe is already optimal and the contribution is explanatory.
Why not just train through a differentiable QWK loss? The theorem says this is unnecessary for consistency. Such a loss also inherits the steepness and batch-dependence problems documented for sigmoidF1. Because QWK's normalizer couples all predictions, minibatch estimates are biased. We will show this directly by sweeping batch size for the Weighted Kappa Loss baseline. If soft-QWK training still wins after our decision rule is applied to all methods, the gain must come from representation learning, and we will say so.
Why not optimize QWK with policy gradients? The file's evidence from coreference, in Optimizing Differentiable Relaxations of Coreference Evaluation Metrics, is that relaxations beat reinforcement learning in both speed and accuracy. Our closed form removes the need for either approach at the decision stage. We therefore include policy gradients only in the synthetic study, as a check that it converges to the same rule.
Why not use cross-entropy with temperature scaling? Temperature scaling changes the spread of the predicted distribution but has only a limited effect on its mean. Under our claim it therefore cannot fix a QWK shortfall caused by using argmax instead of the stretched mean. Pairing cross-entropy's expected value with the stretch rule is included as a baseline and tests this directly.
Proposed Experiments
Synthetic ordinal data with a known $\mu(x)$ and controllable skew is the first decisive check. The stretch rule should match brute-force QWK optimization. Varying predictive variance at a fixed mean should leave QWK unchanged. Test-QWK loss from injected mean noise should grow roughly as $1/\kappa^*$.
The real-data program covers Kaggle-EyePACS and FGADR Grade-set (QWK, with skewed 5-grade labels), Adience (QWK over 8 age groups), and the Amazon, Yelp and SST-5 rating sets. On the text sets we report both QWK and linearly weighted kappa, plus the off-by-one accuracy used in A Simple Log-based Loss Function for Ordinal Text Classification. All methods use three backbones per domain, five seeds, and the shared validation budget.
The crossover experiment is the sharpest falsification test. The theory predicts that distribution-shaping losses give no QWK gain beyond their mean accuracy, measured as held-out squared error of $\hat\mu$, but do give a gain in linearly weighted kappa. If QWK gains from unimodal losses persist after controlling for mean accuracy, the central claim that QWK sees only the mean is wrong in finite samples, and the proposal's framing would have to change.
References
- The Problem with Kappa
- Consistency Conditions for Differentiable Surrogate Losses
- On the Consistency of Ordinal Regression Methods
- Adversarial Surrogate Losses for Ordinal Regression
- Unimodal Probability Distributions for Deep Ordinal Classification
- Unimodal-Concentrated Loss: Fully Adaptive Label Distribution Learning for Ordinal Regression
- A Simple Log-based Loss Function for Ordinal Text Classification
- Ordinal Regression by Extended Binary Classification
- Estimating Class Membership Probabilities using Classifier Learners
- Order Regularization on Ordinal Loss for Head Pose, Age and Gaze Estimation
- sigmoidF1: A Smooth F1 Score Surrogate Loss for Multilabel Classification
- Lesion-Aware Transformers for Diabetic Retinopathy Grading
- A Benchmark for Studying Diabetic Retinopathy: Segmentation, Grading, and Transferability
- Optimizing Differentiable Relaxations of Coreference Evaluation Metrics