Many medical prediction tasks are really ordering problems. A diabetic retinopathy grader, a lung-nodule malignancy scorer, or a cancer-grade classifier is useful when it places sicker patients above healthier ones. Kendall's $\tau$ measures exactly that. It is a rank correlation: the fraction of patient pairs a model orders correctly minus the fraction it orders incorrectly, with no assumption about the spacing between grades. It is therefore tempting to train graders by maximizing a smooth version of $\tau$. We argue that doing this naively fails under the class imbalance typical of clinical data, and for a simple reason. The number of pairs between grades $a$ and $b$ is the product of their class sizes. In a cohort dominated by healthy and mild cases, almost every pair the loss sees is "healthy versus mild." The rare severe cases, which carry the clinical risk, contribute a vanishing share of the gradient.
We propose to fix the objective itself rather than the data. The objective averages concordance separately over each pair of grades, with weights set by clinical risk rather than by prevalence. We optimize it with a smooth surrogate plus a small bank of stored scores for rare-grade patients, so that severe cases take part in every mini-batch without being oversampled. The model outputs a single severity score. Grades, and screening thresholds that meet a required sensitivity, are read off that score afterwards. The claim is testable: the advantage of the balanced objective over plain differentiable $\tau$ should grow with the imbalance ratio and should show up mostly in recall on the severe grades.
Core Claim
Let a network output a scalar severity score $s(x)$ for an image $x$ with grade $y \in \{0,\dots,K-1\}$. The quantity we train and evaluate is
$$C_{\text{bal}} = \frac{\sum_{a<b} w_{ab}\, \Pr\big(s(X') > s(X) \mid Y=a,\ Y'=b\big)}{\sum_{a<b} w_{ab}},$$
where $w_{ab}$ is a weight chosen per grade pair. Population Kendall's $\tau$ is, up to an affine map, the same sum with $w_{ab} \propto n_a n_b$, where $n_a$ is the number of cases in grade $a$. That is the prevalence weighting we remove. Same-grade pairs count as ties and are excluded, as in Kendall's $\tau_b$.
Each term is a per-grade-pair area under the ROC curve (AUC). So $C_{\text{bal}}$ is an ordinal, macro-averaged generalization of the Mann-Whitney statistic that Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes uses for binary critical-versus-non-critical screening. We choose weights that grow with grade distance, $w_{ab} \propto (b-a)$. We add an extra asymmetric factor for pairs involving the top grades, which reflects the clinical view that ranking a severe case below a healthy one is the costliest error. The training loss replaces the indicator inside each probability with a logistic or $\tanh$ comparison. This is the smoothing that Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss applied to class orderings within a single logit vector. We apply it across patients instead.
Relation to Prior Work
Four lines of work motivate or constrain this proposal.
Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss supplies the smoothing technique. That paper ranks classes inside one prediction to match a teacher, on balanced natural-image data. We rank patients against each other under supervision, which is a different object with a different imbalance problem.
Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes is the closest mechanism. It enforces per-sample Mann-Whitney margins with an augmented Lagrangian, which needs four hyperparameters and handles only binary or unordered classes. We keep the rank-statistic view, extend it to ordered grades, and drop the dual variables.
Two papers constrain the design. Imbalance-aware loss functions improve medical image classification shows that losses derived from evaluation metrics help minority classes. It also shows that such losses collapse without cross-entropy alongside them and rely on oversampling. That warns us to test stability and to avoid oversampling. Ordinal Label-Distribution Learning with Constrained Asymmetric Priors for Imbalanced Retinal Grading (CAP-WAE) shows that rare severe grades get absorbed into the healthy class. It fixes this with a parametric skewed latent prior fitted to training prevalence. We make no distributional assumption, so a change of prevalence at deployment only changes thresholds, not the learned ordering.
RACR-MIL uses ranking only among patch attention weights inside one slide, not across patients. Controlling Class Layout for Deep Ordinal Classification via Constrained Proxies Learning fixes grade geometry in advance and was never tested on imbalanced clinical data. Our objective constrains only order, not spacing.
Research Program
The work has three parts.
The first part is analytic. We derive the gradient share that each grade pair receives under population $\tau$ as a function of class sizes. We then show on real label distributions that severe-grade pairs receive a small fraction of it; on retinopathy datasets that fraction is expected to be well under a few percent.
The second part is the estimator. Gap 3 of the research file is real: under a 1:100 skew, most random mini-batches contain no severe case, so no severe pair can be formed. We keep a first-in-first-out bank of detached scores for every grade, holding a few hundred entries per grade. Each current-batch sample is compared against bank entries of all other grades. Gradients flow only through the current sample, and stale bank scores are refreshed as their images recur in the ordinary data stream. Rare images are therefore never shown more often than their natural frequency. That is the concrete difference from oversampling, and it is what should limit memorization. The cost is proportional to batch size times bank size, a modest overhead compared with the $\mathcal{O}(C^2)$ concern raised in the distillation paper.
The third part is readout. Grade cut-points on $s$ are fit on validation data. A separate referral threshold is set to reach 95% or 98% sensitivity for referable disease, which ties the method to the operating points emphasized in the Mann-Whitney constrained-optimization paper.
Baselines
The naive reference is standard cross-entropy on the grades, with the same backbone, augmentation, and epoch budget. On LIDC-IDRI, Meta Ordinal Regression Forest for Medical Image Classification With Ordinal Labels reports 54.2% accuracy for this configuration with ResNet-18. We reproduce it before comparing.
The strongest existing methods are different on each benchmark. For retinal grading, the strongest competitor is CAP-WAE, which reports a quadratic weighted kappa (QWK) of 0.94 on Zenodo-DR-7 (Ordinal Label-Distribution Learning with Constrained Asymmetric Priors for Imbalanced Retinal Grading). For APTOS-2019, IDRiD, and Messidor-2, no reported result was found on Lacuna, so we will report our own reimplementation on identical splits.
For lung nodules, the strongest competitor is the meta ordinal regression forest, which reports 63.4% accuracy and malignant recall of 0.752 on LIDC-IDRI with ResNet-18 MORF.
CAP-WAE and MORF are the strongest competitors because each attacks imbalance and ordering together. Beating them means matching or exceeding QWK and accuracy while clearly improving severe-grade recall at fixed referral sensitivity. We run every method on the same backbone and splits and give each the same hyperparameter-search budget, 20 trials per method. We report means over five seeds.
We also include three metric-driven baselines. The first is cross-entropy combined with soft-$F_1$ and with soft-MCC, both run with oversampling as in Imbalance-aware loss functions improve medical image classification. The second is the augmented-Lagrangian Mann-Whitney method applied to a referable-versus-non-referable split. The only numbers the file gives for it come from binary CIFAR-10 at 1:100, where false-positive rate at 98% true-positive rate fell from 56.0% to 52.0%; those numbers do not transfer to our benchmarks. The third is plain MSE regression on grade index followed by thresholds.
Three ablations isolate the key ingredient. The first is plain differentiable Kendall's $\tau$ with prevalence weighting. The second is plain $\tau$ with class-balanced sampling. The third is our balanced weighting without the score bank, together with a sweep over pair weights: uniform, grade distance, and grade distance with the asymmetric severe-case factor.
Why not just…?
Why not just use class-weighted cross-entropy or focal loss? Both reweight individual samples, but neither knows that grade 4 is further from grade 0 than grade 1 is. CAP-WAE reports that focal loss yields weaker QWK than ordinal-aware methods on retinal grading. The Mann-Whitney constrained-optimization paper reports that reweighting and focal loss leave false-positive rates high at high sensitivity. We include both as baselines. If they match us on severe-grade recall at fixed sensitivity, the ordering objective adds nothing.
Why not just maximize plain differentiable Kendall's $\tau$ with class-balanced sampling? This is the real threat. Balanced sampling roughly equalizes pair counts, so it could recover most of the effect of our weighting. The difference is that it shows rare images many more times, which invites memorization, and Imbalance-aware loss functions improve medical image classification cautions about exactly that. The second ablation settles this directly by comparing test severe-grade recall and the train-test gap on minority grades. If balanced sampling ties, we will report that the contribution reduces to pair weighting and that the score bank is unnecessary.
Why not just regress the grade with MSE? Regression imposes equal spacing between grades. Controlling Class Layout for Deep Ordinal Classification via Constrained Proxies Learning reports regression underperforming proxy-based ordinal models. Under imbalance, the regression target mean also drifts toward the common grades, which is the same under-prediction of the upper tail reported in the clinical literature. It is a baseline in every experiment.
Why not just run the Mann-Whitney augmented-Lagrangian method one-vs-rest? One-vs-rest splits discard the order between intermediate grades, and the method needs four interacting hyperparameters. We give it the same tuning budget. If it wins at the referral threshold, we will report that as a win for constraints over averaged concordance at the extreme operating point.
Proposed Experiments
The main benchmarks are APTOS-2019, Messidor-2, IDRiD, and Zenodo-DR-7 for five-grade retinopathy, and LIDC-IDRI with benign, unsure, and malignant groups following MORF. Metrics are QWK, accuracy, severe-grade recall, false-positive rate at 95% and 98% referral sensitivity, and $C_{\text{bal}}$ itself. We report $C_{\text{bal}}$ because QWK can also be dominated by common grades.
The first decisive check is a controlled imbalance sweep on APTOS-2019. We subsample grades 3 and 4 to imbalance ratios of 1:10, 1:50, and 1:200 relative to grade 0. The hypothesis predicts that the gap in severe recall between balanced and prevalence-weighted $\tau$ widens as imbalance grows. A flat gap would falsify the mechanism.
The second check tests memorization. At each ratio we compare the score bank against class-balanced sampling on the train-test gap in minority recall.
The third check tests robustness to prevalence shift. We train on APTOS-2019 and test on Messidor-2 without refitting, then with only the cut-points refit. The prediction is that our model recovers most of its performance from threshold refitting alone, whereas CAP-WAE's fixed prior should need more than that.
The fourth check tests stability. We train our loss with and without a small cross-entropy term and count seeds in which severe-grade recall collapses. This checks whether our objective inherits the instability reported for soft-$F_1$ and soft-MCC.
References
- Constrained Optimization to Train Neural Networks on Critical and Under-Represented Classes
- Enhancing Logits Distillation with Plug&Play Kendall's $\tau$ Ranking Loss
- Imbalance-aware loss functions improve medical image classification
- Ordinal Label-Distribution Learning with Constrained Asymmetric Priors for Imbalanced Retinal Grading
- RACR-MIL: Rank-aware contextual reasoning for weakly supervised grading of squamous cell carcinoma using whole slide images
- Controlling Class Layout for Deep Ordinal Classification via Constrained Proxies Learning
- Meta Ordinal Regression Forest for Medical Image Classification with Ordinal Labels