Disease severity grading assigns a scan to an ordered scale such as normal, early glaucoma, or progressive glaucoma. In ophthalmology, the abundant and cheap modality is the 2D color fundus photograph, which comes with many expert grades. The more informative modality, the 3D optical coherence tomography (OCT) volume, is scarce and expensive to grade. The standard way to carry knowledge from fundus to OCT is to copy the fundus model's predicted grades or probabilities onto paired OCT scans of the same eyes. We argue this copies the wrong thing. The two modalities see different evidence, so their grade probabilities sit on different, miscalibrated scales. What should transfer is the ordering of patients by severity: if eye A looks clearly worse than eye B on fundus, it should also be worse on OCT. We propose to train the OCT grader from the fundus model's confident pairwise orderings on paired but ungraded scans. A handful of graded OCT "anchor" scans then fixes where the grade boundaries fall on the OCT severity axis.
The idea comes from automated grading research, where the target is agreement with human consensus rather than per-sample accuracy. That literature has shown that ranking objectives tied to agreement metrics beat pointwise losses, and that anchoring to fixed reference examples stabilizes ordinal scores. We bring both lessons into cross-modality transfer, which has so far optimized Dice, accuracy, or cross-entropy. We also treat the fundus grader and the OCT grader as two "raters" of the same eye. Cross-modality consistency is then measured the way inter-rater reliability is measured, using quadratic weighted kappa (QWK), an agreement score that penalizes disagreements by the squared distance between grades.
Core Claim
Our claim is that, under scarce target-modality labels, supervising a target-modality grader with gap-gated pairwise severity orderings from a source-modality grader produces higher QWK against expert grades than distilling the source's soft labels. The gating keeps only pairs whose source scores differ by a wide margin. We further claim that the advantage grows as the source model becomes more miscalibrated relative to the target, because a ranking signal is unchanged by any monotone rescaling of the source scores, while a soft-label signal is not.
The two modalities share only a one-dimensional severity axis. Their feature spaces are never aligned. This design choice follows two findings. The GAMMA Challenge reports that feature-level 2D–3D fusion struggles because fundus coordinates and OCT slices have no direct spatial correspondence. IMDR notes that forcing heterogeneous modalities into identical latent spaces destroys modality-specific diagnostic signal.
Relation to Prior Work
Three papers motivate the gap.
Samba clusters features around learnable severity bases so that grades generalize across hospitals and scanners. It stays within a single 2D modality and trains with classification loss. We cross modalities and optimize an ordinal agreement target.
IMDR handles fundus or OCT missing at test time. It separates modality-common from modality-specific features with a mutual-information bound and anchors classes to learnable proxies. It requires a teacher trained on fully paired, fully graded data, and it reports accuracy rather than agreement. We need grades only on the source modality plus a few target anchors, and we evaluate in kappa.
Harvard-GDP documents the "similarity trap": normal and diseased retinal maps can be about 0.95 similar, so confident pseudo-labels are often wrong. It fixes this with a reinforcement-learned pseudo-labeler. This finding constrains our design: it is why we gate on large source-score gaps instead of trusting per-sample confidence.
Two grading papers supply the method.
Automated Essay Scoring by Maximizing Human-Machine Agreement adapts listwise ranking (LambdaMART) to optimize QWK directly and reaches human-level agreement on the ASAP essay benchmark. It also flags that mapping continuous ranking scores back onto discrete grades introduces errors at the grade boundaries.
ORARS scores pronunciation by ordinal comparison against fixed anchor samples and surpasses teacher inter-rater reliability. Its inference cost grows with the number of anchors.
We borrow the ranking objective from the first and the anchors from the second, but use the anchors differently. They are used once, to fit the $K-1$ cut points between grades, and are never compared against at inference. Each target scan therefore needs one forward pass. This addresses both limitations listed above.
Research Program
The setting is as follows. We have a source modality with grades, here fundus. We have a target modality, here OCT, of which only $k$ scans per grade are graded. We also have paired fundus–OCT scans of the same eyes with no target grades.
We first train a source grader with a QWK-aligned ordinal objective. It outputs a scalar severity score $s_i$ for eye $i$.
We then train a target encoder $f$, which maps an OCT volume $x_i$ to a scalar severity score, on the paired unlabeled set. The loss is
$$ \mathcal{L}_{\text{rank}} = \sum_{(i,j):\, s_i - s_j > \delta} \log\left(1 + e^{-(f(x_i) - f(x_j))}\right), $$
where $\delta$ is the gap threshold that keeps only pairs the source orders with a wide margin. We add a standard ordinal loss on the $k$ graded anchors per grade.
Finally, we choose grade thresholds $\tau_1 < \dots < \tau_{K-1}$ on the target axis to maximize QWK on the anchors. We use leave-one-out to avoid fitting the thresholds to the same anchors that score them.
The program asks three questions:
- Does gap-gating work? We measure, on held-out graded pairs, how the source's pair-ordering accuracy rises with the source-score gap. Gap-gating only makes sense if large-gap pairs are substantially more reliable than the source's grades themselves.
- Does transfer run both ways? The GAMMA single-modality baselines show OCT is the stronger modality (kappa 73.2% vs. 67.7% for fundus). So we test transfer from the weaker to the stronger modality (fundus→OCT) as well as the reverse (OCT→fundus).
- Do the two graders agree on the same eye? We report cross-modality QWK between the fundus grader and the OCT grader as a consistency measure in its own right.
Baselines
The naive reference is an OCT grader trained only on the $k$ anchors per grade, with the same backbone and an ordinal loss. Above it we report a fully supervised OCT-only model as a ceiling. For that ceiling, GAMMA reports a single-modality OCT baseline kappa of 73.2%, against 67.7% for fundus-only. Beating the naive reference shows that the unlabeled pairs carry usable signal. Approaching the 73.2% ceiling with small $k$ is the practically meaningful bar.
The most direct competitor is soft-label cross-modal distillation. Here the fundus grader's grade probabilities on paired eyes train the OCT model through KL divergence. It uses exactly the same data as our method and is the method a reviewer would reach for first. We run it with the same backbones, epochs, anchor set, and an equal hyperparameter search budget of 20 trials over temperature and loss weight; our method gets the same 20 trials over $\delta$ and loss weight. To count as beating it, our method must win on target QWK across three seeds and at least two values of $k$, with non-overlapping bootstrap intervals.
The strongest existing method is IMDR, run with its paired teacher. We give it the same graded fundus data and the same $k$ graded OCT anchors, and evaluate it in QWK rather than accuracy. The file reports that IMDR improves accuracy on Harvard-30k AMD by over 3.58% relative to Eye-Most and M2LC. No reported IMDR result on GAMMA was found on Lacuna, so we will reproduce it.
As a multimodal upper reference we cite GAMMA's winning SmartDSP ensemble, at 85.49% test kappa, and the dual-branch ResNet-34 at 81.2%. Both use both modalities at test time, which our setting does not, so they are context rather than targets.
Two ablations isolate the key ingredient:
- Replacing the pairwise ranking loss with pointwise regression onto the source scores tests whether ordering, rather than score values, is what matters.
- Removing gap-gating ($\delta = 0$) tests whether filtering out the similarity trap matters.
A third ablation, replacing anchor-fitted thresholds with argmax decoding, tests the threshold-mapping fix.
Why not just…?
Why not just distill the fundus model's soft labels? This is the real threat, and the experiments are built around it. Soft-label distillation copies the source's calibration. That is harmful when the source is weaker and differently calibrated, as GAMMA's single-modality kappas suggest fundus is relative to OCT. Our decisive test applies controlled monotone distortions to the source scores, using temperature changes and grade-dependent shifts. Ranking distillation is unchanged by construction, so if soft distillation degrades under these distortions while ours does not, the mechanism is confirmed. If soft distillation matches us on undistorted data and degrades only under artificial distortion, we will say so plainly: the benefit is then robustness rather than accuracy.
Why not just fine-tune an OCT model on the few graded scans with a good ordinal loss? This is our naive reference. The ASAP results in Automated Essay Scoring by Maximizing Human-Machine Agreement show that agreement-aligned objectives help. They do not supply data, though, and with $k$ small per grade the model sees too few examples near grade boundaries. Our learning curves over $k$ will show where, if anywhere, the extra pairs stop mattering. If that happens at small $k$, the method is not worth its complexity.
Why not just prompt a multimodal foundation model with the grading rubric? Your other Left! finds GPT-4o and other vision-language models near chance (about 50%) on spatial judgments in CT slices, driven by memorized textbook priors. Can GPTs Evaluate Graphic Design Based on Design Principles? finds GPT-4o misses 2–3 pixel discrepancies, a scale comparable to early-stage lesions. We include a zero-shot rubric-prompted baseline on fundus as a cheap check, but expect it to be weak.
Why not align the two modalities' features, as in IMDR or Samba-style severity bases? Both GAMMA and IMDR report that feature alignment across 2D and 3D either fails for lack of spatial correspondence or erases modality-specific signal. Our IMDR comparison and the pointwise-regression ablation together test whether a single shared severity axis suffices.
Proposed Experiments
The primary benchmark is GAMMA: 300 paired fundus–OCT scans from 276 patients with 3-tier grades. We hide OCT grades except for $k \in \{3, 5, 10\}$ per grade and evaluate with 5-fold patient-level splits. Because GAMMA is small and drawn from a single cohort, we replicate on the Harvard-30k glaucoma, AMD, and diabetic retinopathy subsets used by IMDR. We use the GAMMA dual-branch backbones (ResNet-34 for fundus, a 3D CNN for OCT).
We report four metrics:
- Target-modality QWK against expert grades.
- Per-grade sensitivity, compared against the junior-ophthalmologist range of 0.694–0.862 reported by GAMMA.
- Cross-modality QWK between the fundus and OCT graders on the same eye.
- Prediction stability across seeds, as an intra-rater reliability analogue.
No reported inter-expert kappa for GAMMA was found on Lacuna. If per-rater grades are not obtainable, expert-parity claims will be limited to the sensitivity comparison.
Three checks decide the proposal:
- Gap-gated pairs must be markedly more accurate than source grades. If they are not, the method has no basis.
- Ranking distillation must beat soft-label distillation on QWK at small $k$, or at least hold up under source miscalibration where soft distillation does not.
- The gain must persist in the OCT→fundus direction and on Harvard-30k. This shows it is not an artifact of GAMMA's size or of which modality is stronger.
References
- GAMMA challenge: Glaucoma grAding from Multi-Modality imAges
- Incomplete Modality Disentangled Representation for Ophthalmic Disease Grading and Diagnosis
- Samba: Severity-aware Recurrent Modeling for Cross-domain Medical Image Grading
- Harvard Glaucoma Detection and Progression: A Multimodal Multitask Dataset and Generalization-Reinforced Semi-Supervised Learning
- Automated Essay Scoring by Maximizing Human-Machine Agreement
- Improving Pronunciation Assessment Via Ordinal Regression with Anchored Reference Samples
- Your other Left! Vision-Language Models Fail to Identify Relative Positions in Medical Images
- Can GPTs Evaluate Graphic Design Based on Design Principles?