Minimax-Guided Ranking-Regression with Adaptive Loss Balancing

Many prediction problems have a peculiar shape: the quantity we want to predict changes sharply between items that are almost identical, while smooth everywhere else. In drug discovery this is called an activity cliff — two molecules whose structures differ by one small edit can differ a hundredfold in potency. Standard regression trained to minimize squared error learns the conditional mean and smooths right over these cliffs, because cliffs are statistically rare; pure ranking losses can spot them but produce scores with no absolute meaning. Hybrid losses that add a pointwise regression anchor to a pairwise or listwise ranking term are the accepted fix, but the literature shows they are fragile in two specific ways: the ranking term is fed pairs chosen by crude straight-line proximity in the model's embedding, and the balance between the two terms is set by hand-tuned weights or outright heuristics — one paper had to postpone a consistency loss until epoch 120 by trial and error to stop training from collapsing.

This proposal argues that both fragilities have the same root and one repair. Minimax path distance — a measure of whether two points can be bridged by a chain of small steps through dense regions of the data, computed as the smallest possible largest hop along any connecting path — is exactly the right notion of "near neighbor" for deciding which pairs deserve ranking pressure. And once the ranking term is fed topology-aware pairs, the composite objective has three heterogeneous losses pulling on one prediction head, which is precisely the setting where adaptive gradient balancing (dynamic weight adjustment based on each loss's gradient norms and learning progress) should replace grid search. No work in the literature combines all three; the closest papers each stop one step short, as detailed below.

Core Claim

The claim is two-part and testable. First, for data on non-convex manifolds where the target changes discontinuously across high-density neighborhoods, selecting ranking pairs by minimax path distance computed on the input descriptor space (for molecules, structural similarity) rather than by Euclidean distance in the learned embedding will improve cliff recovery, because the model stops wasting margin enforcement across low-density voids where its representation is unreliable and concentrates it on genuine cliffs. Second, the training instability documented in hybrid ranking-regression work is largely a gradient magnitude and direction problem that dynamic balancing can absorb, replacing both static weights and hand-tuned activation schedules. The composite objective is:

$$\mathcal{L}(\theta) = w_1\,\mathcal{L}_{\text{reg}} + w_2\,\mathcal{L}_{\text{rank}} + w_3\,\mathcal{L}_{\text{path}}, \qquad w_k = w_k(t) \text{ updated per step},$$

where $\mathcal{L}_{\text{reg}}$ is mean squared error on the target, $\mathcal{L}_{\text{rank}}$ is a thresholded listwise ranking loss restricted to topology-mined pairs, and $\mathcal{L}_{\text{path}}$ preserves the ordering of path distances in the learned embedding.

Research Program

The first component is pair mining. We build a k-nearest-neighbor graph on fixed input descriptors (ECFP Tanimoto similarity for small molecules, sequence-similarity for peptides), compute all-pairs minimax distances via a minimum spanning tree, and rank candidate pairs by the ratio of target gap to minimax distance — pairs close in path distance but far in potency are the cliffs worth training on, following the cliff definition already used in the benchmarks. Pairs whose connecting minimax path is large are excluded from ranking pressure. To guard against the short-circuit failure mode, where a single noisy bridge point lets two medium hops replace one large hop, we bound the number of hops per path using the order-constraint idea of On Order-Constrained Transitive Distance Clustering, which reported that bounding path order raised clustering accuracy on Extended Yale B from 82.81% (unconstrained transitive distance) to 90.64%. Crucially, because the graph lives in fixed input space, no gradient flows through the combinatorial min-max operators; this sidesteps the differentiability barrier that has kept minimax distances out of end-to-end losses (the open problem that Internal Evaluation of Density-Based Clusterings with Noise flags when it suggests its pointwise minimax index could become a training loss). Our graph is a fixed data structure refreshed only by hop-order constraints, not a differentiable operator.

The second component is the path-preservation term. We want the learned embedding to respect the input-space topology: if $\mathcal{D}_{\text{mm}}(i,j) < \mathcal{D}_{\text{mm}}(i,k)$, embedded distances should preserve that order. We implement this as a margin hinge on triplets sampled by path-distance ordering, which is fully differentiable without relaxing the graph itself. As a smoother variant we also implement the power-weighted path metric of Clustering and visualization of single-cell RNA-seq data using path metrics, where the exponent $p$ interpolates between shortest-path distance ($p=1$) and pure minimax ($p \to \infty$); that paper's Elongation Score, which auto-selects $p$ from the skew of neighbor-distance distributions without labels, gives us a principled way to set this without another grid search.

The third component is the balancing engine, and it is where the methodological contribution lives. GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks adjusts weights so that each loss's gradient norm tracks its relative inverse training rate; it was developed and validated only for multi-head vision tasks (roughly 5% depth-error improvement on NYUv2 at ~5% compute overhead), never for heterogeneous structural losses sharing a single scalar output. We extend it here and add a projection step, following Mitigating Gradient Bias in Multi-objective Learning: A Provably Convergent Approach, that activates only when the ranking and regression gradients have negative cosine similarity. The scientific question we will answer along the way is diagnostic: are the oscillations reported by CliffRank: A Dual-Branch Framework for Activity-Cliff Ranking Prediction — which that paper fixed by manually delaying its preference-consistency loss until epoch 120 — driven by gradient magnitude imbalance, directional conflict, or both? We will log per-step gradient norms and pairwise cosines for all three losses and correlate their trajectories with the training-loss oscillations, then test whether GradNorm alone, projection alone, or both is needed.

Baselines

The naive reference point is pure pointwise regression: the same encoder (protein language model for peptides, graph neural network for small molecules) trained with mean squared error only, which the literature already shows smooths over cliffs. The strongest existing competitor is CliffRank itself, the dual-branch architecture combining MSE, thresholded listwise ranking, and preference-consistency losses, which reports a mean Spearman correlation of 0.5393 and Recall@50 of 21.4 on the GRAMPA antimicrobial peptide benchmark with an ESM2-t12 encoder, and a mean Spearman of 0.6890 on matched-molecular-pair small-molecule datasets CliffRank. To make the comparison fair, we run CliffRank with its published architecture and its own delayed-PPC schedule, use identical encoders, splits (the AC Split that puts all cliff pairs in the test set), and tuning budget: whatever compute CliffRank's authors spent searching its hyperparameters, we grant an equal budget to our static-weight ablation, so the adaptive method is never advantaged by more search. We beat CliffRank if mean Spearman improves by at least 0.02 with non-overlapping confidence intervals over five seeds, and, separately for the balancing claim, if performance is stable across a coarse sweep of initial weights where the static-weight version degrades. The classical baseline is ECFP fingerprints with a support vector machine, which Exposing the Limitations of Molecular Machine Learning with Activity Cliffs found to be the best-performing traditional method on activity cliffs on average, beating standard deep architectures — no reported per-target number for this SVM baseline was found on Lacuna, so we will run it ourselves. The key ablation is our full architecture with pair mining switched from minimax path distance to plain Euclidean embedding distance, keeping the adaptive balancing fixed; this isolates the topological ingredient. A second ablation keeps minimax pair mining but replaces adaptive weights with a static grid search, isolating the balancing ingredient.

Why not just…?

Why not just train pure regression? The evidence against it is direct. Temporal Relational Ranking for Stock Prediction showed a model with lower squared error producing roughly a third of the portfolio return of a model with higher error but better top-asset ranking, and Improving Top-N Recommendation with Heterogeneous Loss showed that zeroing out the ranking term collapses precision on Yahoo! Music. Regression optimizes calibration, not discrimination among near neighbors, and cliffs live exactly in that discrimination gap.

Why not just use ECFP plus SVM and skip deep learning? This is a real threat, not a strawman: on the MoleculeACE benchmark, the limitations paper found SVM with ECFP fingerprints beating graph neural networks and transformers on activity cliffs on average. We take it seriously by running that baseline on every dataset with the same splits. Our bet is that the gap the limitations paper found reflects standard training objectives, not deep models per se: their models were trained with pointwise losses that smooth cliffs, whereas ours mines cliff pairs explicitly and enforces their ordering. If the SVM still wins on most of the thirty ChEMBL targets, that bounds the proposal's value and we will say so.

Why not just grid-search static weights? GradNorm's authors report finding better loss balances in a single run than across 100 grid-search runs, at ~5% training-time overhead GradNorm, and Domain Separation Networks documented how slight weight deviations cause training collapse. In our setting the argument is sharper: with three interacting losses the grid is three-dimensional and, per CliffRank's own experience, the optimum moves during training — their epoch-120 threshold is evidence that no static setting works. Our equal-budget static ablation settles this empirically rather than by argument.

Why not just precompute minimax embeddings and feed them in as features? Classification with Minimax Distance Measures showed minimax-embedded features lift linear classifiers dramatically (62.5% to 99.8% on a synthetic manifold), so this is worth running as a baseline. But the embedding is transductive — new test compounds require recomputing graph paths — and it changes the input representation rather than deciding which pairs receive ranking gradients, which is our mechanism. We include it to show where feature-level use of minimax distances ends and pair-level use begins.

Proposed Experiments

The main testbed is GRAMPA, the antimicrobial peptide benchmark with three large subsets (E. coli, S. aureus, P. aeruginosa), evaluated under the AC Split by mean Spearman correlation and Recall@50; the number to beat is CliffRank's mean Spearman of 0.5393 and Recall@50 of 21.4 with ESM2-t12, and for context the plain ESM2 regression result of 0.4669 Spearman on S. aureus reported in AMPCliff. The second testbed is the matched-molecular-pair datasets (thrombin, mu-opioid receptor, melanocortin receptor 4) with a PNA graph network backbone, where CliffRank reports 0.6890 mean Spearman against a prior best of 0.6724. The third is the full ChEMBL/MoleculeACE suite of thirty targets (615 to 3657 molecules each), scored by cliff-compound RMSE against overall RMSE — the benchmark where the ECFP+SVM threat is strongest. Alongside these, two controlled studies: synthetic bridge and spiral manifolds with injected regression cliffs and bridge-noise outliers, measuring whether the regression anchor plus hop-order bounds suppress short-circuits (using block-diagonality of affinity matrices as the diagnostic, per the order-constrained clustering paper), and the gradient-geometry study correlating loss-oscillation episodes with gradient-norm ratios and cosine conflicts between the three terms.

The decisive checks are threefold. Minimax pair mining must beat Euclidean pair mining at matched budget, or the topological claim fails. Adaptive balancing must match or beat both the equal-budget static grid search and CliffRank's delayed-PPC heuristic across all datasets without any dataset-specific schedule, or the balancing claim fails. And the full system must beat ECFP+SVM on a majority of MoleculeACE targets, or the honest conclusion is that this machinery earns its complexity only on the peptide and matched-pair settings — a conclusion we would report as such.