In deep learning, we routinely train models to optimize for downstream objectives like image retrieval, recommendation, or token selection. Yet, a fundamental disconnect persists: while we evaluate these models using rank-based metrics—such as Mean Average Precision (mAP), Normalized Discounted Cumulative Gain (NDCG), or Recall@K—we rarely train them on these metrics directly.
Instead, we rely on proxy losses like cross-entropy or triplet loss. Why? Because sorting and ranking operations are fundamentally non-differentiable. A small change in a model's continuous output score either changes the discrete rank of an item completely or not at all. Mathematically, the derivative of a rank with respect to its score is zero almost everywhere, and undefined at the boundaries. This prevents standard backpropagation from functioning.
To bridge this gap, a major research direction focuses on differentiable ranking approximations. By replacing discrete sorting operations with smooth, continuous surrogates, these methods allow gradients to flow directly from rank-based evaluation metrics back into the neural network.
The Root of the Problem: Step Functions
To understand why ranking is hard to differentiate, consider how we compute the rank of an item $i$ within a set. The rank is simply the count of how many other items have a score higher than $i$:
$$\text{Rank}(x_i) = 1 + \sum_{j \neq i} \mathbb{I}(x_j > x_i)$$
where $\mathbb{I}$ is the indicator function that outputs $1$ if the condition is true and $0$ otherwise.
The indicator function is a step function. If you plot it, it is flat everywhere except for a sharp, discontinuous jump at zero. Because its derivative is zero everywhere it is flat, any loss function built directly on top of this exact rank will yield zero gradients during gradient descent. The model cannot learn which direction to move its parameters to improve the ranking.
Strategy 1: Sigmoid and Softplus Relaxations
The most intuitive way to fix this is to replace the hard indicator function $\mathbb{I}(x_j > x_i)$ with a smooth, continuous approximation. A natural candidate is the sigmoid function, $\sigma(\cdot)$, or the softplus function.
By scaling the difference between scores by a temperature parameter $\tau$, we can write a soft rank approximation:
$$\text{SoftRank}(x_i) = 1 + \sum_{j \neq i} \sigma\left(\frac{x_j - x_i}{\tau}\right)$$
When the temperature $\tau$ is very small, this function closely mimics the sharp step of the true rank. When $\tau$ is larger, it provides a smooth landscape with meaningful gradients, allowing the model to understand how close an item is to overtaking another in rank.
This core mechanism underpins many modern differentiable ranking frameworks. For instance, Karmim et al. utilize sigmoid relaxations to directly optimize NDCG in graph neural networks for top-k recommendation systems. Similarly, in high-dimensional database indexing, Li applies sigmoid relaxations to optimize learned functions for approximate nearest neighbor search, ensuring that the predicted 1D ordering preserves true high-dimensional distances. In anomaly detection, Iwata substitutes the standard step function with a sigmoid to directly maximize the Area Under the Curve (AUC) when dealing with inexact labels.
Strategy 2: Soft Histograms and Binning for Retrieval
For listwise metrics like Average Precision (AP), the challenge is not just computing individual ranks, but counting how many relevant items fall above a certain threshold. To make this differentiable, researchers leverage soft binning or histogram-based similarity assignments.
Instead of assigning an item to a single discrete rank position, its contribution is distributed across neighboring continuous intervals using kernel functions.
Hard Sorting: Score 0.82 ---> [ Rank 2 ] (Discrete slot)
Soft Binning: Score 0.82 ---> [ Bin 1: 0.1 ] [ Bin 2: 0.8 ] [ Bin 3: 0.1 ] (Continuous distribution)
This approach allows the entire ranking pipeline to become an end-to-end differentiable operation. Revaud et al. demonstrated that this soft-binning technique is highly robust to quantization parameters, allowing models to optimize directly for image retrieval metrics. Building on this, the Smooth-AP loss proposed by Brown et al. substituted the discrete indicator function in the AP formula with a continuous sigmoid function, enabling large-scale metric learning to scale efficiently to millions of images.
Strategy 3: Optimal Transport Formulations
A more mathematically rigorous perspective treats sorting as a combinatorial assignment problem. Sorting a vector of numbers can be viewed as finding a permutation matrix that maps the unsorted input vector to a sorted target vector.
Cuturi et al. formulated this permutation search as an optimal transport problem. By applying entropic regularization to the transport objective, the discrete, non-differentiable permutation matrix relaxes into a smooth, doubly stochastic matrix. This operator can be efficiently computed using Sinkhorn iterations, providing high-quality, non-zero gradients that allow neural networks to optimize directly for quantiles, medians, or top-k accuracy.
Strategy 4: Neural Proxies and Critics
Rather than mathematically relaxing the sorting operator, why not train a separate neural network to learn the sorting function itself? Because neural networks are inherently differentiable, a network that successfully mimics sorting can act as a bridge for gradients.
This is the philosophy behind SoDeep, introduced by Engilberge et al.. The authors pre-trained and froze a neural network module tasked solely with mapping a sequence of continuous scores to their corresponding ranks. Once trained, this proxy module is inserted into a larger architecture, allowing gradients to flow from rank-based loss functions back to the primary task network.
A similar concept is applied in reinforcement learning and collaborative filtering by Lobel et al., who trained a differentiable "critic" network to approximate non-differentiable ranking metrics like NDCG, bypassing the sorting bottleneck entirely during backpropagation.
Summary
Differentiable ranking approximations transform the rigid, step-like landscape of sorting operations into a smooth, continuous surface. Whether through sigmoid relaxations, soft histograms, optimal transport, or neural proxies, these techniques align training objectives with real-world evaluation metrics, unlocking direct optimization for the rank-dependent tasks that dominate modern AI applications.