Listwise Learning to Rank Based on Approximate Rank Indicators

Thibaut Thonet, Yagmur Gizem Cinar, Éric Gaussier, Minghan Li, Jean-Michel Renders

Abstract

Learning to rank (LTR) is fundamental to Information Retrieval (IR), yet optimizing standard metrics like NDCG and MAP directly remains challenging due to their non-differentiable nature. This paper introduces SmoothI, a novel, differentiable approximation of the rank indicator function. By smoothing the discrete ranking process, SmoothI enables neural networks to optimize IR metrics directly via gradient descent. We demonstrate that SmoothI achieves state-of-the-art results on both feature-based and text-based ranking benchmarks, including LETOR, MSLR-Web30K, and Robust04.

Introduction

In Information Retrieval, the goal is to order documents such that the most relevant appear at the top. Success is typically measured using metrics like Precision@K, Mean Average Precision (MAP), and Normalized Discounted Cumulative Gain (NDCG). However, these metrics are non-differentiable because they depend on discrete document ranks. Small changes in document scores often result in zero gradient, while score crossovers create discontinuities.

To address this "staircase" loss problem, researchers have traditionally relied on surrogate losses or differentiable approximations. Many existing approximations, however, suffer from mathematical inconsistency, lack of theoretical convergence, or numerical instability when integrated with deep architectures like BERT. SmoothI addresses these issues by providing a theoretically grounded and numerically stable rank indicator approximation.

Methods: The SmoothI Framework

The core contribution of this work is the SmoothI (pronounced "smoothie") rank indicator function. Traditionally, a rank indicator $I_j^r$ is binary: 1 if document $j$ is at rank $r$, and 0 otherwise. SmoothI approximates this using a recursive, parameterized softmax approach.

The SmoothI architecture integrates a neural model with differentiable rank indicator functions to compute ranking loss.

The framework introduces two critical components for stability and accuracy:

  • Recursive Parameterized Softmax: A hyperparameter $\delta$ ensures that once a document is "selected" for a high rank, its influence is removed from subsequent lower-rank distributions, preventing "double-counting."
  • Stop-Gradient (sg) Operator: This operator stabilizes recursive computation, preventing numerical explosions during backpropagation in deep neural networks.

The authors provide theoretical proof that as the temperature parameter $\alpha$ increases, SmoothI converges to the true rank indicator at an exponential rate, offering a more rigorous foundation than many heuristic smoothing methods.

Experimental Evaluation

SmoothI was evaluated across two primary paradigms: feature-based Learning to Rank (LTR) and text-based IR.

Feature-based LTR

The model was tested on standard benchmarks including LETOR 4.0 (MQ2007 and MQ2008), MSLR-Web30K (a large-scale commercial dataset), and Yahoo LTR (YLTR). These datasets represent a wide range of scale and complexity in ranking tasks.

Text-based IR

To test its efficacy with deep language models, the authors applied SmoothI to a "Vanilla BERT" model on the Robust04 dataset, a standard TREC collection.

Key Results and Discussion

SmoothI consistently outperformed or matched top-tier baselines such as LambdaLoss, ApproxNDCG, and SoftSort across various datasets.

Comparison of SmoothI against state-of-the-art ranking baselines across four major LTR datasets.

As shown in the experimental results, SmoothI demonstrates significant dominance on the large-scale MSLR-Web30K dataset across all metrics (P@1, NDCG@1, etc.). Furthermore, when applied to BERT-based rankers, the SmoothI-NDCG loss achieved an NDCG@20 of 0.480 on Robust04, setting a new state-of-the-art for vanilla BERT models on this collection.

A key insight from this research is that approximating the rank indicator itself provides a unified framework. Instead of designing custom approximations for every individual metric, SmoothI allows for the construction of differentiable versions of nearly any rank-based IR metric.

Conclusion and Future Work

SmoothI provides a robust bridge between discrete ranking metrics and gradient-based optimization. While the resulting optimization landscape is non-convex, empirical results suggest it is highly effective for modern deep learning architectures. Future work may involve extending SmoothI to other rank-based domains, such as differentiable K-Nearest Neighbors (K-NN) or general sorting functions, and addressing the memory overhead associated with large listwise batches in models like BERT.

Summary updated

Abstract

We study here a way to approximate information retrieval metrics through a softmax-based approximation of the rank indicator function. Indeed, this latter function is a key component in the design of information retrieval metrics, as well as in the design of the ranking and sorting functions. Obtaining a good approximation for it thus opens the door to differentiable approximations of many evaluation measures that can in turn be used in neural end-to-end approaches. We first prove theoretically that the approximations proposed are of good quality, prior to validate them experimentally on both learning to rank and text-based information retrieval tasks.
BibTeX
@inproceedings{thonet2022listwise,
  author = {Thibaut Thonet and Yagmur Gizem Cinar and Éric Gaussier and Minghan Li and Jean-Michel Renders},
  title = {{Listwise Learning to Rank Based on Approximate Rank Indicators}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i8.20826},
}
Figure