Unbiased Learning to Rank: Counterfactual and Online Approaches

Harrie Oosterhuis, Rolf Jagerman, Maarten de Rijke

Abstract

Learning to Rank (LTR) is a cornerstone of modern information retrieval, yet traditional supervised approaches are limited by the high cost and potential inaccuracy of human annotations. This paper, based on a tutorial presented at The Web Conference 2020, explores the transition toward Unbiased LTR, which leverages implicit user feedback—such as clicks—while mathematically correcting for inherent biases like position and selection bias. We examine the two primary paradigms: Counterfactual LTR, which de-biases historical logs, and Online LTR, which learns through real-time interaction.

Introduction

In the field of Information Retrieval (IR), the gold standard for training ranking models has long been human-labeled datasets. However, these datasets are difficult to scale and often fail to reflect the dynamic preferences of real-world users. Implicit feedback, primarily in the form of user clicks, offers a massive, low-cost alternative.

The challenge lies in the fact that clicks are not direct indicators of relevance; they are heavily influenced by the system's presentation. Position bias, where users are more likely to click top-ranked items regardless of relevance, and selection bias, where users can only interact with displayed items, create a feedback loop that reinforces existing system errors. Unbiased LTR provides the mathematical framework to extract true relevance signals from this noisy, biased interaction data.

Counterfactual Learning to Rank (CLTR)

Counterfactual Learning to Rank is an offline approach that treats de-biasing as a causal inference problem. By analyzing historical interaction logs, CLTR aims to estimate what the user behavior would have been under a different ranking policy.

The core mechanism of CLTR is the use of a propensity model to estimate the probability that a user examined a specific result. By applying Inverse Propensity Weighting (IPW), the learning objective is adjusted so that clicks on lower-ranked (and thus less likely to be seen) items are weighted more heavily. This effectively "undoes" the position bias, allowing the model to learn an unbiased ranking function from biased historical data. The success of CLTR is heavily dependent on the accuracy of the propensity estimation; if the user model is misspecified, the resulting ranker remains biased.

Online Learning to Rank (OLTR)

Unlike the offline nature of CLTR, Online Learning to Rank involves real-time interaction with users. The system learns by actively intervening in the ranking process, showing users slightly modified versions of the current best ranking to observe their preferences.

Historically, Dueling Bandit Gradient Descent (DBGD) was the standard for OLTR. DBGD functions by comparing the current model against a noisy candidate; if the candidate performs better in a "duel," the model moves in that direction. However, recent research has highlighted the instability and slow convergence of DBGD. This has led to the rise of Pairwise Differentiable Gradient Descent (PDGD). PDGD is fully differentiable and utilizes pairwise comparisons between documents to update the model. It is significantly more sample-efficient and stable, making it the preferred choice for modern production environments.

Key Insights and Comparative Analysis

The choice between CLTR and OLTR involves a fundamental trade-off between safety and adaptability:

  • CLTR is safer as it utilizes existing logs without risking the user experience with experimental rankings, but it requires a sophisticated and accurate model of user bias.
  • OLTR is more adaptive to changing trends but requires "exploration," which may temporarily surface lower-quality results to the user to facilitate learning.

A critical finding in the literature is that both paradigms are theoretically capable of converging to the same optimal ranker that would be produced by perfect manual labels. Researchers typically validate these methods using benchmarks like the Yahoo! Learning to Rank Challenge dataset or simulated environments to isolate the effects of specific biases.

Conclusion

Unbiased Learning to Rank represents a significant shift in IR, moving from static, human-labeled datasets to dynamic, user-centric learning. While challenges remain—particularly regarding model misspecification in CLTR and the exploration-exploitation trade-off in OLTR—the development of robust methods like PDGD provides a clear path forward. The next frontier involves hybrid approaches: using historical data to "warm-start" a model via CLTR before refining it through continuous online interaction.

Summary updated

Abstract

This tutorial is about Unbiased Learning to Rank, a recent research field that aims to learn unbiased user preferences from biased user interactions. We will provide an overview of the two main families of methods in Unbiased Learning to Rank: Counterfactual Learning to Rank (CLTR) and Online Learning to Rank (OLTR) and their underlying theory. First, the tutorial will start with a brief introduction to the general Learning to Rank (LTR) field and the difficulties user interactions pose for traditional supervised LTR methods. The second part will cover Counterfactual Learning to Rank (CLTR), a LTR field that sprung out of click models. Using an explicit model of user biases, CLTR methods correct for them in their learning process and can learn from historical data. Besides these methods, we will also cover practical considerations, such as how certain biases can be estimated. In the third part of the tutorial we focus on Online Learning to Rank (OLTR), methods that learn by directly interacting with users and dealing with biases by adding stochasticity to displayed results. We will cover cascading bandits, dueling bandit techniques and the most recent pairwise differentiable approach. Finally, in the concluding part of the tutorial, both approaches are contrasted, highlighting their relative strengths and weaknesses, and presenting future directions of research. For LTR practitioners our comparison gives guidance on how the choice between methods should be made. For the field of Information Retrieval (IR) we aim to provide an essential guide on unbiased LTR to understanding and choosing between methodologies.
BibTeX
@inproceedings{oosterhuis2020unbiased,
  author = {Harrie Oosterhuis and Rolf Jagerman and Maarten de Rijke},
  title = {{Unbiased Learning to Rank: Counterfactual and Online Approaches}},
  booktitle = {The Web Conference},
  year = {2020},
  doi = {10.1145/3366424.3383107},
}
Figure