Abstract
Learning to Rank (LTR) is a cornerstone of modern information retrieval, yet traditional supervised approaches are limited by the high cost and potential inaccuracy of human annotations. This paper, based on a tutorial presented at The Web Conference 2020, explores the transition toward Unbiased LTR, which leverages implicit user feedback—such as clicks—while mathematically correcting for inherent biases like position and selection bias. We examine the two primary paradigms: Counterfactual LTR, which de-biases historical logs, and Online LTR, which learns through real-time interaction.
Introduction
In the field of Information Retrieval (IR), the gold standard for training ranking models has long been human-labeled datasets. However, these datasets are difficult to scale and often fail to reflect the dynamic preferences of real-world users. Implicit feedback, primarily in the form of user clicks, offers a massive, low-cost alternative.
The challenge lies in the fact that clicks are not direct indicators of relevance; they are heavily influenced by the system's presentation. Position bias, where users are more likely to click top-ranked items regardless of relevance, and selection bias, where users can only interact with displayed items, create a feedback loop that reinforces existing system errors. Unbiased LTR provides the mathematical framework to extract true relevance signals from this noisy, biased interaction data.
Counterfactual Learning to Rank (CLTR)
Counterfactual Learning to Rank is an offline approach that treats de-biasing as a causal inference problem. By analyzing historical interaction logs, CLTR aims to estimate what the user behavior would have been under a different ranking policy.
The core mechanism of CLTR is the use of a propensity model to estimate the probability that a user examined a specific result. By applying Inverse Propensity Weighting (IPW), the learning objective is adjusted so that clicks on lower-ranked (and thus less likely to be seen) items are weighted more heavily. This effectively "undoes" the position bias, allowing the model to learn an unbiased ranking function from biased historical data. The success of CLTR is heavily dependent on the accuracy of the propensity estimation; if the user model is misspecified, the resulting ranker remains biased.
Online Learning to Rank (OLTR)
Unlike the offline nature of CLTR, Online Learning to Rank involves real-time interaction with users. The system learns by actively intervening in the ranking process, showing users slightly modified versions of the current best ranking to observe their preferences.
Historically, Dueling Bandit Gradient Descent (DBGD) was the standard for OLTR. DBGD functions by comparing the current model against a noisy candidate; if the candidate performs better in a "duel," the model moves in that direction. However, recent research has highlighted the instability and slow convergence of DBGD. This has led to the rise of Pairwise Differentiable Gradient Descent (PDGD). PDGD is fully differentiable and utilizes pairwise comparisons between documents to update the model. It is significantly more sample-efficient and stable, making it the preferred choice for modern production environments.
Key Insights and Comparative Analysis
The choice between CLTR and OLTR involves a fundamental trade-off between safety and adaptability:
- CLTR is safer as it utilizes existing logs without risking the user experience with experimental rankings, but it requires a sophisticated and accurate model of user bias.
- OLTR is more adaptive to changing trends but requires "exploration," which may temporarily surface lower-quality results to the user to facilitate learning.
A critical finding in the literature is that both paradigms are theoretically capable of converging to the same optimal ranker that would be produced by perfect manual labels. Researchers typically validate these methods using benchmarks like the Yahoo! Learning to Rank Challenge dataset or simulated environments to isolate the effects of specific biases.
Conclusion
Unbiased Learning to Rank represents a significant shift in IR, moving from static, human-labeled datasets to dynamic, user-centric learning. While challenges remain—particularly regarding model misspecification in CLTR and the exploration-exploitation trade-off in OLTR—the development of robust methods like PDGD provides a clear path forward. The next frontier involves hybrid approaches: using historical data to "warm-start" a model via CLTR before refining it through continuous online interaction.
Summary updated