Introduction
Learning to Rank (LTR) systems are the backbone of modern search engines, yet they face a significant hurdle: user click data is inherently biased. Users are naturally inclined to click top-ranked results regardless of relevance, a phenomenon known as position bias. Unbiased Learning to Rank (ULTR) aims to de-bias this data to uncover true document relevance.
However, the field has historically lacked a standardized evaluation environment. Researchers often rely on proprietary logs or oversimplified, "homegrown" simulations. This leads to an evaluation crisis where algorithms appear artificially successful because they share the same underlying assumptions as the simulator—a problem known as "model-simulator matching bias."
The ULTRE Framework
The ULTRE (Unbiased Learning to Rank Evaluation) framework addresses these issues by providing a standardized, multi-model simulation environment. It allows for rigorous comparison of both offline counterfactual and online bandit-based ranking models against diverse, realistic user behaviors.
Methodology
The framework operates in three distinct stages:
- Simulation of Clicks: Rather than relying on a single model, ULTRE utilizes four probabilistic click models: PBM (Position-Based), UBM (User Browsing), DCM (Dependent Click), and MCM (Mobile Click). These are calibrated using real-world query logs to ensure they reflect actual human behavior.
- Training Protocols: The framework supports two primary training modes:
- Offline (Counterfactual): Models are trained on static, biased historical logs.
- Online (Bandit-based): Models interact with the simulator in real-time, iteratively updating their ranking strategy based on simulated feedback.
- Evaluation: Trained models are tested against "gold standard" expert-annotated datasets using standard metrics such as nDCG, MAP, and MRR.
Data and Calibration
To ensure realism, the researchers utilized massive real-world datasets for simulator calibration and model testing. The Sogou.com Query Logs provided over 800,000 sessions to train the probabilistic simulators, while the Sogou-SRR dataset provided expert-level relevance labels for final evaluation.
Key Insights and Results
The Importance of Robustness
The study confirms that an ULTR model’s performance is heavily influenced by whether its assumptions match the simulator's. For instance, a model designed specifically for position bias (PBM-IPS) excels when the simulator only applies position bias but falters under more complex user behaviors.
The Superiority of DLA
The Dual Learning Algorithm (DLA) emerged as the most robust model. Unlike Inverse Propensity Scoring (IPS) methods that rely on fixed bias estimates, DLA jointly learns the ranking and bias models. In offline evaluations using the MCM-simulated dataset, DLA achieved an nDCG@5 of 0.7932, significantly outperforming PBM-IPS (0.7647).
Simulator Realism
The Mobile Click Model (MCM) and User Browsing Model (UBM) were identified as the most realistic simulators, achieving the highest log-likelihood when compared to real human click data. While most ULTR models outperformed the production ranker baseline, a performance gap remains compared to the "Full-info skyline"—a model trained on perfect, unbiased human labels.
Future Directions
While ULTRE represents a significant step forward, the authors plan to incorporate neural user behavior models to capture even more complex nuances than current probabilistic graphical models. Furthermore, while the framework defines an online protocol, future work will focus on large-scale empirical testing of online bandit models. The ULTRE framework is set to become the official benchmark for the NTCIR-16 ULTRE task, driving further community adoption and refinement.
Summary updated