ULTRE Framework: a Framework for Unbiased Learning to Rank Evaluation based on Simulation of User Behavior

Yurou Zhao, Jiaxin Mao, Qingyao AiCSR-Sim4IR @ SIGIR 2021 · OpenReview

Introduction

Learning to Rank (LTR) systems are the backbone of modern search engines, yet they face a significant hurdle: user click data is inherently biased. Users are naturally inclined to click top-ranked results regardless of relevance, a phenomenon known as position bias. Unbiased Learning to Rank (ULTR) aims to de-bias this data to uncover true document relevance.

However, the field has historically lacked a standardized evaluation environment. Researchers often rely on proprietary logs or oversimplified, "homegrown" simulations. This leads to an evaluation crisis where algorithms appear artificially successful because they share the same underlying assumptions as the simulator—a problem known as "model-simulator matching bias."

The ULTRE Framework

The ULTRE (Unbiased Learning to Rank Evaluation) framework addresses these issues by providing a standardized, multi-model simulation environment. It allows for rigorous comparison of both offline counterfactual and online bandit-based ranking models against diverse, realistic user behaviors.

The ULTRE framework standardizes the evaluation process from click simulation to final model assessment.

Methodology

The framework operates in three distinct stages:

  1. Simulation of Clicks: Rather than relying on a single model, ULTRE utilizes four probabilistic click models: PBM (Position-Based), UBM (User Browsing), DCM (Dependent Click), and MCM (Mobile Click). These are calibrated using real-world query logs to ensure they reflect actual human behavior.
  2. Training Protocols: The framework supports two primary training modes:
  3. Offline (Counterfactual): Models are trained on static, biased historical logs.
  4. Online (Bandit-based): Models interact with the simulator in real-time, iteratively updating their ranking strategy based on simulated feedback.
  5. Evaluation: Trained models are tested against "gold standard" expert-annotated datasets using standard metrics such as nDCG, MAP, and MRR.

Data and Calibration

To ensure realism, the researchers utilized massive real-world datasets for simulator calibration and model testing. The Sogou.com Query Logs provided over 800,000 sessions to train the probabilistic simulators, while the Sogou-SRR dataset provided expert-level relevance labels for final evaluation.

Statistics for the Sogou datasets used in the ULTRE framework.

Key Insights and Results

The Importance of Robustness

The study confirms that an ULTR model’s performance is heavily influenced by whether its assumptions match the simulator's. For instance, a model designed specifically for position bias (PBM-IPS) excels when the simulator only applies position bias but falters under more complex user behaviors.

The Superiority of DLA

The Dual Learning Algorithm (DLA) emerged as the most robust model. Unlike Inverse Propensity Scoring (IPS) methods that rely on fixed bias estimates, DLA jointly learns the ranking and bias models. In offline evaluations using the MCM-simulated dataset, DLA achieved an nDCG@5 of 0.7932, significantly outperforming PBM-IPS (0.7647).

Simulator Realism

The Mobile Click Model (MCM) and User Browsing Model (UBM) were identified as the most realistic simulators, achieving the highest log-likelihood when compared to real human click data. While most ULTR models outperformed the production ranker baseline, a performance gap remains compared to the "Full-info skyline"—a model trained on perfect, unbiased human labels.

Future Directions

While ULTRE represents a significant step forward, the authors plan to incorporate neural user behavior models to capture even more complex nuances than current probabilistic graphical models. Furthermore, while the framework defines an online protocol, future work will focus on large-scale empirical testing of online bandit models. The ULTRE framework is set to become the official benchmark for the NTCIR-16 ULTRE task, driving further community adoption and refinement.

Summary updated

BibTeX
@inproceedings{zhao2021ultre,
  author = {Yurou Zhao and Jiaxin Mao and Qingyao Ai},
  title = {{ULTRE Framework: a Framework for Unbiased Learning to Rank Evaluation based on Simulation of User Behavior}},
  booktitle = {CSR-Sim4IR @ SIGIR},
  year = {2021},
  url = {https://openreview.net/forum?id=VU0RHh7iWB},
}
Figure