RUCIR21 at the NTCIR-16 ULTRE Task

Zechun Niu, Yurou Zhao, Feng Wang, Jiaxin MaoNTCIR 2022 · OpenReview

Abstract

The RUCIR21 team participated in the NTCIR-16 Unbiased Learning to Rank Evaluation (ULTRE) task, focusing on the challenge of training search engines using biased user click data. By leveraging the Mobile Click Model (MCM), the team investigated effective strategies for both offline and online learning-to-rank scenarios. Their findings highlight that MCM is a robust tool for estimating click propensity in offline settings and that online models benefit significantly from utilizing cumulative historical interaction data.

Introduction

In modern Information Retrieval, user clicks serve as a primary source of implicit feedback for training ranking models. However, these clicks are inherently biased; for instance, position bias often leads users to click the first few results regardless of their actual relevance. The field of Unbiased Learning to Rank (ULTR) seeks to "debias" this data to recover true document relevance. This paper details the RUCIR21 team's approach to the NTCIR-16 ULTRE task, evaluating several ULTR methods across diverse simulated user behavior datasets.

Methodology

The team explored several sophisticated models to address user behavior complexity:

  • Mobile Click Model (MCM): Unlike simpler models, MCM accounts for the influence of previous clicks, post-click satisfaction, and "cascade" behavior. It was utilized both as a direct relevance estimator and as the foundation for Inverse Propensity Scoring (IPS), where it calculates the probability of a user examining a document to re-weight click importance.
  • Dual Learning Algorithm (DLA): This framework simultaneously learns a ranking model and a bias (propensity) model.
  • Pairwise Differentiable Gradient Descent (PDGD): An online method that updates the ranker by comparing document pairs after each interaction.
  • ODLA-PBM2: A modified Online DLA that employs a cumulative training strategy, incorporating all historical click data from the start of a session rather than just the most recent batch.

Experimental Setup

The study utilized five simulated click datasets provided by the ULTRE task, representing different user behavior models: Position-Based Model (PBM), Dependent Click Model (DCM), User Browsing Model (UBM), Mobile Click Model (MCM), and a heterogeneous FUSION dataset.

Results and Analysis

Offline Subtask

In the offline subtask, the MCM-IPS approach (Run 3) emerged as the top performer. As shown in the results table, it achieved the highest average NDCG@5 of 0.7920. Notably, it excelled on the FUSION dataset (0.8102), demonstrating its robustness in environments with mixed user biases. The results suggest that MCM is more effective at estimating propensity (the likelihood of examination) than at directly predicting relevance scores.

Offline subtask results (NDCG@5) for various models across different click simulations.

Online Subtask

For the online subtask, the team found that "memory" is a critical factor. The ODLA-PBM2 model (Run 3), which utilized cumulative historical data, significantly outperformed standard versions that only considered the current iteration's data. It achieved the best overall performance with an average NDCG@5 of 0.7971.

Online subtask results (NDCG@5) comparing different online learning strategies.

Discussion and Conclusion

A key insight from this research is that while MCM is highly effective for offline propensity estimation, its complexity can be a drawback in online "Dual Learning" settings. In such scenarios, where both the ranker and the bias model are continuously updated, the instability of MCM parameters can hinder performance.

In conclusion, the RUCIR21 team demonstrated that MCM-based IPS is a powerful method for offline debiasing, particularly in realistic, heterogeneous settings. Furthermore, they established that cumulative data utilization is a superior strategy for online learning-to-rank models. Future work will focus on improving the stability of MCM parameter estimation and developing methods to better capture biases in increasingly complex, mixed-user environments.

Summary updated

BibTeX
@inproceedings{niu2022rucir,
  author = {Zechun Niu and Yurou Zhao and Feng Wang and Jiaxin Mao},
  title = {{RUCIR21 at the NTCIR-16 ULTRE Task}},
  booktitle = {NTCIR},
  year = {2022},
  url = {https://openreview.net/forum?id=wjkyLnQLsb},
}
Figure