CIR at the NTCIR-17 ULTRE-2 Task

Lulu Yu, Keping Bi, Jiafeng Guo, Xueqi Cheng

Abstract

In the NTCIR-17 ULTRE-2 ranking challenge, the CIR team from the Chinese Academy of Sciences explored the limitations of traditional Unbiased Learning to Rank (ULTR) models. While most research focuses on mitigating position bias, this work identifies "false negatives"—relevant documents that remain unclicked due to high search engine quality—as a more significant hurdle in mature systems. By integrating a Dual Learning Algorithm (DLA) with a novel label correction strategy and strategic negative sampling, the team achieved a top nDCG@10 score of 0.5355, outperforming the competition's best baseline by 2.66%.

Introduction

Learning to Rank (LTR) systems increasingly rely on user click logs as a source of large-scale, low-cost training data. However, clicks are inherently noisy. The primary focus of the Unbiased Learning to Rank (ULTR) field has historically been position bias, where users are more likely to click top-ranked results regardless of relevance.

In this paper, we argue that for high-quality search engines like Baidu, the noise profile is shifting. The prevalence of false negatives—relevant documents that are ignored because the user's information need was satisfied by a previous result or because they did not scroll far enough—presents a more severe challenge than position bias alone. Standard ULTR models often treat these unclicked items as irrelevant (label 0), which introduces significant noise into the training process. Our approach focuses on "de-noising" these signals through label correction and improved negative sampling.

Methodology

Dual Learning Algorithm with Label Correction (DLA-LC)

The foundation of our approach is the Dual Learning Algorithm (DLA), which jointly learns a ranking model and a propensity model to account for position bias. To address the false negative problem, we introduced Label Correction (DLA-LC).

Instead of treating all unclicked items as irrelevant, we utilized a pre-trained DLA model as an auxiliary judge to assign non-zero labels to unclicked items based on predicted relevance. We evaluated two specific label correction modes:

  1. Sigmoid (sig): Assigning labels based on the raw probability output of the auxiliary model.
  2. Minimum (min): Setting labels to 1 if the predicted relevance exceeds the minimum relevance score of the documents that were actually clicked in that session.

Strategic Negative Sampling

To help the model distinguish between "relevant but unclicked" documents and truly irrelevant ones, we reconstructed the ranking lists using external negative samples. We employed the Galago search engine to identify "hard negatives"—documents that appear relevant based on traditional BM25 scores but are not. These were mixed with random negatives to create a more robust training set that forces the model to learn sharper decision boundaries.

GBDT and Feature Engineering

In addition to neural models, we utilized Gradient Boosting Decision Trees (LightGBM) trained on a small set of human-annotated validation data. This served two purposes: establishing a performance upper bound and testing whether combining neural model scores with traditional heuristic features could provide a competitive boost.

Experimental Setup

Dataset

The study utilized the NTCIR-17 ULTRE-2 dataset, derived from real-world Baidu search logs. The dataset includes:

  • Features: Traditional word-matching features (TF-IDF, BM25) and 768-dimensional BERT embeddings.
  • Heuristics: We extracted 24 additional heuristic features, such as language model smoothing, to calculate scores for the new query-document pairs generated during our negative sampling process.

Results and Discussion

Our experimental results demonstrate the efficacy of addressing false negatives directly. The Scratch-DLA-LC (sig) model achieved the highest performance on the hidden test set with an nDCG@10 of 0.5355. This represents a significant improvement over the standard DLA baseline (0.5247) and the competition organizers' best reported baseline (0.5217).

Key insights from our experiments include:

  • The Impact of Self-Correction: Using a model to relabel its own training data (DLA-LC) significantly improved performance. The "sigmoid" variant proved more effective than the "minimum" threshold variant, suggesting that a continuous relevance probability is more informative than a binary threshold.
  • The Necessity of Hard Negatives: Training solely on search log data is insufficient for high-precision ranking. Incorporating external hard negatives via BM25 allows the model to better differentiate between subtle levels of document relevance.
  • Validation Consistency: Performance on the human-annotated validation set was a reliable predictor of test set success, validating our iterative development process.

Conclusion and Future Work

This work demonstrates that addressing false negatives is critical for advancing ULTR in mature search environments. While our DLA-LC approach yielded top-tier results, several avenues for improvement remain. Due to time constraints, we were unable to generate high-quality BERT embeddings for the newly sampled query-negative pairs, relying instead on traditional features. Future work will focus on integrating large-scale pre-trained models more deeply into the negative sampling process to further bridge the gap between click-based models and human-level relevance judgment.

Summary updated

BibTeX
@inproceedings{yu2023cir,
  author = {Lulu Yu and Keping Bi and Jiafeng Guo and Xueqi Cheng},
  title = {{CIR at the NTCIR-17 ULTRE-2 Task}},
  booktitle = {NTCIR},
  year = {2023},
  url = {https://openreview.net/forum?id=wtlqXlYIXT},
}
Figure