RUCIR at the NTCIR-16 Session Search (SS) Task

Haonan Chen, Zhicheng DouNTCIR 2022 · OpenReview

Abstract

This paper details the top-performing approach by the RUCIR group for the NTCIR-16 Session Search (SS) task. The methodology integrates the COCA neural ranker with traditional BM25 retrieval and human-annotated "usefulness" labels. This combination effectively addresses session noise and contextual ambiguity, leading to state-of-the-art results in both fully and partially observed search sessions.

Introduction

Traditional search engines often treat queries in isolation, failing to capture the evolving intent of a search session. Session search aims to model a sequence of queries and interactions to better predict user needs. However, this task is complicated by contextual ambiguity and "noise"—irrelevant clicks or skipped results. The RUCIR team addresses these challenges by denoising search history and grounding neural predictions with lexical signals.

Methodology

The proposed system is built upon three core components:

1. COCA (Contrastive Learning of User Behavior Sequence)

COCA is a two-stage neural model. During pre-training, it employs contrastive learning to develop robust session representations. The model generates augmented versions of sessions through term masking, query/document deletion, and behavior reordering. A BERT encoder is then trained to recognize these variations as representing the same underlying intent.

Data augmentation strategies for COCA pre-training.

2. BM25 Regularization

To mitigate overfitting in the neural model, COCA scores are blended with traditional BM25 scores. BM25 provides a lexical "sanity check," ensuring that the model remains grounded in exact keyword matches while the neural component handles semantic nuances.

3. Usefulness Filtering (+U)

Recognizing that clicks do not always indicate relevance, the authors utilized human-annotated usefulness labels. By filtering the session history to include only the most useful documents for each historical query, the model is trained on a "denoised" sequence of user behaviors.

Results

The performance of the proposed model was evaluated on the NTCIR-16 Session Search (SS) dataset, which includes Fully Observed (FOSS) and Partially Observed (POSS) subtasks. The "COCA + BM25 + U" configuration outperformed all other variants in the competition.

Performance comparison of model variants on FOSS and POSS subtasks.

As shown in the table above, the inclusion of BM25 and usefulness labels resulted in significant performance gains. In the FOSS task, the full model reached an NDCG@10 of 0.5693, compared to 0.4939 for the base COCA model. In the POSS task, it achieved an RsDCG of 0.5439 and an RsRBP of 0.7466, demonstrating the robustness of the hybrid approach.

Conclusion

The RUCIR team's results demonstrate that combining advanced neural architectures with traditional retrieval wisdom and high-quality data labels is highly effective for session search. Future research will explore optimizing the computational efficiency of the BERT-based ranker for real-time deployment.

Summary updated

BibTeX
@inproceedings{chen2022rucir,
  author = {Haonan Chen and Zhicheng Dou},
  title = {{RUCIR at the NTCIR-16 Session Search (SS) Task}},
  booktitle = {NTCIR},
  year = {2022},
  url = {https://openreview.net/forum?id=eicHTZZtTJ},
}
Figure