Overview of the NTCIR-16 Session Search (SS) Task

Jia Chen, Weihao Wu, Jiaxin Mao, Beining Wang, Fan Zhang, Yiqun LiuNTCIR 2022 · OpenReview

Abstract

The NTCIR-16 Session Search (SS) task marks a significant shift from traditional ad-hoc retrieval to context-aware search, providing the community with a large-scale, realistic benchmark for session-based ranking. By introducing two subtasks—Fully Observed (FOSS) and Partially Observed (POSS) session search—the task demonstrates that while modern transformer models like BERT excel when combined with session context, traditional baselines like BM25 remain surprisingly competitive.

Introduction

Most search evaluation benchmarks focus on "ad-hoc" retrieval, where a user submits a single query and receives a list of results. However, real-world search is rarely a single-shot event. Users often engage in search sessions, submitting multiple related queries as their information needs evolve or as they learn from initial results.

The NTCIR-16 SS task identifies three major gaps in previous session search research (such as the older TREC Session Tracks):

  1. Scale: Previous datasets were too small to effectively train data-hungry neural models.
  2. Realism: Many benchmarks relied on simulated users or crowdsourced tasks rather than natural, log-based search behavior.
  3. Evaluation: Standard metrics often fail to capture session-level effectiveness across multiple interactions.

Task Definition and Scenarios

The NTCIR-16 SS task was organized into two distinct scenarios to evaluate context-aware ranking:

  • Fully Observed Session Search (FOSS): Participants are provided with the full context of a session (previous queries and clicks) to optimize the ranking for the final query. This mimics a system trying to "get it right" at the end of a user journey.
  • Partially Observed Session Search (POSS): Sessions are truncated, and participants must improve document ranking for the last several queries using limited prior context, testing a system's ability to generalize intent.
Example of a search session log from the NTCIR-16 SS dataset showing the sequence of queries and associated document metadata.

Dataset Characteristics

The task provided a massive leap in data availability for the research community. The training set, based on TianGong-ST, contains over 147,000 real-world web search sessions from Sogou logs. The test sets were extracted from field study datasets (TianGong-SS-FSD and TianGong-Qref), totaling 1,817 sessions for FOSS and 1,203 for POSS.

Comparison of the NTCIR-16 SS dataset against previous TREC Session Tracks, highlighting the significant increase in session count and the use of real search logs.

Key Insights and Methodology

The Power of Context

A recurring theme was that utilizing search history and session context significantly boosts performance over treating the last query in isolation. Successful approaches typically combined BERT-based cross-encoders with session-level features, such as previous clicks and query reformulation patterns.

The "Distant History" Trap

In the POSS subtask, researchers found that including search history from queries far removed from the current one could actually degrade performance. This suggests that user intent shifts rapidly, and "recency bias" is a critical feature, not a bug, in session modeling.

Robustness of Traditional Models

A notable finding was the performance of BM25. Despite the prevalence of transformers, a well-tuned BM25 baseline (utilized by the SCIR team) outperformed several complex neural architectures, ranking 7th out of 12 in the final FOSS evaluation.

Results and Evaluation

The competition saw 28 runs from 6 teams, with the RUCIR team emerging as the top performer across both subtasks.

  • FOSS Results: The top run achieved an NDCG@10 of 0.5693, compared to the best "naive" BM25 run of 0.4309.
  • POSS Results: RUCIR led with an RS_DCG of 0.7466.

Evaluation also revealed that preliminary results based on user clicks differed significantly from final evaluations based on expert labels, highlighting the strong influence of position bias in real-world user behavior.

Conclusion and Future Work

While NTCIR-16 SS provides a robust benchmark, several areas for future work remain. These include addressing data sparsity in the POSS task, expanding to multilingual datasets to see if session behaviors generalize across languages, and developing more sophisticated session-level metrics that better account for user effort and information gain across an entire session.

Summary updated

BibTeX
@inproceedings{chen2022overview,
  author = {Jia Chen and Weihao Wu and Jiaxin Mao and Beining Wang and Fan Zhang and Yiqun Liu},
  title = {{Overview of the NTCIR-16 Session Search (SS) Task}},
  booktitle = {NTCIR},
  year = {2022},
  url = {https://openreview.net/forum?id=tyKb7zBNYF},
}
Figure