MM6 at the NTCIR-16 Session Search Task

Shengjie Ma, Chuwei Zeng, Jiaxin MaoNTCIR 2022 · OpenReview

Introduction

Standard search paradigms often treat user queries as isolated events. In practice, however, search is an iterative process where users refine their queries and interact with results over time. This "Session Search" context is critical for understanding true user intent. At the NTCIR-16 Session Search Task, the MM6 team from Renmin University of China investigated several advanced modeling techniques to capture these evolving relationships, focusing on two subtasks: Fully Observed Session Search (FOSS) and Partially Observed Session Search (POSS).

Modeling Search Context

The MM6 team explored three primary architectures for the FOSS task, ranging from sequential models to complex graph-based representations.

Sequential and Contrastive Approaches

The first approach, CARS (Context Attentive document Ranking and query Suggestion), employs a hierarchical RNN. It uses a two-level structure to first encode individual user actions (queries and clicks) and then aggregate them into a session-level representation.

The second approach, COCA (Contrastive learning for Context Aware document ranking), leverages the power of BERT. By applying data augmentation techniques—such as masking terms or reordering sequences—and using contrastive learning, COCA aims to build a robust model capable of understanding diverse user behavior patterns.

The Session Graph (SG) Model

The most innovative contribution was the Session Graph (SG) approach. Unlike linear models, SG treats a session as a heterogeneous graph where nodes represent queries, clicked documents, and keywords (extracted via TF-IDF). This structure allows the model to capture non-linear topological relationships within the search history.

The Session Graph (SG) architecture utilizes heterogeneous graph pooling (HGpool) to generate session embeddings.

As shown in the architecture, the model uses a multi-head self-attention network for node embeddings and a specialized HGpool (Heterogeneous Graph Pooling) mechanism. This process condenses the complex graph into a single session vector used for final document ranking.

Experimental Results

The models were trained using the NTCIR-16 Session Search Task data, with an 8:1:1 split for training, validation, and testing. During the training phase, "satisfied clicks" served as the ground truth labels.

Training Dominance of Graph Models

In internal training and validation, the Session Graph model showed exceptional promise. It significantly outperformed both the RNN-based CARS and the BERT-based COCA models across all standard retrieval metrics.

Internal training results showing the Session Graph (SG) model outperforming CARS and COCA across all metrics.

As illustrated in the training metrics, the SG model achieved an NDCG@10 of 0.8747, a substantial lead over COCA's 0.7613. This suggested that the graph-based approach was highly effective at capturing the patterns leading to user clicks.

The Click-Human Discrepancy

Despite the strong training performance of the Session Graph model, the official NTCIR-16 evaluation—which relied on human-annotated relevance labels rather than clicks—revealed a significant "reality gap."

In the official FOSS results, the COCA model (ranked 6th out of 12) outperformed the SG model (ranked 9th). The SG model's NDCG@10 dropped to 0.4029 in the official evaluation, compared to COCA's 0.4572. This discrepancy suggests that while the graph model was excellent at predicting user click behavior, those clicks do not always align perfectly with objective human relevance judgments.

Furthermore, in the POSS task, the team found that ignoring session history (window size 0) actually yielded better results (RS_DCG 0.423) than using a history of the last three queries (RS_DCG 0.379). This indicates that in many real-world sessions, queries may be independent or driven by external trends rather than a continuous refinement of a single goal.

Conclusion

The MM6 team's participation in NTCIR-16 highlights the potential of heterogeneous graph neural networks in modeling complex search sessions. However, the results also underscore a critical challenge in the field: the distribution mismatch between user click data and human relevance labels. Future work will focus on bridging this gap by incorporating human-labeled data earlier in the training process and refining graph optimization techniques to better generalize across different evaluation paradigms.

Summary updated

BibTeX
@inproceedings{ma2022mm,
  author = {Shengjie Ma and Chuwei Zeng and Jiaxin Mao},
  title = {{MM6 at the NTCIR-16 Session Search Task}},
  booktitle = {NTCIR},
  year = {2022},
  url = {https://openreview.net/forum?id=v86fMOcibe},
}
Figure