Query-oriented Data Augmentation for Session Search

Haonan Chen, Zhicheng Dou, Yutao Zhu, Ji-Rong Wen

Abstract

Modeling contextual information in a search session has drawn more and more attention when understanding complex user intents. Recent methods are all data-driven, i.e., they train different models on large-scale search log data to identify the relevance between search contexts and candidate documents. The common training paradigm is to pair the search context with different candidate documents and train the model to rank the clicked documents higher than the unclicked ones. However, this paradigm neglects the symmetric nature of the relevance between the session context and document, i.e., the clicked documents can also be paired with different search contexts when training. In this work, we propose query-oriented data augmentation to enrich search logs and empower the modeling. We generate supplemental training pairs by altering the most important part of a search context, i.e., the current query, and train our model to rank the generated sequence along with the original sequence. This approach enables models to learn that the relevance of a document may vary as the session context changes, leading to a better understanding of users' search patterns. We develop several strategies to alter the current query, resulting in new training data with varying degrees of difficulty. Through experimentation on two extensive public search logs, we have successfully demonstrated the effectiveness of our model.

Introduction

In modern information retrieval, session search aims to improve ranking by leveraging the sequence of queries and interactions within a user's session. Traditional training paradigms for session search models typically focus on document-oriented negative sampling. In this setup, a fixed search context (the history of queries and clicks) is paired with a clicked document (positive) and an unclicked document (negative). The model is then trained to rank the clicked document higher.

However, this approach overlooks the symmetry of relevance. Just as a specific context makes certain documents relevant, a specific document is only relevant to certain contexts. Existing models often become "lazy," relying heavily on session history while failing to capture how subtle changes in the current query ($q_c$) should drastically alter document relevance. To address this, we introduce QASS (Query-oriented Data Augmentation for Session Search), which shifts the focus to generating "negative" search contexts for a fixed positive document.

Comparison between existing training paradigms and the proposed query-oriented augmented pairs.

The QASS Framework

The core innovation of QASS is the generation of alternative versions of the current query within a session. By keeping the clicked document fixed and modifying the query, the model is forced to learn the precise relationship between the query's intent and the document's content.

Augmentation Strategies

We employ two levels of augmentation to create a diverse set of training pairs with varying difficulty:

  1. Term-level Modification: These are fine-grained changes designed to teach the model nuance. Strategies include masking a term, replacing a term with a random word, or adding a new word to the query.
  2. Query-level Replacement: These are coarser changes. We replace the current query with random queries from other sessions (easy), historical queries from the same session (medium), or ambiguous queries (hard). Ambiguous queries are mined from logs to find queries that result in similar document rankings but represent different intents.
Examples of different query augmentation strategies and their associated difficulty levels.

The model uses a BERT backbone to process the concatenated sequence of history, query, and document. Training is conducted using a pairwise hinge loss, where the margin is adjusted based on the difficulty of the augmented negative sample.

Experimental Evaluation

We evaluated QASS on two large-scale public datasets: AOL (English) and Tiangong-ST (Chinese).

Key Results

QASS consistently outperformed state-of-the-art baselines across both datasets. On the AOL dataset, QASS achieved an NDCG@1 of 0.4266, surpassing previous models like ASE (0.4144) and COCA (0.4024). On Tiangong-ST, it reached an NDCG@10 of 0.9010.

Ablation studies highlighted the critical role of the current query. Removing $q_c$ from the context resulted in a performance drop of over 13% (as seen in the MAP and MRR metrics), confirming it as the most vital component of the session context. Furthermore, the inclusion of "Ambiguous Queries" proved to be the most impactful augmentation strategy, demonstrating that hard negatives are essential for deep session understanding.

Conclusion

QASS demonstrates that query-oriented data augmentation is a powerful tool for session search. By treating the search context as a variable, the model develops a more robust understanding of user intent. While the augmentation increases training time (approximately 11x due to the volume of generated pairs), the inference time remains unchanged, making QASS highly suitable for production environments where low-latency document ranking is critical. Future work will explore more sophisticated masking strategies using attention mechanisms and addressing the "cold start" problem for initial session queries.

Summary updated

Abstract

Modeling contextual information in a search session has drawn more and more attention when understanding complex user intents. Recent methods are all data-driven, i.e., they train different models on large-scale search log data to identify the relevance between search contexts and candidate documents. The common training paradigm is to pair the search context with different candidate documents and train the model to rank the clicked documents higher than the unclicked ones. However, this paradigm neglects the symmetric nature of the relevance between the session context and document, i.e., the clicked documents can also be paired with different search contexts when training. In this work, we propose query-oriented data augmentation to enrich search logs and empower the modeling. We generate supplemental training pairs by altering the most important part of a search context, i.e., the current query, and train our model to rank the generated sequence along with the original sequence. This approach enables models to learn that the relevance of a document may vary as the session context changes, leading to a better understanding of users' search patterns. We develop several strategies to alter the current query, resulting in new training data with varying degrees of difficulty. Through experimentation on two extensive public search logs, we have successfully demonstrated the effectiveness of our model.
BibTeX
@article{chen2024query,
  author = {Haonan Chen and Zhicheng Dou and Yutao Zhu and Ji-Rong Wen},
  title = {{Query-oriented Data Augmentation for Session Search}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2024},
  url = {https://openreview.net/forum?id=Qq0RsTfaQL},
}
Figure