TL;DR
Researchers have developed QASS (Query-oriented Data Augmentation for Session Search), a framework that improves document ranking by generating "negative" search contexts. By strategically altering the current query within a search session, the model learns to distinguish subtle shifts in user intent, significantly outperforming existing state-of-the-art session search models.
What Problem Does This Paper Address?
In session search, a user doesn't just type one query; they engage in a series of interactions (queries and clicks) to find information. To rank documents effectively, models must understand the "context" of the entire session.
The standard way to train these models is to take a fixed search context and pair it with a "clicked" document (positive) and an "unclicked" document (negative). The authors argue this is one-sided. Relevance is symmetric: just as a context has a best document, a document has a "best" matching context. Existing models often fail to realize that a document's relevance can vanish if the current query changes even slightly. This leads to models that are "lazy" in how they process the current query relative to the session history.
Methods
The authors propose QASS, which keeps the clicked document fixed but generates alternative, "negative" versions of the current query ($q_c$). This forces the model to learn why a document was relevant to the original query but not to a modified one.
They use two levels of augmentation:
- Term-level Modification: Subtle changes like masking a word, replacing a word with a random one, or adding a new word. This teaches the model to handle fine-grained nuances.
- Query-level Replacement: Replacing the entire query with:
- Random Queries: Easy negatives from other sessions.
- Historical Queries: Queries from earlier in the same session (medium difficulty).
- Ambiguous Queries: The "secret sauce." They mine queries from the logs that result in similar document rankings to the current query but are not identical. These serve as "hard negatives."
The Architecture: QASS uses BERT as its backbone. It concatenates the session history, the current query, and the candidate document into a single sequence. To handle the different types of augmented data, they use a pair-wise hinge loss with different "margins" (score gaps) based on the difficulty of the negative sample (e.g., a larger margin for random queries, a smaller one for ambiguous ones).
Datasets
The model was evaluated on two major public search logs:
- AOL: A massive, long-standing English search log dataset.
- Tiangong-ST: A Chinese commercial search engine log containing 18 days of user sessions with human-annotated relevance labels for the test set.
Key Insights and Ideas
- Symmetry of Relevance: The core "aha" moment is treating the search context as a variable rather than a constant. By showing the model "near-miss" contexts for a specific document, it becomes much more sensitive to the user's actual intent.
- The Importance of the "Current" Query: Preliminary experiments showed that removing the current query from the context dropped performance by over 13%, far more than removing historical queries. QASS doubles down on this by specifically augmenting the current query.
- Hard Negatives Matter: Not all data augmentation is equal. Replacing a query with a random one helps a little, but replacing it with an "ambiguous" query (one that looks similar to a retriever but has a different intent) provides the most significant boost in ranking accuracy.
Key Results
QASS achieved new state-of-the-art results across both datasets:
- AOL Dataset: QASS achieved an NDCG@1 of 0.4266, a significant jump over the previous best model, ASE (0.4144), and the popular COCA model (0.4024).
- Tiangong-ST Dataset: It reached an NDCG@10 of 0.9010, outperforming all baselines.
- Ablation Studies: Removing the "Ambiguous Query" mining resulted in the largest performance drop among all augmentation strategies, proving that "hard" negatives are crucial for deep learning models in IR.
Limitations and Future Work
- Training Cost: Because QASS generates many supplemental pairs (up to 11x more than standard training), it requires significantly more training time—roughly 11 times that of a naive BERT model. However, inference time remains identical to standard BERT models, making it practical for real-world deployment.
- Heuristic-Based: The current term-level modifications (masking/replacing) are somewhat random. The authors suggest that future work could use attention scores to "smartly" mask the most important words.
- Cold Start: The current augmentation strategy relies on having a session history. The authors aim to develop a specific strategy for "ad-hoc" queries (the very first query in a session) in the future.
Summary updated