Leveraging User Behavior History for Personalized Email Search

Keping Bi, Pavel Metrikov, Chunyuan Li, Byungki ByunWWW 2021 · DOI · OpenReview

Abstract

An effective email search engine can facilitate users’ search tasks and improve their communication efficiency. Users could have varied preferences on various ranking signals of an email, such as relevance and recency based on their tasks at hand and even their jobs. Thus a uniform matching pattern is not optimal for all users. Instead, an effective email ranker should conduct personalized ranking by taking users’ characteristics into account. Existing studies have explored user characteristics from various angles to make email search results personalized. However, little attention has been given to users’ search history for characterizing users. Although users’ historical behaviors have been shown to be beneficial as context in Web search, their effect in email search has not been studied and remains unknown. Given these observations, we propose to leverage user search history as query context to characterize users and build a context-aware ranking model for email search. In contrast to previous context-dependent ranking techniques that are based on raw texts, we use ranking features in the search history. This frees us from potential privacy leakage while giving a better generalization power to unseen users. Accordingly, we propose a context-dependent neural ranking model (CNRM) that encodes the ranking features in users’ search history as query context and show that it can significantly outperform the baseline neural model without using the context. We also investigate the benefit of the query context vectors obtained from CNRM on the state-of-the-art learning-to-rank model LambdaMart by clustering the vectors and incorporating the cluster information. Experimental results show that significantly better results can be achieved on LambdaMart as well, indicating that the query clusters can characterize different users and effectively turn the ranking model personalized.

Introduction

Email search is an inherently personal task. Unlike general web search, where many users might seek the same public information, email searchers are navigating unique, private corpora. A salesperson might prioritize emails from specific clients, while an engineer might look for technical keywords, and a casual user might simply want the most recent message. Standard "one-size-fits-all" ranking models struggle because they apply a uniform matching pattern to everyone, failing to account for these individual preferences.

Personalizing email search presents unique challenges, primarily regarding privacy. Models cannot easily "read" a user's private inbox to understand their interests without risking data leakage. Furthermore, the lack of cross-user interaction—where different users rarely interact with the same document—makes traditional collaborative filtering difficult. This paper proposes bridging this gap by using a user's recent search history to characterize their "matching pattern" and adapt the ranker accordingly.

The Context-dependent Neural Ranking Model (CNRM)

The core innovation of this work is the Context-dependent Neural Ranking Model (CNRM). Unlike traditional models that only consider the current query and document, CNRM leverages the user's recent search history as a query context.

Architecture of the CNRM model, showing how historical query-document matching features are encoded via a Transformer to create a context vector.

To preserve privacy, the model avoids using raw text or n-grams. Instead, it extracts numerical ranking features (e.g., BM25 scores, email length, and recency) from the last $k$ queries and their associated clicked documents. These features are passed through a Transformer encoder to produce a dense "context vector" representing the user's current search state. This approach ensures the model learns how a user searches rather than what they are searching for, allowing it to generalize effectively even to users it has never seen before.

Experimental Setup

The study utilized a massive, proprietary dataset from a major email search engine, comprising hundreds of thousands of queries. The researchers focused on users with 10–20 queries to ensure sufficient history for building context. Relevance labels were derived from real-world user interactions, such as clicks, replies, and dwell time. To account for "position bias"—the tendency for users to click top results regardless of quality—the training process incorporated an examination model to weight samples appropriately.

Results and Discussion

Experimental results demonstrate that incorporating behavioral context significantly improves search relevance. The CNRM outperformed a non-contextual baseline (NRM) by 2.5% to 2.9% in NDCG metrics. Notably, the Transformer-based architecture consistently beat LSTM-based alternatives, suggesting that the specific sequence and relationship of recent searches provide a high-resolution signal of user intent.

t-SNE visualizations of context vectors. Subplot (d) demonstrates how these vectors can be clustered into distinct user cohorts to provide personalized ranking boosts.

One of the most compelling findings is the "Cohort Effect." By clustering the dense context vectors into discrete groups, the researchers identified distinct user behaviors. For example, some clusters showed a high sensitivity to email length while others prioritized recency. When these cluster IDs were integrated as features into a production-grade LambdaMart model, they provided a statistically significant improvement of approximately 0.6% in NDCG@3. While seemingly small, such gains are substantial in large-scale industrial search applications.

Conclusion

This research proves that user search history is a powerful, privacy-preserving signal for personalizing email search. By focusing on abstract ranking features rather than raw content, the CNRM model can adapt to individual user needs without compromising data security. Future work may explore longer-term search histories and more sophisticated methods for integrating dense neural context into traditional gradient-boosted decision tree models like LambdaMart.

Summary updated

Abstract

An effective email search engine can facilitate users' search tasks and improve their communication efficiency. Users could have varied preferences on various ranking signals of an email, such as relevance and recency based on their tasks at hand and even their jobs. Thus a uniform matching pattern is not optimal for all users. Instead, an effective email ranker should conduct personalized ranking by taking users' characteristics into account. Existing studies have explored user characteristics from various angles to make email search results personalized. However, little attention has been given to users' search history for characterizing users. Although users' historical behaviors have been shown to be beneficial as context in Web search, their effect in email search has not been studied and remains unknown. Given these observations, we propose to leverage user search history as query context to characterize users and build a context-aware ranking model for email search. In contrast to previous context-dependent ranking techniques that are based on raw texts, we use ranking features in the search history. This frees us from potential privacy leakage while giving a better generalization power to unseen users. Accordingly, we propose a context-dependent neural ranking model (CNRM) that encodes the ranking features in users' search history as query context and show that it can significantly outperform the baseline neural model without using the context. We also investigate the benefit of the query context vectors obtained from CNRM on the state-of-the-art learning-to-rank model LambdaMart by clustering the vectors and incorporating the cluster information. Experimental results show that significantly better results can be achieved on LambdaMart as well, indicating that the query clusters can characterize different users and effectively turn the ranking model personalized.
BibTeX
@inproceedings{bi2021leveraging,
  author = {Keping Bi and Pavel Metrikov and Chunyuan Li and Byungki Byun},
  title = {{Leveraging User Behavior History for Personalized Email Search}},
  booktitle = {The Web Conference},
  year = {2021},
  doi = {10.1145/3442381.3450110},
}
Figure