TL;DR
This paper introduces a novel embedding-based approach to search personalization that maps users, queries, and documents into a shared "topical interest space." By using user-specific projection matrices and embeddings inspired by knowledge graph models, the authors demonstrate a significant improvement in re-ranking search results compared to traditional topic modeling and learning-to-rank baselines.
What Problem Does This Paper Address?
Search personalization aims to tailor results to an individual's specific needs based on their history. However, doing this effectively is notoriously difficult. Traditional methods often rely on building user profiles from the topics of documents a user has clicked on in the past.
The authors identify two major hurdles in current research:
- Direct Utility: Simple topic-based profiles often provide only marginal improvements or, in some cases, actually degrade search performance because they ignore how a user’s specific interest links a query to a document.
- Feature Obscurity: Many state-of-the-art systems plug user profiles into complex "Learning-to-Rank" (L2R) frameworks alongside dozens of other features. This makes it nearly impossible to determine if the user profile is actually contributing to the performance gain or if the model is just picking up on other signals.
The goal of this paper is to create a user profile that is rich enough to be used directly for re-ranking search results, providing a clear measure of its effectiveness.
Methods
The authors propose a three-step embedding approach:
- Base Embeddings via LDA: They use Latent Dirichlet Allocation (LDA) to learn $k$ topics from a collection of documents. A document's embedding ($v_d$) is its probability distribution over these topics. A query's embedding ($v_q$) is calculated as a weighted mixture of the topic distributions of the top-ranked documents returned for that query.
- The User Interest Subspace: This is the core innovation. Instead of comparing $v_q$ and $v_d$ directly, they represent each user $u$ with two projection matrices ($W_{u,1}, W_{u,2}$) and a user embedding vector ($v_u$).
- The matrices project the query and document into a subspace specific to that user's interests.
- The vector $v_u$ captures the relationship between the query and document within that subspace.
- The Score Function: Inspired by knowledge base embedding models (like TransE and STransE), they define an "implausibility" score: $$f(q, u, d) = ||W_{u,1}v_q + v_u - W_{u,2}v_d||$$ The model is trained using a margin-based loss function, essentially teaching the system that "correct" triples (query, user, clicked document) should have a lower implausibility score than "incorrect" ones.
Datasets
The study uses a real-world dataset consisting of query logs from a major commercial web search engine:
- Scope: 106 anonymous users over 15 days in July 2012.
- Content: The logs include user IDs, queries, top-10 ranked URLs, and click data (including dwell time).
- Ground Truth: The authors use the "SAT criteria"—a click is considered relevant if the user stays on the page for at least 30 seconds or if it is the final click of a search session.
- Scale: After pre-processing, the training set contained 5,658 correct triples, with roughly 1,200 triples each for validation and testing.
Key Insights and Ideas
- Users as Relations: The most clever insight is treating the user as a "relationship" between a query and a document, similar to how "capital_of" relates "Paris" to "France" in knowledge graphs. This allows the model to capture nuance that simple similarity scores miss.
- Topical Projection: By using matrices ($W_u$), the model acknowledges that different users might prioritize different aspects of the same topic. A "Python" query for a programmer vs. a snake enthusiast requires different projections of the same underlying topical data.
- Direct Re-ranking as a Litmus Test: By using the profile to re-rank results directly (rather than as one of many features in an L2R model), the authors prove that their embedding captures genuine user intent.
Key Results
The proposed method was tested against the original search engine ranking (SE), a personalized navigation baseline (CI), and a strong short-term profile baseline (SP).
- Mean Reciprocal Rank (MRR): The embedding approach achieved an MRR of 0.656, a 17.3% relative improvement over the original search engine and a 4% improvement over the strong SP baseline.
- Precision at 1 (P@1): The results were even more pronounced here, with a score of 0.501, representing a 30.3% relative improvement over the original search engine.
- Ablation Study: A simplified version of the model (using identity matrices instead of learned projection matrices) still performed well but was significantly outperformed by the full model, validating the need for user-specific topical projections.
Limitations and Future Work
- User Scale: The study was limited to 106 users. While the results are statistically significant, testing on a larger, more diverse user base would be necessary for commercial deployment.
- LDA Dependency: The model relies on LDA for initial embeddings. While effective in 2017, modern transformer-based embeddings (like BERT or RoBERTa) would likely yield even better results today.
- Cold Start: The paper focuses on users with existing history to build short-term profiles. It does not explicitly address how to handle entirely new users (the "cold start" problem).
- Computational Overhead: Learning two $k \times k$ matrices per user could become computationally expensive as the number of users scales into the millions.
Summary updated