Abstract
Personalized search aims to tailor results to individual user needs, particularly for ambiguous queries. This paper evaluates various personalization strategies using a large-scale MSN query log dataset. We find that while click-based methods consistently improve results, profile-based methods can be unstable. A key metric, "click entropy," is identified as a robust indicator for when personalization is beneficial versus potentially harmful.
Introduction
Traditional search engines typically return a uniform set of results for a given query, regardless of the user's specific intent. For example, a biologist searching for "mouse" has vastly different requirements than a computer gamer. While numerous personalization algorithms have been proposed, the research community has historically lacked a standardized, large-scale framework for evaluation.
This study addresses two primary challenges: scaling evaluation beyond expensive, small-scale user studies and identifying "selective personalization." The authors hypothesize that applying personalization uniformly—especially to navigational queries like "Facebook"—may actually degrade the user experience rather than improve it.
Personalization Strategies
The authors developed a re-ranking evaluation framework using click-through data as a proxy for relevance. Five distinct strategies were tested:
- P-Click (Person-level Click): Re-ranks results based on a specific user's historical clicks for that exact query.
- G-Click (Group-level Click): A collaborative approach using the click history of similar users via K-Nearest Neighbors.
- L-Profile (Long-term Profile): Builds a topical interest profile based on 11 days of user history across 67 categories.
- S-Profile (Short-term Profile): Focuses on clicks within the current search session to capture immediate intent.
- LS-Profile: A linear combination of long-term and short-term profiles.
Performance was measured using Rank Scoring and Average Rank metrics, with Borda ranking fusion used to integrate personalized scores with original search engine rankings.
Dataset Analysis
The study utilized 12 days of MSN search engine query logs from August 2006, involving 10,000 sampled users. The data was split into 11 days for training (profile building) and one day for testing, totaling approximately 55,937 queries and 93,566 clicks.
As shown in Figure 3, the dataset exhibits a wide range of user activity levels. Most users have relatively few historical query days and times, highlighting the sparsity challenge inherent in building robust long-term profiles.
The Personalization Sweet Spot
A major contribution of this work is the identification of Click Entropy as a deciding factor for personalization. Click entropy measures the degree of disagreement among users regarding which result is most relevant for a query.
- Low Entropy: Users consistently click the same link (e.g., "Google"). Personalization here is often redundant or harmful.
- High Entropy: Users click many different links (e.g., "mouse"). This is where personalization provides the most value.
Furthermore, the effectiveness of the S-Profile suggests that immediate context is vital. As Figure 6 illustrates, most search sessions are brief, containing only one or two queries. Capturing intent within these short windows is often more effective than relying on potentially "noisy" long-term history.
Key Results
- Click-based Superiority: Simple click-based methods (P-Click and G-Click) consistently outperformed the standard web baseline. On queries where users did not click the first result, P-Click showed a 3.68% improvement in Rank Scoring.
- High-Entropy Gains: For queries with high click entropy (>1.5), G-Click achieved a significant 23.37% improvement over standard search.
- Profile Instability: Profile-based methods (L-Profile, S-Profile) were found to be unstable. While they offered small improvements for many queries, their occasional large errors resulted in average performance that was worse than the baseline.
Conclusion and Future Work
The study concludes that while personalization is powerful, it must be applied selectively. Click-based strategies are robust, whereas profile-based methods require more sophisticated topic modeling to avoid the "noise" of irrelevant past interests. Future research should focus on real-time "switching" mechanisms that use click entropy to decide when to invoke personalized ranking, alongside addressing the significant privacy concerns associated with logging detailed user histories.
Summary updated