TL;DR
This paper introduces "groupization," a novel approach to enhance personalized web search by leveraging data from groups of users. By analyzing how users with shared traits or tasks interact with search results, they demonstrate that aggregating group preferences can significantly improve search result rankings, especially for queries relevant to the group's shared interests or tasks.
What Problem Does This Paper Address?
Personalized web search aims to deliver more relevant results by understanding individual user preferences. However, building rich user profiles for effective personalization often requires a substantial amount of data. This paper tackles the challenge of augmenting individual user profiles by exploring how data from related groups of people can be used to improve search. The core problem is identifying which groups are valuable for personalization and how to effectively combine their data, especially when explicit group membership is not readily available.
Methods
The authors propose "groupization," an extension of existing personalization algorithms. Instead of solely relying on an individual's profile, groupization aggregates personalization scores from multiple group members. This aggregation can be done by summing personalized scores for each result across group members. They also explore different ways to define and identify groups, categorizing them along two axes: longevity (short-term task-based vs. long-term trait-based) and identification (explicitly defined vs. implicitly inferred).
Key methods include:
- Data Collection: Two datasets were collected. Dataset 1 focused on trait-based groups (demographics, geography, occupation, interest) and Dataset 2 on task-based groups.
- Group Definition: Groups were defined using explicit criteria (e.g., job role, mailing list membership) and implicit indicators (e.g., query selection, desktop content similarity, relevance judgment similarity).
- Similarity Measures: Cosine similarity was used to compare user profiles (desktop indices), and Cohen's Kappa/Fleiss' Kappa for relevance judgments.
- Groupization Algorithm: An algorithm that sums personalized scores from group members to create a unified group ranking.
- Evaluation Metric: Normalized Discounted Cumulative Gain (nDCG) was used to measure the performance of personalization and groupization.
Datasets
- Dataset 1 (Trait-Based): 110 participants from Microsoft, with data on explicit group memberships (demographics, geography, occupation, interest groups) and implicit indicators (query selection, desktop content, relevance judgments).
- Dataset 2 (Task-Based): 30 participants from Microsoft, organized into 10 task-based groups of three people each, with shared work-related tasks and associated queries.
The paper does not introduce a new benchmark dataset but rather uses data collected for the specific purpose of this study.
Key Insights and Ideas
- Groupization Outperforms Personalization: Aggregating preferences from a group generally leads to better search results than relying solely on an individual's profile.
- Task-Based Groups are Highly Cohesive: Groups formed around a shared, short-term task exhibit the strongest agreement on relevance judgments, making them particularly valuable for groupization.
- Explicit Groups are Easier to Leverage: Explicitly defined groups (like occupational or interest groups) show more promise for groupization than implicitly identified ones.
- Relevance is Topic-Dependent: Group cohesion and the effectiveness of groupization are strongest when queries are directly related to the group's shared theme (e.g., work queries for occupational groups, photography queries for photography groups).
- Implicit Group Identification is Challenging: While desktop index similarity can predict group membership to some extent, it doesn't reliably predict similar relevance judgments, suggesting that diverse profiles within a group might actually be beneficial for groupization.
- Web Ranking as a Prior: Incorporating the existing web ranking as a prior further improves groupization performance, suggesting a complementary relationship between aggregate web signals and group-specific signals.
Key Results
- Overall Improvement: Groupization of all participants yielded a normalized DCG of 0.61, significantly better than individual personalization (0.55) and even the Web ranking (0.57).
- Trait-Based Groups: Groupization with trait-based groups provided an average 3.5% boost in nDCG over personalization. Performance was better for work queries (4.4% boost) than social queries (2.9% boost).
- Task-Based Groups: Groupization for task-based groups showed significant improvements, particularly for task-related queries. Within-group groupization improved results by 32% over the Web ranking (to 0.67).
- Implicit Groups: The study found little correlation between index similarity and relevance judgment similarity, suggesting that implicit group identification for groupization is challenging.
- Query Selection vs. Relevance Judgments: While query selection can indicate shared interests (especially for occupational and special-interest groups), it doesn't strongly predict similar relevance judgments.
Limitations and Future Work
- Implicit Group Identification: A significant limitation is the difficulty in reliably identifying valuable implicit groups, especially those with similar relevance judgments. The paper suggests that while index similarity can predict group membership, it doesn't guarantee similar relevance judgments.
- Generalizability of Implicit Measures: The study found that implicit measures like query selection and desktop similarity were not strong predictors of future relevance judgment agreement, limiting their direct use for groupization.
- Future Work: The authors suggest exploring techniques for identifying groups "in the wild" for commercial search engines. This includes leveraging collaborative search tools, social networking sites, and enterprise directories. They also propose further research into combining personalization and groupization scores in more sophisticated ways to achieve even larger performance gains.
Summary updated