TL;DR
This paper tackles the challenge of improving proactive information card recommendations (like those in Google Now or Siri) by leveraging users' past reactive search history. They introduce a novel reranking model, "Carré," that significantly boosts the relevance of these proactive cards by considering user context, temporal dynamics, and crucially, the topical consistency between past searches and current card suggestions.
What Problem Does This Paper Address?
Mobile devices have revolutionized information access, leading to the rise of "proactive" systems that anticipate user needs and present relevant content (information cards) without explicit queries. While these systems are gaining popularity, there's a significant knowledge gap regarding how users interact with them and how to effectively rank these cards. Traditional search relies on explicit queries, but proactive systems operate in a "zero-query" paradigm, making it harder to infer user intent. This paper aims to bridge this gap by studying user engagement with proactive cards and developing a method to improve their ranking by incorporating valuable signals from users' existing reactive search behavior.
Methods
The authors propose a supervised framework called "Carré" for reranking proactive cards. The core idea is to treat proactive card recommendation as a ranking problem and use machine learning to optimize the order of cards presented to a user.
Key methodological contributions include:
- Pseudo-relevance Labeling: Since explicit relevance judgments are scarce for proactive cards, they employ two strategies:
- SAT-Clicks: Similar to reactive search, clicks on cards with a dwell time of at least 30 seconds are considered relevant.
- SAT-Views: Recognizing that some cards (like weather) might satisfy user needs without a click, they use viewport duration (time a card is visible on screen) as a proxy for relevance, again with a 30-second threshold.
- SAT-Hybrid: A combination of the above two.
- Feature Engineering: Carré utilizes a rich set of features, categorized as:
- Reactive History: Features derived from the user's past web searches (e.g., number of past searches, domains matched).
- Proactive History: Features from recent proactive card interactions.
- Lexical/Topical Features: Similarity between card content (URLs, ODP categories) and documents clicked in past searches.
- Local/Temporal Features: Time of day, day of week, and user location.
- Constant Features: Intrinsic properties like card position and card type.
- Learning to Rank Model: They use LambdaMART, a gradient boosted decision tree algorithm, to learn the optimal ranking function based on these features and pseudo-relevance labels.
Datasets
The study utilizes logs from a commercial search engine (Microsoft Cortana for proactive data and Bing for reactive data).
- Proactive Data: 365,612 unique users, with 2,663,472 proactive impressions collected over a month.
- Reactive Data: 4,113,977 reactive search impressions from the same users, collected over a non-overlapping period to avoid temporal bias.
The paper does not introduce a new benchmark dataset but rather uses real-world logs to analyze user behavior and train their model.
Key Insights and Ideas
- Proactive and Reactive Search Share Similarities: User interaction patterns with proactive cards (like click distribution and temporal/local dynamics) mirror those observed in traditional reactive search, suggesting that existing IR principles can be adapted.
- Topical Consistency Across Platforms: A crucial finding is that users tend to interact with similar topics across proactive cards and reactive searches, regardless of the device. This consistency is a strong signal for personalization.
- Reactive History is a Powerful Predictor: The user's past reactive search history provides significant value for reranking proactive cards, even for new users of proactive systems. This is a key differentiator from purely context-aware proactive systems.
- "Good Abandonment" is Relevant: For some proactive cards (e.g., weather), user satisfaction can be achieved without a click. Using viewport duration as a proxy for relevance is a novel and important adaptation for this scenario.
- Position Still Matters: Similar to web search, the position of a card in the recommendation list heavily influences its click-through rate.
Key Results
- Carré Outperforms Baselines: The proposed Carré model consistently outperforms both the default production ranker and a static ranking baseline. Gains in Mean Reciprocal Rank (MRR) range from 6% to 10% across different labeling strategies.
- Feature Importance:
- Constant features (especially position) are the most influential.
- Reactive history features are as effective as proactive history features, highlighting the value of past searches.
- Lexical/Topical features are critical, confirming the importance of topical consistency across platforms.
- Local/Temporal features have the least impact, suggesting they are less card-specific.
- News Click Prediction: When focusing solely on news cards (which are always clickable), Carré achieved an accuracy of 0.8050, with significant drops when key feature groups were excluded.
Limitations and Future Work
- Pseudo-Relevance Labels: The authors acknowledge that click-based and viewport-based labels are proxies and may not perfectly capture true relevance for all card types. They suggest further research into more robust labeling strategies.
- Threshold Sensitivity: The 30-second threshold for dwell time and viewport duration was based on prior work and might need adjustment for proactive mobile settings.
- User Studies: While experiments were conducted on large-scale logs, the authors suggest validating findings with live users or user studies.
- Long-Term History: Investigating the impact of longer and more consistent historical data for both reactive and proactive interactions is a promising future direction.
- Bidirectional Influence: Exploring whether proactive history can also improve reactive search ranking is an interesting avenue for future research.
Summary updated