TL;DR
This paper introduces Lumi Social News, a mobile app that moves beyond traditional recommender systems by learning user interests implicitly from browsing history and Twitter activity. It offers a personalized feed of diverse, crowd-curated content, continuously adapting based on user interactions like swiping to like or skip articles, effectively turning the user community into content curators.
What Problem Does This Paper Address?
The proliferation of online content has made it challenging for users to discover relevant information. Traditional recommender systems often rely on users explicitly defining their interests from predefined lists, which can be tedious and limiting. Furthermore, many existing systems focus on content from specific publishers, neglecting the vast amount of diverse, long-tail, and user-generated content available. This paper addresses the need for a more intuitive and dynamic personalization system that learns user preferences implicitly and leverages the collective intelligence of a user community for content curation.
Methods
Lumi Social News employs a graph processing architecture, similar to Twitter's Storm, for real-time processing. Key components include:
- Ingestion: A multi-stream ingestion process that collects content from RSS feeds, Twitter, and public browsing history. Content undergoes quality filtering, feature extraction, and categorization using SVM classifiers. Named entities are extracted using NLTK and OpenNLP.
- Bootstrapping: At signup, an initial user model is built by analyzing public browsing history and Twitter feeds, creating separate models for each to maintain privacy.
- User Model Update: The user model is continuously updated based on online activities (Twitter feed) and in-app interactions (liking, skipping, sharing recommended items).
- Trending Content Detection: Identifies trending stories by monitoring social media likes and tweets, and uses clustering to detect breaking news published across multiple outlets.
- Offline Recommender: A background process that uses Content-Based (CB) and Collaborative Filtering (CF) recommenders to calculate relevance scores and store top recommendations for users.
- Online Recommender: When a user requests recommendations, the system retrieves top-ranked items from the offline store, trending stories, and recently ingested content, considering relevance, freshness, and popularity.
- Related Content: Suggests articles based on content similarity and topical similarity, as well as other articles from the same site.
- Suggested User to Follow: Recommends users to follow based on the relevance of their liked items to the current user or existing Twitter connections.
Datasets
The paper does not explicitly list specific datasets used for training or evaluation. It mentions using "public pages from user visits" and "public Twitter feed" as sources for user interest learning, and "content from RSS, Twitter and public pages" for ingestion. The system is being evaluated through online experiments with real users using an A/B testing framework.
Key Insights and Ideas
- Implicit Learning is Key: Moving away from explicit topic selection significantly lowers the barrier to entry for users and allows for more nuanced interest discovery.
- Crowd as Curator: The collective actions of users (liking/skipping) effectively curate the content feed, democratizing the recommendation process.
- Privacy-Conscious Design: Maintaining separate models for browsing history and Twitter data, and only linking them when a recommendation is requested, is a crucial step for user privacy.
- Dynamic Adaptation: Continuous learning from user interactions ensures the recommender system remains relevant and adapts to evolving user interests.
- Diverse Content Inclusion: The system's ability to ingest and recommend long-tail and user-generated content broadens the scope beyond traditional publisher-centric systems.
Key Results
The paper focuses on the system's design and methodology rather than presenting specific quantitative results. It mentions that the system is currently undergoing online experiments with real users, and the quality of recommendations is being tracked through metrics like likes, skips, and reading times.
Limitations and Future Work
- Noisy and Sparse Data: The authors acknowledge that a key challenge is assessing how recommender systems perform with noisy and sparse data, especially with online user feedback.
- A/B Testing: The paper states that they are conducting online experiments with real users using an A/B testing framework to evaluate different recommender algorithms, trending story detection methods, and UI features. This implies ongoing work to refine and validate the system's performance.
- Future Work: While not explicitly detailed, the ongoing A/B testing suggests future work will focus on optimizing the algorithms and UI based on experimental results.
Summary updated