TL;DR
This paper tackles the challenge of accurately filtering relevant microblogging messages for social TV applications. They introduce a novel bootstrapping approach that leverages a small set of labeled data and a large corpus of unlabeled messages to iteratively improve a classifier, significantly boosting precision and recall compared to naive keyword matching.
What Problem Does This Paper Address?
Social TV, the integration of social media into television viewing experiences, is a rapidly growing trend. A key feature of these applications is the ability to display microblogging messages (like tweets) related to the TV program currently being watched. However, existing systems often rely on simple keyword matching using the TV show's title, which leads to poor results. This is because show titles can be ambiguous (e.g., "House" can refer to many things) and many relevant messages might not explicitly mention the title, instead referencing characters, actors, or hashtags. This results in low precision (irrelevant messages) and low recall (missed relevant messages). The paper aims to solve this by developing a robust filtering mechanism that can accurately identify and display only the most relevant microblogging content for any given TV show.
Methods
The core of the paper is a bootstrapping approach for building a microblogging message classifier. This method involves two main stages:
- Initial Classifier: They start with a small, manually annotated dataset (DT) where human annotators label messages as relevant or irrelevant to specific TV shows. Using this data, they train an initial binary classifier. The features for this classifier are designed to capture the association between a message and a TV show, including:
- TV-related terms: Presence of general terms like "watching," "episode," "Netflix," or network names.
- Season/Episode indicators: Patterns indicating a specific season and episode (e.g., "S06E07").
- General Positive Rules: Learned patterns that frequently appear in messages discussing TV shows (e.g., "<start> watching <show_name>").
- Show Title Features: Capitalization of show titles and matches with other known TV show titles.
- Domain Knowledge Features: Cosine similarity between the message and crawled information about actors, characters, or Wikipedia pages related to the show.
- Improved Classifier: This stage leverages a large dataset of unlabeled microblogging messages (DL). The initial classifier is used to automatically label these unlabeled messages. This expanded, automatically labeled dataset is then used to derive broader features for a second, improved classifier. These new features include:
- Positive/Negative Rules: More specific rules learned for individual shows (e.g., "episode of House" vs. "in the house").
- User/Hashtag Scores: Identifying frequently used users and hashtags associated with a particular show.
- Rush Period: Detecting spikes in message activity during a show's broadcast time.
This process can be iterated to further refine the features and improve classifier performance.
Datasets
- DT Dataset: A manually labeled dataset created by Amazon Turk workers. It contains 2,629 labeled messages for three TV shows with ambiguous titles: "Fringe," "Heroes," and "Monk." Messages were sampled based on the show's title and then annotated as relevant, irrelevant, or unsure/foreign language.
- DL Dataset: A large corpus of over 10 million unlabeled Twitter messages collected in October 2009 using Twitter's Streaming API (Gardenhose level). This dataset serves as the source for deriving broader features in the bootstrapping process.
Key Insights and Ideas
- Bootstrapping is Key: The paper demonstrates that a bootstrapping approach, iteratively learning from both labeled and unlabeled data, is highly effective for building robust classifiers in domains with limited initial labeled data.
- Beyond Simple Keywords: The success of the improved classifier highlights the inadequacy of naive keyword matching. Incorporating features related to show-specific context, user behavior, and temporal patterns is crucial for accurate filtering.
- Ambiguity is a Major Hurdle: The choice of ambiguous show titles for the DT dataset underscores a critical real-world problem that the proposed method effectively addresses.
- Feature Engineering Matters: The paper showcases how carefully designed features, ranging from linguistic patterns to domain-specific knowledge and social network dynamics, can significantly improve classification performance.
Key Results
- Initial Classifier Performance: Achieved a precision of 0.76 and an F-measure of 0.8 for identifying relevant messages ("Yes" label) on the DT dataset using 10-fold cross-validation.
- Improved Classifier Performance: Significantly boosted performance, reaching a precision of 0.89 and an F-measure of 0.89 for the "Yes" label on the DT dataset.
- Generalization: The improved classifier demonstrated good generalization capabilities, achieving an average precision of 0.84 and an F-measure of 0.85 for "Yes" label when tested on shows not included in the training set (leave-one-show-out evaluation).
Limitations and Future Work
- Limited Initial Training Data: While the bootstrapping method mitigates this, the initial classifier still relies on a small, manually labeled set. The authors acknowledge that obtaining training data for every show is prohibitively expensive.
- API Rate Limits: The paper mentions the challenge of collecting messages due to API rate limits and the need to find an optimal query strategy.
- Future Work: The authors suggest expanding the set of queries for each show by incorporating discovered hashtags and users to further improve recall. They also plan to explore using these features for other applications like measuring program popularity and performing sentiment analysis.
Summary updated