TL;DR
This paper tackles the challenge of accurately classifying sentiment in short, noisy microblog posts by leveraging the sentiment expressed in related news articles. They introduce a novel approach using distributional semantics (word embeddings) to better link and assign news sentiment to microblogs, significantly improving classification performance.
What Problem Does This Paper Address?
In today's digital age, social media platforms like Twitter have become major conduits for news consumption and discussion. However, the brevity and informal nature of microblog posts often make sentiment analysis difficult. While news articles provide a richer source of sentiment, simply averaging news sentiment for a given entity might not accurately reflect microblog sentiment due to differing contexts or the presence of unrelated information. This paper addresses the gap in understanding how to effectively transfer and adapt sentiment from news to microblogs, especially when the textual content isn't directly related beyond a shared entity. They aim to improve microblog sentiment classification by intelligently linking and assigning news sentiment, filtering out noise and focusing on relevant signals.
Methods
The core of this research lies in applying distributional semantics, specifically word embeddings, to bridge the gap between news and microblog sentiment. They build upon previous work that used an entity-based approach to link news sentiment to microblogs. Their novel contribution is the development of "context-based approaches" that go beyond simple entity matching.
Here's a breakdown of their methods:
- Distributional Semantics: They utilize word embeddings (derived from GloVe and converted to Word2Vec) to represent words in a vector space. This allows them to measure semantic similarity between texts.
- Textual Similarity Measures:
- Cosine Similarity: Used to measure the similarity between document embeddings (averaged word embeddings) of microblogs and news articles.
- Word Mover's Distance (WMD): A more sophisticated metric that measures the "distance" between two documents based on the minimum cumulative distance required to move words from one document to match words in the other.
- Context-Based Approaches: These are the key innovations. Instead of just using entity matching, they incorporate textual similarity to:
- Linking: Determine which news articles are truly relevant to a microblog's sentiment.
-
Assigning: Model how news sentiment influences microblog sentiment. They propose four context-based approaches:
-
Threshold-based Cosine Similarity: News articles with a cosine similarity above a certain threshold (0.5) are considered relevant. Their sentiments are averaged.
- Weighted Cosine Similarity: All news articles are considered, but their sentiments are weighted by their cosine similarity score.
- Threshold-based WMD: Similar to approach 1, but using WMD with a threshold.
- Weighted WMD: Similar to approach 2, but using WMD-based similarity as a weighting factor.
- Support Vector Machine (SVM): The final sentiment classification is performed using an SVM, with features derived from the microblog text and the context-based assigned news sentiment.
Datasets
The research utilizes two primary datasets:
- Dataset M: A microblog dataset containing posts from Twitter and StockTwits. This dataset was originally created for the SemEval 2017 Task 5.
- Dataset MRN: A news dataset containing news titles, URLs, timestamps, and sentiment scores. Crucially, all news in MRN are related to at least one microblog in dataset M.
They also created subsets of Dataset M:
- Subset A: Microblogs from Dataset M that have a relation to one or more news articles in MRN.
- Subset B: Microblogs from Subset A that were specifically retrieved from Twitter (to account for StockTwits not being collected in the same period as tweets).
Key Insights and Ideas
- Beyond Entity Matching: Simply linking news and microblogs based on shared entities is insufficient. Textual similarity is crucial for filtering out irrelevant news and accurately transferring sentiment.
- WMD's Potential: The Word Mover's Distance, while computationally more intensive, shows promise in capturing nuanced semantic relationships, leading to improved sentiment assignment.
- Context Matters: Context-based approaches, which consider the semantic relatedness between news and microblogs, significantly outperform simpler entity-based methods.
- Noise Reduction: Distributional semantics effectively helps in reducing the impact of noisy or unrelated news on microblog sentiment classification.
- Surprising Performance on Subset B: The context-based approaches, particularly WMD-based ones, showed the most significant improvements on Subset B, which had fewer microblogs but a higher proportion of related news. This suggests that when entity-based sentiment has less impact, context becomes paramount.
Key Results
The paper demonstrates significant improvements in microblog sentiment classification by incorporating context-based news sentiment.
- Overall Improvement: For Dataset M, the weighted F1-Score increased by 0.17% and Euclidean distance decreased by 7.04% when using context-based approaches compared to entity-based sentiment.
- Comparison to Message Text Only: Context-based approaches improved the weighted F1-Score by 3.13% and decreased Euclidean distance by 13.99% compared to using only microblog message text.
- Subset Performance: Subsets A and B also saw improvements in weighted F1-Score (1.06% and 3.07% respectively) and decreases in Euclidean distance (1.82% and 8.25% respectively).
- Outperforming Previous Work: Their best model achieved a cosine similarity of 0.869 on Dataset M, outperforming the best score of 0.778 reported by Jiang et al. (2017) in the SemEval 2017 Task 5 competition.
- Best Performing Approaches: For Dataset M and Subset A, threshold-based cosine similarity and WMD-S performed best. For Subset B, threshold-based WMD-S achieved the best scores.
Limitations and Future Work
- Dataset Size and Period: The authors acknowledge that the dataset is relatively small and covers a specific period. Future work aims to create a larger dataset covering a single, defined period.
- Hybrid Models: They suggest that hybrid models, combining thresholding and weighting schemes, could further improve classification.
- Similarity Measures: While WMD and cosine similarity were explored, other potentially adequate methods for measuring similarity still require investigation.
- Computational Cost: WMD is computationally more expensive than cosine similarity, which might be a consideration for real-time applications.
- Balanced Sentiment Classes: While they balanced sentiment classes, the impact of imbalanced real-world sentiment distributions on their models is not deeply explored.
Summary updated