TL;DR
This paper presents an ensemble approach for detecting sarcasm in social media, combining multiple models to achieve better performance than any single model. By leveraging conversational context and a variety of features, their ensemble achieved impressive F1-scores of 67% on Reddit and 74% on Twitter, highlighting the importance of context and model diversity.
What Problem Does This Paper Address?
Sarcasm detection in social media is a notoriously difficult task. Sarcasm is often subjective, culturally dependent, and relies heavily on context for accurate interpretation. An utterance that seems straightforward in isolation can be deeply sarcastic when viewed within its conversational thread. Existing sarcasm detection models often struggle with this nuance, leading to lower performance. This paper aims to build a more robust sarcasm detection system by combining the strengths of multiple models and explicitly incorporating conversational context.
Methods
The core of this work is an ensemble learning approach. The authors built an ensemble classifier using an AdaBoost algorithm with decision trees as the base estimator. This ensemble doesn't just combine predictions from other models; it learns when each component model is most reliable.
The ensemble is trained on the predicted sarcasm probabilities from four diverse component models:
- LSTM with hashtag and emoji representations: This model uses a bidirectional LSTM to capture sequential information and explicitly encodes hashtags and emojis, which are common indicators of sarcasm.
- CNN-LSTM with various features: This model combines convolutional neural networks (CNNs) and LSTMs. It incorporates word embeddings along with features like punctuation, casing, sentiment (using NLTK's Vader), and stop words.
- MLP based on Infersent embeddings: This model leverages sentence embeddings from the Infersent model, which is trained on natural language inference tasks. The hypothesis is that Infersent's ability to spot logical discrepancies can help identify sarcasm.
- SVM with stylometric and emotion-based features: This model takes a different approach, focusing on linguistic patterns, part-of-speech tags, function words, and emotion lexicons, without using conversational context.
Crucially, all component models except the SVM utilize the two preceding conversational turns as context. The ensemble itself also incorporates additional features like comment length and source (Reddit/Twitter) to further refine its predictions.
Datasets
The study uses data from the Second Workshop on Figurative Language Processing held in conjunction with ACL 2020.
- Training Data: 5,000 tweets and 4,400 Reddit comments, automatically annotated.
- Test Data: 1,800 tweets and 1,800 Reddit comments, also automatically annotated.
Both training and test datasets were balanced. User mentions and URLs were anonymized. The authors note that participating teams were allowed to collect extra data, but their primary experiments used the provided dataset.
Key Insights and Ideas
- Ensemble Power: Combining multiple diverse models significantly outperforms any single model. This suggests that different models capture different aspects of sarcasm.
- Context is King (Still): The strong performance of models incorporating conversational context reinforces its critical role in understanding sarcasm, especially for utterances that are ambiguous in isolation.
- Reddit vs. Twitter Nuances: Reddit comments are consistently harder to classify than Twitter tweets across all models. This is hypothesized to be due to the longer, more complex nature of Reddit comments, making context interpretation more challenging for models.
- Feature Diversity Matters: The inclusion of various features (word embeddings, punctuation, sentiment, stylometric features, etc.) in the component models and the ensemble's meta-features (length, source) contributes to the overall performance.
Key Results
The ensemble approach achieved the following F1-scores on the official test data:
- Reddit: 67.0%
- Twitter: 74.0%
These scores are higher than any of the individual component models on both datasets. The paper also notes that the 10-fold cross-validation scores on the training data were substantially higher than the test set scores, indicating that the test data was indeed more challenging.
Limitations and Future Work
- Reddit's Complexity: The paper acknowledges the persistent performance gap between Reddit and Twitter data. Further research is needed to understand why Reddit comments are more challenging and how to better model their complexity.
- Model Contributions: While the ensemble improves performance, the paper suggests that removing any component model leads to a decrease, implying each contributes uniquely. However, a more detailed ablation study on the specific contribution of each model could be valuable.
- Future Work: The authors suggest experimenting with other component models to further enhance the ensemble's performance. They also implicitly point to the need for better methods to handle the nuances of longer, context-rich platforms like Reddit.
Summary updated