TL;DR
This paper investigates whether incorporating sentiment analysis can improve sarcasm detection. The authors found that their attempts to use sentiment analysis as an additional feature for both deep learning (CNN) and linear models (SVM, Naive Bayes) actually decreased performance compared to models without sentiment analysis, suggesting that a simple approach to sentiment clash detection isn't sufficient.
What Problem Does This Paper Address?
Sarcasm is notoriously difficult to detect in text because it relies on saying the opposite of what you mean, often for humorous or critical effect. While humans can pick up on vocal cues and facial expressions, these are absent in written language. This makes automated sarcasm detection crucial, especially for moderating online content where sarcasm can be used to mask hate speech or offensive remarks. The core challenge this paper tackles is whether a common linguistic phenomenon in sarcasm – the "connotation clash" (where words with positive connotations are used to express negative sentiment, or vice-versa) – can be effectively leveraged by sentiment analysis to improve sarcasm detection models.
Methods
The researchers explored two main approaches: a Deep Learning model (Convolutional Neural Network - CNN) and linear models (Naive Bayes, Support Vector Machine - SVM, Random Forest).
For the Deep Learning approach, they built a CNN and experimented with adding sentiment analysis results as an extra feature. The sentiment analysis itself was performed using a pre-trained model from the Flair NLP library, which provided both a positive/negative classification and a confidence score. They then categorized sentences into six classes based on this sentiment and confidence, and encoded these as one-hot vectors to be fed into the CNN.
For the Linear Model approach, they focused on English tweets and employed a more involved preprocessing strategy to combat data imbalance. They extracted features using TF-IDF weighting on word n-grams (unigrams to 4-grams). For the sentiment analysis component, they trained a CNN on the IMDB movie review dataset and used its weights to predict sentiment for the sarcasm detection task. The sentiment predictions were then converted into a one-hot encoded vector and appended to the TF-IDF feature matrix. They then trained SVM, Naive Bayes, and Random Forest classifiers with and without these sentiment features.
Datasets
- Training Data: The study used two collections of tweets for the SemEval-2022 Task 6, Subtask A (sarcasm detection): one in English and one in Arabic. The English dataset had approximately 3468 sarcastic and 867 non-sarcastic tweets, while the Arabic dataset had 3102 sarcastic and 2357 non-sarcastic tweets.
- Sentiment Analysis Training Data: For the sentiment analysis component, they used the IMDB movie review sentiment classification dataset (25,000 reviews, split into 20,000 for training and 5,000 for testing).
Key Insights and Ideas
- Sentiment Clash is Tricky: The core idea was that a "connotation clash" (e.g., "I love how ill I became") is a hallmark of sarcasm. However, simply applying a standard sentiment analysis model to the whole sentence and using its output as a feature didn't work as expected.
- Deep Learning vs. Linear Models: The SVM model generally outperformed the CNN models in this study, both with and without sentiment analysis.
- Sentiment Analysis Hurt Performance: Surprisingly, in both the CNN and linear model approaches, adding sentiment analysis features degraded the sarcasm detection performance compared to models that relied solely on text features. This suggests that the way sentiment was incorporated was not effectively capturing the relevant "clash."
- Data Imbalance is a Real Problem: The authors highlighted the significant imbalance in their sarcasm datasets (around 25% sarcastic tweets) and devised a specific preprocessing strategy for their linear models to ensure a 50/50 split in training and testing sets to mitigate bias.
Key Results
- CNN Performance:
- CNN without sentiment analysis (English): F1-score of 0.2024.
- CNN with sentiment analysis (English): F1-score of 0.1679.
- The Arabic CNN also performed worse with sentiment analysis.
- Linear Model Performance (SVM):
- SVM without sentiment analysis (English): F1-score of 0.2738.
- SVM with sentiment analysis (English): F1-score of 0.2721.
- While the SVM without sentiment analysis performed slightly better, the difference was marginal.
- Overall Ranking: The SVM model, without sentiment analysis, achieved the highest F1 score among all tested models.
Limitations and Future Work
- Sentiment Analysis Granularity: The authors acknowledge that performing sentiment analysis on the entire sentence might be too coarse. Future work could involve splitting sentences into sub-parts and analyzing sentiment on those smaller units to better capture localized connotation clashes.
- Sentiment Lexicons: Alternatively, using sentiment lexicons to score sub-parts of sentences could be explored.
- Sentiment Model Domain: For the Deep Learning approach, using sentiment analysis models trained on datasets closer in genre to the sarcasm detection data (e.g., social media text) might yield better results.
- Decision Boundary Adjustment: For the CNN, adjusting the sigmoid output threshold (e.g., between 0.35 and 0.65) could potentially better capture mixed sentiments.
- Weighted Categories: For linear models, weighting the "rather positive" and "rather negative" sentiment categories more heavily might improve performance.
Summary updated