Abstract
In this paper we introduce our system for the task of Irony detection in English tweets, a part of SemEval 2018. We propose representation learning approach that relies on a multi-layered bidirectional LSTM, without using external features that provide additional semantic information. Although our model is able to outperform the baseline in the validation set, our results show limited generalization power over the test set. Given the limited size of the dataset, we think the usage of more pre-training schemes would greatly improve the obtained results.
Introduction
Irony is a major "stumbling block" for sentiment analysis. Because ironic statements often convey the opposite of their literal meaning (e.g., "I love being stuck in traffic"), they can easily trick automated systems into misclassifying negative sentiment as positive. Detecting irony is difficult because it relies on context, shared knowledge, and subtle linguistic cues rather than just keywords. Historically, researchers tackled this using manual "feature engineering." The authors of this paper aim to replace these expensive, manual systems with a representation learning approach, where the model automatically learns the relevant features from the data using neural networks.
Methods
The classification pipeline consists of three main components:
- Pre-processing: To ensure the model wasn't "cheating," hashtags that explicitly label irony (like #sarcasm or #not) were removed. The NLTK TweetTokenizer was used, but text was not lowercased, as capitalization (e.g., "GREAT weather") is often a signal for irony.
- Word Representations: The system used pre-trained GloVe embeddings (trained on 2 billion tweets). To handle Out-of-Vocabulary (OOV) words, vectors were sampled from a sphere centered around the "centroid" of the least frequent words in the GloVe vocabulary.
- Model Architecture: The core of the system is a bidirectional LSTM (BiLSTM). By processing the tweet in both forward and backward directions, the model captures context from the entire sentence. Simple binary features were concatenated to the word vectors, such as token-level (e.g., is the word all caps?) and sentence-level (e.g., are there any all-caps words?) indicators.
Datasets
The study used the SemEval-2018 Task 3 dataset, which consists of 4,792 tweets in total (balanced: 2,396 ironic, 2,396 non-ironic). The ironic tweets were further categorized into types: Verbal irony (polarity contrast), Other verbal irony, and Situational irony.
Results and Analysis
The model's performance was a "tale of two datasets." While the BiLSTM significantly outperformed the SVM baseline on the validation set, achieving an F1-score of 0.7262, its performance plummeted on the hidden test set.
An ablation study revealed a "granularity" trade-off: sentence-level features (like checking if any word is in all-caps) slightly improved performance, while token-level features actually hurt it. This suggests that irony is a global property of a sentence rather than a property of individual words.
Limitations and Future Work
The authors identify the small dataset size as the primary reason for the model's failure to generalize. Deep learning models typically require much larger corpora to learn the subtle nuances of irony. For future work, the authors suggest using transfer learning with models pre-trained on massive social media datasets (e.g., emoji-based datasets) and exploring data augmentation to reduce overfitting. This stark difference in performance underscores how difficult it is to build a robust irony detector without massive amounts of data.
Summary updated