TL;DR
This paper introduces two novel deep learning models, ACE 1 and ACE 2, that significantly improve sarcasm detection by integrating both affective (emotional) and contextual (meaning-based on surrounding words) features. By modifying the BERT architecture and training it from scratch with these combined features, the models achieve state-of-the-art performance, outperforming previous methods.
What Problem Does This Paper Address?
Sarcasm detection in text is a challenging task because written language lacks the vocal intonation and facial cues that help humans understand it. Existing methods often rely on hand-crafted "affective" features (like sentiment words) or pre-trained word embeddings (like Word2vec) that don't fully capture the nuances of emotional expression or context. While recent models like BERT have improved contextual understanding, they haven't been optimally adapted for the specific demands of sarcasm detection, especially by integrating affective information directly into their architecture. This paper aims to bridge this gap by proposing models that deeply combine both affective and contextual information within a BERT-based framework.
Methods
The core innovation lies in two novel deep neural network models, ACE 1 and ACE 2, both built upon the BERT architecture.
- Affective Feature Embedding (AFE): This component extracts emotional information from text. It uses a Bi-LSTM with multi-head attention to process affective features. Two methods are proposed for extracting these features:
- EAISe (Emotion Affective Intensity with Sentiment Feature): Uses the NRC Emotion Intensity Lexicon to score words for basic emotions (anger, fear, sadness, joy) and sentiment (positive, negative), then aggregates these scores.
- EMoSi (Emotion Similarity Feature): Measures semantic similarity between words in a sentence and seed words for each emotion using Word2vec embeddings.
- Contextual Feature Embedding (CFE): This component leverages BERT's powerful contextual understanding.
- ACE 1: This model integrates AFE and CFE by training a modified BERT model from scratch. Affective feature embeddings are added to BERT's input sequence, allowing the model to learn task-specific embeddings that incorporate both types of information. This model is then fine-tuned on a sarcasm detection dataset.
- ACE 2: This model uses a feature-based approach. It utilizes pre-trained BERT (specifically SBERT for sentence embeddings) for CFE and combines its output with the AFE component in a fully connected layer for classification. This approach avoids training BERT from scratch.
Datasets
The authors use several datasets for training and evaluation:
- Training Corpora for Embeddings:
- Wiki: A combination of Wikipedia and a news corpus (142,546 articles from 15 American publications).
- WikiSarc: A more sarcasm-rich corpus created by combining the "Wiki" corpus with two additional datasets:
- IMSDB: Internet Movie Script Database (comedy movie transcripts).
- Riloff: Automatically extracted tweets containing sarcasm hashtags and random tweets.
- Sarcasm Detection Datasets (for evaluation):
- Onion: News headlines from The Onion and non-sarcastic headlines from HuffPost.
- IAC: Internet Argument Corpus (response utterances annotated for sarcasm).
- Reddit: Self-Annotated Reddit Corpus (SARC) (Reddit posts labeled by users).
- Pt'acek: Manually annotated sarcastic tweets.
- SemEval-2018: Tweets labeled for irony detection.
Key Insights and Ideas
- Deep Integration of Affective Features is Key: The study strongly suggests that incorporating affective features during the embedding training phase (as in ACE 1) is more effective than simply concatenating them later with pre-trained contextual embeddings (as in ACE 2). This deep integration allows the model to learn richer, context-aware emotional representations.
- Sarcasm-Rich Corpora Boost Performance: Training embedding models on corpora specifically designed to be rich in sarcasm and emotional content (like WikiSarc) leads to significantly better performance compared to using general-purpose corpora. This highlights the importance of domain-specific data for specialized tasks.
- Transformer Models Benefit from Task-Specific Features: The research demonstrates that even powerful pre-trained transformer models like BERT can be further enhanced for specific tasks (like sarcasm detection) by incorporating relevant, task-specific features.
- EMoSi vs. EAISe: While both affective feature extraction methods showed promise, EMoSi (Emotion Similarity) performed better for ACE 1 when trained on WikiSarc, whereas EAISe (Emotion Affective Intensity) was better for ACE 2. This suggests the optimal affective feature representation can depend on the model architecture and training data.
Key Results
The proposed ACE models significantly outperform existing state-of-the-art methods for sarcasm detection across multiple datasets.
- ACE 1 generally achieved better results than ACE 2, indicating the benefit of deep integration.
- The best performing model, ACE 1 trained on WikiSarc with EMoSi affective features, achieved F1-scores as high as 92.21% on the Onion dataset and 93.14% on the IAC dataset, substantially outperforming baselines.
- Using the WikiSarc corpus for training embeddings consistently led to better results than using the general Wiki corpus.
- The paper systematically evaluated various combinations of training corpora and affective feature representations, providing a comprehensive comparison.
Limitations and Future Work
- Model Complexity: Training ACE 1 from scratch is computationally intensive, requiring significant resources. ACE 2 offers a more efficient alternative by leveraging pre-trained models.
- Domain Specificity: While WikiSarc improved performance, the authors note that the effectiveness of affective features might be more pronounced in datasets with richer emotional content.
- Future Work: The authors plan to investigate whether the learned embeddings (e.g., WikiSarcA-BERT) can improve performance on other tasks, such as emotion detection. They also aim to explore how affective and contextual embeddings can be further leveraged for other NLP tasks.
Summary updated