Neural Sarcasm Detection using Conversation Context

Nikhil JaiswalACL 2020

TL;DR

This paper tackles the challenge of detecting sarcasm in social media by leveraging deep neural networks and, crucially, the conversational context surrounding a tweet. By fine-tuning powerful pre-trained language models like BERT and RoBERTa on Twitter data, the authors demonstrate that incorporating the last three turns of conversation significantly improves sarcasm detection accuracy, achieving a competitive F1 score of 0.790.

What Problem Does This Paper Address?

Sarcasm is a pervasive form of figurative language on social media, making it difficult for machines to accurately interpret sentiment and meaning. While previous work has explored sarcasm detection, many approaches treated each utterance in isolation. This paper addresses the gap in understanding how much conversational context is truly necessary and beneficial for accurately identifying sarcasm in platforms like Twitter, where conversations unfold over multiple turns. The authors aim to build a robust sarcasm detection system that can leverage this contextual information.

Methods

The core of the proposed methodology involves fine-tuning state-of-the-art pre-trained language representation models (PLRMs) on a sarcasm detection task. Specifically, they experiment with:

  • ELMo (Embeddings from Language Models): A bidirectional LSTM-based model that generates deep contextualized word representations.
  • USE (Universal Sentence Encoder): A Transformer-based model that generates sentence embeddings.
  • BERT (Bidirectional Encoder Representations from Transformers): A powerful Transformer model pre-trained on masked language modeling and next sentence prediction tasks. They use the BERT_LARGE variant.
  • RoBERTa (Robustly Optimized BERT Approach): An optimized version of BERT with modifications to its pre-training strategy, including more training epochs, larger batch sizes, and dynamic masking.

The authors frame sarcasm detection as a binary classification problem. They experiment with different lengths of conversational context, from just the immediate response to including the previous three turns and the entire conversation history. For fine-tuning, they add dense layers on top of the PLRMs and train the models end-to-end. A key aspect of their approach is an ensemble strategy called "majority voting," where predictions from multiple models are combined to produce the final classification. They also explore techniques like gradual unfreezing of layers and dynamic learning rates for RoBERTa.

Datasets

The primary dataset used in this paper is from the Sarcasm Shared Task 2020, collected from the Twitter platform.

  • Training Data: 5,000 English Tweets.
  • Testing Data: 1,800 English Tweets.
  • Validation Data: 10% of the training data was used for hyper-parameter tuning.

The data is prepared in various formats to incorporate contextual information, as detailed in Table 1:

  • RESP: Only the response tweet.
  • CON1: Previous 1 turn + response.
  • CON2: Previous 2 turns + response.
  • CON3: Previous 3 turns + response.
  • CON: Entire context + response.

Key Insights and Ideas

  • Context is King (Up to a Point): The most significant finding is that incorporating conversational context dramatically improves sarcasm detection. Specifically, using the previous three turns of conversation (CON3) consistently yielded the best results across most models, suggesting a sweet spot for context length.
  • RoBERTa's Superiority: The RoBERTa model, being an optimized version of BERT, consistently outperformed other models, highlighting the benefits of its enhanced pre-training.
  • Ensembling for the Win: A simple majority voting ensemble of the best-performing RoBERTa models provided a further boost in performance, demonstrating the power of combining multiple models.
  • Transformer Architectures Shine: The paper reinforces the effectiveness of Transformer-based architectures (like USE, BERT, and RoBERTa) over LSTM-based models (like ELMo) for handling sequential and contextual data in this task.

Key Results

The paper reports the following key quantitative results:

  • The best performing ensemble model achieved an overall F1 score of 0.790 on the test data.
  • This F1 score ranked the authors second on the leaderboard of the Sarcasm Shared Task 2020.
  • Individually, the RoBERTa model fine-tuned on CON3 (previous three turns) achieved the highest F1 scores among single models, outperforming other approaches. For instance, one RoBERTa A model achieved an F1 of 0.778 on CON3, and an ensemble of top RoBERTa models reached 0.790.
  • USE generally performed better than ELMo, and BERT_LARGE outperformed USE.

Limitations and Future Work

The authors acknowledge the following:

  • Timing Constraints: They mention that they could not test all their experiments within the timing constraints of the Shared Task, particularly for the Reddit dataset.
  • Focus on Twitter: The current work is focused on Twitter data.

Future work suggested by the authors includes:

  • Exploring other datasets: Applying their methodology to other publicly available sarcasm detection datasets.
  • Deeper context analysis: Investigating how specific contextual elements contribute to sarcasm detection and deriving insights from this information.
  • More complex ensembles: Exploring more sophisticated ensemble techniques to further improve performance.

Summary updated

BibTeX
@inproceedings{jaiswal2020neural,
  author = {Nikhil Jaiswal},
  title = {{Neural Sarcasm Detection using Conversation Context}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure