Context vs. Human Disagreement in Sarcasm Detection

Hyewon Jang, Moritz Jakob, Diego FrassinelliNAACL 2024

TL;DR

This paper investigates how much context is needed for humans and language models to accurately detect sarcasm. They found that while context generally helps, more context doesn't always mean better sarcasm detection for humans. Interestingly, for language models, longer context becomes more beneficial when human evaluators disagree on whether something is sarcastic.

What Problem Does This Paper Address?

Sarcasm detection is a notoriously difficult task, both for humans and machines. Previous research has shown that providing context to the utterance in question can improve sarcasm detection performance. However, there's been less exploration into how much context is optimal, or what constitutes "context" itself. This paper aims to bridge this gap by systematically examining the impact of different amounts of textual context (short vs. long) on sarcasm detection accuracy for both human annotators and pre-trained language models. They also explore how the level of human disagreement on sarcasm judgments influences the effectiveness of context for machine models.

Methods

The researchers created a new dataset by adapting the Multimodal Sarcasm Detection Dataset (MUSTARD). They took existing "contexts" (preceding utterances) from the TV show "Friends" and created two versions: a short context (SC) using the original utterance, and a long context (LC) by providing a narrative description of the scene to restore relevant information. They then collected new "responses" to these contexts from participants.

For human evaluation, participants rated the sarcasm level of responses under three conditions: no context (NC), short context (SC), and long context (LC). For language model evaluation, they fine-tuned three pre-trained models (BERT, RoBERTa, and DistilBERT) on the MUSTARD dataset and then tested them on their newly created data across the three context conditions (NC, SC, LC). They also performed an error analysis, distinguishing between instances where human evaluators agreed on sarcasm labels and where they disagreed.

Datasets

  • MUSTARD Dataset (adapted): The original dataset provided written transcriptions of contexts and responses from TV series, with binary sarcasm labels. The authors adapted this by selecting 24 contexts from "Friends," creating narrative descriptions for long contexts, and collecting new responses.
  • Newly created dataset: This dataset consists of 24 contexts, each with associated short and long versions, and newly collected responses. It was used for both human and language model evaluation.

Key Insights and Ideas

  • Context is generally good, but more isn't always better (for humans): For human evaluators, the presence of context (either short or long) significantly increases the probability of correctly identifying sarcasm compared to no context. However, increasing context from short to long did not yield additional benefits for humans.
  • LM performance benefits from context, especially with human disagreement: Language models also show improved performance with context. Crucially, when human evaluators disagreed on a sarcasm label, longer contexts provided a more significant benefit to the language models compared to shorter or no context.
  • Human disagreement doesn't break LMs, but it reveals nuances: Despite low inter-rater agreement among human evaluators, the overall patterns of sarcasm detection by language models remained consistent across different evaluation sets. This suggests that LMs might be learning generalizable patterns, but their performance can be more sensitive to context when human judgments are ambiguous.

Key Results

  • Human Evaluation:
    • Context (SC or LC) significantly increased the probability of judging a response as sarcastic compared to no context (NC).
  • Around 38% of instances judged as "not sarcastic" in the NC condition were judged as "sarcastic" with context.
  • Long context (LC) and short context (SC) showed similar benefits for humans (LC = SC > NC).
  • Language Model Evaluation (Macro F-scores):
    • Supplying context (LC or SC) consistently improved the performance of BERT, RoBERTa, and DistilBERT compared to no context (NC) across all evaluations.
  • For instances where human evaluators agreed on a label, the benefit of longer context over shorter context was minimal for LMs.
  • For instances where human evaluators disagreed, longer context (LC) showed a greater benefit for LMs compared to shorter (SC) or no context (NC). For example, for BERT on disagreed-upon instances, LC had a 0.54 proportion of majority predictions, while SC and NC had 0.51 and 0.50 respectively.

Limitations and Future Work

  • Information Quantity vs. Quality: The study acknowledges that they did not systematically quantify the amount of information in SC vs. LC. Future work could explore a gradient of context lengths to better understand the relationship between information quantity and sarcasm detection.
  • Dataset Size: The collected dataset is relatively small due to rigorous filtering. Future research should test these effects with larger datasets.
  • Participant Demographics: The data collection primarily involved male participants. Future studies should include gender as a variable for a more comprehensive understanding of sarcasm use.
  • Defining Context: The paper defines context as preceding textual utterances. Future work could explore other forms of context, such as multimodal information (images, audio).

Summary updated

BibTeX
@inproceedings{jang2024context,
  author = {Hyewon Jang and Moritz Jakob and Diego Frassinelli},
  title = {{Context vs. Human Disagreement in Sarcasm Detection}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure