Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?

Maxime Peyrard, Beatriz Borges, Kristina Gligoric, Robert West

Introduction

Humor detection is often considered "AI-complete," requiring deep semantic understanding, common-sense reasoning, and an appreciation of social context. Researchers from EPFL have recently demonstrated that Transformer-based models can detect humor with high accuracy by comparing "minimal pairs" of satirical and serious headlines. Their most striking discovery is the emergence of a "laughing head"—a single attention head that spontaneously learns to pinpoint the specific word that makes a sentence funny, even without explicit supervision.

The Challenge of Computational Humor

Historically, computational humor research has struggled with two main issues: confounded datasets and the "black box" problem. Models trained on disparate sources (e.g., Reddit jokes vs. Reuters news) often learn to distinguish writing styles rather than humor itself. Furthermore, achieving high scores on benchmarks doesn't explain how a model processes the underlying logic of a joke.

To address these issues, the authors utilized the Unfun.me dataset, which contains over 23,000 pairs of headlines. These "minimal pairs" are nearly identical, with humor localized to just one or two edited words, allowing for a precise analysis of the model's internal attention mechanisms.

Methodology: Single-Sentence vs. Paired Setups

The researchers finetuned several architectures, including BERT and RoBERTa, on two distinct tasks:

  • Single-sentence (1S): Classifying a lone headline as funny or serious.
  • Paired setup (PS): Identifying the funny version within a minimal pair using a Siamese network architecture.

Semantic Processing in Deeper Layers

Analysis of attention weights using Jensen-Shannon divergence revealed that the most significant changes during finetuning occur in the later layers of the Transformer. This suggests the model is picking up on high-level semantic contradictions rather than simple grammatical or lexical patterns.

Figure 1: Average attention distance across layers. Plot (a) shows distance to the base BERT model, while (b) shows distance between funny and serious versions. The divergence in later layers indicates semantic-level processing.

Performance and Results

The paired setup proved significantly more effective than single-sentence classification. In the PS task, the best model (distilBERT) achieved 77.7% accuracy, compared to only 64.5% for the single-sentence BERT model. This highlights the importance of context and comparison in humor detection.

Figure 4: Accuracy comparison across different models and modification sizes. The paired setup (PS) consistently outperforms the single-sentence (1S) approach across all architectures.

The "Laughing Head" (Head 10-6)

The study's most notable finding was the specialization of Head 10-6 in BERT. Without explicit training to identify the edited word, this head learned to attend to the "funny token" (the word creating the punchline) with 37% accuracy—nearly five times better than a random baseline. Perturbation analysis confirmed this localization: masking the "funny" word changed the model's decision from funny to serious 24% of the time, compared to only 13% for other words.

Conclusion and Future Work

While the study is limited to short, satirical headlines, it provides a compelling look into how Transformers handle complex semantic tasks. The discovery of the "laughing head" suggests that these models can develop sophisticated internal representations of irony and absurdity. Future research may explore whether similar mechanisms exist for other forms of humor, such as puns or long-form narrative structures, potentially providing tools to help humans write better humor or improve AI-based content moderation.

Summary updated

Abstract

The automatic detection of humor poses a grand challenge for natural language processing. Transformer-based systems have recently achieved remarkable results on this task, but they usually (1) were evaluated in setups where serious vs humorous texts came from entirely different sources, and (2) focused on benchmarking performance without providing insights into how the models work. We make progress in both respects by training and analyzing transformer-based humor recognition models on a recently introduced dataset consisting of minimal pairs of aligned sentences, one serious, the other humorous. We find that, although our aligned dataset is much harder than previous datasets, transformer-based models recognize the humorous sentence in an aligned pair with high accuracy (78\%). In a careful error analysis, we characterize easy vs hard instances. Finally, by analyzing attention weights, we obtain important insights into the mechanisms by which transformers recognize humor. Most remarkably, we find clear evidence that one single attention head learns to recognize the words that make a test sentence humorous, even without access to this information at training time.
BibTeX
@inproceedings{peyrard2021laughing,
  author = {Maxime Peyrard and Beatriz Borges and Kristina Gligoric and Robert West},
  title = {{Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/537},
}
Figure