Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?

Maxime Peyrard, Beatriz Borges, Kristina Gligoric, Robert West

Abstract

The automatic detection of humor poses a grand challenge for natural language processing. Transformer-based systems have recently achieved remarkable results on this task, but they usually (1) were evaluated in setups where serious vs humorous texts came from entirely different sources, and (2) focused on benchmarking performance without providing insights into how the models work. We make progress in both respects by training and analyzing transformer-based humor recognition models on a recently introduced dataset consisting of minimal pairs of aligned sentences, one serious, the other humorous. We find that, although our aligned dataset is much harder than previous datasets, transformer-based models recognize the humorous sentence in an aligned pair with high accuracy (78%). In a careful error analysis, we characterize easy vs hard instances. Finally, by analyzing attention weights, we obtain important insights into the mechanisms by which transformers recognize humor. Most remarkably, we find clear evidence that one single attention head learns to recognize the words that make a test sentence humorous, even without access to this information at training time.

Introduction

Humor is often considered "AI-complete"—a task so complex it requires human-level understanding of common sense, social context, and language nuances. While modern Transformer models have shown high accuracy in humor detection, previous benchmarks were often flawed. They frequently compared funny texts (like Reddit jokes) to serious texts (like news articles) from entirely different sources. In such cases, models could "cheat" by identifying stylistic or topical differences rather than the humor itself.

This research addresses two main gaps: the Data Gap, by using a dataset of "minimal pairs" where sentences are identical except for a tiny edit that removes the humor; and the Interpretability Gap, by moving beyond simple accuracy scores to ask how these models actually recognize a joke.

Dataset and Methodology

The study utilizes an extended version of the Unfun.me dataset, consisting of 23,113 pairs of funny and serious sentences. These pairs are unique because they originate from satirical news headlines (from The Onion) that users have edited to appear serious. Because the changes are minimal—often just one or two words—the model is forced to focus on the semantic "pivot" that creates the humor.

Researchers fine-tuned standard Transformer architectures, including BERT, distilBERT, and RoBERTa, on two tasks:

  1. Single-sentence setup (1S): Determining if a specific sentence is funny or serious.
  2. Paired setup (PS): Identifying the funny version within a nearly identical pair.

To understand the model's internal logic, the team analyzed attention weights using Jensen-Shannon divergence to measure how patterns shifted during fine-tuning. They also performed perturbation analysis, masking specific words to see if the model’s prediction would flip when the punchline was hidden.

Analysis of Attention Mechanisms

A central finding of this work is that humor recognition is a high-level semantic task rather than a syntactic one. As shown in the attention distance analysis, the most significant changes in the model's internal weights occur in the later layers of the network. This suggests that the model builds a complex semantic understanding to "get" the joke rather than relying on simple lexical features.

Average attention distance across layers, showing that humor-related shifts occur primarily in later semantic layers.

Most remarkably, the researchers discovered the emergence of a specialized "Laughing Head" (specifically Head 10-6 in BERT). During the fine-tuning process, this single attention head learned to pinpoint the specific words that trigger a punchline, even though the model was never explicitly told which words were responsible for the humor.

Heatmaps showing the emergence of a specialized attention head (10-6) that focuses on modified "funny" chunks in humorous sentences.

Key Results and Discussion

  • High Accuracy on Hard Data: In the paired setup, the best model (distilBERT) achieved 77.7% accuracy in identifying the funny version of a sentence, proving that Transformers can handle the nuances of minimal pairs.
  • The Power of Head 10-6: This single "laughing head" could identify the "funny token" with 37% accuracy—nearly five times better than a random baseline (8%) and significantly better than a simple "last word" heuristic (12%).
  • Causal Evidence: Masking the "funny" token identified by Head 10-6 caused the model to change its prediction from "funny" to "serious" 24% of the time, confirming that this head is critical to the model's decision-making process.
  • Humor vs. Surprisal: While humorous sentences are often more "surprising" (lower probability) to language models like GPT-2, BERT significantly outperformed GPT-2 based baselines. This indicates that humor requires a specific type of semantic opposition rather than just being unexpected.

Conclusion and Future Work

The discovery of the "laughing head" provides a rare and interpretable glimpse into the "black box" of AI. While this study was limited to short, headline-style humor, it sets the stage for future work in long-form humor and AI-assisted humor generation. By identifying where a serious sentence needs a "semantic pivot," these specialized attention heads could eventually help AI not just detect humor, but create it.

Summary updated

Abstract

The automatic detection of humor poses a grand challenge for natural language processing. Transformer-based systems have recently achieved remarkable results on this task, but they usually (1)~were evaluated in setups where serious vs humorous texts came from entirely different sources, and (2)~focused on benchmarking performance without providing insights into how the models work. We make progress in both respects by training and analyzing transformer-based humor recognition models on a recently introduced dataset consisting of minimal pairs of aligned sentences, one serious, the other humorous. We find that, although our aligned dataset is much harder than previous datasets, transformer-based models recognize the humorous sentence in an aligned pair with high accuracy (78%). In a careful error analysis, we characterize easy vs hard instances. Finally, by analyzing attention weights, we obtain important insights into the mechanisms by which transformers recognize humor. Most remarkably, we find clear evidence that one single attention head learns to recognize the words that make a test sentence humorous, even without access to this information at training time.
BibTeX
@inproceedings{peyrard2021laughing,
  author = {Maxime Peyrard and Beatriz Borges and Kristina Gligoric and Robert West},
  title = {{Laughing Heads: Can Transformers Detect What Makes a Sentence Funny?}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/537},
}
Figure