Comparing Apples to Oranges: A Dataset & Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes

Tyler Loakman, William Thorne, Chenghua Lin2025 · DOI

Introduction

Humor is a hallmark of human intelligence, yet evaluating it in Large Language Models (LLMs) has often been restricted to simple detection tasks. This paper introduces a new benchmark focused on humor explanation—the ability of a model to articulate why a joke is funny. While modern LLMs handle linguistic wordplay with increasing competence, they encounter significant hurdles when faced with topical humor that demands esoteric real-world knowledge and nuanced social reasoning.

The Challenge of Humor Explanation

Traditional computational humor research has focused on "easy mode": binary humor detection. However, recognizing a joke is distinct from understanding its mechanics. Existing datasets are also heavily skewed toward puns, which are linguistically self-contained. In contrast, real-world humor—the kind found on social platforms like Reddit or in stand-up comedy—is often topical. It relies on an intricate web of pop culture, current events, and social constructs. Bridging the gap between recognizing a phonetic double-entendre and understanding a joke about a specific political scandal or internet meme is essential for true AI language understanding.

Methodology and Dataset

The researchers evaluated eight prominent LLMs, ranging from proprietary models like GPT-4o and Gemini 1.5 to open-source weights like Llama 3.1 and reasoning-specialized models like DeepSeek-R1. Models were tasked with generating ~100-word explanations for a curated dataset of 600 jokes.

The dataset is balanced across four distinct categories, each representing a different level of linguistic or contextual complexity:

  1. Homographic Puns: Wordplay based on multiple meanings of the same spelling.
  2. Heterographic Puns: Wordplay based on words that sound alike but are spelled differently.
  3. Non-topical Reddit Jokes: General situational irony or observational humor.
  4. Topical Reddit Jokes: Humor requiring specific knowledge of external entities or events (e.g., Elon Musk, PETA).
Figure 1: Examples of the four joke categories used in the dataset, ranging from simple homographic puns to complex topical humor.

To evaluate the outputs, the authors used human annotators and an "LLM-as-a-judge" (Qwen 2.5 72B) to score explanations on Accuracy (correct identification of humor/context) and Completeness (coverage of all nuances) on a 0-5 scale.

Key Findings

The analysis revealed several critical insights into the current state of LLM reasoning:

  • The Phonetic Blind Spot: LLMs consistently struggle more with heterographic puns than homographic ones. Because these models process text as tokens rather than acoustic signals, they often fail to "hear" the similarity between words like dye and die without explicit phonetic training.
  • The "Esoteric" Wall: Topical humor proved to be the most difficult category. Understanding these jokes requires a two-step process: retrieving a specific, often niche, memory of a real-world event and mapping it to the joke's incongruity. Smaller models frequently failed at the retrieval step, leading to hallucinations. For instance, in a joke about the "Tide Pod Challenge," smaller models often missed the 2018 phenomenon entirely, suggesting instead that the joke was about teenagers being "messy eaters."
  • Reasoning Models vs. Scale: While DeepSeek-R1 70B performed well, it did not significantly outperform standard models of similar size. Interestingly, the smaller 8B reasoning model performed the worst, often "overthinking" simple puns and following incorrect reasoning chains.
Figure 4: Average explanation lengths and performance scores (Accuracy and Completeness) across all models and joke types. GPT-4o emerges as the most consistent performer.

Results Summary

  • GPT-4o Leadership: GPT-4o consistently produced the most accurate and complete explanations across all categories.
  • The Impact of Model Size: Larger models (e.g., Llama 70B vs. 8B) were significantly more likely to generate high-quality explanations (scores of 4+). Statistical analysis confirmed that model size is a primary predictor of success.
  • Success Variance: While success rates for homographic puns often exceeded 80%, they plummeted for topical jokes. Most models, with the exception of GPT-4o, failed to provide adequate explanations for more than half of the topical set.

Limitations and Future Directions

The study notes that topical humor is inherently ephemeral; a joke that resonates in 2019 may be obscure by 2025. The current dataset focuses on jokes from 2008–2019 to align with the models' training cutoffs. Additionally, while 600 jokes allow for deep qualitative analysis, larger scales would be beneficial for broader generalization.

Future research may benefit from Retrieval-Augmented Generation (RAG). By allowing models to "look up" the context of topical jokes in real-time, the gap between simple wordplay and complex cultural humor might finally be closed.

Summary updated

BibTeX
@misc{loakman2025comparing,
  author = {Tyler Loakman and William Thorne and Chenghua Lin},
  title = {{Comparing Apples to Oranges: A Dataset \& Analysis of LLM Humour Understanding from Traditional Puns to Topical Jokes}},
  year = {2025},
  doi = {10.18653/v1/2025.findings-emnlp.505},
}
Figure