TL;DR
This paper introduces a novel, flexible metric to evaluate how well Large Language Models (LLMs) can identify humorous punchlines in stand-up comedy transcripts. The metric, which uses fuzzy string matching, sentence embeddings, or subspace similarity, reveals that even top LLMs struggle with this nuanced task, scoring only around 50%, though this is still better than human performance on the same text-based task.
What Problem Does This Paper Address?
Humor is a fundamental aspect of human interaction, yet it remains a significant challenge for Artificial Intelligence. While LLMs have made impressive strides in understanding and generating human-like text, their ability to grasp the subtleties of humor, especially in complex narrative forms like stand-up comedy, is not well understood. Existing methods for evaluating humor detection often rely on simpler joke formats or artificial laughter, failing to capture the natural flow and audience reaction inherent in live performances. This paper aims to bridge this gap by developing a robust metric to assess LLM performance in identifying humor within stand-up comedy transcripts, a rich source of nuanced comedic expression.
Methods
The core contribution is a novel, modular humor detection metric designed to evaluate LLMs. The process begins with an LLM being prompted to extract humorous lines from a stand-up comedy transcript. The metric then compares these predicted quotes against a "ground truth" set of punchlines, identified through a laughter detection model and forced alignment with the transcript.
The metric offers three distinct scoring modules:
- Fuzzy String Matching: This module uses Levenshtein distance to compare the similarity of predicted and ground truth quotes at a character level. It's a straightforward approach but can be overly punitive if the LLM makes minor errors.
- Vector Embedding: This module leverages sentence embeddings to capture semantic similarity, allowing for a more flexible assessment that focuses on the meaning of the quotes rather than exact wording. This is particularly useful when LLMs provide explanations alongside quotes.
- Subspace Similarity: This novel approach analyzes the structural similarity between subspaces derived from the LLM's feature vectors and the ground truth. It aims to provide a more generalized assessment of the model's ability to capture underlying patterns.
The final score is an aggregation of these modules, with a penalty for overgeneration of quotes. The paper also explores prompt engineering techniques and human-machine agreement.
Datasets
The primary dataset used is the Open Mic dataset (Mittal et al., 2021), which consists of audio recordings and transcripts from stand-up comedy performances. For their experiments, the authors randomly selected 51 transcripts.
Key Insights and Ideas
- LLMs Still Struggle with Nuanced Humor: Despite their advanced capabilities, even leading LLMs like ChatGPT, Claude, and DeepSeek achieve relatively low scores (around 50%) in identifying humorous punchlines from stand-up comedy transcripts. This highlights the inherent difficulty of computational humor.
- Text-Based Analysis Limitations: The study suggests that LLMs excel at text-based tasks and linguistic cues, which can outperform humans in a text-only evaluation. However, this might be at the expense of capturing the broader context of live performance, which humans implicitly consider.
- Subjectivity of Humor: The research underscores the subjective nature of humor, as evidenced by the variability in agreement between human evaluators and LLMs, and even among human evaluators themselves.
- Prompt Engineering Matters, But Not Universally: While prompt engineering can influence LLM performance, the study found that simple persona-based prompts or humor-type specialization did not consistently improve humor detection scores.
- Metric Design is Crucial: The paper's modular metric design is a key innovation, allowing for a more comprehensive evaluation by combining different similarity measures and acknowledging the inherent subjectivity of humor.
Key Results
- LLM Performance: Leading LLMs (ChatGPT, Claude, DeepSeek) achieved scores of at most 51% in humor detection using the proposed metric.
- Human Performance: Human evaluators, when assessed on the same text-based task, achieved a score of 41%.
- Human-Machine Agreement: The average agreement rate between human evaluators and LLMs was around 59.9%, indicating a notable but not perfect alignment in identifying humorous content.
- Model Comparison: Gemma 2 models showed strong performance, particularly with the subspace similarity module. ChatGPT performed well with fuzzy string matching but declined with semantic similarity, suggesting a strength in lexical matching over deeper semantic understanding.
- Prompt Engineering: Prompt 2 ("When performed in front of a live audience, which jokes do you think made the audience laugh?") yielded the highest score for Gemma 2b-instruct among the tested prompts.
Limitations and Future Work
- Ground Truth Derivation: The ground truth is derived from laughter timestamps, assuming the preceding sentence is the punchline. This might not always be accurate, and subtle humor or delayed laughter could be missed.
- Laughter Magnitude: The metric does not differentiate between varying intensities of laughter, potentially overlooking jokes with subtler audience reactions.
- Text-Based Analysis: The study's reliance on text-based analysis is a limitation, as stand-up comedy is heavily influenced by non-textual elements like delivery, tone, and audience reactions.
- Future Work: The authors propose extending the metric to evaluate humor detection in video formats, incorporating non-textual elements. They also plan to explore how human perception of humor differs when viewing stand-up without background laughter and to investigate the impact of muted laughter on LLM predictions.
Summary updated