Abstract
Humor classification remains a complex challenge for artificial intelligence, requiring a nuanced understanding of cultural context, double meanings, and linguistic subversion. This paper evaluates two distinct paradigms for multiclass humor classification: a "fast" approach using the efficient fastText classifier and a "slow" approach utilizing Retrieval-Augmented Generation (RAG) with state-of-the-art Large Language Models (LLMs). Evaluating on the JOKER 2024 Task 2 dataset, we find that while RAG significantly enhances LLM performance, a well-tuned fastText baseline achieves parity with the most advanced models, reaching an accuracy of 0.5725.
1. Introduction
Humor is often considered the final frontier for AI, as it demands more than mere vocabulary; it requires grasping the delicate subversion of expectations. In this study, we explore the trade-offs between lightweight, efficient classifiers and deep-thinking LLMs for categorizing humor types. Inspired by Daniel Kahneman’s "Thinking, Fast and Slow" framework, we compare the performance of fastText against RAG-equipped models like GPT-4 and Llama-3.
2. Problem Statement
The study addresses Task 2 of the JOKER 2024 competition: multiclass humor classification. Unlike binary classification ("Is this funny?"), models must categorize text into one of six categories: irony (IR), sarcasm (SC), exaggeration (EX), incongruity-absurdity (AID), self-deprecation (SD), or wit-surprise (WS). These categories often overlap, presenting a significant challenge for automated systems that must distinguish between subtle intents and tones.
3. Methodology
The research follows a two-pronged approach to evaluate the efficiency-depth trade-off:
3.1 The "Fast" Approach: fastText
We utilized the fastText library for supervised learning, known for its speed and efficiency. To maximize performance, we conducted extensive hyperparameter optimization, exploring various learning rates, n-gram lengths, and word count thresholds to squeeze the most performance out of a relatively simple architecture.
3.2 The "Slow" Approach: LLMs + RAG
We evaluated several state-of-the-art models, including GPT-3.5-Turbo, GPT-4o, GPT-4, and Llama-3 (70b). We tested these models in two settings:
- Zero-shot: Direct classification without examples.
- Retrieval-Augmented Generation (RAG): For each query, the system retrieved the most similar examples from the training set using the
mxbai-embed-large-v1embedding model to provide few-shot context.
4. Dataset
The experiments were conducted on the JOKER 2024 Task 2 dataset, comprising 1,742 training samples and 6,642 test samples. The small size of the training set poses a challenge for both traditional learning and RAG-based retrieval, as the "knowledge base" for context is relatively limited.
5. Results and Discussion
The experimental results reveal several key insights regarding the capabilities of current NLP models in humor analysis.
5.1 The Impact of RAG
Zero-shot LLMs performed poorly, with accuracies hovering around 20-27%. This suggests that the academic definitions of humor types do not align perfectly with the general knowledge baked into these models. However, the introduction of RAG acted as a massive force multiplier, more than doubling the accuracy for most models. GPT-4 + RAG achieved the highest accuracy among LLMs at 0.5725.
5.2 The Baseline Paradox
Despite the complexity and parameter count of GPT-4, the optimized fastText model achieved the exact same top accuracy of 0.5725. This serves as a sobering reminder for practitioners to establish strong, simple baselines. Furthermore, LLMs occasionally "refused" to classify ambiguous or confusing inputs, whereas the fastText model provided a prediction for every sample.
5.3 Model-Specific Observations
Interestingly, GPT-4o performed significantly worse than the standard GPT-4 when combined with RAG (0.4695 vs 0.5725), indicating that newer "omni" models may not always be superior for specialized linguistic tasks like humor categorization.
6. Conclusion
Our findings suggest that for niche tasks like humor classification, a well-tuned "fast" baseline remains highly competitive with "slow" LLM pipelines. While RAG provides a necessary boost for LLMs, the performance ceiling (~57%) suggests that future work should explore fine-tuning to better align model weights with the specific nuances of humorous text.
Summary updated