Thinking, Fast and Slow: From the Speed of FastText to the Depth of Retrieval Augmented Large Language Models For Humour Classification

Jana Viktória Kováciková, Marek SuppaCLEF (Working Notes) 2024 · OpenReview

Abstract

Humor classification remains a complex challenge for artificial intelligence, requiring a nuanced understanding of cultural context, double meanings, and linguistic subversion. This paper evaluates two distinct paradigms for multiclass humor classification: a "fast" approach using the efficient fastText classifier and a "slow" approach utilizing Retrieval-Augmented Generation (RAG) with state-of-the-art Large Language Models (LLMs). Evaluating on the JOKER 2024 Task 2 dataset, we find that while RAG significantly enhances LLM performance, a well-tuned fastText baseline achieves parity with the most advanced models, reaching an accuracy of 0.5725.

1. Introduction

Humor is often considered the final frontier for AI, as it demands more than mere vocabulary; it requires grasping the delicate subversion of expectations. In this study, we explore the trade-offs between lightweight, efficient classifiers and deep-thinking LLMs for categorizing humor types. Inspired by Daniel Kahneman’s "Thinking, Fast and Slow" framework, we compare the performance of fastText against RAG-equipped models like GPT-4 and Llama-3.

2. Problem Statement

The study addresses Task 2 of the JOKER 2024 competition: multiclass humor classification. Unlike binary classification ("Is this funny?"), models must categorize text into one of six categories: irony (IR), sarcasm (SC), exaggeration (EX), incongruity-absurdity (AID), self-deprecation (SD), or wit-surprise (WS). These categories often overlap, presenting a significant challenge for automated systems that must distinguish between subtle intents and tones.

3. Methodology

The research follows a two-pronged approach to evaluate the efficiency-depth trade-off:

3.1 The "Fast" Approach: fastText

We utilized the fastText library for supervised learning, known for its speed and efficiency. To maximize performance, we conducted extensive hyperparameter optimization, exploring various learning rates, n-gram lengths, and word count thresholds to squeeze the most performance out of a relatively simple architecture.

Hyperparameter search space for the fastText model optimization.

3.2 The "Slow" Approach: LLMs + RAG

We evaluated several state-of-the-art models, including GPT-3.5-Turbo, GPT-4o, GPT-4, and Llama-3 (70b). We tested these models in two settings:

  1. Zero-shot: Direct classification without examples.
  2. Retrieval-Augmented Generation (RAG): For each query, the system retrieved the most similar examples from the training set using the mxbai-embed-large-v1 embedding model to provide few-shot context.

4. Dataset

The experiments were conducted on the JOKER 2024 Task 2 dataset, comprising 1,742 training samples and 6,642 test samples. The small size of the training set poses a challenge for both traditional learning and RAG-based retrieval, as the "knowledge base" for context is relatively limited.

5. Results and Discussion

The experimental results reveal several key insights regarding the capabilities of current NLP models in humor analysis.

Performance comparison of various LLM configurations on the humor classification task.

5.1 The Impact of RAG

Zero-shot LLMs performed poorly, with accuracies hovering around 20-27%. This suggests that the academic definitions of humor types do not align perfectly with the general knowledge baked into these models. However, the introduction of RAG acted as a massive force multiplier, more than doubling the accuracy for most models. GPT-4 + RAG achieved the highest accuracy among LLMs at 0.5725.

5.2 The Baseline Paradox

Despite the complexity and parameter count of GPT-4, the optimized fastText model achieved the exact same top accuracy of 0.5725. This serves as a sobering reminder for practitioners to establish strong, simple baselines. Furthermore, LLMs occasionally "refused" to classify ambiguous or confusing inputs, whereas the fastText model provided a prediction for every sample.

5.3 Model-Specific Observations

Interestingly, GPT-4o performed significantly worse than the standard GPT-4 when combined with RAG (0.4695 vs 0.5725), indicating that newer "omni" models may not always be superior for specialized linguistic tasks like humor categorization.

6. Conclusion

Our findings suggest that for niche tasks like humor classification, a well-tuned "fast" baseline remains highly competitive with "slow" LLM pipelines. While RAG provides a necessary boost for LLMs, the performance ceiling (~57%) suggests that future work should explore fine-tuning to better align model weights with the specific nuances of humorous text.

Summary updated

BibTeX
@inproceedings{kovacikova2024thinking,
  author = {Jana Viktória Kováciková and Marek Suppa},
  title = {{Thinking, Fast and Slow: From the Speed of FastText to the Depth of Retrieval Augmented Large Language Models For Humour Classification}},
  booktitle = {CLEF (Working Notes)},
  year = {2024},
  url = {https://openreview.net/forum?id=O3sDhlZA9v},
}
Figure