TL;DR
This paper tackles the challenging task of humor detection in text by combining the power of pre-trained language models (DistilBERT) with a novel multi-scale Convolutional Neural Network (CNN) architecture. Their approach achieved state-of-the-art results, demonstrating the effectiveness of capturing diverse textual features for humor identification.
What Problem Does This Paper Address?
Humor detection in Natural Language Processing (NLP) is notoriously difficult. Humor often relies on subtle cues, context, sarcasm, and shared cultural knowledge, making it hard for algorithms to grasp. The SemEval-2021 Task 7-1a specifically aimed to classify short English phrases based on whether the writer intended them to be humorous, rather than whether the reader found them funny. This distinction is crucial, as humor can be subjective and offensive. The dataset provided consisted of labeled phrases, with additional ratings for funniness and offensiveness for humorous texts.
Methods
The authors explored a range of models, from traditional shallow methods like Logistic Regression, Naive Bayes, Random Forests, and Support Vector Machines (SVMs), to deep learning approaches like Long Short-Term Memory (LSTM) networks and Convolutional Neural Networks (CNNs).
The core of their winning approach, DistilBERT+Multi-Scale CNN, leverages two key ideas:
- Pre-trained Language Models (DistilBERT): They used DistilBERT, a distilled (smaller, faster) version of BERT, to generate rich, contextualized vector representations of the input text. This is a significant step up from static word embeddings like GloVe, as DistilBERT captures nuances of word meaning based on surrounding words.
- Multi-Scale CNN: To further enhance feature extraction, they employed a CNN architecture with multiple kernel sizes (1, 2, 3, 4, and 5). This allows the network to capture patterns and features at different granularities – from short n-grams to longer sequences – within the text. These multi-scale features are then concatenated and fed into dense layers for classification.
They also experimented with a MultiPool CNN variant, which uses different pooling strategies instead of varying kernel sizes to achieve multi-scale feature extraction.
Datasets
The primary dataset used was provided for SemEval-2021 Task 7-1a. This dataset comprised:
- A training set of 8000 labeled sentences.
- A development set of 1000 sentences.
- A test set of 1000 sentences.
The labels indicated whether a phrase was intended to be humorous. For humorous phrases, additional labels for funniness and offensiveness were also available.
Key Insights and Ideas
- Pre-trained models are essential: The paper strongly emphasizes that using pre-trained language models like DistilBERT significantly boosts performance. Without them, even sophisticated models struggle to achieve good results.
- CNNs are well-suited for humor detection: Compared to LSTMs and fully-connected networks, CNNs, especially when combined with multi-scale feature extraction, proved to be more effective for this task. This suggests that local patterns and feature hierarchies are important for identifying humor.
- Multi-scale feature extraction is key: The success of their best model highlights the benefit of capturing features at various scales. Humor can manifest in different ways, and a multi-scale approach allows the model to be sensitive to these diverse patterns.
- Subjectivity of humor: The authors acknowledge that humor labels can be subjective, leading to misclassifications even for human annotators. This inherent ambiguity is a fundamental challenge in humor detection.
Key Results
The DistilBERT+Multi-Scale CNN model achieved the best performance, reaching:
- 93.7% F1-score on the test set.
- 92.1% accuracy on the test set.
This performance significantly surpassed the official baseline (85.7% F1, 88.4% accuracy) and other implemented models. For instance, their official submission, DistilBERT+FC, achieved 92.41% F1, ranking 47th in the task. The DistilBERT+Multi-Scale CNN model also outperformed a GloVe-based LSTM model, reinforcing the advantage of transformer-based pre-trained representations.
Limitations and Future Work
- Hardware Limitations: The authors mention that hardware constraints limited them to using the "light" version of BERT (DistilBERT) and shallower models. They suggest that exploring the full BERT model or deeper networks could yield further improvements.
- Multi-scale Dimension: They explored multi-scale strategies in the "wide" dimension (kernel sizes). They propose that multi-scale approaches could also be applied in the "deep" dimension (network layers) for future work.
- Subjectivity of Labels: The paper acknowledges that the subjective nature of humor labels can lead to misclassifications. This inherent ambiguity is a limitation of the dataset and the task itself.
- Error Analysis: The authors performed an error analysis, identifying false positives (classifying non-humorous as humorous) and false negatives (missing humorous sentences). They noted that some misclassified sentences were difficult even for humans to understand, highlighting the complexity of humor.
Summary updated