TL;DR
This paper explores the effectiveness of traditional machine learning (ML) approaches for humor and offense detection tasks in 2021, comparing them against more modern deep learning models. The DUTH team found that while conventional ML methods are computationally efficient and faster to train, they struggle with outliers and complex nuances that deep learning models can capture, suggesting a continued role for ML in specific research contexts.
What Problem Does This Paper Address?
Humor detection and understanding its nuances, particularly its relationship with offensiveness, remain challenging areas in Natural Language Processing (NLP). While deep learning models like BERT have shown superior performance in recent humor detection tasks, this paper investigates whether conventional machine learning techniques, often favored for their efficiency, are still relevant and capable of achieving competitive results. The authors aim to assess if these established methods are "enough" in 2021 for tasks involving humor and offense classification and rating.
Methods
The DUTH team employed a comprehensive approach involving various preprocessing techniques, vectorization methods, and conventional machine learning algorithms.
Preprocessing techniques included: * Tokenization and Lowercasing: Standardizing text by breaking it into words and converting them to lowercase. * Stemming/Lemmatization: Reducing words to their root form to decrease noise. * Stopword Removal: Eliminating common words that don't carry significant meaning. * Capital Tagging: Identifying words with initial capital letters (excluding sentence beginnings). * Emoji Replacement: Substituting emojis with their sentiment equivalents. * Contraction Expansion: Replacing contractions (e.g., "don't") with their full forms. * Integer Removal: Removing numbers, as they are deemed to have no emotional value. * Part-of-Speech (POS) Tagging: Two approaches were used: appending POS tags to words as features and removing words based on their POS tags. * Numeric Feature Extraction: Extracting counts of characters, words, punctuation, sentence types, and specific word categories (verbs, nouns, adjectives).
Vectorization methods included: * TF-IDF Vectorizer: Converting word counts into TF-IDF features. * Delta TF-IDF Vectorizer: A variation that weights features based on their difference in occurrence between classes. * Word2Vec Embeddings: Using pre-trained GloVe embeddings to represent words as dense vectors, specifically for LSTM models.
Machine Learning Algorithms: A range of probabilistic, linear, and tree-based algorithms were explored, including Linear SVM, Bayesian Ridge Regression, LASSO, Naive Bayes, LightGBM, and XGBoost. For classification tasks, outputs from these models were combined using majority voting, often enhanced by small Neural Networks (NNs). For regression tasks, the mean of model outputs was used. Dense and Long Short-Term Memory (LSTM) networks were also utilized.
Datasets
The SemEval-2021 Task 7 dataset was used. This dataset comprised:
- Training Data: 9,000 annotated texts.
- Test Data: 1,000 texts.
The training data was provided in three parts: trial, development, and evaluation. The authors also mention using their development test set for validation during training and tuning.
Key Insights and Ideas
- Conventional ML's Efficiency Advantage: The paper highlights that conventional ML models, particularly boosting algorithms like LightGBM and XGBoost, offer significant advantages in training speed and hardware requirements compared to deep learning models. This makes them a practical choice for research tasks where computational resources are limited or rapid iteration is needed.
- Outlier Sensitivity: A key observation is that conventional ML models, while achieving respectable performance, struggle with outliers. This is particularly evident in humor detection, where ironic, reference-based, or highly controversial jokes can be misclassified. Similarly, in offense rating, outliers can disproportionately affect RMSE metrics.
- Deep Learning's Nuance Capture: The authors acknowledge that transformer-based models (like BERT) and even basic LSTM models can better capture the subtle nuances and complexities of humor and offense, leading to superior performance, especially with challenging cases.
- Ensemble Power: Combining the outputs of multiple models (through majority voting or averaging) proved crucial for improving performance, especially when dealing with a large number of individual models.
- Task-Specific Performance: While conventional ML struggled with outliers in humor classification, their performance on humor rating (Subtask 1b) and offense rating (Subtask 2) was closer to superior models, suggesting that the subjective nature of humor itself poses a greater challenge than the classification task's complexity.
Key Results
- Subtask 1a (Humor Classification): The best single non-NN model achieved an F1 score of 0.8706, while the best single NN model reached 0.8952. The winning submission achieved 0.9820 (F1). The DUTH team's best submission was ranked 52 out of 58.
- Subtask 1b (Humor Rating): The best non-NN model achieved an RMSE of 0.5411, and the best NN model achieved 0.5492. The winning submission achieved 0.4959 (RMSE). The DUTH team's best submission was ranked 12 out of 50.
- Subtask 1c (Controversial Humor Classification): The best non-NN model achieved an F1 of 0.6455, and the best NN model achieved 0.6491. The winning submission achieved 0.6302 (F1). The DUTH team's best submission was ranked 12 out of 36.
- Subtask 2 (Offense Rating): The best non-NN model achieved an RMSE of 0.8313, and the best NN model achieved 0.7552. The winning submission achieved 0.4120 (RMSE). The DUTH team's best submission was ranked 40 out of 48.
- Preprocessing Impact: Table 4 shows that different preprocessing methods had varying impacts, with some (like integer removal and capital tagging) showing improvements over plain tokenization and lowercasing.
Limitations and Future Work
Limitations: * Outlier Handling: Conventional ML models demonstrated a clear limitation in handling outliers, leading to misclassifications of nuanced or complex humor. * Performance Gap: The DUTH team's conventional ML models were generally outperformed by their own NN models and significantly by state-of-the-art transformer-based models. * Subjectivity: The inherent subjectivity of humor and controversiality makes these tasks particularly challenging, even for advanced models.
Future Work: The authors suggest expanding their preprocessing methods, exploring more advanced ensemble techniques, and investigating stacking methods to further improve performance. They also propose continuing to explore the trade-offs between computational efficiency and accuracy in NLP tasks.
Summary updated