UMUTeam at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Linguistic Features and Word Embeddings

José Antonio García-Díaz, Rafael Valencia-GarcíaACL 2021

TL;DR

This paper presents UMUTeam's approach to detecting and rating humor and offense in text for SemEval-2021. They combined linguistic features with pre-trained and contextual word embeddings, achieving their best results on the controversial humor subtask. Their work highlights the ongoing challenge of robustly understanding nuanced language like humor for NLP systems.

What Problem Does This Paper Address?

Humor and offense in text are notoriously difficult for Natural Language Processing (NLP) systems to grasp. This is largely due to the figurative language, subjectivity, and context-dependency inherent in these forms of communication. Unlike literal language, where words have a direct meaning, humor often relies on words and expressions taking on a different, often ironic or exaggerated, meaning. This ambiguity makes tasks like sentiment analysis or emotion detection less effective. The HaHackathon 2021 challenge aimed to address this by focusing on identifying and rating humorous and offensive content, pushing the boundaries of NLP's understanding of these complex linguistic phenomena.

Methods

UMUTeam's approach involved a multi-faceted strategy combining two main types of features:

  • Linguistic Features (LF): These were extracted using a tool called UMUTextStats, which was inspired by LIWC but developed from scratch for Spanish (though adapted for English in this task). The LF captured various aspects of text, including:
  • Expressive lengthening (drawing out words for emphasis).
  • Common typing mistakes (e.g., lowercase starts, repeated words).
  • Corpus statistics (like type/token ratio, sentence length).
  • Punctuation and emoticons.
  • Social media specific elements (hashtags, jargon).
  • Word Embeddings:
    • Pre-trained Word Embeddings (PWE): They utilized embeddings from fastText and GloVe.
  • Contextual Word Embeddings (CWE): They employed BERT, a powerful transformer-based model, to capture context-dependent word meanings.

These features were then fed into different neural network architectures. They explored combining LF with PWE using various architectures (MLP, CNN, LSTM, BiLSTM, GRU, BiGRU) and also evaluated fine-tuning BERT with LF.

Datasets

The primary dataset used was from the HaHackathon 2021 challenge. This dataset consisted of 10,000 English tweets, split into training (80%), development (10%), and testing (10%) sets. The tweets were annotated for humor (binary classification and a 1-5 rating) and offense (binary classification and a 1-5 rating). The organizers also provided pre-trained word embedding models and BERT.

Key Insights and Ideas

  • The Synergy of Features: While contextual embeddings like BERT often achieve state-of-the-art results, UMUTeam found that combining linguistic features with word embeddings generally yielded better performance across most subtasks. This suggests that explicit linguistic cues, which might be missed by purely data-driven embedding models, are still valuable for understanding humor and offense.
  • Subtask Specific Architectures: The optimal neural network architecture varied significantly depending on the subtask. For instance, MLP and CNN performed well for humor detection (subtask 1a), while recurrent networks (RNNs, LSTMs) were more effective for humor and offense rating (subtasks 1b, 1c, 2a).
  • The Importance of Context (and its Limits): BERT's contextual understanding was powerful, but it didn't always outperform simpler models when combined with linguistic features. This hints that the specific nature of humor and offense might require more than just contextual word meaning.
  • Hyperparameter Sensitivity: The performance was highly sensitive to hyperparameter choices, including network shape, number of layers, dropout rates, and learning rates, underscoring the need for thorough tuning.

Key Results

UMUTeam achieved their best performance on subtask 1c (controversial humor), where they trained a binary classifier. While specific metrics for this subtask aren't detailed in the summary, they state it was their strongest result. In the overall comparison with other participants on the test dataset:

  • Subtask 1a (Humor Detection): They achieved an F1-score of 91.60% and an accuracy of 93.25%, ranking 45th out of 53.
  • Subtask 1b (Humor Rating): Their RMSE was 0.8847, placing them 47th, below the baseline.
  • Subtask 1c (Offense Detection): They achieved an F1-score of 57.22% and an accuracy of 46.50%, ranking 14th.
  • Subtask 2a (Offense Rating): Their RMSE was 0.8740, ranking 46th, also below the baseline.

Limitations and Future Work

The authors acknowledge several limitations:

  • Time Constraints: They were unable to perform all desired experiments, including re-fitting models with the final development dataset labels once they were released.
  • Limited Runs: They submitted only one run for the final stage, whereas other participants submitted multiple runs, potentially exploring more hyperparameter combinations.
  • Feature Set: They only evaluated a subset of linguistic features for English, suggesting that a more comprehensive set, potentially adapted from their Spanish UMUTextStats tool, could improve results.
  • Regression Tasks: Their performance on regression subtasks (1b and 2a) was below the baseline, indicating potential implementation or conceptualization errors that need investigation.

Future work directions include:

  • Reviewing their pipeline to identify weaknesses, particularly for the regression tasks.
  • Adapting their UMUTextStats tool for English and comparing its reliability with established tools like LIWC.
  • Focusing on model interpretability, as linguistic features are believed to lead to more interpretable models.
  • Evaluating machine learning ensembles as a method for combining different feature sets.

Summary updated

BibTeX
@inproceedings{garciadiaz2021umuteam,
  author = {José Antonio García-Díaz and Rafael Valencia-García},
  title = {{UMUTeam at SemEval-2021 Task 7: Detecting and Rating Humor and Offense with Linguistic Features and Word Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure