UPB at SemEval-2021 Task 7: Adversarial Multi-Task Learning for Detecting and Rating Humor and Offense

Răzvan-Alexandru Smădu, Dumitru-Clementin Cercel, Mihai DascaluACL 2021

TL;DR

This paper introduces AMTL-Humor, a novel adversarial multi-task learning framework for detecting and rating humor and offense in text. By combining BERTweet embeddings with BiLSTM and Capsule layers, and incorporating adversarial training and orthogonality constraints, the model achieves state-of-the-art results on SemEval-2021 Task 7, demonstrating the effectiveness of this approach for nuanced sentiment analysis.

What Problem Does This Paper Address?

Detecting and understanding humor and offense in text is a notoriously difficult task, even for humans. This challenge stems from the inherent ambiguity of language, where words can carry multiple meanings and their interpretation is heavily context-dependent. A statement that is humorous in one situation might be offensive in another, and individual backgrounds, cultural nuances, and even current moods can influence perception. SemEval-2021 Task 7 specifically aimed to tackle this by introducing four sub-tasks: classifying text as humorous or not (Task 1a), rating the degree of humor (Task 1b), classifying controversial texts (Task 1c), and rating the level of offensiveness (Task 2). Existing methods often struggle to capture these subtle distinctions, highlighting the need for more sophisticated models that can learn from multiple related tasks simultaneously.

Methods

The core of the proposed solution is the AMTL-Humor architecture, which leverages adversarial multi-task learning (MTL). The model starts with BERTweet, a Transformer-based model pre-trained on a massive corpus of English tweets, to generate contextualized embeddings. These embeddings are then fed into five branches: one shared branch for non-task-specific information and four task-specific branches. Each task-specific branch employs a Bidirectional LSTM (BiLSTM) for feature extraction, followed by Capsule layers. Capsule layers are a novel approach inspired by human vision, designed to capture hierarchical relationships and local order of words, offering a more robust representation than traditional pooling methods.

An adversarial component is introduced through a discriminator that tries to distinguish between features from the different branches. This adversarial training encourages the model to learn task-specific representations while preventing information leakage between tasks. Additionally, an orthogonality constraint is applied to further enforce the separation between shared and private representations. The overall loss function is a combination of task-specific losses, the adversarial loss, and the orthogonality constraint, with parameters to balance their influence. For the final submission, an ensemble of different configurations of this architecture was used, combining predictions from multiple models to achieve more robust results.

Datasets

The primary dataset used in this work is the SemEval-2021 Task 7 dataset, which comprises 10,000 English texts. This dataset is annotated for humor classification (Task 1a), humor rating (Task 1b), controversy classification (Task 1c), and offensiveness rating (Task 2). The dataset was pre-split into training (8,000 samples), development (1,000 samples), and testing (1,000 samples) sets. Notably, the annotations for the test set were provided only after the evaluation phase.

Key Insights and Ideas

  • Adversarial MTL for Nuance: The paper demonstrates that adversarial learning, when applied to multi-task learning for text analysis, can significantly improve the model's ability to disentangle complex linguistic features related to humor and offense.
  • Capsule Networks for Text: The successful integration of Capsule layers in text processing, beyond their typical use in computer vision, highlights their potential for capturing hierarchical and positional information in sequential data.
  • Task Interdependence: The finding that training Task 1c data on the Task 1a branch improved Task 1a performance suggests that even seemingly distinct tasks can benefit from shared learning, especially when they involve subjective interpretations.
  • Ensembling for Robustness: The ensemble approach proved crucial for achieving the best overall performance, underscoring the benefit of combining diverse model predictions for more stable and accurate outcomes.

Key Results

The AMTL-Humor model achieved impressive results on SemEval-2021 Task 7. The best performing configuration, an ensemble of models, achieved a 95.66% F1-score and 94.70% accuracy for Task 1a (humor classification). For the regression tasks, the ensemble model obtained RMSE scores of 0.6200 for Task 1b (humor rating) and 0.5318 for Task 2 (offensiveness rating). The paper also notes that while adversarial learning showed a small improvement on Task 1a, removing it led to a decrease in RMSE for regression tasks, indicating a trade-off that needs careful tuning.

Limitations and Future Work

The authors acknowledge that the model's performance on Task 1c (controversy classification) was relatively low, suggesting potential bias in the dataset towards neutral or missing values for this task. They also note that the AMTL-Humor model, particularly AMTL-Humor-1, registered higher RMSE scores for Task 1b compared to some baselines, indicating areas for improvement in humor rating.

For future work, the authors propose adapting hierarchical multi-task learning techniques to further enhance performance, especially for challenging tasks like Task 1c. They also suggest exploring different configurations and combinations of adversarial learning, orthogonality constraints, and Capsule layer variations to optimize the architecture.

Summary updated

BibTeX
@inproceedings{smadu2021upb,
  author = {Răzvan-Alexandru Smădu and Dumitru-Clementin Cercel and Mihai Dascalu},
  title = {{UPB at SemEval-2021 Task 7: Adversarial Multi-Task Learning for Detecting and Rating Humor and Offense}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure