ZYJ at SemEval-2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense with ALBERT-Based Model

Yingjia Zhao, Xin TaoACL 2021

TL;DR

This paper presents a deep learning model for detecting and rating humor and offense in text, leveraging the ALBERT language model. The authors propose a custom network architecture built on top of ALBERT to enhance semantic feature extraction, achieving competitive results in the SemEval-2021 HaHackathon task.

What Problem Does This Paper Address?

Humorous language, while often enjoyable, can easily lead to offense. Distinguishing between humor and offense, and accurately rating their intensity, is a complex challenge due to the subjective nature of language, cultural nuances, and the need for deep semantic understanding. This paper tackles this challenge by developing a system for SemEval-2021 Task 7: HaHackathon, which aims to automatically detect and rate humor and offense in text. The task is broken down into subtasks: predicting if text is humorous, rating its humor intensity, determining if humor is controversial, and rating offensiveness.

Methods

The core of the proposed system is the ALBERT (A Lite BERT for self-supervised Learning of Language) model. ALBERT is chosen for its efficiency compared to BERT, utilizing techniques like cross-layer parameter sharing and factorized embedding parameterization to reduce model size and training time while maintaining strong performance.

The authors extend ALBERT by adding a custom network structure to the upper layers for more sophisticated semantic information processing. Specifically, the output from the last ALBERT hidden layers is fed into a BiLSTM (Bidirectional Long Short-Term Memory) network. This is followed by a ReLU (Rectified Linear Unit) activation function for non-linear mapping to a lower dimension. To extract features, both average pooling and maximum pooling are applied. The outputs from these pooling layers are then concatenated. Finally, this combined feature vector is concatenated with the original output from ALBERT to perform the classification or regression tasks.

For the binary classification subtasks (1a and 1c), the researchers employed Stratified K-fold cross-validation with K=5. This technique ensures that the proportion of each class in the training and validation sets remains consistent with the original dataset, preventing data distribution disorders.

Datasets

The system was trained and evaluated on the dataset provided for SemEval-2021 Shared Task 7: HaHackathon. This dataset consists of 8000 pieces of data for training, 1000 for validation, and 1000 for testing. Each data instance includes four tags related to humor and offense.

Key Insights and Ideas

  • ALBERT as an efficient backbone: The choice of ALBERT highlights the trend towards more parameter-efficient transformer models for NLP tasks, balancing performance with computational resources.
  • Custom architecture for enhanced semantics: Simply using a pre-trained language model is often not enough. The addition of a BiLSTM layer and pooling mechanisms demonstrates a strategy to further refine and aggregate semantic features extracted by ALBERT.
  • Stratified K-fold for robust classification: For imbalanced datasets or binary classification tasks, Stratified K-fold cross-validation is crucial for ensuring reliable model evaluation and preventing bias.

Key Results

The ZYJ team achieved the following results in the SemEval-2021 HaHackathon task:

  • Task 1a Humor Detection: F-Score of 0.9348 (Rank 41)
  • Task 1b Average Humor Score: RMSE of 0.7214 (Rank 43)
  • Task 1c Humor Controversy: F-Score of 0.4603 (Rank 33)
  • Task 2 Average Offensiveness Score: RMSE of 0.5204 (Rank 31)

While these results are competitive, the authors acknowledge that the system's performance needs improvement.

Limitations and Future Work

The authors explicitly state that although their system is relatively lightweight and easier to train due to ALBERT's efficiency, its performance requires further enhancement.

For future work, they plan to optimize the system structure further. The goal is to enable the model to capture richer semantic information characteristics, which is crucial for accurately understanding and rating humor and offense. This could involve exploring different network architectures, attention mechanisms, or incorporating external knowledge sources.

Summary updated

BibTeX
@inproceedings{zhao2021zyj,
  author = {Yingjia Zhao and Xin Tao},
  title = {{ZYJ at SemEval-2021 Task 7: HaHackathon: Detecting and Rating Humor and Offense with ALBERT-Based Model}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure