Amobee at SemEval-2020 Task 7: Regularization of Language Model Based Classifiers

Alon Rozental, Dadi Biton, Ido BlankCOLING 2020

TL;DR

This paper tackles the challenge of assessing humor in edited news headlines by fine-tuning large language models (LMs). They introduce a novel L2-regularization technique that specifically targets the LM's weights during fine-tuning, preventing overfitting on small datasets. This approach, combined with an ensemble of different LMs, achieved impressive second-place rankings in both sub-tasks of the SemEval-2020 Task 7.

What Problem Does This Paper Address?

The paper addresses the intriguing problem of automatically detecting and quantifying humor in text, specifically focusing on how small edits to news headlines can alter their funniness. The SemEval-2020 Task 7 presented two sub-tasks: (1) predicting the funniness score of a single edited headline and (2) choosing which of two edited headlines is funnier. A key challenge in this domain is the inherent subjectivity of humor and the difficulty of capturing it with traditional NLP methods. Furthermore, state-of-the-art language models, while powerful, are notoriously prone to overfitting when fine-tuned on relatively small datasets, which is a common scenario in specialized NLP tasks like humor detection.

Methods

The authors' approach centers on fine-tuning pre-trained language models (BERT, XL-NET, and Roberta) for the humor assessment task. Their core innovation lies in a novel L2-regularization strategy. Instead of applying standard L2 regularization to the entire model (which would be meaningless given the vast number of parameters in LMs and small datasets), they apply it to the change in weights caused by the fine-tuning process. This regularization is applied with a higher coefficient to layers closer to the input embedding layer and a lower coefficient to upper layers, reflecting the idea that lower layers might be more involved in capturing syntactic and dependency information, which could be crucial for understanding humor. They combine this with a Mean Squared Error (MSE) loss function.

Additionally, they employ a "freeze weights" strategy for the initial epoch of training, preventing the random weights of the newly initialized classifier layers from negatively impacting the pre-trained LM. Finally, they utilize an ensemble method, combining the predictions of multiple fine-tuned LMs (BERT, XL-NET, and Roberta) with a weighted average, where weights were determined by validation performance.

Datasets

The primary dataset used is from the SemEval-2020 Task 7: "Assessing Humor in Edited News Headlines." * Sub-task 1: 12,071 training samples and 3,024 test samples. * Sub-task 2: 11,736 training samples and 2,960 test samples.

The authors note that for sub-task 2, they did not train a separate model but rather used the predictions from their sub-task 1 model, relying on the test set for evaluation. They also set aside 1% of the training data for validation during their experimental setup.

Key Insights and Ideas

  • Targeted Regularization is Crucial for LM Fine-tuning: The paper strongly suggests that standard regularization techniques are insufficient for fine-tuning large LMs on small datasets. Their novel approach of regularizing the change in LM weights is a key insight for preventing overfitting and improving performance.
  • Layer-Specific Regularization Matters: Applying different regularization strengths to different layers of the LM, based on their presumed function (e.g., lower layers for syntax, higher for semantics), appears to be an effective strategy.
  • Ensembling Boosts Performance: Combining multiple fine-tuned LMs, even with similar architectures, provides a significant performance boost, highlighting the benefit of diverse model perspectives.
  • Humor is a Complex Linguistic Phenomenon: The task itself underscores the difficulty of capturing subjective qualities like humor with computational models, requiring sophisticated techniques beyond simple text classification.

Key Results

The proposed system achieved strong results in both sub-tasks of SemEval-2020 Task 7, securing second place in both:

  • Sub-task 1 (Funniness Prediction): The ensemble model achieved a Root Mean Squared Error (RMSE) of 0.507, outperforming individual models and a baseline.
  • Sub-task 2 (Funnier Headline Selection): The ensemble model achieved an accuracy of 65.8%, also outperforming individual models and a baseline.

The paper also shows that the ensemble based on Roberta alone performed nearly as well as the full ensemble, suggesting Roberta's strong individual performance on this task.

Limitations and Future Work

The authors acknowledge that their system, while successful, is an ensemble of models. They plan to release the complete, fully trained versions of their models in the future. They also express interest in testing the effectiveness of their L2-regularization approach on other NLP tasks, such as topic classification and sentiment analysis, to further validate its general applicability.

Summary updated

BibTeX
@inproceedings{rozental2020amobee,
  author = {Alon Rozental and Dadi Biton and Ido Blank},
  title = {{Amobee at SemEval-2020 Task 7: Regularization of Language Model Based Classifiers}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}
Figure