This joke is [MASK]: Recognizing Humor and Offense with Prompting

Junze Li, Mengjie Zhao, Yubo Xie, Antonis Maronikolakis, Pearl Pu, Hinrich Schuetze

TL;DR

This paper explores "prompting" as a new transfer learning technique for humor recognition, showing it rivals traditional fine-tuning in high-resource settings and significantly outperforms it in low-resource scenarios. Crucially, they reveal that humor recognition models, even when trained on non-offensive data, may inadvertently learn to associate offensive content with humor, a concerning bias for real-world applications.

What Problem Does This Paper Address?

Humor is a vital part of human communication, and building NLP systems that can understand and generate humor is a key goal for more engaging human-computer interactions. A fundamental step in this is humor recognition – identifying whether a piece of text is humorous. While fine-tuning large pre-trained language models (PLMs) like BERT has been the standard approach, this paper investigates a newer technique called "prompting." Prompting reformulates tasks to fit the PLM's pre-training objective (like masked language modeling) without requiring a separate classifier head, potentially offering advantages, especially when labeled data is scarce.

Beyond just recognizing humor, the paper also tackles a critical ethical concern: the potential for humor recognition systems to inadvertently learn and perpetuate offensive biases. Humor is subjective, and datasets might contain jokes that are offensive, particularly along lines of gender. The authors aim to understand if models learn to associate offensive content with humor and how this might be mitigated.

Methods

The paper compares two transfer learning paradigms for humor recognition:

  • Finetuning: This is the standard approach where a PLM (BERT in this case) is adapted to a specific task by adding a new classification layer and training all parameters on the humor recognition dataset.
  • Prompting: This method leverages the PLM's existing masked language modeling (MLM) capabilities. A text is framed as a "prompt" with a [MASK] token, and the model is asked to predict a token from a predefined set (e.g., "funny" or "normal") to fill the blank. This avoids training a new classifier and reuses the MLM head.

To investigate the relationship between humor and offense, the authors employ Influence Functions (IFs). IFs are a technique to identify which training examples had the most significant impact on a model's prediction for a specific test example. By analyzing the most influential training examples, the researchers can understand what data points the model is "learning from" the most. They apply IFs to both finetuning and prompting methods to see if offensive training examples are disproportionately influential for humor recognition.

Datasets

The study uses two datasets:

  • HaHackathon Dataset (HHD): This is a SemEval 2021 dataset containing texts with binary humor labels and an offense score (1-5).
  • Gender Humor Dataset (GHD): This is a subset of HHD specifically curated for gender-related humor. The authors filtered for texts related to "Sexism" and ensured a balanced distribution of humorous and non-humorous examples, as well as a similar number of gender-related and non-gender-related texts to prevent simple heuristic-based predictions. For GHD, 80% of the data was used for training and 20% for testing.

Key Insights and Ideas

  • Prompting shines in low-resource settings: While prompting performs comparably to finetuning when ample data is available, its real strength emerges when training data is limited. This suggests prompting is a more data-efficient transfer learning strategy for humor recognition.
  • Offense is a learned signal for humor: A significant finding is that models, when identifying influential training examples, often find offensive texts to be highly important for predicting humor. This is true for both finetuning and prompting, but more pronounced in low-resource scenarios for finetuning.
  • Undesirable bias: The fact that models rely on offensive content to recognize humor is problematic. Even if a dataset contains non-offensive humorous examples, the model might still learn to associate offensive content with humor, leading to potentially harmful outputs in real-world applications.
  • Prompting's advantage in IF analysis: In low-resource scenarios, prompting combined with influence functions provides a clearer signal, correctly identifying influential examples for humorous texts as offensive and for non-humorous texts as less offensive. Finetuning, in contrast, struggles to disentangle this in low-resource settings.

Key Results

  • RQ1 (Prompting vs. Finetuning): In full data scenarios, prompting and finetuning achieve similar humor recognition performance. However, in few-shot scenarios (e.g., 16, 32, 64, 128 training examples), prompting consistently outperforms finetuning. For instance, with 128-shot data, prompting's performance is better than finetuning's.
  • RQ2 & RQ3 (Offensive Influential Examples):
    • Full Data: For humorous test examples, the top influential training examples for both finetuning and prompting show significantly higher average offense scores (e.g., 1.39 for FT Top10, 1.02 for PT Top10) compared to the overall average offense score of training examples (0.64). For non-humorous test examples, influential examples have lower offense scores.
  • Few-Shot Data (128-shot): Finetuning struggles, with influential examples for non-humorous test cases also showing high offense scores. Prompting, however, maintains a better distinction, with influential examples for humorous texts being more offensive than the reference, and those for non-humorous texts being less offensive.

Limitations and Future Work

  • Limitations:
    • The study focuses on gender-related humor and offense. The findings might not generalize to other types of offense or humor.
  • While prompting shows promise, the paper acknowledges that finetuning can still be competitive in high-resource settings.
  • The influence function analysis, while insightful, is an approximation and doesn't fully capture the model's internal workings.
  • Future Work:
    • Develop methods to mitigate the learned offense bias in humor recognition systems.
  • Investigate offense related to aspects beyond gender/sexism.
  • Explore other transfer learning paradigms and their effectiveness for humor recognition.

Summary updated

Abstract

Humor is a magnetic component in everyday human interactions and communications. Computationally modeling humor enables NLP systems to entertain and engage with users. We investigate the effectiveness of prompting, a new transfer learning paradigm for NLP, for humor recognition. We show that prompting performs similarly to finetuning when numerous annotations are available, but gives stellar performance in low-resource humor recognition. The relationship between humor and offense is also inspected by applying influence functions to prompting; we show that models could rely on offense to determine humor during transfer.
BibTeX
@inproceedings{li2022this,
  author = {Junze Li and Mengjie Zhao and Yubo Xie and Antonis Maronikolakis and Pearl Pu and Hinrich Schuetze},
  title = {{This joke is [MASK]: Recognizing Humor and Offense with Prompting}},
  booktitle = {TL4NLP},
  year = {2022},
  url = {https://openreview.net/forum?id=tQCDQ5j6VC},
}
Figure