“President Vows to Cut <Taxes> Hair”: Dataset and Analysis of Creative Text Editing for Humorous Headlines

NAACL 2019

TL;DR

This paper introduces Humicroedit, a novel dataset of edited news headlines designed to be humorous, along with an analysis of what makes them funny. They also develop baseline classifiers to predict headline funniness, paving the way for automated humor generation.

What Problem Does This Paper Address?

Creating and understanding humor computationally is a notoriously difficult AI problem. While progress has been made in humor detection, automated humor generation lags significantly. A major bottleneck is the scarcity of high-quality, publicly available datasets that go beyond simple templates. Existing datasets often focus on specific joke structures or require extensive manual annotation. This paper aims to address this by creating a dataset of humorous headlines derived from real news, allowing for a more nuanced understanding of what makes text funny, particularly through simple, targeted edits.

Methods

The core of this work is the creation of the Humicroedit dataset. This involved a two-stage crowdsourcing process:

  1. Headline Editing: Qualified editors on Amazon Mechanical Turk (MTurk) were tasked with making existing news headlines funny by applying a single "micro-edit." A micro-edit was defined as a single-word noun or verb replacement, with strict rules against phrase manipulation or sub-token changes to ensure focused analysis.
  2. Headline Grading: Another set of qualified MTurk judges then rated the funniness of these edited headlines on a scale of 0-3.

The authors also performed several analyses:

  • Humor Theory Analysis: They investigated how their data supports established humor theories like incongruity, superiority, and setup/punchline.
  • Clustering of Replacement Words: They used GloVe embeddings and k-means clustering to group the replacement words used by editors, identifying common humor generation strategies.
  • Baseline Classifiers: They developed and evaluated several machine learning models (logistic regression, random forest, and a bi-directional LSTM) to predict whether an edited headline is funny, using both n-gram and GloVe embedding features.

Datasets

The primary dataset introduced in this paper is Humicroedit.

  • Source: Original news headlines were collected from popular subreddits (r/worldnews, r/politics) on Reddit.
  • Content: The dataset consists of 15,095 unique edited headlines, each paired with five funniness scores from human judges.
  • Distinction: Unlike previous datasets, Humicroedit focuses on simple, single-word edits to real news headlines, allowing for a fine-grained analysis of humor mechanisms and providing a foundation for generating topical humor.

Key Insights and Ideas

  • Micro-edits Reveal Humor Mechanisms: The constraint of single-word edits proved effective in isolating specific linguistic and semantic changes that contribute to humor, making it easier to analyze what makes a headline funny.
  • Humor Theories Supported: The data quantitatively supports classic humor theories, such as incongruity (where distant word replacements lead to more humor) and superiority (evidenced by the prevalence of "insults" and "human deficiencies" in replacement word clusters).
  • Setup and Punchline Matters: Editors tended to place their humorous edits towards the end of headlines, aligning with the "setup and punchline" theory of humor.
  • Context is Key for Humor: Longer headlines, with more contextual information, generally offered higher potential for humor, as they provided more opportunities for editors to create surprising or incongruous edits.
  • Entities vs. Nouns/Verbs: The LSTM classifier performed better when the replaced word was a noun or verb rather than an entity, suggesting that entity-based humor might rely more heavily on external world knowledge that current models struggle to capture.

Key Results

  • The Humicroedit dataset contains 15,095 unique edited headlines with five funniness scores per headline.
  • The authors developed baseline classifiers, with a bi-directional LSTM achieving a best accuracy of 68.54% in predicting whether an edited headline is funny.
  • Analysis showed that replacements creating incongruity (measured by GloVe vector distance) correlated positively with humor scores.
  • The study identified common humor generation strategies, such as creating meaningful n-grams, using semantically distant or phonetically similar replacements, and making strong connections with headline entities.

Limitations and Future Work

  • Annotation Bias: The authors acknowledge that annotator preferences, biases, and stances can influence humor judgment, leading to a modest inter-annotator agreement score.
  • Brittleness of Models: The LSTM classifier, while achieving decent accuracy, was found to be somewhat brittle, with minor text modifications sometimes fixing classification errors. This suggests that current models may not fully grasp the nuances of humor.
  • World Knowledge Gap: The models struggled with humor that relies heavily on common sense, world knowledge, and cultural references, especially when entities were involved.
  • Future Work: The authors suggest exploring deeper features that incorporate world knowledge and current events. They also propose using the dataset to automatically generate personalized humorous headlines and to analyze humor in professional comedians' work.

Summary updated

BibTeX
@inproceedings{2019president,
  title = {{“President Vows to Cut <Taxes> Hair”: Dataset and Analysis of Creative Text Editing for Humorous Headlines}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure