In traditional natural language processing tasks, human disagreement is often treated as noise. If two annotators disagree on whether a word is a noun or a verb, the solution is typically to refine the guidelines or rely on a majority vote to establish a single "gold standard" label.
However, when evaluating creative, subjective, or culturally dependent text, this paradigm breaks down. Humor is a prime example: a joke that makes one person laugh might leave another completely indifferent or even offended. In computational humor, treating human disagreement as a bug rather than a core feature limits a model's ability to understand how humor actually operates in the real world.
Instead of trying to force a consensus where none exists, a growing body of research focuses on explicitly modeling humor subjectivity and annotator disagreement.
The Illusion of the Gold Standard
The challenge of subjective variance in humor has been recognized since the earliest days of computational creativity. When developing an automated system for generating punning riddles, Binsted et al. observed extreme variance in human evaluations, noting that the exact same generated output could be rated as high-quality by one judge and entirely non-humorous by another.
As the field shifted toward deep learning and large-scale datasets, this variance did not disappear; it became more measurable. For example, when Hossain et al. introduced the Humicroedit dataset—where humor is induced by replacing a single word in a news headline—they found only modest agreement among human judges. The ratings were heavily influenced by the individual annotators' personal preferences, backgrounds, and political stances.
When models are trained on datasets where the ground truth is inherently unstable, their performance hits an artificial ceiling. As Liu et al. point out, the ambiguity of humor labels frequently leads to misclassifications, not because the model fails to understand the text, but because the dataset itself contains conflicting human perspectives.
Quantifying Friction via Humor Controversy
To move beyond simple binary classification ("is it funny?"), researchers formalized the variance in human perception into a distinct metric: humor controversy.
As defined by Meaney et al. during the SemEval-2021 HaHackathon, humor controversy is the degree of disagreement among annotators regarding the comedic value of a text. Under this framework, a joke's reception is characterized not just by its average score, but by the spread of its distribution.
[Joke A] Annotator Ratings: [ 3, 3, 3, 3 ] --> Mean: 3, Variance: 0 (Universal Consensus)
[Joke B] Annotator Ratings: [ 1, 1, 5, 5 ] --> Mean: 3, Variance: 4 (High Controversy)
Predicting this variance turns out to be significantly harder than predicting the mean. While large transformer models excel at identifying intended humor, they struggle to predict whether a joke will divide an audience. Chi et al. found that complex deep learning architectures often fail to outperform simple, heuristic-based baselines when predicting controversy.
Similarly, Xie et al. observed that even state-of-the-art models like DeBERTa struggle with controversy detection because the task lacks objective textual criteria, relying instead on the unmapped social and psychological profiles of the readers.
Why Standard NLP Paradigms Fail
Why do standard machine learning techniques struggle to capture this subjectivity?
- Data Augmentation Distorts Nuance: In many NLP tasks, swapping words or paraphrasing sentences is a reliable way to expand training data. However, Gupta et al. demonstrated that standard data augmentation can "kill the joke" by disrupting the delicate linguistic balance required for humor. Furthermore, the authors found no clear correlation between how offensive a joke is and how controversial it is, proving that subjectivity cannot simply be treated as a proxy for toxicity.
- Noisy Social Media Proxies: To bypass expensive human annotation, many datasets rely on proxy metrics like upvotes or retweets. However, as Weller et al. and Ofer et al. argue, upvote counts reflect community-specific trends, algorithmic feed distribution, and audience reach rather than an inherent comedic consensus.
Embracing the Distribution
The solution to the problem of subjectivity is a paradigm shift: learning with disagreements.
Instead of flattening human annotations into a single majority-vote label, models can be trained to predict the full distribution of human opinions. As Uma et al. demonstrate across the Le-wi-Di shared tasks, preserving the complete distribution of annotations allows models to capture the inherent ambiguity of real-world data.
This distribution-centric view is especially vital as computational humor expands into multimodal domains. Whether evaluating the visual irony of internet memes (Bejan; Wang et al.), the complex narrative juxtapositions in comics (Hu et al.), or the timing of laughter in short-form videos (Ko et al.; Hyun et al.), the goal is no longer to teach a machine what is universally "funny." Instead, the goal is to teach the machine to understand who will find it funny, and why.
Key Foundational Work
An Implemented Model of Punning Riddles (1994) — contributed 1 concept elements to this direction.