The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation

Tyler Loakman, Aaron Maladry, Chenghua Lin

Abstract

Human evaluation is widely considered the "gold standard" for assessing Natural Language Generation (NLG) systems. However, for subjective tasks such as humor, irony, and sarcasm generation, the validity of this standard depends heavily on who is performing the evaluation. This paper presents a systematic review of 18 recent studies in the field, revealing a significant transparency gap: researchers rarely report the demographic or cultural backgrounds of their evaluators. By failing to document these variables, the community risks developing AI that caters to narrow subsets of the population while ignoring the diverse ways figurative language is perceived globally. The authors propose a standardized "Evaluation Statement" to improve reporting and replicability in future research.

Introduction

In the pursuit of creating AI that can truly mimic human wit, researchers rely on human judgment to determine if a generated pun is funny or if a sarcastic retort is successful. Unlike objective tasks like machine translation or summarization—where accuracy can be measured against a reference text—humor and irony are "in the eye of the beholder." A joke's success depends on the evaluator's age, cultural background, political leanings, and linguistic nuances.

Despite this inherent subjectivity, the NLP community often treats human evaluation as a monolithic, objective truth. This paper addresses the resulting transparency and replicability crisis, arguing that without knowing the demographic makeup of the "human" in human evaluation, we cannot truly understand or compare the performance of generative models.

Methodology: Auditing the State of the Field

The authors conducted a critical survey and systematic review of the figurative language generation landscape. The study followed a multi-step process:

  1. Literature Selection: The researchers filtered the ACL Anthology for papers published between 2018 and 2023 specifically focusing on the generation of humor, irony, or sarcasm.
  2. Taxonomy Development: They analyzed how specific linguistic features, such as puns or situational irony, rely on "world knowledge" and social conventions.
  3. Reporting Audit: A pool of 18 core papers (including works like Pun-GAN and ExPUNations) was evaluated against a rigorous checklist of transparency criteria, including evaluator demographics, recruitment methods, and compensation.
  4. Framework Proposal: Based on the findings, the authors developed a proposal for a standardized "Evaluation Statement" for NLG research.

The Subjectivity Challenge: Why Demographics Matter

The paper highlights several ways in which evaluator background influences the perception of figurative language:

  • Knowledge Priors: Humor often relies on "incongruity resolution." For example, a pun involving English homophones requires specific linguistic knowledge, while political satire requires familiarity with specific regional events. If an evaluator lacks these priors, the AI's output is judged as a failure, regardless of its technical quality.
  • The "WEIRD" Bias: Much of current AI evaluation relies on participants from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies. This creates a cultural feedback loop where models are optimized for a specific cultural palate, potentially alienating or misinterpreting other global perspectives.
  • Perspectivism vs. Noise: Traditional NLP treats evaluator disagreement as "noise" to be averaged out. The authors argue for a "perspectivist" approach, where disagreement is viewed as a valuable signal indicating how different demographic groups interpret the same text.

Key Results: A Crisis of Transparency

The audit of the 18 generation papers revealed a startling lack of detail regarding the human element of evaluation:

  • Demographic Blind Spots: Less than 42% of the papers reported any demographic information. Most shockingly, 0% of the surveyed papers reported the age, gender, or social class of their evaluators.
  • Recruitment Trends: Over 70% of studies relied on crowdsourcing platforms like Amazon Mechanical Turk (AMT) or Prolific. While efficient, these platforms often suffer from "demand characteristics," where workers provide the answers they believe researchers want to hear to ensure payment.
  • Compensation and Training: Only 30% of papers explicitly mentioned evaluator compensation (with rates varying from $9 to $20 USD per hour). Furthermore, only 24% provided evidence of training materials or qualifier tasks to ensure evaluators understood the specific nuances of the task (e.g., the difference between irony and sarcasm).

Discussion and Future Directions

The authors conclude that the current "black box" approach to human evaluation undermines the progress of the field. To move forward, they propose that all future NLG research include a formal Evaluation Statement covering:

  1. Logistics: Detailed counts of evaluators and sample distribution.
  2. Demographics: Mandatory reporting of age range, native language (L1), and cultural/geographic background.
  3. Process Transparency: Clear definitions of the phenomena being studied and the specific training materials provided to participants.
  4. Stratified Sampling: A shift away from convenience sampling on crowdsourcing platforms toward intentionally diverse evaluator panels.

By adopting these standards, the NLP community can transition from a narrow "melting pot" that erases difference to a more robust, inclusive, and scientifically sound framework for evaluating the complexities of human language.

Summary updated

Abstract

Human evaluation in often considered to be the gold standard method of evaluating a Natural Language Generation system. However, whilst its importance is accepted by the community at large, the quality of its execution is often brought into question. In this position paper, we argue that the generation of more esoteric forms of language - humour, irony and sarcasm - constitutes a subdomain where the characteristics of selected evaluator panels are of utmost importance, and every effort should be made to report demographic characteristics wherever possible, in the interest of transparency and replicability. We support these claims with an overview of each language form and an analysis of examples in terms of how their interpretation is affected by different participant variables. We additionally perform a critical survey of recent works in NLG to assess how well evaluation procedures are reported in this subdomain, and note a severe lack of open reporting of evaluator demographic information, and a significant reliance on crowdsourcing platforms for recruitment.
BibTeX
@inproceedings{loakman2023iron,
  author = {Tyler Loakman and Aaron Maladry and Chenghua Lin},
  title = {{The Iron(ic) Melting Pot: Reviewing Human Evaluation in Humour, Irony and Sarcasm Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.444},
}
Figure