How Did This Get Funded?! Automatically Identifying Quirky Scientific Achievements

Chen Shani, Nadav Borenstein, Dafna Shahaf

TL;DR

This paper tackles the challenge of automatically identifying "quirky" and humorous scientific papers, inspired by the Ig Nobel Prize. They introduce a novel dataset and a hybrid approach combining linguistic/psychological features with state-of-the-art NLP models to detect these unconventional achievements. Their work demonstrates that even simple models incorporating domain-specific insights can rival complex deep learning architectures in this nuanced task.

What Problem Does This Paper Address?

Computational humor is a notoriously difficult problem, often considered "AI-complete." While much research focuses on specific joke formats like knock-knock jokes or one-liners, this paper addresses a more sophisticated and subtle form of humor: the quirky, often unexpected, scientific achievement that makes people laugh and then think. Identifying such papers is valuable not only for understanding humor but also for potentially highlighting creative, boundary-pushing research that might otherwise be overlooked. The authors are motivated by the Ig Nobel Prize, which celebrates these unusual scientific contributions.

Methods

The authors propose a novel task: automatically detecting funny scientific papers, primarily by analyzing their titles. Their approach is two-fold:

  1. Literature-Inspired Features: They engineer 127 features based on established theories of humor from psychology and linguistics. These include measures of:
  2. Unexpected Language: Using various language models (N-gram, POS-tag based, and Transformer-based like BERT and GPT-2) to quantify the "perplexity" or unexpectedness of words in a title.
  3. Simple Language: Metrics like title length, word length variance, and automated readability indices.
  4. Crude Language: A classifier trained on toxic comments, combined with perplexity to identify unexpected crude terms.
  5. Funny Language: Quantifying the inherent "funniness" of nouns in a title.
  6. NLP Models: They compare several models:
  7. Iggy: A simple Multi-Layer Perceptron (MLP) classifier trained on the engineered features.
  8. Pre-trained Transformer Models: BERT and SciBERT (a BERT variant trained on scientific text), both fine-tuned for the task.
  9. Hybrid Models: BERT/SciBERT models that combine the pre-trained embeddings with the literature-inspired features.

Datasets

  • Positive Examples: The dataset includes titles of Ig Nobel winners, supplemented by humorous papers found online (totaling 1,707 papers).
  • Negative Examples: 1,707 titles randomly sampled from Semantic Scholar, balanced across scientific fields.
  • Large Corpus: A dataset of 630,000 titles from Semantic Scholar was used for training language models and for the "in the wild" evaluation.

Key Insights and Ideas

  • Titles as Proxies: The authors found that for many Ig Nobel-worthy papers, the title alone is sufficient to convey the humor, making title analysis a viable and efficient approach.
  • Domain Expertise Matters: The engineered features, derived from humor theory, proved to be highly informative, even for complex Transformer models. This suggests that combining traditional knowledge with modern NLP can yield superior results.
  • Simplicity and Unexpectedness are Key: Features related to linguistic simplicity and unexpectedness (perplexity) were consistently strong predictors of humor.
  • Humor is Subjective, but Detectable: While humor is subjective, the study demonstrates that models can learn to identify patterns associated with it, especially when guided by domain insights.
  • Distinguishing Funny Titles from Funny Topics: The authors highlight the challenge of differentiating between a research topic that is inherently funny and a title that is cleverly worded to be humorous.

Key Results

  • High Accuracy: All models achieved high accuracy on the labeled dataset, with SciBERT generally outperforming others.
  • Feature Importance: The literature-inspired features were crucial, significantly improving the performance of BERT and SciBERT when combined.
  • "In the Wild" Performance: In a realistic setting evaluating performance on a large corpus, the "Iggy" model (feature-based MLP) showed strong performance, particularly at the top of the ranked list, outperforming even some Transformer-based models for the first 50 results.
  • Overlap Analysis: The overlap between feature-based models was high, suggesting the features captured similar aspects of humor. BERT and SciBERT showed less overlap, indicating they might rely more on their pre-trained knowledge.
  • Qualitative Observations: The models sometimes confused funny titles with funny topics, particularly with catchy editorial titles. They also learned that longer titles tend to be more serious.

Limitations and Future Work

  • Subjectivity of Humor: The inherent subjectivity of humor makes perfect classification challenging, as evidenced by the low agreement among human annotators on topic funniness.
  • Dataset Bias: The dataset might contain biases, and the authors acknowledge that filtering out editorials from real research papers was not straightforward.
  • Model Confusion: Models sometimes confused funny titles with funny topics, and the rudeness classifier's role was inconclusive.
  • Future Directions:
    • Creating aligned corpora of funny/serious titles to pinpoint exact humor-inducing words.
  • Investigating whether funny science receives disproportionate attention.
  • Expanding analysis to paper abstracts and full texts.
  • Exploring multi-task learning for more robust humor classification.
  • Promoting the integration of data-driven research with insights from traditional fields like psychology.

Summary updated

BibTeX
@inproceedings{shani2021how,
  author = {Chen Shani and Nadav Borenstein and Dafna Shahaf},
  title = {{How Did This Get Funded?! Automatically Identifying Quirky Scientific Achievements}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=jkrzgRta0T},
}
Figure