TL;DR
This paper tackles the challenge of automatically identifying "quirky" and humorous scientific papers, inspired by the Ig Nobel Prize. They introduce a novel dataset and a hybrid approach combining linguistic/psychological features with state-of-the-art NLP models to detect these unconventional achievements. Their work demonstrates that even simple models incorporating domain-specific insights can rival complex deep learning architectures in this nuanced task.
What Problem Does This Paper Address?
Computational humor is a notoriously difficult problem, often considered "AI-complete." While much research focuses on specific joke formats like knock-knock jokes or one-liners, this paper addresses a more sophisticated and subtle form of humor: the quirky, often unexpected, scientific achievement that makes people laugh and then think. Identifying such papers is valuable not only for understanding humor but also for potentially highlighting creative, boundary-pushing research that might otherwise be overlooked. The authors are motivated by the Ig Nobel Prize, which celebrates these unusual scientific contributions.
Methods
The authors propose a novel task: automatically detecting funny scientific papers, primarily by analyzing their titles. Their approach is two-fold:
- Literature-Inspired Features: They engineer 127 features based on established theories of humor from psychology and linguistics. These include measures of:
- Unexpected Language: Using various language models (N-gram, POS-tag based, and Transformer-based like BERT and GPT-2) to quantify the "perplexity" or unexpectedness of words in a title.
- Simple Language: Metrics like title length, word length variance, and automated readability indices.
- Crude Language: A classifier trained on toxic comments, combined with perplexity to identify unexpected crude terms.
- Funny Language: Quantifying the inherent "funniness" of nouns in a title.
- NLP Models: They compare several models:
- Iggy: A simple Multi-Layer Perceptron (MLP) classifier trained on the engineered features.
- Pre-trained Transformer Models: BERT and SciBERT (a BERT variant trained on scientific text), both fine-tuned for the task.
- Hybrid Models: BERT/SciBERT models that combine the pre-trained embeddings with the literature-inspired features.
Datasets
- Positive Examples: The dataset includes titles of Ig Nobel winners, supplemented by humorous papers found online (totaling 1,707 papers).
- Negative Examples: 1,707 titles randomly sampled from Semantic Scholar, balanced across scientific fields.
- Large Corpus: A dataset of 630,000 titles from Semantic Scholar was used for training language models and for the "in the wild" evaluation.
Key Insights and Ideas
- Titles as Proxies: The authors found that for many Ig Nobel-worthy papers, the title alone is sufficient to convey the humor, making title analysis a viable and efficient approach.
- Domain Expertise Matters: The engineered features, derived from humor theory, proved to be highly informative, even for complex Transformer models. This suggests that combining traditional knowledge with modern NLP can yield superior results.
- Simplicity and Unexpectedness are Key: Features related to linguistic simplicity and unexpectedness (perplexity) were consistently strong predictors of humor.
- Humor is Subjective, but Detectable: While humor is subjective, the study demonstrates that models can learn to identify patterns associated with it, especially when guided by domain insights.
- Distinguishing Funny Titles from Funny Topics: The authors highlight the challenge of differentiating between a research topic that is inherently funny and a title that is cleverly worded to be humorous.
Key Results
- High Accuracy: All models achieved high accuracy on the labeled dataset, with SciBERT generally outperforming others.
- Feature Importance: The literature-inspired features were crucial, significantly improving the performance of BERT and SciBERT when combined.
- "In the Wild" Performance: In a realistic setting evaluating performance on a large corpus, the "Iggy" model (feature-based MLP) showed strong performance, particularly at the top of the ranked list, outperforming even some Transformer-based models for the first 50 results.
- Overlap Analysis: The overlap between feature-based models was high, suggesting the features captured similar aspects of humor. BERT and SciBERT showed less overlap, indicating they might rely more on their pre-trained knowledge.
- Qualitative Observations: The models sometimes confused funny titles with funny topics, particularly with catchy editorial titles. They also learned that longer titles tend to be more serious.
Limitations and Future Work
- Subjectivity of Humor: The inherent subjectivity of humor makes perfect classification challenging, as evidenced by the low agreement among human annotators on topic funniness.
- Dataset Bias: The dataset might contain biases, and the authors acknowledge that filtering out editorials from real research papers was not straightforward.
- Model Confusion: Models sometimes confused funny titles with funny topics, and the rudeness classifier's role was inconclusive.
- Future Directions:
- Creating aligned corpora of funny/serious titles to pinpoint exact humor-inducing words.
- Investigating whether funny science receives disproportionate attention.
- Expanding analysis to paper abstracts and full texts.
- Exploring multi-task learning for more robust humor classification.
- Promoting the integration of data-driven research with insights from traditional fields like psychology.
Summary updated