TL;DR
This paper introduces a large, crowd-annotated Spanish Twitter corpus for humor analysis, featuring over 27,000 tweets with humor/non-humor labels and funniness scores. This dataset aims to address the scarcity of Spanish humor resources and provide a foundation for building better humor detection models and exploring the subjective nature of humor.
What Problem Does This Paper Address?
Computational humor research, encompassing tasks like humor recognition, generation, and scoring, heavily relies on human-annotated data. While existing English humor datasets exist, there's a significant lack of comparable resources for Spanish. Previous Spanish humor datasets have suffered from limited annotations per instance, low annotator agreement, and a narrow range of tweet sources. This paper aims to fill this gap by creating a more robust and diverse Spanish humor corpus.
Methods
The authors employed a crowd-sourcing approach to annotate tweets. They developed a web-based annotation tool that presented tweets to users, asking them to first classify if a tweet intended to be humorous ("Yes," "No," or "Skip"). If marked as humorous, annotators then rated its funniness on a 1-5 scale using emojis. The interface was designed to be intuitive and minimize bias. To ensure data quality, they implemented "test tweets" at the beginning of each session and filtered out sessions with consistently low quality. They also adjusted their annotation strategy mid-process to address an imbalance in non-humorous votes.
Datasets
The paper introduces a new dataset consisting of 27,282 Spanish tweets. This dataset is divided into two CSV files: one for tweets (containing identifier and origin) and another for annotations (tweet ID, session ID, date, and annotation value). The tweets were sourced from both selected humorous and non-humorous Twitter accounts and from random real-time samples. The dataset contains approximately 107,634 annotations in total, with an average of 3.8 annotations per tweet.
Key Insights and Ideas
- The Subjectivity of Humor: The paper highlights that while annotators can generally agree on whether a tweet is intended to be humorous (moderate to substantial agreement), they struggle significantly to agree on the degree of funniness. This underscores the inherently subjective nature of humor.
- Crowdsourcing for Scale: The use of crowdsourcing allowed the researchers to gather a substantial number of annotations (over 100,000) relatively efficiently, which is crucial for building robust machine learning models.
- Interface Design Matters: The authors suggest that their carefully designed, intuitive user interface, which used emojis for funniness ratings, contributed to better annotator agreement compared to previous work.
- Bias in Humor Datasets: The dataset reveals a bias towards "bad jokes" in humorous tweets, as indicated by the low average funniness score.
Key Results
- The dataset contains 27,282 Spanish tweets with 107,634 annotations.
- The inter-annotator agreement for humor/non-humor classification (using Krippendorff's alpha) was 0.5710, considered "moderate" to "substantial." This is significantly higher than previous Spanish humor datasets.
- The funniness score inter-annotator agreement was much lower (0.1625), indicating high subjectivity.
- Approximately 27.01% of the tweets were classified as humorous, with a low average funniness score of 1.35.
- The dataset is balanced between humorous and non-humorous accounts.
Limitations and Future Work
The authors acknowledge that while they aimed for five annotations per tweet, the average is 3.8. They suggest that more annotations per tweet would be beneficial for more robust subjectivity analysis, potentially enabling techniques like label distribution learning. Future work includes gathering more annotations, exploring how social strata might influence humor perception, and building similar datasets for other languages to facilitate cross-lingual humor analysis and leverage advanced Deep Learning techniques.
Summary updated