TL;DR
This paper introduces the first freely available corpus for humor detection in English-Hindi code-mixed social media content (often called "Hinglish"). By combining manual annotation of 3,543 tweets with a baseline supervised learning framework, the researchers achieved a 69.3% accuracy in distinguishing humorous from non-humorous content, providing a foundation for linguistic analysis in multilingual digital spaces.
What Problem Does This Paper Address?
Humor is one of the most complex linguistic phenomena to automate because it relies heavily on semantic depth, cultural context, and irony. The challenge intensifies on social media, where users frequently engage in code-mixing—the practice of interchanging two or more languages within a single conversation or even a single sentence.
In regions like South Asia, users often transliterate Hindi into the Latin script and mix it with English. This creates significant hurdles for standard Natural Language Processing (NLP) tools due to:
- Spelling variations: Non-standardized phonetic spellings (e.g., "dis" for "this" or "h" for "hai").
- Grammatical inconsistency: Sentences that don't follow the formal rules of either language.
- Ambiguity: Words that exist in both languages but have different meanings (e.g., the word "is" functions as a verb in English but as a determiner meaning "this" in Hindi).
While humor detection has been studied in monolingual contexts (like English puns or Yelp reviews), this paper fills the gap for the massive amount of code-mixed data generated by millions of bilingual users.
Methods
The authors developed a supervised classification pipeline to tackle the problem:
- Pre-processing: Beyond standard tokenization and punctuation removal, the authors employed hashtag decomposition. For example,
#AadabArzHaiwas split into three distinct tokens ("Aadab", "Arz", "Hai") based on camel casing, as hashtags often carry the "punchline" or essential context for humor. - Feature Engineering: They extracted three main types of features:
- N-grams: Specifically character trigrams, which are effective for code-mixed text because they are somewhat language-independent and robust against spelling variations.
- Bag-of-words: Using a three-word window to capture local context.
- Common Words and Hashtags: Identifying specific keywords and idioms frequently associated with jokes.
- Classification: The team tested four models: Support Vector Machines (SVM) with a Radial Basis Function (RBF) kernel, Random Forest, Extra Tree, and Naive Bayes. They used 10-fold cross-validation to ensure the reliability of their results.
Datasets
The researchers created a new dataset by scraping 10,478 tweets across domains like sports, politics, and entertainment. After filtering for purely English or Hindi tweets, they arrived at 3,543 English-Hindi code-mixed tweets.
- Annotation: Three bilingual university students spent approximately 60 hours tagging the tweets.
- Humor Tags: Tweets were classified as Humorous (H) or Non-humorous (N). The final set was nearly balanced: 1,755 (H) and 1,698 (N).
- Language Tags: Crucially, every word was tagged at the token level as English (En), Hindi (Hi), or Other (Ot - for emojis, URLs, etc.). This allowed the model to process features for both languages separately before combining them into a single vector.
Key Insights and Ideas
- The "Ambiguity" Challenge: The authors highlight that words like "to" (English preposition vs. Hindi for "but") require context-aware annotation. This underscores why simple dictionary-based language identification fails in code-mixed settings.
- Feature Performance: N-grams proved to be the most powerful feature across all models. This suggests that in noisy, code-mixed environments, sub-word information (character patterns) is more reliable than word-level semantics, which are often obscured by creative spellings.
- Context is King: The authors noted that some humor is "observational" and tied to specific events (like the 2015 ICC Cricket World Cup). Without external knowledge or hashtag analysis, these tweets might appear non-humorous to a machine.
Key Results
The Kernel SVM emerged as the top performer, particularly when utilizing all available features. The researchers attributed this success to the fact that the number of observations in their dataset was greater than the number of features, a scenario where RBF kernels typically excel.
As shown in the table above, the combination of all features with a Kernel SVM yielded the highest accuracy of 69.3%. Interestingly, Naive Bayes performed remarkably well with the Bag-of-words feature compared to the more complex tree-based models.
Limitations and Future Work
While a significant first step, the authors acknowledge several areas for improvement:
- Dataset Size: 3,500 tweets is a relatively small corpus for deep learning approaches.
- Unimodal Focus: The study removed tweets containing images to keep the corpus "unimodal." However, social media humor is often multimodal (memes), where the text only becomes funny when paired with a specific image.
- Linguistic Depth: Future iterations could benefit from word-level Part-of-Speech (POS) tagging to better handle language identification and semantic nuances.
- Expansion: The authors suggest extending this work to other code-mixed pairs, such as Arabic-English or Greek-English, where transliteration issues are similarly prevalent.
Summary updated