Humor Detection in English-Hindi Code-Mixed Social Media Content : Corpus and Baseline System

Ankush Khandelwal, Sahil Swami, Syed Sarfaraz Akhtar, Manish Shrivastava

TL;DR

This paper introduces the first freely available corpus for humor detection in English-Hindi code-mixed social media content (often called "Hinglish"). By combining manual annotation of 3,543 tweets with a baseline supervised learning framework, the researchers achieved a 69.3% accuracy in distinguishing humorous from non-humorous content, providing a foundation for linguistic analysis in multilingual digital spaces.

What Problem Does This Paper Address?

Humor is one of the most complex linguistic phenomena to automate because it relies heavily on semantic depth, cultural context, and irony. The challenge intensifies on social media, where users frequently engage in code-mixing—the practice of interchanging two or more languages within a single conversation or even a single sentence.

In regions like South Asia, users often transliterate Hindi into the Latin script and mix it with English. This creates significant hurdles for standard Natural Language Processing (NLP) tools due to:

  1. Spelling variations: Non-standardized phonetic spellings (e.g., "dis" for "this" or "h" for "hai").
  2. Grammatical inconsistency: Sentences that don't follow the formal rules of either language.
  3. Ambiguity: Words that exist in both languages but have different meanings (e.g., the word "is" functions as a verb in English but as a determiner meaning "this" in Hindi).

While humor detection has been studied in monolingual contexts (like English puns or Yelp reviews), this paper fills the gap for the massive amount of code-mixed data generated by millions of bilingual users.

Methods

The authors developed a supervised classification pipeline to tackle the problem:

  • Pre-processing: Beyond standard tokenization and punctuation removal, the authors employed hashtag decomposition. For example, #AadabArzHai was split into three distinct tokens ("Aadab", "Arz", "Hai") based on camel casing, as hashtags often carry the "punchline" or essential context for humor.
  • Feature Engineering: They extracted three main types of features:
  • N-grams: Specifically character trigrams, which are effective for code-mixed text because they are somewhat language-independent and robust against spelling variations.
  • Bag-of-words: Using a three-word window to capture local context.
  • Common Words and Hashtags: Identifying specific keywords and idioms frequently associated with jokes.
  • Classification: The team tested four models: Support Vector Machines (SVM) with a Radial Basis Function (RBF) kernel, Random Forest, Extra Tree, and Naive Bayes. They used 10-fold cross-validation to ensure the reliability of their results.

Datasets

The researchers created a new dataset by scraping 10,478 tweets across domains like sports, politics, and entertainment. After filtering for purely English or Hindi tweets, they arrived at 3,543 English-Hindi code-mixed tweets.

  • Annotation: Three bilingual university students spent approximately 60 hours tagging the tweets.
  • Humor Tags: Tweets were classified as Humorous (H) or Non-humorous (N). The final set was nearly balanced: 1,755 (H) and 1,698 (N).
  • Language Tags: Crucially, every word was tagged at the token level as English (En), Hindi (Hi), or Other (Ot - for emojis, URLs, etc.). This allowed the model to process features for both languages separately before combining them into a single vector.

Key Insights and Ideas

  • The "Ambiguity" Challenge: The authors highlight that words like "to" (English preposition vs. Hindi for "but") require context-aware annotation. This underscores why simple dictionary-based language identification fails in code-mixed settings.
  • Feature Performance: N-grams proved to be the most powerful feature across all models. This suggests that in noisy, code-mixed environments, sub-word information (character patterns) is more reliable than word-level semantics, which are often obscured by creative spellings.
  • Context is King: The authors noted that some humor is "observational" and tied to specific events (like the 2015 ICC Cricket World Cup). Without external knowledge or hashtag analysis, these tweets might appear non-humorous to a machine.

Key Results

The Kernel SVM emerged as the top performer, particularly when utilizing all available features. The researchers attributed this success to the fact that the number of observations in their dataset was greater than the number of features, a scenario where RBF kernels typically excel.

Accuracy of each feature using different classifiers

As shown in the table above, the combination of all features with a Kernel SVM yielded the highest accuracy of 69.3%. Interestingly, Naive Bayes performed remarkably well with the Bag-of-words feature compared to the more complex tree-based models.

Limitations and Future Work

While a significant first step, the authors acknowledge several areas for improvement:

  • Dataset Size: 3,500 tweets is a relatively small corpus for deep learning approaches.
  • Unimodal Focus: The study removed tweets containing images to keep the corpus "unimodal." However, social media humor is often multimodal (memes), where the text only becomes funny when paired with a specific image.
  • Linguistic Depth: Future iterations could benefit from word-level Part-of-Speech (POS) tagging to better handle language identification and semantic nuances.
  • Expansion: The authors suggest extending this work to other code-mixed pairs, such as Arabic-English or Greek-English, where transliteration issues are similarly prevalent.

Summary updated

Abstract

The tremendous amount of user generated data through social networking sites led to the gaining popularity of automatic text classification in the field of computational linguistics over the past decade. Within this domain, one problem that has drawn the attention of many researchers is automatic humor detection in texts. In depth semantic understanding of the text is required to detect humor which makes the problem difficult to automate. With increase in the number of social media users, many multilingual speakers often interchange between languages while posting on social media which is called code-mixing. It introduces some challenges in the field of linguistic analysis of social media content (Barman et al., 2014), like spelling variations and non-grammatical structures in a sentence. Past researches include detecting puns in texts (Kao et al., 2016) and humor in one-lines (Mihalcea et al., 2010) in a single language, but with the tremendous amount of code-mixed data available online, there is a need to develop techniques which detects humor in code-mixed tweets. In this paper, we analyze the task of humor detection in texts and describe a freely available corpus containing English-Hindi code-mixed tweets annotated with humorous(H) or non-humorous(N) tags. We also tagged the words in the tweets with Language tags (English/Hindi/Others). Moreover, we describe the experiments carried out on the corpus and provide a baseline classification system which distinguishes between humorous and non-humorous texts.
BibTeX
@misc{khandelwal2018humor,
  author = {Ankush Khandelwal and Sahil Swami and Syed Sarfaraz Akhtar and Manish Shrivastava},
  title = {{Humor Detection in English-Hindi Code-Mixed Social Media Content : Corpus and Baseline System}},
  year = {2018},
  eprint = {1806.05513},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1806.05513},
}
Figure