Deep Learning Techniques for Humor Detection in Hindi-English Code-Mixed Tweets

Sushmitha Reddy Sane, Suraj Tripathi, Koushik Reddy Sane, Radhika MamidiNAACL 2019

TL;DR

This paper tackles the challenge of detecting humor in code-mixed Hindi-English tweets, a common linguistic phenomenon in multilingual communities. The authors propose and evaluate three deep learning models, leveraging novel bilingual word embeddings, and achieve state-of-the-art results, with an attention-based BiLSTM model reaching 73.6% accuracy.

What Problem Does This Paper Address?

The internet has become a primary platform for communication, leading to a surge in user-generated content. This content often exhibits various linguistic nuances, including humor, irony, and sarcasm. Automatically detecting these elements is a growing area of research. However, a significant challenge arises in multilingual contexts, particularly in regions like India, where code-mixing (blending multiple languages within a single utterance) is prevalent. Existing natural language processing (NLP) research and tools are heavily biased towards monolingual English data, leaving code-mixed text underserved. This paper specifically addresses the problem of humor detection in Hindi-English code-mixed tweets, a domain that suffers from a lack of readily available bilingual word embeddings and annotated datasets.

Methods

The core of this paper lies in developing and applying deep learning techniques to code-mixed humor detection. The authors' approach involves:

  • Bilingual Word Embeddings: Recognizing the limitations of standard word embeddings for code-mixed data, they propose using bilingual word embeddings generated from code-mixed tweets. They experiment with two popular word embedding models:
  • Word2Vec: Utilizing CBOW and Skip-gram architectures.
  • FastText: Leveraging its ability to handle out-of-vocabulary words and subword information, which is crucial for informal language and novel word formations common in social media.
  • Deep Learning Architectures: Three distinct deep learning models are proposed and evaluated, all taking the generated bilingual word embeddings as input:
  • Convolutional Neural Network (CNN): A model employing parallel 1D convolution layers to extract features, followed by max-pooling and fully connected layers for classification.
  • Bidirectional Long Short-Term Memory (BiLSTM): A standard BiLSTM network designed to capture compositional semantics and long-range temporal dependencies in the code-mixed text.
  • Attention-based BiLSTM: An enhancement of the BiLSTM model that incorporates an attention mechanism. This mechanism allows the model to dynamically weigh the importance of different words in a sentence, helping to filter out noise and focus on critical elements for humor detection.

All models are trained using a cross-entropy loss function and the Adam optimizer. Techniques like dropout and batch normalization are employed to improve training stability and performance.

Datasets

  • Training Data: A corpus of approximately 200,000 Hindi-English tweets was collected using Twitter's API. This corpus was filtered by extracting tweets containing the 1000 most common words, after removing stop words. Preprocessing involved removing Twitter handles and special symbols, and replacing URLs with the token "URL".
  • Evaluation Data: The benchmark dataset used for evaluation was previously published online by Khandelwal et al. (2018). This dataset comprises 3543 annotated tweets, with 1755 labeled as humorous and 1698 as non-humorous. The authors utilized 5-fold cross-validation for experimental results.

Key Insights and Ideas

  • Bilingual Embeddings are Crucial: The paper strongly demonstrates that using bilingual word embeddings, trained on code-mixed data, is essential for effectively processing and understanding such text. Standard monolingual embeddings are insufficient.
  • FastText Outperforms Word2Vec for Similarity: The study found that FastText embeddings generally showed better similarity scores between Hindi and English words of similar meaning, suggesting that subword information is beneficial for capturing nuances in code-mixed language.
  • Attention Enhances BiLSTM: The introduction of an attention mechanism to the BiLSTM architecture significantly improved performance. This highlights the importance of selectively focusing on relevant parts of the text for accurate humor detection.
  • Deep Learning Surpasses Baselines: The proposed deep learning models consistently outperformed traditional machine learning baselines (SVM, Random Forest, Naive Bayes) on the code-mixed humor detection task.

Key Results

The proposed deep learning models achieved superior performance compared to existing state-of-the-art results. Specifically:

  • The Attention-based BiLSTM model (Model 3(b) with FastText embeddings) achieved the highest accuracy of 73.6%.
  • This represents an improvement of over 4% compared to the current state-of-the-art.
  • The other proposed models also showed strong performance:
  • CNN with FastText (Model 1(b)): 71.3%
    • BiLSTM with FastText (Model 2(b)): 72.2%
  • The baseline systems (Kernel SVM, Random Forest, etc.) from Khandelwal et al. (2018) achieved a best accuracy of 69.3%.

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that a significant challenge in this task is the lack of large, clean, and annotated code-mixed datasets. They suggest that larger corpora are needed from geographies where multilingualism is common.
  • Linguistic Complexity: The inherent linguistic complexity of code-mixed data, including phrasal repetitions, informal constructions, and spelling errors, poses ongoing challenges.
  • Future Directions: The authors propose several promising avenues for future research:
  • Generating aligned multilingual word embeddings.
  • Comparing these embeddings with those aligned using MUSE (Multilingual Unsupervised and Supervised Embeddings).
  • Investigating pre-aligned FastText embeddings.

Summary updated

BibTeX
@inproceedings{sane2019deep,
  author = {Sushmitha Reddy Sane and Suraj Tripathi and Koushik Reddy Sane and Radhika Mamidi},
  title = {{Deep Learning Techniques for Humor Detection in Hindi-English Code-Mixed Tweets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure