PANDAS@TamilNLP-ACL2022: Emotion Analysis in Tamil Text using Language Agnostic Embeddings

Divyasri K, Gayathri G L, Krithika Swaminathan, Thenmozhi Durairaj, Bharathi B, Senthil Kumar BACL 2022

TL;DR

This paper tackles the challenge of emotion analysis in Tamil text, a low-resource language for NLP tasks. They propose a language-agnostic approach using the LaBSE model for feature extraction, combined with an SVM classifier, to identify 11 distinct emotions in user-generated content. While achieving a modest F1-score of 0.010, the work highlights the potential of multilingual embeddings for under-resourced languages.

What Problem Does This Paper Address?

The digital age has led to an explosion of user-generated content on social media, making emotion analysis a crucial tool for various applications, from public health monitoring to business intelligence. However, for many languages, particularly those with fewer digital resources like Tamil, developing effective emotion analysis models is a significant challenge. Existing models often struggle with the nuances of these languages, and the availability of labeled datasets is limited. This paper aims to bridge this gap by developing a language-agnostic model for emotion analysis in Tamil, making it more accessible and applicable.

Methods

The core of the proposed methodology lies in leveraging a pre-trained, language-agnostic model for feature extraction. Specifically, the authors employ LaBSE (Language-Agnostic BERT Sentence Embedding). LaBSE is a multilingual BERT model that encodes sentences into a shared embedding space, allowing for similarity comparisons across languages. This approach bypasses the need for language-specific training data for the embedding layer.

The extracted features from LaBSE are then fed into a Support Vector Machine (SVM) classifier. SVM is a well-established supervised learning algorithm for classification tasks, known for its effectiveness in mapping data to high-dimensional spaces for better categorization.

A significant challenge identified was the imbalanced nature of the dataset, with a disproportionate number of "Neutral" emotion examples. To address this, the authors applied the Synthetic Minority Oversampling Technique (SMOTE). SMOTE is an oversampling method that generates synthetic data points for minority classes, helping to balance the dataset and prevent the model from being biased towards the majority class.

The preprocessing pipeline involved several steps:

  1. Checking for inconsistencies: Removing empty rows or mismatched values.
  2. Removal of punctuation and special characters: Including emoticons, to focus on textual content.
  3. Transformation of data: Converting all text to lowercase for uniformity.
  4. Reduction of data: Removing common Tamil stop words.
  5. Balancing of the dataset: Using SMOTE.

Datasets

The dataset used in this research consists of user-generated comments from YouTube in the Tamil language. These comments are categorized into 11 distinct emotion classes: Neutral, Joy, Ambiguous, Trust, Disgust, Anger, Anticipation, Sadness, Love, Surprise, and Fear. The paper provides data distribution for both the training and development datasets, illustrating the imbalance, particularly the dominance of the "Neutral" category.

Key Insights and Ideas

  • Multilingual Embeddings for Low-Resource Languages: The paper demonstrates the viability of using language-agnostic models like LaBSE for tasks in languages with limited NLP resources. This opens doors for applying advanced NLP techniques to a wider range of languages.
  • The Power of Preprocessing: The meticulous preprocessing steps, especially the removal of noise and stop words, are crucial for preparing the data for effective feature extraction and classification.
  • Addressing Data Imbalance is Critical: The significant improvement observed when using SMOTE highlights the importance of handling imbalanced datasets in emotion analysis, preventing models from being skewed towards the most frequent emotion.
  • LaBSE + SVM as a Promising Combination: The combination of LaBSE for robust feature representation and SVM for classification proved to be a strong approach, outperforming other experimented models.

Key Results

The paper reports the following key results:

  • The proposed model, using LaBSE feature extraction with an SVM classifier, achieved an F1-score of 0.010 on the test dataset.
  • The Precision score was 0.080, and the Recall score was 0.070.
  • The authors noted that the accuracy was slightly increased when the data was further preprocessed using the SMOTE technique. Table 2 shows a comparison:
  • LaBSE + SVM (without SMOTE): F1 = 0.23, Accuracy = 0.43
    • LaBSE + SVM (with SMOTE): F1 = 0.27, Accuracy = 0.44
  • The submission secured the 8th rank in Task B (Emotion Analysis on a Tamil dataset) of the DravidianTech-Lang ACL 2022 shared task.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Modest Performance: The reported F1-score of 0.010 is relatively low, indicating room for significant improvement.
  • Custom Embeddings: The authors believe that results can be highly improved by using custom embeddings that are based on a deeper statistical analysis of the Tamil language. This suggests that while generic multilingual embeddings are useful, language-specific fine-tuning or custom-built embeddings could yield better performance.
  • Further Exploration of Models: While LaBSE and SVM showed promise, exploring other transformer models or advanced classification techniques could lead to better outcomes.
  • Dataset Size and Quality: The performance is inherently tied to the size and quality of the dataset. Expanding the dataset or improving its annotation could be beneficial.

Summary updated

BibTeX
@inproceedings{k2022pandas,
  author = {Divyasri K and Gayathri G L and Krithika Swaminathan and Thenmozhi Durairaj and Bharathi B and Senthil Kumar B},
  title = {{PANDAS@TamilNLP-ACL2022: Emotion Analysis in Tamil Text using Language Agnostic Embeddings}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure