Optimize_Prime@DravidianLangTech-ACL2022: Emotion Analysis in Tamil

Omkar Gokhale, Shantanu Patankar, Onkar Litake, Aditya Mandke, Dipali KadamACL 2022

TL;DR

This paper tackles the challenge of emotion analysis in Tamil social media comments, a crucial task for understanding public sentiment in a widely spoken language. The authors explored various transformer and RNN-based models, finding that XLM-ROBERTa and MuRIL achieved the best performance on different subtasks, highlighting the potential of advanced language models for low-resource languages.

What Problem Does This Paper Address?

In today's digital age, social media platforms have become primary channels for individuals to express their opinions and emotions. Emotion analysis, or opinion mining, aims to automatically identify these sentiments from text. While significant progress has been made for high-resource languages like English, analyzing emotions in languages with fewer digital resources, such as Tamil, remains a considerable challenge. This paper addresses this gap by developing and evaluating models for emotion analysis specifically for Tamil social media comments, contributing to the understanding of sentiment in this under-resourced linguistic context.

Methods

The researchers employed a multi-pronged approach, experimenting with three main categories of models:

  • Transformer-based Models: They utilized state-of-the-art transformer architectures, including XLM-ROBERTa, MuRIL, M-BERT, and Indic BERT. These models are pre-trained on massive text corpora and are known for their strong performance in various NLP tasks.
  • Recurrent Neural Networks (RNNs): Long Short-Term Memory (LSTM) networks and ULM-Fit were explored as baseline RNN approaches. ULM-Fit, in particular, leverages a universal language model fine-tuning strategy.
  • Ensemble Models: To potentially boost performance, they combined transformer embeddings with traditional machine learning classifiers like Logistic Regression, Support Vector Machines (SVM), and Multi-Layer Perceptrons (MLP).

Data preprocessing involved cleaning text by removing punctuation, URLs, and stop words, and replacing emojis with textual equivalents. They also addressed class imbalance using techniques like over-sampling, under-sampling, SMOTE, and assigning class weights.

Datasets

The study used datasets from the "Emotion Analysis in Tamil-ACL 2022" shared task. These datasets consist of Tamil social media comments labeled with emotions.

  • Task A: This task involved classifying comments into 11 broad emotion categories (e.g., Neutral, Joy, Anger, Sadness). The dataset comprised 14,208 training, 3,552 development, and 4,440 test data points.
  • Task B: This task focused on classifying comments into 31 more specific emotion categories. The dataset contained 30,180 training, 4,269 development, and 4,269 test data points.

A significant class imbalance was noted in both datasets, reflecting the natural distribution of emotions in real-world social media data.

Key Insights and Ideas

  • Transformer Dominance: Advanced transformer models, particularly XLM-ROBERTa and MuRIL, consistently outperformed traditional RNNs and simpler ensemble models, underscoring their effectiveness for complex language understanding tasks.
  • Task-Specific Best Models: Interestingly, XLM-ROBERTa achieved the best results for the broader 11-class emotion classification (Task A), while MuRIL excelled in the more granular 31-class classification (Task B). This suggests that model suitability can depend on the complexity and granularity of the classification task.
  • Data Cleaning Nuances: While data cleaning generally improved RNN and some transformer models, it surprisingly degraded MuRIL's performance. This highlights that optimal preprocessing steps can be model-dependent and require careful tuning.
  • Class Imbalance Mitigation: Techniques like over-under sampling and assigning class weights were crucial for improving performance, especially for transformer and ensemble models, demonstrating their importance in handling imbalanced datasets.
  • Complex Emotions are Harder: The models showed better accuracy in identifying simpler emotions like Joy, Sadness, and Neutral, as well as ambiguity. However, classifying more nuanced emotions like anger and fear proved more challenging.

Key Results

  • Task A (11 classes): XLM-ROBERTa achieved the highest macro-averaged F1 score of 0.32, with MuRIL performing comparably.
  • Task B (31 classes): MuRIL achieved the highest macro-averaged F1 score of 0.125.
  • For ensemble models, Logistic Regression performed best on Task A (macro-avg F1: 0.23) and Task B (macro-avg F1: 0.1).
  • For RNNs, ULM-Fit performed best on Task A (macro-avg F1: 0.27), while LSTM performed best on Task B (macro-avg F1: 0.11).

Limitations and Future Work

The authors acknowledge that while their models perform well on certain emotions, they struggle with more complex ones like anger, fear, and sadness.

Future work suggested includes:

  • Exploring more sophisticated ensembling techniques, such as genetic algorithm-based ensembling, to further improve performance.
  • Investigating other advanced transformer architectures and fine-tuning strategies.
  • Expanding the research to other low-resource Indic languages to build more comprehensive multilingual emotion analysis capabilities.

Summary updated

BibTeX
@inproceedings{gokhale2022optimize,
  author = {Omkar Gokhale and Shantanu Patankar and Onkar Litake and Aditya Mandke and Dipali Kadam},
  title = {{Optimize\_Prime@DravidianLangTech-ACL2022: Emotion Analysis in Tamil}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure