GJG@TamilNLP-ACL2022: Emotion Analysis and Classification in Tamil using Transformers

Janvi Prasad, Gaurang Prasad, Gunavathi CACL 2022

TL;DR

This paper tackles the challenge of emotion analysis in Tamil, a low-resource language, by fine-tuning powerful transformer models like XLM-ROBERTa and DeBERTa. They achieved top rankings in a shared task, demonstrating the effectiveness of these models for nuanced emotion classification in Tamil social media text.

What Problem Does This Paper Address?

Emotion analysis, a crucial task in Natural Language Processing (NLP), helps us understand user-generated content, which is vital for applications like public opinion analysis, healthcare, and building better AI assistants. While significant research exists for high-resource languages like English, low-resource languages, particularly Dravidian languages like Tamil, have been largely under-explored. This lack of research is often due to a scarcity of annotated datasets. This paper addresses this gap by participating in a shared task focused on emotion analysis in Tamil, aiming to leverage state-of-the-art transformer models for this under-resourced language.

Methods

The core methodology involves fine-tuning pre-trained transformer models for text classification. Specifically, the authors experimented with two powerful multilingual models: XLM-ROBERTa and DeBERTa. These models were fine-tuned on the provided Tamil datasets for two distinct sub-tasks: a coarse-grained classification with 11 emotion categories and a fine-grained classification with 31 emotion categories. The researchers chose not to pre-process the text by removing special characters, emojis, or extended punctuation, believing these elements provide valuable context for emotion analysis.

Datasets

The study utilized datasets provided by the organizers of the "Emotion Analysis in Tamil" shared task at the Second Workshop on Speech and Language Technologies for Dravidian Languages (ACL 2022).

  • Sub-task A (Coarse-grained): Contained 11 emotion classes. The training dataset had 14,208 rows, the development dataset had 3,552 rows, and the test dataset had 4,440 rows. The labels were in English.
  • Sub-task B (Fine-grained): Contained 31 emotion classes. The training dataset was larger with 30,179 rows, the development dataset had 4,269 rows, and the test dataset had 4,269 rows. The labels for this sub-task were in Tamil.

The datasets consisted of Tamil sentences extracted from social media comments, with annotations made at the comment/post level.

Key Insights and Ideas

  • Transformer models excel even in low-resource settings: The study demonstrates that powerful, pre-trained transformer models like XLM-ROBERTa can achieve competitive results in emotion analysis for a low-resource language like Tamil, even without extensive language-specific pre-training.
  • Preserving context is crucial: The decision to retain special characters and emojis in the text proved beneficial, as these elements often carry significant emotional cues in social media communication.
  • Fine-grained classification is significantly harder: The performance metrics for sub-task B (31 classes) were considerably lower than for sub-task A (11 classes), highlighting the increased complexity of distinguishing between more nuanced emotions.
  • XLM-ROBERTa shows a slight edge: Across both sub-tasks, XLM-ROBERTa consistently outperformed DeBERTa, suggesting its suitability for this specific task and dataset.

Key Results

The paper reports the following performance metrics:

  • Sub-task A (11 classes):
    • XLM-ROBERTa achieved an accuracy of 0.46 and a weighted F1-score of 0.44.
  • DeBERTa achieved an accuracy of 0.45 and a weighted F1-score of 0.38.
  • The team using XLM-ROBERTa secured the best classification performance out of 11 teams for this sub-task.
  • Sub-task B (31 classes):
    • XLM-ROBERTa achieved an accuracy of 0.33 and a weighted F1-score of 0.26.
  • DeBERTa achieved an accuracy of 0.26 and a weighted F1-score of 0.2.
  • The team ranked 2nd out of 7 teams for this sub-task.

Limitations and Future Work

  • Limitations: The authors acknowledge that the overall classification performance for the fine-grained sub-task (Sub-task B) was lower, indicating the inherent difficulty of distinguishing between a larger number of emotions. While they achieved strong results, the absolute accuracy scores are still modest, suggesting room for improvement.
  • Future Work: The authors express their hope to build upon the learnings from this shared task to architect and develop models specifically for downstream Tamil NLP tasks. They have also open-sourced their code to encourage further research in this area.

Summary updated

BibTeX
@inproceedings{prasad2022gjg,
  author = {Janvi Prasad and Gaurang Prasad and Gunavathi C},
  title = {{GJG@TamilNLP-ACL2022: Emotion Analysis and Classification in Tamil using Transformers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure