UMUTeam@TamilNLP-ACL2022: Emotional Analysis in Tamil

José García-Díaz, Miguel Ángel Rodríguez García, Rafael Valencia-GarcíaACL 2022

TL;DR

This paper details UMUTeam's participation in the TamilNLP-ACL 2022 shared task on emotion analysis in Tamil. They achieved first place in the 30-emotion classification subtask by combining linguistic features with various sentence embeddings using a knowledge integration strategy.

What Problem Does This Paper Address?

Emotion analysis, a subfield of sentiment analysis, aims to identify specific emotions expressed in text. While research in this area is growing, it's particularly challenging for low-resource languages like Tamil, which lack extensive annotated datasets and pre-trained models compared to languages like English. The TamilNLP-ACL 2022 shared task specifically targeted emotion analysis in Tamil, presenting two subtasks: one with 11 emotions and another with a more granular 30 emotions. The challenge lies in developing robust models that can accurately capture nuanced emotional expressions in a language with unique linguistic characteristics.

Methods

The UMUTeam's approach is grounded in a neural network classifier that integrates four distinct feature sets. These include:

  • Linguistic Features (LF): A curated subset of language-independent features encompassing stylometric metrics (word/sentence length), Part-of-Speech (PoS) tags (using the Stanza toolkit for Tamil), and social media specific features like emoji detection. This set also includes features capturing jargon and hyperlinks.
  • Sentence Embeddings (SE): Non-contextual sentence embeddings derived from a pre-trained fastText model for Tamil.
  • BERT Embeddings (BF): Sentence embeddings generated from a multilingual BERT model, fine-tuned for the specific task.
  • RoBERTa Embeddings (RF): Sentence embeddings generated from a multilingual RoBERTa model, also fine-tuned.

The core of their strategy involves a "knowledge integration" approach where each feature set is fed into a separate hidden layer of a neural network, with their outputs then combined in subsequent hidden layers. They also explored ensemble learning methods (mode and averaging of probabilities) as alternative combination strategies. Hyperparameter optimization was performed using RayTune with a Tree of Parzen Estimators (TPE) strategy.

Datasets

The paper utilizes datasets provided by the TamilNLP-ACL 2022 shared task, which included training, development, and test splits for both subtasks.

  • Subtask 1: Distinguishes among 11 emotions.
  • Subtask 2: Distinguishes among 30 emotions.

The paper notes that the distribution of emotions in the validation and test sets appeared similar.

Key Insights and Ideas

  • The Power of Integration: Combining diverse feature sets (linguistic, contextual, and non-contextual embeddings) proved more effective than using any single feature set in isolation, especially for the more complex 30-emotion subtask.
  • Language-Independent Features' Resilience: The language-independent linguistic features, while a smaller subset, still contributed to the overall performance, suggesting their potential utility even in low-resource languages.
  • Fine-tuning is Crucial: Fine-tuning pre-trained models like BERT and RoBERTa on the specific Tamil emotion analysis task was essential for extracting effective sentence embeddings.
  • Subtle Differences in Embeddings: For the first subtask, RoBERTa embeddings (RF) alone performed very well, outperforming BERT and fastText embeddings. However, when integrated, the knowledge integration strategy's performance was slightly lower than RF alone, suggesting potential challenges in effectively combining strong individual features.
  • Confusion with Neutral: In the 11-emotion subtask, a significant portion of misclassifications involved the "neutral" class, with neutral documents being incorrectly labeled as joy or disgust. Surprise was also underrepresented and often misclassified.

Key Results

  • Subtask 2 (30 Emotions): UMUTeam achieved 1st place with a macro F1-score of 15.1% on the official leaderboard. This was a notable improvement over the second-best result.
  • Subtask 1 (11 Emotions): While their official submission for this subtask was not recorded due to an apparent formatting error, their results on the validation split showed a macro F1-score of 32.36% using the knowledge integration strategy. This was slightly lower than the performance of the RoBERTa embeddings (RF) used in isolation (33.74%).
  • Individual Emotion Performance: In the 11-emotion subtask, "joy" and "ambiguous" had the highest F1-scores (57.63% and 57.38%), while "surprise" had the lowest (8.16%). For the 30-emotion subtask, "thanksgiving" and "admiration" showed the best performance.

Limitations and Future Work

  • Unreported Subtask 1 Results: The primary limitation acknowledged is the inability to report official results for the 11-emotion subtask due to a submission error.
  • Knowledge Integration vs. Isolation: For the 11-emotion subtask, the knowledge integration strategy did not outperform the best individual feature set (RF), suggesting room for improvement in how features are combined.
  • Future Work: The authors plan to:
  • Extend the architecture by incorporating new feature extraction techniques.
  • Focus on interpretability techniques to understand model decisions.
  • Evaluate the correlation of linguistic features with hate-speech detection in Spanish, leveraging existing datasets.
  • Explore the impact of multilingual RoBERTa on the task, noting its strong performance when used alone.

Summary updated

BibTeX
@inproceedings{garciadiaz2022umuteam,
  author = {José García-Díaz and Miguel Ángel Rodríguez García and Rafael Valencia-García},
  title = {{UMUTeam@TamilNLP-ACL2022: Emotional Analysis in Tamil}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure