Overview of the Shared Task on Sentiment Analysis in Tamil and Tulu

Durairaj Thenmozhi, Bharathi Raja Chakravarthi, Asha Hegde, Hosahalli Lakshmaiah Shashirekha, Rajeswari Natarajan, Sajeetha Thavareesan, Ratnasingam Sakuntharaj, Krishnakumari Kalyanasundaram, Charmathi Rajkumar, Poorvi Shetty, Harshitha S KumarNAACL 2025

TL;DR

This paper introduces a shared task focused on sentiment analysis for Tamil and Tulu, two low-resource Dravidian languages, specifically addressing the challenges of code-mixing. The task involved classifying sentiment in user-generated social media content, leading to the development of new datasets and a comparative analysis of various machine learning and deep learning approaches.

What Problem Does This Paper Address?

Sentiment analysis, the process of understanding opinions and emotions in text, is crucial for many applications. However, analyzing sentiment in languages with limited digital resources, especially when users mix multiple languages within a single text (code-mixing), presents a significant challenge. This paper tackles this by organizing a shared task to encourage research into sentiment analysis for Tamil and Tulu, two languages with rich cultural heritage but limited NLP resources. The prevalence of code-mixing, where English words and scripts are interspersed with Tamil and Tulu, makes traditional sentiment analysis methods struggle to accurately capture the intended meaning and emotional tone.

Methods

The shared task invited participants to develop models for sentiment analysis in Tamil-English and Tulu-English code-mixed text. The approaches employed by the participating teams were diverse, showcasing a range of techniques:

  • Traditional Machine Learning: Teams utilized algorithms like Support Vector Machines (SVM), Naive Bayes, Logistic Regression, and Random Forests, often combined with TF-IDF vectorization for feature extraction.
  • Deep Learning: Recurrent Neural Networks (RNNs) such as LSTMs and BiLSTMs were used, leveraging word-level embeddings (Word2Vec) and character-level features.
  • Transformer Models: A significant portion of the participants fine-tuned pre-trained multilingual transformer models like XLM-RoBERTa, mBERT, and distilRoBERTa. These models, often pre-trained on large multilingual datasets (e.g., AI4Bharat), demonstrated strong performance.
  • Hybrid Approaches: Some teams combined elements of traditional and deep learning methods, for instance, using attention mechanisms with LSTMs or integrating TF-IDF with BERT embeddings.

The common preprocessing steps included stop word removal, hashtag and URL removal, and lemmatization. Techniques like random upsampling and SMOTE were employed to address class imbalance in the datasets.

Datasets

The paper describes the creation of two new datasets for this shared task:

  • Tamil-English Code-Mixed Dataset: Collected from YouTube comments related to Tamil films, encompassing discussions, reviews, and audience opinions.
  • Tulu-English Code-Mixed Dataset: Similarly sourced from YouTube comments about Tulu films, covering discussions, reviews, and audience opinions.

These datasets were designed to be multi-class classification challenges, with sentiments categorized as positive, negative, neutral, or mixed emotions. The data was aggregated from social media platforms to ensure linguistic diversity and real-world applicability.

Key Insights and Ideas

  • Code-mixing is a major hurdle: The inherent complexity of interweaving languages and scripts significantly challenges standard sentiment analysis techniques.
  • Transformer models show promise: Fine-tuning pre-trained multilingual transformer models, particularly XLM-RoBERTa, proved highly effective for sentiment analysis in these code-mixed Dravidian languages.
  • Low-resource languages benefit from shared tasks: Organizing such tasks is crucial for driving research and developing resources for languages that are traditionally underserved in NLP.
  • Data diversity is key: User-generated content from social media provides a realistic and linguistically rich source for training sentiment analysis models.

Key Results

The shared task saw participation from 21 teams for the Tamil sub-task and 20 teams for the Tulu sub-task. Submissions were evaluated based on macro F1-score.

  • Tamil Task: The winning team, byteSizedLLM, achieved a macro F1-score of 0.5036. This team utilized a fine-tuned XLM-ROBERTa base model with a custom attention BiLSTM network.
  • Tulu Task: The top-performing team, lowes, secured a macro F1-score of 0.5938. This team used a language-specific BERT model pre-trained by 13cube-pune.

These results highlight the effectiveness of advanced transformer models and language-specific fine-tuning for sentiment analysis in these challenging linguistic contexts.

Limitations and Future Work

The paper acknowledges that the datasets, while valuable, are derived from social media and movie reviews, which might not cover the full spectrum of sentiment expression across all domains. The performance of models, especially for the Tulu language, is still relatively modest, indicating room for improvement. Future work could involve:

  • Expanding the datasets to include more diverse domains and linguistic phenomena.
  • Developing more robust methods for handling extreme code-mixing and transliteration variations.
  • Investigating unsupervised or semi-supervised learning techniques to further alleviate the reliance on labeled data for low-resource languages.
  • Exploring cross-lingual transfer learning more deeply to leverage resources from higher-resource languages.

Summary updated

BibTeX
@inproceedings{thenmozhi2025overview,
  author = {Durairaj Thenmozhi and Bharathi Raja Chakravarthi and Asha Hegde and Hosahalli Lakshmaiah Shashirekha and Rajeswari Natarajan and Sajeetha Thavareesan and Ratnasingam Sakuntharaj and Krishnakumari Kalyanasundaram and Charmathi Rajkumar and Poorvi Shetty and Harshitha S Kumar},
  title = {{Overview of the Shared Task on Sentiment Analysis in Tamil and Tulu}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure