TL;DR
This paper tackles the challenge of sentiment analysis in code-mixed Dravidian languages (Tamil-English and Tulu-English), which are notoriously difficult for traditional NLP models due to linguistic mixing and non-native scripts. By fine-tuning the XLM-RoBERTa model, the researchers achieved competitive results, placing third in the DravidianLangTech@NAACL 2025 competition, highlighting the effectiveness of large pre-trained models for low-resource, complex language scenarios.
What Problem Does This Paper Address?
Sentiment analysis, the task of understanding opinions and emotions in text, is crucial for social media analysis. However, analyzing sentiment in languages like Tamil and Tulu, especially when mixed with English (code-switching), presents significant hurdles. These languages often feature blended sentences and non-standard scripts, which traditional monolingual sentiment analysis models struggle to handle. This research aims to bridge this gap by developing a robust sentiment analysis approach for these low-resource, code-mixed Dravidian languages.
Methods
The core of this research is the application and fine-tuning of the XLM-RoBERTa model. XLM-RoBERTa is a powerful, pre-trained transformer model designed for cross-lingual understanding, making it a strong candidate for handling code-mixed text. The authors utilized the XLM-RoBERTa tokenizer for text preprocessing, ensuring alignment with the model's architecture. For comparison, they also evaluated traditional machine learning models like Logistic Regression and Random Forest, along with BERT base and RoBERTa base transformer models, using TF-IDF vectorization. The fine-tuning process involved training the XLM-RoBERTa model on specific Tamil and Tulu datasets using the AdamW optimizer with a linear warm-up schedule, a batch size of 16, and early stopping.
Datasets
The study utilized two distinct datasets:
- Tamil Dataset: Consisting of social media comments with four sentiment classes: Positive, Negative, Mixed Feelings, or Unknown State.
- Tulu Dataset: Comprising social media comments with five sentiment classes: Not Tulu, Positive, Negative, Neutral, or Mixed.
These datasets were part of the Sentiment Analysis in Tamil and Tulu Dravidian-LangTech@NAACL 2025 competition. The paper notes that the Tamil dataset exhibits significant class imbalance, with a high frequency of positive class labels, which impacts model performance.
Key Insights and Ideas
- XLM-RoBERTa's Strength in Code-Mixing: The research demonstrates that a large, pre-trained cross-lingual model like XLM-RoBERTa can effectively handle the complexities of code-mixed Dravidian languages, outperforming traditional models and even other transformer baselines.
- Class Imbalance is a Major Hurdle: The significant imbalance in the Tamil dataset (overrepresentation of positive sentiment) directly correlates with lower accuracy, underscoring the need for strategies to address this issue in low-resource settings.
- Fine-tuning is Key: While XLM-RoBERTa is powerful out-of-the-box, fine-tuning it on domain-specific, code-mixed data is crucial for achieving optimal performance.
Key Results
The fine-tuned XLM-RoBERTa model achieved a macro F1-score of 59% for the Tulu dataset and 49% for the Tamil dataset. This performance secured the researchers third place in the DravidianLangTech@NAACL 2025 competition. The paper also presents a comparative table showing XLM-RoBERTa outperforming RoBERTa base, BERT base, and TF-IDF based models on both datasets. For instance, on the Tulu dataset, XLM-RoBERTa achieved an accuracy of 0.71 and a macro F1 of 0.59, compared to BERT base's 0.69 accuracy and 0.59 macro F1.
Limitations and Future Work
The authors acknowledge that class imbalance in the Tamil dataset significantly affects model performance. They suggest future work could explore techniques like:
- Re-sampling: Adjusting the distribution of classes in the training data.
- Cost-sensitive learning: Assigning different misclassification costs to different classes.
- Advanced data augmentation: Creating more training data, especially for underrepresented classes.
- Developing comprehensive lexical resources: Building richer dictionaries and linguistic tools for these languages would further enhance transformer-based sentiment analysis.
Summary updated