TL;DR
This paper tackles the challenge of sentiment analysis in code-mixed Dravidian languages (Tamil and Tulu) by fine-tuning a multilingual BERT (mBERT) model. Despite class imbalance and noisy social media data, their approach achieved competitive results, ranking 5th and 6th in the respective tasks, highlighting mBERT's effectiveness for low-resource languages.
What Problem Does This Paper Address?
Sentiment analysis, the task of understanding the emotional tone of text, is crucial for gauging public opinion, especially on social media. However, analyzing sentiment in code-mixed languages, where multiple languages are blended within a single utterance, presents significant challenges. This is particularly true for Dravidian languages like Tamil and Tulu, which often have limited annotated datasets and exhibit diverse linguistic characteristics. The paper specifically addresses the need for effective sentiment analysis tools for code-mixed Tamil-English and Tulu-English social media text, a domain characterized by informal language, transliteration, spelling variations, and class imbalance.
Methods
The core of the proposed solution is the fine-tuning of a pre-trained Multilingual BERT (mBERT) model. mBERT, trained on 104 languages, is chosen for its inherent multilingual capabilities, making it well-suited for handling code-mixed text.
Key methodological steps include:
- Data Preprocessing: This involved cleaning the text by removing punctuation and special characters to reduce noise. Tokenization was performed using
BertTokenizerfrom the pre-trained mBERT model, with sequences padded or truncated to a maximum length of 128 tokens for consistent input. - Synthetic Sample Generation (Addressing Class Imbalance): A significant challenge was the class imbalance in the datasets. To mitigate this, the authors employed RandomOverSampler from the
unbalanced-learnlibrary. This technique duplicates samples from minority classes to create a more balanced distribution, preventing the model from being biased towards majority classes. - Model Training: The mBERT model was trained for 10 epochs using the AdamW optimizer with a learning rate of 3e-5. Batch size was set to 16, and maximum token length to 128. Class weights were computed to further adjust the loss function.
The authors also experimented with Tamil BERT and IndicBERT but found mBERT to be superior due to its broader multilingual coverage.
Datasets
The study utilized datasets provided for the DravidianLangTech@NAACL 2025 shared task. These datasets consist of code-mixed Tamil-English and Tulu-English text sourced from social media platforms like YouTube.
- Tamil Dataset:
- Training: 31,122 rows
- Development: 1,643 rows
- Test: 3,459 rows
- Tulu Dataset:
- Training: 13,301 rows
- Development: 1,643 rows
- Test: 1,479 rows
The datasets are described as real-world, informal, and noisy, presenting challenges like transliteration, spelling variations, and grammatical inconsistencies. A significant class imbalance was observed in both datasets.
Key Insights and Ideas
- mBERT's Robustness for Low-Resource Languages: The study strongly suggests that pre-trained multilingual models like mBERT are highly effective for sentiment analysis in low-resource languages and code-mixed scenarios, even with limited annotated data.
- The Power of Upsampling for Imbalance: RandomOverSampler proved to be a crucial technique for addressing class imbalance, enabling the model to learn from minority sentiment categories and improve overall performance.
- Code-Mixing as a Major Hurdle: The inherent complexities of code-switching, transliteration, and informal language in social media text pose significant challenges that require specialized approaches beyond standard sentiment analysis.
- Multilingualism Trumps Language Specificity (in this context): While language-specific models like Tamil BERT exist, mBERT's broader multilingual training provided an advantage in handling the code-mixed nature of the datasets.
Key Results
The paper reports competitive performance on the DravidianLangTech@NAACL 2025 shared task:
- Tamil: Achieved an F1-score of 0.4461, securing 6th place in the ranklist.
- Tulu: Achieved an F1-score of 0.569, securing 5th place in the ranklist.
The model demonstrated particular strength in identifying nuanced sentiment categories like mixed feelings, unknown states, and neutral sentiments.
Limitations and Future Work
Limitations:
- Dataset Structure Dependency: The approach's generalizability might be limited by its reliance on a predefined dataset structure, potentially struggling with datasets that have different distributions.
- Computational Complexity: The model's computational demands increase with input length, impacting scalability for very long texts without optimization.
- Hardware Dependence: Efficient training and inference require high-performance hardware (GPUs), which could be a barrier for deployment in resource-constrained environments.
Future Work:
- Domain-Adaptive Pretraining: Fine-tuning models on social media-specific corpora to improve understanding of informal and code-mixed text.
- External Linguistic Resources: Incorporating code-mixed lexicons and transliteration models to enhance accuracy, especially for low-resource languages like Tulu.
- Multimodal Sentiment Analysis: Integrating textual, visual, and audio cues from social media posts for a more comprehensive sentiment understanding.
Summary updated