TL;DR
This paper tackles the challenge of analyzing political sentiment in Tamil, a low-resource language, using social media data. They developed a system that combines advanced transformer embeddings with Support Vector Machines (SVM) to achieve effective sentiment classification, ranking 5th in a shared task.
What Problem Does This Paper Address?
Political sentiment analysis is crucial for understanding public opinion, but it's particularly challenging for low-resource languages like Tamil due to linguistic and cultural complexities, as well as data scarcity. Social media platforms like X (formerly Twitter) are rich sources of political discourse, but extracting nuanced sentiment from these short, often code-mixed (Tamil and English) posts requires sophisticated techniques. Existing methods often struggle with the unique characteristics of such data, necessitating new approaches to accurately gauge public mood.
Methods
The authors employed a multi-stage methodology:
- Preprocessing: This involved cleaning the text by replacing emojis with their textual equivalents, marking hashtags with
<hashtag>, converting text to lowercase, and removing special characters, URLs, and mentions to reduce noise and normalize the data. They also experimented with ADASYN for oversampling to address class imbalance. - Embedding Extraction: They utilized several pre-trained transformer models and static embedding techniques to generate rich feature representations of the text. These included:
- Transformer-based models: XLM-ROBERTa, MuRIL, IndicBERT, TamilSBERT, and STSb-XLM-R. For these, they used techniques like mean pooling of the last hidden state or the
[CLS]token representation. - Static embedding models: FastText was used for its ability to handle morphologically rich languages.
- LaBSE: Sentence representations were extracted and normalized for stability.
- Classification: The extracted embeddings were then fed into a Support Vector Machine (SVM) classifier. They specifically used Scikit-learn's
LinearSVCfor efficiency. To handle class imbalance, class weights were inversely proportional to class frequencies. Features were normalized usingRobustScaler, and grid search was used to tune regularization parameters. They also experimented with XGBoost with LaBSE embeddings and TF-IDF features.
Datasets
The dataset used for this study was provided by the 2025 Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics. It was pre-divided into training and test sets, with approximately 4,300 Tamil sentences in the training set and 430 in the test set. Each sentence contained hashtags and emojis and was categorized into seven distinct classes: Substantiated, Sarcastic, Opinionated, Positive, Negative, Neutral, and None of the Above. The dataset was noted to be imbalanced, with more samples in the "Opinionated" category than in "None of the Above."
Key Insights and Ideas
- Transformer embeddings are powerful for low-resource languages: The study highlights that advanced transformer models, when used to generate embeddings, can significantly improve sentiment analysis performance even with limited data.
- LaBSE shows promise for multilingual sentiment: The LaBSE embeddings, optimized for multilingual tasks, demonstrated strong performance when combined with SVM, indicating their effectiveness in handling code-mixed Tamil-English text.
- Class imbalance is a significant challenge: The imbalanced nature of the dataset, particularly the similarity between "Substantiated" and "Opinionated" classes, led to lower F1 scores for these categories and impacted overall performance.
- Code-mixing adds complexity: The inherent variations in grammar, transliteration, and word order in code-mixed Tamil-English text pose a substantial hurdle for sentiment analysis models.
Key Results
- The system achieved a Rank 5 in the NAACL shared task, demonstrating its effectiveness in a real-world setting.
- The Sentence-Transformers-LaBSE model with an SVM classifier reported the highest weighted F1 score of 0.310 among the SVM-based approaches.
- Post-shared task experiments with XGBoost using LaBSE embeddings and TF-IDF achieved the highest F1 score of 0.33.
- FastText also performed well with a weighted F1 score of 0.280, showing its capability in handling morphologically rich Tamil.
- MuRIL, despite being designed for Indian languages, yielded the lowest score (0.150), suggesting its pre-training might not sufficiently cover Tamil-English code-mixed text.
- The "None of the Above" class had a high score (0.79), while "Substantiated" had a low score (0.11), attributed to the similarity between these classes.
Limitations and Future Work
- Dataset Size and Diversity: The authors acknowledge that the dataset volume could be expanded to capture greater variability and improve model generalization.
- Multimodal Elements: The current text-only approach does not account for multimodal elements like images, videos, and memes, which are prevalent in political tweets and can influence sentiment.
- Concept Drift: Political narratives evolve, and models trained on past data can become outdated. Sentiment models need continuous updating to reflect current socio-political dynamics.
- Future Work: The paper suggests that using large language models could be a promising direction to address the challenges of code-mixed text and improve performance. Further research is also needed to develop more sophisticated classification approaches beyond basic positive/negative sentiment.
Summary updated