KEC-Elite-Analysts@DravidianLangTech 2025: Deciphering Emotions in Tamil-English and Code-Mixed Social Media Tweets

Malliga Subramanian, Aruna A, Anbarasan T, Amudhavan M, Jahaganapathi S, Kogilavani ShanmugavadivelNAACL 2025

TL;DR

This paper tackles the challenge of sentiment analysis in Tamil-English code-mixed social media text, a complex task due to informal language and code-switching. The authors demonstrate that transformer-based models, particularly mBERT, significantly outperform traditional machine learning approaches in accurately classifying sentiment, paving the way for more nuanced understanding of online discourse in multilingual communities.

What Problem Does This Paper Address?

Social media has become a melting pot of languages, especially in multilingual communities. For instance, users often blend Tamil and English within a single sentence or tweet, a phenomenon known as code-mixing. This presents a significant hurdle for traditional Natural Language Processing (NLP) techniques, which are typically designed for monolingual text. Sentiment analysis, the task of determining the emotional tone of text, becomes particularly challenging in this context due to informal grammar, slang, cultural nuances, and the inherent difficulty of understanding mixed-language structures. Existing models struggle to capture these complexities, leading to inaccurate sentiment classifications. This paper aims to address this gap by exploring and evaluating various machine learning and transformer-based models for sentiment analysis on Tamil-English code-mixed social media data.

Methods

The researchers employed a comparative approach, evaluating both traditional machine learning models and advanced transformer-based models.

  • Traditional Models: Logistic Regression, Support Vector Machines (SVM), K-Nearest Neighbors (KNN), Random Forest, and XGBoost were used as baselines. These models typically rely on hand-crafted features or simpler vectorization techniques like TF-IDF and n-grams.
  • Transformer Models: The study focused on BERT and its multilingual variant, mBERT. These models leverage self-attention mechanisms to capture long-range dependencies and contextual relationships within text, making them well-suited for complex linguistic phenomena like code-mixing. The mBERT model was fine-tuned on the specific Tamil-English dataset.

The paper also details preprocessing steps, including removing special characters, emojis, numbers, and URLs, as well as transliteration and tokenization. They also investigated the impact of hyperparameter tuning (learning rate, batch size, dropout rate) and preprocessing techniques (stopword removal, subword tokenization, lowercasing) through ablation studies to understand their influence on model performance.

Datasets

The primary dataset used in this study was collected from social media platforms, specifically Twitter (X) and Facebook. It consists of Tamil-English code-mixed comments related to socio-political and cultural discussions. This dataset was part of the Shared Task on Sentiment Analysis in Tamil at DravidianLangTech@NAACL 2025. The dataset contains 31,122 comments, with a distribution of 58.30% positive, 13.34% negative, and no neutral or mixed sentiments explicitly mentioned in the provided text (though the authors discuss sentiment classification into Positive, Negative, Mixed, and Unknown). The annotation process involved manual labeling by linguists and NLP experts, with inter-annotator agreement measured using Cohen's Kappa score.

Key Insights and Ideas

  • Transformers Reign Supreme: The most significant insight is the clear superiority of transformer-based models (mBERT) over traditional machine learning models for Tamil-English code-mixed sentiment analysis. This highlights the power of self-attention mechanisms in understanding complex linguistic structures.
  • Code-Mixing is a Major Hurdle: The study underscores the inherent difficulty of code-mixed text for sentiment analysis, with informal grammar, slang, and script mixing posing substantial challenges.
  • Preprocessing Matters, But Selectively: While preprocessing is crucial, the impact of different techniques varies. Subword tokenization (like BPE and WordPiece) showed a positive impact, whereas lowercasing had minimal effect due to case-insensitive embeddings.
  • Class Imbalance is a Concern: The dataset exhibits class imbalance, which can skew model performance. The authors addressed this by using SMOTE during training and employing macro-averaged F1-score for evaluation.
  • Nuance is Hard to Capture: The error analysis reveals that sarcasm, mixed sentiments, and negation phrases are particularly difficult for models to interpret correctly, indicating areas for future improvement.

Key Results

  • mBERT Outperforms: mBERT achieved the highest accuracy of 80%, with a precision of 78%, recall of 75%, and an F1-score of 76%. This significantly surpasses the performance of traditional models.
  • Traditional Model Performance: Logistic Regression achieved an accuracy of 72%, Random Forest 70%, SVM 68%, and XGBoost 65%.
  • Preprocessing Impact: The ablation study showed that subword tokenization improved accuracy to 80.3%, while no preprocessing yielded 77.5%.
  • Computational Efficiency: The paper analyzes computational efficiency, with "Our Model" (likely mBERT or a fine-tuned variant) showing an inference time of 95ms and 1100 tokens/sec, indicating good performance for real-world deployment.

Limitations and Future Work

Limitations: * The paper acknowledges the challenge of class imbalance in the dataset. * Error analysis highlights difficulties in handling sarcasm, mixed sentiments, and negation. * The reliance on large labeled datasets and high computational requirements for transformer models is mentioned as a challenge.

Future Work: * Expand the dataset to include more diverse linguistic styles and emerging slang. * Improve adaptability to evolving language patterns. * Explore unsupervised domain adaptation techniques for better performance on unseen contexts. * Optimize transformer architectures (e.g., DistilBERT, TinyBERT) to reduce inference time for real-world applications. * Develop an interactive API or dashboard for practical use in social media monitoring and sentiment analysis. * Investigate hybrid models and advanced model architectures to address code-switching and class imbalance.

Summary updated

BibTeX
@inproceedings{subramanian2025kec,
  author = {Malliga Subramanian and Aruna A and Anbarasan T and Amudhavan M and Jahaganapathi S and Kogilavani Shanmugavadivel},
  title = {{KEC-Elite-Analysts@DravidianLangTech 2025: Deciphering Emotions in Tamil-English and Code-Mixed Social Media Tweets}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure