TL;DR
This paper investigates sentiment analysis for code-mixed Tamil and Tulu social media text, comparing traditional Machine Learning (ML) with Deep Learning (DL) approaches. Surprisingly, ML models, particularly Logistic Regression with TF-IDF, outperformed DL models (BiLSTM with Attention) on these low-resource languages, highlighting the effectiveness of simpler methods when data is scarce.
What Problem Does This Paper Address?
The increasing use of code-mixed languages on social media, where speakers seamlessly blend words and phrases from multiple languages within a single sentence, presents a significant challenge for Natural Language Processing (NLP) tasks like sentiment analysis. This is especially true for low-resource languages like Tamil and Tulu, which often lack standardized grammar, exhibit diverse transliteration styles, and have limited publicly available annotated datasets. Existing sentiment analysis tools, often trained on monolingual data, struggle to accurately interpret the nuances and complexities of these mixed-language texts. This paper aims to bridge this gap by comparing the effectiveness of established ML techniques against more advanced DL models for sentiment classification in Tamil-English and Tulu-English code-mixed datasets.
Methods
The study employs two distinct methodologies for sentiment classification:
- Machine Learning (ML) Approach: This method utilizes a classic pipeline. Text is first preprocessed, including transliteration to English script, tokenization, lowercasing, and removal of punctuation. Feature extraction is performed using TF-IDF (Term Frequency-Inverse Document Frequency) with character-level n-grams (ranging from 1 to 4) to capture word-level and subword-level patterns crucial for code-mixed text. A Logistic Regression classifier is then trained on these TF-IDF vectors. This approach is chosen for its known efficiency with smaller datasets.
- Deep Learning (DL) Approach: This method leverages more advanced techniques. After similar preprocessing steps (transliteration, tokenization, lowercasing, punctuation removal), the text is converted into sequences and mapped to word embeddings using FastText. These embeddings are then fed into a Bidirectional LSTM (BiLSTM) model enhanced with an attention mechanism. The BiLSTM captures forward and backward dependencies, while the attention mechanism allows the model to focus on the most informative words. The architecture includes dense layers with ReLU activation and L2 regularization, culminating in a softmax output layer for classification.
Datasets
The paper utilizes two code-mixed datasets:
- Tamil-English Code-Mixed Dataset: Consists of 31,122 samples for training and 3,843 for validation.
- Tulu-English Code-Mixed Dataset: Consists of 13,308 samples for training and 1,643 for validation.
These datasets appear to be existing benchmarks for Dravidian code-mixed sentiment analysis, as referenced in the related work section.
Key Insights and Ideas
- ML's Resilience in Low-Resource Settings: The most striking insight is that traditional ML models (TF-IDF + Logistic Regression) significantly outperformed DL models (FastText + BiLSTM + Attention) on both Tamil and Tulu code-mixed datasets. This challenges the common assumption that DL models are always superior, especially in scenarios with limited training data.
- Data Scarcity Favors Simplicity: The performance disparity was more pronounced for Tulu, which has a smaller dataset. This strongly suggests that DL models, which require substantial data to learn robust representations and avoid overfitting, struggle when data is scarce. ML models, on the other hand, can generalize better with less data by relying on statistical feature extraction.
- Importance of Character-Level Features: The use of character-level n-grams in the ML approach was crucial for capturing subword patterns, which are prevalent in code-mixed text and often missed by word-level embeddings alone.
Key Results
The experimental results, summarized in Table 2, show:
- Tamil:
- ML (Logistic Regression): Macro F1 Score = 0.46, Accuracy = 0.54
- DL (BiLSTM with Attention): Macro F1 Score = 0.43, Accuracy = 0.58
- Tulu:
- ML (Logistic Regression): Macro F1 Score = 0.60, Accuracy = 0.69
- DL (BiLSTM with Attention): Macro F1 Score = 0.48, Accuracy = 0.65
The ML model achieved higher Macro F1 scores for both languages, with a notable lead in Tulu (0.60 vs. 0.48). While the DL model achieved slightly higher accuracy for Tamil, the Macro F1 score is a more robust metric for imbalanced datasets, making the ML model's performance more compelling.
Limitations and Future Work
The authors acknowledge several limitations:
- Dataset Size: The relatively small size of the datasets, particularly for Tulu, limits the generalizability of the models.
- Language Scope: The study is confined to Tamil-English and Tulu-English code-mixed texts, limiting its applicability to other Dravidian languages.
- ML Model Limitations: The TF-IDF approach lacks deep contextual understanding.
- DL Model Limitations: The DL model was not fine-tuned on Dravidian corpora, and transformer-based models (like mBERT, XLM-R) were not explored. Transliteration variations were also not explicitly handled.
Future work could focus on:
- Creating larger and more diverse datasets.
- Exploring transfer learning or pre-trained multilingual models for improved performance in low-resource scenarios.
- Investigating transformer-based models and fine-tuning them on Dravidian corpora.
- Developing methods to explicitly handle transliteration variations.
Summary updated