TL;DR
This paper presents a pioneering approach to sentiment analysis for code-mixed social media text, specifically focusing on English-Bengali-Hindi Facebook posts. By combining extensive text normalization with a Multilayer Perceptron (MLP) model and a custom-built, manually labeled corpus, the authors achieved a classification accuracy of 68.5%, significantly outperforming baseline models.
What Problem Does This Paper Address?
Most Natural Language Processing (NLP) tools are designed for monolingual, formal text. However, social media users—particularly in multilingual regions like India—frequently engage in code-mixing, the practice of alternating between two or more languages (e.g., English and Bengali) within a single post.
These texts are notoriously difficult to analyze because they are:
- Informal: Lacking standard grammar and spelling.
- Noisy: Filled with abbreviations, repetitions, and arbitrary punctuation.
- Resource-Poor: Standard sentiment lexicons for languages like Bengali are often too formal to capture colloquial social media expressions.
The authors set out to bridge this gap by creating a specialized pipeline to identify whether a code-mixed post is positive, negative, or neutral.
Methods
The authors developed a three-stage pipeline: preprocessing, feature extraction, and classification.
1. Preprocessing
To handle the "noise" of Facebook posts, the team implemented:
- Abbreviation Expansion: Converting "btw" to "by the way" or "clg" to "college."
- Punctuation Handling: Removing punctuation while recording the count of exclamation and question marks as features.
- Character Normalization: Reducing repetitions (e.g., "loooool" becomes "lool") to help with dictionary matching.
2. Feature Engineering
The model relies on 13 distinct features grouped into three categories:
- Lexical (Group 1): Matches against SentiWordNet, Opinion Lexicon, and custom-built English and Bengali sentiment word lists. Crucially, they developed a transliterated Bengali sentiment list to handle Bengali words written in Roman script.
- Syntactic (Group 2): Part-of-Speech (POS) tags, specifically focusing on adjectives and adverbs (JJ, RB) which often carry sentiment.
- Stylistic (Group 3): Density of uppercase words, exclamation points, question marks, character repetitions, and the frequency of code-switching points.
3. Classification
The researchers used a Multilayer Perceptron (MLP) model implemented in WEKA. The network utilized sigmoid nodes and backpropagation, trained over 500 epochs to classify posts into positive, negative, or neutral categories.
Datasets
The study utilized a modified version of the ICON-2015 Shared Task corpus, consisting of 882 Facebook posts from the "JU Confession" group. The corpus is primarily English-Bengali with occasional Hindi.
Two human annotators labeled the data. Due to the inherent complexity of code-mixed text and sarcasm, the initial inter-annotator agreement (Kappa) was relatively low at 0.4354. To ensure high-quality training data, the authors restricted their final dataset to the 565 instances where both annotators were in total agreement.
Key Insights and Ideas
- The Failure of Formal Lexicons: The authors found that standard Bengali sentiment resources were too formal. They had to create a "Colloquial Bengali Sentiment Word" list to capture social media slang like jata (hopeless) or hebby (excellent).
- Style Isn't Everything: Surprisingly, adding stylistic features (Group 3, like capitalization and punctuation density) actually decreased the model's accuracy. The most robust results came from combining lexical and syntactic features.
- Code-Switching as a Signal: The frequency of switching between languages was included as a feature, based on the observation that users often shift languages to clarify an opinion or emphasize a point.
- Neutral Bias: The model showed a slight bias toward the "neutral" class, likely because many positive and negative posts were misidentified as neutral due to subtle linguistic cues.
Key Results
The system achieved a top accuracy of 68.5% using a combination of word-based (lexical) and syntactic features. This is a notable improvement over the baseline accuracy of 55.2% (which assumes all instances are neutral).
| Class | Precision | Recall | F-measure |
|---|---|---|---|
| Positive | 0.641 | 0.481 | 0.55 |
| Neutral | 0.716 | 0.857 | 0.78 |
| Negative | 0.588 | 0.455 | 0.513 |
The confusion matrix revealed that the model is most effective at identifying neutral posts but struggles more with the nuances of negative sentiment in a code-mixed context.
Limitations and Future Work
The authors acknowledge several hurdles for future research:
- Dataset Size: The current dataset is small (565 agreed-upon posts). Larger datasets are needed to reduce bias and improve deep learning performance.
- Sarcasm and Negation: The current model does not explicitly handle sarcasm or the inversion of sentiment through negation (e.g., "not good"), which are common in social media.
- Entity Targeting: Future work aims to identify not just the sentiment, but the specific entity or object the sentiment is directed toward.
- Distant Supervision: The team suggests using emoticons as "noisy labels" to automatically curate much larger training sets in the future.
Summary updated