Sentiment Identification in Code-Mixed Social Media Text

Souvick Ghosh, Satanu Ghosh, Dipankar DasarXiv 2017 · arXiv

TL;DR

This paper presents a pioneering approach to sentiment analysis for code-mixed social media text, specifically focusing on English-Bengali-Hindi Facebook posts. By combining extensive text normalization with a Multilayer Perceptron (MLP) model and a custom-built, manually labeled corpus, the authors achieved a classification accuracy of 68.5%, significantly outperforming baseline models.

What Problem Does This Paper Address?

Most Natural Language Processing (NLP) tools are designed for monolingual, formal text. However, social media users—particularly in multilingual regions like India—frequently engage in code-mixing, the practice of alternating between two or more languages (e.g., English and Bengali) within a single post.

These texts are notoriously difficult to analyze because they are:

  1. Informal: Lacking standard grammar and spelling.
  2. Noisy: Filled with abbreviations, repetitions, and arbitrary punctuation.
  3. Resource-Poor: Standard sentiment lexicons for languages like Bengali are often too formal to capture colloquial social media expressions.

The authors set out to bridge this gap by creating a specialized pipeline to identify whether a code-mixed post is positive, negative, or neutral.

Methods

The authors developed a three-stage pipeline: preprocessing, feature extraction, and classification.

1. Preprocessing

To handle the "noise" of Facebook posts, the team implemented:

  • Abbreviation Expansion: Converting "btw" to "by the way" or "clg" to "college."
  • Punctuation Handling: Removing punctuation while recording the count of exclamation and question marks as features.
  • Character Normalization: Reducing repetitions (e.g., "loooool" becomes "lool") to help with dictionary matching.

2. Feature Engineering

The model relies on 13 distinct features grouped into three categories:

  • Lexical (Group 1): Matches against SentiWordNet, Opinion Lexicon, and custom-built English and Bengali sentiment word lists. Crucially, they developed a transliterated Bengali sentiment list to handle Bengali words written in Roman script.
  • Syntactic (Group 2): Part-of-Speech (POS) tags, specifically focusing on adjectives and adverbs (JJ, RB) which often carry sentiment.
  • Stylistic (Group 3): Density of uppercase words, exclamation points, question marks, character repetitions, and the frequency of code-switching points.
Fig. 1: Overview of the System Architecture.

3. Classification

The researchers used a Multilayer Perceptron (MLP) model implemented in WEKA. The network utilized sigmoid nodes and backpropagation, trained over 500 epochs to classify posts into positive, negative, or neutral categories.

Datasets

The study utilized a modified version of the ICON-2015 Shared Task corpus, consisting of 882 Facebook posts from the "JU Confession" group. The corpus is primarily English-Bengali with occasional Hindi.

Two human annotators labeled the data. Due to the inherent complexity of code-mixed text and sarcasm, the initial inter-annotator agreement (Kappa) was relatively low at 0.4354. To ensure high-quality training data, the authors restricted their final dataset to the 565 instances where both annotators were in total agreement.

Key Insights and Ideas

  • The Failure of Formal Lexicons: The authors found that standard Bengali sentiment resources were too formal. They had to create a "Colloquial Bengali Sentiment Word" list to capture social media slang like jata (hopeless) or hebby (excellent).
  • Style Isn't Everything: Surprisingly, adding stylistic features (Group 3, like capitalization and punctuation density) actually decreased the model's accuracy. The most robust results came from combining lexical and syntactic features.
  • Code-Switching as a Signal: The frequency of switching between languages was included as a feature, based on the observation that users often shift languages to clarify an opinion or emphasize a point.
  • Neutral Bias: The model showed a slight bias toward the "neutral" class, likely because many positive and negative posts were misidentified as neutral due to subtle linguistic cues.

Key Results

The system achieved a top accuracy of 68.5% using a combination of word-based (lexical) and syntactic features. This is a notable improvement over the baseline accuracy of 55.2% (which assumes all instances are neutral).

Class Precision Recall F-measure
Positive 0.641 0.481 0.55
Neutral 0.716 0.857 0.78
Negative 0.588 0.455 0.513

The confusion matrix revealed that the model is most effective at identifying neutral posts but struggles more with the nuances of negative sentiment in a code-mixed context.

Limitations and Future Work

The authors acknowledge several hurdles for future research:

  • Dataset Size: The current dataset is small (565 agreed-upon posts). Larger datasets are needed to reduce bias and improve deep learning performance.
  • Sarcasm and Negation: The current model does not explicitly handle sarcasm or the inversion of sentiment through negation (e.g., "not good"), which are common in social media.
  • Entity Targeting: Future work aims to identify not just the sentiment, but the specific entity or object the sentiment is directed toward.
  • Distant Supervision: The team suggests using emoticons as "noisy labels" to automatically curate much larger training sets in the future.

Summary updated

Abstract

Sentiment analysis is the Natural Language Processing (NLP) task dealing with the detection and classification of sentiments in texts. While some tasks deal with identifying the presence of sentiment in the text (Subjectivity analysis), other tasks aim at determining the polarity of the text categorizing them as positive, negative and neutral. Whenever there is a presence of sentiment in the text, it has a source (people, group of people or any entity) and the sentiment is directed towards some entity, object, event or person. Sentiment analysis tasks aim to determine the subject, the target and the polarity or valence of the sentiment. In our work, we try to automatically extract sentiment (positive or negative) from Facebook posts using a machine learning approach.While some works have been done in code-mixed social media data and in sentiment analysis separately, our work is the first attempt (as of now) which aims at performing sentiment analysis of code-mixed social media text. We have used extensive pre-processing to remove noise from raw text. Multilayer Perceptron model has been used to determine the polarity of the sentiment. We have also developed the corpus for this task by manually labeling Facebook posts with their associated sentiments.
BibTeX
@misc{ghosh2017sentiment,
  author = {Souvick Ghosh and Satanu Ghosh and Dipankar Das},
  title = {{Sentiment Identification in Code-Mixed Social Media Text}},
  year = {2017},
  eprint = {1707.01184},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1707.01184},
}

Loading related content...

Figure