IndiSentiment140: Sentiment Analysis Dataset for Indian Languages with Emphasis on Low-Resource Languages using Machine Translation

Saurabh Kumar, Sanasam Ranbir Sanasam, Sukumar NandiNAACL 2024 · DOI

TL;DR

This paper tackles the challenge of sentiment analysis for low-resource Indian languages by leveraging machine translation to create a large-scale dataset, IndiSentiment140. They demonstrate that this MT-generated dataset, when trained with standard NLP models, can effectively capture sentiment, paving the way for broader NLP applications in India's diverse linguistic landscape.

What Problem Does This Paper Address?

Sentiment analysis, a crucial NLP task for understanding opinions and emotions in text, faces a significant hurdle in India due to its immense linguistic diversity. While resource-rich languages like English have abundant labeled datasets, many Indian languages suffer from a scarcity of such resources. This "low-resource" problem makes it difficult to build accurate sentiment analysis models for these languages. Manually creating datasets for each of India's many languages is a monumental and often infeasible task. The authors propose using machine translation (MT) as a practical solution to bridge this resource gap.

Methods

The core of this research lies in using machine translation to generate a sentiment analysis dataset. The authors took the widely-used English Sentiment140 dataset, a large collection of tweets, and translated it into 22 Indian languages using Google Translate. They then evaluated the effectiveness of this translated dataset by training two common NLP models: a Multi-Layer Perceptron (MLP) and a Long Short-Term Memory (LSTM) network. The choice of these simpler models was deliberate, aiming to assess the dataset's inherent quality and its ability to preserve sentiment and sequential information, rather than testing the limits of advanced architectures. They also compared the performance of models trained on their MT-generated dataset with models trained on a human-translated dataset (IndicSentiment) to gauge the quality of the machine translation.

Datasets

  • Sentiment140: The original English dataset used as the source for translation. It's a large collection of tweets labeled with positive or negative sentiment.
  • IndiSentiment140: The newly created dataset, consisting of the Sentiment140 dataset translated into 22 Indian languages using Google Translate.
  • IndicSentiment: A human-translated sentiment dataset for 13 Indian languages, used for comparative evaluation.
  • Flores-200, PMIndia, IndicSentiment (existing datasets): These publicly available parallel corpora were used to evaluate the quality of Google Translate's output (via BLEU scores) before creating IndiSentiment140.

Key Insights and Ideas

  • Machine Translation as a Viable Solution: The study strongly suggests that machine translation, specifically Google Translate, can be a practical and effective method for bootstrapping sentiment analysis datasets for low-resource languages.
  • Preservation of Sentiment and Structure: The translated datasets not only retain the sentiment characteristics of the original English data but also preserve the sequential structure of the text, which is crucial for NLP tasks.
  • Simpler Models Suffice for Quality Assessment: Using basic MLP and LSTM models proved sufficient to demonstrate the usability of the MT-generated dataset, indicating its potential for more complex models as well.
  • Context Matters for Embeddings: The performance differences observed between models using different embedding spaces (fastText vs. IndicBERT v2) highlight the importance of choosing appropriate embeddings for capturing sentiment in Indian languages.
  • Noise Impact: The presence of untranslated English words (noise) in the translated text can negatively impact model performance, though the effect varies across languages.

Key Results

  • Models trained on the IndiSentiment140 dataset achieved consistent accuracy across various Indian languages, generally around 75% ± 2% for MLP models using fastText embeddings and 76% ± 5% for MLP models using IndicBERT v2 embeddings.
  • LSTM models showed a notable performance improvement, suggesting that Google Translate adequately preserves sequential information.
  • The performance on the original English Sentiment140 dataset was significantly higher than on the translated Indian language datasets, as expected.
  • IndicBERT v2 embeddings generally yielded slightly better results than fastText embeddings.
  • BLEU scores for machine translation varied, with some low-resource languages showing lower scores, but the sentiment analysis performance remained surprisingly robust.

Limitations and Future Work

  • Noise in Translation: A significant limitation is the presence of untranslated English words within the translated datasets, which can act as noise and potentially affect model performance.
  • Lack of Human-Translated Benchmark: Direct comparison with a fully human-translated dataset for the exact same content is challenging due to the absence of such a benchmark, making a definitive assessment of translation quality difficult.
  • Future Work: The authors plan to incorporate more machine translation tools and APIs and expand the dataset to include more Indian languages. They also aim to further investigate the impact of noise and explore more advanced NLP models.

Summary updated

BibTeX
@inproceedings{kumar2024indisentiment,
  author = {Saurabh Kumar and Sanasam Ranbir Sanasam and Sukumar Nandi},
  title = {{IndiSentiment140: Sentiment Analysis Dataset for Indian Languages with Emphasis on Low-Resource Languages using Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.425},
}
Figure