GSAC: A Gujarati Sentiment Analysis Corpus from Twitter

Monil Gokani, Radhika MamidiACL 2023

TL;DR

This paper introduces GSAC, the first large-scale, manually annotated Gujarati sentiment analysis corpus sourced from Twitter. They demonstrate its utility by establishing baselines with various models, finding that pre-trained multilingual models fine-tuned on Gujarati data perform best, with IndicBERT and GujaratiBERT leading the pack.

What Problem Does This Paper Address?

Sentiment analysis, the task of determining the emotional tone of text, is crucial for understanding online opinions and for applications like content moderation. While significant resources exist for many languages, there's a notable lack of publicly available, large-scale datasets for Gujarati, a language spoken by over 55 million people. This gap hinders research and development of sentiment analysis tools for the Gujarati-speaking community. Previous efforts in Gujarati sentiment analysis have relied on small datasets or proprietary resources, making reproducibility and comparison difficult.

Methods

The authors collected tweets from Twitter using a sentiment lexicon derived from machine-translated English words to ensure a balanced representation of sentiments. They employed a multi-stage filtering process to clean the data, removing duplicates, short tweets, and gibberish. The core of their methodology lies in the manual annotation process, where four native Gujarati speakers were trained to classify tweets into three categories: positive, negative, and neutral, with an additional "unfit" category for unusable tweets. Inter-annotator agreement was measured using Fleiss' Kappa, showing moderate to strong agreement after training. To establish baselines, they experimented with two types of models: traditional feature-based models (Naive Bayes, Logistic Regression, SVM, Random Forests, MLP) using Bag-of-Words and TF-IDF features, and deep contextualized models, specifically fine-tuning multilingual transformer models like mBERT, XLM-ROBERTa, XLM-T, GujaratiBERT, and IndicBERT.

Datasets

The primary dataset introduced in this paper is the Gujarati Sentiment Analysis Corpus (GSAC), containing 6,575 manually annotated tweets. This dataset is publicly available on GitHub. The authors also reference existing datasets for other Indian languages (Hindi, Telugu, Marathi, Bengali, Tamil, Malayalam) in their related work section, but GSAC is the novel contribution.

Key Insights and Ideas

  • The critical need for language-specific resources: The paper strongly highlights how the lack of dedicated datasets for languages like Gujarati impedes progress in NLP tasks.
  • Effectiveness of sentiment lexicons for data collection: Using a sentiment lexicon, even a machine-translated one, proved effective in guiding the collection of a diverse set of tweets with varying sentiments.
  • Pre-trained multilingual models are key for low-resource languages: The experiments clearly show that models pre-trained on a vast number of languages, especially those that include Gujarati in their training or fine-tuning data (like IndicBERT and GujaratiBERT), significantly outperform general multilingual models or traditional methods.
  • Domain-specific pre-training matters: XLM-T, despite having fewer Gujarati samples than some other models, performed comparably well, suggesting that pre-training on domain-specific data (like Twitter) can be highly beneficial.
  • Traditional models still have a place: While deep learning models dominate, the Naive Bayes model with TF-IDF features achieved the highest precision among all models, demonstrating that simpler, computationally less expensive methods can still be competitive for specific metrics.

Key Results

  • The GSAC dataset consists of 6,575 manually annotated tweets, split into training (70%), development (10%), and testing (20%) sets.
  • The class distribution in GSAC is approximately 45.12% neutral, 30.05% positive, and 24.83% negative.
  • Among the traditional feature-based models, Random Forests with Bag-of-Words features achieved the best performance (Accuracy: 0.58, Weighted F1: 0.55).
  • IndicBERT and GujaratiBERT achieved the best overall performance on the test set, with IndicBERT reaching an Accuracy of 0.66 and a Weighted F1 score of 0.66.
  • mBERT and XLM-ROBERTa performed poorly, likely due to insufficient representation of Gujarati in their training data.
  • XLM-T showed comparable performance to the top models, suggesting the value of domain-specific pre-training.

Limitations and Future Work

  • Limitations: The authors acknowledge that mBERT and XLM-ROBERTa underperformed, potentially due to the vast number of languages they were trained on, leading to insufficient Gujarati representation. They also note that while XLM-T performed well, its training data was limited to ~10,000 Gujarati samples.
  • Future Work: The authors plan to explore methods for automatically extending the GSAC dataset, possibly through bootstrapping or by translating existing datasets from languages like English or Hindi. They also aim to investigate how to better leverage pre-trained models for low-resource languages.

Summary updated

BibTeX
@inproceedings{gokani2023gsac,
  author = {Monil Gokani and Radhika Mamidi},
  title = {{GSAC: A Gujarati Sentiment Analysis Corpus from Twitter}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure