Robust Sentiment Analysis for Low Resource languages Using Data Augmentation Approaches: A Case Study in Marathi

Aabha Pingle, Aditya Vyawahare, Isha Joshi, Rahul Tangsali, Geetanjali Kale, Raviraj JoshiarXiv 2023 · DOI · arXiv

Introduction

Marathi, despite having over 83 million speakers, remains a "low-resource" language in the field of Natural Language Processing (NLP). The scarcity of high-quality annotated datasets presents two primary challenges: overfitting and domain sensitivity. Models trained on limited data often memorize specific phrases instead of learning generalizable sentiment patterns, leading to poor performance when applied to data from different domains—for instance, transitioning from Twitter-based datasets to Reddit comments. This study investigates the efficacy of various data augmentation strategies, including BERT-based masking and GPT-driven generation, to build more robust sentiment analysis models for Marathi.

Methodology

The researchers established a baseline using the marathi-bert-v2 model and evaluated several advanced data augmentation techniques designed to improve generalization.

BERT-based Masking Strategies

  • Random Masking: 40% of words in a sentence were masked, with the model predicting replacements. This was tested in Sequential mode (predicting one word at a time) and Parallel mode (predicting all masked words simultaneously).
  • Named Entity Masking: A targeted approach using Named Entity Recognition (NER) to identify and mask specific entities such as people, places, or organizations. This forces the model to focus on the linguistic context of sentiment rather than specific entity associations.

Generative and Semantic Approaches

  • GPT-based Generation: Utilizing the l3cube-pune/marathi-gpt model for label generation (pseudo-labeling unlabeled data) and text completion (generating sentiment-consistent sentence endings).
  • Back-Translation & Paraphrasing: Standard techniques involving translation to English and back to Marathi, or using specialized Indic-language paraphrasing models to create semantic variations.

Datasets and Experimental Setup

The study utilized two primary datasets to evaluate cross-domain generalization:

  1. L3CubeMahaSent: Approximately 16,000 Marathi tweets covering politics and social issues.
  2. GoEmotions (Marathi): A Marathi translation of Reddit comments, simplified to a three-way sentiment scale (positive, negative, neutral).

By training on one domain and testing on the other, the researchers measured the model's ability to generalize beyond its training distribution.

Results and Discussion

The experimental results highlight the significant impact of data augmentation on model robustness, particularly in cross-domain scenarios.

Confusion matrices for the GoEmotions and MahaSent datasets across validation and test sets.

Performance Gains

The most notable improvements were observed in cross-domain tests. For a model fine-tuned on GoEmotions (Reddit) and tested on MahaSent (Twitter), the baseline accuracy was 0.6882. Implementing Sequential Named Entity Masking increased this accuracy to 0.7362, demonstrating that masking entities helps the model learn structural sentiment cues rather than memorizing entity-sentiment correlations.

In-domain performance also saw gains. For models trained and tested within the MahaSent domain, Sequential Random Masking achieved a peak accuracy of 0.8430. Furthermore, GPT-based label generation proved highly effective, yielding an in-domain accuracy of 0.8435.

Sequential vs. Parallel Masking

Across nearly all tests, sequential masking outperformed parallel masking. This is likely because the model can use previously predicted words as context for subsequent predictions, resulting in more coherent and linguistically sound synthetic data.

Conclusion

This research demonstrates that advanced data augmentation techniques can significantly mitigate the challenges of low-resource NLP for Marathi. By focusing on context over specific content through methods like Named Entity Masking, models become more resilient to domain shifts. While currently limited by the scarcity of advanced preprocessing tools for Marathi, these strategies provide a scalable framework for improving sentiment analysis in other Indic and low-resource languages.

Summary updated

BibTeX
@misc{pingle2023robust,
  author = {Aabha Pingle and Aditya Vyawahare and Isha Joshi and Rahul Tangsali and Geetanjali Kale and Raviraj Joshi},
  title = {{Robust Sentiment Analysis for Low Resource languages Using Data Augmentation Approaches: A Case Study in Marathi}},
  year = {2023},
  eprint = {2310.00734},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00734},
}
Figure