Experimental Evaluation of Deep Learning Models for Marathi Text Classification

Atharva Kulkarni, Meet Mandhane, Manali Likhitkar, Gayatri Kshirsagar, Jayashree Jagdale, Raviraj JoshiLecture Notes in Networks and Systems 2022 · DOI · arXiv · OpenAlex

Abstract

Marathi, an Indo-Aryan language spoken by over 83 million people, remains underrepresented in mainstream Natural Language Processing (NLP) research. This study provides a comprehensive empirical benchmark for Marathi text classification, evaluating architectures ranging from traditional deep learning models (CNN, LSTM) to state-of-the-art Transformers (BERT, IndicBERT). Our findings demonstrate that for low-resource languages like Marathi, the choice of pre-trained embeddings is often as critical as the model architecture itself, with specialized linguistic resources significantly boosting the performance of simpler models.

Introduction

Despite its large speaker base, Marathi is classified as a "low-resource" language in NLP due to the scarcity of large-scale annotated datasets. Its morphological richness and use of the Devanagari script present unique challenges for automated text processing. While the global NLP community has shifted toward massive Transformer-based models, their efficacy on regional Indian languages requires systematic validation. This paper addresses this gap by comparing various deep learning methods on standard Marathi classification tasks.

Methods

The researchers evaluated several architectures and embedding strategies to identify the most effective configurations for Marathi text classification.

Architectures

  • Baseline Models: 1D Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM), and Bidirectional LSTM (BiLSTM) networks.
  • Transfer Learning: Universal Language Model Fine-tuning (ULMFiT) utilizing the iNLTK pre-trained language model.
  • Transformers: Multilingual BERT (mBERT) and IndicBERT, an ALBERT-based model trained on 11 Indian languages.

Embedding Strategies

To handle Marathi's complex morphology, the study tested several initialization methods:

  1. Random Initialization: Starting from scratch with word or sub-word units.
  2. FB FastText: General-purpose pre-trained embeddings from Facebook.
  3. IndicNLP FastText: Embeddings specifically trained on a large corpus of Indian languages.
  4. Sub-word Tokenization: Utilizing SentencePiece to decompose words into smaller units, mitigating the out-of-vocabulary (OOV) problem.

Datasets

Models were benchmarked on two publicly available datasets:

  • IndicNLP News Article Dataset: Longer documents (avg. 250 words) across three categories: Sports, Entertainment, and Lifestyle.
  • iNLTK Headline Dataset: Short texts (avg. 7 words) across Entertainment, Sports, and State news, representing a more challenging context-limited task.

Results

The experimental results indicate a high level of performance across most models, particularly when paired with language-specific embeddings.

Performance comparison of various deep learning models and embedding strategies on Marathi news datasets.

As shown in the table above, the integration of IndicNLP FastText embeddings significantly improved the performance of CNN and LSTM architectures. Notably, a CNN model using these embeddings achieved 99.37% accuracy on news articles and 94.88% on headlines, outperforming or matching much more complex Transformer models like IndicBERT (94.50% on headlines).

Key Insights

  • Domain-Specific Embeddings are Critical: The study highlights that for Marathi, the quality of the initial word representations matters more than the depth of the neural network. Simple models using IndicNLP FastText often outperformed complex Transformers.
  • The Power of "Indic" Pre-training: IndicBERT consistently outperformed the standard Multilingual BERT (mBERT), proving that pre-training on a cluster of related languages is more effective than training on a massive, generic global set.
  • Efficiency vs. Accuracy: For many applications, a single-layer CNN or LSTM with good embeddings is preferable to a Transformer. They are faster to train and easier to deploy while providing nearly identical accuracy.

Conclusion

This study establishes a robust baseline for Marathi text classification, suggesting that specialized linguistic resources can be more impactful than raw model complexity for low-resource languages. Future work should expand these benchmarks to noisier domains like social media and more complex tasks such as Named Entity Recognition (NER).

Summary updated

BibTeX
@inproceedings{kulkarni2022experimental,
  author = {Atharva Kulkarni and Meet Mandhane and Manali Likhitkar and Gayatri Kshirsagar and Jayashree Jagdale and Raviraj Joshi},
  title = {{Experimental Evaluation of Deep Learning Models for Marathi Text Classification}},
  booktitle = {Lecture Notes in Networks and Systems},
  year = {2022},
  doi = {10.1007/978-981-16-6407-6\_53},
}

Loading related content...

Figure