L3Cube-MahaNews: News-Based Short Text and Long Document Classification Datasets in Marathi

Saloni Mittal, Vidula Magdum, Sharayu Hiwarkhedkar, Omkar Dhekane, Raviraj JoshiCommunications 2024 · DOI · arXiv · OpenAlex

Introduction

Marathi is the third most spoken language in India, with over 83 million speakers, yet it remains categorized as a "low-resource" language in the field of Natural Language Processing (NLP). Existing Marathi datasets for text classification often suffer from two primary limitations: they are frequently too simplistic, with few target labels leading to near-perfect model accuracy, and they lack diversity in text length. To address these issues, this research introduces L3Cube-MahaNews, the largest supervised dataset for Marathi text classification to date, featuring over 108,000 records across 12 diverse categories.

The L3Cube-MahaNews Dataset

The researchers curated the dataset by scraping news articles from the Marathi news site Lokmat, utilizing the site's original human-labeled categories as ground truth. The corpus is uniquely structured into three sub-datasets based on text length to test the sensitivity of Transformer-based models to document size:

  • Short Headlines Classification (SHC): Contains only the headlines of news articles.
  • Long Paragraph Classification (LPC): Contains 53,593 individual paragraphs extracted from articles.
  • Long Document Classification (LDC): Contains the full text of entire news articles.

All three versions utilize the same 12 categories: Auto, Bhakti (Spirituality), Crime, Education, Fashion, Health, International, Manoranjan (Entertainment), Politics, Sports, Tech, and Travel.

Experimental Methodology

To establish a baseline for the dataset, the authors fine-tuned three state-of-the-art BERT-based models:

  1. MahaBERT: A monolingual model specifically fine-tuned on Marathi corpora.
  2. IndicBERT: A multilingual model trained on 12 major Indian languages.
  3. MuRIL: A massive multilingual model trained on 17 Indian languages.

Evaluation was conducted using standard metrics, including Accuracy, F1 Score, Recall, and Precision (all Macro-averaged).

Key Results and Insights

The Monolingual Advantage

The specialized MahaBERT model consistently outperformed the broader multilingual models (IndicBERT and MuRIL) across all datasets. For example, on the SHC dataset, MahaBERT reached 91.1% accuracy, while IndicBERT trailed at 89.3%. This suggests that for low-resource languages, a deep monolingual understanding is often more effective than the broad but potentially shallower coverage of multilingual models.

The Role of Context and Length

The models performed best on the Long Document Classification (LDC) dataset, with MahaBERT achieving its highest accuracy of 94.7%. This indicates that longer documents provide richer context and more keyword clues, aiding the model in distinguishing between similar categories. Conversely, the Long Paragraph Classification (LPC) dataset proved the most difficult, with accuracy dropping to 86.7%. The researchers noted that individual paragraphs can often be generic or lack the specific context of the full story, leading to increased model confusion.

The Generalization Gap

Cross-dataset testing revealed that models trained on headlines (SHC) performed poorly when tested on full documents (LDC), and vice versa. This highlights that news classification is not a monolithic task; the linguistic features of a headline are fundamentally different from those of a full-length article, requiring models to adapt to specific document structures.

Conclusion

L3Cube-MahaNews provides a much-needed benchmark for Marathi NLP, moving beyond simple classification tasks to a more complex, 12-category challenge. The results underscore the superiority of monolingual models like MahaBERT and the significant impact of text length on classification accuracy. Future work will focus on manual data cleaning to remove "noise" from scraped content and exploring architectures that can better bridge the gap between short-text and long-text classification. The researchers have released the datasets and models publicly to support the Marathi NLP community.

Summary updated

BibTeX
@inproceedings{mittal2024l,
  author = {Saloni Mittal and Vidula Magdum and Sharayu Hiwarkhedkar and Omkar Dhekane and Raviraj Joshi},
  title = {{L3Cube-MahaNews: News-Based Short Text and Long Document Classification Datasets in Marathi}},
  booktitle = {Communications},
  year = {2024},
  doi = {10.1007/978-3-031-58495-4\_4},
}
Figure