Abstract
Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), yet its application to low-resource languages remains a significant challenge. This study benchmarks various BERT-based transformer models for Hindi and Marathi, two major Indian languages written in the Devanagari script. By evaluating both monolingual and multilingual architectures, the research establishes new performance baselines and explores the efficacy of cross-language generalization. Findings reveal that while specialized monolingual models like MahaRoBERTa lead in Marathi NER, large-scale multilingual models such as XLM-RoBERTa remain superior for Hindi, highlighting a disparity in current monolingual model development for these languages.
Introduction
Named Entity Recognition—the process of identifying and categorizing entities such as persons, locations, and organizations—is a cornerstone of modern information extraction. While English NER has reached high levels of maturity, languages like Hindi and Marathi present unique linguistic and resource-based hurdles. Researchers from the Pune Institute of Computer Technology and IIT Madras address this gap by providing a comprehensive benchmark of transformer-based models, comparing the strengths of specialized monolingual training against broad multilingual pre-training.
Linguistic Challenges in Devanagari NER
Developing NER systems for Hindi and Marathi is complicated by several factors that distinguish them from Western languages:
- Lack of Capitalization: Unlike English, where capitalization serves as a primary signal for proper nouns, the Devanagari script does not distinguish between upper and lower case. This forces models to rely entirely on semantic and syntactic context.
- Equivocation: These languages frequently use common nouns as proper names (e.g., a word meaning "flower" or "light" functioning as a person's name). Distinguishing these requires deep contextual understanding.
- Resource Scarcity: Despite being spoken by hundreds of millions, Hindi and Marathi lack the high-quality, "gold-standard" annotated datasets and standardized toolsets available for English. Existing datasets are often smaller, noisier, or imbalanced.
Methodology
The study focused on the Transformer architecture, specifically variations of the Bidirectional Encoder Representations from Transformers (BERT) model. The evaluated models were categorized into two distinct groups:
Multilingual Models
These models are pre-trained on massive, multi-language corpora:
- mBERT: Trained on 104 languages.
- XLM-RoBERTa: A robustly optimized approach trained on 100 languages.
- IndicBERT: A model specifically targeting 12 major Indian languages, designed to capture regional linguistic nuances.
Monolingual Models
These models are pre-trained specifically on a single target language:
- Marathi: The MahaBERT suite, including MahaBERT, MahaRoBERTa, and MahaAlBERT.
- Hindi: Various RoBERTa-based models trained exclusively on Hindi corpora.
In addition to standard fine-tuning, the researchers conducted cross-language evaluation to determine if models trained on one Devanagari language could generalize to the other, leveraging their shared script and structural similarities.
Datasets and Experimental Setup
The researchers utilized several datasets, noting the necessity of significant data cleaning to address inconsistent tagging:
- Hindi Datasets: The IJCNLP 2008 dataset (featuring 12 complex entity categories) and the WikiAnn "silver-standard" dataset (3 categories: Person, Organization, Location).
- Marathi Datasets: The IIT Bombay Marathi NER Corpus and the Marathi subset of WikiAnn.
Notably, the Hindi training sets were approximately double the size of the Marathi sets, a factor that influenced the comparative performance of the models.
Results and Discussion
The benchmarking process yielded several key insights into the behavior of transformer models in low-resource settings:
The Monolingual Advantage in Marathi
For Marathi, the language-specific MahaRoBERTa model outperformed all multilingual alternatives. It achieved a peak F1 score of 64.34 on the IIT Bombay corpus and 88.90 on WikiAnn. This suggests that for certain low-resource languages, specialized pre-training on local corpora captures nuances that broad multilingual models fail to resolve.
The Hindi Paradox
Conversely, Hindi results favored the multilingual approach. XLM-RoBERTa led the Hindi benchmarks, reaching an F1 of 75.90 on the IJCNLP dataset and 83.04 on WikiAnn. The authors hypothesize that this is due to the maturity of the models: while the Marathi monolingual models were fine-tuned on top of existing robust architectures, many current monolingual Hindi models were trained from scratch with less optimization.
Asymmetrical Generalization
Cross-language tests revealed an interesting asymmetry: Marathi models generalized surprisingly well when tested on Hindi data. However, Hindi models performed poorly on Marathi. This indicates that the Marathi models may be capturing more robust, script-level features that are transferable across the Devanagari family, whereas the Hindi models may be more overfit to specific lexical patterns.
Conclusion
This research successfully establishes reproducible baselines for Hindi and Marathi NER, demonstrating that even "silver-standard" data can yield high accuracy (90%+) when paired with the correct transformer architecture. The study highlights a clear need for more sophisticated monolingual Hindi models and the development of larger "gold-standard" annotated datasets to further advance Indian language NLP. Future work will investigate the underlying reasons for Marathi's superior cross-lingual generalization and seek to bridge the performance gap between monolingual and multilingual models in the Hindi context.
Summary updated