On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi

Harsh Chaudhari, Anuja Patil, Dhanashree Lavekar, Pranav Khairnar, Raviraj Joshi, Sachin PandearXiv 2023 · DOI · arXiv

Abstract

Named Entity Recognition (NER) is a fundamental task in Natural Language Processing, yet it remains challenging for morphologically rich, low-resource languages like Marathi. While state-of-the-art Transformer models offer high accuracy, their computational demands are often prohibitive for resource-constrained environments. This paper investigates a hybrid approach that combines the efficiency of shallow neural networks—such as Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) networks—with the linguistic advantages of subword tokenization derived from pre-trained models like MahaBERT and IndicBERT. Our findings demonstrate that simply replacing word-based tokenizers with subword-based ones significantly improves performance, closing the gap between lightweight models and heavy-duty Transformers.

Introduction

Named Entity Recognition (NER) involves identifying and classifying key entities—such as names of people, organizations, and locations—within a text. For Marathi, an Indo-Aryan language spoken by over 80 million people, NER is complicated by two primary factors: resource scarcity and morphological complexity. As a morphologically rich language, Marathi words frequently change form through prefixes and suffixes, leading to a high rate of "Out-of-Vocabulary" (OOV) words when using traditional word-based tokenizers.

While deep Transformer-based models (e.g., BERT) handle these complexities well, they require significant computational power and memory. Conversely, shallow architectures like CNNs or LSTMs are computationally efficient but typically struggle with the linguistic nuances of Marathi. This study proposes a middle ground: leveraging the sophisticated vocabulary breakdown of subword tokenizers within the framework of efficient, shallow neural architectures.

Methodology

The core of the proposed approach is a Hybrid Model that decouples the tokenizer from the full Transformer architecture. Instead of processing text as space-separated words, the model utilizes subword tokenization to break complex Marathi words into smaller, meaningful units (sub-tokens).

Subword Tokenization and Label Management

The researchers utilized tokenizers from pre-trained models, specifically MahaBERT (monolingual) and IndicBERT (multilingual). Subword tokenization mitigates the OOV problem by ensuring that even unseen words can be decomposed into known sub-units (e.g., a complex verb form might be split into its root and its suffix).

To maintain the integrity of NER labels during this process, the authors implemented a label-alignment strategy. During training, if a single word is split into multiple sub-tokens, the original entity label is assigned to every sub-token. During inference, these sub-tokens are aggregated back into the original word format to provide a final entity prediction.

Shallow Architectures

The study evaluated three standard shallow architectures to process these sub-tokens:

  1. 1D CNN: Utilizing 512 filters to extract local features.
  2. LSTM: A single-layer Long Short-Term Memory network for sequence modeling.
  3. BiLSTM: A bidirectional LSTM to capture context from both preceding and succeeding tokens.

All models utilized 300-dimensional embeddings as input, providing a lightweight alternative to the 768+ dimensions typically found in Transformer models.

Experimental Setup

The models were trained and evaluated using the L3Cube-MahaNER dataset, the first gold-standard NER corpus for Marathi. The dataset consists of 25,000 sentences sourced from news data, categorized into eight target entity types. The data was split into 21,500 sentences for training, 1,500 for validation, and 2,000 for testing.

Results and Discussion

The experimental results highlight the critical role of tokenization in NER performance. The MahaBERT tokenizer paired with a CNN emerged as the most effective hybrid combination.

Performance Gains

  • F1-Score Improvement: The hybrid CNN model achieved an 82.1% F1-score, representing a significant increase over the 79.5% achieved by the standard word-based CNN.
  • Efficiency vs. Accuracy: While the full MahaBERT model reaches an F1-score of 86.8%, the hybrid approach recovers approximately 50% of the performance gap between shallow models and state-of-the-art Transformers, while remaining significantly faster and less resource-intensive.
  • Architecture Insights: Interestingly, the CNN generally outperformed the LSTM variants when paired with subword tokenizers. This suggests that local feature extraction is highly effective when the input tokens are already morphologically descriptive.

Monolingual Advantage

The study found that tokenizers from monolingual models (MahaBERT) consistently outperformed multilingual ones (IndicBERT). This indicates that language-specific subword patterns are vital for capturing the unique morphological structures of Marathi.

Conclusion and Future Work

This research demonstrates that subword tokenization is a "secret sauce" for improving NER in morphologically rich, low-resource languages. By integrating subword tokenizers into shallow neural networks, developers can create NER systems that are both accurate and efficient enough for deployment on edge devices or low-power servers.

Future work will focus on closing the remaining 4.7% performance gap between hybrid models and full Transformers, potentially by exploring hybrid architectures that better capture long-range dependencies. Additionally, the authors plan to extend this methodology to other Indian languages to verify if these benefits are universal across the subcontinent's diverse linguistic landscape.

Summary updated

BibTeX
@misc{chaudhari2023significance,
  author = {Harsh Chaudhari and Anuja Patil and Dhanashree Lavekar and Pranav Khairnar and Raviraj Joshi and Sachin Pande},
  title = {{On Significance of Subword tokenization for Low Resource and Efficient Named Entity Recognition: A case study in Marathi}},
  year = {2023},
  eprint = {2312.01306},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.01306},
}
Figure