Abstract
Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this simple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consistently underperforms models trained on monolingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to this problem, in which polyglot models are fine-tuned on monolingual data to consistently and significantly outperform their monolingual counterparts. To explain this phenomena, we explore the sources of multilingual transfer in polyglot NER models and examine the weight structure of polyglot models compared to their monolingual counterparts. We find that polyglot models efficiently share many parameters across languages and that fine-tuning may utilize a large number of those parameters.
Introduction
Named Entity Recognition (NER) is a fundamental task in natural language processing, yet it faces a significant data scarcity challenge. While high-resource languages like English benefit from extensive labeled datasets, the vast majority of the world's 7,000+ languages lack sufficient annotated corpora.
Polyglot learning—training a single model on a mixture of multiple languages—offers a theoretical solution by allowing models to learn universal entity patterns. However, in practice, naive polyglot models often underperform compared to specialized monolingual models. This research addresses this performance gap by proposing a two-stage training regime and investigating the underlying mechanisms that enable successful cross-lingual transfer.
Methodology
The study evaluates four distinct neural NER architectures: Word-level CRF (BiLSTM-CRF), CharNER (character-level inference), Byte-to-Span (BTS), and Multilingual BERT (mBERT). The researchers propose a two-stage training approach:
- Polyglot Pre-training: Training a model on a concatenated dataset of multiple languages.
- Language-Specific Fine-tuning: Using the polyglot model as an initialization for further training on a specific target language.
To analyze these models, the authors utilized Magnitude Weight Pruning to assess parameter efficiency and the Fisher Information Matrix to identify language-specific weight importance. Experiments were conducted on the CoNLL benchmark (English, German, Dutch, Spanish) and the LORELEI benchmark (10 diverse, low-resource languages).
Analysis of Multilingual Transfer
The "Common Entity" Advantage
A primary source of transfer in polyglot models is the presence of "common entities"—names like "London" or "Apple" that appear across multiple linguistic corpora. Polyglot models benefit from increased exposure to these entities during the pre-training phase. Analysis shows that polyglot models, particularly after fine-tuning, achieve significantly lower error rates on these shared entities compared to monolingual baselines.
Parameter Efficiency and Weight Sharing
One of the most striking findings is the extreme parameter efficiency of polyglot models. The research demonstrates that a single polyglot model requires only approximately 30% of the total parameters that four separate monolingual models would need to reach equivalent performance levels. This suggests that different languages share a vast amount of underlying structural information within the model's weights.
Further analysis using the Fisher Information Matrix reveals that while the top 5% of weights are often specialized for a specific language, the overlap in "important" weights increases significantly as more parameters are considered (e.g., the top 25%).
Key Results
Fine-tuned polyglot models consistently outperformed monolingual baselines, frequently yielding improvements of up to 3 F1 points. While transformer-based models like mBERT showed strong results, simpler architectures like Byte-CRFs also saw significant gains from the polyglot-then-fine-tune approach.
In zero-shot experiments on novel languages (e.g., Yoruba or Bengali), the polyglot models provided a superior starting point for learning. Even when zero-shot performance was initially low, fine-tuning from a polyglot initialization led to much faster and more effective adaptation than starting from scratch.
Conclusion
The failure of naive polyglot models is likely due to an optimization bottleneck rather than a lack of transferable knowledge. By utilizing a two-stage fine-tuning process, researchers can unlock the latent benefits of multilingual training. Future work may explore multi-objective optimization to achieve these gains without requiring separate fine-tuned models for every language, or utilize techniques like Elastic Weight Consolidation (EWC) to preserve multilingual knowledge during adaptation.
Summary updated