Abstract
Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this simple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consistently underperforms models trained on monolingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to this problem, in which polyglot models are fine-tuned on monolingual data to consistently and significantly outperform their monolingual counterparts. To explain this phenomena, we explore the sources of multilingual transfer in polyglot NER models and examine the weight structure of polyglot models compared to their monolingual counterparts. We find that polyglot models efficiently share many parameters across languages and that fine-tuning may utilize a large number of those parameters.
Introduction
Named Entity Recognition (NER) is a fundamental NLP task, but annotating data for every language is expensive and time-consuming. While "polyglot" learning—training one model on a mix of many languages—promises to share knowledge across linguistic boundaries, these models often paradoxically underperform specialized monolingual models. This research identifies that the gap is primarily an optimization issue rather than a lack of learned knowledge. By introducing a simple fine-tuning step on target-language data, polyglot models can consistently outperform monolingual baselines.
Methodology
The authors evaluate several neural NER architectures, including Word-Level CRFs, CharNER, Byte-to-Span (BTS), and Multilingual BERT (mBERT). The core technique introduced is Target Language Polyglot Adaptation, where a polyglot model is used as an initialization for fine-tuning on monolingual data. To probe the internal behavior of these models, the researchers use Magnitude Weight Pruning to measure parameter efficiency and the Fisher Information Matrix to identify language-specific critical weights.
Mechanisms of Multilingual Transfer
The paper highlights three key reasons why fine-tuned polyglot models achieve superior performance:
1. Overcoming the Optimization Bottleneck
The initial underperformance of joint training is due to the complexity of the optimization landscape when handling multiple languages simultaneously. Fine-tuning acts as a local search that finds the optimal performance within the multilingual space already mapped out during joint training.
2. Extreme Parameter Efficiency
Polyglot models are remarkably lean. Analysis shows that a single polyglot model for four languages requires only about 30% of the total parameters that four separate monolingual models would need to achieve the same performance. This high degree of sharing proves that languages possess significant underlying structural commonalities.
3. The Common Entity Advantage
A major source of transfer is the overlap of entities across languages. Names like "Apple" or "Paris" appear in multiple contexts across the training set. Polyglot models exhibit lower error rates on these "common entities" because they have been exposed to them more frequently and in more diverse linguistic settings than monolingual models.
Key Results
- Performance Boost: Fine-tuning a polyglot model consistently outperformed monolingual baselines, often by up to 3 $F_1$ points.
- mBERT Dominance: While mBERT remained the strongest performer overall, byte-level models were surprisingly competitive in low-resource settings (LORELEI).
- Transfer to Unseen Languages: Transfer is most effective when there is script or family overlap, such as Hindi models transferring knowledge to Bengali.
Conclusion
Fine-tuning is the key to unlocking the latent potential of polyglot NER models. While this requires maintaining language-specific versions of a model, the gains in accuracy and the massive parameter efficiency make it a superior approach to training from scratch. Future work will explore multi-objective optimization to achieve these gains within a single, unified model.
Summary updated