Sources of Transfer in Multilingual Named Entity Recognition

David Mueller, Nicholas Andrews, Mark Dredze

Abstract

Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this simple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consistently underperforms models trained on monolingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to this problem, in which polyglot models are fine-tuned on monolingual data to consistently and significantly outperform their monolingual counterparts. To explain this phenomena, we explore the sources of multilingual transfer in polyglot NER models and examine the weight structure of polyglot models compared to their monolingual counterparts. We find that polyglot models efficiently share many parameters across languages and that fine-tuning may utilize a large number of those parameters.

Introduction

Named Entity Recognition (NER) is a fundamental NLP task, but annotating data for every language is expensive and time-consuming. While "polyglot" learning—training one model on a mix of many languages—promises to share knowledge across linguistic boundaries, these models often paradoxically underperform specialized monolingual models. This research identifies that the gap is primarily an optimization issue rather than a lack of learned knowledge. By introducing a simple fine-tuning step on target-language data, polyglot models can consistently outperform monolingual baselines.

Methodology

The authors evaluate several neural NER architectures, including Word-Level CRFs, CharNER, Byte-to-Span (BTS), and Multilingual BERT (mBERT). The core technique introduced is Target Language Polyglot Adaptation, where a polyglot model is used as an initialization for fine-tuning on monolingual data. To probe the internal behavior of these models, the researchers use Magnitude Weight Pruning to measure parameter efficiency and the Fisher Information Matrix to identify language-specific critical weights.

Mechanisms of Multilingual Transfer

The paper highlights three key reasons why fine-tuned polyglot models achieve superior performance:

1. Overcoming the Optimization Bottleneck

The initial underperformance of joint training is due to the complexity of the optimization landscape when handling multiple languages simultaneously. Fine-tuning acts as a local search that finds the optimal performance within the multilingual space already mapped out during joint training.

2. Extreme Parameter Efficiency

Polyglot models are remarkably lean. Analysis shows that a single polyglot model for four languages requires only about 30% of the total parameters that four separate monolingual models would need to achieve the same performance. This high degree of sharing proves that languages possess significant underlying structural commonalities.

Pruning thresholds for Byte CRF models on CoNLL, demonstrating the high parameter efficiency of polyglot models.

3. The Common Entity Advantage

A major source of transfer is the overlap of entities across languages. Names like "Apple" or "Paris" appear in multiple contexts across the training set. Polyglot models exhibit lower error rates on these "common entities" because they have been exposed to them more frequently and in more diverse linguistic settings than monolingual models.

Error rates on common entities across languages for monolingual, polyglot, and fine-tuned polyglot models.

Key Results

  • Performance Boost: Fine-tuning a polyglot model consistently outperformed monolingual baselines, often by up to 3 $F_1$ points.
  • mBERT Dominance: While mBERT remained the strongest performer overall, byte-level models were surprisingly competitive in low-resource settings (LORELEI).
  • Transfer to Unseen Languages: Transfer is most effective when there is script or family overlap, such as Hindi models transferring knowledge to Bengali.

Conclusion

Fine-tuning is the key to unlocking the latent potential of polyglot NER models. While this requires maintaining language-specific versions of a model, the gains in accuracy and the massive parameter efficiency make it a superior approach to training from scratch. Future work will explore multi-objective optimization to achieve these gains within a single, unified model.

Summary updated

Abstract

Named-entities are inherently multilingual, and annotations in any given language may be limited. This motivates us to consider polyglot named-entity recognition (NER), where one model is trained using annotated data drawn from more than one language. However, a straightforward implementation of this simple idea does not always work in practice: naive training of NER models using annotated data drawn from multiple languages consistently underperforms models trained on monolingual data alone, despite having access to more training data. The starting point of this paper is a simple solution to this problem, in which polyglot models are fine-tuned on monolingual data to consistently and significantly outperform their monolingual counterparts. To explain this phenomena, we explore the sources of multilingual transfer in polyglot NER models and examine the weight structure of polyglot models compared to their monolingual counterparts. We find that polyglot models efficiently share many parameters across languages and that fine-tuning may utilize a large number of those parameters.
BibTeX
@inproceedings{mueller2020sources,
  author = {David Mueller and Nicholas Andrews and Mark Dredze},
  title = {{Sources of Transfer in Multilingual Named Entity Recognition}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  url = {https://openreview.net/forum?id=CXZAXSjv3p5},
}
Figure