Abstract
Recently, neural methods have achieved state-of-the-art (SOTA) results in Named Entity Recognition (NER) tasks for many languages without the need for manually crafted features. However, these models still require manually annotated training data, which is not available for many languages. In this paper, we propose an unsupervised cross-lingual NER model that can transfer NER knowledge from one language to another in a completely unsupervised way without relying on any bilingual dictionary or parallel data. Our model achieves this through word-level adversarial learning and augmented fine-tuning with parameter sharing and feature augmentation. Experiments on five different languages demonstrate the effectiveness of our approach, outperforming existing models by a good margin and setting a new SOTA for each language pair.
Introduction
Named Entity Recognition (NER) is a fundamental task in Natural Language Processing (NLP), identifying names of people, places, and organizations. While deep learning has significantly advanced NER accuracy, these models are notoriously data-hungry. For the vast majority of the world’s 7,000+ languages, manually labeled datasets do not exist. Traditional cross-lingual transfer methods often rely on "crutches" like bilingual dictionaries or parallel corpora. This research addresses the "Zero-Resource" scenario: training an NER model for a target language using only raw text and a labeled dataset from a source language (e.g., English).
Methodology
The proposed framework utilizes a multi-stage pipeline to bridge the gap between source and target languages without explicit translation resources.
1. Word-level Adversarial Mapping
The model learns to map word embeddings from the source and target languages into a shared vector space. Using an adversarial approach similar to the MUSE framework, a generator attempts to align embeddings via rotation or flipping, while a discriminator tries to distinguish between the two languages.
2. Source Model Pre-training with Weight Sharing
A Bi-LSTM-CRF model is trained on English data. To enhance transferability, the weights of the forward and backward LSTMs are bound. This forces the model to learn order-invariant features, making it less sensitive to the specific syntax of English.
3. Augmented Fine-tuning
This stage involves three critical components:
- Pseudo-labeling: The English-trained model predicts labels on raw target language text.
- Stochastic Selection: A curriculum-based approach where the model initially trains on shorter, more reliably tagged sentences.
- Joint Training: The model is trained simultaneously on original source data and noisy pseudo-labeled target data to prevent "catastrophic forgetting."
Experimental Results
The model was evaluated by transferring knowledge from English (CoNLL-2003) to Spanish, Dutch, German, Arabic, and Finnish. These languages represent diverse families and morphological complexities.
The proposed "Cross-Augmented" model significantly outperformed previous unsupervised SOTA methods:
- Spanish & Dutch: Improved F1 scores by 2.43 and 2.21 points, respectively.
- German: Saw a substantial 6.14 point F1 improvement.
- Arabic: Achieved an F1 of 36.91, a significant breakthrough for this language pair in an unsupervised setting.
- Finnish: Improved F1 score by 5.73 points.
Discussion and Key Insights
A core finding of this work is the relationship between sentence length and tagging accuracy. As shown in Figure 2, shorter sentences yield a higher ratio of correctly tagged entities. This observation justifies the stochastic selection strategy, allowing the model to bootstrap its learning from simpler examples before tackling complex syntax.
Furthermore, the research highlights that while character embeddings assist languages with similar scripts (like English and Spanish), they can introduce noise for languages with different scripts (like Arabic). The "Gap" theory, inspired by Second-Language Acquisition, suggests that models learn best by using the source language as a scaffold to identify and bridge knowledge gaps in the target language.
Conclusion
This paper presents a robust framework for zero-resource cross-lingual NER. Despite its success, challenges remain, particularly regarding word-order sensitivity in languages like Arabic and capitalization quirks in German. Future work may involve integrating large-scale pre-trained multilingual language models to further enhance performance in zero-resource settings.
Reference: Bari, M. S., Joty, S., & Jwalapuram, P. (2020). Zero-Resource Cross-Lingual Named Entity Recognition. Proceedings of the AAAI Conference on Artificial Intelligence.
Summary updated