Abstract
Named Entity Recognition (NER) is a fundamental NLP task, yet most languages lack the annotated data required for modern neural models. This paper introduces a novel unsupervised framework for cross-lingual NER that requires zero labeled data, parallel corpora, or bilingual dictionaries for the target language. By integrating adversarial word mapping with an "augmented fine-tuning" strategy—featuring weight sharing for order invariance and stochastic sentence-length sampling—the model achieves state-of-the-art performance across five diverse language pairs.
Introduction
While neural models have achieved high performance in NER for high-resource languages like English, they remain heavily dependent on large-scale manual annotations. For the vast majority of the world's languages, such resources are unavailable. Previous cross-lingual transfer methods often rely on "crutches" such as bilingual dictionaries or parallel bi-texts, which are also scarce for low-resource languages. This work addresses the "zero-resource" challenge, transferring NER capabilities from a source language (English) to a target language using only unlabeled text.
Methodology
The proposed framework employs a multi-stage pipeline designed for deep task transfer beyond simple word translation.
1. Word-level Adversarial Mapping
Using an unsupervised approach similar to the MUSE framework, the model learns to map word embeddings from source and target languages into a shared vector space. A discriminator is employed to align the distributions, followed by refinement using the Procrustes solution.
2. Shared Character Embeddings
To leverage morphological similarities such as common prefixes or suffixes, the model utilizes a shared character embedding matrix across languages.
3. Augmented Fine-tuning
The core innovation, "augmented fine-tuning," consists of three primary components:
- Weight Sharing for Order Invariance: Forward and backward LSTM layer weights are bound during source model training. This encourages the learning of features less dependent on specific syntax, facilitating transfer to languages with different word orders.
- Pseudo-Label Generation: The source model, operating in the shared embedding space, generates initial tags for unlabeled target language sentences.
- Joint Training with Stochastic Sampling: To mitigate noise in pseudo-labels, the authors implement a stochastic schedule. Recognizing that shorter sentences are tagged more accurately, the model is initially trained on shorter sentences to bootstrap the learning process.
Experimental Setup
The model was trained using English (CoNLL-2003) as the source language. Evaluation was conducted on five target languages: Spanish and Dutch (CoNLL-2002), German (CoNLL-2003), Arabic (AQMAR), and Finnish (Ruokolainen et al., 2019). Notably, validation was performed using only the source-language development set, ensuring a truly zero-resource evaluation.
Results and Discussion
The proposed model significantly outperformed previous unsupervised state-of-the-art (SOTA) models across all tested languages. Improvements in F1 scores ranged from 2.21 points for Dutch to over 6 points for German and Finnish. Even for Arabic, which uses a different script, the model achieved a notable F1 of 36.91.
A key insight from the experiments is the "short sentence advantage." As shown in the figure above, there is a clear negative correlation between sentence length and tagging accuracy in zero-shot settings. By focusing on shorter, more reliable sentences during early fine-tuning, the model avoids the confusion introduced by the noisy labels of complex, long sentences. Furthermore, the weight-sharing constraint in the LSTMs effectively prevented overfitting to English-specific word order, resulting in more universal feature representations.
Conclusion
This work demonstrates that effective cross-lingual NER transfer is possible without any bilingual supervision. While character embeddings can introduce noise for languages with different alphabets, the combination of adversarial mapping and robust fine-tuning strategies provides a powerful baseline. Future research may explore the integration of pre-trained multilingual language models like mBERT to capture even deeper linguistic nuances.
Summary updated