TL;DR
This paper demonstrates that incorporating embeddings from auxiliary languages significantly boosts sequence tagging performance (NER and POS), but surprisingly, the most linguistically "related" language isn't always the best choice. By using attention-based meta-embeddings to dynamically weight contributions from multiple languages, the authors achieved new state-of-the-art results for part-of-speech tagging across five different languages.
What Problem Does This Paper Address?
State-of-the-art sequence tagging models—used for tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging—rely heavily on pre-trained word embeddings. While researchers have previously found that adding embeddings from a "related" language (e.g., using Portuguese to help a Spanish model) can improve performance, two major questions remained:
- Predictability: Can we use standard language distance measures (like vocabulary overlap or perplexity) to pick the best auxiliary language in advance?
- Combination: What is the most effective way to integrate these multiple sources of information? Is simple concatenation enough, or do we need something more sophisticated?
Methods
The authors use a standard BiLSTM-CRF architecture as their backbone. The core of their experimentation lies in how they represent input words by combining different embeddings.
They compare two primary methods for combining a source language embedding with one or more auxiliary language embeddings:
- Concatenation ($e^{CONCAT}$): Simply stacking the vectors together.
- Meta-Embeddings ($e^{ATT}$): Using an attention mechanism to compute a weighted sum of the embeddings. This allows the model to dynamically decide which language's representation is most useful for a given word in context.
To analyze language relationships, they tested several distance metrics, including perplexity-based measures (how well a language model trained on Language A predicts Language B) and vocabulary overlap.
Datasets
The study covers five languages: German (De), English (En), Spanish (Es), Finnish (Fi), and Dutch (Nl).
- NER: Evaluated on the CoNLL 2002/03 datasets and the FiNER corpus for Finnish.
- POS Tagging: Evaluated using the Universal Dependencies (v2.0) treebanks.
- Embeddings: They utilized both BPEmb (subword embeddings) and FLAIR (contextualized string embeddings) for all languages.
Key Insights and Ideas
- Linguistic intuition isn't a perfect guide: The most surprising finding is that the "closest" language according to linguistic metrics is not always the best auxiliary language for a specific task. For example, in Dutch NER, German was the best auxiliary language, even though Dutch is technically closer to other languages by some metrics.
- Weak Correlation: As shown in Figure 2, the correlation between language distance metrics and actual model performance is only moderate (Spearman's $\rho \approx 0.5$). This suggests that predicting the "best" helper language is harder than it looks.
- Meta-embeddings are robust: Instead of agonizing over which single language to add, the authors found that including all available languages and using an attention-based meta-embedding layer is often the superior strategy. The attention mechanism learns to ignore noise and focus on helpful signals automatically.
- Context Matters: The model assigns different weights to different languages depending on the specific word. For instance, when processing a German word in an English sentence, the attention weight for the German embedding naturally increases.
Key Results
- New State-of-the-Art: The meta-embedding approach set new SOTA records for POS tagging in all five tested languages when using FLAIR embeddings.
- Meta vs. Concatenation: Meta-embeddings provided statistically significant improvements over the best "Source + 1 Auxiliary" concatenation models in 12 out of 20 settings.
- Multilingual Synergy: Combining monolingual embeddings with multilingual ones (like the multilingual version of BPEmb) consistently outperformed using either one alone.
Limitations and Future Work
- Character Set Constraint: The current method requires at least some character overlap between languages. This means the approach, as described, cannot easily use a language with a completely different script (e.g., Chinese) to help a Latin-script language (e.g., English) without additional tools like bilingual dictionaries or machine translation.
- Computational Cost: While meta-embeddings are effective, they increase the number of parameters and the memory footprint of the model, as multiple sets of embeddings must be loaded.
- Practical Guidance: The authors provide a flowchart to help practitioners choose their strategy based on available resources.
Summary updated