On the Choice of Auxiliary Languages for Improved Sequence Tagging

Lukas Lange, Heike Adel, Jannik Strötgen

TL;DR

This paper demonstrates that incorporating embeddings from auxiliary languages significantly boosts sequence tagging performance (NER and POS), but surprisingly, the most linguistically "related" language isn't always the best choice. By using attention-based meta-embeddings to dynamically weight contributions from multiple languages, the authors achieved new state-of-the-art results for part-of-speech tagging across five different languages.

What Problem Does This Paper Address?

State-of-the-art sequence tagging models—used for tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging—rely heavily on pre-trained word embeddings. While researchers have previously found that adding embeddings from a "related" language (e.g., using Portuguese to help a Spanish model) can improve performance, two major questions remained:

  1. Predictability: Can we use standard language distance measures (like vocabulary overlap or perplexity) to pick the best auxiliary language in advance?
  2. Combination: What is the most effective way to integrate these multiple sources of information? Is simple concatenation enough, or do we need something more sophisticated?

Methods

The authors use a standard BiLSTM-CRF architecture as their backbone. The core of their experimentation lies in how they represent input words by combining different embeddings.

They compare two primary methods for combining a source language embedding with one or more auxiliary language embeddings:

  • Concatenation ($e^{CONCAT}$): Simply stacking the vectors together.
  • Meta-Embeddings ($e^{ATT}$): Using an attention mechanism to compute a weighted sum of the embeddings. This allows the model to dynamically decide which language's representation is most useful for a given word in context.
Overview of the model architecture and embedding combination methods.

To analyze language relationships, they tested several distance metrics, including perplexity-based measures (how well a language model trained on Language A predicts Language B) and vocabulary overlap.

Datasets

The study covers five languages: German (De), English (En), Spanish (Es), Finnish (Fi), and Dutch (Nl).

  • NER: Evaluated on the CoNLL 2002/03 datasets and the FiNER corpus for Finnish.
  • POS Tagging: Evaluated using the Universal Dependencies (v2.0) treebanks.
  • Embeddings: They utilized both BPEmb (subword embeddings) and FLAIR (contextualized string embeddings) for all languages.

Key Insights and Ideas

  • Linguistic intuition isn't a perfect guide: The most surprising finding is that the "closest" language according to linguistic metrics is not always the best auxiliary language for a specific task. For example, in Dutch NER, German was the best auxiliary language, even though Dutch is technically closer to other languages by some metrics.
  • Weak Correlation: As shown in Figure 2, the correlation between language distance metrics and actual model performance is only moderate (Spearman's $\rho \approx 0.5$). This suggests that predicting the "best" helper language is harder than it looks.
Figure 2: Spearman’s rank correlation between lan- guage distance and model performance rankings for NER and POS tasks for different language distances.
  • Meta-embeddings are robust: Instead of agonizing over which single language to add, the authors found that including all available languages and using an attention-based meta-embedding layer is often the superior strategy. The attention mechanism learns to ignore noise and focus on helpful signals automatically.
  • Context Matters: The model assigns different weights to different languages depending on the specific word. For instance, when processing a German word in an English sentence, the attention weight for the German embedding naturally increases.

Key Results

  • New State-of-the-Art: The meta-embedding approach set new SOTA records for POS tagging in all five tested languages when using FLAIR embeddings.
  • Meta vs. Concatenation: Meta-embeddings provided statistically significant improvements over the best "Source + 1 Auxiliary" concatenation models in 12 out of 20 settings.
  • Multilingual Synergy: Combining monolingual embeddings with multilingual ones (like the multilingual version of BPEmb) consistently outperformed using either one alone.

Limitations and Future Work

  • Character Set Constraint: The current method requires at least some character overlap between languages. This means the approach, as described, cannot easily use a language with a completely different script (e.g., Chinese) to help a Latin-script language (e.g., English) without additional tools like bilingual dictionaries or machine translation.
  • Computational Cost: While meta-embeddings are effective, they increase the number of parameters and the memory footprint of the model, as multiple sets of embeddings must be loaded.
  • Practical Guidance: The authors provide a flowchart to help practitioners choose their strategy based on available resources.
Figure 3: Proposal for auxiliary embedding selection.

Summary updated

Abstract

Recent work showed that embeddings from related languages can improve the performance of sequence tagging, even for monolingual models. In this analysis paper, we investigate whether the best auxiliary language can be predicted based on language distances and show that the most related language is not always the best auxiliary language. Further, we show that attention-based meta-embeddings can effectively combine pre-trained embeddings from different languages for sequence tagging and set new state-of-the-art results for part-of-speech tagging in five languages.
BibTeX
@inproceedings{lange2020choice,
  author = {Lukas Lange and Heike Adel and Jannik Strötgen},
  title = {{On the Choice of Auxiliary Languages for Improved Sequence Tagging}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}

Loading related content...

Figure