More Embeddings, Better Sequence Labelers?

Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu

Abstract

This paper investigates the common practice of stacking multiple word embeddings in sequence labeling tasks. While generally beneficial, an extensive empirical study across 18 datasets and 8 languages reveals that "more" is not always better. In extremely low-resource settings, complex contextual embeddings can introduce noise and degrade accuracy. Furthermore, the study identifies diminishing returns when adding redundant embeddings of the same category and highlights the continued importance of static word embeddings for model grounding.

Introduction

Sequence labeling tasks, such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging, have been revolutionized by contextual embeddings like BERT and Flair. These representations capture nuanced word meanings based on surrounding context, a significant advancement over static embeddings like fastText. A common research strategy is to concatenate (stack) various embedding types, assuming that more features will inevitably lead to superior performance. This study empirically tests this assumption across different languages, domains, and resource levels.

Methods

The authors utilized a standard BiLSTM architecture with either a Softmax or Conditional Random Field (CRF) output layer. They systematically evaluated combinations of four embedding categories:

  1. Contextual Sub-word Embeddings (CSE): Multilingual BERT (M-BERT).
  2. Contextual Character Embeddings (CCE): Flair.
  3. Non-contextual Word Embeddings (NWE): fastText.
  4. Non-contextual Character Embeddings (NCE): Standard CNN or BiLSTM-based character representations.

The study spanned 18 datasets across 8 languages, simulating different resource levels by varying training set sizes from 10 sentences to full "rich-resource" datasets.

Key Insights and Results

The Low-Resource Trap

A critical finding is that in "extremely" low-resource scenarios (10–50 sentences), adding complex contextual sub-word embeddings (CSEs) like M-BERT can actually decrease accuracy compared to using Flair and fastText alone. CSEs appear to be sample-inefficient, requiring more data to provide a useful signal; in tiny datasets, they primarily introduce noise.

Figure 1: Relative scores across different training set sizes for NER, POS, and Chunking. Note the performance drop for complex combinations in low-resource settings.

As shown in Figure 1, for NER with only 10 sentences, the combination of Flair and fastText (F+W) significantly outperforms the "All" combination. This "flip" in performance highlights the risks of embedding bloat when data is scarce.

Static Embeddings and Redundancy

Despite the power of contextual models, static embeddings like fastText remain essential. They provide a stable grounding for word meanings that contextual models still find valuable. Conversely, the study found that adding redundant embeddings of the same category (e.g., adding an English-specific BERT to a model already using Multilingual BERT) rarely leads to significant gains.

Figure 4: Performance of various embedding combinations across NER, POS, and Chunking tasks.

Table 1 (Figure 4) demonstrates that while the "All" combination (Row 7) generally performs best in high-resource settings, it offers no improvement over the M+F+W combination (Row 6). This indicates that non-contextual character embeddings (C) become redundant when contextual sub-word (M) and character (F) embeddings are already present.

Conclusion

While stacking embeddings is a powerful technique for rich-resource and cross-domain sequence labeling, it is not a universal solution. Practitioners should favor simpler, more robust combinations like Flair and fastText in low-resource settings. Future work should explore more sample-efficient fusion methods and investigate whether Transformer-based labelers exhibit similar sensitivities to embedding redundancy.

Summary updated

BibTeX
@inproceedings{wang2020more,
  author = {Xinyu Wang and Yong Jiang and Nguyen Bach and Tao Wang and Zhongqiang Huang and Fei Huang and Kewei Tu},
  title = {{More Embeddings, Better Sequence Labelers?}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=S4xm61ro6xq},
}
Figure