Abstract
This paper investigates the common practice of stacking multiple word embeddings in sequence labeling tasks. While generally beneficial, an extensive empirical study across 18 datasets and 8 languages reveals that "more" is not always better. In extremely low-resource settings, complex contextual embeddings can introduce noise and degrade accuracy. Furthermore, the study identifies diminishing returns when adding redundant embeddings of the same category and highlights the continued importance of static word embeddings for model grounding.
Introduction
Sequence labeling tasks, such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging, have been revolutionized by contextual embeddings like BERT and Flair. These representations capture nuanced word meanings based on surrounding context, a significant advancement over static embeddings like fastText. A common research strategy is to concatenate (stack) various embedding types, assuming that more features will inevitably lead to superior performance. This study empirically tests this assumption across different languages, domains, and resource levels.
Methods
The authors utilized a standard BiLSTM architecture with either a Softmax or Conditional Random Field (CRF) output layer. They systematically evaluated combinations of four embedding categories:
- Contextual Sub-word Embeddings (CSE): Multilingual BERT (M-BERT).
- Contextual Character Embeddings (CCE): Flair.
- Non-contextual Word Embeddings (NWE): fastText.
- Non-contextual Character Embeddings (NCE): Standard CNN or BiLSTM-based character representations.
The study spanned 18 datasets across 8 languages, simulating different resource levels by varying training set sizes from 10 sentences to full "rich-resource" datasets.
Key Insights and Results
The Low-Resource Trap
A critical finding is that in "extremely" low-resource scenarios (10–50 sentences), adding complex contextual sub-word embeddings (CSEs) like M-BERT can actually decrease accuracy compared to using Flair and fastText alone. CSEs appear to be sample-inefficient, requiring more data to provide a useful signal; in tiny datasets, they primarily introduce noise.
As shown in Figure 1, for NER with only 10 sentences, the combination of Flair and fastText (F+W) significantly outperforms the "All" combination. This "flip" in performance highlights the risks of embedding bloat when data is scarce.
Static Embeddings and Redundancy
Despite the power of contextual models, static embeddings like fastText remain essential. They provide a stable grounding for word meanings that contextual models still find valuable. Conversely, the study found that adding redundant embeddings of the same category (e.g., adding an English-specific BERT to a model already using Multilingual BERT) rarely leads to significant gains.
Table 1 (Figure 4) demonstrates that while the "All" combination (Row 7) generally performs best in high-resource settings, it offers no improvement over the M+F+W combination (Row 6). This indicates that non-contextual character embeddings (C) become redundant when contextual sub-word (M) and character (F) embeddings are already present.
Conclusion
While stacking embeddings is a powerful technique for rich-resource and cross-domain sequence labeling, it is not a universal solution. Practitioners should favor simpler, more robust combinations like Flair and fastText in low-resource settings. Future work should explore more sample-efficient fusion methods and investigate whether Transformer-based labelers exhibit similar sensitivities to embedding redundancy.
Summary updated