Abstract: Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, Kewei Tu. Findings of the Association for Computational Linguistics: EMNLP 2020.
2020.
Introduction
In contemporary Natural Language Processing (NLP), researchers have access to a diverse array of embedding techniques, ranging from contextual sub-word representations like BERT to static word vectors like fastText. A common heuristic in the field is that concatenating these various embeddings will monotonically improve model performance. However, this "folk wisdom" has lacked a rigorous, systematic evaluation across different languages, resource levels, and domains. This paper investigates whether there is a point of diminishing returns or even performance degradation when "overdosing" on embeddings.
Methodology
The authors employ a standard BiLSTM architecture as the backbone for three primary sequence labeling tasks: Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and Chunking. They categorize embeddings into four distinct types:
- Contextual Sub-word (CSE): Multilingual BERT (M-BERT).
- Contextual Character (CCE): Flair.
- Non-contextual Word (NWE): fastText.
- Non-contextual Character (NCE): BiLSTM-encoded character sequences.
The study evaluates various combinations of these embeddings across 8 languages (including Arabic, English, Tamil, and others) and 18 datasets. Performance is measured in three scenarios: Rich-resource (full datasets), Low-resource (10 to 1000 training sentences), and Cross-domain transfer.
Analysis of Embedding Synergy
The Low-Resource Trap
A critical finding of this research is that BERT-style contextual sub-word embeddings are "sample inefficient." In extremely low-resource settings, adding M-BERT to a model can actually decrease accuracy. For instance, in a 10-sentence NER task, a combination of Flair and fastText achieved a 35.5 F1 score, which plummeted to 25.4 F1 upon the addition of M-BERT.
Complementary and Redundant Information
The experiments reveal that different embedding types provide unique inductive biases:
- Static Vectors are Anchors: Even when using powerful contextual models, adding static fastText vectors consistently improves results, likely by providing stable word-level identity that contextual models may occasionally lose.
- Character Redundancy: Non-contextual character embeddings (NCE) become largely redundant when contextual character embeddings (Flair) are present, as Flair captures similar morphological information more effectively.
- Diversity over Quantity: Stacking highly similar embeddings (e.g., M-BERT and English BERT) yields negligible gains compared to combining diverse types.
Key Results
In rich-resource settings, the general rule of "more is better" typically holds. Concatenating all four embedding types generally yielded the highest performance across tasks. As shown in the table below, the "All" combination (Row 7) achieves the highest average scores, though the marginal gain from adding non-contextual characters (NCE) to a mix of M-BERT, Flair, and fastText (Row 6) is often zero.
Conclusion
While stacking embeddings is a powerful technique for boosting sequence labeling performance, it is not a universal panacea. Practitioners must be wary of the "BERT penalty" in low-resource environments where sample inefficiency can hinder learning. Furthermore, the computational cost of high-dimensional concatenated vectors—leading to slower training and inference—must be weighed against the marginal accuracy gains, especially when using redundant embedding types.
Summary updated