Abstract: Genta Indra Winata, Zhaojiang Lin, Jamin Shin, Zihan Liu, Pascale Fung. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Code-switching—the practice of alternating between two or more languages within a single conversation—is a prevalent linguistic phenomenon in multilingual communities. However, it presents a significant challenge for standard Natural Language Processing (NLP) systems. Named Entity Recognition (NER) is particularly difficult in these contexts because entities often act as the "anchor" points where speakers switch languages.
Traditional NER approaches typically rely on word-level embeddings for specific languages. These methods often fail to capture shared morphological structures across languages, such as the shared root between the English "possible" and Spanish "posible." This paper introduces Hierarchical Meta-Embeddings (HME), a framework designed to effectively combine word, subword, and character-level representations across multiple languages into a single, language-agnostic vector.
The Hierarchical Meta-Embedding (HME) Framework
The HME architecture is designed to fuse information from diverse linguistic levels without requiring explicit language identifiers. The system consists of three primary components integrated into a sequence labeling pipeline.
1. Multilingual Meta-Embeddings (MME)
Instead of selecting a single language's embedding, the model incorporates pre-trained embeddings from several languages. A projection matrix maps these into a shared space, and an attention mechanism learns to weight the most relevant language embedding for each specific word.
2. Subword and Character Mapping
To handle varying segmentation across languages, the authors utilize Byte-Pair Encodings (BPEs). A Transformer encoder maps these subword sequences into fixed-length vectors. This is combined with character-level encodings to capture fine-grained orthographic features.
3. Hierarchical Fusion and Sequence Labeling
The final word representation is a concatenation of word-level MME, subword-level MME, and character-level features. These fused representations are fed into a Transformer-CRF (Conditional Random Field) architecture to predict entity tags (e.g., B-LOCATION).
Experimental Results
The model was evaluated on the English-Spanish code-switching tweets dataset from the 2018 CALCS shared task. The HME model achieved a state-of-the-art F1 score of 67.8 (reaching 69.17 with an ensemble), a significant improvement over previous benchmarks that hovered around 62-63.
Key findings include:
- Subword Importance: Incorporating BPE-level features consistently provided a boost of ~1-2 F1 points, suggesting that subwords act as the "glue" between languages with shared roots.
- Cross-Lingual Robustness: In a zero-shot setting where the model was trained on Romance and Germanic languages but tested on English-Spanish data, it still achieved a 64.52 F1 score, outperforming many models that had access to the target languages.
Analysis: Implicit Language Detection
A critical advantage of the HME framework is its ability to perform implicit language detection. By visualizing the attention weights, the researchers demonstrated that the model naturally learns to attend to the correct language features based on the input text.
As shown in the attention heatmaps, the model correctly identifies Spanish words like "tengo sueño" by assigning higher weights to Spanish (ES) embeddings, while switching focus to English (EN) or GloVe embeddings for English segments. This behavior emerges purely from data without explicit language labels.
Conclusion
Hierarchical Meta-Embeddings provide a robust solution for code-switching NER by leveraging multi-level linguistic features. By effectively fusing word, subword, and character information, the model achieves state-of-the-art performance and demonstrates remarkable generalization capabilities across different language families.
Summary updated