Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition

Genta Indra Winata, Zhaojiang Lin, Jamin Shin, Zihan Liu, Pascale Fung

Abstract: Genta Indra Winata, Zhaojiang Lin, Jamin Shin, Zihan Liu, Pascale Fung. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Code-switching—the practice of alternating between two or more languages within a single conversation—is a prevalent linguistic phenomenon in multilingual communities. However, it presents a significant challenge for standard Natural Language Processing (NLP) systems. Named Entity Recognition (NER) is particularly difficult in these contexts because entities often act as the "anchor" points where speakers switch languages.

Traditional NER approaches typically rely on word-level embeddings for specific languages. These methods often fail to capture shared morphological structures across languages, such as the shared root between the English "possible" and Spanish "posible." This paper introduces Hierarchical Meta-Embeddings (HME), a framework designed to effectively combine word, subword, and character-level representations across multiple languages into a single, language-agnostic vector.

The Hierarchical Meta-Embedding (HME) Framework

The HME architecture is designed to fuse information from diverse linguistic levels without requiring explicit language identifiers. The system consists of three primary components integrated into a sequence labeling pipeline.

The HME architecture (right) integrated into a Transformer-CRF sequence labeling pipeline (left).

1. Multilingual Meta-Embeddings (MME)

Instead of selecting a single language's embedding, the model incorporates pre-trained embeddings from several languages. A projection matrix maps these into a shared space, and an attention mechanism learns to weight the most relevant language embedding for each specific word.

2. Subword and Character Mapping

To handle varying segmentation across languages, the authors utilize Byte-Pair Encodings (BPEs). A Transformer encoder maps these subword sequences into fixed-length vectors. This is combined with character-level encodings to capture fine-grained orthographic features.

3. Hierarchical Fusion and Sequence Labeling

The final word representation is a concatenation of word-level MME, subword-level MME, and character-level features. These fused representations are fed into a Transformer-CRF (Conditional Random Field) architecture to predict entity tags (e.g., B-LOCATION).

Experimental Results

The model was evaluated on the English-Spanish code-switching tweets dataset from the 2018 CALCS shared task. The HME model achieved a state-of-the-art F1 score of 67.8 (reaching 69.17 with an ensemble), a significant improvement over previous benchmarks that hovered around 62-63.

Key findings include:

  • Subword Importance: Incorporating BPE-level features consistently provided a boost of ~1-2 F1 points, suggesting that subwords act as the "glue" between languages with shared roots.
  • Cross-Lingual Robustness: In a zero-shot setting where the model was trained on Romance and Germanic languages but tested on English-Spanish data, it still achieved a 64.52 F1 score, outperforming many models that had access to the target languages.

Analysis: Implicit Language Detection

A critical advantage of the HME framework is its ability to perform implicit language detection. By visualizing the attention weights, the researchers demonstrated that the model naturally learns to attend to the correct language features based on the input text.

Visualization of attention weights for word-level (left) and BPE-level (right) embeddings across a code-switched sentence.

As shown in the attention heatmaps, the model correctly identifies Spanish words like "tengo sueño" by assigning higher weights to Spanish (ES) embeddings, while switching focus to English (EN) or GloVe embeddings for English segments. This behavior emerges purely from data without explicit language labels.

Conclusion

Hierarchical Meta-Embeddings provide a robust solution for code-switching NER by leveraging multi-level linguistic features. By effectively fusing word, subword, and character information, the model achieves state-of-the-art performance and demonstrates remarkable generalization capabilities across different language families.

Summary updated

BibTeX
@inproceedings{winata2019hierarchical,
  author = {Genta Indra Winata and Zhaojiang Lin and Jamin Shin and Zihan Liu and Pascale Fung},
  title = {{Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=FfeNlRMbU1L},
}
Figure