Multilingual Code-switching Identification via LSTM Recurrent Neural Networks

Younes Samih, Suraj Maharjan, Mohammed Attia, Laura Kallmeyer, Thamar SolorioCodeSwitch @ EMNLP 2016 · DOI

Introduction

Code-switching—the practice of alternating between multiple languages or dialects within a single conversation—is a common phenomenon in globalized social media environments. For Natural Language Processing (NLP) systems, this presents a significant challenge, as standard tools typically assume a monolingual input. Traditional approaches to code-switching identification relied heavily on hand-crafted features such as dictionaries and part-of-speech taggers, which are costly to develop and often fail to capture the informal nature of social media text. This paper presents a unified neural network architecture designed to identify language transitions without extensive language-specific linguistic tools.

Methodology

The proposed system, designated as HHU-UH-G, utilizes a deep learning approach to minimize manual feature engineering. The architecture integrates both word-level and character-level information to capture semantic and morphological patterns.

The HHU-UH-G system architecture, featuring parallel word and character LSTM layers followed by a concatenation and a CRF output layer.

Dual-Input LSTM Architecture

The core of the model consists of two parallel Long Short-Term Memory (LSTM) networks. One branch processes word embeddings to capture broad semantic meaning, while the other processes character embeddings. The character-level representation is a critical innovation, allowing the model to learn internal word structures (e.g., prefixes and suffixes) that distinguish languages. This capability is particularly effective for handling out-of-vocabulary (OOV) words, such as slang or typos common on social media.

Sequence Modeling with CRF

To account for the "stickiness" of language—the tendency for speakers to stay in one language for several tokens before switching—the system employs a Conditional Random Field (CRF) as the final output layer. Unlike a standard Softmax layer that predicts labels independently, the CRF considers the sequence of labels, statistically favoring transitions that align with observed language patterns.

Experimental Setup

The system was evaluated on datasets from the EMNLP 2016 Second Workshop on Computational Approaches to Code Switching, focusing on two language pairs:

  • Spanish-English: 11,446 training/dev tweets and over 10,000 test tweets.
  • Arabic (MSA-Egyptian): Approximately 10,000 tweets mixing Modern Standard Arabic and Egyptian Dialect.

Word embeddings were initialized using large-scale external corpora, including the English/Spanish Gigaword and a 383-million-word Arabic dataset comprising news articles and Facebook posts.

Results and Discussion

The HHU-UH-G system achieved top-tier performance in the EMNLP 2016 shared task. For the Arabic (MSA-Egyptian) task, the system ranked first with an F1-score of 0.83. In the Spanish-English task, it ranked second with an F1-score of 0.90 and a token-level accuracy of 96.7%.

Latent Space Visualization

To understand how the model distinguishes languages, the authors used Principal Component Analysis (PCA) to project internal representations into a 2D space.

PCA visualizations of the model's latent space for Spanish-English (top) and MSA-Egyptian (bottom), showing distinct clustering by language.

As shown in the visualization above, the model naturally clusters words by language. Tokens labeled as "ambiguous" or "mixed" typically reside in the transition zones between these clusters, demonstrating the model's ability to learn meaningful linguistic boundaries.

Limitations

Despite its success, the system faces challenges with "mixed" tokens—single words containing elements of both languages (e.g., an English root with a Spanish suffix)—due to their rarity in training data. Additionally, identifying Named Entities (NE) remains difficult, as the model occasionally mislabels proper nouns as belonging to one of the primary languages.

Conclusion

The HHU-UH-G system demonstrates that a deep learning architecture combining word and character-level LSTMs can effectively identify code-switching across diverse language pairs. By moving away from manual feature engineering, the model provides a robust, language-independent solution for processing multilingual social media text. Future work will focus on improving Named Entity recognition and handling rare mixed-language tokens.

Summary updated

BibTeX
@inproceedings{samih2016multilingual,
  author = {Younes Samih and Suraj Maharjan and Mohammed Attia and Laura Kallmeyer and Thamar Solorio},
  title = {{Multilingual Code-switching Identification via LSTM Recurrent Neural Networks}},
  booktitle = {CodeSwitch @ EMNLP},
  year = {2016},
  doi = {10.18653/v1/w16-5806},
}
Figure