Abstract
Linguistic Code Switching (CS) occurs when multilingual speakers alternate between two or more languages or dialects within a single conversation. Processing CS data is challenging for state-of-the-art monolingual NLP technologies, which are typically designed for one language at a time. This paper addresses Part-of-Speech (POS) tagging in the context of CS by leveraging various neural network architectures and pre-trained embedding methods. We investigate four language pairs: Spanish-English (SPA-ENG), Hindi-English (HIN-ENG), Modern Standard Arabic-Egyptian (MSA-EGY), and Modern Standard Arabic-Levantine (MSA-LEV). Our results demonstrate that multilingual embeddings significantly benefit closely related dialects but can introduce noise for linguistically distant pairs. Notably, our proposed models achieve state-of-the-art performance for the MSA-EGY language pair.
Introduction
Most Natural Language Processing (NLP) tools assume a monolingual input. However, multilingualism is a global norm, and speakers frequently engage in Code-Switching (CS). Standard POS taggers often fail on CS data due to out-of-vocabulary (OOV) words, structural ambiguities in hybrid grammars, and shared cognates that function differently across languages. Given the scarcity of annotated CS data, this research explores using large-scale unlabeled text to create robust word representations (embeddings) that capture the nuances of mixed-language environments.
Methodology
The study evaluates three neural architectures and several embedding strategies based on FastText.
Architectures
- BiLSTM-CRF: A standard sequence labeling model combining a Bidirectional Long Short-Term Memory network with a Conditional Random Field layer.
- MTL-POS: A Multi-Task Learning model that learns POS tags for related language pairs simultaneously.
- MTL-POS+LID: A multi-task model that predicts both POS tags and Language Identifiers (LID) for each token.
Embedding Strategies
- Monolingual (Baseline): Embeddings trained on single languages.
- Combined Filtered Monolingual (CFM): Trained on a mix of monolingual data to handle inter-sentential switching.
- Pure Code Switched (PCS): Trained exclusively on code-switched data to capture intra-sentential dynamics.
- PseudoCS: A hybrid approach combining CFM and PCS data to maximize vocabulary and capture how words interact across language boundaries.
- Projected Bilingual: Aligning separate monolingual spaces into a shared space using the MUSE tool.
Experimental Setup
The models were tested on four datasets representing varying linguistic distances:
- MSA-EGY: Modern Standard Arabic and Egyptian Arabic (LDC ARZ1-5).
- MSA-LEV: Modern Standard Arabic and Levantine Arabic (Curras Corpus).
- SPA-ENG: Spanish-English (Miami Bangor Corpus).
- HIN-ENG: Hindi-English (UD-HIN-ENG Twitter data).
Results and Discussion
The experimental results highlight the importance of linguistic proximity. Multilingual embeddings provided the most substantial gains for closely related dialects (Arabic pairs), where shared homographs and cognates are prevalent.
As shown in the results, the PseudoCS strategy was consistently the most effective, as it captures cross-language interactions while maintaining a broad vocabulary. For the MSA-EGY pair, the proposed model reached 92.90% accuracy, establishing a new state-of-the-art by outperforming previous systems by approximately 2%. For SPA-ENG, the model achieved a competitive 96.55%. However, the HIN-ENG pair (86.01%) remained challenging, likely due to the linguistic distance and different scripts involved, which can introduce noise into the shared embedding space.
Error Analysis
An analysis of the tagging errors reveals common patterns across the language pairs. In the Arabic dialects, the most frequent error is misidentifying Adjectives as Nouns (19-21%). In the Hindi-English data, a significant portion of errors involves misclassifying Nouns as Verbs (27%).
These errors often stem from shared cognates or short code-switched fragments that lack sufficient context for disambiguation. For instance, the specialized embeddings helped reduce the OOV rate for MSA-EGY from 10% to 8%, which directly improved tagging accuracy, yet short fragments (1-2 words) remain a persistent challenge for the classifiers.
Conclusion
This research demonstrates that specialized pre-trained embeddings, particularly the PseudoCS approach, can significantly improve POS tagging for code-switched data. While the method is highly effective for closely related dialects, further refinement is needed for linguistically distant pairs. Future work will focus on more sophisticated alignment techniques and applying these embeddings to other downstream tasks like dependency parsing and text segmentation in code-switched environments.
Summary updated