Abstract
Code-switching (CS)—the practice of alternating between two or more languages in a single conversation—is a linguistic norm for multilingual populations. However, Natural Language Processing (NLP) tools often fail to handle CS due to the "one-document-one-language" assumption and a lack of high-quality annotated data. This paper introduces a novel, user-centric approach to building high-density CS corpora from Twitter for Spanish-English and Nepali-English language pairs. By identifying frequent code-switchers rather than searching for specific keywords, we created a more natural dataset. We demonstrate that while word-level language identification is maturing, detecting code-switching at the tweet level remains a significant challenge for state-of-the-art models.
1. Introduction
The primary bottleneck in improving NLP tools for multilingual populations is the scarcity of high-quality, annotated code-switching data. Previous collection efforts often relied on keyword searches (e.g., searching for "Spanglish"), which tend to yield "self-conscious" or unnatural data. Furthermore, existing datasets often exhibit a low density of actual code-switching, sometimes as low as 11%, hindering the training of robust machine learning models. This work addresses these issues through a user-centric collection strategy and rigorous token-level annotation.
2. Methodology
We moved away from keyword-based filtering toward a user-centric strategy. Frequent code-switchers were identified by observing users who naturally toggled between languages in their timelines. For Nepali-English, a "snowball" approach was used starting from known bilingual users. For Spanish-English, geographical filters (California and Texas) were cross-referenced with frequent bilingual vocabulary.
Token-level annotation was performed via crowdsourcing. To ensure quality, we employed a "gold standard" system where expert-annotated tweets were used to filter out unreliable workers. Our universal tagging schema includes Lang1, Lang2, Ambiguous (cross-lingual homographs), Mixed (intra-word switching), NE (Named Entities), and Other (URLs, emojis, etc.).
3. Dataset Composition
The study produced two primary corpora: a Spanish-English (Es-En) set of ~14,400 tweets and a Nepali-English (Ne-En) set of ~12,800 tweets. Notably, the Nepali data is written in Romanized script, reflecting actual social media usage despite the lack of a standardized spelling system.
As shown in the tag distribution, the datasets capture a variety of linguistic elements, including a significant portion of "Other" tokens and a small but critical number of "Mixed" and "Ambiguous" tokens that represent the core challenges of code-switching identification.
4. Experimental Results
We evaluated two benchmarks: a simple Dictionary-based approach and a Conditional Random Field (CRF) model with Generalized Expectation. At the word level, the CRF model performed well, achieving 86% accuracy for Spanish-English and 87% for Nepali-English.
However, tweet-level detection—identifying if a tweet contains any code-switching—proved surprisingly difficult. For Nepali-English, the simple Dictionary approach significantly outperformed the CRF model in accuracy and recall.
This suggests that the CRF relies too heavily on long monolingual contexts and fails to detect switches occurring at a fine-grained level. Additionally, lexical overlap analysis revealed that Spanish and English share more n-gram overlaps than Nepali and English, making the Spanish-English pair a more challenging task for language identification due to increased ambiguity.
5. Conclusion and Future Work
This work provides a foundation for more sophisticated CS analysis. Future research should move beyond identification to explore the syntax of code-switching. Addressing the lack of standardized Romanized spelling for languages like Nepali may require character-level modeling. Furthermore, analyzing entire user timelines rather than isolated tweets could improve prediction accuracy by leveraging individual linguistic patterns and historical context.
Summary updated