Developing Language-tagged Corpora for Code-switching Tweets

Suraj Maharjan, Elizabeth Blair, Steven Bethard, Thamar SolorioLAW @ NAACL 2015 · DOI

Abstract

Code-switching (CS)—the practice of alternating between two or more languages in a single conversation—is a linguistic norm for multilingual populations. However, Natural Language Processing (NLP) tools often fail to handle CS due to the "one-document-one-language" assumption and a lack of high-quality annotated data. This paper introduces a novel, user-centric approach to building high-density CS corpora from Twitter for Spanish-English and Nepali-English language pairs. By identifying frequent code-switchers rather than searching for specific keywords, we created a more natural dataset. We demonstrate that while word-level language identification is maturing, detecting code-switching at the tweet level remains a significant challenge for state-of-the-art models.

1. Introduction

The primary bottleneck in improving NLP tools for multilingual populations is the scarcity of high-quality, annotated code-switching data. Previous collection efforts often relied on keyword searches (e.g., searching for "Spanglish"), which tend to yield "self-conscious" or unnatural data. Furthermore, existing datasets often exhibit a low density of actual code-switching, sometimes as low as 11%, hindering the training of robust machine learning models. This work addresses these issues through a user-centric collection strategy and rigorous token-level annotation.

2. Methodology

We moved away from keyword-based filtering toward a user-centric strategy. Frequent code-switchers were identified by observing users who naturally toggled between languages in their timelines. For Nepali-English, a "snowball" approach was used starting from known bilingual users. For Spanish-English, geographical filters (California and Texas) were cross-referenced with frequent bilingual vocabulary.

Token-level annotation was performed via crowdsourcing. To ensure quality, we employed a "gold standard" system where expert-annotated tweets were used to filter out unreliable workers. Our universal tagging schema includes Lang1, Lang2, Ambiguous (cross-lingual homographs), Mixed (intra-word switching), NE (Named Entities), and Other (URLs, emojis, etc.).

3. Dataset Composition

The study produced two primary corpora: a Spanish-English (Es-En) set of ~14,400 tweets and a Nepali-English (Ne-En) set of ~12,800 tweets. Notably, the Nepali data is written in Romanized script, reflecting actual social media usage despite the lack of a standardized spelling system.

Distribution of language tags across the training and test sets for both language pairs.

As shown in the tag distribution, the datasets capture a variety of linguistic elements, including a significant portion of "Other" tokens and a small but critical number of "Mixed" and "Ambiguous" tokens that represent the core challenges of code-switching identification.

4. Experimental Results

We evaluated two benchmarks: a simple Dictionary-based approach and a Conditional Random Field (CRF) model with Generalized Expectation. At the word level, the CRF model performed well, achieving 86% accuracy for Spanish-English and 87% for Nepali-English.

However, tweet-level detection—identifying if a tweet contains any code-switching—proved surprisingly difficult. For Nepali-English, the simple Dictionary approach significantly outperformed the CRF model in accuracy and recall.

Performance metrics for tweet-level code-switching detection in the Nepali-English corpus, highlighting the discrepancy between Dictionary and CRF models.

This suggests that the CRF relies too heavily on long monolingual contexts and fails to detect switches occurring at a fine-grained level. Additionally, lexical overlap analysis revealed that Spanish and English share more n-gram overlaps than Nepali and English, making the Spanish-English pair a more challenging task for language identification due to increased ambiguity.

5. Conclusion and Future Work

This work provides a foundation for more sophisticated CS analysis. Future research should move beyond identification to explore the syntax of code-switching. Addressing the lack of standardized Romanized spelling for languages like Nepali may require character-level modeling. Furthermore, analyzing entire user timelines rather than isolated tweets could improve prediction accuracy by leveraging individual linguistic patterns and historical context.

Summary updated

BibTeX
@inproceedings{maharjan2015developing,
  author = {Suraj Maharjan and Elizabeth Blair and Steven Bethard and Thamar Solorio},
  title = {{Developing Language-tagged Corpora for Code-switching Tweets}},
  booktitle = {LAW @ NAACL},
  year = {2015},
  doi = {10.3115/v1/w15-1608},
}
Figure