Overview for the First Shared Task on Language Identification in Code-Switched Data

Thamar Solorio, Elizabeth Blair, Suraj Maharjan, Steven Bethard, Mona T. Diab, Mahmoud Ghoneim, Abdelati Hawwari, Fahad AlGhamdi, Julia Hirschberg, Alison Chang, Pascale FungCodeSwitch @ EMNLP 2014 · OpenReview

Introduction

Code-switching (CS)—the practice of alternating between two or more languages in a single conversation or text—is ubiquitous in social media and spoken communication. However, most Natural Language Processing (NLP) pipelines assume a single language per document. Standard Language Identification (LID) tools often fail when faced with "intrasentential" code-switching (switching within a sentence). To build effective translation, sentiment analysis, or parsing tools for the modern web, we first need to accurately label every single token in a CS string.

The First Shared Task on Language Identification in Code-Switched Data, held in 2014, was a landmark event that established benchmarks and provided new datasets for identifying language at the token level in multilingual social media text.

Datasets and Task Setup

The task introduced several newly annotated datasets primarily sourced from Twitter, covering four diverse language pairs. These pairs represent different levels of linguistic distance, which significantly impacts the difficulty of the task.

Examples of code-switched text for different language pairs.

The data was labeled with six categories: lang1, lang2, other (punctuation/numbers), ambiguous, mixed (words containing morphemes from both languages), and NE (Named Entities). Additionally, a "Surprise Genre" dataset (Facebook comments and blogs) was provided to test how well systems could generalize beyond Twitter.

Methodology

Seven teams submitted 42 different system runs, utilizing a variety of approaches:

  • Algorithms: Most teams used sequence labeling models like Conditional Random Fields (CRF), though some employed Support Vector Machines (SVM) or k-Nearest Neighbors (kNN).
  • Features: The most effective systems relied on character n-grams, casing information, and contextual features (using the labels of surrounding words to predict the current one).
  • Resources: Several systems integrated external "gazetteers" (lists of names), morphological analyzers, and unsupervised data crawled from the web.

Key Results and Insights

The difficulty of LID is directly proportional to the linguistic distance between the languages. Distinguishing Nepali from English (different scripts and families) is much easier than distinguishing Modern Standard Arabic from Egyptian Arabic, which share significant vocabulary and phonology.

Performance (F-measure) of participating systems across the four language pairs, highlighting the difficulty of the MSA-DA pair.

As shown in the results, systems performed exceptionally well on NEP-EN and SPA-EN, with top performers achieving weighted F-measures of 0.959 and 0.940, respectively. In MAN-EN, the difference in scripts (Chinese characters vs. Latin alphabet) made the task easier, with most systems reaching an F-measure above 0.80.

However, the MSA-DA pair proved the most difficult by a wide margin. The best system reached only an F-measure of 0.417 for identifying CS instances in one of the test sets, barely outperforming the baseline. This reveals that identifying closely related variants remains a formidable challenge for current NLP models.

Challenges and Limitations

Several factors contributed to system errors and task complexity:

  • Named Entities (NEs): Proper nouns are a major source of error. They often don't follow the phonotactics of either language and are frequently borrowed, leading systems to misclassify the surrounding language.
  • Minority Classes: Systems struggled to correctly identify "minority" classes like ambiguous or mixed tokens, which are rare in training data but crucial for understanding the mechanics of code-switching.
  • Genre Sensitivity: Systems tuned for Twitter saw a performance drop of roughly 10% when tested on the "Surprise Genre" (Facebook/blogs), highlighting that LID models are highly sensitive to the specific slang and formatting of a platform.

Conclusion

The First Shared Task provided a critical foundation for code-switching research. While performance is high for typologically distinct languages, closely related dialects require more sophisticated modeling. Future work aims to move beyond identification toward parsing code-switched data and exploring the socio-pragmatic reasons why users switch languages at specific points.

Summary updated

BibTeX
@inproceedings{solorio2014overview,
  author = {Thamar Solorio and Elizabeth Blair and Suraj Maharjan and Steven Bethard and Mona T. Diab and Mahmoud Ghoneim and Abdelati Hawwari and Fahad AlGhamdi and Julia Hirschberg and Alison Chang and Pascale Fung},
  title = {{Overview for the First Shared Task on Language Identification in Code-Switched Data}},
  booktitle = {CodeSwitch @ EMNLP},
  year = {2014},
  url = {https://openreview.net/forum?id=UawP9RtF08},
}
Figure