TL;DR
This paper introduces "Any-English" (AnE), a novel system for identifying English code-switching in multilingual text. By unifying existing datasets and training a flexible model, AnE achieves state-of-the-art performance on low-resource language pairs and offers a competitive baseline for future research, bridging a significant gap in code-switching NLP.
What Problem Does This Paper Address?
Code-switching, the practice of alternating between languages within a single conversation or utterance, is increasingly prevalent, especially on social media. However, existing Natural Language Processing (NLP) tools struggle with this phenomenon, often performing poorly even on large language models. A key bottleneck is the lack of fine-grained, multilingual language identification systems that can accurately distinguish between languages at a token level within code-switched text. Current language identification systems are typically language-pair specific, making it difficult to research code-switching in low-resource languages where annotated data is scarce. This creates a circular problem: we need better language identification to create more data, but we need more data to train better language identification systems.
Methods
The authors propose an "Any-English" (AnE) system, which is an ensemble of two classifiers:
- Code-Switching Identification (AnELID): This classifier distinguishes between English, non-English languages, words that mix English and another language within the word ("Mixed"), and other tokens like punctuation or emojis ("Other").
- Binary Named Entity Recognition (AnENER): This classifier identifies whether a word is part of a named entity or not.
To achieve this, they adapt and unify existing code-switching corpora. For the AnELID classifier, they create a unified labelset that includes "English," "notEnglish," "Mixed," and "Other." For the AnENER classifier, they create a binary distinction for named entities. The AnE system then combines these two classifiers, using the AnENER output to refine the AnELID predictions. They also explore a zero-shot evaluation approach for low-resource languages, where the AnE system is evaluated without explicit fine-tuning on code-switching data for those specific language pairs.
Datasets
The paper utilizes several existing code-switching corpora, primarily focusing on those involving English and other languages. These include:
- High-resource language pairs: Hindi-English, Spanish-English, Nepali-English (from the LinCE benchmark), and German-English (Denglisch corpus).
- Low-resource language pairs: Indonesian-English, Turkish-English, and Vietnamese-English.
The authors preprocess these corpora to match their unified labelset for training. They also note that the Vietnamese-English corpus is unique as it's derived from spoken conversations and is "silver-standard" data.
Key Insights and Ideas
- Unified approach for diverse corpora: The core innovation is the ability to train a single system that can handle code-switching across various language pairs involving English, by harmonizing disparate annotation schemes.
- AnE as a versatile tool: The AnE system demonstrates strong performance not only on high-resource pairs but also excels in zero-shot evaluations on low-resource pairs, suggesting its potential as a general-purpose tool for code-switching research.
- Surprising correlation between lexical similarity and distinguishability: Counter-intuitively, the study found a negative correlation between lexical similarity and English recall. This suggests that lexically dissimilar languages are easier to distinguish, which the authors hypothesize might be due to the monolingual pre-training of the underlying language model.
Key Results
- High-resource pairs: The AnE system achieves overall F₁ scores very close to, and sometimes slightly below, language-pair-specific state-of-the-art (SoTA) baselines, with differences typically less than 1% absolute F₁. For example, for Hindi-English, the AnE F₁ is 98.40% compared to a baseline of 98.43%.
- Low-resource pairs (zero-shot): AnE significantly outperforms existing SoTA fine-tuned models on Indonesian-English (93.45% vs. 88.86% F₁) and Turkish-English (97.9% vs. 95.6% F₁).
- Named Entity Recognition (NER): The AnENER component performs comparably to baseline NER systems, indicating its effectiveness as a complementary module.
- Lexical Similarity: A significant negative Spearman's correlation (p=-0.82) was found between lexical distance and English recall, suggesting lexically similar languages are easier to distinguish.
Limitations and Future Work
- English-centric: The current AnE system is designed to identify English code-switching. It does not support code-switching between other language pairs (e.g., Romance vs. non-Romance languages).
- Data availability: The primary motivation for focusing on English was the availability of suitable high-resource code-switching data. The authors acknowledge that similar systems for other language families are currently hindered by a lack of comparable data.
- Vietnamese-English "Other" category: The AnE system's performance on the Vietnamese-English corpus is affected by a mismatch in the "Other" category, which in that corpus includes language-neutral words rather than just punctuation or emojis.
- Future Work: The authors suggest exploring code-switching between language families beyond English and investigating the surprising correlation between lexical similarity and distinguishability further. They also highlight the potential of AnE to facilitate the compilation of new, large-scale code-switching corpora for underrepresented language pairs.
Summary updated