Much Gracias: Semi-supervised Code-switch Detection for Spanish-English: How far can we get?

Dana-Maria Iliescu, Rasmus Grand, Sara Qirko, Rob van der GootNAACL 2021

TL;DR

This paper explores semi-supervised methods for detecting language switches within Spanish-English text, leveraging readily available monolingual data instead of expensive annotated code-switched datasets. The Viterbi decoding model achieved the best performance among the semi-supervised approaches, demonstrating the potential of these methods for broader language pair applicability.

What Problem Does This Paper Address?

Globalization has led to an increase in code-switching, where individuals mix multiple languages within a single utterance. This phenomenon presents a significant challenge for Natural Language Processing (NLP) systems, as it introduces unique linguistic structures. Existing language identification models for code-switched data are typically supervised, requiring large amounts of annotated training data. However, such annotated datasets are scarce, especially for many language pairs. This paper aims to bridge this gap by investigating semi-supervised approaches that utilize out-of-domain monolingual data, making code-switch detection more accessible for a wider range of languages. The core research question is how semi-supervised models compare to each other and to supervised models in detecting language switches in English-Spanish data.

Methods

The authors experiment with a variety of semi-supervised methods, all leveraging monolingual Wikipedia data for training:

  • Word uni-grams and n-grams: These models calculate word probabilities based on frequency in the monolingual data, with Laplace smoothing applied.
  • Character n-grams: This approach uses character sequences within words to estimate language probabilities.
  • Viterbi Decoding: This method frames code-switching as a Hidden Markov Model (HMM) problem, where languages are hidden states. It uses the Viterbi algorithm to find the most probable sequence of language labels for a given sequence of words. Emission probabilities are derived from word dictionaries, with character n-grams used as a fallback.
  • Latent Dirichlet Allocation (LDA): LDA is adapted to find topics (languages) within words, using character n-grams as features.
  • Support Vector Machine (SVM) and Logistic Regression: These traditional machine learning models are used with TF-IDF vectorized character n-gram features extracted from words.
  • Ensemble Model: The authors also explore combining the predictions of multiple models using majority voting and an "oracle" approach (which always picks the correct label if available from any model).

Datasets

  • Test Data: The Spanish-English (SPA-EN) portion of the LinCE benchmark dataset is used for evaluation. This dataset consists of 32,651 posts. For this study, only three labels are considered: lang1 (English), lang2 (Spanish), and other (punctuation, numbers, symbols, emoticons).
  • Monolingual Data: Wikipedia dumps from September 1st, 2020, are used as the source of monolingual data for training. After cleaning, the English dataset contains approximately 420K distinct words, and the Spanish dataset contains about 610K distinct words.

Key Insights and Ideas

  • Semi-supervised is promising: The Viterbi decoding model, a semi-supervised approach, achieves a respectable weighted F1 score of 92.23% on the test set, demonstrating that significant performance can be attained without extensive manual annotation.
  • Viterbi shines: Among the tested semi-supervised methods, Viterbi decoding emerged as the top performer, highlighting its effectiveness in modeling sequential language transitions.
  • Monolingual data is a valuable resource: The study successfully leverages readily available monolingual data (Wikipedia) to train models for code-switch detection, circumventing the need for costly annotated code-switched datasets.
  • Ensembling has potential, but needs refinement: While majority voting ensembling didn't improve performance, the "oracle" ensemble achieved a near state-of-the-art score, suggesting that smarter ensemble strategies could further boost performance.
  • The "other" class is a challenge: The Viterbi model shows more confusion with the "other" class compared to the supervised BERT model, indicating that accurately classifying non-linguistic tokens remains an area for improvement.

Key Results

  • The Viterbi decoding model achieved the highest weighted F1 score among the semi-supervised models, scoring 95.76% on validation data and 92.23% on test data.
  • A supervised BERT-based model (MaChAmp) achieved a state-of-the-art weighted F1 score of 98.43% on the test data.
  • The oracle ensemble model reached 98.47% on validation data, indicating the potential for combining multiple models.
  • The authors note that their semi-supervised models, while promising, still underperform compared to the supervised BERT model.
  • The Viterbi model is significantly faster to train than the supervised MaChAmp model, completing in 1,805 seconds compared to MaChAmp's 53,990 seconds.

Limitations and Future Work

  • Domain Mismatch: The study acknowledges a domain difference between the monolingual Wikipedia training data and the social media-like test data (LinCE). Collecting monolingual data directly from social media platforms is noted as non-trivial.
  • Monolingual Data Purity: Wikipedia datasets are not perfectly monolingual, which could negatively impact performance.
  • Class Imbalance and Confusion: The paper notes that performance varies across language classes (English vs. Spanish) and that the "other" class presents a significant challenge for accurate classification. Investigating discrepancies in label distribution between datasets is suggested for future work.
  • Ensemble Strategy: The majority voting ensemble did not yield improvements, suggesting the need for more sophisticated ensemble methods.
  • Data Size for LDA: The authors suggest that the relatively small amount of data used for LDA might have contributed to its low performance.
  • Future Directions: The authors propose exploring larger monolingual datasets, corpora more similar to the test set (e.g., social media data), and more complex methods for handling the "other" class. The potential to train CS-detection for many more language pairs using these semi-supervised methods is highlighted.

Summary updated

BibTeX
@inproceedings{iliescu2021much,
  author = {Dana-Maria Iliescu and Rasmus Grand and Sara Qirko and Rob van der Goot},
  title = {{Much Gracias: Semi-supervised Code-switch Detection for Spanish-English: How far can we get?}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure