Methods of Automatic Matrix Language Determination for Code-Switched Speech

Olga Iakovenko, Thomas HainEMNLP 2024 · DOI

Abstract

In code-switched (CS) speech, where speakers alternate between languages, identifying the "Matrix Language" (ML)—the language providing the grammatical framework—is critical for accurate linguistic analysis and speech recognition. This paper presents automated methods for ML determination based on Matrix Language Frame (MLF) theory. By implementing principles such as singleton insertions, token order fluency, and function word analysis, the authors develop both text-based and acoustic predictors. Experimental results on English-Mandarin and English-Spanish corpora show that these ML-focused approaches significantly outperform traditional Language Identification (LID) systems in identifying the structural driver of mixed utterances.

Introduction

Code-switching is a pervasive linguistic phenomenon where speakers transition between two or more languages within a single conversation or sentence. While common in multilingual societies, it poses a significant challenge for modern Natural Language Processing (NLP) and Automatic Speech Recognition (ASR) systems. Traditional Language Identification (LID) systems typically treat CS as a competition between languages, often defaulting to the language with the most words or the strongest acoustic presence. However, this approach ignores the underlying grammatical hierarchy of the utterance.

The Matrix Language Frame

Linguistic research, specifically Matrix Language Frame (MLF) theory, suggests that code-switched sentences are not random mixtures. Instead, one language—the Matrix Language—provides the morphosyntactic frame, while the other—the Embedded Language—contributes specific insertions. Identifying the ML is essential for understanding the syntax of the speaker. This paper, presented at EMNLP 2024, bridges the gap between theoretical linguistics and computational efficiency by proposing automated methods to determine the "grammatical boss" from text and audio.

Methodology

The researchers implemented three core principles derived from MLF theory to identify the Matrix Language:

  1. The Singleton Principle (P1.1): This identifies the ML by detecting "singleton insertions"—single words from one language embedded within a sequence of another. For example, in a Mandarin sentence containing the English word "blog," Mandarin is identified as the ML because it provides the surrounding grammatical context.
  2. The Token Order Principle (P1.2): This novel approach uses Neural Machine Translation (NMT) to translate a CS utterance into two monolingual versions. Language Models (LMs) then evaluate the "naturalness" or fluency of each version's word order. The language resulting in the more likely word order is designated as the ML.
  3. The System Word Principle (P2): This focuses on "function words" (e.g., determiners, auxiliaries, conjunctions), which MLF theory posits must originate from the Matrix Language. The system employs Part-of-Speech (POS) tagging to determine which language provides these grammatical "glue" words.
Workflow for determining the Matrix Language using the Token Order Principle (P1.2).

To extend these principles to speech, the authors developed Acoustic MLID. They trained a Multi-Layer Perceptron (MLP) on features extracted from a pre-trained LID system (ECAPA-TDNN) to predict these textual ML principles directly from audio signals.

Experimental Results

The study utilized the SEAME (English-Mandarin) and Miami (English-Spanish) corpora. A key finding was that the acoustically dominant language is not always the grammatical driver. While traditional LID often favored English in these datasets, MLID analysis revealed that Mandarin and Spanish frequently served as the structural backbone.

Comparison of Matthews Correlation Coefficient (MCC) across different MLID predictors and standard LID baselines.

As shown in the performance metrics, predictors trained on ML principles (MLID_P1.1, MLID_P1.2, MLID_P2) consistently showed higher correlations with linguistic ground truth compared to standard LID. On a manually annotated subset of the Miami data, the audio-based MLID predictors achieved an F1 macro score of 60% and a correlation score of 0.38, significantly outperforming traditional methods.

Conclusion

This research demonstrates that automated systems can successfully incorporate complex linguistic principles to identify the Matrix Language in code-switched speech. By moving beyond simple word counts to structural analysis, these methods provide a more nuanced understanding of multilingual communication. Future work will focus on integrating these MLID components into end-to-end ASR systems to improve recognition accuracy in code-switched contexts.

Summary updated

BibTeX
@inproceedings{iakovenko2024methods,
  author = {Olga Iakovenko and Thomas Hain},
  title = {{Methods of Automatic Matrix Language Determination for Code-Switched Speech}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.330},
}
Figure