UniCoM: A Universal Code-Switching Speech Generator

Sangmin Lee, Woojin Chung, Seyun Um, Hong-Goo KangEMNLP 2025

TL;DR

This paper introduces UniCoM, a novel pipeline for generating natural-sounding, intra-sentential code-switching (CS) speech across diverse languages. By developing a new algorithm called SWORDS and a comprehensive CS speech corpus (CS-FLEURS), they aim to overcome the scarcity of CS data and advance multilingual speech technologies.

What Problem Does This Paper Address?

Code-switching (CS), the practice of mixing languages within a single conversation, is common in multilingual communities but poses a significant challenge for speech technology. Existing research has largely focused on major language pairs and inter-sentential CS (switching between sentences), leaving intra-sentential CS (switching within a sentence) and CS across a wider range of languages underexplored. This gap is primarily due to the scarcity and difficulty of creating diverse CS datasets. Current methods for generating CS data either struggle with naturalness and speaker consistency or are limited in their language coverage.

Methods

The core of UniCoM is a three-stage pipeline:

  • Preprocessing: This stage cleans the input speech data by applying bandpass filtering to remove noise and amplitude normalization to ensure consistent volume levels.
  • Source Mixing (SWORDS Algorithm): This is the novel contribution. SWORDS generates intra-sentential CS by replacing words with their semantically equivalent translations from another language. It focuses on word-level substitution based on part-of-speech (POS) tags, which are more linguistically consistent across languages than phrase-level substitutions. The process involves:
  • Sampling Equivalent Sentence Pairs: Gathering parallel sentences with similar meanings.
  • Wordpair Mapping Generation: Using a large language model (LLM) like GPT-4o-mini to find equivalent word pairs and classify them by POS.
  • Segmentation Using Forced Alignment: Employing MMS-FA to align speech segments with the target words.
  • Completion Through Recombination: Rearranging the clipped segments, randomly selecting a matrix language, and substituting words from the word pairs into the matrix language utterance.
  • Style Unification: This stage uses a kNN-VC (k-nearest neighbor voice conversion) model to unify speaker styles across the generated segments, ensuring naturalness and speaker consistency.

Datasets

  • Baseline Dataset: The authors selected FLEURS-R as their baseline for constructing the new dataset. FLEURS-R is a denoised version of the FLEURS dataset, which is a 102-language multilingual speech-text dataset known for its n-way parallel sentences suitable for ASR and S2TT tasks.
  • New Dataset: CS-FLEURS: The paper introduces CS-FLEURS, a large-scale, massively multilingual CS speech corpus. It spans 253 language pairs and is constructed using the UniCoM pipeline. It includes over 70 n-way parallel sentences, offering potential for CS-S2TT research.

Key Insights and Ideas

  • Word-level substitution is key for natural intra-sentential CS: Unlike phrase-level substitution, which can disrupt sentence structure due to cross-linguistic syntactic variations, word-level substitution based on POS tags preserves naturalness and meaning more effectively.
  • LLMs are powerful for generating word-level semantic mappings: The use of GPT-4o-mini to create a dictionary of equivalent word pairs, classified by POS, is a novel and effective approach for enabling targeted substitutions.
  • A universal pipeline is achievable: UniCoM's design aims for broad applicability across a wide range of languages, moving beyond the limitations of previous language-pair-specific methods.
  • CS-FLEURS bridges a critical data gap: The creation of a large, multilingual CS corpus is a significant contribution, directly addressing the scarcity of data that has hindered research in this area.

Key Results

  • High Intelligibility and Naturalness: CS-FLEURS achieved comparable intelligibility (average RER of 31%) and naturalness (average MOS of 4.44) to human-generated CS datasets. Speaker identity consistency (average SIS of 4.74) was also high.
  • Competitive Performance: In comparisons with existing human-generated CS datasets (Spoken Wikipedia, Miami-Bangor, ASCEND, MUCS2021), CS-FLEURS demonstrated competitive or superior performance, especially in in-domain language pairs.
  • Effective for CS-ASR: Training a CS-ASR model with CS-FLEURS improved transcription performance, both as a standalone training resource and as a data augmentation method for existing datasets. For the English-German pair, using CS-FLEURS alongside SWC yielded better results than SWC alone.
  • Efficient Voice Conversion: The kNN-VC model was chosen for its effective balance of intelligibility and fast generation speed, outperforming diffusion-based alternatives in real-time factor (RTF).

Limitations and Future Work

  • Out-of-Domain Language Limitations: The performance of UniCoM is limited for out-of-domain languages in the voice conversion stage. This constrained the language span of CS-FLEURS to languages within the voice conversion model's in-domain set.
  • Orthographic Bias: CS-FLEURS primarily includes languages using the Latin script, with only a few exceptions (Bulgarian, Greek).
  • Future Work: The authors aim to expand UniCoM's scope to include languages with diverse orthographies and a wider linguistic coverage, striving for a truly universal method. They also suggest future research exploring CS-S2TT tasks using the CS-FLEURS dataset.

Summary updated

BibTeX
@inproceedings{lee2025unicom,
  author = {Sangmin Lee and Woojin Chung and Seyun Um and Hong-Goo Kang},
  title = {{UniCoM: A Universal Code-Switching Speech Generator}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure