Speech collage: code-switched audio generation by collaging monolingual corpora

Amir Hussein, Dorsa Zeinali, Ondrej Klejch, Matthew Wiesner, Brian Yan, Shammur Absar Chowdhury, Ahmed M. Ali, Shinji Watanabe, Sanjeev KhudanpurICASSP 2023 · OpenReview

TL;DR

This paper introduces "Speech Collage," a novel method for generating synthetic code-switched (CS) audio data by splicing segments from existing monolingual speech corpora. This approach effectively addresses the scarcity of labeled CS data, leading to significant improvements in code-switching Automatic Speech Recognition (ASR) systems, particularly in zero-shot scenarios.

What Problem Does This Paper Address?

Code-switching, the practice of interweaving languages within a conversation or utterance, is common in multilingual communities but poses a significant challenge for speech technologies like ASR. This is primarily due to the scarcity of transcribed code-switched speech datasets. While monolingual data is abundant, directly training ASR models on it can lead to a "monolingual bias," hindering their ability to handle language switches. Existing methods for generating CS data often rely on text-to-speech (TTS) synthesis, which can lack speaker variability, or simple concatenation, which may not capture intra-sentential switches effectively. This paper tackles the problem of generating realistic and diverse CS audio data from readily available monolingual resources to improve CS ASR performance, especially in low-resource or zero-shot settings.

Methods

The core of the proposed method, "Speech Collage," involves synthesizing CS audio by splicing audio segments from monolingual corpora. This process is guided by code-switched text, which can be either real or synthetically generated.

Here's a breakdown of the key techniques:

  • Audio Segment Splicing: The system extracts audio units (word-units for English/Arabic, characters for Mandarin) from monolingual data based on alignments derived from Hidden Markov Model-Gaussian Mixture Model (HMM-GMM) systems. These units are then spliced together to form CS utterances.
  • Overlap-Add: To ensure smooth transitions between spliced audio segments and mitigate discontinuity artifacts, an overlap-add technique with a Hamming window is employed. Segments are extended slightly to create an overlap region for smoother blending.
  • Energy Normalization: To compensate for potential energy variations between spliced segments, the synthesized utterances are normalized by the average energy of their constituent segments.
  • N-gram Units: The method explores using n-grams (unigrams and bigrams) for splicing, aiming to capture more context than single units. If a specific n-gram is unavailable, the system "backs off" to a smaller unit.
  • Zero-Shot CS Text Generation: For scenarios where no real CS text is available, the paper proposes a method to generate synthetic CS text. This involves translating monolingual text, obtaining word-level alignments, and then randomly replacing a percentage of words with their counterparts from the other language.

Datasets

The paper utilizes several datasets for training and evaluation:

  • Monolingual Training Data:
    • Chinese AISHELL-1 (for Mandarin-English CS)
    • Tedlium3 (for Arabic-English CS)
    • MGB-2 (for Arabic-English CS)
  • In-Domain CS Data:
    • SEAME (Mandarin-English CS) - used for both generating CS text and evaluation.
  • Zero-Shot CS Evaluation Data:
    • ESCWA (Arabic-English CS) - a real CS dataset.

The paper does not introduce a new benchmark dataset but rather focuses on generating synthetic CS data to augment existing resources.

Key Insights and Ideas

  • Splicing is a Viable Alternative to TTS: The paper demonstrates that splicing audio segments from monolingual corpora can be a powerful data augmentation technique for CS ASR, offering advantages in terms of speaker variability compared to TTS-based methods.
  • Audio Quality Matters: Enhancing the quality of the generated audio through techniques like overlap-add and energy normalization significantly boosts ASR performance, highlighting that simply generating data isn't enough; it needs to be high-fidelity.
  • Zero-Shot CS is Achievable: The proposed method shows promising results in zero-shot CS scenarios, where no real CS training data is available, by leveraging synthetically generated CS text. This opens doors for building CS ASR systems in truly low-resource settings.
  • CS Augmentation Reduces Monolingual Bias: The experiments suggest that using CS-augmented data not only improves CS performance but also makes the ASR models more adept at handling code-switching and less biased towards a single language.

Key Results

  • In-Domain CS: Speech Collage achieved up to a 34.4% relative reduction in Mixed-Error Rate (MER) compared to a baseline trained on monolingual data.
  • Zero-Shot CS: The method resulted in up to a 16.2% relative reduction in Word-Error Rate (WER) and a 27.6% reduction in Character-Error Rate (CER) compared to a monolingual baseline on the ESCWA dataset.
  • Impact of Data Size: Even a small percentage (10%) of generated CS data led to substantial improvements, with diminishing returns as more data was added.
  • Code-Mixing Index (CMI): Analysis showed that CS data augmentation consistently increased the CMI score, indicating an enhanced ability of the models to handle code-switching.

Limitations and Future Work

  • Audio Mismatches: The paper acknowledges that a gap in MER (up to 8.9%) in the in-domain scenario might be attributed to audio mismatches between the generated data and the target SEAME dataset. Future work could focus on further aligning audio characteristics.
  • Monolingual Performance Deterioration: It was observed that on purely monolingual tasks, performance can sometimes degrade with CS augmentation. This suggests a trade-off that needs further investigation.
  • Further Enhancements: The authors suggest exploring more sophisticated splicing strategies, potentially incorporating linguistic information, and investigating the impact of larger n-grams or other unit types. They also propose further research into the zero-shot CS framework and its limitations.

Summary updated

Abstract

Designing effective automatic speech recognition (ASR) systems for Code-Switching (CS) often depends on the availability of the transcribed CS resources. To address data scarcity, this paper introduces Speech Collage, a method that synthesizes CS data from monolingual corpora by splicing audio segments. We further improve the smoothness quality of audio generation using an overlap-add approach. We investigate the impact of generated data on speech recognition in two scenarios: using in-domain CS text and a zero-shot approach with synthesized CS text. Empirical results highlight up to 34.4% and 16.2% relative reductions in Mixed-Error Rate and Word-Error Rate for in-domain and zero-shot scenarios, respectively. Lastly, we demonstrate that CS augmentation bolsters the model's code-switching inclination and reduces its monolingual bias.
BibTeX
@inproceedings{hussein2023speech,
  author = {Amir Hussein and Dorsa Zeinali and Ondrej Klejch and Matthew Wiesner and Brian Yan and Shammur Absar Chowdhury and Ahmed M. Ali and Shinji Watanabe and Sanjeev Khudanpur},
  title = {{Speech collage: code-switched audio generation by collaging monolingual corpora}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=mRqirCrk9n},
}
Figure