Abstract
This work evaluates several cutting-edge large-scale foundation models based on self-supervision or weak supervision, including SeamlessM4T, SeamlessM4T v2, and Whisper-large-v3, on three code-switched corpora. We found that self-supervised models can achieve performances close to the supervised model, indicating the effectiveness of multilingual self-supervised pre-training. We also observed that these models still have room for improvement as they kept making similar mistakes and had unsatisfactory performances on modeling intra-sentential code-switching. In addition, the validity of several variants of Whisper was explored, and we concluded that they remained effective in a code-switching scenario, and similar techniques for self-supervised models are worth studying to boost the performance of code-switched tasks.
Introduction
In our increasingly globalized world, "code-switching" (CS)—the fluid act of mixing multiple languages in a single conversation—is a daily reality for millions. Yet, for speech technology, it remains a formidable challenge. Most Automatic Speech Recognition (ASR) systems are trained on single-language datasets, leaving them prone to failure when a speaker pivots from Mandarin to English mid-sentence.
A primary bottleneck in code-switching research is the scarcity of high-quality labeled data; collecting thousands of hours of natural, bilingual conversation is both expensive and logistically difficult. This study investigates whether the current generation of "foundation models"—massive AI models like OpenAI’s Whisper and Meta’s SeamlessM4T—can handle Mandarin-English code-switching in a zero-shot manner, leveraging their massive pre-training without specific fine-tuning on mixed-language datasets.
Methodology
Foundation Models
The researchers evaluated models representing two primary training paradigms:
- Self-Supervised Learning (SSL): Models like SeamlessM4T (v1 and v2) and MMS (Massively Multilingual Speech), which learn from raw, unlabeled audio to capture universal speech representations.
- Weakly Supervised Learning: Models like Whisper-large-v3, which are trained on astronomical amounts of noisy, web-scraped audio-text pairs.
Prompting and In-Context Learning
To enhance Whisper's performance, the study explored several prompting strategies:
- Language Token Concatenation (
concat): Forcing the model to consider both<|zh|>and<|en|>tokens simultaneously to acknowledge the bilingual nature of the input. - Speech-based In-Context Learning (SICL): Providing the model with a short audio-transcript example of code-switching as a prefix before transcribing the target audio.
- Clairaudience Prompting: Utilizing domain-specific tags (e.g., "domain: machine learning") to prime the model for specialized vocabulary.
Experimental Setup
The models were benchmarked across three distinct Mandarin-English corpora to test various aspects of linguistic complexity:
- ASCEND: Spontaneous conversational speech covering general topics like education and technology.
- CSZS-correct: A challenging benchmark specifically designed to evaluate intra-sentential code-switching (switching languages within a single sentence).
- NTUML2021: A real-world dataset consisting of noisy Machine Learning lecture recordings, testing the models' ability to handle technical terminology.
Key Insights and Results
Performance Metrics
The study utilized the Mixed Error Rate (MER), which treats Chinese characters and English words as individual units.
- Whisper's Gains: On the ASCEND dataset, the baseline Whisper-large-v3 recorded a 30.29% MER. However, by applying prompting and SICL, this error rate dropped significantly to 13.53%.
- The SSL Leap: Meta’s SeamlessM4T v2 achieved a 25.86% MER on ASCEND without specialized prompting, outperforming its predecessor and the MMS model. This suggests that SSL pre-training is increasingly effective at capturing the structure of multilingual speech, even with less labeled data than Whisper.
- Speech-to-Text Translation (ST): When translating code-switched speech into pure English, Whisper-large-v3 with SICL achieved a BLEU score of 31.57, approaching the performance of top-tier text-to-text models like GPT-3.5.
Qualitative Observations
- Data Scaling vs. Language Coverage: SeamlessM4T v2 (trained on more data) outperformed MMS (which supports more languages), indicating that for code-switching, data depth and volume are more critical than the sheer number of supported languages.
- The "Auto-Translation" Trap: A recurring error pattern involved models spontaneously translating English segments into Mandarin (or vice versa) instead of transcribing them. This is likely a side-effect of multi-task training where models learn to both transcribe and translate.
- Domain Sensitivity: Models struggled with technical terms in the NTUML2021 dataset, often hallucinating or mistranscribing terms (e.g., transcribing "autoencoder" as "posenco").
Conclusion
While foundation models show remarkable zero-shot capabilities, intra-sentential code-switching remains a significant hurdle. The research demonstrates that while SSL models are rapidly closing the gap with weakly supervised models, they currently lack the "instruction-following" flexibility that allows Whisper to be tuned via prompting. Future work should focus on developing SICL-like techniques for self-supervised architectures to further boost performance in complex, multi-lingual scenarios.
Summary updated