Introduction
Cantonese (Yue Chinese), spoken by nearly 85 million people, has historically been a low-resource language in the field of AI. It presents unique linguistic challenges, including a complex nine-tone system, a sharp divide between literary and colloquial forms, and frequent "code-switching" with English. Existing datasets are often limited in scale, domain diversity, or lack the rich metadata required for modern generative speech tasks.
To bridge this gap, researchers have introduced WenetSpeech-Yue, the largest open-source Cantonese speech corpus to date. It features over 21,800 hours of audio across 10 diverse domains, accompanied by a robust automated annotation pipeline.
The WenetSpeech-Pipe Framework
The researchers developed WenetSpeech-Pipe, a modular framework designed to transform "in-the-wild" audio into a gold-standard dataset through multi-dimensional labeling and advanced transcription techniques.
The pipeline follows a structured five-step process:
- Audio Collection & Segmentation: Scrapes audio from 10 domains (vlogs, drama, news, etc.) and uses Voice Activity Detection (VAD) to create manageable clips.
- Multi-Dimensional Labeling: Employs diarization and classification models to estimate speaker ID, age, and gender, alongside quality metrics like Signal-to-Noise Ratio (SNR) and DNSMOS.
- Transcription Ensemble: Uses three distinct ASR models (SenseVoice, Whisper, and TeleASR) to generate candidate transcripts.
- ROVER + LLM Refinement: Candidate outputs are aligned and voted upon (ROVER). A 4B-parameter Large Language Model (Qwen) then acts as a contextual corrector, resolving homophone errors and ensuring grammatical consistency.
- Forced Alignment: Generates character-level timestamps, which are essential for precise Text-to-Speech (TTS) and synchronization.
Dataset Characteristics and Insights
WenetSpeech-Yue provides 21,800 hours of training data. By utilizing the pipeline's quality metadata, researchers can filter for high-quality subsets, such as a 12,000-hour set optimized for TTS. The corpus's diversity across 10 domains allows models to generalize better to real-world scenarios compared to those trained on traditional "read speech" datasets.
A key innovation is the use of LLMs as intelligent error correctors. By providing the LLM with the outputs of three different ASR systems as context, it can resolve linguistic ambiguities that traditional models struggle with. Additionally, the study demonstrates that automated quality filtering (SNR and MOS) can effectively extract high-quality generative data from noisy, real-world sources.
Key Results
The impact of WenetSpeech-Yue is demonstrated through state-of-the-art performance on the new WSYue-eval benchmark:
- ASR Performance: Models trained on this corpus, such as the U2pp-Conformer-LLM-Yue, achieved a Mixed Error Rate (MER) of 4.73% on short utterances, significantly outperforming commercial baselines and larger models like Whisper-large-v3 (12.95%).
- TTS Quality: Fine-tuning models like CosyVoice2 on this data led to dramatic improvements, reaching an intelligibility score (I-MOS) of 4.45, surpassing many commercial systems.
Limitations and Future Work
While WenetSpeech-Yue represents a massive leap forward, the authors noted a gender imbalance, with a predominance of middle-aged male speakers. Future work will focus on capturing more dialectal variations within Cantonese and continuing to scale the corpus to match the resources available for high-resource languages like English or Mandarin.
Summary updated