WenetSpeech-Yue: A Large-scale Cantonese Speech Corpus with Multi-dimensional Annotation

Longhao Li, Zhao Guo, Hongjie Chen, Yuhang Dai, Ziyu Zhang, Hongfei Xue, Tianlun Zuo, Chengyou Wang, Shuiyuan Wang, Jie Li, Jian Kang, Xin Xu, Hui Bu, Binbin Zhang, Ruibin Yuan, Ziya Zhou, Wei Xue, Lei Xie

Introduction

Cantonese (Yue Chinese), spoken by nearly 85 million people, has historically been a low-resource language in the field of AI. It presents unique linguistic challenges, including a complex nine-tone system, a sharp divide between literary and colloquial forms, and frequent "code-switching" with English. Existing datasets are often limited in scale, domain diversity, or lack the rich metadata required for modern generative speech tasks.

To bridge this gap, researchers have introduced WenetSpeech-Yue, the largest open-source Cantonese speech corpus to date. It features over 21,800 hours of audio across 10 diverse domains, accompanied by a robust automated annotation pipeline.

The WenetSpeech-Pipe Framework

The researchers developed WenetSpeech-Pipe, a modular framework designed to transform "in-the-wild" audio into a gold-standard dataset through multi-dimensional labeling and advanced transcription techniques.

Overview of the WenetSpeech-Pipe architecture, illustrating the stages from audio collection to forced alignment.

The pipeline follows a structured five-step process:

  1. Audio Collection & Segmentation: Scrapes audio from 10 domains (vlogs, drama, news, etc.) and uses Voice Activity Detection (VAD) to create manageable clips.
  2. Multi-Dimensional Labeling: Employs diarization and classification models to estimate speaker ID, age, and gender, alongside quality metrics like Signal-to-Noise Ratio (SNR) and DNSMOS.
  3. Transcription Ensemble: Uses three distinct ASR models (SenseVoice, Whisper, and TeleASR) to generate candidate transcripts.
  4. ROVER + LLM Refinement: Candidate outputs are aligned and voted upon (ROVER). A 4B-parameter Large Language Model (Qwen) then acts as a contextual corrector, resolving homophone errors and ensuring grammatical consistency.
  5. Forced Alignment: Generates character-level timestamps, which are essential for precise Text-to-Speech (TTS) and synchronization.

Dataset Characteristics and Insights

WenetSpeech-Yue provides 21,800 hours of training data. By utilizing the pipeline's quality metadata, researchers can filter for high-quality subsets, such as a 12,000-hour set optimized for TTS. The corpus's diversity across 10 domains allows models to generalize better to real-world scenarios compared to those trained on traditional "read speech" datasets.

Distribution of audio hours across different age groups and genders in the WenetSpeech-Yue corpus.

A key innovation is the use of LLMs as intelligent error correctors. By providing the LLM with the outputs of three different ASR systems as context, it can resolve linguistic ambiguities that traditional models struggle with. Additionally, the study demonstrates that automated quality filtering (SNR and MOS) can effectively extract high-quality generative data from noisy, real-world sources.

Key Results

The impact of WenetSpeech-Yue is demonstrated through state-of-the-art performance on the new WSYue-eval benchmark:

  • ASR Performance: Models trained on this corpus, such as the U2pp-Conformer-LLM-Yue, achieved a Mixed Error Rate (MER) of 4.73% on short utterances, significantly outperforming commercial baselines and larger models like Whisper-large-v3 (12.95%).
  • TTS Quality: Fine-tuning models like CosyVoice2 on this data led to dramatic improvements, reaching an intelligibility score (I-MOS) of 4.45, surpassing many commercial systems.

Limitations and Future Work

While WenetSpeech-Yue represents a massive leap forward, the authors noted a gender imbalance, with a predominance of middle-aged male speakers. Future work will focus on capturing more dialectal variations within Cantonese and continuing to scale the corpus to match the resources available for high-resource languages like English or Mandarin.

Summary updated

BibTeX
@inproceedings{li2026wenetspeech,
  author = {Longhao Li and Zhao Guo and Hongjie Chen and Yuhang Dai and Ziyu Zhang and Hongfei Xue and Tianlun Zuo and Chengyou Wang and Shuiyuan Wang and Jie Li and Jian Kang and Xin Xu and Hui Bu and Binbin Zhang and Ruibin Yuan and Ziya Zhou and Wei Xue and Lei Xie},
  title = {{WenetSpeech-Yue: A Large-scale Cantonese Speech Corpus with Multi-dimensional Annotation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i37.40429},
}
Figure