AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines

Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, Ming Li

Abstract

In this paper, we present AISHELL-3, a large-scale and high-fidelity multi-speaker Mandarin speech corpus which could be used to train multi-speaker Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese mandarin speakers. Their auxiliary attributes such as gender, age group and native accents are explicitly marked and provided in the corpus. Accordingly, transcripts in Chinese character-level and pinyin-level are provided along with the recordings. We present a baseline system that uses AISHELL-3 for multi-speaker Madarin speech synthesis. The multi-speaker speech synthesis system is an extension on Tacotron-2 where a speaker verification model and a corresponding loss regarding voice similarity are incorporated as the feedback constraint. We aim to use the presented corpus to build a robust synthesis model that is able to achieve zero-shot voice cloning. The system trained on this dataset also generalizes well on speakers that are never seen in the training process. Objective evaluation results from our experiments show that the proposed multi-speaker synthesis system achieves high voice similarity concerning both speaker embedding similarity and equal error rate measurement.

Introduction

While neural Text-to-Speech (TTS) systems like Tacotron-2 have achieved near-human naturalness, most high-quality models remain limited to single-speaker scenarios. Developing systems capable of mimicking diverse voices or performing "zero-shot" voice cloning—synthesizing a new voice from a brief audio sample—requires vast amounts of labeled data. For Mandarin Chinese, this task is further complicated by its tonal nature, where pitch contours define word meanings. Existing Mandarin datasets often lack the scale or speaker diversity necessary for robust multi-speaker research.

AISHELL-3 addresses this gap by providing a massive, high-fidelity Mandarin speech corpus. Alongside the data, we introduce a baseline architecture designed to leverage this diversity for high-quality synthesis and zero-shot cloning.

The AISHELL-3 Dataset

The AISHELL-3 corpus consists of 88,035 recordings, totaling approximately 85 hours of audio from 218 native speakers. Each speaker's metadata, including age, gender, and accent, is meticulously recorded. Transcriptions are provided at both the Chinese character and pinyin levels. To ensure high quality, the speaker encoder was pre-trained on auxiliary datasets like aidataTang and MAGICDATA before integration.

Proposed Baseline System

The baseline system is a multi-stage pipeline designed for Mandarin-specific challenges:

  1. Textual Frontend: To handle Mandarin's complex prosody, the system uses a pinyin parser followed by an LSTM-based prosody prediction model. This hierarchical approach ensures natural rhythmic patterns rather than monotonic output.
  2. Speaker-Aware Acoustic Model: Built upon Tacotron-2, the model is conditioned on a ResNet-based Speaker Encoder. A key innovation is the Feedback Constraint: during training, the system extracts an embedding from the generated mel-spectrogram and compares it to the target speaker's embedding. A similarity loss term forces the model to minimize the difference, effectively teaching it to prioritize speaker identity.
  3. Neural Vocoder: MelGAN is employed to convert the model's internal acoustic representations back into high-fidelity waveforms.
Architecture of the multi-speaker TTS system featuring the Speaker Encoder and Feedback Constraint loop.

Key Insights and Experimental Results

Several practical optimizations were identified during development. Trimming silence from the start of training samples improved alignment speed by a factor of ten. Additionally, data augmentation—concatenating short samples into longer sequences—helped the model maintain stable attention during long-form synthesis.

The system was evaluated using Cosine Similarity and Speaker Verification Equal Error Rate (SV-EER).

  • Seen Speakers: The system achieved a cosine similarity of 0.918, indicating high fidelity to the original speaker's identity.
  • Unseen Speakers (Zero-Shot): For speakers not present in the training set, the model maintained a similarity of 0.868, demonstrating strong generalization for voice cloning.
t-SNE visualization of speaker embeddings, illustrating the model's ability to cluster and distinguish between different speaker identities.

Conclusion

AISHELL-3 provides the community with the scale and quality needed to advance multi-speaker Mandarin synthesis. While the current corpus is primarily emotion-neutral and skewed toward younger demographics, it establishes a robust foundation for future research into expressive and diverse Mandarin TTS.

Summary updated

Abstract

In this paper, we present AISHELL-3, a large-scale and high-fidelity multi-speaker Mandarin speech corpus which could be used to train multi-speaker Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese mandarin speakers. Their auxiliary attributes such as gender, age group and native accents are explicitly marked and provided in the corpus. Accordingly, transcripts in Chinese character-level and pinyin-level are provided along with the recordings. We present a baseline system that uses AISHELL-3 for multi-speaker Madarin speech synthesis. The multi-speaker speech synthesis system is an extension on Tacotron-2 where a speaker verification model and a corresponding loss regarding voice similarity are incorporated as the feedback constraint. We aim to use the presented corpus to build a robust synthesis model that is able to achieve zero-shot voice cloning. The system trained on this dataset also generalizes well on speakers that are never seen in the training process. Objective evaluation results from our experiments show that the proposed multi-speaker synthesis system achieves high voice similarity concerning both speaker embedding similarity and equal error rate measurement. The dataset, baseline system code and generated samples are available online.
BibTeX
@misc{shi2020aishell,
  author = {Yao Shi and Hui Bu and Xin Xu and Shaoji Zhang and Ming Li},
  title = {{AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines}},
  year = {2020},
  eprint = {2010.11567},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.11567},
}
Figure