AISHELL-3: A Multi-Speaker Mandarin TTS Corpus

Yao Shi, Hui Bu, Xin Xu, Shaoji Zhang, Ming Li2021 · DOI

TL;DR

This paper introduces AISHELL-3, a large-scale, open-source Mandarin Chinese speech corpus designed for training multi-speaker Text-to-Speech (TTS) systems. By providing 85 hours of recordings from 218 native speakers with detailed attribute annotations, AISHELL-3 addresses a critical gap in Mandarin TTS research, enabling more flexible and controllable speech synthesis.

What Problem Does This Paper Address?

Text-to-Speech (TTS) systems have made remarkable progress, largely thanks to deep learning models. However, a key challenge is controlling the output beyond just the text itself. While systems can generate speech, making them produce specific voices, emotions, or speaking styles consistently remains an active research area. Multi-speaker TTS systems aim to disentangle speaker identity from textual content, allowing for more personalized and flexible speech generation. Training these advanced systems requires substantial, well-annotated datasets. While English has datasets like VCTK, Mandarin Chinese, a tonal language with complex prosody, has lacked a comparable large-scale, multi-speaker corpus. This deficit hinders research and development in Mandarin TTS, making it difficult to build systems that can synthesize speech with diverse speaker characteristics.

Methods

The paper introduces the AISHELL-3 corpus, which contains approximately 85 hours of Mandarin speech from 218 native speakers. The recordings are emotion-neutral and include detailed auxiliary attributes like gender, age group, and native accents, along with character-level and pinyin-level transcriptions.

To demonstrate the utility of AISHELL-3, the authors train two mainstream TTS systems: Tacotron-2 (an RNN-based seq2seq model) and Fastspeech (a feed-forward Transformer-based model). Both models are adapted for multi-speaker synthesis using a common one-hot embedding approach, where speaker-specific embeddings are conditioned on the encoder's output.

A novel technique explored is "cross-speaker duration migration" (DM) with Fastspeech. This method allows for replacing "bad" duration predictions with more fluent ones from a different speaker during inference. This is particularly useful when dealing with amateur recordings that might have hesitant or mechanical tones, aiming to improve prosody without sacrificing the target speaker's voice.

For Fastspeech, a specific method for duration extraction is proposed to address issues with attention mechanisms in Tacotron-2, leading to more accurate duration labels.

Datasets

  • AISHELL-3: The newly introduced dataset. It comprises 85 hours of Mandarin speech from 218 native speakers, with character-level and pinyin-level transcriptions, and speaker attributes (gender, age, accent).
  • VCTK: Mentioned as a comparable English multi-speaker corpus.
  • DiDi-Speech: A concurrent work also addressing the Mandarin TTS corpus gap, noted for its larger scale but with fewer utterances per speaker and higher background noise.
  • Voxceleb2, AISHELL-2: Used for training speaker verification (SV) systems for objective evaluation.

Key Insights and Ideas

  • The critical need for Mandarin multi-speaker TTS data: The paper highlights the significant gap in publicly available resources for Mandarin, which AISHELL-3 aims to fill.
  • Manual transcription is crucial for Mandarin: The authors emphasize the challenges of automatic pinyin generation for Mandarin due to homographs, tone sandhi, and erization, making manual transcripts invaluable.
  • Cross-speaker duration migration (DM) can improve prosody: The DM technique shows promise in enhancing the naturalness of synthesized speech, especially when dealing with less-than-ideal speaker recordings, by leveraging prosodic information from other speakers.
  • Objective evaluation metrics are key: The use of speaker verification systems (resnet-se, ecapa-tdnn) for evaluating speaker similarity provides a robust, quantitative measure of the TTS models' ability to capture target voices.
  • Generalization potential is high: The experiments suggest that even with a limited number of training speakers, the embedding-based approach allows TTS systems to generalize to a wider range of potential voices.

Key Results

  • MOS Scores: Tacotron-2 achieved a MOS score of 4.21 (±0.09), and Fastspeech achieved 4.08 (±0.11) for synthesized speech from seen speakers. For unseen speakers, Tacotron-2 scored 3.89 (±0.11) and Fastspeech 4.14 (±0.11).
  • Speaker Similarity: The trained models demonstrated strong speaker similarity, with the highest EER (Equal Error Rate) for Fastspeech with DM being 1.51%, indicating that voices from seen speakers can be stably reproduced.
  • Generalization: The models showed significant generalization potential, with the number of distinct generalized voices (Vc) greatly exceeding the number of training speakers (161). Fastspeech, despite a slightly lower Ve count than Tacotron, achieved higher MOS scores.

Limitations and Future Work

  • Data Quality: While AISHELL-3 is designed for quiet environments, the paper acknowledges that amateur speakers might exhibit hesitant or mechanical tones, which can impact naturalness.
  • Vocoder Artifacts: The vocoder (HiFi-GAN) fine-tuned with Tacotron-GTA features showed irregular performance when used with ground-truth spectrograms, suggesting potential artifacts.
  • Generalization Interpretation: The interpretation of generalization potential is limited by the clique-finding algorithm and the number of sampled embeddings.
  • Future Work: The authors suggest that the corpus is valuable for more sophisticated TTS approaches. The success of DM hints at further exploration of disentangling voice and prosody. The high generalization potential also opens avenues for research into few-shot or zero-shot TTS with even fewer training speakers.

Summary updated

BibTeX
@misc{shi2021aishell,
  author = {Yao Shi and Hui Bu and Xin Xu and Shaoji Zhang and Ming Li},
  title = {{AISHELL-3: A Multi-Speaker Mandarin TTS Corpus}},
  year = {2021},
  doi = {10.21437/interspeech.2021-755},
}
Figure