MSDTRON: A High-Capability Multi-Speaker Speech Synthesis System for Diverse Data Using Characteristic Information

Qinghua Wu, Quanbo Shen, Jian Luan, Yujun Wang

TL;DR

This paper introduces two novel techniques to improve multi-speaker speech synthesis: an Excitation Spectrogram that explicitly models the harmonic structure of speech, and a Conditional Gated LSTM (CGLSTM) that uses speaker identity to control information flow rather than just adding it to the text. These methods significantly reduce reconstruction errors and boost intelligibility scores (MOS) by up to 0.81 points while actually reducing the total number of model parameters.

What Problem Does This Paper Address?

Multi-speaker Text-to-Speech (TTS) faces a "one-to-many" mapping problem: the same sequence of text can be spoken in countless ways depending on the speaker's age, style, pitch, and emotion.

Current state-of-the-art systems usually handle this by taking a "control signal" (like a speaker embedding or pitch value) and simply concatenating or adding it to the text representation. The authors argue this is inefficient. Text content is the primary driver of intelligibility, while speaker identity and pitch drive voice quality and timbre. By mixing them together too early or too simply, the model struggles to disentangle "what is being said" from "who is saying it," often leading to muffled audio or poor speaker similarity.

Methods

The authors propose a modified encoder-decoder framework (similar to Tacotron 2 but with a duration predictor instead of traditional attention) featuring two key innovations:

  1. Excitation Spectrogram Generator: Instead of feeding raw pitch ($f_0$) and energy into the decoder, the authors create an "excitation spectrogram." This acts as a synthetic resonator that explicitly maps out the harmonics (multiples of the fundamental frequency). This provides the decoder with a structural "skeleton" of the speech's harmonic content, making it much easier for the model to predict the final mel-spectrogram.
  2. Conditional Gated LSTM (CGLSTM): In a standard LSTM, all inputs are treated similarly. In the proposed CGLSTM, the speaker embedding is used specifically to re-weight the gates (input, output, and forget gates). This means the speaker identity acts as a controller that decides how the text information flows through the network, without directly altering the hidden states that represent the text content itself.

Datasets

The researchers used AISHELL-3, a large-scale public Mandarin Chinese multi-speaker corpus.

  • Scale: ~85 hours of recordings from 218 native speakers.
  • Training Set: 57,304 utterances from 165 speakers (133 female, 32 male).
  • Test Set: 8 speakers (4 female, 4 male) used for speaker adaptation and evaluation.
  • Preprocessing: Audio was down-sampled to 16kHz, and silence was trimmed from the ends of utterances.

Key Insights and Ideas

  • Differentiated Roles: The core philosophy is that control information (speaker/pitch) and content information (text) should play different roles in the architecture. Content should define the state, while control should define the flow.
  • Harmonics are a Shortcut: By explicitly calculating harmonics via the Excitation Spectrogram, the model doesn't have to "guess" the spectral structure from scratch. This leads to much cleaner voiced segments in the synthesized speech.
  • Efficiency through Design: Interestingly, "System-3" (which uses both new features) achieved the best performance despite having 10% fewer parameters than the baseline. This suggests that better architectural priors are more effective than simply making a model larger.

Key Results

The authors compared their full system (System-3) against a baseline and versions using only one of the two innovations.

  • Objective Performance: System-3 showed the lowest Mean Square Error (MSE) in mel-spectrogram reconstruction across the entire training process.
  • Subjective Intelligibility (MOS): The Mean Opinion Score for intelligibility jumped from 3.30 (Baseline) to 4.11 (System-3).
  • Voice Quality: For female speakers, the quality score rose from 2.54 to 3.93.
  • Speaker Similarity: Similarity scores for unseen speakers improved significantly, moving from 2.76 to 4.05 for female voices, indicating the model is much better at "voice cloning" with limited data.

Limitations and Future Work

  • Gender Imbalance: The performance for male speakers was notably lower than for female speakers (e.g., a voice quality score of 3.10 vs 3.93). The authors attribute this to the AISHELL-3 dataset's 9:2 female-to-male ratio.
  • Gate Analysis: While the CGLSTM shows clear empirical benefits, the authors note that more investigation is needed to visualize and prove exactly how the control information is modulating the gates in different scenarios.
  • Language Scope: The study was conducted exclusively on Mandarin; testing the Excitation Spectrogram on non-tonal languages or languages with different phonological structures remains for future work.

Summary updated

Abstract

In multi-speaker speech synthesis, data from a number of speakers usually tend to have great diversity due to the fact that the speakers may differ largely in ages, speaking styles, emotions, and so on. It is important but challenging to improve the modeling capabilities for multi-speaker speech synthesis. To address the issue, this paper proposes a high-capability speech synthesis system, called Msdtron, in which 1) a representation of the harmonic structure of speech, called excitation spectrogram, is designed to directly guide the learning of harmonics in mel-spectrogram. 2) conditional gated LSTM (CGLSTM) is proposed to control the flow of text content information through the network by re-weighting the gates of LSTM using speaker information. The experiments show a significant reduction in reconstruction error of mel-spectrogram in the training of the multi-speaker model, and a great improvement is observed in the subjective evaluation of speaker adapted model.
BibTeX
@inproceedings{wu2022msdtron,
  author = {Qinghua Wu and Quanbo Shen and Jian Luan and Yujun Wang},
  title = {{MSDTRON: A High-Capability Multi-Speaker Speech Synthesis System for Diverse Data Using Characteristic Information}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747032},
}

Loading related content...

Figure