Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis

Qinghua Wu, Quanbo Shen, Jian Luan, Yujun Wang

Abstract

In multi-speaker speech synthesis, data from a number of speakers usually tend to have great diversity due to the fact that the speakers may differ largely in ages, speaking styles, emotions, and so on. It is important but challenging to improve the modeling capabilities for multi-speaker speech synthesis. To address the issue, this paper proposes a high-capability speech synthesis system, called Msdtron, in which 1) a representation of the harmonic structure of speech, called excitation spectrogram, is designed to directly guide the learning of harmonics in mel-spectrogram. 2) conditional gated LSTM (CGLSTM) is proposed to control the flow of text content information through the network by re-weighting the gates of LSTM using speaker information. The experiments show a significant reduction in reconstruction error of mel-spectrogram in the training of the multi-speaker model, and a great improvement is observed in the subjective evaluation of speaker adapted model.

Introduction

Multi-speaker Text-to-Speech (TTS) faces a significant "one-to-many" mapping challenge: a single text sequence can be realized in numerous ways depending on a speaker's age, style, pitch, and emotional state. Traditional state-of-the-art systems typically handle this by concatenating or adding control signals (such as speaker embeddings or pitch values) to the text representation.

However, this approach can be inefficient. While text content primarily drives intelligibility, speaker identity and pitch are the main drivers of voice quality and timbre. By mixing these signals too simply, models often struggle to disentangle "what is being said" from "who is saying it," frequently resulting in muffled audio or poor speaker similarity. This paper introduces Msdtron, a system designed to differentiate the roles of content and control information.

Proposed Methodology: Msdtron

The Msdtron architecture, illustrated in Figure 1, is built upon an encoder-decoder framework that replaces traditional attention with a duration predictor. It introduces two primary innovations to better handle multi-speaker data:

  1. Excitation Spectrogram Generator: Rather than feeding raw pitch ($f_0$) and energy directly into the decoder, the system generates an "excitation spectrogram." This serves as a synthetic resonator that explicitly maps out the speech's harmonic structure (multiples of the fundamental frequency). By providing this structural "skeleton," the model can more easily predict the final mel-spectrogram.
  2. Conditional Gated LSTM (CGLSTM): In a standard LSTM, all inputs are processed uniformly. The proposed CGLSTM uses the speaker embedding ($spk$) to specifically re-weight the input, output, and forget gates. This allows speaker identity to act as a controller for information flow without directly altering the hidden states that represent text content.
The Msdtron architecture featuring the Excitation Spectrogram Generator and CGLSTM Decoder.

Experimental Setup

The researchers evaluated the system using AISHELL-3, a large-scale Mandarin Chinese multi-speaker corpus containing approximately 85 hours of recordings from 218 speakers. The training set consisted of 57,304 utterances from 165 speakers, while 8 speakers (4 female, 4 male) were reserved for speaker adaptation and evaluation. Audio was down-sampled to 16kHz with silence trimmed from the ends of utterances.

Results and Discussion

The proposed system (referred to as System-2 in evaluations) was compared against a baseline and an intermediate version (System-1).

Subjective Evaluation

As shown in Figure 3, the proposed system significantly outperformed the baseline in both voice quality and speaker similarity. For female speakers, the quality score improved from 2.71 to 3.97, while similarity scores for unseen speakers rose from 3.39 to 4.06. While performance for male speakers was also improved, it remained lower than for female speakers (3.58 quality vs 3.97), likely due to the 9:2 female-to-male ratio in the AISHELL-3 dataset.

Subjective evaluation results for quality and similarity across different systems.

Efficiency and Design

One of the most notable findings is that the full proposed system (System-2) achieved superior performance while utilizing approximately 10% fewer parameters than the baseline (10.08 M vs 11.24 M). This suggests that incorporating better architectural priors—such as explicit harmonic modeling—is more effective than simply increasing model scale. By differentiating the roles of content (defining state) and control (defining flow), Msdtron achieves cleaner voiced segments and better "voice cloning" capabilities with limited data.

Conclusion

Msdtron demonstrates that explicit harmonic modeling and gated control of information flow can significantly improve multi-speaker TTS. Future work will focus on addressing the gender performance gap and extending the excitation spectrogram approach to non-tonal languages. Additionally, further analysis is needed to visualize how the CGLSTM gates modulate information across different speaking styles.

Summary updated

Abstract

In multi-speaker speech synthesis, data from a number of speakers usually tend to have great diversity due to the fact that the speakers may differ largely in ages, speaking styles, emotions, and so on. It is important but challenging to improve the modeling capabilities for multi-speaker speech synthesis. To address the issue, this paper proposes a high-capability speech synthesis system, called Msdtron, in which 1) a representation of the harmonic structure of speech, called excitation spectrogram, is designed to directly guide the learning of harmonics in mel-spectrogram. 2) conditional gated LSTM (CGLSTM) is proposed to control the flow of text content information through the network by re-weighting the gates of LSTM using speaker information. The experiments show a significant reduction in reconstruction error of mel-spectrogram in the training of the multi-speaker model, and a great improvement is observed in the subjective evaluation of speaker adapted model.
BibTeX
@inproceedings{wu2022effective,
  author = {Qinghua Wu and Quanbo Shen and Jian Luan and Yujun Wang},
  title = {{Effective and Differentiated Use of Control Information for Multi-speaker Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747032},
}
Figure