TL;DR
This paper introduces two novel techniques to improve multi-speaker speech synthesis: an Excitation Spectrogram that explicitly models the harmonic structure of speech, and a Conditional Gated LSTM (CGLSTM) that uses speaker identity to control information flow rather than just adding it to the text. These methods significantly reduce reconstruction errors and boost intelligibility scores (MOS) by up to 0.81 points while actually reducing the total number of model parameters.
What Problem Does This Paper Address?
Multi-speaker Text-to-Speech (TTS) faces a "one-to-many" mapping problem: the same sequence of text can be spoken in countless ways depending on the speaker's age, style, pitch, and emotion.
Current state-of-the-art systems usually handle this by taking a "control signal" (like a speaker embedding or pitch value) and simply concatenating or adding it to the text representation. The authors argue this is inefficient. Text content is the primary driver of intelligibility, while speaker identity and pitch drive voice quality and timbre. By mixing them together too early or too simply, the model struggles to disentangle "what is being said" from "who is saying it," often leading to muffled audio or poor speaker similarity.
Methods
The authors propose a modified encoder-decoder framework (similar to Tacotron 2 but with a duration predictor instead of traditional attention) featuring two key innovations:
- Excitation Spectrogram Generator: Instead of feeding raw pitch ($f_0$) and energy into the decoder, the authors create an "excitation spectrogram." This acts as a synthetic resonator that explicitly maps out the harmonics (multiples of the fundamental frequency). This provides the decoder with a structural "skeleton" of the speech's harmonic content, making it much easier for the model to predict the final mel-spectrogram.
- Conditional Gated LSTM (CGLSTM): In a standard LSTM, all inputs are treated similarly. In the proposed CGLSTM, the speaker embedding is used specifically to re-weight the gates (input, output, and forget gates). This means the speaker identity acts as a controller that decides how the text information flows through the network, without directly altering the hidden states that represent the text content itself.
Datasets
The researchers used AISHELL-3, a large-scale public Mandarin Chinese multi-speaker corpus.
- Scale: ~85 hours of recordings from 218 native speakers.
- Training Set: 57,304 utterances from 165 speakers (133 female, 32 male).
- Test Set: 8 speakers (4 female, 4 male) used for speaker adaptation and evaluation.
- Preprocessing: Audio was down-sampled to 16kHz, and silence was trimmed from the ends of utterances.
Key Insights and Ideas
- Differentiated Roles: The core philosophy is that control information (speaker/pitch) and content information (text) should play different roles in the architecture. Content should define the state, while control should define the flow.
- Harmonics are a Shortcut: By explicitly calculating harmonics via the Excitation Spectrogram, the model doesn't have to "guess" the spectral structure from scratch. This leads to much cleaner voiced segments in the synthesized speech.
- Efficiency through Design: Interestingly, "System-3" (which uses both new features) achieved the best performance despite having 10% fewer parameters than the baseline. This suggests that better architectural priors are more effective than simply making a model larger.
Key Results
The authors compared their full system (System-3) against a baseline and versions using only one of the two innovations.
- Objective Performance: System-3 showed the lowest Mean Square Error (MSE) in mel-spectrogram reconstruction across the entire training process.
- Subjective Intelligibility (MOS): The Mean Opinion Score for intelligibility jumped from 3.30 (Baseline) to 4.11 (System-3).
- Voice Quality: For female speakers, the quality score rose from 2.54 to 3.93.
- Speaker Similarity: Similarity scores for unseen speakers improved significantly, moving from 2.76 to 4.05 for female voices, indicating the model is much better at "voice cloning" with limited data.
Limitations and Future Work
- Gender Imbalance: The performance for male speakers was notably lower than for female speakers (e.g., a voice quality score of 3.10 vs 3.93). The authors attribute this to the AISHELL-3 dataset's 9:2 female-to-male ratio.
- Gate Analysis: While the CGLSTM shows clear empirical benefits, the authors note that more investigation is needed to visualize and prove exactly how the control information is modulating the gates in different scenarios.
- Language Scope: The study was conducted exclusively on Mandarin; testing the Excitation Spectrogram on non-tonal languages or languages with different phonological structures remains for future work.
Summary updated