Representation Mixing for TTS Synthesis.

Kyle Kastner, João Felipe Santos, Yoshua Bengio, Aaron C. Courville

Abstract

This paper introduces "representation mixing," a novel technique for Text-to-Speech (TTS) synthesis that enables a single model to process flexible combinations of character and phoneme inputs. By randomly mixing these representations during training and utilizing a masking mechanism, the model not only resolves pronunciation ambiguities at inference time but also benefits from a powerful data augmentation effect that significantly improves overall audio quality.

The Challenge of Input Representation

In modern TTS systems, text input is typically represented as either graphemes (characters) or phonemes. Both approaches have inherent limitations. Characters are easily extracted but suffer from pronunciation ambiguity; for instance, the word "wind" requires context to distinguish between its noun and verb forms. Conversely, phonemes provide precise instructions but rely on external Grapheme-to-Phoneme (G2P) models or dictionaries, which are often brittle or incomplete.

The authors propose a unified approach to bridge these methods, building a single encoder capable of handling characters, phonemes, or a mix of both. This allows users to explicitly control the pronunciation of ambiguous words while defaulting to standard characters for the remainder of the text.

Technical Methodology

Representation Mixing

The core innovation is the Representation Mixing module. During training, input sequences are randomly mixed at the word level. A binary mask sequence tracks the source of each symbol (0 for character, 1 for phoneme). The network utilizes separate embedding matrices for characters and phonemes, combines them based on the mask, and adds a learned "mask embedding" to help the network distinguish the input type.

Architecture of the representation mixing module, showing the combination of character (e_c) and phoneme (e_p) embeddings via a mask (m).

Encoder and Attention Mechanisms

The mixed embeddings are processed through a Stacked Multi-scale Residual Convolution (SMRC) network followed by a Bidirectional LSTM. For the attention mechanism, the authors employ Gaussian Mixture (GM) attention. A key refinement involves replacing the traditional exponential activation for the GM mean-step with a softplus activation, which substantially reduces training instability and improves alignment.

The SMRC encoder and attention-based decoder pipeline.

Robustness and Audio Inversion

To enhance robustness, the model applies dropout to the pre-net audio inputs at every layer during training (Noisy Teacher Forcing). The final audio is generated via a two-stage inversion process: log mel spectrograms are first converted to a moderate-quality waveform using L-BFGS optimization and a modified Griffin-Lim algorithm. This waveform is then re-converted to a spectrogram to condition a pre-trained WaveNet vocoder, ensuring high-fidelity output.

Experimental Results

The model was trained on the LJSpeech dataset (24 hours of English speech). An A/B user preference study involving 22 users and 429 responses yielded the following insights:

  • Data Augmentation Effect: Even when evaluated using only character inputs, the model trained with representation mixing was preferred 73.7% of the time over a baseline trained strictly on characters. This suggests that mixing acts as a powerful regularizer.
  • Phoneme with Character Backoff (PWCB): When using a strategy where known words are phonemes and unknown words are characters, the model was preferred 73.1% of the time over a PWCB-only baseline.
  • Explicit Control: Within the representation mixing model, users preferred audio generated using PWCB inputs over pure character inputs 58.5% of the time, confirming the value of explicit phonetic injection.

Conclusion and Future Outlook

Representation mixing provides a flexible and robust framework for TTS, offering both improved audio quality through data augmentation and fine-grained control over pronunciation. While the current high-quality inversion pipeline is computationally intensive (~183x slower than real-time), the underlying architectural improvements to the encoder and attention mechanism are highly effective. Future work may extend this technique to multi-speaker datasets and end-to-end waveform generation models.

Summary updated

Abstract

Recent character and phoneme-based parametric TTS systems using deep learning have shown strong performance in natural speech generation. However, the choice between character or phoneme input can create serious limitations for practical deployment, as direct control of pronunciation is crucial in certain cases. We demonstrate a simple method for combining multiple types of linguistic information in a single encoder, named representation mixing, enabling flexible choice between character, phoneme, or mixed representations during inference. Experiments and user studies on a public audiobook corpus show the efficacy of our approach.
BibTeX
@inproceedings{kastner2018representation,
  author = {Kyle Kastner and João Felipe Santos and Yoshua Bengio and Aaron C. Courville},
  title = {{Representation Mixing for TTS Synthesis.}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  doi = {10.1109/icassp.2019.8682880},
}
Figure