Abstract
This paper introduces "representation mixing," a novel technique for Text-to-Speech (TTS) synthesis that enables a single model to process flexible combinations of character and phoneme inputs. By randomly mixing these representations during training and utilizing a masking mechanism, the model not only resolves pronunciation ambiguities at inference time but also benefits from a powerful data augmentation effect that significantly improves overall audio quality.
The Challenge of Input Representation
In modern TTS systems, text input is typically represented as either graphemes (characters) or phonemes. Both approaches have inherent limitations. Characters are easily extracted but suffer from pronunciation ambiguity; for instance, the word "wind" requires context to distinguish between its noun and verb forms. Conversely, phonemes provide precise instructions but rely on external Grapheme-to-Phoneme (G2P) models or dictionaries, which are often brittle or incomplete.
The authors propose a unified approach to bridge these methods, building a single encoder capable of handling characters, phonemes, or a mix of both. This allows users to explicitly control the pronunciation of ambiguous words while defaulting to standard characters for the remainder of the text.
Technical Methodology
Representation Mixing
The core innovation is the Representation Mixing module. During training, input sequences are randomly mixed at the word level. A binary mask sequence tracks the source of each symbol (0 for character, 1 for phoneme). The network utilizes separate embedding matrices for characters and phonemes, combines them based on the mask, and adds a learned "mask embedding" to help the network distinguish the input type.
Encoder and Attention Mechanisms
The mixed embeddings are processed through a Stacked Multi-scale Residual Convolution (SMRC) network followed by a Bidirectional LSTM. For the attention mechanism, the authors employ Gaussian Mixture (GM) attention. A key refinement involves replacing the traditional exponential activation for the GM mean-step with a softplus activation, which substantially reduces training instability and improves alignment.
Robustness and Audio Inversion
To enhance robustness, the model applies dropout to the pre-net audio inputs at every layer during training (Noisy Teacher Forcing). The final audio is generated via a two-stage inversion process: log mel spectrograms are first converted to a moderate-quality waveform using L-BFGS optimization and a modified Griffin-Lim algorithm. This waveform is then re-converted to a spectrogram to condition a pre-trained WaveNet vocoder, ensuring high-fidelity output.
Experimental Results
The model was trained on the LJSpeech dataset (24 hours of English speech). An A/B user preference study involving 22 users and 429 responses yielded the following insights:
- Data Augmentation Effect: Even when evaluated using only character inputs, the model trained with representation mixing was preferred 73.7% of the time over a baseline trained strictly on characters. This suggests that mixing acts as a powerful regularizer.
- Phoneme with Character Backoff (PWCB): When using a strategy where known words are phonemes and unknown words are characters, the model was preferred 73.1% of the time over a PWCB-only baseline.
- Explicit Control: Within the representation mixing model, users preferred audio generated using PWCB inputs over pure character inputs 58.5% of the time, confirming the value of explicit phonetic injection.
Conclusion and Future Outlook
Representation mixing provides a flexible and robust framework for TTS, offering both improved audio quality through data augmentation and fine-grained control over pronunciation. While the current high-quality inversion pipeline is computationally intensive (~183x slower than real-time), the underlying architectural improvements to the encoder and attention mechanism are highly effective. Future work may extend this technique to multi-speaker datasets and end-to-end waveform generation models.
Summary updated