Abstract
Microsoft researchers have introduced E2 TTS (Embarrassingly Easy TTS), a zero-shot text-to-speech system that achieves state-of-the-art naturalness by stripping away the complex "plumbing" of traditional models. By using a simple flow-matching Transformer and a character-based input with filler tokens, it matches or surpasses the performance of much more complex systems like Voicebox and NaturalSpeech 3.
Introduction
Modern zero-shot TTS aims to clone a person's voice from just a few seconds of audio. Currently, researchers face a trade-off: Autoregressive (AR) models (like VALL-E) offer high quality but suffer from high latency because they generate tokens one by one. Non-Autoregressive (NAR) models are faster but struggle with the "alignment problem"—the fact that a short sentence of text corresponds to a much longer sequence of audio frames.
To solve this, most NAR models rely on external "crutches": phoneme aligners, duration predictors, or complex algorithms like Monotonic Alignment Search (MAS). E2 TTS asks: Can we get rid of all that extra machinery and still get human-level results?
Methodology
The architecture of E2 TTS is minimalist, consisting of only two main parts: a flow-matching-based mel spectrogram generator and a vocoder (BigVGAN).
Core Design
- Input Representation: Instead of complex phoneme sequences, E2 TTS uses raw characters. To handle the length mismatch, the authors append "filler tokens" $\langle F \rangle$ to the text until the length of the text sequence matches the number of frames in the target audio.
- Flow-Matching: The model uses conditional flow-matching, a generative technique that learns to transform a simple noise distribution into a structured mel spectrogram.
- The Infilling Task: The model is trained on a speech-infilling task. Given a mel spectrogram where some parts are masked, the model must predict the missing audio based on the provided text and the unmasked audio "prompt."
Model Extensions
- E2 TTS X1: Removes the need to transcribe the audio prompt during inference.
- E2 TTS X2: Allows users to manually specify the pronunciation of specific words by wrapping phonemes in parentheses within the character string, providing granular control over the output.
Key Results and Insights
E2 TTS achieved impressive numbers across both objective and subjective metrics on the LibriSpeech-PC benchmark:
- Naturalness (CMOS): In subjective listening tests, E2 TTS achieved a score of -0.05, effectively meaning listeners found it indistinguishable from ground-truth human speech. It significantly outperformed Voicebox (-0.78) and NaturalSpeech 3 (-0.98).
- Intelligibility: It achieved a Word Error Rate (WER) of 1.9%, which is better than the ground truth audio (2.0%) in the same test environment.
- Speaker Similarity: It reached a similarity score (SIM-o) of 0.708, surpassing the ground truth's own internal similarity score (0.695).
One of the most striking findings is that explicit duration models and phoneme aligners are not just unnecessary—they might actually be harmful. By removing them, the model learns a more fluid, natural way to map text to speech jointly within a single Transformer.
Limitations and Future Work
During inference, E2 TTS still requires the user to provide a "target duration." While the model is robust to various durations (0.7x to 1.3x speed), it doesn't yet have a built-in mechanism to predict the most natural duration for a given sentence. Additionally, the "easy" architecture shifts the burden from the engineer to the GPU, requiring more training iterations to converge compared to models with explicit frame-wise alignment. Future work will explore using this simplified representation for large-scale speech foundation models.
Summary updated