TL;DR
Guided-TTS introduces a framework to build high-quality text-to-speech (TTS) systems for target speakers using only raw, untranscribed, and unsegmented audio. By decoupling speech generation into an unconditional diffusion model trained on audio chunks and a phoneme classifier trained on separate multi-speaker data, it achieves speech quality and intelligibility on par with fully supervised TTS baselines.
What Problem Does This Paper Address?
State-of-the-art neural TTS models generally require dozens of hours of high-quality paired <speech, transcript> data for each target speaker. While vast amounts of untranscribed speech exist (e.g., audiobooks, podcasts), using them directly has traditionally required labor-intensive preprocessing: segmenting continuous audio into sentence-level chunks and generating highly accurate transcripts.
Guided-TTS tackles this bottleneck by asking: Can we train an unconditional generative speech model entirely without transcripts or sentence alignments, and then guide it to speak specific text at inference time?
Methods
Guided-TTS splits the TTS pipeline into four components:
- Unconditional Diffusion Model (DDPM): Models the unconditional distribution of mel-spectrograms $p(X)$ using continuous score-based stochastic differential equations (SDEs). Because it requires no text, it trains directly on random 5-second audio crops from the target speaker. The architecture is a 2D U-Net adapted from standard image diffusion models.
- Framewise Phoneme Classifier: A WaveNet-like classifier trained on noisy mel-spectrograms $X_t$ to predict frame-level phoneme labels $\hat{y}$. It conditions on diffusion timestep $t$ and speaker embeddings.
- Duration Predictor & Speaker Encoder: A Glow-TTS-style duration predictor maps input phoneme sequences to framewise phoneme sequences $\hat{y}$. A pre-trained speaker encoder (trained with GE2E loss) extracts speaker embeddings $e_S$ to make the classifier and duration predictor speaker-adaptive.
- Norm-Based Guidance: Standard classifier guidance adds a scaled gradient $\nabla_{X_t} \log p(\hat{y}|X_t)$ to the score during reverse sampling. However, the authors observed that the unconditional score norm $\|\nabla_{X_t} \log p_\theta(X_t)\|$ spikes dramatically as $t \to 0$ (up to ~70$\times$ the classifier gradient norm), rendering the text guidance ineffective near clean data and causing severe mispronunciations. Norm-based guidance dynamically rescales the classifier gradient in proportion to the unconditional score norm at each step, ensuring stable text conditioning throughout sampling.
+-------------------+
| Text / Phonemes y |
+---------+---------+
|
v
+-------------------+
| Duration Predictor|
+---------+---------+
|
v
+-------------------+
| Framewise Labels ŷ|
+---------+---------+
|
Random Noise X_T v
| +-------------------+
+------------> |Phoneme Classifier | ---> Classifier Gradient
| +-------------------+ |
v v
+--------------+ +-----------------+
|Unconditional | ---> Unconditional Score -----> | Norm-Based | ---> Speech Mel X_0
| DDPM | | Guidance Solver | (to HiFi-GAN)
+--------------+ +-----------------+
Datasets
- Target Speaker Unpaired Audio:
- LJSpeech: 24 hours of single-speaker female speech (used both transcribed and speech-only for comparisons).
- Hi-Fi TTS: Multi-speaker dataset; three speakers were evaluated (two male, one female; 27.3 to 58.0 hours).
- Blizzard 2013: 147 hours of audiobook speech read by a single female speaker (evaluated using random 5-second crops from unsegmented data).
- Guidance & Multi-Speaker Pretraining:
- LibriTTS: ~585 hours across 2,456 speakers, aligned with Montreal Forced Aligner (MFA), used to train the speaker-dependent phoneme classifier and duration predictor.
- VoxCeleb2: Over 1 million utterances across 6,112 speakers, used to train the speaker verification encoder.
No new datasets were introduced.
Key Insights and Ideas
- Decoupled Text Conditioning: Text understanding and acoustic distribution modeling do not need to be learned jointly. A single multi-speaker classifier trained on LibriTTS can steer an unconditional diffusion model trained exclusively on raw audio from an entirely unseen speaker.
- Handling Long-Form and Unsegmented Audio: Because unconditional diffusion does not need sentence boundaries or transcript alignment, arbitrary audio slicing (e.g., 5-second random windows from raw audiobooks) suffices for training.
- Norm-Based Scaling is Essential: In audio diffusion, the unconditional score norm grows steeply near $t=0$. Naive classifier guidance fails (producing high character error rates), whereas scaling the gradient norm relative to the score norm keeps guidance effective across all noise schedules.
Key Results
- Comparable to Supervised Baselines: On LJSpeech, Guided-TTS without target transcripts (Guided-TTS-U) achieved a 5-scale MOS of 4.18 ± 0.07, matching fully transcribed Guided-TTS-T (4.18 ± 0.07), Grad-TTS (4.25 ± 0.09), and Glow-TTS (4.10 ± 0.10).
- High Intelligibility: Guided-TTS-U achieved a 1.26% CER and 4.61% WER (using a Conformer ASR model), outperforming supervised Grad-TTS (1.31% CER / 5.55% WER).
- Zero-Transcript Generalization: Guided-TTS successfully generalized to unsegmented Blizzard 2013 audio (3.85 MOS) and multiple Hi-Fi TTS speakers (3.72–3.97 MOS).
- Sampling Speed: Generates audio in real time on an NVIDIA RTX 8000 using $N=50$ steps, with a real-time factor (RTF) of 0.56 (Guided-TTS-T) and 0.63 (Guided-TTS-U).
Limitations and Future Work
- Large Model Capacity Required: Downscaling the unconditional U-Net architecture to match the smaller Grad-TTS decoder caused a sharp drop in MOS (down to 4.01) and increased CER/WER (1.65% / 5.97%), indicating that unconditional speech generation demands substantially higher model capacity than conditional models.
- Reliance on External Multi-Speaker Transcriptions: While the target voice requires no text, the guidance framework still depends on a large paired multi-speaker corpus (LibriTTS) and forced-alignment tools (MFA).
- Sampling Latency: Like most diffusion models, reverse-step iterative sampling ($N=50$) is inherently slower than single-pass feed-forward or GAN-based acoustic models.
Summary updated