Guided-TTS:Text-to-Speech with Untranscribed Speech

Heeseung Kim, Sungwon Kim, Sungroh Yoon

TL;DR

Guided-TTS introduces a framework to build high-quality text-to-speech (TTS) systems for target speakers using only raw, untranscribed, and unsegmented audio. By decoupling speech generation into an unconditional diffusion model trained on audio chunks and a phoneme classifier trained on separate multi-speaker data, it achieves speech quality and intelligibility on par with fully supervised TTS baselines.

What Problem Does This Paper Address?

State-of-the-art neural TTS models generally require dozens of hours of high-quality paired <speech, transcript> data for each target speaker. While vast amounts of untranscribed speech exist (e.g., audiobooks, podcasts), using them directly has traditionally required labor-intensive preprocessing: segmenting continuous audio into sentence-level chunks and generating highly accurate transcripts.

Guided-TTS tackles this bottleneck by asking: Can we train an unconditional generative speech model entirely without transcripts or sentence alignments, and then guide it to speak specific text at inference time?

Methods

Guided-TTS splits the TTS pipeline into four components:

  1. Unconditional Diffusion Model (DDPM): Models the unconditional distribution of mel-spectrograms $p(X)$ using continuous score-based stochastic differential equations (SDEs). Because it requires no text, it trains directly on random 5-second audio crops from the target speaker. The architecture is a 2D U-Net adapted from standard image diffusion models.
  2. Framewise Phoneme Classifier: A WaveNet-like classifier trained on noisy mel-spectrograms $X_t$ to predict frame-level phoneme labels $\hat{y}$. It conditions on diffusion timestep $t$ and speaker embeddings.
  3. Duration Predictor & Speaker Encoder: A Glow-TTS-style duration predictor maps input phoneme sequences to framewise phoneme sequences $\hat{y}$. A pre-trained speaker encoder (trained with GE2E loss) extracts speaker embeddings $e_S$ to make the classifier and duration predictor speaker-adaptive.
  4. Norm-Based Guidance: Standard classifier guidance adds a scaled gradient $\nabla_{X_t} \log p(\hat{y}|X_t)$ to the score during reverse sampling. However, the authors observed that the unconditional score norm $\|\nabla_{X_t} \log p_\theta(X_t)\|$ spikes dramatically as $t \to 0$ (up to ~70$\times$ the classifier gradient norm), rendering the text guidance ineffective near clean data and causing severe mispronunciations. Norm-based guidance dynamically rescales the classifier gradient in proportion to the unconditional score norm at each step, ensuring stable text conditioning throughout sampling.
                      +-------------------+
                      | Text / Phonemes y |
                      +---------+---------+
                                |
                                v
                      +-------------------+
                      | Duration Predictor|
                      +---------+---------+
                                |
                                v
                      +-------------------+
                      | Framewise Labels ŷ|
                      +---------+---------+
                                |
Random Noise X_T                v
       |              +-------------------+
       +------------> |Phoneme Classifier | ---> Classifier Gradient
       |              +-------------------+               |
       v                                                  v
+--------------+                                 +-----------------+
|Unconditional | ---> Unconditional Score -----> | Norm-Based      | ---> Speech Mel X_0
|     DDPM     |                                 | Guidance Solver |      (to HiFi-GAN)
+--------------+                                 +-----------------+

Datasets

  • Target Speaker Unpaired Audio:
  • LJSpeech: 24 hours of single-speaker female speech (used both transcribed and speech-only for comparisons).
  • Hi-Fi TTS: Multi-speaker dataset; three speakers were evaluated (two male, one female; 27.3 to 58.0 hours).
  • Blizzard 2013: 147 hours of audiobook speech read by a single female speaker (evaluated using random 5-second crops from unsegmented data).
  • Guidance & Multi-Speaker Pretraining:
  • LibriTTS: ~585 hours across 2,456 speakers, aligned with Montreal Forced Aligner (MFA), used to train the speaker-dependent phoneme classifier and duration predictor.
  • VoxCeleb2: Over 1 million utterances across 6,112 speakers, used to train the speaker verification encoder.

No new datasets were introduced.

Key Insights and Ideas

  • Decoupled Text Conditioning: Text understanding and acoustic distribution modeling do not need to be learned jointly. A single multi-speaker classifier trained on LibriTTS can steer an unconditional diffusion model trained exclusively on raw audio from an entirely unseen speaker.
  • Handling Long-Form and Unsegmented Audio: Because unconditional diffusion does not need sentence boundaries or transcript alignment, arbitrary audio slicing (e.g., 5-second random windows from raw audiobooks) suffices for training.
  • Norm-Based Scaling is Essential: In audio diffusion, the unconditional score norm grows steeply near $t=0$. Naive classifier guidance fails (producing high character error rates), whereas scaling the gradient norm relative to the score norm keeps guidance effective across all noise schedules.

Key Results

  • Comparable to Supervised Baselines: On LJSpeech, Guided-TTS without target transcripts (Guided-TTS-U) achieved a 5-scale MOS of 4.18 ± 0.07, matching fully transcribed Guided-TTS-T (4.18 ± 0.07), Grad-TTS (4.25 ± 0.09), and Glow-TTS (4.10 ± 0.10).
  • High Intelligibility: Guided-TTS-U achieved a 1.26% CER and 4.61% WER (using a Conformer ASR model), outperforming supervised Grad-TTS (1.31% CER / 5.55% WER).
  • Zero-Transcript Generalization: Guided-TTS successfully generalized to unsegmented Blizzard 2013 audio (3.85 MOS) and multiple Hi-Fi TTS speakers (3.72–3.97 MOS).
  • Sampling Speed: Generates audio in real time on an NVIDIA RTX 8000 using $N=50$ steps, with a real-time factor (RTF) of 0.56 (Guided-TTS-T) and 0.63 (Guided-TTS-U).

Limitations and Future Work

  • Large Model Capacity Required: Downscaling the unconditional U-Net architecture to match the smaller Grad-TTS decoder caused a sharp drop in MOS (down to 4.01) and increased CER/WER (1.65% / 5.97%), indicating that unconditional speech generation demands substantially higher model capacity than conditional models.
  • Reliance on External Multi-Speaker Transcriptions: While the target voice requires no text, the guidance framework still depends on a large paired multi-speaker corpus (LibriTTS) and forced-alignment tools (MFA).
  • Sampling Latency: Like most diffusion models, reverse-step iterative sampling ($N=50$) is inherently slower than single-pass feed-forward or GAN-based acoustic models.

Summary updated

Abstract

Most neural text-to-speech (TTS) models require $\langle$speech, transcript$\rangle$ paired data from the desired speaker for high-quality speech synthesis, which limits the usage of large amounts of untranscribed data for training. In this work, we present Guided-TTS, a high-quality TTS model that learns to generate speech from untranscribed speech data. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier for text-to-speech. By modeling the unconditional distribution for speech, our model can utilize the untranscribed data for training. For text-to-speech synthesis, we guide the generative process of the unconditional DDPM via phoneme classification to produce mel-spectrograms from the conditional distribution given transcript. We show that Guided-TTS achieves comparable performance with the existing methods without any transcript for LJSpeech. Our results further show that a single speaker-dependent phoneme classifier trained on multispeaker large-scale data can guide unconditional DDPMs for various speakers to perform TTS.
BibTeX
@inproceedings{kim2022guided,
  author = {Heeseung Kim and Sungwon Kim and Sungroh Yoon},
  title = {{Guided-TTS:Text-to-Speech with Untranscribed Speech}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=W3SxRos66mT},
}

Loading related content...

Figure