Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance

Heeseung Kim, Sung‐Won Kim, Sungroh Yoon

Abstract

We propose Guided-TTS, a high-quality text-to-speech (TTS) model that does not require any transcript of target speaker using classifier guidance. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier for classifier guidance. Our unconditional diffusion model learns to generate speech without any context from untranscribed speech data. For TTS synthesis, we guide the generative process of the diffusion model with a phoneme classifier trained on a large-scale speech recognition dataset. We present a norm-based scaling method that reduces the pronunciation errors of classifier guidance in Guided-TTS. We show that Guided-TTS achieves a performance comparable to that of the state-of-the-art TTS model, Grad-TTS, without any transcript for LJSpeech. We further demonstrate that Guided-TTS performs well on diverse datasets including a long-form untranscribed dataset.

Introduction

Traditional Text-to-Speech (TTS) models are typically designed as conditional generative models, requiring large datasets of paired audio and transcripts (<audio, transcript>). While high-quality raw audio is increasingly abundant through podcasts and audiobooks, obtaining accurate human-verified transcripts remains an expensive and time-consuming bottleneck.

Existing attempts to circumvent this requirement, such as using automated speech recognition (ASR) for "noisy" transcripts or few-shot speaker adaptation, often compromise audio quality or pronunciation accuracy. Guided-TTS addresses this challenge by shifting from conditional modeling to guided unconditional modeling, allowing a model to learn a speaker's voice from raw audio alone.

Methodology

Guided-TTS decouples the learning of voice identity (timbre) from the learning of linguistic structure (pronunciation). The framework consists of four primary components:

  1. Unconditional DDPM (Diffusion Model): The core generative engine, trained on random chunks of untranscribed audio from the target speaker. It learns to produce high-fidelity mel-spectrograms characteristic of the speaker's voice without any textual context.
  2. Phoneme Classifier: Trained on a separate, large-scale transcribed dataset (e.g., LibriSpeech), this component acts as the "brain," recognizing phonemes even within the noisy intermediate steps of the diffusion process.
  3. Duration Predictor: This module determines the temporal length of each phoneme to construct a frame-level sequence for guidance.
  4. Speaker Encoder: Extracts speaker embeddings to maintain identity across different components.
The Guided-TTS architecture, showing the interaction between the unconditional diffusion model, phoneme classifier, and duration predictor.

Norm-Based Guidance

A key innovation in this work is the norm-based scaling method. Standard classifier guidance, which simply multiplies the classifier gradient by a constant, often fails in speech synthesis. The authors observed that as the diffusion process approaches completion (low noise levels), the unconditional score of the diffusion model tends to overpower the classifier's gradient, leading to significant mispronunciations. The proposed norm-based scaling dynamically balances these forces, ensuring that linguistic guidance remains effective throughout the entire generation process.

Experimental Results

The model was evaluated across several datasets, including LJSpeech (female), Hi-Fi TTS (multi-speaker), and the 147-hour Blizzard 2013 audiobook dataset. For Guided-TTS training, all transcripts for these target speakers were ignored.

Performance and Robustness

Guided-TTS demonstrates remarkable parity with state-of-the-art supervised models. On the LJSpeech benchmark, it achieved a Mean Opinion Score (MOS) of 4.25, matching the performance of Grad-TTS despite the latter having access to full transcripts. Furthermore, Guided-TTS achieved a Character Error Rate (CER) of 1.03%, outperforming baseline models that relied on ASR-generated transcripts (which had a CER of 1.16%).

Comparison of Guided-TTS against ground truth and baseline models across multiple datasets.

The model also proved robust when synthesizing "Out-of-Distribution" text, indicating that the phoneme classifier generalizes effectively across different vocal identities. This decoupling allows researchers to leverage massive public datasets for pronunciation while using small amounts of specialized data for specific voices.

Conclusion and Future Work

Guided-TTS represents a significant step toward transcript-free speech synthesis. By leveraging diffusion models and intelligent guidance, it removes the need for expensive paired data without sacrificing quality.

However, challenges remain. Like other diffusion-based models, inference speed is currently slower than one-shot GAN-based alternatives. Additionally, the model's accuracy is heavily tied to the quality of the phoneme classifier; reducing the classifier's training data from 100% to 1% of LibriSpeech causes the CER to jump from 1.03% to 4.24%. Future research will likely focus on improving inference efficiency and capturing the complex prosodic and emotional nuances found in long-form, untranscribed audio.

Summary updated

Abstract

We propose Guided-TTS, a high-quality text-to-speech (TTS) model that does not require any transcript of target speaker using classifier guidance. Guided-TTS combines an unconditional diffusion probabilistic model with a separately trained phoneme classifier for classifier guidance. Our unconditional diffusion model learns to generate speech without any context from untranscribed speech data. For TTS synthesis, we guide the generative process of the diffusion model with a phoneme classifier trained on a large-scale speech recognition dataset. We present a norm-based scaling method that reduces the pronunciation errors of classifier guidance in Guided-TTS. We show that Guided-TTS achieves a performance comparable to that of the state-of-the-art TTS model, Grad-TTS, without any transcript for LJSpeech. We further demonstrate that Guided-TTS performs well on diverse datasets including a long-form untranscribed dataset.
BibTeX
@inproceedings{kim2022guided,
  author = {Heeseung Kim and Sung‐Won Kim and Sungroh Yoon},
  title = {{Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=W3SxRos66mT},
}
Figure