Autotts: End-to-End Text-to-Speech Synthesis Through Differentiable Duration Modeling

Bac Nguyen, Fabien Cardinaux, Stefan Uhlich

TL;DR

This paper introduces AutoTTS, a novel end-to-end text-to-speech (TTS) system that directly generates audio waveforms from text. It achieves this by proposing a "differentiable duration modeling" technique, which allows the model to learn alignments between text and speech without relying on external, pre-trained alignment models, leading to a simpler and more efficient training pipeline.

What Problem Does This Paper Address?

Traditional text-to-speech (TTS) systems often fall into two categories: autoregressive models (like Tacotron) that produce natural speech but are slow due to their sequential nature, and non-autoregressive (parallel) models that are fast but struggle with alignment and can produce less natural speech. A key challenge for parallel TTS is learning how to align input text (phonemes) to the output audio frames. Existing parallel models often rely on external alignment models or information from slower autoregressive "teacher" models, which are not jointly optimized with the main TTS model. This can lead to suboptimal alignments and a more complex training setup. AutoTTS aims to solve this by enabling the TTS model to learn these crucial alignments directly from data in an end-to-end fashion.

Methods

The core innovation of AutoTTS is its differentiable duration modeling. Instead of predicting a fixed duration for each phoneme, it models duration as a stochastic process. This means it calculates the probability of a phoneme lasting a certain number of audio frames. This probabilistic approach allows gradients to flow back through the duration prediction mechanism during training, enabling the model to learn these alignments end-to-end.

AutoTTS is built on an encoder-aligner-decoder architecture:

  • Encoder: Converts input phonemes into hidden representations.
  • Aligner: Uses the differentiable duration model to map these hidden representations to a lower-resolution sequence that aligns with the audio.
  • Decoder: Upsamples this aligned sequence into the final raw audio waveform.

The system is trained using an adversarial learning scheme. A discriminator tries to distinguish between real and synthesized audio, while the generator (AutoTTS) learns to produce realistic audio. This is combined with:

  • Adversarial Loss: For realistic audio generation.
  • Length Loss: Encourages the expected total duration of synthesized speech to match the ground truth.
  • Duration Loss: Helps the duration predictor align with the aligner's output.
  • Reconstruction Loss: Ensures the synthesized speech is similar to the ground truth audio in terms of features and mel-spectrograms.

Datasets

The paper uses the LJSpeech dataset for training and evaluation. This is a widely used, single-speaker dataset containing English audio clips and corresponding text.

Key Insights and Ideas

  • End-to-End Alignment Learning: The most significant insight is that a TTS model can learn accurate, monotonic alignments between text and speech without external alignment tools or teacher models. This simplifies the training pipeline considerably.
  • Stochastic Duration Modeling: Treating duration as a probabilistic process is key to enabling end-to-end gradient flow for alignment. This is a departure from simply predicting discrete durations.
  • Simplicity and Efficiency: By integrating alignment learning directly, AutoTTS achieves a much simpler training setup compared to multi-stage or teacher-student approaches, while still delivering competitive performance.

Key Results

  • Speech Quality: AutoTTS achieved a Mean Opinion Score (MOS) of 4.28 ±0.06, which is significantly better than other two-stage models like Tacotron 2 (3.92) and FastSpeech 2 (3.75), and only slightly below the ground truth audio (4.81).
  • Inference Speed: AutoTTS is notably faster than competing methods on both CPU and GPU, thanks to its parallel, end-to-end nature. For example, it synthesizes 10 seconds of speech in 0.12 seconds on a GPU, compared to 1.72 seconds for Tacotron 2.
  • Alignment Quality: The paper shows that AutoTTS can learn hard and monotonic alignment matrices, similar to what is achieved with external aligners.

Limitations and Future Work

  • Model Size: AutoTTS has slightly more parameters than some acoustic models because it generates raw audio directly, whereas others output mel-spectrograms that require a separate vocoder.
  • Future Work: The authors aim to further reduce the model size while maintaining audio quality for deployment on smaller devices. They also suggest exploring their duration-based alignment mechanism for other domains.

Summary updated

Abstract

Parallel text-to-speech (TTS) models have recently enabled fast and highly-natural speech synthesis. However, they typically require external alignment models, which are not necessarily optimized for the decoder as they are not jointly trained. In this paper, we propose a differentiable duration method for learning monotonic alignments between input and output sequences. Our method is based on a soft-duration mechanism that optimizes a stochastic process in expectation. Using this differentiable duration method, we introduce AutoTTS, a direct text-to-waveform speech synthesis model. AutoTTS enables high-fidelity speech synthesis through a combination of adversarial training and matching the total ground-truth duration. Experimental results show that our model obtains competitive results while enjoying a much simpler training pipeline. Audio samples are available online <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
BibTeX
@inproceedings{nguyen2023autotts,
  author = {Bac Nguyen and Fabien Cardinaux and Stefan Uhlich},
  title = {{Autotts: End-to-End Text-to-Speech Synthesis Through Differentiable Duration Modeling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=Auia1wz4qeX},
}
Figure