TTS-Transducer: End-to-End Speech Synthesis with Neural Transducer

Vladimir Bataev, Subhankar Ghosh, Vitaly Lavrukhin, Jason Li

TL;DR

This paper introduces TTS-Transducer, a novel end-to-end text-to-speech (TTS) system that leverages the strengths of neural transducers and audio codecs. By using a transducer to align text with the first codebook of audio tokens and a non-autoregressive Transformer for subsequent codes, it achieves high-quality, robust speech synthesis that is competitive with state-of-the-art models, even on challenging texts, without requiring massive pretraining.

What Problem Does This Paper Address?

Generating natural-sounding speech from text (TTS) is a complex sequence-to-sequence problem. A key challenge is aligning the discrete units of text (phonemes or characters) with the continuous, frame-based nature of speech. Traditional autoregressive TTS models implicitly learn this alignment but can suffer from issues like hallucination or repetition. Non-autoregressive models often rely on explicit duration predictors, which can be brittle.

Recent advancements in neural audio codecs have enabled TTS systems to predict discrete audio tokens instead of continuous spectrograms, simplifying the pipeline and allowing the application of language modeling techniques. However, audio codecs often produce multiple tokens per frame, leading to a massive search space for traditional transducer models, making them computationally expensive. This paper aims to bridge this gap by proposing a more efficient transducer-based approach for discrete audio token prediction in TTS.

Methods

The core innovation is the TTS-Transducer architecture, which tackles the multi-token-per-frame problem by splitting the prediction into two stages:

  1. Transducer for First Codebook: A neural transducer (specifically, an RNNT-based architecture) is used to predict the tokens for the first codebook of the audio codec. This component learns a monotonic alignment between the input text and these initial audio tokens. This is a key departure, as transducers are naturally suited for monotonic alignment.
  2. Non-Autoregressive Transformer (RCH) for Residual Codebooks: A separate, non-autoregressive Transformer module, termed the Residual Codebook Head (RCH), is responsible for predicting the remaining audio tokens for subsequent codebooks. This RCH module uses the aligned encoder output from the transducer and the previously predicted codes to iteratively generate the rest of the audio tokens for each frame.

Both components are conditioned on a speaker embedding, generated using Global Style Tokens (GST) from a reference speech sample, allowing for speaker style control. The entire system is trained end-to-end.

Datasets

  • LibriTTS-R: An improved 24 kHz version of the LibriTTS dataset was used for training and evaluating on seen speakers. This dataset contains a diverse set of speakers reading English audiobooks.
  • LibriTTS-R dev-clean subset: Used for evaluating performance on unseen speakers.
  • VCTK dataset: Used for evaluating out-of-domain generalization capabilities.

The paper also mentions using reference audios of 3-5 seconds for speaker embedding generation.

Key Insights and Ideas

  • Transducers as an Alignment Tool for Discrete Audio: The paper demonstrates that transducers, known for their robustness in ASR, are a powerful tool for learning monotonic alignments in TTS when applied to discrete audio tokens. This bypasses the need for explicit duration predictors.
  • Decomposing Multi-Codebook Prediction: Splitting the prediction of multiple codebooks into a transducer-first stage and a non-autoregressive residual stage significantly reduces computational complexity compared to a single transducer trying to predict all tokens.
  • Codec Agnosticism: The TTS-Transducer architecture shows robustness and adaptability to different residual vector quantization (RVQ) audio codecs, highlighting its flexibility.
  • Competitive Performance with Less Data: The proposed model achieves state-of-the-art results, particularly in intelligibility on challenging texts, without the massive pretraining datasets often required by other large language model-based TTS systems.

Key Results

  • Character Error Rate (CER): The TTS-Transducer achieved a CER of 3.94% on challenging texts using the EnCodec codec and IPA tokenization, surpassing larger state-of-the-art models trained on significantly more data.
  • Word Error Rate (WER) and SSIM: The models consistently produced intelligible speech with low WER and maintained good speaker similarity (SSIM) across various audio codecs (EnCodec, NeMo-Codec, DAC) and tokenization methods (BPE, IPA). IPA tokenization generally led to better intelligibility.
  • Naturalness (MOS): The proposed BPE-based TTS-Transducer models were comparable to state-of-the-art models like Bark and VALL-E-X in naturalness, with MOS scores around 3.8.
  • Model Size Impact: Ablation studies showed that increasing the number of layers in the encoder, predictor, and residual codebook head generally improved performance, with the full 12-layer encoder and RCH showing the best results.

Limitations and Future Work

  • Decoding Strategy: While nucleus sampling is used for the first codebook, the remaining codebooks are predicted greedily during inference. The authors note that using an autoregressive Transformer as the second component could enable streaming mode, which is left for future work.
  • Computational Cost: Although the proposed method is more efficient than a single transducer for all codebooks, the RCH still involves iterative prediction, which can be computationally intensive during inference.
  • Future Work: The authors suggest exploring streaming inference and further investigating the codec-agnostic nature of their model. They also plan to open-source their code in the NeMo toolkit.

Summary updated

Abstract

This work introduces TTS-Transducer - a novel architecture for text-to-speech, leveraging the strengths of audio codec models and neural transducers. Transducers, renowned for their superior quality and robustness in speech recognition, are employed to learn monotonic alignments and allow for avoiding using explicit duration predictors. Neural audio codecs efficiently compress audio into discrete codes, revealing the possibility of applying text modeling approaches to speech generation. However, the complexity of predicting multiple tokens per frame from several codebooks, as necessitated by audio codec models with residual quantizers, poses a significant challenge. The proposed system first uses a transducer architecture to learn monotonic alignments between tokenized text and speech codec tokens for the first codebook. Next, a non-autoregressive Transformer predicts the remaining codes using the alignment extracted from transducer loss. The proposed system is trained end-to-end. We show that TTS-Transducer is a competitive and robust alternative to contemporary TTS systems.
BibTeX
@inproceedings{bataev2025tts,
  author = {Vladimir Bataev and Subhankar Ghosh and Vitaly Lavrukhin and Jason Li},
  title = {{TTS-Transducer: End-to-End Speech Synthesis with Neural Transducer}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10890256},
}
Figure