Non-Autoregressive Neural Text-to-Speech

Kainan Peng, Wei Ping, Zhao Song, Kexin ZhaoICML 2020 · OpenReview

Introduction

Most state-of-the-art Text-to-Speech (TTS) systems, such as Tacotron 2 or Deep Voice 3, rely on autoregressive architectures. These models generate each frame of audio conditioned on all previously generated frames. While this approach yields high-quality, natural-sounding speech, it introduces a severe computational bottleneck: it cannot leverage the parallel processing capabilities of modern GPUs during synthesis. Furthermore, autoregressive models often suffer from "exposure bias"—a discrepancy between training (using ground-truth previous frames) and inference (using potentially erroneous self-predictions)—which can lead to stability issues like skipping or repeating words in complex sentences.

To address these limitations, researchers have proposed ParaNet, a fully convolutional, non-autoregressive TTS model that generates mel-spectrograms in parallel rather than frame-by-frame. By distilling attention from a teacher model and refining it layer-by-layer, ParaNet achieves a massive 46.7x speedup over its autoregressive predecessor while maintaining comparable speech quality.

Methodology: The ParaNet Architecture

ParaNet converts text to spectrograms in a single feed-forward pass, fundamentally overhauling traditional TTS pipelines.

1. Architecture Overhaul

Built upon the foundation of Deep Voice 3, ParaNet introduces critical structural changes. Unlike standard decoders that use "causal" convolutions (restricted to past context), ParaNet employs non-causal convolutions. This allows the model to analyze both future and past contexts simultaneously, predicting the entire spectrogram at once.

Because the decoder is non-causal and processes the entire context globally, the authors found that the separate post-processing "Converter" network—typically required in previous models to refine the output—could be entirely removed.

Comparison of the baseline autoregressive model (a) and the proposed ParaNet (b), highlighting the removal of the Converter and the use of attention distillation from a teacher model.

2. Parallel Attention & Distillation

The most significant challenge in non-autoregressive TTS is "alignment": determining which part of the input text corresponds to which segment of the output audio without the benefit of sequential generation. ParaNet solves this via Attention Distillation. The model learns to align text and audio by mimicking the attention distributions of a pre-trained, autoregressive "teacher" model.

The ParaNet Decoder architecture, illustrating the iterative refinement of attention across multiple stacked blocks.

Crucially, this alignment is refined iteratively. As shown in the decoder architecture, the model stacks multiple attention blocks. The first layer starts with a rough positional guess, and subsequent layers progressively sharpen the focus until a clear, precise alignment emerges.

3. WaveVAE for Vocoding

To convert the generated spectrograms into actual sound waves, the authors also address the "vocoder" stage. Fast vocoders typically require a complex distillation process from a slow WaveNet teacher. To simplify this, the authors propose WaveVAE, a Variational Autoencoder framework that allows training fast vocoders from scratch, streamlining the overall TTS pipeline.

Key Insights and Results

The model was trained and evaluated on an internal English speech dataset comprising approximately 20 hours of high-quality audio from a single female speaker (sampled at 48 kHz, downsampled to 24 kHz).

  • Massive Speedups: ParaNet is exceptionally fast, achieving synthesis speeds 254.6 times faster than real-time on a standard NVIDIA 1080 Ti GPU. Compared directly to Deep Voice 3, it offers a 46.7x speedup.
  • Competitive Quality: In Mean Opinion Score (MOS) evaluations, ParaNet paired with a WaveNet vocoder scored 4.01, closely trailing the autoregressive baseline's score of 4.09.
  • Enhanced Robustness: Surprisingly, the non-autoregressive approach proved more stable on challenging sentences (e.g., those containing numerous numbers or abbreviations). Because it does not rely on its own previous outputs, it avoids spiraling into cascading errors. On a test set of 100 "hard" sentences, ParaNet produced only 12 attention errors (skips/repeats) compared to 37 from Deep Voice 3.
  • Attention Masking: The authors found that applying a "windowing" trick during inference—restricting the model's attention to a small window around the expected text position rather than the whole sentence—significantly reduced pronunciation errors and mumbling.

Limitations and Future Work

While ParaNet represents a major breakthrough for the TTS "front-end" (text-to-spectrogram generation), the authors acknowledge that parallel vocoders (the "back-end") still lag behind slower, autoregressive models in robustness. The proposed WaveVAE simplified training but yielded a lower MOS (3.31) compared to distilled vocoder versions. Future research will likely focus on closing this quality gap for distillation-free vocoders and exploring highly efficient architectures for production environments.

Summary updated

Abstract

In this work, we propose ParaNet, a non-autoregressive seq2seq model that converts text to spectrogram. It is fully convolutional and brings 46.7 times speed-up over the lightweight Deep Voice 3 at...
BibTeX
@inproceedings{peng2020non,
  author = {Kainan Peng and Wei Ping and Zhao Song and Kexin Zhao},
  title = {{Non-Autoregressive Neural Text-to-Speech}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=83TqgtmvjlI},
}
Figure