FeatherTTS: Robust and Efficient attention based Neural TTS

Qiao Tian, Zewang Zhang, Chao Liu, Heng Lu, Linghui Chen, Bin Wei, Pujiang He, Shan Liu

Abstract

Attention based neural TTS is elegant speech synthesis pipeline and has shown a powerful ability to generate natural speech. However, it is still not robust enough to meet the stability requirements for industrial products. Besides, it suffers from slow inference speed owning to the autoregressive generation process. In this work, we propose FeatherTTS, a robust and efficient attention-based neural TTS system. Firstly, we propose a novel Gaussian attention which utilizes interpretability of Gaussian attention and the strict monotonic property in TTS. By this method, we replace the commonly used stop token prediction architecture with attentive stop prediction. Secondly, we apply block sparsity on the autoregressive decoder to speed up speech synthesis. The experimental results show that our proposed FeatherTTS not only nearly eliminates the problem of word skipping, repeating in particularly hard texts and keep the naturalness of generated speech, but also speeds up acoustic feature generation by 3.5 times over Tacotron. Overall, the proposed FeatherTTS can be 35x faster than real-time on a single CPU.

Introduction

Modern neural Text-to-Speech (TTS) systems have achieved human-like naturalness. However, industrial deployment remains challenging due to two primary issues: instability and efficiency. Content-based attention mechanisms often fail on complex or long sentences, leading to word skipping or repetitions. Furthermore, the autoregressive nature of these models results in high computational costs. FeatherTTS addresses these challenges through architectural innovations that enforce alignment stability and reduce computational overhead.

Methods

The FeatherTTS architecture improves upon the standard Tacotron2 framework with three main components:

Gaussian Attention with Monotonicity

Instead of complex attention mixtures, FeatherTTS employs a single Gaussian attention mechanism. To ensure stability, the model enforces strict monotonicity: the attention center (mean) moves forward by adding a non-negative offset at each step. This prevents the model from skipping text or looping back.

Attentive Stop Loss (ATL)

Traditional systems rely on a "stop token" classifier, which often triggers prematurely. FeatherTTS replaces this with an Attentive Stop Loss. This mechanism ties the end of audio generation directly to the attention mean reaching the end of the input phoneme sequence.

Sparse Autoregressive Decoder

To optimize inference speed, block sparsity is applied to the LSTM layers within the decoder. By pruning 90% of the weights, the model significantly reduces the number of operations required per frame without degrading speech quality.

FeatherTTS Architecture Overview

Experimental Results

The system was evaluated on a 20-hour Mandarin corpus. Performance was measured across three dimensions: robustness, naturalness, and speed.

Robustness and Quality

In stress tests using "hard" sentences, FeatherTTS reduced the Word Error Rate (WER) from 4.1% (baseline Tacotron2) to 0.9%. Despite the high sparsity and simplified attention, the Mean Opinion Score (MOS) remained high at 4.33, indicating that naturalness was preserved or slightly improved.

Inference Efficiency

The combination of block sparsity and architectural optimizations led to dramatic speed improvements. FeatherTTS generates acoustic features 3.5x faster than Tacotron2. On a single CPU core, it achieves 35x real-time speed, which increases to 60x when utilizing Intel's BF16 format.

Inference Speed Comparison

Conclusion

FeatherTTS demonstrates that constraints on attention mechanisms can lead to significantly more robust TTS systems. By integrating attentive stop prediction and block sparsity, the model achieves production-grade stability and exceptional inference speeds, making it highly suitable for real-time applications on standard hardware.

Summary updated

Abstract

Attention based neural TTS is elegant speech synthesis pipeline and has shown a powerful ability to generate natural speech. However, it is still not robust enough to meet the stability requirements for industrial products. Besides, it suffers from slow inference speed owning to the autoregressive generation process. In this work, we propose FeatherTTS, a robust and efficient attention-based neural TTS system. Firstly, we propose a novel Gaussian attention which utilizes interpretability of Gaussian attention and the strict monotonic property in TTS. By this method, we replace the commonly used stop token prediction architecture with attentive stop prediction. Secondly, we apply block sparsity on the autoregressive decoder to speed up speech synthesis. The experimental results show that our proposed FeatherTTS not only nearly eliminates the problem of word skipping, repeating in particularly hard texts and keep the naturalness of generated speech, but also speeds up acoustic feature generation by 3.5 times over Tacotron. Overall, the proposed FeatherTTS can be $35$x faster than real-time on a single CPU.
BibTeX
@inproceedings{tian2021feathertts,
  author = {Qiao Tian and Zewang Zhang and Chao Liu and Heng Lu and Linghui Chen and Bin Wei and Pujiang He and Shan Liu},
  title = {{FeatherTTS: Robust and Efficient attention based Neural TTS}},
  booktitle = {SSW},
  year = {2021},
  doi = {10.21437/ssw.2021-35},
}
Figure