RobuTrans: A Robust Transformer-Based Text-to-Speech Model

Naihan Li, Yanqing Liu, Yu Wu, Shujie Liu, Sheng Zhao, Ming Liu

Abstract

RobuTrans is a robust text-to-speech (TTS) model designed to eliminate common alignment errors like word skipping or repeating in standard Transformer-based architectures. By replacing soft attention with a duration-based hard alignment and a pseudo non-causal decoder, RobuTrans achieves state-of-the-art naturalness while maintaining high robustness on challenging inputs such as URLs and long numerical sequences.

Introduction

Neural TTS models, such as Tacotron 2 and TransformerTTS, have set high standards for speech naturalness. However, these models are often fragile due to their reliance on "soft attention" mechanisms inherited from Neural Machine Translation (NMT). While NMT requires flexibility for word reordering, speech is inherently monotonic. This mismatch leads to significant failure modes, including word skipping, repeating, and "unstoppable" predictions where the model fails to identify the end of a sequence.

Common failure modes in soft attention mechanisms, such as skipping or repeating words.

Furthermore, standard position embeddings often fail to generalize to sequences longer than those encountered during training, leading to garbled output for long texts.

Methods

RobuTrans introduces several architectural innovations to enforce stability and robustness while preserving naturalness.

Duration-Based Hard Attention

To solve alignment issues, RobuTrans replaces soft attention with a Duration Predictor. This module determines the exact number of frames each phoneme should occupy. The encoder states are then replicated (tiled) according to these predicted durations, ensuring a strictly monotonic alignment between text and audio.

Pseudo Non-Causal Attention (PNCA)

While hard alignment ensures robustness, it can lead to robotic-sounding speech if the decoder loses the holistic context provided by soft attention. RobuTrans introduces Pseudo Non-Causal Attention (PNCA). Unlike standard causal decoders that only look at past frames, PNCA allows the decoder to attend to the entire sequence of encoder hidden states for every generated frame. This restores the "global" context necessary for natural prosody without reintroducing the risk of alignment errors.

The RobuTrans architecture featuring the Duration Predictor and Pseudo Non-Causal (PNC) Self-Attention.

Linguistic and Positional Enhancements

RobuTrans utilizes rich linguistic features—including phonemic categories and prosodic markers like tone and break indices—rather than raw phonemes. Additionally, the model replaces fixed position embeddings with 1D Convolutional Neural Networks (CNNs) in the pre-nets to capture relative positional information, enabling better generalization to long sequences.

Experimental Setup

The model was trained on a 20-hour internal US English dataset from a single professional speaker. Evaluation was conducted across two sets:

  1. General Set: 81 sentences for naturalness testing.
  2. Bad-case Set: 327 "stress-test" sentences containing URLs, long number sequences, and complex command lines designed to trigger failures in traditional models.

Key Results

RobuTrans demonstrated superior robustness and competitive naturalness:

  • Robustness: In the 327-sentence stress test, RobuTrans achieved zero failures. In comparison, TransformerTTS failed on 237 cases and Tacotron 2 on 35.
  • Naturalness: RobuTrans achieved a Mean Opinion Score (MOS) of 4.36, statistically equivalent to TransformerTTS (4.37) and Tacotron 2 (4.37). It also outperformed FastSpeech (4.31 MOS).
  • Generalization: The model successfully synthesized sequences up to 30 seconds long, a task where baseline models typically fail due to positional embedding constraints.

Conclusion and Future Work

RobuTrans proves that explicitly constraining TTS models to follow monotonic alignments significantly improves robustness without sacrificing naturalness. By combining duration prediction with pseudo non-causal attention, the model bridges the gap between the stability of traditional systems and the quality of modern neural architectures.

Future research will focus on reducing the dependency on external alignment tools for training the Duration Predictor and exploring faster, non-autoregressive vocoders to replace the computationally intensive WaveNet.

Summary updated

Abstract

Recently, neural network based speech synthesis has achieved outstanding results, by which the synthesized audios are of excellent quality and naturalness. However, current neural TTS models suffer from the robustness issue, which results in abnormal audios (bad cases) especially for unusual text (unseen context). To build a neural model which can synthesize both natural and stable audios, in this paper, we make a deep analysis of why the previous neural TTS models are not robust, based on which we propose RobuTrans (Robust Transformer), a robust neural TTS model based on Transformer. Comparing to TransformerTTS, our model first converts input texts to linguistic features, including phonemic features and prosodic features, then feed them to the encoder. In the decoder, the encoder-decoder attention is replaced with a duration-based hard attention mechanism, and the causal self-attention is replaced with a "pseudo non-causal attention" mechanism to model the holistic information of the input. Besides, the position embedding is replaced with a 1-D CNN, since it constrains the maximum length of synthesized audio. With these modifications, our model not only fix the robustness problem, but also achieves on parity MOS (4.36) with TransformerTTS (4.37) and Tacotron2 (4.37) on our general set.
BibTeX
@inproceedings{li2020robutrans,
  author = {Naihan Li and Yanqing Liu and Yu Wu and Shujie Liu and Sheng Zhao and Ming Liu},
  title = {{RobuTrans: A Robust Transformer-Based Text-to-Speech Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i05.6337},
}
Figure