Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS

Mutian He, Yan Deng, Lei He

Abstract

Neural TTS has demonstrated strong capabilities to generate human-like speech with high quality and naturalness, while its generalization to out-of-domain texts is still a challenging task, with regard to the design of attention-based sequence-to-sequence acoustic modeling. Various errors occur in those inputs with unseen context, including attention collapse, skipping, repeating, etc., which limits the broader applications. In this paper, we propose a novel stepwise monotonic attention method in sequence-to-sequence acoustic modeling to improve the robustness on out-of-domain inputs. The method utilizes the strict monotonic property in TTS with constraints on monotonic hard attention that the alignments between inputs and outputs sequence must be not only monotonic but allowing no skipping on inputs. Soft attention could be used to evade mismatch between training and inference. The experimental results show that the proposed method could achieve significant improvements in robustness on out-of-domain scenarios for phoneme-based models, without any regression on the in-domain naturalness test.

Introduction

Modern neural Text-to-Speech (TTS) systems, such as Tacotron2, have achieved remarkable success in producing natural, human-like speech. However, these models often exhibit a "glass jaw" when encountering out-of-domain (OOD) text. When faced with long sentences, complex technical jargon, or semantically unusual phrases, standard attention mechanisms can fail catastrophically. These failures typically manifest as attention collapse (unintelligible gibberish), word skipping, or infinite repetition of phonemes.

The root of this instability lies in the flexibility of standard soft attention. While this flexibility enables expressive prosody, it allows the model to attend to any part of the input sequence at any time. In unfamiliar contexts, this lack of constraint leads to misalignments that break the speech synthesis process.

Methodology: Stepwise Monotonic Attention (SMA)

To address these robustness issues, we propose that TTS alignment must adhere to three fundamental principles: locality, monotonicity, and completeness. While existing Monotonic Attention (MA) mechanisms enforce the first two, they often fail at completeness—the requirement that every input token must be processed. Standard MA allows the model to skip an arbitrary number of input tokens in a single decoding step.

Stepwise Monotonic Attention (SMA) introduces a stricter constraint: at each decoding step, the model can either remain on the current input token or advance to the immediately following token. This "stepwise" constraint ensures that no phonemes are skipped, effectively enforcing completeness.

Comparison of Soft, Monotonic, and Stepwise Monotonic Attention mechanisms.

Furthermore, SMA addresses the "context mismatch" between training and inference. Traditionally, models are trained with soft attention but may use hard attention during inference. Because SMA's constraints are so rigorous, the learned attention weights become highly discrete. This allows for the use of soft attention during inference without the typical stability risks, bridging the gap between training and deployment.

Experimental Evaluation

The proposed SMA method was evaluated using a 20.8-hour proprietary English speech corpus. Testing was conducted across three distinct sets:

  1. General Sentences: 80 in-domain sentences.
  2. News Test Set: 80 out-of-domain sentences.
  3. Intelligibility Test Set: 292 complex or semantically abnormal utterances designed to stress-test the model's robustness.

Robustness and Naturalness

The results demonstrate a dramatic improvement in stability. On the Intelligibility Test Set, the baseline Tacotron2 model suffered a 28.42% unintelligible rate. SMA (hard) reduced this failure rate to a mere 0.68%. In a large-scale word error test, standard Monotonic Attention skipped 2,660 words, whereas SMA (soft) skipped only 10.

Spectrogram comparison between hard and soft Stepwise Monotonic Attention.

Crucially, this robustness does not come at the cost of quality. Crowdsourced naturalness tests (Mean Opinion Score) showed that SMA-based models perform on par with standard Tacotron2 for in-domain sentences.

Conclusion

Stepwise Monotonic Attention provides a robust solution for neural TTS by enforcing strict alignment constraints that mirror the physical reality of speech. By ensuring that every token is processed in sequence, SMA eliminates common failure modes like skipping and collapse. Future work will explore the application of SMA to character-based models and other sequence-to-sequence tasks requiring strict monotonic alignment, such as online handwriting generation.

Summary updated

Abstract

Neural TTS has demonstrated strong capabilities to generate human-like speech with high quality and naturalness, while its generalization to out-of-domain texts is still a challenging task, with regard to the design of attention-based sequence-to-sequence acoustic modeling. Various errors occur in those inputs with unseen context, including attention collapse, skipping, repeating, etc., which limits the broader applications. In this paper, we propose a novel stepwise monotonic attention method in sequence-to-sequence acoustic modeling to improve the robustness on out-of-domain inputs. The method utilizes the strict monotonic property in TTS with constraints on monotonic hard attention that the alignments between inputs and outputs sequence must be not only monotonic but allowing no skipping on inputs. Soft attention could be used to evade mismatch between training and inference. The experimental results show that the proposed method could achieve significant improvements in robustness on out-of-domain scenarios for phoneme-based models, without any regression on the in-domain naturalness test.
BibTeX
@inproceedings{he2019robust,
  author = {Mutian He and Yan Deng and Lei He},
  title = {{Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=9p0VqfNXWCd},
}
Figure