Forward Attention in Sequence- To-Sequence Acoustic Modeling for Speech Synthesis

Jingxuan Zhang, Zhen-Hua Ling, Li-Rong Dai

Abstract

This paper proposes a forward attention method for the sequence-to-sequence acoustic modeling of speech synthesis. This method is motivated by the nature of the monotonic alignment from phone sequences to acoustic sequences. Only the alignment paths that satisfy the monotonic condition are taken into consideration at each decoder timestep. The modified attention probabilities at each timestep are computed recursively using a forward algorithm. A transition agent for forward attention is further proposed, which helps the attention mechanism to make decisions whether to move forward or stay at each decoder timestep. Experimental results show that the proposed forward attention method achieves faster convergence speed and higher stability than the baseline attention method. Besides, the method of forward attention with transition agent can also help improve the naturalness of synthetic speech and control the speed of synthetic speech effectively.

Introduction

Standard sequence-to-sequence (seq2seq) models, such as Tacotron, typically rely on content-based attention mechanisms. While flexible, this flexibility can be a disadvantage in Text-to-Speech (TTS) synthesis. Unlike machine translation, where word order can change significantly between languages, speech synthesis is inherently monotonic: phones are produced in a strict chronological order. When a standard attention mechanism "wanders," it leads to common synthesis errors like repeating phones, skipping words, or slow training convergence as the model struggles to learn the basic left-to-right progression.

Methods

To address these issues, the researchers introduce two primary modifications to the attention mechanism:

Forward Attention (FA)

Inspired by Hidden Markov Models (HMMs), Forward Attention enforces a monotonic constraint. Instead of allowing the attention to jump anywhere, the probability of attending to a specific phone at time $t$ is calculated recursively based on the attention state at time $t-1$. Specifically, the model only considers the current phone or the immediately preceding one as valid transitions.

Trellis diagram illustrating the monotonic alignment paths in Forward Attention.

Transition Agent (TA)

The Transition Agent (TA) is a small neural network that predicts the probability of moving to the next phone versus staying on the current one at each decoder step. This "pacing" mechanism allows the model to implicitly learn the duration of each phone, eliminating the need for separate duration models.

Key Results

The proposed methods were evaluated on a 19.8-hour Mandarin speech database.

Faster Convergence and Stability

One of the most striking results is the speed of convergence. As shown in the attention alignment heatmaps, the FA-TA model develops a sharp, diagonal alignment as early as the 3rd epoch, whereas the baseline model remains significantly more diffused and unstable even at later stages.

Comparison of attention alignments between Baseline and FA-TA models across training epochs, showing faster convergence for FA-TA.

In terms of stability, the Forward Attention models demonstrated a dramatic reduction in synthesis failures. In a test of 120 utterances, the baseline model suffered from 54 failures (skips or repetitions). In contrast, the FA and FA-TA models, when combined with convolutional features, achieved zero failures.

Naturalness and Speed Control

Human listening tests indicated that the FA-TA system significantly outperformed the baseline in terms of naturalness. Furthermore, the Transition Agent provides a unique benefit: by adjusting the transition probability during inference, the system can control the speaking rate (speeding up or slowing down by over 10%) without degrading audio quality.

Conclusion

By incorporating the monotonic nature of speech as a structural prior, Forward Attention significantly improves the robustness and training efficiency of seq2seq TTS models. The addition of a Transition Agent further enhances naturalness and provides a mechanism for fine-grained speed control. Future work will explore applying these monotonic constraints to other sequence-to-sequence tasks like Automatic Speech Recognition (ASR).

Summary updated

Abstract

This paper proposes a forward attention method for the sequence-to-sequence acoustic modeling of speech synthesis. This method is motivated by the nature of the monotonic alignment from phone sequences to acoustic sequences. Only the alignment paths that satisfy the monotonic condition are taken into consideration at each decoder timestep. The modified attention probabilities at each timestep are computed recursively using a forward algorithm. A transition agent for forward attention is further proposed, which helps the attention mechanism to make decisions whether to move forward or stay at each decoder timestep. Experimental results show that the proposed forward attention method achieves faster convergence speed and higher stability than the baseline attention method. Besides, the method of forward attention with transition agent can also help improve the naturalness of synthetic speech and control the speed of synthetic speech effectively.
BibTeX
@inproceedings{zhang2018forward,
  author = {Jingxuan Zhang and Zhen-Hua Ling and Li-Rong Dai},
  title = {{Forward Attention in Sequence- To-Sequence Acoustic Modeling for Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=9wBtYL2osK},
}
Figure