Abstract
This paper proposes a forward attention method for the sequence-to-sequence acoustic modeling of speech synthesis. This method is motivated by the nature of the monotonic alignment from phone sequences to acoustic sequences. Only the alignment paths that satisfy the monotonic condition are taken into consideration at each decoder timestep. The modified attention probabilities at each timestep are computed recursively using a forward algorithm. A transition agent for forward attention is further proposed, which helps the attention mechanism to make decisions whether to move forward or stay at each decoder timestep. Experimental results show that the proposed forward attention method achieves faster convergence speed and higher stability than the baseline attention method. Besides, the method of forward attention with transition agent can also help improve the naturalness of synthetic speech and control the speed of synthetic speech effectively.
Introduction
Modern end-to-end speech synthesis systems, such as Tacotron, typically utilize an encoder-decoder architecture with a content-based attention mechanism. While flexible, standard attention often fails to capture the inherent monotonicity of speech—where phonemes are produced in a strict sequential order. This lack of constraint frequently leads to artifacts like word skipping or repetition, and makes training convergence slow as the model must learn the monotonic property from scratch.
Methods: Enforcing Monotonicity
The core of the proposed approach is Forward Attention (FA). Unlike standard attention which can attend to any part of the input sequence at any time, FA restricts the alignment to paths that move forward or stay in place.
The mechanism is implemented through three key components:
- Forward Algorithm: Attention probabilities are computed recursively. The probability of attending to a phoneme at time $t$ depends on the attention state at $t-1$, ensuring that the model only considers the current or the next phoneme.
- Transition Agent (TA): A small integrated neural network predicts a transition probability ($u_t$), representing the likelihood of moving to the next phoneme at the current timestep.
- Product-of-Experts (PoE): The final attention weight combines standard content-based scores with these monotonic constraints.
Results and Discussion
The researchers evaluated the model using a 19.8-hour Mandarin speech database featuring a professional female speaker. The results demonstrate significant improvements in both training efficiency and synthesis stability.
Convergence and Stability
One of the most striking benefits of Forward Attention is the speed of convergence. As shown in the attention heatmaps, the FA-TA model establishes a clear monotonic alignment as early as epoch 3, whereas the baseline model remains noisy even at epoch 7.
In terms of stability, the baseline model suffered 54 failures (skips or repeats) in a 120-utterance test set. In contrast, the FA model reduced this to 5 failures, and the addition of convolutional features eliminated failures entirely.
Naturalness and Speed Control
Subjective listening tests indicated that the FA-TA system significantly outperformed the baseline in naturalness, achieving scores comparable to traditional LSTM-based systems. Furthermore, the Transition Agent allows for explicit speed control. By adding a bias to the transition probability $u_t$, the researchers could adjust the speech rate by approximately ±10% without sacrificing audio quality.
Conclusion
Forward Attention provides a robust solution to the alignment issues common in seq2seq speech synthesis. By incorporating the monotonic nature of speech as a mathematical prior, the model achieves faster training, higher stability, and improved naturalness. Future work will explore applying this mechanism to other monotonic tasks like Automatic Speech Recognition (ASR).
Summary updated