Forward Attention in Sequence- To-Sequence Acoustic Modeling for Speech Synthesis

Jing-Xuan Zhang, Zhen-Hua Ling, Li-Rong Dai

TL;DR

Researchers have developed a "forward attention" mechanism for sequence-to-sequence (seq2seq) speech synthesis that enforces the naturally monotonic order of speech. By using a recursive forward algorithm and a "transition agent," the model achieves faster convergence, eliminates common artifacts like skipping or repeating words, and enables fine-grained control over speech speed.

What Problem Does This Paper Address?

Modern "end-to-end" speech synthesis (like the famous Tacotron model) uses an encoder-decoder architecture with an attention mechanism. In theory, attention allows the model to "look" at the right part of the input text while generating the corresponding audio frames.

However, standard attention (content-based) is "too flexible." It treats the input as a bag of items rather than an ordered sequence. Because speech is inherently monotonic—we say words in the order they are written—standard models often stumble. They might get "stuck" on a phoneme (repeating it) or "jump" ahead (skipping words). Additionally, these models usually lack a way to explicitly control how fast the synthetic voice speaks without a separate, complex duration model.

Methods

The authors propose Forward Attention (FA), which replaces the standard "look anywhere" approach with a constraint: the model can only stay on the current phoneme or move to the next one.

  1. The Forward Algorithm: Borrowing logic from Hidden Markov Models (HMMs) and Connectionist Temporal Classification (CTC), the model computes attention probabilities recursively. It calculates the probability of being at a specific input position at time $t$ based on where it was at time $t-1$.
  2. The Transition Agent (TA): This is a small, integrated neural network that predicts a transition probability ($u_t$). This value represents the likelihood that the model should move to the next phoneme at the current timestep.
  3. Product-of-Experts (PoE): The final attention weight is essentially a combination of what the "content" suggests (the standard way) and what the "monotonic constraint" allows.

Datasets

The researchers used a Mandarin speech database featuring a professional female speaker.

  • Size: 19.8 hours (13,334 utterances).
  • Split: 12,219 utterances for training, 1,115 for testing.
  • Features: Log magnitude spectrograms (for the seq2seq model) and 41-dimension mel-cepstral coefficients (for the baseline LSTM comparison).

Key Insights and Ideas

  • Monotonicity as a Prior: By baking the "forward-only" nature of speech into the math of the attention mechanism, the model doesn't have to spend thousands of iterations learning that it shouldn't jump backward. This leads to much faster training convergence.
  • The "Aha" Moment on Speed Control: Because the Transition Agent produces a probability ($u_t$) of moving forward, the researchers realized they could manually "nudge" this value. Adding a positive bias makes the model move through phonemes faster (faster speech); a negative bias slows it down.
  • Stability vs. Flexibility: While techniques like "windowing" (limiting the attention range) exist, they are often rigid. Forward attention is a "soft" constraint that is learned and integrated into the recursive flow of the decoder.

Key Results

  • Stability: In a test of 120 utterances, the "Plain Baseline" model had 54 failures (skips/repeats). The Forward Attention (FA) model had only 5, and adding convolutional features reduced that to 0.
  • Naturalness: In listening tests, the FA system with a Transition Agent (FA-TA) significantly outperformed the baseline and achieved naturalness scores comparable to traditional LSTM-based systems that use much richer linguistic input.
  • Speed Control: The model successfully demonstrated a speed adjustment range of roughly ±10% by simply modifying the bias of the transition agent, all without degrading the quality of the voice.
  • Convergence: The FA models "aligned" much earlier in the training process (visible by epoch 3) compared to the baseline, which still looked noisy at epoch 7.

Limitations and Future Work

  • Scope of Monotonicity: This method is specifically designed for tasks where the input and output are monotonic (like TTS or Speech Recognition). It wouldn't work for tasks like machine translation, where word order often flips between languages.
  • Speed Range: While 10% speed control is a great start, it is relatively modest compared to traditional vocoders.
  • Future Directions: The authors suggest applying this forward attention mechanism to Automatic Speech Recognition (ASR) and other seq2seq problems that share this monotonic nature.

Summary updated

Abstract

This paper proposes a forward attention method for the sequence-to-sequence acoustic modeling of speech synthesis. This method is motivated by the nature of the monotonic alignment from phone sequences to acoustic sequences. Only the alignment paths that satisfy the monotonic condition are taken into consideration at each decoder timestep. The modified attention probabilities at each timestep are computed recursively using a forward algorithm. A transition agent for forward attention is further proposed, which helps the attention mechanism to make decisions whether to move forward or stay at each decoder timestep. Experimental results show that the proposed forward attention method achieves faster convergence speed and higher stability than the baseline attention method. Besides, the method of forward attention with transition agent can also help improve the naturalness of synthetic speech and control the speed of synthetic speech effectively.
BibTeX
@inproceedings{zhang2018forward,
  author = {Jing-Xuan Zhang and Zhen-Hua Ling and Li-Rong Dai},
  title = {{Forward Attention in Sequence- To-Sequence Acoustic Modeling for Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2018},
  url = {https://openreview.net/forum?id=9wBtYL2osK},
}

Loading related content...

Figure