PAMA-TTS: Progression-Aware Monotonic Attention for Stable Seq2Seq TTS With Accurate Phoneme Duration Control

Yunchao He, Jian Luan, Yujun Wang

Abstract

Sequence expansion between encoder and decoder is a critical challenge in sequence-to-sequence TTS. Attention-based methods achieve great naturalness but suffer from unstable issues like missing and repeating phonemes, not to mention accurate duration control. Duration-informed methods, on the contrary, seem to easily adjust phoneme duration but show obvious degradation in speech naturalness. This paper proposes PAMA-TTS to address the problem. It takes the advantage of both flexible attention and explicit duration models. Based on the monotonic attention mechanism, PAMA-TTS also leverages token duration and relative position of a frame, especially countdown information, i.e. in how many future frames the present phoneme will end. They help the attention to move forward along the token sequence in a soft but reliable control. Experimental results prove that PAMA-TTS achieves the highest naturalness, while has on-par or even better duration controllability than the duration-informed model.

Introduction

In the world of Text-to-Speech (TTS), researchers typically face a trade-off between two paradigms:

  1. Attention-based models (e.g., Tacotron 2): These produce highly natural-sounding speech but are notoriously unstable. The flexible attention mechanism can "lose its place," leading to skipped words, repeated phonemes, or awkward silences.
  2. Duration-informed models (e.g., FastSpeech): These are robust because they use a separate model to predict exactly how many frames each phoneme should last. However, this "hard" control often degrades speech naturalness, making transitions between sounds feel mechanical.

PAMA-TTS (Progression-Aware Monotonic Attention) bridges this gap. It introduces a system that is as natural as an attention-based system but as controllable and robust as a duration-informed one by incorporating a "countdown" mechanism and fuzzy alignment guidance.

Methods and Architecture

PAMA-TTS builds on the Tacotron 2 architecture but introduces three key innovations to make the attention "progression-aware."

PAMA-TTS architecture overview showing the integration of duration prediction and relative position encoding.

Key Innovations

  • Fuzzy Guidance Attention Matrix: During training, most models use "hard" alignment (a binary 0 or 1) from forced-alignment tools. PAMA-TTS uses a "fuzzy" matrix where boundaries between phonemes transition linearly over six frames. This acknowledges that speech sounds do not have instantaneous breaks, allowing for smoother acoustic transitions.
Comparison between (a) a standard hard guidance matrix and (b) the proposed fuzzy guidance matrix used in PAMA-TTS.
  • Latent Duration Representation: The model includes an auxiliary duration predictor. The hidden state (latent code) from this predictor is added to the encoder's output, giving the attention mechanism a "hint" about the expected duration of a specific phoneme.
  • Backward Frame Position (The "Countdown"): This is the most novel feature. The model injects a "countdown" into the attention query—specifically, how many frames are left before the current phoneme should end. This helps the decoder "premeditate" the transition to the next phoneme, ensuring it doesn't stay on one sound for too long or jump away too early.

The system utilizes Stepwise Monotonic Attention as a baseline, ensuring the model only moves forward through the text sequence.

Experimental Setup

The researchers evaluated PAMA-TTS using an internal Mandarin Chinese corpus consisting of approximately 10 hours of speech (12,000 utterances) from a non-professional female speaker. Audio was sampled at 16 kHz, and phoneme duration labels were generated using a Kaldi-based forced-alignment tool. LPCNet was employed as the vocoder to convert predicted Mel-spectrograms into audio.

Key Results

PAMA-TTS was compared against two baselines: TLR (Duration-informed) and TSW (Stepwise Monotonic Attention).

  • Naturalness: PAMA-TTS achieved a Mean Opinion Score (MOS) of 4.41, significantly higher than the duration-informed TLR (4.22) and standard attention TSW (4.38), nearing the ground-truth audio score of 4.54.
  • Duration Accuracy: In terms of Mean Absolute Error (MAE) for phoneme duration, PAMA-TTS (6.68ms) performed on par with or better than the duration-informed model (7.53ms) and was nearly 10x more accurate than standard monotonic attention (65.92ms).
  • Robustness: Using Automated Speech Recognition (ASR) to check for errors, PAMA-TTS showed a lower Phoneme Error Rate (2.12%) than both baselines.
  • Speed Control: Unlike standard attention models, PAMA-TTS successfully handled speed modulations (0.75x and 1.5x) without synthesis collapse.

Conclusion

PAMA-TTS demonstrates that soft constraints can be superior to hard constraints in TTS. By feeding duration information into the attention mechanism rather than using it to force-multiply frames, the model retains the flexibility needed for natural prosody while gaining the stability of duration-informed systems. Future work may explore extending this progression-aware logic to other sequence-to-sequence tasks where timing is a critical factor.

Summary updated

Abstract

Sequence expansion between encoder and decoder is a critical challenge in sequence-to-sequence TTS. Attention-based methods achieve great naturalness but suffer from unstable issues like missing and repeating phonemes, not to mention accurate duration control. Duration-informed methods, on the contrary, seem to easily adjust phoneme duration but show obvious degradation in speech naturalness. This paper proposes PAMA-TTS to address the problem. It takes the advantage of both flexible attention and explicit duration models. Based on the monotonic attention mechanism, PAMA-TTS also leverages token duration and relative position of a frame, especially countdown information, i.e. in how many future frames the present phoneme will end. They help the attention to move forward along the token sequence in a soft but reliable control. Experimental results prove that PAMA-TTS achieves the highest naturalness, while has on-par or even better duration controllability than the duration-informed model.
BibTeX
@inproceedings{he2022pama,
  author = {Yunchao He and Jian Luan and Yujun Wang},
  title = {{PAMA-TTS: Progression-Aware Monotonic Attention for Stable Seq2Seq TTS With Accurate Phoneme Duration Control}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746202},
}
Figure