PAMA-TTS: Progression-Aware Monotonic Attention for Stable SEQ2SEQ TTS with Accurate Phoneme Duration Control

Yunchao He, Jian Luan, Yujun Wang

TL;DR

PAMA-TTS introduces "Progression-Aware Monotonic Attention" to bridge the gap between highly natural but unstable attention-based TTS and stable but robotic duration-informed models. By incorporating a "countdown" mechanism (backward frame position) and fuzzy alignment guidance, it achieves state-of-the-art naturalness while maintaining precise control over phoneme duration.

What Problem Does This Paper Address?

In the world of Text-to-Speech (TTS), researchers usually choose between two "evils":

  1. Attention-based models (e.g., Tacotron 2): These produce very natural-sounding speech but are notoriously unstable. Because the attention mechanism is flexible, it can "lose its place," leading to skipped words, repeated phonemes, or awkward silences.
  2. Duration-informed models (e.g., FastSpeech): These are rock-solid because they use a separate model to predict exactly how many frames each phoneme should last. However, this "hard" control often degrades speech naturalness, making the transitions between sounds feel mechanical.

The authors of PAMA-TTS aim to solve this trade-off. They want a model that is as natural as an attention-based system but as controllable and robust as a duration-informed one.

Methods

PAMA-TTS builds on the Tacotron 2 architecture but introduces three key innovations to make the attention "progression-aware":

  • Fuzzy Guidance Attention Matrix: During training, most models use "hard" alignment (a binary 0 or 1) from forced-alignment tools. PAMA-TTS uses a "fuzzy" matrix where boundaries between phonemes transition linearly over six frames. This acknowledges that speech sounds don't have instantaneous, pixel-perfect breaks, allowing for smoother acoustic transitions.
  • Latent Duration Representation: The model includes an auxiliary duration predictor. The "latent code" (hidden state) from this predictor is added to the encoder's output. This gives the attention mechanism a "hint" about how much "memory" or time a specific phoneme should occupy.
  • Backward Frame Position (The "Countdown"): This is the most novel feature. The model injects a "countdown" into the attention query—specifically, how many frames are left before the current phoneme should end. This helps the decoder "premeditate" the transition to the next phoneme, ensuring it doesn't stay on one sound for too long or jump away too early.

The system uses Stepwise Monotonic Attention as a baseline, ensuring the model only moves forward through the text, never backward.

Datasets

The researchers evaluated PAMA-TTS using an internal Mandarin Chinese corpus:

  • Size: ~10 hours of speech (approx. 12,000 utterances).
  • Speaker: A non-professional female speaker.
  • Processing: Audio was sampled at 16 kHz. Phoneme duration labels were generated using a Kaldi-based forced-alignment tool.
  • Vocoder: LPCNet was used to convert the predicted Mel-spectrograms into actual audio.

Key Insights and Ideas

  • Soft Constraints > Hard Constraints: The "aha moment" here is that explicit duration control doesn't have to be a "hard" override. By feeding duration information into the attention mechanism rather than using it to force-multiply frames, the model retains the flexibility needed for natural prosody.
  • The Power of the Countdown: Knowing the "distance to the end" of a phoneme is a powerful signal for stability. It acts as a rhythmic guide for the decoder, preventing the "stuck" or "skipping" behaviors common in standard Seq2Seq models.
  • Fuzziness is Realistic: Real human speech is fluid. By using fuzzy guidance, the model avoids the "robotic" artifacts that occur when a model is forced to adhere to rigid, potentially slightly inaccurate, forced-alignment boundaries.

Key Results

The authors compared PAMA-TTS against two baselines: TLR (Duration-informed) and TSW (Stepwise Monotonic Attention).

  • Naturalness (Mean Opinion Score): PAMA-TTS achieved a 4.41 MOS, significantly higher than the duration-informed TLR (4.22) and the standard attention TSW (4.38). It came remarkably close to the ground-truth audio (4.54).
  • Duration Accuracy: In terms of Mean Absolute Error (MAE) for phoneme duration, PAMA-TTS (6.68ms) performed on par with or better than the duration-informed model (7.53ms) and was nearly 10x more accurate than standard monotonic attention (65.92ms).
  • Robustness: Using an Automated Speech Recognition (ASR) tool to check for errors, PAMA-TTS showed a lower Phoneme Error Rate (2.12%) than both baselines, proving it is more stable during synthesis.
  • Speed Control: Unlike standard attention models, PAMA-TTS successfully handled speed modulations (0.75x and 1.5x) without collapsing.

Limitations and Future Work

  • Dependency on Alignment: The model still requires ground-truth alignment labels for training, which necessitates an external tool like Kaldi.
  • Language Specificity: The results are demonstrated on a Mandarin Chinese dataset; while the architecture is language-agnostic, performance on highly non-phonetic languages or different prosodic structures remains to be seen.
  • Future Directions: The authors suggest that this "progression-aware" logic could be extended to other sequence-to-sequence tasks beyond TTS where duration or timing is a critical factor.

Summary updated

Abstract

Sequence expansion between encoder and decoder is a critical challenge in sequence-to-sequence TTS. Attention-based methods achieve great naturalness but suffer from unstable issues like missing and repeating phonemes, not to mention accurate duration control. Duration-informed methods, on the contrary, seem to easily adjust phoneme duration but show obvious degradation in speech naturalness. This paper proposes PAMA-TTS to address the problem. It takes the advantage of both flexible attention and explicit duration models. Based on the monotonic attention mechanism, PAMA-TTS also leverages token duration and relative position of a frame, especially countdown information, i.e. in how many future frames the present phoneme will end. They help the attention to move forward along the token sequence in a soft but reliable control. Experimental results prove that PAMA-TTS achieves the highest naturalness, while has on-par or even better duration controllability than the duration-informed model.
BibTeX
@inproceedings{he2022pama,
  author = {Yunchao He and Jian Luan and Yujun Wang},
  title = {{PAMA-TTS: Progression-Aware Monotonic Attention for Stable SEQ2SEQ TTS with Accurate Phoneme Duration Control}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746202},
}

Loading related content...

Figure