Introduction
Modern end-to-end text-to-speech (TTS) systems, such as Tacotron, rely on attention mechanisms to align input text with generated audio frames. While effective for short sentences, these systems often struggle with long-form content, leading to alignment failures like word skipping or repetition. This paper proposes a shift from content-based attention to location-relative attention, introducing Dynamic Convolution Attention (DCA) and improved GMM-based mechanisms to achieve robust, monotonic alignment for extremely long passages.
The Alignment Challenge
Standard TTS models typically use "content-based" attention, where a decoder query is compared against text keys. This approach is brittle; for out-of-domain or long sequences, the model often loses its place. The authors argue that alignment should instead exploit the inherent monotonicity and locality of speech—the fact that speech moves forward through text and the next word is usually adjacent to the current one.
Proposed Location-Relative Mechanisms
The authors explore two primary families of location-relative attention:
1. Improved GMM-Based Attention
Building on Alex Graves' mixture of Gaussians approach, the authors introduce several refinements to stabilize the mean offset and variance, including normalization and stable activation functions like softplus. Crucially, they introduce Initial Biases—small forward-moving biases at the start of training—to "nudge" the model toward the correct one-phoneme-per-step progression.
2. Dynamic Convolution Attention (DCA)
DCA is the paper's core contribution. It refines the "location-sensitive" attention of Tacotron 2 by removing content terms entirely, preventing the model from "jumping" to similar-sounding words elsewhere in the text. It utilizes dynamic filters predicted from the decoder state and a causal prior filter that strictly enforces forward progression.
Experimental Results
The models were trained on the Lessac (37h) and LJ Speech (23h) datasets and evaluated on a "Harry Potter" test set containing utterances up to 84 seconds long.
Alignment Stability and Naturalness
DCA and GMMv2 with bias demonstrated significantly faster and more consistent alignment during training compared to standard methods. In Mean Opinion Score (MOS) evaluations, these models achieved naturalness scores (~4.3) comparable to ground truth and standard Tacotron 2, proving that removing content-based queries does not degrade audio quality.
Robustness to Utterance Length
The most significant finding is the models' ability to generalize to long sequences. As shown in the robustness trials, the Character Error Rate (CER) for standard content-based models increases dramatically once text length exceeds the training distribution (typically 5–10 seconds). In contrast, DCA and GMMv2b maintain near-zero error rates even for utterances exceeding 80 seconds.
Conclusion
By prioritizing location over content for alignment, DCA and improved GMM mechanisms solve the long-standing issue of alignment failure in TTS. These methods enable the synthesis of entire book chapters with high naturalness and perfect word order, while also offering potential for computational optimizations through "hard windowing" in production environments.
Summary updated