Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis

Eric Battenberg, R. J. Skerry-Ryan, Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, Tom Bagby

Introduction

Modern end-to-end text-to-speech (TTS) systems, such as Tacotron, rely on attention mechanisms to align input text with generated audio frames. While effective for short sentences, these systems often struggle with long-form content, leading to alignment failures like word skipping or repetition. This paper proposes a shift from content-based attention to location-relative attention, introducing Dynamic Convolution Attention (DCA) and improved GMM-based mechanisms to achieve robust, monotonic alignment for extremely long passages.

The Alignment Challenge

Standard TTS models typically use "content-based" attention, where a decoder query is compared against text keys. This approach is brittle; for out-of-domain or long sequences, the model often loses its place. The authors argue that alignment should instead exploit the inherent monotonicity and locality of speech—the fact that speech moves forward through text and the next word is usually adjacent to the current one.

Proposed Location-Relative Mechanisms

The authors explore two primary families of location-relative attention:

1. Improved GMM-Based Attention

Building on Alex Graves' mixture of Gaussians approach, the authors introduce several refinements to stabilize the mean offset and variance, including normalization and stable activation functions like softplus. Crucially, they introduce Initial Biases—small forward-moving biases at the start of training—to "nudge" the model toward the correct one-phoneme-per-step progression.

2. Dynamic Convolution Attention (DCA)

DCA is the paper's core contribution. It refines the "location-sensitive" attention of Tacotron 2 by removing content terms entirely, preventing the model from "jumping" to similar-sounding words elsewhere in the text. It utilizes dynamic filters predicted from the decoder state and a causal prior filter that strictly enforces forward progression.

Alignment trials showing MCD-DTW over training steps. DCA and GMMv2+Bias exhibit more consistent and faster convergence compared to content-based baselines.

Experimental Results

The models were trained on the Lessac (37h) and LJ Speech (23h) datasets and evaluated on a "Harry Potter" test set containing utterances up to 84 seconds long.

Alignment Stability and Naturalness

DCA and GMMv2 with bias demonstrated significantly faster and more consistent alignment during training compared to standard methods. In Mean Opinion Score (MOS) evaluations, these models achieved naturalness scores (~4.3) comparable to ground truth and standard Tacotron 2, proving that removing content-based queries does not degrade audio quality.

Robustness to Utterance Length

The most significant finding is the models' ability to generalize to long sequences. As shown in the robustness trials, the Character Error Rate (CER) for standard content-based models increases dramatically once text length exceeds the training distribution (typically 5–10 seconds). In contrast, DCA and GMMv2b maintain near-zero error rates even for utterances exceeding 80 seconds.

Character Error Rate (CER) as a function of utterance length. While standard models fail on long sequences, DCA and GMMv2b maintain robustness far beyond training lengths.

Conclusion

By prioritizing location over content for alignment, DCA and improved GMM mechanisms solve the long-standing issue of alignment failure in TTS. These methods enable the synthesis of entire book chapters with high naturalness and perfect word order, while also offering potential for computational optimizations through "hard windowing" in production environments.

Summary updated

BibTeX
@inproceedings{battenberg2020location,
  author = {Eric Battenberg and R. J. Skerry-Ryan and Soroosh Mariooryad and Daisy Stanton and David Kao and Matt Shannon and Tom Bagby},
  title = {{Location-Relative Attention Mechanisms For Robust Long-Form Speech Synthesis}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=-2KvTN9C1tY},
}
Figure