When humans read a sentence aloud, our eyes move strictly from left to right. We map discrete linguistic units (words or phonemes) to a continuous sequence of acoustic sounds in a strictly forward-moving, sequential trajectory.
Early end-to-end neural text-to-speech (TTS) architectures, such as Tacotron introduced by Wu in Tacotron: Towards End-to-End Speech Synthesis, revolutionized the field by replacing complex, hand-engineered pipelines with a single sequence-to-sequence network. These models rely on soft, content-based attention mechanisms to implicitly learn how text aligns with audio.
However, standard attention mechanisms lack an inherent inductive bias for linear progression. Because a Transformer or recurrent network can theoretically attend to any part of the input sequence at any time, the alignment between text and speech is inherently unstable. This architectural freedom leads to a notorious class of synthesis failures known as alignment instability: word skipping, infinite loops (syllable repetition), or sudden silences.
To solve this, researchers have developed mechanisms that enforce monotonic alignment—constraining the model's attention to ensure that speech generation moves forward in perfect lockstep with the text.
The Mechanics of Attention Collapse
To understand why speech models hallucinate, we must look at the structural mismatch between language modeling and speech generation. In a standard autoregressive decoder-only model (like VALL-E), the network predicts high-dimensional acoustic tokens or spectrogram frames one after another. As noted by Zhou et al. in Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis, predicting these noisy representations causes cumulative errors over time.
Without an explicit alignment constraint, the attention matrix can degrade in three distinct ways:
Skipping: Text: The [quick brown] fox jumps... -> Audio: "The fox jumps..."
Looping: Text: The quick [brown] fox jumps... -> Audio: "The quick brown brown brown..."
Collapse: Text: The quick brown fox jumps... -> Audio: [Sudden Silence]
As Tian demonstrates in FeatherTTS: Robust and Efficient attention based Neural TTS, content-based attention struggles particularly with long, complex, or out-of-domain sentences. When the model encounters an unfamiliar word or a stylistic deviation, the attention weights become scattered. The model "loses its place" in the text sequence, either jumping forward prematurely (skipping) or getting trapped in a localized attention pocket (looping).
Enforcing Monotonicity at Inference Time
One elegant way to fix this instability without re-training massive models is to constrain the attention weights during inference. Since we know speech must be monotonic, we can mathematically forbid the model from looking backward or too far ahead.
In Attention-Constrained Inference for Robust Decoder-Only Text-to-Speech, Wang et al. introduce Attention-Constrained Inference (ACI). They observe that within a decoder-only speech Transformer, specific attention heads naturally specialize in tracking text-to-audio alignment. By identifying these "alignment-aware" heads and applying a dynamic mask during inference, they physically prevent the model from attending to non-monotonic text tokens.
Standard Attention Matrix Attention-Constrained Inference (ACI)
Text Tokens Text Tokens
[T1][T2][T3][T4] [T1][T2][T3][T4]
A1 | X . . . | A1 | X . . . |
A2 | . X . . | A2 | . X . . | (Masks out illegal
A3 | X . . . | <-- Loop! A3 | . . X . | backward/forward
A4 | . . . X | A4 | . . . X | jumps)
A similar approach is taken by Xin et al. in RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis. Instead of letting the model guess the timing implicitly, they use a "Chain-of-Thought" style prompt where the model explicitly predicts phoneme durations first. These predicted durations are then used to construct a hard, duration-guided attention mask that bounds the generation window, slashing error rates on difficult sentences from 68% to just 4%.
Architectural Monotonicity: Hard Alignments and Transducers
While inference-time masking acts as a safety rail, a more permanent solution is to bake monotonicity directly into the model's architecture, making alignment failure mathematically impossible.
Stepwise Monotonic Attention (SMA)
Instead of computing a global softmax over all text tokens, Stepwise Monotonic Attention (SMA) forces the model to make a binary decision at each acoustic step: either stay on the current phoneme or advance to the next one. As explored by He et al. in Robust Sequence-to-Sequence Acoustic Modeling with Stepwise Monotonic Attention for Neural TTS and implemented in Transformers by Liang et al. in Enhancing Monotonicity for Robust Autoregressive Transformer TTS, this constraint eliminates attention drift entirely by removing the model's ability to skip backward or jump arbitrarily forward.
Monotonic Alignment Search (MAS)
For non-autoregressive models that generate entire sentences in parallel, soft attention is replaced by explicit duration matching. In Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search, Kim et al. utilize Monotonic Alignment Search (MAS). MAS uses dynamic programming to find the most probable monotonic path between text features and audio frames during training.
Phonemes (Text)
|
v
[ P1 ] [ P2 ] [ P3 ]
| | |
+--+ | |
v v v
[ F1 ][ F2 ][ F3 ][ F4 ]
^
|
Audio Frames (Time)
By finding a hard, discrete assignment where each audio frame maps to exactly one text token in chronological order, models like Glow-TTS and EfficientTTS (Miao et al.) bypass soft attention entirely during inference, ensuring absolute robustness against loops and skips.
Generative Transducers
The ultimate convergence of these ideas is the integration of Automatic Speech Recognition (ASR) architectures into synthesis. In VALL-T: Decoder-Only Generative Transducer for Robust and Decoding-Controllable Text-to-Speech, Du et al. replace the standard autoregressive attention mechanism with a Neural Transducer architecture.
The transducer explicitly operates on a grid of text steps and audio steps. At each point, it emits either an acoustic token (remaining on the current text token) or a blank token (signaling a transition to the next text token). This structural constraint guarantees that the model processes the input text sequentially, achieving a massive reduction in word error rates while preserving the high-fidelity, zero-shot capabilities of modern generative speech models.