In speech processing, we face a fundamental "synchronization" problem. Text is a sequence of discrete symbols (phonemes or characters), while audio is a continuous stream of frames. A single phoneme like "s" might span ten audio frames in one sentence and thirty in another.
Historically, researchers solved this by using external "aligners"—pre-trained models or manual labels that told the system exactly which audio frame corresponded to which text token. However, this created a bottleneck: if the aligner was wrong, the model was wrong. A major research direction has emerged to move this alignment process inside the neural network, allowing models to autonomously discover the temporal relationship between text and sound.
The Foundation of Latent Alignment
The most influential tool for this discovery is Connectionist Temporal Classification (CTC). As Awni Y. Hannun and Andrew Y. Ng demonstrated in First-Pass Large Vocabulary Continuous Speech Recognition using Bi-Directional Recurrent DNNs, CTC allows a model to be trained on unaligned data by considering every possible way the text could be "stretched" to fit the audio.
The intuition behind CTC is marginalization. Instead of asking the model to predict the exact frame where a "b" starts, CTC asks the model to output a probability for each character (plus a special "blank" symbol) at every frame. It then sums the probabilities of all valid paths that collapse into the target transcript. This "many-to-one" property, as noted by Ji Won Yoon in the Oracle Teacher paper, forces the model to learn the underlying timing of speech to maximize the total probability of the correct sequence.
From Recognition to Synthesis
While CTC revolutionized speech recognition (ASR), text-to-speech (TTS) synthesis faced a different challenge. Early neural TTS models like Tacotron used attention mechanisms to "look" at the text while generating audio. However, attention is notoriously finicky; it can skip words or get stuck in loops.
To fix this, researchers like Jonathan Shen and Heiga Zen proposed the Non-Attentive Tacotron. Instead of a soft attention "glance," they used an explicit duration predictor. But how do you train a duration predictor without duration labels?
The answer lies in Monotonic Alignment Search (MAS). In the VITS model, Jaehyeon Kim and colleagues used MAS to find the single most likely monotonic path between text and audio during training. By searching for the path that maximizes the likelihood of the audio given the text, the model endogenously learns how long each phoneme should last. This same principle was later applied to diffusion-based models like Grad-TTS by Vadim Popov, proving that unsupervised alignment is robust across different generative architectures.
Making Alignment Differentiable
The next evolution was making these alignment searches fully differentiable. Standard MAS or Dynamic Time Warping (DTW) involves a "hard" choice (picking the best path), which can be difficult to optimize with backpropagation.
In Parallel Tacotron 2, Isaac Elias and Heiga Zen introduced a Soft-DTW loss. Instead of picking one path, Soft-DTW considers a weighted average of all possible paths. This creates a smooth "loss landscape" that the model can follow to learn timing.
Building on this, Lei He introduced SoftSpeech, which uses a "Soft-LengthRegulator." This allows the model to "smear" a phoneme's representation across multiple frames in a way that is mathematically differentiable. The result is a system that can learn to speak a new language without ever seeing a single timestamped transcript.
Why This Works: The Power of Monotonicity
The common thread across these papers—from Sound Event Detection to Non-Autoregressive Machine Translation—is the monotonicity constraint.
Human speech (and language) generally moves forward in time. By forcing the latent alignment to be monotonic (it can stay on a token or move to the next, but never go back), researchers provide the model with a powerful inductive bias. This constraint simplifies the search space enough that the model can distinguish between linguistic content and temporal variation, even when the only supervision is the final sequence of words.
By treating alignment as a latent variable to be optimized rather than a fixed input to be followed, these models achieve a level of robustness and naturalness that "hand-aligned" systems cannot match.