Generative Flow Matching and Language Models for Speech Synthesis

56 concepts55 papers1 proposal

To understand why modern text-to-speech (TTS) systems work, we must first understand the fundamental challenge they solve: the one-to-many mapping problem. A single string of text can be spoken in an infinite number of ways, varying in pitch, speed, and emotional inflection.

Early speech synthesis relied on unit selection, where segments of recorded speech were stitched together, as explored by Ann K. Syrdal et al. in Text-to-speech intelligibility across speech rates and Jian Li et al. in The Toshiba Mandarin TTS System for the Blizzard Challenge 2008. While intelligible, these systems lacked flexibility. Statistical parametric methods, such as the trajectory tiling approach utilized by Yao Qian et al. in An HMM Trajectory Tiling (HTT) Approach to High Quality TTS - Microsoft Entry to Blizzard Challenge 2010, smoothed out waveforms but often introduced a robotic buzz.

The advent of deep learning brought neural acoustic models and vocoders. Early neural architectures, such as the Deep Feed-Forward Sequential Memory Networks designed by Heng Lu et al. in Deep Feed-Forward Sequential Memory Networks for Speech Synthesis, significantly improved generation speed and efficiency over recurrent networks. However, these early neural systems still relied on a complex, multi-stage pipeline: text analysis, duration prediction, acoustic feature generation, and waveform synthesis.

Collapsing the Pipeline via Joint Alignment

Traditional multi-stage neural TTS models suffered from error propagation. If the text analysis or external alignment tool made a mistake mapping a phoneme to a time duration, the acoustic model would generate a distorted spectrogram, and the vocoder would produce unnatural audio.

To build intuition for why modern systems are so robust, consider the shift toward single-stage, end-to-end architectures that learn alignment internally. Instead of relying on pre-computed alignments, the authors of JDI-T: Jointly Trained Duration Informed Transformer for Text-To-Speech without Explicit Alignment (Dan Lim et al.) demonstrated that a duration predictor could be trained jointly with a feed-forward Transformer. This allowed the model to extract phoneme durations on-the-fly, eliminating synthesis errors.

This concept was unified further by Dan Lim et al. in JETS: Jointly Training FastSpeech2 and HiFi-GAN for End to End Text to Speech, which combined the acoustic model and the vocoder into a single, jointly trained network. By optimizing both components simultaneously with an internal alignment module, the system bypassed the mismatch between acoustic features and waveform generation.

Similarly, Bac Nguyen et al. developed Autotts: End-to-End Text-to-Speech Synthesis Through Differentiable Duration Modeling, proving that differentiable duration modeling allows a model to learn text-to-audio alignments directly from raw data, drastically simplifying the training pipeline while boosting speech quality. Even when data is imperfect, automated alignment pipelines can match the quality of manually curated datasets, as shown by Mahta Fetrat Qharabagh in ManaTTS Persian: a recipe for creating TTS datasets for lower resource languages.

Resolving the One-to-Many Problem with Flow Matching

Even with perfect alignment, traditional models trained with deterministic regression losses (like Mean Squared Error) fail to capture true human expressiveness. When forced to predict a single deterministic value for a phoneme that could be spoken with multiple pitches or durations, the model defaults to the statistical average. This average represents a flat, lifeless drone.

To overcome this, modern speech synthesis treats audio generation as a continuous probabilistic trajectory using Conditional Flow Matching (CFM) and diffusion models. Instead of predicting a static spectrogram, the model learns a vector field that smoothly deforms a simple prior distribution (like standard Gaussian noise) into a highly detailed, expressive speech distribution.

By utilizing CFM, Haoyu Wang et al. introduced ProsodyFlow: High-fidelity Text-to-Speech through Conditional Flow Matching and Prosody Modeling with Large Speech Language Models, achieving human-level speech quality and rapid synthesis speeds. CFM allows the model to sample distinct, crisp prosodic paths rather than blurring them together. To maintain temporal coherence across long sentences, Hui Wang et al. proposed FELLE: Autoregressive Speech Synthesis with Token-Wise Coarse-to-Fine Flow Matching, which applies flow matching in a coarse-to-fine manner across discrete tokens.

Furthermore, factorizing speech into independent components makes this generative process highly efficient. In DiFlow-TTS: Compact and Low-Latency Zero-Shot Text-to-Speech with Factorized Discrete Flow Matching, Ngoc-Son Nguyen et al. demonstrated that separating speech into prosody and acoustic details before applying discrete flow matching yields state-of-the-art naturalness with minimal computational latency. This capacity to model diverse trajectories is crucial for conversational speech synthesis, where contextual coherence dictates how an utterance should sound, as seen in the diffusion-based framework by Weihao Wu et al. in DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models.

Contextual Awareness via Speech Language Models

A generative model can produce diverse speech trajectories, but it needs a mechanism to choose the correct trajectory based on semantic context. This is where Large Speech Language Models (SLMs) and LLM backbones enter the pipeline.

By leveraging self-supervised speech representations and large language model backbones, synthesis engines gain a deep understanding of linguistic consistency and macro-context. For example, Cong Han et al. developed StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models, which uses style diffusion alongside adversarial training driven by large SLMs. This enables the model to synthesize lifelike speech without needing any reference audio, occasionally outperforming human recordings in subjective naturalness evaluations.

To refine this control, researchers have adapted reinforcement learning concepts to speech. In Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization, Xiaoxue Gao et al. applied Direct Preference Optimization (DPO) to speech synthesis. Instead of merely mimicking a dataset, the model is explicitly trained to maximize the likelihood of preferred emotional expressions while minimizing flat or neutral ones. This optimization ensures high linguistic consistency and superior instruction-following capabilities, allowing the system to capture subtle prosodic nuances that standard supervised learning completely misses.

Reference-Free Evaluation via Conditional Token Prediction

As speech synthesis reaches near-human fidelity, evaluating these models presents a paradox. Traditional objective metrics like Word Error Rate (WER) or Mel-Cepstral Distortion (MCD) rely on comparing the generated audio directly to a single human reference recording. However, because flow matching and diffusion models generate highly diverse, valid prosodic variations, a synthesized utterance might sound incredibly natural but differ significantly from the reference recording's specific pitch contour.

To solve this evaluation bottleneck, the field has shifted toward reference-free metrics that evaluate speech based on its intrinsic linguistic and prosodic plausibility. The authors of Objective Evaluation of Prosody and Intelligibility in Speech Synthesis via Conditional Prediction of Discrete Tokens (Ismail Rasim Ulgen et al.) introduced TTScore, a framework that evaluates synthesized speech by predicting the likelihood of discrete speech tokens conditioned on the input text.

By decoupling the evaluation into separate metrics for intelligibility and prosody, this approach measures whether the generated audio accurately reflects the intended text while maintaining natural prosodic flow. This method correlates far more closely with human quality judgments than traditional metrics, providing a reliable compass for the next generation of generative speech models.

Key Foundational Work

Development of a Kiswahili text to speech system (2005) — contributed 1 concept elements to this direction.
A bi-lingual Mandarin-to-taiwanese text-to-speech system (2005) — contributed 1 concept elements to this direction.
Phonetically enriched labeling in unit selection TTS synthesis (2006) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_state_of_the_art_speech_synthesis_qualit_5c93ec59,
  title = {state-of-the-art speech synthesis quality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-speech-synthesis-quality/rdr_5c93ec590f9c45628f277cfcd5507c48},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_state_of_the_art_speech_synthesis_qualit_5c93ec59_v1,
  title = {state-of-the-art speech synthesis quality},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/state-of-the-art-speech-synthesis-quality/rdr_5c93ec590f9c45628f277cfcd5507c48/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake