Expressive Speech Synthesis and the Prosodic Data Gap

This direction is split into other directions: manual annotation bottleneck (manual prosodic annotation bottleneck). Its last version stays readable and citable here.

Version 1 of 1216 papers242 conceptsfrozen 2026-10-04

Modern text-to-speech (TTS) systems have achieved remarkable phonetic clarity, yet they often fall into the "neutrality trap." While a model might perfectly pronounce every syllable, the resulting audio frequently sounds robotic or detached because it lacks prosody—the rhythm, stress, and intonation that convey emotion and intent.

The core challenge in this research direction is not just architectural; it is a fundamental data bottleneck. Most high-quality speech corpora are recorded in a strictly neutral, professional tone, which prevents models from learning the nuances of human affective states.

The Neutrality Bottleneck

Standard datasets like LibriTTS or LJSpeech are "flat." As Adaeze Adigwe and colleagues note in The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems, these neutral recordings make it nearly impossible for models to generalize to emotional speech. When a model only hears "reading-style" narration, it fails to capture the acoustic variance of a sigh, a happy exclamation, or a whispered secret.

This scarcity extends to linguistic structures. For instance, Tom Ko and his team found that "declarative questions"—sentences that are grammatically statements but use rising intonation to ask a question—are so rare in standard corpora that models default to flat intonation even when the context demands a rise (A Study of Modeling Rising Intonation in Cantonese Neural Speech Synthesis).

Modeling the Latent Vibe

To overcome the lack of explicit labels, researchers are moving toward modeling speech in "latent style spaces." Instead of telling a model to be "happy" (a rigid category), systems like InstructTTS, developed by Dongchao Yang and others, allow users to provide natural language prompts like "full of sadness and helplessness" (InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Prompt).

The intuition here is to decouple the content (what is said) from the style (how it is said). Shengpeng Ji demonstrates this in ControlSpeech, which uses a decoupled codec to ensure that changing the emotion doesn't accidentally change the speaker's identity (ControlSpeech: Towards Simultaneous Zero-shot Speaker Cloning and Zero-shot Language Style Control With Decoupled Codec).

From Deterministic to Generative Prosody

Traditional TTS models are "one-size-fits-all": for a given sentence, they predict a single, average pitch contour. This leads to "over-smoothing," where the voice sounds safe but boring.

To inject life back into synthetic voices, researchers are turning to generative frameworks:

Scaling via Automated Annotation

The final hurdle is the cost of labeling. Manually marking every "pitch accent" or "emphasis" in a 100-hour dataset is prohibitively expensive. Ani Nenkova showed early on that modeling these prominences is vital for naturalness (Modelling prominence and emphasis improves unit-selection synthesis), but how do we scale this?

Recent work suggests using "weak supervision" from other domains. Taesik Gong and his team developed LanSER, which uses Large Language Models (LLMs) to look at speech transcripts and "guess" the emotion, providing labels for massive amounts of previously unannotated audio (LanSER: Language-Model Supported Speech Emotion Recognition). Similarly, Ruiqi Li uses robust transcription models to automate the alignment of lyrics and notes in singing synthesis, bypassing the manual annotation bottleneck (Robust Singing Voice Transcription Serves Synthesis).

By combining specialized expressive datasets with generative modeling and automated labeling, the field is moving toward voices that don't just speak, but communicate.

Papers in this version

Showing 60 of 216 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_expressive_speech_synthesis_and_the_pros_3a923bb4,
  title = {Expressive Speech Synthesis and the Prosodic Data Gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-speech-synthesis-and-the-prosodic-data-gap/rdr_3a923bb492c04243aca7d754a7180996},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_expressive_speech_synthesis_and_the_pros_3a923bb4_v1,
  title = {Expressive Speech Synthesis and the Prosodic Data Gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-speech-synthesis-and-the-prosodic-data-gap/rdr_3a923bb492c04243aca7d754a7180996/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}