Expressive Speech Synthesis via Natural Language Instructions

This direction is split into other directions: reference-dependent style transfer. Its last version stays readable and citable here.

Version 1 of 134 papers34 conceptsfrozen 2026-10-04

Traditional text-to-speech (TTS) systems face a "control bottleneck." To generate speech that sounds angry, whispered, or accented, users typically have to provide a reference audio clip or select from a rigid set of categorical labels. As noted by Yichong Leng and Kai Shen in PromptTTS 2, finding a perfect reference clip is often impractical, and rigid labels fail to capture the nuance of human expression.

A new generation of models is shifting this paradigm by allowing users to describe the desired vocal output using natural language prompts—such as "a sighing tone, full of sadness"—and mapping these semantic descriptions directly to acoustic styles.

Mapping Semantics to Style Embeddings

The core challenge in instruction-based synthesis is translating high-level text (semantics) into low-level acoustic variations (prosody). Early attempts, such as the Text-Predicted Global Style Tokens (TP-GST) introduced by Yuxuan Wang and Daisy Stanton, demonstrated that a model could learn to predict a "style embedding" directly from the input text.

Modern frameworks like PromptStyle and InstructTTS refine this by using a two-stage process. First, a text prompt is mapped to a shared style latent space. Second, this latent representation guides the speech generator. By modeling speech in a discrete latent space, Dongchao Yang and colleagues showed that models can achieve much higher expressiveness than those relying on one-hot emotion labels.

Solving the One to Many Problem

A single text prompt like "speak excitedly" can correspond to many valid acoustic realizations. This is known as the "one-to-many" ambiguity problem. To solve this, researchers have integrated diffusion models and Large Language Models (LLMs).

In PL-TTS, Shuhua Li and Qirong Mao replaced standard encoders with fine-tuned LLMs (like Llama 2) to better grasp the creative nuances of prompts. Meanwhile, PromptTTS 2 uses a variation network based on diffusion to sample from the distribution of possible voices, ensuring that the generated speech is both diverse and aligned with the prompt's description.

Disentangling Identity from Delivery

For these systems to work across different speakers, the model must separate who is speaking (timbre) from how they are speaking (style/accent). This is critical for tasks like cross-speaker style transfer, where an emotional delivery from one person is applied to the neutral voice of another.

Songxiang Liu and the team behind Referee achieved this by cascading a text-to-style model with a style-to-wave model. Similarly, for accent control, Dongya Jia and Jiaxin Li developed the Pseudo Siamese Disentanglement Network (PSDN) in Zero-Shot Accent Conversion. By using adversarial training, these networks learn to strip away speaker-specific traits, leaving a "pure" representation of the style or accent that can be applied to any target voice.

Beyond Speech to Singing and Music

The shift from rigid inputs to flexible instructions is also transforming musical generation. Traditional singing voice synthesis (SVS) requires precise MIDI scores. However, systems like SongCreator by Shiyin Kang and Max W. Y. Lam allow for song creation directly from lyrics.

This direction culminates in unified frameworks like InstructAudio, where Chunyu Qiang and Chen Zhang demonstrate that a single interface can handle speech, music, and even complex multi-speaker dialogues using only natural language instructions. By removing the need for specialized musical notation or reference audio, these models make high-fidelity audio creation accessible to anyone who can describe what they want to hear.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_expressive_speech_synthesis_via_natural__9e1de21b,
  title = {Expressive Speech Synthesis via Natural Language Instructions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-speech-synthesis-via-natural-language-instructions/rdr_9e1de21ba2af496593937fbceeb8577d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_expressive_speech_synthesis_via_natural__9e1de21b_v1,
  title = {Expressive Speech Synthesis via Natural Language Instructions},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/expressive-speech-synthesis-via-natural-language-instructions/rdr_9e1de21ba2af496593937fbceeb8577d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}