Towards Controllable Speech Synthesis in the Era of Large Language Models: A Survey

Tianxin Xie, Yan Rong, Pengfei Zhang, Li Liu

Abstract

Text-to-speech (TTS), also known as speech synthesis, is a prominent research area that aims to generate natural-sounding human speech from text. Recently, with the increasing industrial demand, TTS technologies have evolved beyond synthesizing human-like speech to enabling controllable speech generation. This includes fine-grained control over various attributes of synthesized speech such as emotion, prosody, timbre, and duration. In addition, advancements in deep learning, such as diffusion and large language models, have significantly enhanced controllable TTS over the past several years. In this work, we conduct a comprehensive survey of controllable TTS, covering approaches ranging from basic control techniques to methods utilizing natural language prompts, aiming to provide a clear understanding of the current state of research. We examine the general controllable TTS pipeline, challenges, model architectures, and control strategies, offering a comprehensive and clear taxonomy of existing methods. Additionally, we provide a detailed summary of datasets and evaluation metrics and shed some light on the applications and future directions of controllable TTS. To the best of our knowledge, this survey paper provides the first comprehensive review of emerging controllable TTS methods, which can serve as a beneficial resource for both academic researchers and industrial practitioners.

1. Introduction: The Shift Toward Controllability

For decades, the primary goal of speech synthesis was achieving high naturalness and intelligibility—essentially making machines sound human. Having largely achieved this milestone, the research frontier has shifted toward controllability. Modern applications require the ability to precisely manipulate attributes such as emotion, timbre, prosody, and acoustic environment (e.g., "make it sound like they are in a park").

While previous surveys focused on traditional deep learning TTS, they often overlook the seismic shift caused by Large Language Models (LLMs). Today's users increasingly demand "instruction-guided" synthesis, moving away from rigid pitch sliders toward intuitive natural language commands like, "Read this like a news anchor who just had too much coffee." This survey addresses the lack of a unified taxonomy for these emerging methods and bridges the gap between current academic research and industrial needs.

2. Evolution and Feature Representations

The landscape of controllable TTS has evolved rapidly since 2018. A key trend is the transition from Continuous Feature Representations (like Mel-spectrograms), which preserve fine acoustic detail, to Discrete Tokens (quantized units). The latter allows TTS models to be trained using the same autoregressive principles as text-based LLMs, enabling powerful "in-context learning" capabilities.

Timeline and statistics of controllable TTS methods (2018-2025), illustrating the shift from continuous to discrete feature representations and the surge in research activity.

3. Model Architectures and Control Strategies

The survey categorizes controllable TTS through two primary lenses: architecture and control strategy.

3.1 Model Architectures

  • Non-Autoregressive (NAR): Models like FastSpeech and newer Diffusion/Flow-based systems (e.g., Matcha-TTS) generate speech in parallel. They offer high speed but can struggle with the complex prosody required for expressive control.
  • Autoregressive (AR): The "LLM-style" approach (e.g., VALL-E) treats speech as a sequence of tokens. These models excel at zero-shot voice cloning, often requiring only a 3-second audio prompt to mimic a speaker's voice.
  • Hybrids: Emerging state-of-the-art models like CosyVoice combine the semantic logic of LLMs with the high-fidelity generation of flow-based models.

3.2 Control Strategies

The field has moved through four distinct stages of control, as illustrated in the taxonomy below:

  1. Style Tagging: Using predefined labels (e.g., "Happy," "Fast").
  2. Reference Prompts: Mimicking a provided short audio clip.
  3. Natural Language Descriptions: Describing the desired voice in text (e.g., "A deep, gravelly voice").
  4. Instruction-Guided Synthesis: Interpreting free-form instructions to modify specific nuances of delivery.
Taxonomy of control strategies for speech synthesis, categorizing methods from predefined style tags to sophisticated instruction-guided control.

4. Datasets and the Evaluation Crisis

Research in this area relies on three dataset categories: Tag-based (e.g., AISHELL-3), Description-based (e.g., PromptSpeech), and Dialogue-based (e.g., DailyTalk).

However, traditional metrics like Mel Cepstral Distortion (MCD) are insufficient for measuring subjective qualities like "sarcasm" or "authority." To address this, the authors propose a novel evaluation pipeline using Multimodal LLMs (MLLMs) like Gemini as automated judges. Experiments show that MLLM ratings for instruction following align more closely with human perception than older automated tools like NISQA.

5. Key Results and Future Directions

Benchmarking of 10 prominent models revealed several key insights:

  • Zero-Shot Performance: Vevo leads in naturalness and expressiveness without explicit guidance.
  • Instruction Following: CosyVoice achieved the highest scores (4.81/5.0) in adhering to complex user directives.
  • The Instruction Bonus: Models specifically tuned for instructions (e.g., VoxInstruct) consistently outperform general zero-shot models, proving that explicit instruction-tuning is essential for fine-grained control.

Future Challenges: Achieving true "disentangled" control—changing one attribute like pitch without affecting another like emotion—remains an open challenge. Additionally, the field faces a shortage of high-quality datasets for subtle vocal nuances (e.g., "breathiness"). Future work will likely focus on multimodal integration, generating speech that matches the visual mood of video or images.

Summary updated

Abstract

Text-to-speech (TTS), also known as speech synthesis, is a prominent research area that aims to generate natural-sounding human speech from text. Recently, with the increasing industrial demand, TTS technologies have evolved beyond synthesizing human-like speech to enabling controllable speech generation. This includes fine-grained control over various attributes of synthesized speech such as emotion, prosody, timbre, and duration. In addition, advancements in deep learning, such as diffusion and large language models, have significantly enhanced controllable TTS over the past several years. In this work, we conduct a comprehensive survey of controllable TTS, covering approaches ranging from basic control techniques to methods utilizing natural language prompts, aiming to provide a clear understanding of the current state of research. We examine the general controllable TTS pipeline, challenges, model architectures, and control strategies, offering a comprehensive and clear taxonomy of existing methods. Additionally, we provide a detailed summary of datasets and evaluation metrics and shed some light on the applications and future directions of controllable TTS. To the best of our knowledge, this survey paper provides the first comprehensive review of emerging controllable TTS methods, which can serve as a beneficial resource for both academic researchers and industrial practitioners.
BibTeX
@misc{xie2024towards,
  author = {Tianxin Xie and Yan Rong and Pengfei Zhang and Li Liu},
  title = {{Towards Controllable Speech Synthesis in the Era of Large Language Models: A Survey}},
  year = {2024},
  eprint = {2412.06602},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.06602},
}
Figure