Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models

Weiqin Li, Peiji Yang, Yicheng Zhong, Yixuan Zhou, Zhisheng Wang, Zhiyong Wu, Xixin Wu, Helen Meng

Introduction

Traditional Text-to-Speech (TTS) systems have long been optimized for "read-style" audio—the clear, rhythmic, and emotionally neutral delivery found in audiobooks or news broadcasts. While these systems are highly intelligible, they often fall into an "uncanny valley" of robotic perfection when applied to conversational contexts. Real human speech is inherently messy, characterized by a variety of spontaneous behaviors including disfluencies (fillers like "um" and "uh"), interjections ("oh!", "wow"), and non-speech sounds such as laughter or throat clearing.

While recent neural codec language models, such as VALL-E, have revolutionized zero-shot voice cloning, they struggle to generate these spontaneous behaviors intentionally. Often, these models treat such quirks as noise to be smoothed over, or they fail to capture the subtle variations in pitch and timing (prosody) that give spontaneous speech its authentic feel. This paper introduces a new TTS framework designed to bridge the gap between mere intelligibility and true conversational authenticity through controllable spontaneous behavior modeling.

Methodology

The proposed system builds upon the VALL-E architecture—a neural codec language model—and introduces three primary innovations to handle the complexities of spontaneous speech.

1. Syntactic-aware Spontaneous Behavior Modeling

Spontaneous behaviors are not distributed randomly throughout a sentence; their meaning and prosodic realization are often tied to their syntactic position. For example, a filler word at the beginning of a sentence serves a different pragmatic function than one occurring mid-thought. The researchers identified a taxonomy of 19 specific spontaneous behaviors in Mandarin and developed a specialized encoder that processes both the behavior label and its syntactic context, allowing the model to learn the structural logic of human disfluency.

2. Fine-grained Spontaneous Prosody Modeling

To capture the non-linear pitch contours and rhythmic shifts typical of real-life talking, the system employs a dual-component prosody module:

  • Spontaneous Prosody Extractor: Used during training to extract high-fidelity "P-embeddings" from ground-truth audio.
  • Autoregressive (AR) Prosody Predictor: Used during inference to predict these expressive embeddings based on the input text and behavior labels. This explicit modeling ensures that when a behavior like "laughter" is requested, the resulting audio carries the correct acoustic texture rather than just a generic vocalization.

3. Three-Stage Training Strategy

The model undergoes a rigorous training pipeline to balance general linguistic knowledge with specialized stylistic control:

  1. Pre-training: The model is first trained on the WenetSpeech dataset, a 10,000-hour multi-domain Mandarin corpus, to establish a robust foundation for speech synthesis.
  2. Teaching: The prosody extractor is used to "teach" the model the specific acoustic signatures of spontaneous behaviors.
  3. Fine-tuning: The predictors and the acoustic model are jointly fine-tuned on a high-quality spontaneous dataset to ensure the final output is both natural and highly controllable.

Experimental Setup

The researchers utilized two primary datasets:

  • WenetSpeech: 10,000 hours of open-source Mandarin speech for backbone pre-training.
  • Internal Spontaneous Dataset: A specialized 5.4-hour dataset (4,968 records) from a native Mandarin female speaker. This dataset features meticulous manual annotations for 19 spontaneous behaviors, totaling 5,907 individual labels.

Results and Analysis

The proposed model was evaluated against VALL-E and FastSpeech 2 using Mean Opinion Scores (MOS) and objective acoustic metrics. The results demonstrate a significant leap in performance:

  • Prosody Naturalness (PN-MOS): The proposed model achieved a score of 4.09, vastly outperforming the VALL-E baseline of 3.30.
  • Behavior Naturalness (LN-MOS): In terms of how naturally the spontaneous behaviors were integrated, the model scored 4.046, compared to 3.324 for VALL-E.
  • Objective Accuracy: The model recorded the lowest Mel Cepstral Distortion (MCD) of 4.879, indicating that its synthesized waveforms were the most similar to the original human recordings.

An ablation study confirmed that both the explicit behavior labels and the fine-grained prosody embeddings are essential. Removing prosody modeling resulted in a 0.320 CMOS drop, while removing behavior modeling caused a 0.408 drop, highlighting that labels alone are insufficient without the underlying acoustic nuance.

Conclusion

This research demonstrates that spontaneous speech synthesis requires more than just large-scale data; it requires explicit, syntactically-aware modeling of human disfluencies and fine-grained control over prosody. By moving beyond "read-style" TTS, this system opens the door for more relatable AI assistants and more realistic digital humans. Future work will focus on scaling these techniques to multi-speaker datasets and exploring automated labeling methods to reduce the reliance on manual annotation.

Summary updated

BibTeX
@inproceedings{li2024spontaneous,
  author = {Weiqin Li and Peiji Yang and Yicheng Zhong and Yixuan Zhou and Zhisheng Wang and Zhiyong Wu and Xixin Wu and Helen Meng},
  title = {{Spontaneous Style Text-to-Speech Synthesis with Controllable Spontaneous Behaviors Based on Language Models}},
  booktitle = {Interspeech},
  year = {2024},
  doi = {10.21437/interspeech.2024-1989},
}
Figure