FillerSpeech: Towards Human-Like Text-to-Speech Synthesis with Filler Insertion and Filler Style Control

Seung-Bin Kim, Jun-Hyeok Cha, Hyung-Seok Oh, Heejin Choi, Seong-Whan LeeEMNLP 2025

TL;DR

This paper introduces FillerSpeech, a novel text-to-speech (TTS) framework that seamlessly integrates natural-sounding fillers (like "um" and "uh") into synthesized speech and allows for fine-grained control over their style (pitch and duration). By constructing a unique filler-inclusive dataset and leveraging advanced techniques like flow matching and LLM-based prediction, FillerSpeech significantly enhances the naturalness and expressiveness of conversational TTS.

What Problem Does This Paper Address?

While modern TTS systems excel at generating high-quality audio, synthesized speech often lacks the natural hesitations, pauses, and non-verbal cues that characterize human conversation. Fillers, such as "um," "uh," and "well," are crucial for signaling thought processes, buying time, and maintaining conversational flow. Their absence in synthesized speech can make it sound robotic and less engaging, hindering its effectiveness in applications requiring natural human interaction. Existing approaches to filler synthesis have been limited in their ability to model diverse filler styles or achieve precise, text-driven control.

Methods

FillerSpeech tackles this challenge with a multi-pronged approach:

  • Filler-Inclusive Dataset Construction: The authors created a novel dataset by extracting and annotating filler instances from a large-scale speech corpus. Crucially, these fillers are labeled with pitch (F0) and duration information, eliminating the need for manual annotation.
  • Filler Style Control: To control the style of inserted fillers, the framework tokenizes filler style attributes (pitch and duration) and uses cross-attention mechanisms to integrate these style tokens with the input text. A dedicated pitch predictor is also incorporated to enhance prosodic control.
  • LLM-based Filler Prediction: To enable natural filler insertion based solely on text input, they fine-tune a large language model (LLM), specifically Vicuna-7B, using a LoRA adapter. This LLM predicts the position, type, duration, and pitch of fillers, considering the surrounding textual context. This allows for automatic filler insertion without requiring explicit filler markers in the input text.
  • Flow Matching Decoder: The core TTS synthesis is handled by a flow-matching decoder, a generative diffusion model known for its efficiency and high-quality output.

Datasets

  • LibriHeavy Corpus: The primary dataset used for training and evaluation was derived from the large-scale LibriHeavy corpus.
  • Newly Constructed Filler-Inclusive Dataset: The authors created a custom dataset by extracting and annotating filler instances from LibriHeavy. This dataset includes fillers with pitch and duration information, differing from standard TTS datasets.

Key Insights and Ideas

  • Fillers are Key to Conversational Naturalness: The paper strongly emphasizes that fillers are not mere imperfections but integral components of natural human speech, and their inclusion is vital for realistic conversational TTS.
  • Style Control is More Than Just Insertion: Simply inserting fillers isn't enough; controlling their prosodic characteristics (pitch and duration) is crucial for naturalness and expressiveness. The paper demonstrates that fine-grained control over these attributes significantly improves perceived naturalness.
  • LLMs are Powerful for Contextual Filler Prediction: Leveraging LLMs for filler prediction proves highly effective, allowing the system to infer appropriate filler placement and style based on textual context, moving beyond simple rule-based or statistical methods.
  • Pitch Predictor's Trade-off: While the pitch predictor significantly improves filler style control, it can slightly degrade overall pronunciation accuracy by shifting the text encoder's focus towards acoustic features. This highlights a nuanced trade-off in TTS model design.

Key Results

  • Subjective Evaluation (MOS): FillerSpeech achieved high MOS scores, indicating naturalness comparable to or exceeding baselines. The inclusion of pitch information led to improvements in UTMOS (an objective measure of speech quality).
  • Filler Naturalness and Controllability: Subjective evaluations (fMOS and cMOS) showed that FillerSpeech successfully generates natural-sounding fillers and provides effective control over their style, outperforming simpler filler tokenization methods.
  • LLM-based Filler Prediction: The fine-tuned Vicuna-7B model significantly outperformed baseline LLMs and random sampling in predicting filler position, type, duration, and pitch, demonstrating its ability to generate meaningful and contextually appropriate fillers.
  • Efficiency: The flow-matching decoder contributes to efficient synthesis, with rapid performance improvements observed even with fewer sampling steps.

Limitations and Future Work

  • Discrete Style Labels: The current approach uses discrete labels for pitch and duration. The authors acknowledge that this limits the model's ability to achieve extremely fine-grained or continuous stylistic control.
  • LLM Efficiency: While Vicuna-7B provides excellent performance, its large size can be a bottleneck for inference speed and resource usage. The authors suggest that their LoRA-based fine-tuning approach is applicable to smaller LLMs, offering a trade-off between performance and efficiency.
  • Future Work: The authors plan to explore more expressive speech synthesis and investigate control methods based on continuous values rather than discrete labels. They also aim to address potential malicious uses of realistic synthesized speech through detection and watermarking techniques.

Summary updated

BibTeX
@inproceedings{kim2025fillerspeech,
  author = {Seung-Bin Kim and Jun-Hyeok Cha and Hyung-Seok Oh and Heejin Choi and Seong-Whan Lee},
  title = {{FillerSpeech: Towards Human-Like Text-to-Speech Synthesis with Filler Insertion and Filler Style Control}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure