Bayesian Speech synthesizers Can Learn from Multiple Teachers

Ziyang Zhang, Yifan Gao, Xuenan Xu, Baoxiangli, Wen Wu, Chao Zhang

Abstract

Codec-based text-to-speech (TTS) models have recently gained traction for their efficiency and strong performance in voice cloning. However, codec-based TTS faces limitations due to the challenges of pretraining robust speech codecs and the quality degradation introduced by quantization errors. Emerging evidence suggests that continuous-valued generative models can alleviate these issues and serve as a promising alternative. Yet, effectively modelling diverse speech patterns and developing reliable sampling strategies for continuous-valued autoregressive (AR) TTS remains underexplored. In this work, we propose BELLE, Bayesian evidential learning with language modelling for TTS, a novel continuous-valued AR framework that directly predicts mel-spectrograms from textual input. BELLE treats each mel-spectrogram frame as a Gaussian distribution sampled from a learned hyper distribution, enabling principled uncertainty estimation, particularly in scenarios with parallel data (i.e., one text-audio prompt paired with multiple speech samples). To obtain such data, diverse speech samples are synthesized using multiple pre-trained TTS models given the same text-audio prompts, which are distilled into BELLE via Bayesian evidential learning. Experimental results indicate that BELLE demonstrates highly competitive performance compared with the current best open-source TTS models, even though BELLE is trained on a large amount of synthetic data and uses only approximately one-tenth of their training data.

Introduction

Text-to-Speech (TTS) is fundamentally a "one-to-many" problem: there are infinite ways to say the same sentence depending on prosody, rhythm, and emotion. However, most modern TTS systems oversimplify this into a "one-to-one" mapping. Current continuous-valued models typically rely on a fixed-variance prior, which forces the model toward a static point estimate. This leads to "oversmoothing," where the model produces a boring "average" of all possible prosodies to minimize error. Furthermore, standard datasets only provide one recording per text, making it nearly impossible for a model to learn the natural variance of human speech.

Methodology: The BELLE Framework

The proposed framework, BELLE, shifts the paradigm from deterministic prediction to principled Bayesian inference. Unlike standard regression models, BELLE explicitly models "aleatoric uncertainty"—the inherent randomness in speech data.

Bayesian Evidential Learning

Instead of predicting a single acoustic value, BELLE predicts the parameters of a Normal-Inverse-Gamma (NIG) distribution ($\gamma, \nu, \alpha, \beta$). This allows the model to capture the distributional envelope of potential speech outputs.

The architecture of BELLE, featuring the Bayesian Evidential sampling module and hierarchical sampling process.

Hierarchical Sampling and Student-t Predictive Distribution

During generation, the model samples a variance from an Inverse-Gamma distribution, then a mean from a Gaussian, and finally the output frame. This process effectively results in a Student-t predictive distribution, which is more expressive and handles "heavy-tailed" outliers (like expressive intonations) better than a standard Gaussian. As the model becomes more certain (evidence $\nu \to \infty$), it naturally reverts to a Gaussian, but it retains the flexibility to be "non-Gaussian" when the data is ambiguous.

One-to-Many Training Strategy

To teach the model about variance when only one human recording is available, the researchers used six different pre-trained TTS models (the "teachers") to generate synthetic versions of the same text. These synthetic samples act as a "statistical support set," helping the model map out the boundaries of what is "plausible" speech.

Experimental Setup

BELLE was trained on approximately 706 hours of human data from the LibriSpeech training set and synthetic data generated by six models (CosyVoice2, IndexTTS, SparkTTS, F5TTS, MaskGCT, and XTTS-v2), totaling 4,817 hours. Evaluation was conducted on the LibriSpeech test-clean subset using "Continuation" (completing a prompt) and "Cross-sentence" (zero-shot voice cloning) tasks.

Key Results

Experimental results indicate that BELLE demonstrates highly competitive performance compared with the current best open-source TTS models, despite using significantly less training data.

Performance comparison of BELLE against state-of-the-art TTS models on LibriSpeech test-clean.
  • Data Efficiency: BELLE, trained on ~5k hours, outperformed leading open-source models (like F5-TTS) trained on 50k hours.
  • Robustness: It achieved a 25.8% relative reduction in Word Error Rate (WER) compared to the MELLE baseline under identical conditions.
  • Speaker Similarity: In subjective tests, BELLE achieved the highest speaker similarity score (4.13 SMOS), even beating the Ground Truth (3.81), likely because it replicates the reference's acoustic characteristics more stably.
  • Streaming Performance: The "BELLE-stream" variant achieved a Real-Time Factor (RTF) of 0.55 and a first-packet latency of ~440ms, proving it is ready for interactive dialogue systems.

Conclusion

BELLE introduces a robust Bayesian framework for continuous-valued AR TTS. By leveraging multiple teachers and evidential learning, it avoids the "mean-seeking" behavior of standard regression and achieves state-of-the-art performance with high data efficiency. Future work will explore multilingual generalization and further latency reductions for real-time applications.

Summary updated

Abstract

Codec-based text-to-speech (TTS) models have recently gained traction for their efficiency and strong performance in voice cloning. However, codec-based TTS faces limitations due to the challenges of pretraining robust speech codecs and the quality degradation introduced by quantization errors. Emerging evidence suggests that continuous-valued generative models can alleviate these issues and serve as a promising alternative. Yet, effectively modelling diverse speech patterns and developing reliable sampling strategies for continuous-valued autoregressive (AR) TTS remains underexplored. In this work, we propose BELLE, Bayesian evidential learning with language modelling for TTS, a novel continuous-valued AR framework that directly predicts mel-spectrograms from textual input. BELLE treats each mel-spectrogram frame as a Gaussian distribution sampled from a learned hyper distribution, enabling principled uncertainty estimation, particularly in scenarios with parallel data (i.e., one text-audio prompt paired with multiple speech samples). To obtain such data, diverse speech samples are synthesized using multiple pre-trained TTS models given the same text-audio prompts, which are distilled into BELLE via Bayesian evidential learning. Experimental results indicate that BELLE demonstrates highly competitive performance compared with the current best open-source TTS models, even though BELLE is trained on a large amount of synthetic data and uses only approximately one-tenth of their training data. Audio samples generated by BELLE are available at https://belletts.github.io/Belle/. The code, checkpoints, and synthetic data will be released after the paper is accepted.
BibTeX
@misc{zhang2025bayesian,
  author = {Ziyang Zhang and Yifan Gao and Xuenan Xu and Baoxiangli and Wen Wu and Chao Zhang},
  title = {{Bayesian Speech synthesizers Can Learn from Multiple Teachers}},
  year = {2025},
  eprint = {2510.24372},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.24372},
}
Figure