HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis

Sang-Hoon Lee, Ha-Yeong Choi, Seung-Bin Kim, Seong-Whan Lee

TL;DR

HierSpeech++ introduces a novel hierarchical speech synthesis framework that significantly improves zero-shot text-to-speech (TTS) and voice conversion (VC) by effectively bridging semantic and acoustic representations. This approach achieves human-level quality, faster inference, and greater robustness compared to existing LLM-based and diffusion models, even with limited data.

What Problem Does This Paper Address?

The field of speech synthesis, particularly for zero-shot scenarios (where the model synthesizes speech for a speaker it hasn't been explicitly trained on), faces several challenges. Large Language Model (LLM)-based approaches, while powerful, often suffer from slow inference speeds and a lack of robustness, leading to issues like repetition or skipping. Existing methods also struggle with accurately capturing speaker identity and prosody in zero-shot settings, often requiring large datasets and extensive training. Furthermore, many models have a "train-inference mismatch" problem, where the way speech is processed during training differs from how it's generated during inference, impacting quality. This paper aims to address these limitations by proposing a faster, more robust, and higher-quality zero-shot speech synthesis system.

Methods

HierSpeech++ is built upon a hierarchical speech synthesis framework, combining three key components:

  • Hierarchical Speech Synthesizer: This is the core of the system, employing a hierarchical variational autoencoder (VAE) to disentangle semantic and acoustic information. It uses a dual-audio acoustic encoder to capture richer acoustic representations and a source-filter multi-path semantic encoder to separate speaker-related and speaker-agnostic information. This hierarchical approach allows for better control over prosody and voice style.
  • Text-to-Vec (TTV): For TTS tasks, this module generates a semantic representation and fundamental frequency (F0) from text input, leveraging self-supervised speech representations to capture linguistic information and prosody.
  • Speech Super-Resolution (SpeechSR): This component efficiently upsamples 16 kHz audio to 48 kHz, improving audio quality and enabling the use of lower-resolution datasets for training.

A key innovation is the use of self-supervised speech representations (specifically from the MMS model) to bridge the gap between text and acoustic features, improving expressiveness and robustness. They also introduce a bidirectional Transformer-based normalizing flow (BiT-Flow) to mitigate the train-inference mismatch problem.

Datasets

The primary dataset used for training the hierarchical speech synthesizer is LibriTTS (train-clean-100, train-clean-360, and train-other-500 subsets). For TTV, only the LibriTTS dataset was used. For SpeechSR, the VCTK dataset was used for evaluation, and a larger dataset including MSSS and EXPRESSO was used for better performance. They also mention using NIKL and AIHub datasets.

Key Insights and Ideas

  • Hierarchy is Key: The paper strongly advocates for hierarchical modeling in speech synthesis, showing it significantly improves robustness and expressiveness, especially in zero-shot scenarios.
  • Bridging the Semantic-Acoustic Gap: Effectively using self-supervised speech representations (like MMS) is crucial for capturing rich linguistic and prosodic information, which is vital for natural-sounding speech.
  • Dual-Audio Encoder for Richer Acoustics: Combining waveform and linear spectrogram information in the acoustic encoder leads to a more comprehensive representation, boosting reconstruction quality.
  • Bidirectional Flow for Robustness: Employing BiT-Flow helps reduce the train-inference mismatch, leading to more consistent and higher-quality synthesized speech.
  • Efficient Super-Resolution: The SpeechSR module offers a fast and effective way to enhance audio quality, making it practical for real-world applications and enabling better data utilization.
  • Style Prompt Replication (SPR): A simple yet effective trick to improve style transfer from very short prompts (even 1 second) by replicating the prompt sequence.

Key Results

  • Human-Level Quality: HierSpeech++ achieves human-level quality in zero-shot TTS and VC tasks, outperforming LLM-based and diffusion models.
  • Improved Metrics: The model shows significant improvements across various objective metrics like Mel error, PESQ, and pitch consistency, as well as subjective MOS scores for naturalness and speaker similarity.
  • Faster Inference: HierSpeech++ offers a much faster inference speed compared to previous state-of-the-art models, particularly for SpeechSR which is significantly faster than AudioSR.
  • Robustness: The model demonstrates strong robustness to noisy speech prompts and achieves better CER/WER scores than even ground truth in some cases.
  • Data Efficiency: The framework can achieve high performance even with limited data, and the SpeechSR component allows for effective use of low-resolution datasets.

Limitations and Future Work

  • Noise Handling: While the model is robust to noisy prompts, it can still synthesize repeated background noise from the global voice style representation. The paper explores using a denoiser but notes it can degrade pronunciation. Future work could focus on better disentangling voice and noise.
  • Cross-lingual and Emotion Control: The authors suggest extending the model to cross-lingual and emotion-controllable speech synthesis by leveraging pre-trained models.
  • Speech-to-Speech Translation: The hierarchical framework could be adapted for speech-to-speech translation using non-autoregressive generation.
  • Data Scale: While the model performs well with available datasets, the authors acknowledge that increasing the dataset scale further could yield even better results.

Summary updated

Abstract

Large language model (LLM)-based speech synthesis has been widely adopted in zero-shot speech synthesis. However, they require a large-scale data and possess the same limitations as previous autoregressive speech models, including slow inference speed and lack of robustness. This article proposes HierSpeech++, a fast and strong zero-shot speech synthesizer for text-to-speech (TTS) and voice conversion (VC). We verified that hierarchical speech synthesis frameworks could significantly improve the robustness and expressiveness of the synthetic speech. Furthermore, we significantly improve the naturalness and speaker similarity of synthetic speech even in zero-shot speech synthesis scenarios. For TTS, we adopt the text-to-vec (TTV) framework, which generates a self-supervised speech representation and an F0 representation based on text representations and prosody prompts. Then, HierSpeech++ generates speech from the generated vector, F0, and voice prompt. We further introduce a high-efficient speech super-resolution (SpeechSR) framework from 16 to 48 kHz. The experimental results demonstrated that the hierarchical variational autoencoder could be a strong zero-shot speech synthesizer given that it outperforms LLM-based and diffusion-based models. Moreover, we achieved the first human-level quality zero-shot speech synthesis. Audio samples and source code are available at https://github.com/hierspeechpp/code
BibTeX
@article{lee2025hierspeech,
  author = {Sang-Hoon Lee and Ha-Yeong Choi and Seung-Bin Kim and Seong-Whan Lee},
  title = {{HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2025},
  doi = {10.1109/tnnls.2025.3584944},
}
Figure