TL;DR
HierSpeech++ introduces a novel hierarchical speech synthesis framework that significantly improves zero-shot text-to-speech (TTS) and voice conversion (VC) by effectively bridging semantic and acoustic representations. This approach achieves human-level quality, faster inference, and greater robustness compared to existing LLM-based and diffusion models, even with limited data.
What Problem Does This Paper Address?
The field of speech synthesis, particularly for zero-shot scenarios (where the model synthesizes speech for a speaker it hasn't been explicitly trained on), faces several challenges. Large Language Model (LLM)-based approaches, while powerful, often suffer from slow inference speeds and a lack of robustness, leading to issues like repetition or skipping. Existing methods also struggle with accurately capturing speaker identity and prosody in zero-shot settings, often requiring large datasets and extensive training. Furthermore, many models have a "train-inference mismatch" problem, where the way speech is processed during training differs from how it's generated during inference, impacting quality. This paper aims to address these limitations by proposing a faster, more robust, and higher-quality zero-shot speech synthesis system.
Methods
HierSpeech++ is built upon a hierarchical speech synthesis framework, combining three key components:
- Hierarchical Speech Synthesizer: This is the core of the system, employing a hierarchical variational autoencoder (VAE) to disentangle semantic and acoustic information. It uses a dual-audio acoustic encoder to capture richer acoustic representations and a source-filter multi-path semantic encoder to separate speaker-related and speaker-agnostic information. This hierarchical approach allows for better control over prosody and voice style.
- Text-to-Vec (TTV): For TTS tasks, this module generates a semantic representation and fundamental frequency (F0) from text input, leveraging self-supervised speech representations to capture linguistic information and prosody.
- Speech Super-Resolution (SpeechSR): This component efficiently upsamples 16 kHz audio to 48 kHz, improving audio quality and enabling the use of lower-resolution datasets for training.
A key innovation is the use of self-supervised speech representations (specifically from the MMS model) to bridge the gap between text and acoustic features, improving expressiveness and robustness. They also introduce a bidirectional Transformer-based normalizing flow (BiT-Flow) to mitigate the train-inference mismatch problem.
Datasets
The primary dataset used for training the hierarchical speech synthesizer is LibriTTS (train-clean-100, train-clean-360, and train-other-500 subsets). For TTV, only the LibriTTS dataset was used. For SpeechSR, the VCTK dataset was used for evaluation, and a larger dataset including MSSS and EXPRESSO was used for better performance. They also mention using NIKL and AIHub datasets.
Key Insights and Ideas
- Hierarchy is Key: The paper strongly advocates for hierarchical modeling in speech synthesis, showing it significantly improves robustness and expressiveness, especially in zero-shot scenarios.
- Bridging the Semantic-Acoustic Gap: Effectively using self-supervised speech representations (like MMS) is crucial for capturing rich linguistic and prosodic information, which is vital for natural-sounding speech.
- Dual-Audio Encoder for Richer Acoustics: Combining waveform and linear spectrogram information in the acoustic encoder leads to a more comprehensive representation, boosting reconstruction quality.
- Bidirectional Flow for Robustness: Employing BiT-Flow helps reduce the train-inference mismatch, leading to more consistent and higher-quality synthesized speech.
- Efficient Super-Resolution: The SpeechSR module offers a fast and effective way to enhance audio quality, making it practical for real-world applications and enabling better data utilization.
- Style Prompt Replication (SPR): A simple yet effective trick to improve style transfer from very short prompts (even 1 second) by replicating the prompt sequence.
Key Results
- Human-Level Quality: HierSpeech++ achieves human-level quality in zero-shot TTS and VC tasks, outperforming LLM-based and diffusion models.
- Improved Metrics: The model shows significant improvements across various objective metrics like Mel error, PESQ, and pitch consistency, as well as subjective MOS scores for naturalness and speaker similarity.
- Faster Inference: HierSpeech++ offers a much faster inference speed compared to previous state-of-the-art models, particularly for SpeechSR which is significantly faster than AudioSR.
- Robustness: The model demonstrates strong robustness to noisy speech prompts and achieves better CER/WER scores than even ground truth in some cases.
- Data Efficiency: The framework can achieve high performance even with limited data, and the SpeechSR component allows for effective use of low-resolution datasets.
Limitations and Future Work
- Noise Handling: While the model is robust to noisy prompts, it can still synthesize repeated background noise from the global voice style representation. The paper explores using a denoiser but notes it can degrade pronunciation. Future work could focus on better disentangling voice and noise.
- Cross-lingual and Emotion Control: The authors suggest extending the model to cross-lingual and emotion-controllable speech synthesis by leveraging pre-trained models.
- Speech-to-Speech Translation: The hierarchical framework could be adapted for speech-to-speech translation using non-autoregressive generation.
- Data Scale: While the model performs well with available datasets, the authors acknowledge that increasing the dataset scale further could yield even better results.
Summary updated