HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models

Ji-Sang Hwang, Sang-Hoon Lee, Seong-Whan Lee

TL;DR

This paper introduces HiddenSinger, a novel singing voice synthesis (SVS) system that leverages neural audio codecs and latent diffusion models to achieve high-quality and expressive singing. It also presents HiddenSinger-U, an unsupervised framework that enables training with unlabeled singing voice data, addressing the challenge of data scarcity.

What Problem Does This Paper Address?

Synthesizing realistic and expressive singing voices from musical scores (SVS) is a challenging task. Traditional SVS systems often rely on a two-stage pipeline: first generating an intermediate acoustic representation (like a Mel-spectrogram) and then using a separate vocoder to convert it into audio. This approach suffers from limitations such as a training-inference mismatch, difficulty in applying latent learning for audio generation, and the need for large, high-quality paired datasets of musical scores and singing voices. More recent end-to-end systems, while improving quality, still face computational challenges due to high-dimensional audio processing and can exhibit pitch inaccuracies. Furthermore, acquiring sufficient paired data for training is a significant bottleneck, especially for new speakers or styles.

Methods

HiddenSinger tackles these issues with a multi-component approach:

  • Neural Audio Codec: The system employs an audio autoencoder that compresses audio into a low-dimensional latent representation using residual vector quantization (RVQ). This RVQ-based approach is crucial for achieving high-fidelity audio generation and computational efficiency by creating a more compact and manageable latent space compared to standard vector quantization or KL-regularized spaces.
  • Latent Diffusion Models: A latent generator, powered by diffusion models, is used to sample a latent representation conditioned on a musical score. This allows for a more robust and controllable generation process. The diffusion model starts denoising from a "data-driven prior" distribution, which is closer to the target latent representation, making the denoising process more efficient and stable.
  • Condition Encoder: To guide the generation process, a condition encoder extracts relevant information from the musical score, including lyrics and melody. This encoder is enhanced with an "enhanced condition encoder" that combines lyric and melody representations, which the authors found crucial for accurate pronunciation and overall synthesis quality.
  • Unsupervised Singing Voice Learning Framework (HiddenSinger-U): To address the data scarcity problem, HiddenSinger-U extends the core model to train using unpaired singing voice data. It introduces unsupervised lyrics and melody encoders that leverage self-supervised speech representations (from models like XLS-R) and pitch information, respectively. Contrastive learning is employed to align these unsupervised representations with their paired counterparts, further stabilizing training.

Datasets

The primary dataset used is the Guide vocal dataset, containing approximately 157.39 hours of audio for 4,000 paired Korean songs. This dataset was segmented and split into training, validation, and testing sets.

For training the audio autoencoder, a multi-speaker singing dataset and a children singing dataset were also used, totaling 285.1 hours of audio for 8,781 K-pop songs.

For evaluating the unsupervised framework (HiddenSinger-U), an internal singing voice dataset was used, containing approximately 3.30 hours of audio for 316 Korean songs without musical scores. This dataset was crucial for assessing the model's ability to learn from unlabeled data.

Key Insights and Ideas

  • RVQ for Latent Space Stability: The authors found that Residual Vector Quantization (RVQ) in the audio autoencoder significantly improves the stability and quality of latent space sampling for diffusion models, outperforming standard VQ or KL-regularization. This is key for generating high-fidelity audio.
  • Data-Driven Priors in Diffusion: Using "data-driven priors" for the diffusion model's starting point (instead of standard Gaussian noise) makes the denoising process more efficient and stable, leading to better generation quality.
  • Enhanced Condition Encoder is Crucial: The proposed "enhanced condition encoder" that fuses lyric and melody information was found to be essential for accurate pronunciation and overall synthesis quality, highlighting the importance of rich conditioning signals.
  • Unsupervised Learning for Data Scarcity: HiddenSinger-U demonstrates that effective SVS can be achieved even with a significant portion of unlabeled data, making the technology more accessible and adaptable to new speakers or styles where paired data is scarce.
  • Contrastive Learning for Alignment: In the unsupervised setting, contrastive learning plays a vital role in aligning the learned representations from unpaired data, improving the overall coherence and quality of the synthesized singing.

Key Results

  • Subjective Quality: HiddenSinger and HiddenSinger-U outperformed existing SVS models (including HiFi-GAN, FastSpeech 2 + HiFi-GAN, DiffSinger + HiFi-GAN, and VISinger) in terms of naturalness (nMOS scores). HiddenSinger achieved an nMOS of 3.80±0.08, and HiddenSinger-U achieved 3.83±0.08, both close to the ground truth (4.48±0.06).
  • Objective Metrics: While VISinger showed slightly better performance in MAE and periodicity error, HiddenSinger and HiddenSinger-U maintained competitive scores across objective metrics like pitch error and V/UV F1.
  • Audio Autoencoder Performance: The audio autoencoder, when used with the HiFi-GAN V1 generator, achieved comparable objective metrics to HiFi-GAN itself and slightly better subjective naturalness.
  • Unsupervised Learning Effectiveness: HiddenSinger-U showed statistically insignificant degradation in nMOS even with up to 20% unlabeled data, demonstrating its robustness. Similarity scores (SMOS) did decrease with higher unlabeled ratios, but the framework remained effective.

Limitations and Future Work

  • Iterative Generation: The diffusion models require a number of iterative processes for generation. The authors suggest exploring consistency models to distill teacher diffusion models for single-step generation.
  • Singing Style Adaptation: The current model focuses on adapting to new speakers but not necessarily new singing styles. Future work aims to implement zero-shot singing style transfer using style-generalized generative models.
  • Similarity Degradation in Unsupervised Learning: While nMOS remained high, the similarity of the synthesized singing voice decreased with an increased ratio of unlabeled data in HiddenSinger-U. This is attributed to subtle differences between musical score pitch and human F0, leading to slight speaker identity variations. The authors believe contrastive learning and information perturbation help stabilize training but acknowledge this limitation.

Summary updated

Abstract

Highlights•Introduce HiddenSinger, a high-quality singing voice synthesis model.•Propose HiddenSinger-U, an unsupervised learning framework to train with unlabeled datasets.•Audio samples are available at https://jisang93.github.io/hiddensinger-demo/.
BibTeX
@article{hwang2023hiddensinger,
  author = {Ji-Sang Hwang and Sang-Hoon Lee and Seong-Whan Lee},
  title = {{HiddenSinger: High-Quality Singing Voice Synthesis via Neural Audio Codec and Latent Diffusion Models}},
  journal = {Neural Networks},
  year = {2023},
  doi = {10.1016/j.neunet.2024.106762},
}
Figure