Introduction
Singing Voice Conversion (SVC) is the task of transforming a source singer's voice into a target singer's voice while strictly maintaining the original linguistic content and melodic structure. Unlike standard speech conversion, SVC must handle the complex acoustic dynamics, extended pitch ranges, and expressive nuances inherent in singing. Traditional approaches using Generative Adversarial Networks (GANs) or regression models often struggle with training stability or produce "blurry" acoustic features.
DiffSVC addresses these challenges by being the first to apply Denoising Diffusion Probabilistic Models (DDPM) to the SVC task. By framing mel spectrogram generation as an iterative refinement process from Gaussian noise, DiffSVC achieves state-of-the-art performance in both naturalness and voice similarity.
Methodology
The Diffusion Framework
DiffSVC utilizes a diffusion-based generative framework consisting of two processes:
- Forward Process: Clean mel spectrograms are gradually corrupted with Gaussian noise until they reach a state of isotropic Gaussian noise.
- Reverse Process: A neural network is trained to iteratively "denoise" the signal. Starting from pure noise, the model predicts and removes noise at each step to reconstruct a high-fidelity mel spectrogram.
Architecture and Conditioning
To ensure the generated voice preserves the source's performance while adopting the target's timbre, the model incorporates several conditioning signals into its Diffusion Decoder.
As shown in the architecture diagram, the Diffusion Decoder is a bidirectional residual convolutional network. It is conditioned on three key auxiliary inputs:
- Phonetic Posteriorgrams (PPGs): Extracted from an ASR model to represent linguistic content.
- Fundamental Frequency (Log-F0): To capture the melody.
- Loudness: Extracted via A-weighting to preserve vocal dynamics.
The final mel spectrogram is converted into raw audio using a pre-trained HiFi-GAN vocoder.
Key Results
The performance of DiffSVC was evaluated against three strong baselines: BLSTM-SVC, Seq2seq-SVC, and FastSVC (a GAN-based model). Evaluation metrics included subjective Mean Opinion Scores (MOS) for naturalness and similarity, as well as objective metrics like Mel-Cepstrum Distortion (MCD) and F0 correlation (FPC).
DiffSVC significantly outperformed the baselines in subjective naturalness (3.97 MOS) and voice similarity (4.67 MOS). Objectively, it achieved the best MCD score (6.307), demonstrating its superior ability to reconstruct the target singer's acoustic characteristics. While GAN-based models like FastSVC offer faster inference, DiffSVC provides higher-fidelity output by avoiding the common pitfalls of GAN training and the blurriness of simple regression.
Conclusion and Future Work
DiffSVC demonstrates that diffusion models are highly effective for the detailed textures required in singing voice conversion. While the current implementation focuses on any-to-one conversion with 100 refinement steps, future research aims to extend the model to any-to-any conversion and improve inference speed using advanced sampling techniques like DDIM. Additionally, exploring low-data scenarios remains a key area for development to make high-quality SVC more accessible.
Summary updated