Abstract
Singing Voice Conversion (SVC) typically requires clean vocal inputs to accurately preserve melody and lyrics. However, real-world applications often involve vocals mixed with background music (BGM), where traditional pitch extraction fails and source separation introduces unwanted artifacts. This paper presents a robust any-to-one SVC framework that extracts melody features directly from mixed audio using fine-tuned Self-Supervised Learning (SSL) representations. By leveraging a learnable weighted sum of SSL layers, the system achieves superior pitch accuracy and naturalness in noisy environments compared to traditional pipelines.
Introduction
Singing Voice Conversion (SVC) is the task of transforming a source singer's timbre into that of a target singer while maintaining the original linguistic content and melodic structure. While SVC has seen significant progress with clean audio, its performance degrades sharply when the input contains background music (BGM). BGM interferes with fundamental frequency (F0) estimation, leading to "off-key" synthesis.
Current solutions—either noise-robust pitch trackers or pre-processing via music source separation—often fall short. Pitch trackers struggle with complex instrumental interference, while source separation algorithms introduce audio artifacts that degrade the final synthesized voice. This work proposes a method to perform SVC directly on mixed audio by utilizing robust melody features derived from SSL models like WavLM and HuBERT, bypassing the need for explicit source separation.
Methodology
SSL-Based Melody Extraction
The core innovation of this framework is a robust Melody Extractor built on top of pretrained SSL models. Rather than relying on the final layer of the SSL model, the system calculates a learnable weighted sum of all hidden layers. This approach acknowledges that different layers of a transformer-based model capture different acoustic properties. These representations are passed through Feed Forward Transformer (FFT) blocks to predict pitch, energy, and voiced/unvoiced flags directly from the mixed signal.
Feature Disentanglement and Conversion
To ensure high-quality conversion, the system must disentangle the singer's identity from the content and melody:
- Content Extraction: An Automatic Speech Recognition (ASR) model extracts Bottleneck Features (BNFs) to capture phonetic information (lyrics).
- Melody Encoding: The melody features are processed by an encoder using Conditional Instance Normalization to strip away the original singer's timbre.
- Adversarial Decoding: A decoder combines these features to generate Mel-spectrograms. The training process employs three discriminators: a Real/Fake discriminator for audio quality, a Conversion discriminator for target singer similarity, and an Embedding discriminator to ensure the source timbre is fully removed.
- Synthesis: A HiFi-GAN vocoder is used to synthesize the final waveform from the generated Mel-spectrograms.
Experimental Setup
The Melody Extractor was trained using MUSDB (150 professional songs) and OpenSinger (50 hours of clean vocals), with BGM from MUSDB randomly added to OpenSinger to simulate noisy conditions. For the SVC framework, Opencpop (5.2 hours of professional female vocals) served as the target data, while OpenSinger provided the source data.
Key Insights
The Role of SSL Layers
Analysis of the SSL models revealed that in their pretrained state, melody-related information is primarily concentrated in the shallow layers (CNN extractor and lower transformer layers). However, the authors found that fine-tuning the SSL model specifically for melody extraction "wakes up" the higher transformer layers, allowing them to contribute significantly to pitch accuracy in the presence of BGM.
Performance in Noisy Conditions
The proposed model (using WavLM) was evaluated against state-of-the-art baselines, including the Crepe neural pitch extractor and a pipeline involving source separation followed by Crepe. In 10dB SNR conditions, the proposed method achieved a Naturalness Mean Opinion Score (NMOS) of 3.49, whereas the Crepe baseline without separation collapsed to 2.40.
In terms of pitch accuracy, the model achieved an F0 Root Mean Square Error (F0RMSE) of 0.174 at 15dB SNR, outperforming both the standard Crepe (0.208) and the Separated+Crepe pipeline (0.176). Importantly, the model maintains high performance on clean audio (3.54 NMOS), proving it does not sacrifice quality for robustness.
Conclusion and Future Work
This research demonstrates that directly extracting melody features from mixed audio using SSL representations is more effective than traditional source-separation pipelines. By leveraging the hierarchical information within SSL models, the system provides a more practical solution for real-world SVC applications. Future work will focus on expanding the architecture from any-to-one to any-to-many conversion and testing robustness in more diverse acoustic environments.
Summary updated