TL;DR
SYKI-SVC is a high-fidelity singing voice conversion (SVC) system that achieves state-of-the-art naturalness by fusing ContentVec and Whisper features with a VITS-based architecture. Its standout innovations include a Mel-spectrogram supervisory loss to prevent information loss and a "borrowing" post-processing technique that restores high-frequency details directly from the source audio.
What Problem Does This Paper Address?
Singing Voice Conversion (SVC) is significantly more complex than standard speech-to-speech voice conversion. It requires the model to change the singer's identity (timbre) while perfectly preserving the source's melody, lyrics, and—most difficult of all—expressive vocal techniques like vibrato, falsetto, and growling.
Existing systems often suffer from two main issues:
- The "Muffled" Audio Problem: Neural vocoders often struggle to reconstruct high-frequency details (above 10 kHz), leading to a loss of "air" and clarity.
- Feature Entanglement: It is difficult to completely separate the singer's identity from the linguistic content, often resulting in a synthesized voice that sounds like a "leaky" hybrid of the source and target.
Furthermore, the authors identify a lack of professional-grade datasets specifically designed to test how well models handle advanced singing techniques.
Methods
The SYKI-SVC framework follows a recognition-synthesis paradigm and consists of three main stages:
- Feature Extraction: The system extracts linguistic features by fusing outputs from two pre-trained models: ContentVec (a self-supervised model) and Whisper (an ASR model). These are combined via element-wise addition. For pitch, it uses RMVPE, a robust neural network-based F0 extractor.
- Voice Converter (VITS-based): The core engine is based on the VITS architecture (Variational Inference with adversarial learning for end-to-end Text-to-Speech). To ensure the latent representation ($z$) doesn't lose critical content information, the authors added a Mel-spectrogram reconstruction loss as an additional supervisory signal during training.
- High-Frequency Post-Processor: This is a clever, low-complexity signal processing step. The authors observed that timbre information is mostly concentrated below 10 kHz. Therefore, they use high-pass and low-pass filters to take the high-frequency components (>10 kHz) directly from the source audio, scale them to match the synthesized volume, and mix them back into the target output.
Datasets
The model was trained on a massive hybrid dataset including VCTK, NUS48e, Opencpop, M4singer, and Opensinger, along with a closed-source male database.
The authors also introduced a new Open-Source Professional Testset. This benchmark features 6 professional singers recorded in a studio, covering 21 distinct vocal techniques including:
- Extreme techniques: Growling, Ultra High/Low Pitch, Opera.
- Stylistic nuances: Vibrato, Breathy Voice, Crying Tone, Sobbing, and Rap.
Key Insights and Ideas
- The "High-Freq Borrowing" Hack: Instead of forcing a neural network to hallucinate complex high-frequency noise (which is computationally expensive and prone to artifacts), simply "borrowing" it from the source works remarkably well because those frequencies contribute to "fidelity" but not "identity."
- Feature Synergy: By combining ContentVec (which excels at disentangling speaker info) with Whisper (which provides robust linguistic grounding), the model achieves better "bite" and articulation in the converted voice.
- Supervised Latent Space: Adding the Mel-loss forces the posterior encoder to be more descriptive. This ensures that the nuances of the source performance—the "how" of the singing—are captured in the latent space before being transformed by the target singer's timbre.
Key Results
The system was evaluated by both ordinary listeners and professional musicians across four dimensions: vocal naturalness, "bite" (articulation) reproduction, technique reproduction, and tone similarity.
- Superior Naturalness: SYKI-SVC achieved a Mean Opinion Score (MOS) of 4.11, significantly outperforming popular models like
so-vits(3.67) andDDSP-SVC(2.42). - Technique Preservation: It scored 4.09 in technique reproduction, proving it can handle the professional nuances in their new testset.
- Ablation Impact: Removing the post-processor caused the naturalness score to drop from 4.11 to 3.82, highlighting that high-frequency restoration is critical for the "high-fidelity" feel.
- High Resolution: While trained on 24kHz data, the post-processing allows the system to output audio with a perceived quality reaching 48kHz.
Limitations and Future Work
While the system performs exceptionally well, the authors note that the current implementation focuses on a many-to-one conversion system (converting many source singers to a specific target). Future work could involve expanding this to a more robust "any-to-any" framework. Additionally, while the post-processing trick is effective, a native 48kHz end-to-end training approach might eventually yield even more seamless results if the computational hurdles are cleared.
Summary updated