SYKI-SVC: Advancing Singing Voice Conversion with Post-Processing Innovations and an Open-Source Professional Testset

Yiquan Zhou, Wenyu Wang, Hongwu Ding, Jiacheng Xu, Jihua Zhu, Xin Gao, Shihao Li

TL;DR

SYKI-SVC is a high-fidelity singing voice conversion (SVC) system that achieves state-of-the-art naturalness by fusing ContentVec and Whisper features with a VITS-based architecture. Its standout innovations include a Mel-spectrogram supervisory loss to prevent information loss and a "borrowing" post-processing technique that restores high-frequency details directly from the source audio.

What Problem Does This Paper Address?

Singing Voice Conversion (SVC) is significantly more complex than standard speech-to-speech voice conversion. It requires the model to change the singer's identity (timbre) while perfectly preserving the source's melody, lyrics, and—most difficult of all—expressive vocal techniques like vibrato, falsetto, and growling.

Existing systems often suffer from two main issues:

  1. The "Muffled" Audio Problem: Neural vocoders often struggle to reconstruct high-frequency details (above 10 kHz), leading to a loss of "air" and clarity.
  2. Feature Entanglement: It is difficult to completely separate the singer's identity from the linguistic content, often resulting in a synthesized voice that sounds like a "leaky" hybrid of the source and target.

Furthermore, the authors identify a lack of professional-grade datasets specifically designed to test how well models handle advanced singing techniques.

Methods

The SYKI-SVC framework follows a recognition-synthesis paradigm and consists of three main stages:

  • Feature Extraction: The system extracts linguistic features by fusing outputs from two pre-trained models: ContentVec (a self-supervised model) and Whisper (an ASR model). These are combined via element-wise addition. For pitch, it uses RMVPE, a robust neural network-based F0 extractor.
  • Voice Converter (VITS-based): The core engine is based on the VITS architecture (Variational Inference with adversarial learning for end-to-end Text-to-Speech). To ensure the latent representation ($z$) doesn't lose critical content information, the authors added a Mel-spectrogram reconstruction loss as an additional supervisory signal during training.
  • High-Frequency Post-Processor: This is a clever, low-complexity signal processing step. The authors observed that timbre information is mostly concentrated below 10 kHz. Therefore, they use high-pass and low-pass filters to take the high-frequency components (>10 kHz) directly from the source audio, scale them to match the synthesized volume, and mix them back into the target output.

Datasets

The model was trained on a massive hybrid dataset including VCTK, NUS48e, Opencpop, M4singer, and Opensinger, along with a closed-source male database.

The authors also introduced a new Open-Source Professional Testset. This benchmark features 6 professional singers recorded in a studio, covering 21 distinct vocal techniques including:

  • Extreme techniques: Growling, Ultra High/Low Pitch, Opera.
  • Stylistic nuances: Vibrato, Breathy Voice, Crying Tone, Sobbing, and Rap.

Key Insights and Ideas

  • The "High-Freq Borrowing" Hack: Instead of forcing a neural network to hallucinate complex high-frequency noise (which is computationally expensive and prone to artifacts), simply "borrowing" it from the source works remarkably well because those frequencies contribute to "fidelity" but not "identity."
  • Feature Synergy: By combining ContentVec (which excels at disentangling speaker info) with Whisper (which provides robust linguistic grounding), the model achieves better "bite" and articulation in the converted voice.
  • Supervised Latent Space: Adding the Mel-loss forces the posterior encoder to be more descriptive. This ensures that the nuances of the source performance—the "how" of the singing—are captured in the latent space before being transformed by the target singer's timbre.

Key Results

The system was evaluated by both ordinary listeners and professional musicians across four dimensions: vocal naturalness, "bite" (articulation) reproduction, technique reproduction, and tone similarity.

  • Superior Naturalness: SYKI-SVC achieved a Mean Opinion Score (MOS) of 4.11, significantly outperforming popular models like so-vits (3.67) and DDSP-SVC (2.42).
  • Technique Preservation: It scored 4.09 in technique reproduction, proving it can handle the professional nuances in their new testset.
  • Ablation Impact: Removing the post-processor caused the naturalness score to drop from 4.11 to 3.82, highlighting that high-frequency restoration is critical for the "high-fidelity" feel.
  • High Resolution: While trained on 24kHz data, the post-processing allows the system to output audio with a perceived quality reaching 48kHz.

Limitations and Future Work

While the system performs exceptionally well, the authors note that the current implementation focuses on a many-to-one conversion system (converting many source singers to a specific target). Future work could involve expanding this to a more robust "any-to-any" framework. Additionally, while the post-processing trick is effective, a native 48kHz end-to-end training approach might eventually yield even more seamless results if the computational hurdles are cleared.

Summary updated

Abstract

Singing voice conversion aims to transform a source singing voice into that of a target singer while preserving the original lyrics, melody, and various vocal techniques. In this paper, we propose a high-fidelity singing voice conversion system. Our system builds upon the SVCC T02 framework and consists of three key components: a feature extractor, a voice converter, and a post-processor. The feature extractor utilizes the ContentVec and Whisper models to derive F0 contours and extract speaker-independent linguistic features from the input singing voice. The voice converter then integrates the extracted timbre, F0, and linguistic content to synthesize the target speaker's waveform. The post-processor augments high-frequency information directly from the source through simple and effective signal processing to enhance audio quality. Due to the lack of a standardized professional dataset for evaluating expressive singing conversion systems, we have created and made publicly available a specialized test set. Comparative evaluations demonstrate that our system achieves a remarkably high level of naturalness, and further analysis confirms the efficacy of our proposed system design.
BibTeX
@misc{zhou2025syki,
  author = {Yiquan Zhou and Wenyu Wang and Hongwu Ding and Jiacheng Xu and Jihua Zhu and Xin Gao and Shihao Li},
  title = {{SYKI-SVC: Advancing Singing Voice Conversion with Post-Processing Innovations and an Open-Source Professional Testset}},
  year = {2025},
  eprint = {2501.02953},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.02953},
}
Figure