Abstract
Recently, phonetic posteriorgrams (PPGs) based methods have been quite popular in non-parallel singing voice conversion systems. However, due to the lack of acoustic information in PPGs, style and naturalness of the converted singing voices are still limited. To solve these problems, in this paper, we utilize an acoustic reference encoder to implicitly model singing characteristics. We experiment with different auxiliary features, including mel spectrograms, HuBERT, and the middle hidden feature (PPG-Mid) of pretrained automatic speech recognition (ASR) model, as the input of the reference encoder, and finally find the HuBERT feature is the best choice. In addition, we use contrastive predictive coding (CPC) module to further smooth the voices by predicting future observations in latent space. Experiments show that, compared with the baseline models, our proposed model can significantly improve the naturalness of converted singing voices and the similarity with the target singer. Moreover, our proposed model can also make the speakers with just speech data sing.
Introduction
Non-parallel singing voice conversion (SVC) aims to transform the vocal timbre of a source singer to match a target singer without requiring parallel training data. A common approach involves using Phonetic Posteriorgrams (PPGs) extracted from Automatic Speech Recognition (ASR) models to represent linguistic content. While PPGs effectively decouple speaker identity, they often discard essential acoustic and musical nuances such as prosody, rhythm, and vocal style.
Previous attempts to reintroduce these elements via reference encoders using mel spectrograms often suffer from "source leakage," where the source singer's identity persists in the output. This paper proposes an end-to-end architecture that leverages HuBERT features for acoustic reference and Contrastive Predictive Coding (CPC) to enhance audio stability and naturalness.
Methodology
The proposed architecture integrates several components to balance identity disentanglement with musical fidelity.
Acoustic Reference Encoder with HuBERT
To capture musicality without leaking source identity, the model employs an auxiliary acoustic reference encoder. After evaluating mel spectrograms and internal ASR features (PPG-Mid), the authors found that HuBERT features provide an optimal balance. HuBERT's self-supervised representations retain rich prosody and rhythm while being relatively speaker-agnostic. To further ensure purity, adversarial singer and pitch confusion modules are applied to the encoder's output.
Contrastive Predictive Coding (CPC)
Local recognition errors in ASR-derived PPGs can lead to audible artifacts in synthesized singing. The system incorporates a CPC module as a regularization loss. By training the model to predict future latent representations from current context, it learns to ignore transient noise and local errors, resulting in smoother vocal synthesis. The final output is generated using a HiFi-GAN vocoder.
Experiments and Results
The model was trained on a combination of the VCTK Corpus (speech) and NUS-48E (singing/speech) to support both in-domain and cross-domain conversion tasks.
Performance Comparison
As shown in the experimental results, the proposed model (Prop_hub_cpc) significantly outperforms standard baselines in both naturalness and target similarity.
Key findings include:
- Feature Selection: HuBERT achieved a Mean Opinion Score (MOS) of 3.62 for naturalness, surpassing Mel spectrograms (3.56) and PPG-Mid (3.49).
- CPC Impact: The addition of the CPC module further improved naturalness to 3.74 and similarity to 3.87.
- Cross-Domain Conversion: The model successfully enabled "speech-to-singing" conversion, allowing non-singing speakers to produce singing voices with a Cosine Similarity (COS-SIM) of 0.67, outperforming baselines.
Conclusion
This work demonstrates that combining HuBERT-based acoustic references with CPC regularization effectively addresses the limitations of PPG-based singing voice conversion. The system achieves high-fidelity results and robust cross-domain performance. Future research will focus on further narrowing the quality gap between in-domain and cross-domain conversion.
Summary updated