PitchNet: Unsupervised Singing Voice Conversion with Pitch Adversarial Network

Chengqi Deng, Chengzhu Yu, Heng Lu, Chao Weng, Dong Yu

Abstract

Singing voice conversion aims to transform a singer's vocal identity while preserving the original content and melody. Traditional unsupervised methods often suffer from "out of key" artifacts due to poor disentanglement of pitch and phonetic content. We propose PitchNet, which utilizes an adversarially trained pitch regression network to learn pitch-invariant representations. By feeding explicitly extracted pitch directly to the decoder, PitchNet achieves superior conversion quality and enables flexible pitch manipulation. Our evaluation demonstrates a significant improvement in Mean Opinion Score (MOS) from 2.92 to 3.75 compared to baseline methods.

The Challenge of Singing Voice Conversion

Singing voice conversion (SVC) is inherently more complex than speech conversion. Singing involves a broader frequency range, sustained vowels, and precise melodic contours. Existing unsupervised approaches, typically based on autoencoders, often fail to separate the phonetic content (the lyrics) from the pitch (the melody). This lack of disentanglement leads to "pitch leakage," where the converted voice sounds robotic or dissonant because the model cannot distinguish between the source singer's identity and the intended melody.

Methods: Achieving Disentanglement

PitchNet addresses these issues by introducing a novel adversarial training strategy within a WaveNet-based autoencoder framework.

The PitchNet architecture featuring an encoder, adversarial networks (Singer and Pitch), and a decoder with explicit pitch injection.

The architecture consists of three core components:

  1. Encoder with Bottleneck: The encoder extracts high-level features from the source audio. To prevent the model from simply passing through raw audio information, a bottleneck (average pooling) is applied.
  2. Adversarial Networks: During training, two auxiliary networks are attached to the encoder's output:
  3. Singer Classification Network: Attempts to identify the original singer.
  4. Pitch Regression Network: Attempts to predict the pitch from the latent features. The encoder is trained to minimize its own reconstruction loss while maximizing the error of these adversarial networks. This forces the latent representation to be both singer-invariant and pitch-invariant.

  5. Decoder (WaveNet): Since the encoder features are now "clean" of pitch and identity, the decoder is provided with three explicit inputs: the phonetic features, a target singer embedding from a lookup table (LUT), and the original pitch extracted from the source audio.

Key Results

The researchers evaluated PitchNet against a baseline Unsupervised Singing Voice Conversion (USVC) model using the NUS-48E dataset.

Subjective Evaluation

Human listening tests (Mean Opinion Score) showed that PitchNet significantly outperformed the baseline in both naturalness and similarity to the target singer.

Subjective evaluation results comparing PitchNet to baseline USVC models in terms of Naturalness and Similarity.

Objective Evaluation and Pitch Control

PitchNet achieved a Normalized Cross Correlation (NCC) score of 0.855 for conversion, compared to the baseline's 0.821, indicating much higher melodic accuracy.

Furthermore, because pitch is an explicit input to the decoder, PitchNet allows for direct pitch manipulation. By scaling the input pitch (e.g., 0.7x or 1.3x), the model can shift the key of the entire performance without degrading vocal quality or phonetic clarity. This decoupling proves that the adversarial training successfully isolated the melody from the vocal identity.

Conclusion and Future Work

PitchNet demonstrates that adversarial disentanglement is a powerful tool for high-quality singing voice conversion. By treating pitch as a controllable parameter rather than a latent feature, the model avoids the "out of key" pitfalls of previous unsupervised methods.

Future research may focus on improving audio fidelity beyond 8-bit mu-law encoding and exploring faster, non-autoregressive decoders (like GAN-based vocoders) to enable real-time conversion. Additionally, scaling the model to larger, more diverse datasets will be crucial for broader applicability across different musical genres.

Summary updated

Abstract

Singing voice conversion is to convert a singer's voice to another one's voice without changing singing content. Recent work shows that unsupervised singing voice conversion can be achieved with an autoencoder-based approach [1]. However, the converted singing voice can be easily out of key, showing that the existing approach cannot model the pitch information precisely. In this paper, we propose to advance the existing unsupervised singing voice conversion method proposed in [1] to achieve more accurate pitch translation and flexible pitch manipulation. Specifically, the proposed PitchNet added an adversarially trained pitch regression network to enforce the encoder network to learn pitch invariant phoneme representation, and a separate module to feed pitch extracted from the source audio to the decoder network. Our evaluation shows that the proposed method can greatly improve the quality of the converted singing voice (2.92 vs 3.75 in MOS). We also demonstrate that the pitch of converted singing can be easily controlled during generation by changing the levels of the extracted pitch before passing it to the decoder network.
BibTeX
@inproceedings{deng2020pitchnet,
  author = {Chengqi Deng and Chengzhu Yu and Heng Lu and Chao Weng and Dong Yu},
  title = {{PitchNet: Unsupervised Singing Voice Conversion with Pitch Adversarial Network}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  doi = {10.1109/icassp40776.2020.9054199},
}
Figure