Preserving background sound in noise-robust voice conversion via multi-task learning

Jixun Yao, Yi Lei, Qing Wang, Pengcheng Guo, Ziqian Ning, Lei Xie, Hai Li, Junhui Liu, Danming Xie

Abstract

This paper presents an end-to-end framework for "noisy-to-noisy" voice conversion (VC) designed to alter a speaker's identity while maintaining the integrity of the original background audio. Traditional "cascade" methods—which separate, convert, and then remix audio—often suffer from "cascade mismatch" and speech distortion due to error propagation between independent models. By employing a multi-task learning approach that jointly optimizes source separation (SS) and voice conversion, the proposed system eliminates these issues. Experimental results demonstrate that this unified framework significantly improves both subjective speech quality and background sound preservation compared to state-of-the-art cascaded baselines.

Introduction

Most contemporary voice conversion research focuses on "clean" input audio. However, real-world applications such as movie dubbing or podcast editing often involve "noisy" audio where speech is entangled with background music (BGM) or environmental sounds. The standard solution is a cascade approach: first, separate the voice from the background; second, convert the voice; and third, remix the converted voice with the original background.

This approach is fundamentally flawed because source separation models are imperfect. They introduce digital artifacts and phase distortions. When these "cleaned" but distorted voices are processed by a VC model, errors propagate, resulting in unnatural, robotic speech. Furthermore, because the separation and conversion models are trained independently, they lack a shared representation, leading to a performance gap known as cascade mismatch.

Methods

The proposed end-to-end (E2E) multi-task framework integrates three primary components to address these challenges:

  1. Source Separation (SS) Module: Utilizing a DCCRN (Deep Complex Convolution Recurrent Network), the system processes complex-valued spectra. This is crucial for preserving phase information, which prevents the "underwater" sound quality typical of magnitude-only separation models.
  2. Bottleneck Feature (BN) Extraction: A pre-trained, frozen ASR encoder from the WeNet toolkit is used to extract speaker-independent linguistic content.
  3. Voice Conversion (VC) Module: This module consists of a convolutional LSTM encoder and a HiFiGAN-based decoder, which reconstructs the voice using the extracted content and a target speaker ID.
Overview of the proposed multi-task learning framework for voice conversion in noisy environments.

Multi-Task Joint Training

The core innovation lies in the joint optimization of three tasks simultaneously:

  • Source Separation: Optimized via a "Power-law Compressed Phase-Aware Asymmetric" (PLCPA-ASYM) loss to ensure clean separation without over-suppressing speech.
  • Voice Conversion: Uses standard reconstruction and adversarial losses for natural-sounding speech.
  • Unified Reconstruction: A loss function evaluates the final "remixed" audio (converted voice + original background) against the original input, ensuring all modules work in harmony and the VC module learns to be robust to separation artifacts.

Experimental Setup

The model was trained and evaluated using the MUSDB18 dataset for music source separation and the VCTK dataset for speech. Evaluation involved mixing VCTK speech with MUSDB18 background tracks at Signal-to-Noise Ratios (SNR) ranging from 0 to 10 dB to simulate realistic noisy environments.

Key Results

The proposed model was compared against several baselines, including a standard separation-then-conversion cascade ("Sep") and an advanced separation+denoising cascade ("Sep+Denoise").

Subjective evaluation results (CMOS, Quality, and Similarity) comparing the proposed method against baselines.
  • Subjective Quality: In Comparative Mean Opinion Score (CMOS) tests, the proposed model achieved a score of -0.12, where the theoretical "Upper Bound" (converting clean speech then adding noise) is 0. This significantly outperformed the best baseline score of -0.35.
  • Objective Metrics: The model achieved a PESQ score of 3.43, matching specialized denoising models. Crucially, it showed superior background preservation with an SI-SDR of 11.11 dB, compared to 9.85 dB for the baseline separation models.

Conclusion

This research demonstrates that joint training of source separation and voice conversion modules effectively mitigates the cascade mismatch problem. By preserving phase information and optimizing for unified reconstruction, the framework achieves high-fidelity voice conversion in noisy environments. Future work will focus on increasing the sampling rate for high-fidelity applications, fine-tuning the content encoder, and reducing computational complexity for real-time use.

Summary updated

BibTeX
@inproceedings{yao2023preserving,
  author = {Jixun Yao and Yi Lei and Qing Wang and Pengcheng Guo and Ziqian Ning and Lei Xie and Hai Li and Junhui Liu and Danming Xie},
  title = {{Preserving background sound in noise-robust voice conversion via multi-task learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=QgZi5NL35l},
}
Figure