Stepback: Enhanced Disentanglement for Voice Conversion via Multi-Task Learning

Qian Yang, Calbert GrahamarXiv 2025 · DOI · arXiv

Introduction

Voice Conversion (VC) is the process of altering the speaker's identity (timbre) in an audio signal while maintaining the underlying linguistic content. This task is particularly challenging in "non-parallel" scenarios, where the source and target speakers are not uttering the same sentences. The fundamental difficulty lies in disentanglement: isolating the speaker's identity from the phonetic content. Traditional models often suffer from "residual speaker traces" or "oversmoothing," which degrades the naturalness and accuracy of the converted speech.

The Stepback Methodology

The researchers introduce the Stepback network, a novel training strategy designed to force the model to abandon speaker-specific shortcuts in favor of robust linguistic representations. The training follows a two-stage process:

  1. Preparatory Stage: A standard Variational Autoencoder (VAE) is trained to reconstruct speech where the input and target speakers are identical. This phase stabilizes the encoder and decoder components.
  2. The Stepback Stage: This core innovation utilizes a "dual flow" architecture. While one flow continues standard reconstruction to maintain audio quality, the second flow—the "self-destructive" part—uses two identical decoders. One decoder attempts to reconstruct the original speaker, while the other converts the speech to a random target speaker.
The Stepback architecture featuring dual decoders and a speaker classifier to guide disentanglement.

In this stage, the model intentionally maximizes the distance between input and output for the same-speaker flow. By making the encoder "worse" at identifying the original speaker, the system is pressured to extract a more speaker-independent representation of the spoken words.

Technical Architecture

The system is built upon a sophisticated stack of convolutional and recurrent layers. The encoder utilizes a combination of 1D convolution banks, dense blocks with residual connections, and bi-directional GRUs to capture both local features and long-term dependencies. The decoder mirrors this complexity to reconstruct high-fidelity audio from the latent space.

Detailed layer specifications for the Encoder, Decoder/Generator, and Discriminator components.

Experimental Setup and Results

The model was evaluated using the CSTR VCTK Corpus, focusing on a subset of 20 speakers with diverse accents. The training was conducted in a non-parallel setup to simulate real-world data constraints.

Key Findings:

  • Training Efficiency: The Stepback approach achieved convergence in approximately 548,000 mini-batches, representing a 30% reduction in training costs compared to baseline models (such as Chou’s model), which required over 800,000 mini-batches.
  • Linguistic Preservation: Subjective testing indicated superior preservation of intonation, stress, and pitch. By preventing the encoder from "remembering" the speaker's voice, the model focused more effectively on linguistic uniformity.
  • Quality and Stability: The "Stepback" logic acts as a regularizer, preventing the overfitting often seen in more aggressive encoder optimization strategies.

Conclusion and Future Work

The Stepback network demonstrates that strategic "sabotage" during training can lead to more effective feature disentanglement in voice conversion. While the current implementation uses standard VAE/GAN components, future iterations aim to integrate more advanced architectures like ECAPA-TDNN to better characterize speakers from variable-length utterances and further improve the scalability of the efficiency gains.

Summary updated

BibTeX
@misc{yang2025stepback,
  author = {Qian Yang and Calbert Graham},
  title = {{Stepback: Enhanced Disentanglement for Voice Conversion via Multi-Task Learning}},
  year = {2025},
  eprint = {2501.15613},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.15613},
}
Figure